|
|
| (4 intermediate revisions by 3 users not shown) |
| Line 1: |
Line 1: |
| − | | + | {{GlossaryTermTemp |
| | + | |Abbreviation=DB |
| | + | |Ambiguities=Data bank |
| | + | |SubtermOf=Server, Data storage, Data preparation,Data security |
| | + | |Definition=0. A database is an organized collection of data. It is the collection of schemas, tables, queries, reports, views and other objects |
| | + | 1. A comprehensive collection of related data organized for convenient access, generally in a computer. |
| | + | 2. A database is a collection of information that is organized so that it can be easily accessed, managed and updated |
| | + | |Sources=Wikipedia https://en.wikipedia.org/wiki/Database (0.); http://www.dictionary.com/browse/database (1.); http://searchsqlserver.techtarget.com/definition/database (2.) |
| | + | }} |
| | (platform.openmod-initiative.org/data)<br/> | | (platform.openmod-initiative.org/data)<br/> |
| | | | |
| − | '''Finding and downloading data (web)'''<br/> | + | <br/> |
| | + | |
| | + | == Database Setup == |
| | + | |
| | + | The name of the database is OpenEnergyDataBase '''oedb'''. |
| | + | |
| | + | It contains original data (orig), processed data (calc) and results (res). |
| | + | |
| | + | |
| | + | |
| | + | == Database Rules == |
| | + | |
| | + | [[DatabaseRules|Here]]. Work in progess, feel free to discuss. |
| | + | |
| | + | <br/> |
| | + | |
| | + | |
| | + | == Finding and downloading data (web)<br/> == |
| | | | |
| | As before the user finds an overview (like e.g. [https://www.govdata.de/ here]) of data and their sources for different kind of data (replacing the [http://wiki._openmod-initiative.org/wiki/Data wiki page Data]) and an overview of data and their sources for grid data (replacing the [[Transmission network datasets|wiki page Transmission network datasets]]) including data models (sometimes also called grid models). The latter are understood as collections of all relevant information about a grid / network, separated from the tools for network calculations / analysis. ''Are theses tools models, that are presented in fact sheets?''<br/> | | As before the user finds an overview (like e.g. [https://www.govdata.de/ here]) of data and their sources for different kind of data (replacing the [http://wiki._openmod-initiative.org/wiki/Data wiki page Data]) and an overview of data and their sources for grid data (replacing the [[Transmission network datasets|wiki page Transmission network datasets]]) including data models (sometimes also called grid models). The latter are understood as collections of all relevant information about a grid / network, separated from the tools for network calculations / analysis. ''Are theses tools models, that are presented in fact sheets?''<br/> |
| Line 15: |
Line 40: |
| | *evtl. feedin data (?)<br/> | | *evtl. feedin data (?)<br/> |
| | | | |
| − | In addition, the user finds two schemata for every model (called app) in the working database (wdb) on the OpenEnergy Platform, which the model developers create and fill with model specific content. | + | In addition, the user finds two schemata for every model (called app) <s>in the working database (wdb)</s> on the OpenEnergy Platform, which the model developers create and fill with model specific content. |
| | | | |
| | The user searches for certain data (using the data tags named above), assumptions or results (using provided tags like "scenario input" or "scenario results" etc.). She/he uses also the data view functions, which give an overview of the data itself in form of a table or in form of a map or graphs. Doing so, the user might limit the data to a certain geographical area using the map provided for the selection. The user might also download the (selected) data as a text, csv or json file.<br/> | | The user searches for certain data (using the data tags named above), assumptions or results (using provided tags like "scenario input" or "scenario results" etc.). She/he uses also the data view functions, which give an overview of the data itself in form of a table or in form of a map or graphs. Doing so, the user might limit the data to a certain geographical area using the map provided for the selection. The user might also download the (selected) data as a text, csv or json file.<br/> |
| | | | |
| − | <u>'''Implementation:'''</u>
| + | === Implementation: === |
| | | | |
| | Assuming that the openmod database includes or links to all relevant data the lists on the wiki pages are no longer necessary. On overview pages the database entries might be ranked by access frequency in order to reduce the overwhelming effect of the database. | | Assuming that the openmod database includes or links to all relevant data the lists on the wiki pages are no longer necessary. On overview pages the database entries might be ranked by access frequency in order to reduce the overwhelming effect of the database. |
| Line 25: |
Line 50: |
| | On top of the PostgreSQL database CKAN with a faceted search will be installed.<br/> | | On top of the PostgreSQL database CKAN with a faceted search will be installed.<br/> |
| | | | |
| − | '''Creating model specific schemata'''<br/>
| + | |
| | + | == Creating model specific schemata<br/> == |
| | | | |
| | In addition to the activities of non-modellers, modellers of a model can make model specific configurations and store model specific data and assumptions in the xy_app_wd schema and results in the xy_app_res schema. | | In addition to the activities of non-modellers, modellers of a model can make model specific configurations and store model specific data and assumptions in the xy_app_wd schema and results in the xy_app_res schema. |
| | | | |
| − | '''Downloading data (API)'''
| + | == Downloading data (API) == |
| | | | |
| | Modellers can use the API for data exchange between the databases (oedb and wdb) and the model. Models written in different languages can communicate with the database using a function or module that generates and sends http queries like for instance: “GET openmod.org/data/oedb/schema/table?fields=id,name,date”. Python model might for instance use a function based url lib. | | Modellers can use the API for data exchange between the databases (oedb and wdb) and the model. Models written in different languages can communicate with the database using a function or module that generates and sends http queries like for instance: “GET openmod.org/data/oedb/schema/table?fields=id,name,date”. Python model might for instance use a function based url lib. |
| | | | |
| − | '''Uploading data (API and Web-GUI)'''
| + | == Uploading data (API and Web-GUI) == |
| | | | |
| | A user can add new rows in the tables of the database as well as new tables. To upload the data she/he can use a function / module, sending PUT queries in http to the database API (see also “Downloading data (API)”), or the data importer (GUI) on the online presence. In the first step the importer can cope with data in csv fomat. The user can submit data as a batch and edit (or delete) the tables manually before she/he sends them. When the user accidentally leave the input site, while data is still pending, she/he will be warned and can cancel the exit. | | A user can add new rows in the tables of the database as well as new tables. To upload the data she/he can use a function / module, sending PUT queries in http to the database API (see also “Downloading data (API)”), or the data importer (GUI) on the online presence. In the first step the importer can cope with data in csv fomat. The user can submit data as a batch and edit (or delete) the tables manually before she/he sends them. When the user accidentally leave the input site, while data is still pending, she/he will be warned and can cancel the exit. |
| Line 39: |
Line 65: |
| | Before theses changes will be available in the oedb, they are saved in a test databse and must pass tests, which a test server runs regularly. Data, that is made available by OpenPowerSystenData (OPSD), will in the same way be checked and saved in the database. | | Before theses changes will be available in the oedb, they are saved in a test databse and must pass tests, which a test server runs regularly. Data, that is made available by OpenPowerSystenData (OPSD), will in the same way be checked and saved in the database. |
| | | | |
| − | '''Changing data'''
| + | == Changing data == |
| | | | |
| | The documentation of changes of the data in the database means saving an enormous amount of information. Therefore, changing the data in the database must be organised very well. | | The documentation of changes of the data in the database means saving an enormous amount of information. Therefore, changing the data in the database must be organised very well. |
| | | | |
| − | <u>'''Implementation:'''</u>
| + | == Implementation: == |
| | | | |
| | The API and the data importer will be implemented following the RESTFul paradigms. | | The API and the data importer will be implemented following the RESTFul paradigms. |
| | | | |
| − | Nightly tests check, whether the new data fulfil the requirements and work with the models and delivers plausible results.<br/> | + | Nightly tests check, whether the new data fulfil the requirements and work with the models and deliver plausible results.<br/> |
Latest revision as of 11:59, 6 November 2017
[edit] Definition
0. A database is an organized collection of data. It is the collection of schemas, tables, queries, reports, views and other objects
1. A comprehensive collection of related data organized for convenient access, generally in a computer.
2. A database is a collection of information that is organized so that it can be easily accessed, managed and updated
[edit] Abbreviation
DB
[edit] Synonyms
Data bank
[edit] Superterms
Server, Data storage, Data preparation, Data security
[edit] Subterms
Database management system, Database schema, DatabaseRules, Master Salve
[edit] Sources
Wikipedia https://en.wikipedia.org/wiki/Database (0.); http://www.dictionary.com/browse/database (1.); http://searchsqlserver.techtarget.com/definition/database (2.)
(platform.openmod-initiative.org/data)
[edit] Database Setup
The name of the database is OpenEnergyDataBase oedb.
It contains original data (orig), processed data (calc) and results (res).
[edit] Database Rules
Here. Work in progess, feel free to discuss.
[edit] Finding and downloading data (web)
As before the user finds an overview (like e.g. here) of data and their sources for different kind of data (replacing the wiki page Data) and an overview of data and their sources for grid data (replacing the wiki page Transmission network datasets) including data models (sometimes also called grid models). The latter are understood as collections of all relevant information about a grid / network, separated from the tools for network calculations / analysis. Are theses tools models, that are presented in fact sheets?
The OpenEnergy Platform provides additionally a database, which links to and if possible includes data, which are of general interest for energy system modellers. This database is called oedb (open energy database) and includes original open data collected from different sources as well as processed data. The datasets might be grouped / tagged as follows:
- climate data
- existing power stations (Kraftwerksliste and EEG-Anlagenregister: installed capacities and efficiencies)
- demand data / energy consumption
- transmission network
- flexibility options
- evtl. feedin data (?)
In addition, the user finds two schemata for every model (called app) in the working database (wdb) on the OpenEnergy Platform, which the model developers create and fill with model specific content.
The user searches for certain data (using the data tags named above), assumptions or results (using provided tags like "scenario input" or "scenario results" etc.). She/he uses also the data view functions, which give an overview of the data itself in form of a table or in form of a map or graphs. Doing so, the user might limit the data to a certain geographical area using the map provided for the selection. The user might also download the (selected) data as a text, csv or json file.
[edit] Implementation:
Assuming that the openmod database includes or links to all relevant data the lists on the wiki pages are no longer necessary. On overview pages the database entries might be ranked by access frequency in order to reduce the overwhelming effect of the database.
On top of the PostgreSQL database CKAN with a faceted search will be installed.
[edit] Creating model specific schemata
In addition to the activities of non-modellers, modellers of a model can make model specific configurations and store model specific data and assumptions in the xy_app_wd schema and results in the xy_app_res schema.
[edit] Downloading data (API)
Modellers can use the API for data exchange between the databases (oedb and wdb) and the model. Models written in different languages can communicate with the database using a function or module that generates and sends http queries like for instance: “GET openmod.org/data/oedb/schema/table?fields=id,name,date”. Python model might for instance use a function based url lib.
[edit] Uploading data (API and Web-GUI)
A user can add new rows in the tables of the database as well as new tables. To upload the data she/he can use a function / module, sending PUT queries in http to the database API (see also “Downloading data (API)”), or the data importer (GUI) on the online presence. In the first step the importer can cope with data in csv fomat. The user can submit data as a batch and edit (or delete) the tables manually before she/he sends them. When the user accidentally leave the input site, while data is still pending, she/he will be warned and can cancel the exit.
Before theses changes will be available in the oedb, they are saved in a test databse and must pass tests, which a test server runs regularly. Data, that is made available by OpenPowerSystenData (OPSD), will in the same way be checked and saved in the database.
[edit] Changing data
The documentation of changes of the data in the database means saving an enormous amount of information. Therefore, changing the data in the database must be organised very well.
[edit] Implementation:
The API and the data importer will be implemented following the RESTFul paradigms.
Nightly tests check, whether the new data fulfil the requirements and work with the models and deliver plausible results.
