|
|
| Line 151: |
Line 151: |
| | </pre> | | </pre> |
| | <br/> | | <br/> |
| | + | |
| | | | |
| | ==== Processed Data (calc) - Row Annotation ==== | | ==== Processed Data (calc) - Row Annotation ==== |
| Line 165: |
Line 166: |
| | ****solution: Method that was used to generate date to fill this gap (e.g. linear interpolation) | | ****solution: Method that was used to generate date to fill this gap (e.g. linear interpolation) |
| | | | |
| − | *
| |
| − | *
| |
| | *multiplicity: A field could be filled by several values | | *multiplicity: A field could be filled by several values |
| − | ** | + | **values: Possible Values that could have been used |
| − | *
| + | ***solution: Method that was used to select one value (e.g. Minimum) |
| − | *values: Possible Values that could habe been used | + | |
| − | ****solution: Method that was used to select one value (e.g. Minimum)
| + | |
| | | | |
| | <br/>An examplatory dictionary:<br/><pre> | | <br/>An examplatory dictionary:<br/><pre> |
Revision as of 06:35, 10 June 2016
Data Documentation
All data included in the databases has be documented!
All abreviations have to be documentated in the Glossary!
Naming of Data
Database Name
The name of the database is oedb
Database Schema
The structure of the database is realised via the naming of the schema!
- always lower case
- no points, no commas
- no spaces
- no dates
- use underscores
- name starts with type of schema
- orig for original data
- calc for processed data
- name includes distinct subject area or source
Example: orig_vg250
Database Table
- always lower case
- no points, no commas
- no spaces
- no dates
- use underscores
- name starts with the source (e.g. zensus)
- main value (e.g. population)
- if separated by [attribute] (e.g. by_gender)
- with resolution [tupel] (e.g. per_mun)
Example: zensus_population_by_gender_per_mun
Data Integrity
General
- Primary Key [PK]
- Grants to oeuser
Geografic Data
- WGS84 - EPSG: 4326 (http://spatialreference.org/ref/epsg/4326/)
- ETRS89 / ETRS-LAEA - EPSG: 3035 (http://spatialreference.org/ref/epsg/3035/)
Data Referencing
Original Data (orig)
Tables are annotated by a comment in form of a json string:
{"Name": "The Full Name",
"Source": ["Name", "www.website.com / registation required"],
"Reference date": ["2013"],
"Date of collection": ["01.08.2013"],
"Original file": ["346-22-5.xls"],
"Spatial resolution": ["Germany"],
"Description": ["Example Data (annual totals)", "Regional level: national"],
"Table fields": [
{"name":"id",
"description"Unique identifier"",
"description_german":"",
"unit":"" },
{"name":"year",
"description"Reference Year"",
"description_german":"",
"unit":"" },
{"name":"example_value",
"description"Some important value"",
"description_german":"",
"unit":"EUR" }],
"Changes":[
{ "name":"Joe Nobody",
"mail":"joe.nobody@gmail.com (fake)",
"date":"16.06.2014",
"comment":"Created table" },
{ "name":"Joana Anybody",
"mail":"joana.anybody@gmail.com (fake)",
"date":"17.07.2014",
"comment":"Translated field names"}],
"ToDo": ["Some datasets are odd -> Check numbers against another data"],
"Licence": ["Licence – Version 2.0 (dl-de/by-2-0; [http://www.govdata.de/dl-de/by-2-0])"],
"Instructions for proper use": ["Always state licence"]}
Processed Data (calc)
{"Name": "Results",<br/>"Date of collection": ["01.08.2013"],<br/>"Spatial resolution": ["Germany"],<br/>"Description": ["Financial key figures of German municipalities (annual totals)", "Regional level: municipalities, association of municipalities"],<br/>
"Table fields": [<br/>{"name":"id",<br/>"description"Unique identifier"",<br/>"description_german":"",<br/>"unit":"" },
{"name":"year",<br/>"description"Reference Year"",<br/>"description_german":"",<br/>"unit":"" },
{"name":"example_value",<br/>"description"Some important value"",<br/>"description_german":"",<br/>"unit":"EUR" }],
"Changes":[<br/> { "name":"Autor1",<br/> "mail":"Autor1@e-mail.com",<br/> "date":"16.06.2014",<br/> "comment":"Created table" },
{ "name":"Autor2",<br/> "mail":"Autor2@e-mail.com",<br/> "date":"17.07.2014",<br/> "comment":"Translated field names"}],
"ToDo": ["Some datasets are odd -> Check numbers against another data"],<br/>"Licence": ["Datenlizenz Deutschland – Namensnennung – Version 2.0 (dl-de/by-2-0; [http://www.govdata.de/dl-de/by-2-0 http://www.govdata.de/dl-de/by-2-0])"],<br/>"Instructions for proper use": ["Always state licence"]}<br/>
Processed Data (calc) - Row Annotation
Each row has to be annotated by a json dictionary that must contain the following fields:
- origin: Link or textual description of the data set this row origins from.
- method: Method used to calculate this row from above origin (e.g. Link to a python script)
- assumption: A list of dictionaries. Each dictionary describes an assumption and annotates the affected rows.
- begin: First column affected by the assumption
- end: Last column affected by the assumption
- type: Type of the problem that had to be solved. Each type requires one or more additional keys in this dictionary. Possible types and their required additional keys are:
- gap: A not all fields could be calculated and/or filled,
- solution: Method that was used to generate date to fill this gap (e.g. linear interpolation)
- multiplicity: A field could be filled by several values
- values: Possible Values that could have been used
- solution: Method that was used to select one value (e.g. Minimum)
An examplatory dictionary:
{
"origin":"https://data.openmod-initiative.org/data/oedb/orig_db/table
"method":"https://github.com/openego/data_processing/blob/master/calc_ego_substation/Voronoi_ehv.sql"
"assumptions":
[
{
"type": "gap"
"begin": "step_15"
"end": "step_34"
"solution": "linear_interpolation"
}
]
}
