Data Documentation
All data included in the databases has be documented!
All abreviations have to be documentated in the Glossary!
Naming of Data
Database Name
The name of the database is oedb
Database Schema
The structure of the database is realised via the naming of the schema!
- always lower case
- no points, no commas
- no spaces
- no dates
- use underscores
- name starts with type of schema
- orig for original data
- calc for processed data
- name includes distinct subject area or source
Example: orig_vg250
Database Table
- always lower case
- no points, no commas
- no spaces
- no dates
- use underscores
- name starts with the source (e.g. zensus)
- main value (e.g. population)
- if separated by [attribute] (e.g. by_gender)
- with resolution [tupel] (e.g. per_mun)
Example: zensus_population_by_gender_per_mun
Data Integrity
General
- Primary Key [PK]
- Grants to oeuser
Geografic Data
The CRS is defined
Data Referencing
Original Data (orig)
Tables are annotated by a comment in form of a json string:
{"Name": "Wirtschaftsdaten pro Region",
"Source": ["Regionaldatenbank Deutschland", "www.regionalstatistik.de / registation required"],
"Reference date": ["2013"],
"Retrieved": ["04.06.2014"],
"Date of collection": ["01.08.2013"],
"Original file": ["346-22-5.xls"],
"Spatial resolution": ["Germany"],
"Description": ["Financial key figures of German municipalities (annual totals)", "Regional level: municipalities, association of municipalities"],
"Table fields": [
{"name":"id",
"description"Unique identifier"",
"description_german":"",
"unit":"" },
{"name":"year",
"description"Reference Year"",
"description_german":"",
"unit":"" },
{"name":"example_value",
"description"Some important value"",
"description_german":"",
"unit":"EUR" }],
"Changes":[
{ "name":"Wolf-Dieter Bunke",
"mail":"wd.bunke@gmail.com",
"date":"16.06.2014",
"comment":"Created table" },
{ "name":"Cord Kaldemeyer",
"mail":"cord.kaldemeyer@fh-flensburg.de",
"date":"17.07.2014",
"comment":"Translated field names"}],
"ToDo": ["Some datasets are odd -> Check numbers against another data"],
"Licence": ["Datenlizenz Deutschland – Namensnennung – Version 2.0 (dl-de/by-2-0; [http://www.govdata.de/dl-de/by-2-0 http://www.govdata.de/dl-de/by-2-0])"],
"Instructions for proper use": ["Always state licence"]}
Processed Data (calc)
{"Name": "Results",
"Date of collection": ["01.08.2013"],
"Spatial resolution": ["Germany"],
"Description": ["Financial key figures of German municipalities (annual totals)", "Regional level: municipalities, association of municipalities"],
"Table fields": [
{"name":"id",
"description"Unique identifier"",
"description_german":"",
"unit":"" },
{"name":"year",
"description"Reference Year"",
"description_german":"",
"unit":"" },
{"name":"example_value",
"description"Some important value"",
"description_german":"",
"unit":"EUR" }],
"Changes":[
{ "name":"Wolf-Dieter Bunke",
"mail":"wd.bunke@gmail.com",
"date":"16.06.2014",
"comment":"Created table" },
{ "name":"Cord Kaldemeyer",
"mail":"cord.kaldemeyer@fh-flensburg.de",
"date":"17.07.2014",
"comment":"Translated field names"}],
"ToDo": ["Some datasets are odd -> Check numbers against another data"],
"Licence": ["Datenlizenz Deutschland – Namensnennung – Version 2.0 (dl-de/by-2-0; http://www.govdata.de/dl-de/by-2-0)"],
"Instructions for proper use": ["Always state licence"]}
Processed Data (calc) - Row Annotation
Each row has to be annotated by a json dictionary that must contain the following fields:
- origin: Link or textual description of the data set this row origins from.
- method: Method used to calculate this row from above origin (e.g. Link to a python script)
- assumption: A list of dictionaries. Each dictionary describes an assumption and annotates the affected rows.
- begin: First column affected by the assumption
- end: Last column affected by the assumption
- type: Type of the problem that had to be solved. Each type requires one or more additional keys in this dictionary. Possible types and their required additional keys are:
- gap: A not all fields could be calculated and/or filled,
- solution: Method that was used to generate date to fill this gap (e.g. linear interpolation)
- multiplicity: A field could be filled by several values
- values: Possible Values that could habe been used
- solution: Method that was used to select one value (e.g. Minimum)
An examplatory dictionary:
{
"origin":"https://data.openmod-initiative.org/data/oedb/orig_db/table
"method":"https://github.com/openego/data_processing/blob/master/calc_ego_substation/Voronoi_ehv.sql"
"assumptions":
[
{
"type": "gap"
"begin": "step_15"
"end": "step_34"
"solution": "linear_interpolation"
}
]
}
