|
|
| Line 83: |
Line 83: |
| | | | |
| | = Data Referencing = | | = Data Referencing = |
| | + | |
| | | | |
| | === Original Data (orig)<br/> === | | === Original Data (orig)<br/> === |
| Line 88: |
Line 89: |
| | Tables are annotated by a comment in form of a json string:<br/> | | Tables are annotated by a comment in form of a json string:<br/> |
| | | | |
| − | <pre>{"Name": "Wirtschaftsdaten pro Region", | + | <pre>{"Name": "The Full Name", |
| − | "Source": ["Regionaldatenbank Deutschland", "www.regionalstatistik.de / registation required"], | + | "Source": ["Name", "www.website.com / registation required"], |
| | "Reference date": ["2013"], | | "Reference date": ["2013"], |
| | "Retrieved": ["04.06.2014"], | | "Retrieved": ["04.06.2014"], |
| Line 114: |
Line 115: |
| | | | |
| | "Changes":[ | | "Changes":[ |
| − | { "name":"Wolf-Dieter Bunke", | + | { "name":"Joe Nobody", |
| − | "mail":"wd.bunke@gmail.com", | + | "mail":"joe.nobody@gmail.com (fake)", |
| | "date":"16.06.2014", | | "date":"16.06.2014", |
| | "comment":"Created table" }, | | "comment":"Created table" }, |
| | | | |
| − | { "name":"Cord Kaldemeyer", | + | { "name":"Joana Anybody", |
| − | "mail":"cord.kaldemeyer@fh-flensburg.de", | + | "mail":"joana.anybody@gmail.com (fake)", |
| | "date":"17.07.2014", | | "date":"17.07.2014", |
| | "comment":"Translated field names"}], | | "comment":"Translated field names"}], |
| | | | |
| | "ToDo": ["Some datasets are odd -> Check numbers against another data"], | | "ToDo": ["Some datasets are odd -> Check numbers against another data"], |
| − | "Licence": ["Datenlizenz Deutschland – Namensnennung – Version 2.0 (dl-de/by-2-0; [http://www.govdata.de/dl-de/by-2-0 http://www.govdata.de/dl-de/by-2-0])"], | + | "Licence": ["Datenlizenz Deutschland – Namensnennung – Version 2.0 (dl-de/by-2-0; [http://www.govdata.de/dl-de/by-2-0])"], |
| | "Instructions for proper use": ["Always state licence"]}</pre><br/> | | "Instructions for proper use": ["Always state licence"]}</pre><br/> |
| | | | |
| | <br/> | | <br/> |
| | + | |
| | | | |
| | === Processed Data (calc)<br/> === | | === Processed Data (calc)<br/> === |
Revision as of 10:15, 27 May 2016
Data Documentation
All data included in the databases has be documented!
All abreviations have to be documentated in the Glossary!
Naming of Data
Database Name
The name of the database is oedb
Database Schema
The structure of the database is realised via the naming of the schema!
- always lower case
- no points, no commas
- no spaces
- no dates
- use underscores
- name starts with type of schema
- orig for original data
- calc for processed data
- name includes distinct subject area or source
Example: orig_vg250
Database Table
- always lower case
- no points, no commas
- no spaces
- no dates
- use underscores
- name starts with the source (e.g. zensus)
- main value (e.g. population)
- if separated by [attribute] (e.g. by_gender)
- with resolution [tupel] (e.g. per_mun)
Example: zensus_population_by_gender_per_mun
Data Integrity
General
- Primary Key [PK]
- Grants to oeuser
Geografic Data
- WGS84 - EPSG: 4326 (http://spatialreference.org/ref/epsg/4326/)
- ETRS89 / ETRS-LAEA - EPSG: 3035 (http://spatialreference.org/ref/epsg/3035/)
Data Referencing
Original Data (orig)
Tables are annotated by a comment in form of a json string:
{"Name": "The Full Name",
"Source": ["Name", "www.website.com / registation required"],
"Reference date": ["2013"],
"Retrieved": ["04.06.2014"],
"Date of collection": ["01.08.2013"],
"Original file": ["346-22-5.xls"],
"Spatial resolution": ["Germany"],
"Description": ["Financial key figures of German municipalities (annual totals)", "Regional level: municipalities, association of municipalities"],
"Table fields": [
{"name":"id",
"description"Unique identifier"",
"description_german":"",
"unit":"" },
{"name":"year",
"description"Reference Year"",
"description_german":"",
"unit":"" },
{"name":"example_value",
"description"Some important value"",
"description_german":"",
"unit":"EUR" }],
"Changes":[
{ "name":"Joe Nobody",
"mail":"joe.nobody@gmail.com (fake)",
"date":"16.06.2014",
"comment":"Created table" },
{ "name":"Joana Anybody",
"mail":"joana.anybody@gmail.com (fake)",
"date":"17.07.2014",
"comment":"Translated field names"}],
"ToDo": ["Some datasets are odd -> Check numbers against another data"],
"Licence": ["Datenlizenz Deutschland – Namensnennung – Version 2.0 (dl-de/by-2-0; [http://www.govdata.de/dl-de/by-2-0])"],
"Instructions for proper use": ["Always state licence"]}
Processed Data (calc)
{"Name": "Results",
"Date of collection": ["01.08.2013"],
"Spatial resolution": ["Germany"],
"Description": ["Financial key figures of German municipalities (annual totals)", "Regional level: municipalities, association of municipalities"],
"Table fields": [
{"name":"id",
"description"Unique identifier"",
"description_german":"",
"unit":"" },
{"name":"year",
"description"Reference Year"",
"description_german":"",
"unit":"" },
{"name":"example_value",
"description"Some important value"",
"description_german":"",
"unit":"EUR" }],
"Changes":[
{ "name":"Autor1",
"mail":"Autor1@e-mail.com",
"date":"16.06.2014",
"comment":"Created table" },
{ "name":"Autor2",
"mail":"Autor2@e-mail.com",
"date":"17.07.2014",
"comment":"Translated field names"}],
"ToDo": ["Some datasets are odd -> Check numbers against another data"],
"Licence": ["Datenlizenz Deutschland – Namensnennung – Version 2.0 (dl-de/by-2-0; http://www.govdata.de/dl-de/by-2-0)"],
"Instructions for proper use": ["Always state licence"]}
Processed Data (calc) - Row Annotation
Each row has to be annotated by a json dictionary that must contain the following fields:
- origin: Link or textual description of the data set this row origins from.
- method: Method used to calculate this row from above origin (e.g. Link to a python script)
- assumption: A list of dictionaries. Each dictionary describes an assumption and annotates the affected rows.
- begin: First column affected by the assumption
- end: Last column affected by the assumption
- type: Type of the problem that had to be solved. Each type requires one or more additional keys in this dictionary. Possible types and their required additional keys are:
- gap: A not all fields could be calculated and/or filled,
- solution: Method that was used to generate date to fill this gap (e.g. linear interpolation)
-
-
- multiplicity: A field could be filled by several values
- values: Possible Values that could habe been used
- solution: Method that was used to select one value (e.g. Minimum)
An examplatory dictionary:
{
"origin":"https://data.openmod-initiative.org/data/oedb/orig_db/table
"method":"https://github.com/openego/data_processing/blob/master/calc_ego_substation/Voronoi_ehv.sql"
"assumptions":
[
{
"type": "gap"
"begin": "step_15"
"end": "step_34"
"solution": "linear_interpolation"
}
]
}
