Difference between revisions of "GCat Background"

Revision as of 15:46, 17 October 2016

** THIS PAGE IS UNDER CONSTRUCTION **

gCube Data Catalogue

D4Science offers services for seamless access and analysis to a wide spectrum of data including biological and ecological data, geospatial data, statistical data and semi-structured data from multiple authoritative data providers and information systems. These services can be exploited both via web based graphical user interfaces and web based protocols for programmatic access, e.g. OAI-PMH, CSW, WFS, SDMX. This offering nicely complements specific and community-specific applications. The gCube Data Catalogue catalogue contains a wealth of resources resulting from several activities, projects and communities including BlueBRIDGE (www.bluebridge-vres.eu/), i-Marine (www.i-marine.eu), SoBigData.eu (www.sobigdata.eu), and FAO (www.fao.org). All the products are accompanied with rich descriptions capturing general attributes, e.g. title and creator(s), as well as usage policies and licences.

The gCube Data Catalogue is builded using and extending CKAN platform. CKAN is a powerful DMS (data management system) that makes data accessible – by providing tools to streamline publishing, sharing, finding and using data. CKAN is an open-source DMS for powering data hubs and data portals. CKAN makes it easy to publish, share and use data see: http://ckan.org/

CKAN model is made by the following entities (and their relations):

CKAN: 'Entities and Relations'

Metadata

A Metadata in the gCube Data Catalogue is made by two parts: CKAN's default metadata fields and gCube Metadata Profile.

CKAN's default metadata fields

Those are metadata fields common for all metadata types in the gCube Data Catalogue (and used by default in the CKAN platform).

Label	Field Name (API)	Definition	Guidelines	Example
Title*	title	Name given to the dataset.	Short phrase, written in plain language. Should be sufficiently descriptive to allow for search and discovery.	Aquaculture Production and Consumption in Africa (2011)
Description	description	Short description explaining the content and its origins.	Description of a few sentences, written in plain language. Should,provide a sufficiently comprehensive overview of the resource for anyone,to understand its content, origins, and any continuing work on it. The,description can be written at the end, since it summarizes key,information from the other metadata fields.	This dataset contains attributes of aquaculture production and,consumption for each of Africa’s provinces in 2011. The data was,provided by………
Tags	tags	An array of Taxonomic terms stored as tags	Taxonomic terms	Access to education, Bamboo
License*	lincese_title	the license that applies to published dataset.
Organization*	organization	Organization the datasets belongs to	See list of organizations on https://ckan-d-d4s.d4science.org/organization	D4Science
Version	version	Version of dataset	Increase manually after editing	1.0
Author*		Owner of dataset	The person who created the dataset in the format: Surname, Name	Bloggs, Joe
Author Contact		Contact details of owner	The email or other contact details of the person who created the dataset.	joe@example.com
Mantainer		Mantainer of the dataset	The person or the authority that maintains the dataset	A person: Bloggs, Joe. An authority: D4Science
Mantainer Contact		Contact details of mantainer	The email or other contact details of the person who maintains the dataset.	joe@example.com

mandatory fields are marked with an asterisk (*)

gCube Metadata Profile

gCube Metadata Profile defines a Metadata schema XML-based for adding custom metadata fields.

A gCube Metadata Profile is composed by one Metadata Format (<metadataformat>) that contains one or many (<metadatafield>). The schema is shown below.

Metadata Profile v.2

Relased in gCube 4.1. What's new:

Added datatype field (<datatype>). A valid datatype must be equal to one of the values {String, Time, Time_Interval, Times_ListOf, Text, Boolean, Number}. When data type is not specified the metadata field has default value as "String". DataType values:
- String: is a string;
- Time: an instant time that follows the general format: YYYY-MM-DD [HH:MM] where: YYYY: 4-digit year, MM: 2-digit month, DD: 2-digit day, [optional HH: 2-digit hour], [optional MM: 2-digit minute] (e.g. "2005-03-01");
- Time_Interval: a continuous interval instead of a single instant by specifying a start and end time, separated by one '/' ('slash') character (e.g. "2005-03-01/2006-05-11");
- Times_ListOf: a list of discrete time values, separated by a ',' ('comma') character (e.g. "2005-03-01, 2006-05-11, 2006-05-11-2007-04-12");
- Text: is a text;
- Boolean: is True/False;
- Number: is a valid Java number, see: Apache Commons NumberUtils.isNumber.

Added multi selection attribute ('isMultiSelection=true|false') to vocabulary.

<?xml version="1.0" encoding="UTF-8"?>
<metadataformat>
	<metadatafield>
		<fieldName>Name</fieldName>
		<mandatory>true</mandatory>
		<dataType>String|Time|Time_Interval|Times_ListOf|Text|Boolean|Number</dataType>
		<defaultValue>default value</defaultValue>
		<note>shown as suggestions in the insert/update metadata form of CKAN
		</note>
		<vocabulary isMultiSelection="true|false">
			<vocabularyField>field1</vocabularyField>
			<vocabularyField>field2</vocabularyField>
			<vocabularyField>field3</vocabularyField>
		</vocabulary>
		<validator>
			<regularExpression>a regular expression for validating values
			</regularExpression>
		</validator>
	</metadatafield>
</metadataformat>

It's possible to validate a Metadata Format schema using following DTD v2

<?xml version="1.0" encoding="UTF-8"?>
<!ELEMENT metadataformat (metadatafield+)>
<!ELEMENT metadatafield (fieldName, mandatory, datatype?, defaultValue?, note?, vocabulary?, validator?)>
<!ELEMENT fieldName (#PCDATA)>
<!ELEMENT mandatory (#PCDATA)>
<!ELEMENT dataType (#PCDATA)>  
<!ELEMENT defaultValue (#PCDATA)> 
<!ELEMENT note (#PCDATA)> 
<!ELEMENT vocabulary (vocabularyField+)> 
<!ATTLIST vocabulary isMultiSelection (true|false) "false">
<!ELEMENT vocabularyField (#PCDATA)> 
<!ELEMENT validator (regularExpression)> 
<!ELEMENT regularExpression (#PCDATA)> 

<!--
Where datatype element is the enum: {String, Time, Time_Interval, Times_ListOf, Text, Boolean, Number}
Its xml schema:
<xs:element name="dataType">
  <xs:simpleType>
    <xs:restriction base="xs:string">
      <xs:enumeration value="String"/>
      <xs:enumeration value="Time"/>
      <xs:enumeration value="Time_Interval"/>
      <xs:enumeration value="Times_ListOf"/>
      <xs:enumeration value="Text"/>
      <xs:enumeration value="Boolean"/>
      <xs:enumeration value="Number"/>
    </xs:restriction>
  </xs:simpleType>
</xs:element> 
-->

Metadata Profile v.1

Relased in gCube 4.0. The schema is:

<?xml version="1.0" encoding="UTF-8">
<metadataformat>
    <metadatafield>
        <fieldName>Name</fieldName>
        <mandatory>true</mandatory>
        <isBoolean>false</isBoolean>
        <defaultValue>default value</defaultValue>
        <note>shown as suggestions in the insert/update metadata form of CKAN</note>
        <vocabulary>
            <vocabularyField>field1</vocabularyField>
            <vocabularyField>field2</vocabularyField>
            <!-- ... others vocabulary fields -->
        </vocabulary>
        <validator>
            <regularExpression>a regular expression for validating values</regularExpression>
        </validator>
    </metadatafield>
     <!-- ... others metadata fields -->
</metadataformat>

It's possible to validate a Metadata Format schema using following DTD v1


<?xml version="1.0" encoding="UTF-8"?>
<!ELEMENT metadataformat (metadatafield+)>
<!ELEMENT metadatafield (fieldName, mandatory, isBoolean?, defaultValue?, note?, vocabulary?, validator?)>
<!ELEMENT fieldName (#PCDATA)>
<!ELEMENT mandatory (#PCDATA)>
<!ELEMENT isBoolean (#PCDATA)>  <!-- MUST BE (true|false) -->
<!ELEMENT defaultValue (#PCDATA)> 
<!ELEMENT note (#PCDATA)> 
<!ELEMENT vocabulary (vocabularyField+)> 
<!ELEMENT vocabularyField (#PCDATA)> 
<!ELEMENT validator (regularExpression)> 
<!ELEMENT regularExpression (#PCDATA)>

A possible instance of Metadata Field (<metadatafield>):

<metadatafield>
   <fieldName>Accessibility</fieldName>
   <mandatory>true</mandatory>
   <defaultValue>virtual/public</defaultValue>
   <vocabulary>
       <vocabularyField>virtual/public</vocabularyField>
       <vocabularyField>virtual/private</vocabularyField>
       <vocabularyField>transactional</vocabularyField>
   </vocabulary>
</metadatafield>

Instances of Metadata Profile

The following list shows the instances of metadata profile created in the D4Sciece infrastructure and currently used in gCube Data Catalogue:

SoBigData.eu

SoBigData.eu: Metadata Profile for gCube Data Catalogue

Ckan Connector

TODO @Lucio

Geo Harvesting

This extension contains plugins like ckanext-geonetwork (and others) which add geospatial capabilities to CKAN.

Several harvesters to import geospatial metadata (like ISO 19139 format) into CKAN from other sources have been created in gCube Data Catalogue. In particular all metadata created into gCube Geonetwork (GeoNetwork is the catalog application to manage spatially referenced resources generated into D4Science Infrastructure) are harvested through the 'Geonetwork Resolver' a "middle tier" able to:

use the Geonetwork Manager in order to harvest private metadata (via authentication) stored in gCube Geonetwork on CKAN Data Catalogue (e.g. http://data-d.d4science.org/geonetwork/gcube_devsec_devVRE to harvest private metadata generated from scope /gcube/devsec/devVRE);

create a CKAN Harvester that skip all public metadata via configuration during scope harvesting (e.g. http://data-d.d4science.org/geonetwork/gcube_devsec_devVRE%23filterpublicids to filter public ids during harvesting of /gcube/devsec/devVRE);

create a CKAN Harvester to harvest only public metadata (saved on Geonetwork) avoiding the Geonetwork authentication via configuration (e.g. http://data-d.d4science.org/geonetwork/gcube_devsec_devVRE%23noauthentication).

Mapping (among fields) from an ISO19139 Metadata to Ckan Dataset via ckanext-geonetwork is showed in the following table:

ISO19139	Ckan Dataset
Title	Title
Description	Description
bbox	spatial
Descriptive Keywords
gmd:keyword	Tag
	Additional Info
metadata language, age, reference system, etc.	key/value
Digital Transfer Option	Data and Resource
CI_OnlineResource
gmd:url	URL
gmd:name	Name
gmd:description	Description

Geo Datasets

In order to make a dataset queryable by location (geospatial dataset), a special extra must be defined, with its key named ‘spatial’. The value must be a valid GeoJSON geometry, for example:

{
  "type":"Polygon",
  "coordinates":[[[2.05827, 49.8625],[2.05827, 55.7447], [-6.41736, 55.7447], [-6.41736, 49.8625], [2.05827, 49.8625]]]
}

[Note: the polygon must be closed]

or

{
  "type": "Point",
  "coordinates": [-3.145,53.078]
}

GeoJSON Format Specification are available here: http://geojson.org/geojson-spec.html Datasets with spatial values are automatically geo-indexed, for example so that they can be searched using spatial filters.

GeoSpatial search for datasets: via API or Search Widget

Once your datasets are geo-indexed, you can perform spatial queries by bounding box (coordinates format is [LONG, LAT]), via the following API call:

/api/2/search/dataset/geo?bbox={minx,miny,maxx,maxy}[&crs={srid}]

If the bounding box coordinates are not in the same projection as the one defined in the database, a CRS must be provided, in one of the following forms:

    urn:ogc:def:crs:EPSG::4326
    EPSG:4326
    4326

Otherwise default bounding box is 4326. CKAN Wiki page for Legacy API

Moreover, you can perform spatial queries using an integrated map widget available on CKAN, which allows filtering results by an area of interest. You can try it on D4Science Data Catalogue

CKAN Wiki page for Spatial Search Widget

Users, Roles and Groups

Three roles are envisaged to capture the actions users are allowed to execute by the catalogue in the context of each VRE:

Catalogue_Member - users with this role are allowed to:
- View the organization’s private datasets.

Catalogue_Editor - users with this role are allowed to:
- View the organization’s private datasets;
- Publish new datasets (into the organization);
- Edit or delete the organization’s datasets the user is owner of.

Catalogue_Admin - users with this role are allowed to:
- View the organization’s private datasets;
- Publish new datasets (into the organization);
- Edit or delete any of the organization’s datasets;
- Make datasets public or private.

The default role assigned to every VRE user is Catalogue_Member, i.e. every user of a VRE is entitled to view the private datasets published in the VRE scope in addition to any public dataset. VRE Managers can assign other roles to selected users to enlarge their capabilities.

CKAN Groups can used to create and manage collections of datasets. This could be to catalogue datasets for a particular project or team, or on a particular theme, or as a very simple way to help people find and search your own published datasets.

Related Issues

[1] gCube Data Catalogue for Global Record of Stocks and Fisheries - https://wiki.gcube-system.org/gcube/GCube_Data_Catalogue_for_GRSF

@@ Line 272: / Line 272: @@
 | '''bbox'''
 | spatial
+|-
+| style="font-style: italic;" | Descriptive Keywords
+|
+|-
+| '''gmd:keyword'''
+| Tag
+|-
+|-
+|
+| style="font-style: italic;" | Additional Info
+|-
+| metadata language, age,
+reference system, etc.
+| key/value
 |-
 |-
@@ Line 288: / Line 302: @@
 | '''gmd:description'''
 | Description
-|-
-|
-|
-|-
-| style="font-style: italic;" | Descriptive Keywords
-|
-|-
-| '''gmd:keyword'''
-| Tag
-|-
-|
-| style="font-style: italic;" | Additional Info
-|-
-| metadata language, age,
-reference system, etc.
-| key/value
 |}

Difference between revisions of "GCat Background"

Revision as of 15:46, 17 October 2016

Contents

gCube Data Catalogue

Metadata

CKAN's default metadata fields

gCube Metadata Profile

Metadata Profile v.2

Metadata Profile v.1

Instances of Metadata Profile

SoBigData.eu

Ckan Connector

Geo Harvesting

Geo Datasets

GeoSpatial search for datasets: via API or Search Widget

Users, Roles and Groups

Related Issues

Navigation menu

Views

Personal tools

gCube Wiki

gCube features

gCube documentation

Integration and Distribution

Search

Tools