View source for GCat Background

'''** THIS PAGE IS UNDER CONSTRUCTION **''' 
== gCube Data Catalogue ==

D4Science offers services for seamless access and analysis to a wide spectrum of data including biological and ecological data, geospatial data, statistical data and semi-structured data from multiple authoritative data providers and information systems. These services can be exploited both via web based graphical user interfaces and web based protocols for programmatic access, e.g. OAI-PMH, CSW, WFS, SDMX. This offering nicely complements specific and community-specific applications.
The gCube Data Catalogue catalogue contains a wealth of resources resulting from several activities, projects and communities including BlueBRIDGE (www.bluebridge-vres.eu/), i-Marine (www.i-marine.eu), SoBigData.eu (www.sobigdata.eu), and FAO (www.fao.org). All the products are accompanied with rich descriptions capturing general attributes, e.g. title and creator(s), as well as usage policies and licences. 



The gCube Data Catalogue is built using and extending CKAN platform. CKAN is a powerful DMS (data management system) that makes data accessible – by providing tools to streamline publishing, sharing, finding and using data. 
CKAN is an open-source DMS for powering data hubs and data portals. CKAN makes it easy to publish, share and use data see: http://ckan.org/

CKAN model is made by the following entities (and their relations):

[[File:ckan_entities.png|1000px|thumb|center|CKAN: 'Entities and Relations']]

== Available Catalogues and their public locations ==

''BLUEBRIDGE Catalogue''

* https://bluebridge.d4science.org/catalogue
* https://i-marine.d4science.org/catalogue

''D4Science Catalogue'' 
* https://services.d4science.org/catalogue



== Metadata ==

A Metadata in the gCube Data Catalogue is made by two parts: [[#CKAN's default metadata fields | CKAN's default metadata fields]] and [[#gCube Metadata Profile | gCube Metadata Profile]].

=== CKAN's default metadata fields ===

Those are metadata fields common for all metadata types in the gCube Data Catalogue (and used by default in the CKAN platform).

{| class="wikitable"
! style="font-weight: bold;" | Label
! style="font-weight: bold;" | Field Name (API)
! style="font-weight: bold;" | Definition
! style="font-weight: bold;" | Guidelines
! style="font-weight: bold;" | Example
|-
| Title*
| title
| Name given to the dataset.
| Short phrase, written in plain language. Should be sufficiently descriptive to allow for search and discovery.
| Aquaculture Production and Consumption in Africa (2011)
|-
| Description
| description
| Short description explaining the content and its origins.
| Description of a few sentences, written in plain language. Should,provide a sufficiently comprehensive overview of the resource for anyone,to understand its content, origins, and any continuing work on it. The,description can be written at the end, since it summarizes key,information from the other metadata fields.
| This dataset contains attributes of aquaculture production and,consumption for each of Africa’s provinces in 2011. The data was,provided by………
|-
| Tags
| tags
| An array of Taxonomic terms stored as tags
| Taxonomic terms
| Access to education, Bamboo
|-
| License*
| lincese_title
| the license that applies to published dataset.
| 
| 
|-
| Organization*
| organization
| Organization the datasets belongs to
| See list of organizations on 
https://ckan-d-d4s.d4science.org/organization
| D4Science
|-
| Version
| version
| Version of dataset
| Increase manually after editing
| 1.0
|-
| Author*
| 
| Owner of dataset
| The person who created the dataset in the format: Surname, Name
| Bloggs, Joe
|-
| Author Contact
| 
| Contact details of owner
| The email or other contact details of the person who created the dataset. 
| joe@example.com
|-
| Mantainer
| 
| Mantainer of the dataset
| The person or the authority that maintains the dataset
| A person: Bloggs, Joe. An authority: D4Science
|-
| Mantainer
Contact
| 
| Contact details of mantainer
| The email or other contact details of the person who maintains the dataset.
| joe@example.com
|}

''mandatory fields are marked with an asterisk (*)''

=== gCube Metadata Profile ===

gCube Metadata Profile defines a Metadata schema XML-based for adding custom metadata fields.

A gCube Metadata Profile is composed by one Metadata Format (<metadataformat>) containing an ordered list of '''(at least) one Metadata Field (<metadatafield>)'''. 
From version 3 a Metadata Field can contain also a reference (categoryref="category_id_#") to an entity "Category" using the Namespace of the Category (<namespace id="category_id_#">). 
Add a Category Reference to a Metadata Field means that the "field" belongs to the Category referred by Category Identifier (id="category_id_#). 
See Metadata Profile v.3. for more details.

===== Metadata Profile v.3 =====

Metadata Profile v.3 is a XML file having the format:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<metadataformat type="YOUR TYPE HERE">
    <metadatafield categoryref="category_id_#">
        <fieldName>Name of Metadata Field</fieldName>
        <mandatory>true|false</mandatory>
        <dataType>String|Time|Time_Interval|Times_ListOf|Text|Boolean|Number|GeoJSON</dataType>
        <maxOccurs>N|*</maxOccurs>
        <defaultValue>default value</defaultValue>
        <note>shown as suggestions in the insert/update metadata form of CKAN
		</note>
        <vocabulary isMultiSelection="true|false">
            <vocabularyField>field1</vocabularyField>
            <vocabularyField>field2</vocabularyField>
            <vocabularyField>field3</vocabularyField>
        </vocabulary>
        <validator>
            <regularExpression>a regular expression for validating values</regularExpression>
        </validator>
        <tagging create="true|false" separator="char_to_separate">onFieldName|onValue|onFieldName_onValue|onValue_onFieldName</tagging> 
        <grouping create="true|false">onFieldName|onValue|onFieldName_onValue|onValue_onFieldName</grouping>
    </metadatafield>
</metadataformat>
</pre>

What's new:

'''maxOccurs Indicator''':

The <maxOccurs> indicator specifies the maximum number of times that <metadatafield> can occur:
* N (as number): if the field must appear N  times;
* * (as char asterisk): if the field can appear an unlimited number of times.


'''Categories as "Namespaces"''':

* the Namespace of a Category declares a "class" for metadata fields having particular characteristics. It has been introduced in order to group metadata fields for categories and displaying them in a dedicated area through advanced GUI provided by CKAN D4Science plugin.

Namespaces (for Categories) are defined in an XML file made by one Namespaces element (<namespaces>) containing a list of (at least) one or many Namespace (<namespace>). The file has the format:
<pre>
<?xml version="1.0" encoding="UTF-8"?>
<namespaces xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<namespace id="category_id_#">
		<name>Category Name</name>
		<title>Category Title</title>
		<description>This section is about Category description</description>
	</namespace>
</namespaces>
</pre>

A Namespace element (<namespace>) has an attribute (id) and three entities. The attribute "id" must be unique in the file [[#Namespaces_Categories_schema:_NamespacesCatalogueCategories.xsd]], it represents the category identifier for the Category. The elements are: name (is mandatory), title (is mandatory), description (is optional).

Metadata Field and Category Reference (categoryref="category_id_#"):
* categoryref is an optional attribute. It is a unique id (id="category_id_#"). A metadata field can belong to only one Namespace of a Category referring it via idref (categoryref="id category to which metadata field belongs one").

Type of (meta)data (is Mandatory):
* type: a Metadata Format (metadataformat) must have a unique 'type' (as a xml attribute) that declares a "type" for it. This mandatory information is saved as custom key (system:type="value of type") of the item stored in the Data Catalogue.

'''Tagging''':
* It is used by gCube Data Catalogue front-end for adding a metadata field as a Tag of the metadata. A Tag is a string between 2 and 100 characters long containing only alphanumeric characters and '-' (hyphen), '_' (underscore), . (dot). Tagging element in the Metadata Profile schema v3 must have a value equal to one of the values: {onFieldName, onValue, onFieldName_onValue, onValue_onFieldName}. Tagging values meanings:
** onFieldName: (only) the fieldName specified to metadata field must be added as a Tag;
** onValue: (only) the value specified to metadata field must be added as a Tag;
** onFieldName_onValue: both the fieldName and the value (in this order) specified to metadata field must be added as a Tag. They are separated by string used as separator (<tagging create="true|false" separator="char_to_separate">{onFieldName_onValue}</tagging>);
** onValue_onFieldName: both the value and the fieldName (in this order) specified to metadata field  must be added as a Tag. They are separated by string used as separator (<tagging create="true|false" separator="char_to_separate">{onValue_onFieldName}</tagging>). 
* Moreover, Tagging has two (optional) attribute: 'create' and 'separator'. The first one (create="true"|"false") is used to mean: create the Tag if does not exist, no otherwise. The second one (separator="char_to_separate") is the string that will be used to separate the FieldName from its value. Default value for separator is the character '-' if it is not specified.

Tagging example: using following instance of metadata field
<pre>
	<metadatafield categoryref="contact">
		<fieldName>Name</fieldName>
		<dataType>String</dataType>
		<defaultValue>My Name</defaultValue>
		<note>Insert your Name</note>
		<tagging create="true" separator="-">onFieldName_onValue</tagging>
	</metadatafield>
</pre>

where My Name is "Francesco", gCube Data Catalogue adds the tag Name-Francesco to metadata field if it does not exist

'''Grouping''':
* It is used by Data Catalogue fron-end for adding a metadata field to a Group of Data Catalogue. Data Catalogue [https://ckan-d4s.d4science.org/group groups] are used for browsing. Grouping element in the Metadata Profile schema v3 must have a value equal to one of the values: {onFieldName, onValue, onFieldName_onValue, onValue_onFieldName}. The (optional) attribute create="true" is used to mean: create the Group if does not exist, no otherwise. Grouping values meanings:
** onFieldName: (only) the fieldName specified to metadata field must be added to a Group;
** onValue: (only) the value specified to metadata field must be added to a Group;
** onFieldName_onValue: both the fieldName and the value (in this order) specified to metadata field must be added to a Group (<grouping create="true|false">{onFieldName_onValue}</grouping>);
** onValue_onFieldName: both the value and the fieldName (in this order) specified to metadata field must be added to a Group (<grouping create="true|false">{onValue_onFieldName}</grouping>).
* Moreover, Grouping has one (optional) attribute: 'propagateUp' This property will let a user specify if an item, that is going to be added to that group, must be also added to the hierarchical chain of groups involving it. For instance, if we have group B as child of group A, and item I is going to be added to B, then it will also be added to A. 

'''DataType values''':
* GeoJSON: is a string. [http://geojson.org/geojson-spec.html GeoJSON] is a format for encoding a variety of geographic data structures.

GeoJSON example: an instance of MedataField for GeoSpatial
<pre>
    <metadatafield idref="category_id_#">
        <fieldName>spatial</fieldName>  <!--'spatial' is mandatory name of GeoSpatial field  -->
        <dataType>GeoJSON</dataType>
        <defaultValue>{"type": "Point","coordinates": [-20.145,74.078]}</defaultValue>
        <note>shown as suggestions in the insert/update metadata form of CKAN</note>
    </metadatafield>

</pre>

====== Metadata Profile schema: gcdcmetadataprofilev3.xsd ======

The gCube Data Catalogue Metadata Profile (v.3) schema:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified"
	elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
	<xs:include schemaLocation="NamespacesCatalogueCategories.xsd"/>
	<xs:element name="metadataformat">
		<xs:complexType>
			<xs:sequence>
				<xs:element name="metadatafield" minOccurs="0" maxOccurs="unbounded">
					<xs:complexType>
						<xs:sequence>
							<xs:element type="xs:string" name="fieldName" />
							<xs:element type="xs:boolean" name="mandatory"
								minOccurs="0" maxOccurs="1" />
							<xs:element name="dataType">
								<xs:simpleType>
									<xs:restriction base="xs:string">
										<xs:enumeration value="String" />
										<xs:enumeration value="Time" />
										<xs:enumeration value="Time_Interval" />
										<xs:enumeration value="Times_ListOf" />
										<xs:enumeration value="Text" />
										<xs:enumeration value="Boolean" />
										<xs:enumeration value="Number" />
										<xs:enumeration value="GeoJSON" />
									</xs:restriction>
								</xs:simpleType>
							</xs:element>
							<xs:element type="xs:string" name="maxOccurs"
								minOccurs="0" maxOccurs="1" />
							<xs:element type="xs:string" name="defaultValue"
								minOccurs="0" maxOccurs="1" />
							<xs:element type="xs:string" name="note" minOccurs="0"
								maxOccurs="1" />
							<xs:element name="vocabulary" minOccurs="0" maxOccurs="1">
								<xs:complexType>
									<xs:sequence>
										<xs:element type="xs:string" name="vocabularyField"
											minOccurs="1" maxOccurs="unbounded" />
									</xs:sequence>
									<xs:attribute type="xs:boolean" name="isMultiSelection" />
								</xs:complexType>
							</xs:element>
							<xs:element name="validator" minOccurs="0" maxOccurs="1">
								<xs:complexType>
									<xs:sequence>
										<xs:element type="xs:string" name="regularExpression" />
									</xs:sequence>
								</xs:complexType>
							</xs:element>
							<xs:element name="tagging" type="TaggingType"
								minOccurs="0" maxOccurs="1">
							</xs:element>
							<xs:element name="grouping" type="GroupingType"
								minOccurs="0" maxOccurs="1">
							</xs:element>
						</xs:sequence>
						<xs:attribute name="categoryref" use="optional" type="xs:string" />
					</xs:complexType>
				</xs:element>
			</xs:sequence>
			<xs:attribute type="NotEmpty" use="required" name="type" />
		</xs:complexType>
	</xs:element>
	<xs:simpleType name="TaggingGroupingValue">
		<xs:restriction base="xs:string">
			<xs:enumeration value="onFieldName" />
			<xs:enumeration value="onValue" />
			<xs:enumeration value="onFieldName_onValue" />
			<xs:enumeration value="onValue_onFieldName" />
		</xs:restriction>
	</xs:simpleType>
	<xs:complexType name="TaggingType">
		<xs:simpleContent>
			<xs:extension base="TaggingGroupingValue">
				<xs:attribute type="xs:boolean" name="create" />
				<xs:attribute type="NotEmpty" name="separator" />
			</xs:extension>
		</xs:simpleContent>
	</xs:complexType>
	<xs:complexType name="GroupingType">
		<xs:simpleContent id="TaggingGroupingValue">
			<xs:extension base="TaggingGroupingValue">
				<xs:attribute type="xs:boolean" name="create" />
				<xs:attribute type="xs:boolean" name="propagateUp" />
			</xs:extension>
		</xs:simpleContent>
	</xs:complexType>
	<xs:simpleType name="NotEmpty">
		<xs:restriction base="xs:string">
			<xs:minLength value="1" />
		</xs:restriction>
	</xs:simpleType>
</xs:schema>
</pre>

You can download it by clicking on [https://wiki.gcube-system.org/images_gcube/e/e8/Gcdcmetadataprofilev3.xsd Gcdcmetadataprofilev3.xsd]

A "generic" example of MetadataProfile.xml: 

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<metadataformat type="the_metadata_type" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="gcdcmetadataprofilev3.xsd">
  <metadatafield categoryref="idvalue0">
    <fieldName>fieldName</fieldName>
    <dataType>String</dataType>
    <defaultValue>defaultValue</defaultValue>
    <note>note</note>
    <vocabulary isMultiSelection="true">
      <vocabularyField>vocabularyField</vocabularyField>
    </vocabulary>
    <validator>
      <regularExpression>regularExpression</regularExpression>
    </validator>
    <tagging create="true" separator="-">onFieldName_onValue</tagging>
    <grouping create="true">onFieldName</grouping>
  </metadatafield>
</metadataformat>
</pre>

Another example (of MetadataProfile.xml ) is the following one:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<metadataformat type="the_metadata_type"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:noNamespaceSchemaLocation="gcdcmetadataprofilev3.xsd">
	<metadatafield categoryref="contact">
		<fieldName>Name</fieldName>
		<dataType>String</dataType>
		<defaultValue>My Name</defaultValue>
		<note>Insert your Name</note>
		<tagging create="true" separator="-">onFieldName_onValue</tagging>
	</metadatafield>
	<metadatafield categoryref="contact">
		<fieldName>Surname</fieldName>
		<dataType>String</dataType>
		<defaultValue>My Surname</defaultValue>
		<note>Insert your Surname</note>
	</metadatafield>
</metadataformat>
</pre>

====== Namespaces Categories schema: NamespacesCatalogueCategories.xsd ======

The Namespaces Catalogue Categories schema: 

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified"
	elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema"
	xmlns:category="http://www.w3.org/TR/html4/">
	<xs:element name="namespaces">
		<xs:complexType>
			<xs:sequence>
				<xs:element name="namespace" minOccurs="1" maxOccurs="unbounded">
					<xs:complexType>
						<xs:sequence>
							<xs:element type="xs:string" name="name" minOccurs="1"
								maxOccurs="1" />
							<xs:element type="xs:string" name="title" minOccurs="1"
								maxOccurs="1" />
							<xs:element type="xs:string" name="description"
								minOccurs="0" maxOccurs="1" />
						</xs:sequence>
						<xs:attribute type="xs:string" name="id" use="required" />
					</xs:complexType>
				</xs:element>
			</xs:sequence>
		</xs:complexType>
		<xs:unique name="unique-namespace-id">
			<xs:selector xpath="namespace" />
			<xs:field xpath="@id" />
		</xs:unique>
	</xs:element>
</xs:schema>
</pre>

You can download it by clicking on [https://wiki.gcube-system.org/images_gcube/d/d5/NamespacesCatalogueCategories.xsd NamespacesCatalogueCategories]

An example of valid Namespaces.xml:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<namespaces xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:noNamespaceSchemaLocation="NamespacesCatalogueCategories.xsd">
	<namespace id="contact">
		<name>Contact</name>
		<title>Contact Title</title>
		<description>This section is about Contact(s)</description>
	</namespace>
	<namespace id="developer_information">
		<name>Developer</name>
		<title>Developer Title</title>
		<description>This section is about Developer(s)</description>
	</namespace>
	<namespace id="extra_information">
		<name>Extra</name>
		<title>Extra Title</title>
		<description>This section is about Extra(s)</description>
	</namespace>
</namespaces>
</pre>

===== Metadata Profile v.2 =====

Relased in gCube 4.1. What's new:

* Added datatype field (<datatype>). A valid datatype must be equal to one of the values {String, Time, Time_Interval, Times_ListOf, Text, Boolean, Number}. When data type is not specified the metadata field has default value as "String". DataType values:
** String: is a string;
** Time: an instant time that follows the general format: YYYY-MM-DD [HH:MM] where: YYYY: 4-digit year, MM: 2-digit month, DD: 2-digit day, [optional HH: 2-digit hour], [optional MM: 2-digit minute] (e.g. "2005-03-01");
** Time_Interval: a continuous interval instead of a single instant by specifying a start and end time, separated by one '/' ('slash') character (e.g. "2005-03-01/2006-05-11");
** Times_ListOf: a list of discrete time values, separated by a ',' ('comma') character (e.g. "2005-03-01, 2006-05-11, 2006-05-11-2007-04-12");
** Text: is a text;
** Boolean: is True/False;
** Number: is a valid Java number, see: Apache Commons NumberUtils.isNumber.

* Added multi selection attribute ('isMultiSelection=true|false') to vocabulary.
Values of Time, Time_Interval and Times_ListOf are based on [https://en.wikipedia.org/wiki/ISO_8601 ISO 8601]

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<metadataformat>
	<metadatafield>
		<fieldName>Name</fieldName>
		<mandatory>true</mandatory>
		<dataType>String|Time|Time_Interval|Times_ListOf|Text|Boolean|Number</dataType>
		<defaultValue>default value</defaultValue>
		<note>shown as suggestions in the insert/update metadata form of CKAN
		</note>
		<vocabulary isMultiSelection="true|false">
			<vocabularyField>field1</vocabularyField>
			<vocabularyField>field2</vocabularyField>
			<vocabularyField>field3</vocabularyField>
		</vocabulary>
		<validator>
			<regularExpression>a regular expression for validating values
			</regularExpression>
		</validator>
	</metadatafield>
</metadataformat>
</pre>

It's possible to validate a Metadata Format schema using following DTD v2

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<!ELEMENT metadataformat (metadatafield+)>
<!ELEMENT metadatafield (fieldName, mandatory, datatype?, defaultValue?, note?, vocabulary?, validator?)>
<!ELEMENT fieldName (#PCDATA)>
<!ELEMENT mandatory (#PCDATA)>
<!ELEMENT dataType (#PCDATA)>  
<!ELEMENT defaultValue (#PCDATA)> 
<!ELEMENT note (#PCDATA)> 
<!ELEMENT vocabulary (vocabularyField+)> 
<!ATTLIST vocabulary isMultiSelection (true|false) "false">
<!ELEMENT vocabularyField (#PCDATA)> 
<!ELEMENT validator (regularExpression)> 
<!ELEMENT regularExpression (#PCDATA)> 

<!--
Where datatype element is the enum: {String, Time, Time_Interval, Times_ListOf, Text, Boolean, Number}
Its xml schema:
<xs:element name="dataType">
  <xs:simpleType>
    <xs:restriction base="xs:string">
      <xs:enumeration value="String"/>
      <xs:enumeration value="Time"/>
      <xs:enumeration value="Time_Interval"/>
      <xs:enumeration value="Times_ListOf"/>
      <xs:enumeration value="Text"/>
      <xs:enumeration value="Boolean"/>
      <xs:enumeration value="Number"/>
    </xs:restriction>
  </xs:simpleType>
</xs:element> 
-->
</pre>

===== Metadata Profile v.1 =====

Relased in gCube 4.0. The schema is:

<pre>
<?xml version="1.0" encoding="UTF-8">
<metadataformat>
    <metadatafield>
        <fieldName>Name</fieldName>
        <mandatory>true</mandatory>
        <isBoolean>false</isBoolean>
        <defaultValue>default value</defaultValue>
        <note>shown as suggestions in the insert/update metadata form of CKAN</note>
        <vocabulary>
            <vocabularyField>field1</vocabularyField>
            <vocabularyField>field2</vocabularyField>
            <!-- ... others vocabulary fields -->
        </vocabulary>
        <validator>
            <regularExpression>a regular expression for validating values</regularExpression>
        </validator>
    </metadatafield>
     <!-- ... others metadata fields -->
</metadataformat>
</pre>

It's possible to validate a Metadata Format schema using following DTD v1
<pre>

<?xml version="1.0" encoding="UTF-8"?>
<!ELEMENT metadataformat (metadatafield+)>
<!ELEMENT metadatafield (fieldName, mandatory, isBoolean?, defaultValue?, note?, vocabulary?, validator?)>
<!ELEMENT fieldName (#PCDATA)>
<!ELEMENT mandatory (#PCDATA)>
<!ELEMENT isBoolean (#PCDATA)>  <!-- MUST BE (true|false) -->
<!ELEMENT defaultValue (#PCDATA)> 
<!ELEMENT note (#PCDATA)> 
<!ELEMENT vocabulary (vocabularyField+)> 
<!ELEMENT vocabularyField (#PCDATA)> 
<!ELEMENT validator (regularExpression)> 
<!ELEMENT regularExpression (#PCDATA)> 

</pre>

A possible instance of Metadata Field (<metadatafield>):

<pre>
<metadatafield>
   <fieldName>Accessibility</fieldName>
   <mandatory>true</mandatory>
   <defaultValue>virtual/public</defaultValue>
   <vocabulary>
       <vocabularyField>virtual/public</vocabularyField>
       <vocabularyField>virtual/private</vocabularyField>
       <vocabularyField>transactional</vocabularyField>
   </vocabulary>
</metadatafield>
</pre>

=== Instances of Metadata Profile ===

The following list shows the instances of metadata profile created in the D4Sciece infrastructure and currently used in gCube Data Catalogue:

===== SoBigData.eu =====

[[SoBigData.eu: Metadata Profile for gCube Data Catalogue]]

== Ckan Connector ==

TODO @Lucio

== Geo Harvesting ==

This extension contains plugins like [https://github.com/geosolutions-it/ckanext-geonetwork/wiki ckanext-geonetwork] (and others) which add geospatial capabilities to CKAN.

Several harvesters to import geospatial metadata (like ISO 19139 format) into CKAN from other sources have been created in gCube Data Catalogue.
In particular all metadata created into gCube Geonetwork (GeoNetwork is the catalog application to manage spatially referenced resources generated into D4Science Infrastructure) are harvested through the '''Geonetwork Resolver'''.

Mapping (among fields) from an ISO19139 Metadata to Ckan Dataset via ckanext-geonetwork is shown in the following table:

{| class="wikitable"
! style="font-weight: bold;" | ISO19139
! style="font-weight: bold;" | Ckan Dataset
|-
| '''Title'''
| Title
|-
| '''Description'''
| Description
|-
| '''bbox'''
| spatial
|-
| style="font-style: italic;" | Descriptive Keywords
| 
|-
| '''gmd:keyword'''
| Tag
|-
|-
| 
| style="font-style: italic;" | Additional Info
|-
| metadata language, age,
reference system, etc.
| key/value
|-
|-
| style="font-style: italic;" | Digital Transfer Option
| style="font-style: italic;" | Data and Resource
|-
| CI_OnlineResource
| 
|-
| '''gmd:url'''
| URL
|-
| '''gmd:name'''
| Name
|-
| '''gmd:description'''
| Description
|}

=== Geonetwork Resolver ===

The 'Geonetwork Resolver' is a "middle tier" to perform queries on gCube Geonetwork services. It uses the 'Geonetwork Manager library' to discovery Geonetwork configurations like: Geonetwork service end-point, Geonetwork accounts and so on.
Using the Geonetwork configurations discovered the resolver performs authentication on Geonetwork (if needed) and then forward the requests received to harvesting metadata on that Geonetwork. 
Authentication is needed to get layers stored as private on Geonetwork.

The resolver is reachable by HTTP GET request and supports the following protocol: 
http[s]://[GEONETWORK_RESOLVER_BASE_URL]/[SCOPE]/[MODE]/[VISIBILITY]/[OWNER]/$$

In detail, the protocol parameters are:

* SCOPE (mandatory): the gCube scope used via 'Geonetwork Manager library' to discovery the Geonetwork service end-point and other configurations (like Geonetwork accounts and so on). It must be separated by character pipe '|'. i.e. gcube|devNext|NextNext;
* MODE (mandatory): must be 'VRE' or 'HARVEST'. To harvest at VRE level or External GEONETWORK level;
* VISIBILITY (mandatory): must be 'PUB' or 'PRV'. When PUB: only public layer/s is/are returned. When PRV: only private layer/s is/are returned;
* OWNER (optional): to filter by owner passed as parameter. Only layer/s whose ownership belongs to owner passed are returned;
* REQUEST DELIMITER (mandatory). must be '/$$'. It is the delimiter to specify the end of the query.

i.e. CKAN harvester configured to retrieve metadata by the URL: http://data-d.d4science.org/geonetwork/d4science.research-infrastructures.eu|gCubeApps|RPrototypingLab/VRE/PRV/$$

Where:
* GEONETWORK_RESOLVER_BASE_URL has value: 'http://data-d.d4science.org/geonetwork';
* SCOPE has value: 'd4science.research-infrastructures.eu|gCubeApps|RPrototypingLab';
* MODE has value: 'VRE';
* VISIBILITY has value: 'PRV';

It retrieves all private layers of the VRE RPrototypingLab.

== Geo Datasets ==

In order to make a dataset queryable by location (geospatial dataset), a special extra must be defined, with its key named ‘spatial’. The value must be a valid GeoJSON geometry, for example:

<pre>
{
  "type":"Polygon",
  "coordinates":[[[2.05827, 49.8625],[2.05827, 55.7447], [-6.41736, 55.7447], [-6.41736, 49.8625], [2.05827, 49.8625]]]
}
</pre>

[Note: the polygon must be closed]

or

<pre>
{
  "type": "Point",
  "coordinates": [-3.145,53.078]
}
</pre>

GeoJSON Format Specification are available here: http://geojson.org/geojson-spec.html
Datasets with spatial values are automatically geo-indexed, for example so that they can be searched using spatial filters.

=== GeoSpatial search for datasets: via API or Search Widget === 
 
Once your datasets are geo-indexed, you can perform spatial queries by bounding box (coordinates format is [LONG, LAT]), via the following API call:

<pre>
/api/2/search/dataset/geo?bbox={minx,miny,maxx,maxy}[&crs={srid}]
</pre>

If the bounding box coordinates are not in the same projection as the one defined in the database, a CRS must be provided, in one of the following forms:

<pre>
    urn:ogc:def:crs:EPSG::4326
    EPSG:4326
    4326
</pre>

Otherwise default bounding box is 4326. CKAN Wiki page for [http://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html#legacy-api Legacy API]

Moreover, you can perform spatial queries using an integrated map widget available on CKAN, which allows filtering results by an area of interest. You can try it on [https://ckan-d4s.d4science.org/dataset D4Science Data Catalogue]

CKAN Wiki page for [http://docs.ckan.org/projects/ckanext-spatial/en/latest/spatial-search.html#spatial-search-widget Spatial Search Widget ]

== Users, Roles and Groups ==

Three roles are envisaged to capture the actions users are allowed to execute by the catalogue in the context of each VRE:

* '''''Catalogue_Member''''' - users with this role are allowed to:
** View the organization’s private datasets.

* '''''Catalogue_Editor''''' - users with this role are allowed to:
** View the organization’s private datasets;
** Publish new datasets (into the organization);
** Edit or delete the organization’s datasets the user is owner of.

* '''''Catalogue_Admin''''' - users with this role are allowed to:
** View the organization’s private datasets;
** Publish new datasets (into the organization);
** Edit or delete any of the organization’s datasets;
** Make datasets public or private.
   
The default role assigned to every VRE user is ''Catalogue_Member'', i.e. every user of a VRE is entitled to view the private datasets published in the VRE scope in addition to any public dataset. VRE Managers can assign other roles to selected users to enlarge their capabilities.

'''CKAN Groups''' can used to create and manage collections of datasets. This could be to catalogue datasets for a particular project or team, or on a particular theme, or as a very simple way to help people find and search your own published datasets.

== D4Science CKAN plugin ==

@TODO Francesco

=== Release procedure to D4Science CKAN plugin ===


@TODO Francesco

== Access the Catalogue via RESTful service ==
'''Since gCube 4.5'''
You can find more details about the Catalogue Web Service at the following [http://wiki.gcube-system.org/index.php?title=Catalogue_restful_service page].

== Related Issues ==

[1] gCube Data Catalogue for Global Record of Stocks and Fisheries - https://wiki.gcube-system.org/gcube/GCube_Data_Catalogue_for_GRSF

[2] Data Catalogue Resolver - https://wiki.gcube-system.org/gcube/URI_Resolver#CATALOGUE_Resolver