Difference between revisions of "Darwin Core Terms"

Revision as of 16:25, 11 September 2013

This quick guide provides a list of all current Darwin Core terms we are using in Species Product Discovery Service.

Darwin Core Archive is used to export taxonomic (species) data, while Simple Darwin Core is used to export species occurrences.

Darwin Core Archive

DarwinCore Archive (DwC-A) is a Biodiversity informatics data standard that makes use of the DarwinCore terms to produce a single, self contained dataset for taxonomic (species) data. The GBIF GNA format consists of a set of files where one (or more) files represents the 'core' taxonomic data where a single row represents a single taxon reference. The DarwinCore Taxon class provides the majority of concepts supported in the format that enable taxonomic and nomenclatural semantics and syntax (classification, taxonomic and nomenclatural synonymy, status, etc.) to be expressed.

Other files represent "extensions" to this core table and allow additional data elements to be linked to a taxon in the core table with a many to one relationship. The overall topology of one or more of these extensions to the core table is referred to as a "star schema" and provides a compromise between an overly simple flat-file representation of data and more complex multi-related files. In addition to these files, an additional descriptor file serves as a key to the other files. Collectively, these files can be further zipped into a single compressed archive file for portability. This compressed file is known as a DarwinCore Archive (DwCA) file.

Example: Sarda from OBIS

The example represents the sarda species checklist from OBIS.

The process to create a Darwin Core Archive is simple:

• Identify the standard terms and extensions required to map the biodiversity data you wish to share in your archive;

• Export your data as a set of one or more text (CSV) files;

taxonID		parentNameUsageID	scientificName		scientificNameAuthorship	kingdom		phylum		class		   order		taxonRank	
__________________________________________________________________________________________________________________________________________________________________________________
OBIS:506000	OBIS:758818		Sarda			Cuvier, 1829			Animalia	Chordata	Actinopterygii	   Perciformes	        genus		
OBIS:758818	OBIS:755264		Scombridae						Animalia	Chordata	Actinopterygii	   Perciformes          family	
OBIS:755264	OBIS:737387		Perciformes						Animalia	Chordata	Actinopterygii                          order	
OBIS:737387	OBIS:755917		Actinopterygii						Animalia        Chordata                                                class	
.........

• Add XML descriptor files to inform others how your files are organized;
• Zip the folder into a single archive and you are done!

Files

A DarwinCore Archive file generated by SPD Service (in our case representing Darwin Core Taxa) contains the following files:

• a metadata file (eml.xml) that describes the data resource;
• a metafile (meta.xml) file that describes the content and relationship of the text data files;
• a core data file in CSV (taxa.txt) consisting of a standard set of DarwinCore terms;
• a vernacular names extension file (VernacularName.txt) that supports the description of common name properties that might be related to a species described in the core data file.

Metafile (meta.xml)

The DarwinCore Archive format relies on a special file - an XML descriptor file, called the "metafile" (typically named meta.xml). The metafile is used as a map to describe the core taxon file and any extensions that collectively form the specific data profile that will be produced by the user.

<?xml version='1.0' encoding='utf-8'?>
<archive xmlns="http://rs.tdwg.org/dwc/text/" metadata="eml.xml">
	<core encoding="UTF-8" linesTerminatedBy="\n" fieldsTerminatedBy="\t" fieldsEnclosedBy="" ignoreHeaderLines="1" rowType="http://rs.tdwg.org/dwc/terms/Taxon">
		<files>
			<location>taxa.txt</location>
		</files>
		<id index="0"/>
		<field index="0" term="http://rs.tdwg.org/dwc/terms/taxonID"/>
		<field index="1" term="http://rs.tdwg.org/dwc/terms/acceptedNameUsageID"/>
		<field index="2" term="http://rs.tdwg.org/dwc/terms/parentNameUsageID"/>
		<field index="3" term="http://rs.tdwg.org/dwc/terms/scientificName"/>
		<field index="4" term="http://rs.tdwg.org/dwc/terms/scientificNameAuthorship"/>
		<field index="5" term="http://rs.tdwg.org/dwc/terms/nameAccordingTo"/>
		<field index="6" term="http://rs.tdwg.org/dwc/terms/kingdom"/>
		<field index="7" term="http://rs.tdwg.org/dwc/terms/phylum"/>
		<field index="8" term="http://rs.tdwg.org/dwc/terms/class"/>
		<field index="9" term="http://rs.tdwg.org/dwc/terms/order"/>
		<field index="10" term="http://rs.tdwg.org/dwc/terms/family"/>
		<field index="11" term="http://rs.tdwg.org/dwc/terms/genus"/>
		<field index="12" term="http://rs.tdwg.org/dwc/terms/subgenus"/>
		<field index="13" term="http://rs.tdwg.org/dwc/terms/specificEpithet"/>
		<field index="14" term="http://rs.tdwg.org/dwc/terms/infraspecificEpithet"/>
		<field index="15" term="http://rs.tdwg.org/dwc/terms/verbatimTaxonRank"/>
		<field index="16" term="http://rs.tdwg.org/dwc/terms/taxonRank"/>
		<field index="17" term="http://rs.tdwg.org/dwc/terms/taxonomicStatus"/>
		<field index="18" term="http://purl.org/dc/terms/modified"/>
		<field index="19" term="http://purl.org/dc/terms/bibliographicCitation"/>
		<field index="20" term="http://rs.tdwg.org/dwc/terms/taxonRemarks"/>
		<field index="21" term="http://rs.tdwg.org/dwc/terms/scientificNameID"/>
	</core>
	<extension encoding="UTF-8" linesTerminatedBy="\n" fieldsTerminatedBy="\t" fieldsEnclosedBy="" ignoreHeaderLines="1" rowType="http://rs.gbif.org/terms/1.0/VernacularName">
		<files>
			<location>VernacularName.txt</location>
		</files>
		<coreid index="0"/>
		<field index="1" term="http://rs.tdwg.org/dwc/terms/vernacularName"/>
		<field index="2" term="http://purl.org/dc/terms/language"/>
		<field index="3" term="http://rs.tdwg.org/dwc/terms/locality"/>
	</extension>
</archive>

Core Data File (taxa.txt)

A required core data file consisting of a standard set of DarwinCore terms.

A data file is formatted as fielded text, where data records are expressed as rows of text, and data fields (column) are separated with a standard delimiter (a tab).

In a DarwinCore data file the first row of the file contains the names of the DarwinCore terms represented in the succeeding rows of the data.

Repository URL: http://rs.gbif.org/core/dwc_taxon.xml

Vernacular Names Extension File (VernacularName.txt)

Extension file are also simple text files that can visualised as a spreadsheet. They are tied to the core taxon file through a copy of the taxonID used in the core taxon file that is repeated once for each row in the extension file in a manner similar to foreign keys in a relational database. An extension file may include Darwin Core terms as well as terms defined through other means.

The use of extension files allows checklist information to be represented in a one-to-many relation ship between the core taxon file and the extension. For example, Vernacular Names Extension provides the means to share information related to common (vernacular) names linked to taxa in the core data file. Multiple vernacular names can be linked to the same taxon via the taxonID.

Repository: http://rs.gbif.org/extension/gbif/1.0/vernacular.xml

Validating

You can test your archive for structural or data problems by checking it with the GBIF Darwin Core Archive Validator.

This tool will perform several tests:

• ensure the archive is valid by decompressing it and confirming the presence of a meta.xml descriptor file;
• validate the meta.xml file according to the DwC-A meta file XML Schema;
• verify that required fields are included;
• validate certain field values such as data types and licenses;
• perform some data type-specific validation such as verify text object contain descriptions and images contain accessURIs.

Simple Darwin Core

The Simple Darwin Core has minimal restrictions on which fields are required (none). By having no required field restriction, the Simple Darwin Core can be used to share any meaningful combination of fields - in our case, occurrences.

The following schema specifies the fields contained in a Simple Darwin Core file.