Difference between revisions of "Data Access and Storage APIs"

From Gcube Wiki
Jump to: navigation, search
(→‎Overview)
(→‎SPD Plugin API)
 
(41 intermediate revisions by 6 users not shown)
Line 1: Line 1:
−
Data Access and Storage APIs provide access to a wide range of system processes, including indexing, transfer, transformation and presentation. They make up a main driver for clients that interface the resources managed by the system or accessible through facilities available within the system.
+
Data Access and Storage APIs support both client and back-end access to a range of structured and unstructured data sources available in the Data e-Infrastructure. APIs include remote Web Service APIs, local client APIs, data model implementations, auxiliary APIs, and development frameworks for service plugins.  
  
−
A large number of APIs are dedicated to meet data access requirements including APIs of services, service plugins, client-side libraries, server-side libraries and front-end interfaces.  
+
This document lists all the available APIs in these areas along with their key features, pointing to external documentation for details on their design and usage.  
  
−
This document outlines the design rational and the key features for those APIs.
+
==API Overview==
−
 
+
−
==Overview==
+
  
 
{| {{table}}
 
{| {{table}}
Line 14: Line 12:
 
| align="center" style="background:#f0f0f0;"|'''Framework'''
 
| align="center" style="background:#f0f0f0;"|'''Framework'''
 
|-
 
|-
−
| [[#Tree Manager API|Tree Manager]]||Tree-based CRUD operations over remote sources of structured data||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
+
| [[#Tree Manager API|Tree Manager]]||Tree-based CRUD operations over pluggable sources of structured data||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
 
|-
 
|-
−
| [[#Tree Manager Client API|Tree Manager Client]]||Client front-end to Tree Manager endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#Tree Manager Client API|Tree Manager Client]]||Client proxies for [[#Tree Manager API|Tree Manager]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
|-
 
|-
−
| [[#Tree Manager Plugin API|Tree Manager Plugin]]||Development framework for Tree Manager plugins||Java||Local||none
+
| [[#Tree Manager Plugin API|Tree Manager Plugin]]||Development framework for [[#Tree Manager API|Tree Manager]] plugins||Java||Local||none
 
|-
 
|-
 
| [[#Trees API|Trees]]||Tree and tree pattern model implementations||Java||Local||none
 
| [[#Trees API|Trees]]||Tree and tree pattern model implementations||Java||Local||none
 
|-
 
|-
−
| [[#gDoc API|gDoc]]||gDoc model implementation||Java||Local||none
+
| [[#Trees API|Streams]]||Conversion, transformation, and publication of data streams||Java||Local||none
 
|-
 
|-
−
| [[#gDoc Client API|gDoc Client]]||gDoc client front-end to Tree Manager endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#gDoc API|gDoc]]||<code>gDoc</code> model implementation||Java||Local||none
 
|-
 
|-
−
| [[#View Manager API|View Manager]]||View management for data sources accessible via Tree Manager endpoints||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
+
| [[#gDoc Client API|gDoc Client]]||<code>gDoc</code>-based client proxies for [[#Tree Manager API|Tree Manager]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
|-
 
|-
−
| [[#View Manager Client API|View Manager Client]]||Client front-end to View Manager endpoints||Java||Java||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#View Manager API|View Manager]]||Views over data sources accessible via [[#Tree Manager API|Tree Manager]] endpoints||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
 
|-
 
|-
−
| [[#Storage Manager API|Storage Manager]]||TODO||Java||Java||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#View Manager Client API|View Manager Client]]||Client proxies for [[#View Manager API|View Manager]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
|-
 
|-
−
| [[#SPD API|Species Products Discovery]]||TODO||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
+
| [[#Storage Manager API|Storage Manager]]||Access and storage of unstructured bytestreams, or files implementation||Java||Local||none
 
|-
 
|-
−
| [[#SPD Plugin API|SPD Plugin]]||TODO||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#SPD API|Species Products Discovery (SPD)]]||Read access to pluggable sources of biodiversity data||WS||SOAP||[https://gcore.wiki.gcube-system.org/gCube gCore]
 
|-
 
|-
−
| [[#SPD Client API|SPD Client]]||TODO||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
+
| [[#SPD Plugin API|SPD Plugin]]||Development framework for [[#SPD API|SPD]] plugins||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 +
|-
 +
| [[#SPD Client API|SPD Client]]||Client proxies for [[#SPD API|SPD]] endpoints||Java||Local||[[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
|}
 
|}
 +
 +
 +
 +
The picture below offers an immediate view of the area and where each component is placed and which other interfaces:
 +
 +
 +
[[File:DataManagementAPIs.png]]
  
 
==Tree Manager API==
 
==Tree Manager API==
−
* '''Description''': Tree-based read and write access to structured data sources.
+
* '''Description''': Tree-based CRUD operations over pluggable sources of structured data.
 
* '''Type''': Web Service
 
* '''Type''': Web Service
 
* '''Protocol''': SOAP
 
* '''Protocol''': SOAP
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
−
* [[The_Tree_Manager|'''Wiki Documentation''']]
+
* [[The_Tree_Manager|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
 
:* ''lookup of whole trees as well as individual nodes''
 
:* ''lookup of whole trees as well as individual nodes''
−
:* ''sophisticated tree pattern matching on lookups and queries''
+
:* ''tree pattern matching on lookups and queries''
−
:* ''in-place updates and deletions''
+
:* ''in-place updates''
−
:* ''optional streaming for all operations''
+
:* ''streaming on all operations''
 +
:* ''native storage based on graph database technology''
  
 
==Tree Manager Client API==
 
==Tree Manager Client API==
  
−
* '''Description''': Client front-end to Tree Manager endpoints.
+
* '''Description''': Client proxies for [[#Tree Manager API|Tree Manager]] endpoints.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
−
* [[The_Tree_Manager_Library|'''Wiki Documentation''']]
+
* [[The_Tree_Manager_Library|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''high-level proxy API''
−
:TODO
+
:* ''transparent endpoint discovery and caching''
 +
:* ''transparent fault tolerance over endpoint replicas''
 +
:* ''polling and notifications for asynchronous bindings''
 +
:* ''fluent proxy configuration API''
  
 
==Tree Manager Plugin API==
 
==Tree Manager Plugin API==
−
* '''Description''': Development framework for Tree Manager plugins.
+
 
 +
* '''Description''': Development framework for [[#Tree Manager API|Tree Manager]] plugins.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* [[The_Tree_Manager_Framework|'''Wiki Documentation''']]
+
* [[The_Tree_Manager_Framework|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''support for a wide range of plugins: source-specific, source-generic, type-specific, type-generic, multi-type, etc''
−
:TODO
+
:* ''interface-based POJO framework, with no dependency on service implementation''
 +
:* ''transparent property publication and state persistence"
 +
:* ''callback-based and event-based interaction with service"
 +
:* ''full lifecycle support''
 +
:* ''partial interface implementations''
 +
:* ''derived implementation of node-based and streamed operations''
  
 
==Trees API==
 
==Trees API==
Line 75: Line 92:
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* [[The_Trees_Library|'''Wiki Documentation''']]
+
* [[The_Trees_Library|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''untyped tree model with attributed nodes, labelled edges, and string-valued leaves''
−
:TODO
+
:* ''pattern language for characterisation of tree topology and leaf values''
 +
:* ''tree matching and pruning operations based on pattern language''
 +
:* ''tree difference and in-place update operations''
 +
:* ''fluent API for tree construction, tree navigation, and tree pattern construction''
 +
:* ''predefined XML bindings for trees and patterns''
 +
:* ''synthetic tree generation facilities for load testing''
 +
 
 +
==Streams API==
 +
 
 +
* '''Description''': Conversion, transformation, and publication of data streams.
 +
* '''Type''': Java
 +
* '''Protocol''': Local
 +
* [[The_Streams_Library|'''Documentation''']]
 +
* '''Key Features'''
 +
:* ''high-level stream model based on <code>Iterator</code>-style interfaces''
 +
:* ''predefined implementations over standard <code>Iterator</code>s and <code>gRS2</code> resultsets''
 +
:* ''piping, folding, and unfolding stream operations''
 +
:* ''configurable error handling policies''
 +
:* ''stream lifecycle event notifications''
 +
:* ''publication as <code>gRS2</code> resultset''
 +
:* ''fluent API for all stream operations''
  
 
==gDoc API==
 
==gDoc API==
−
* '''Description''': gDoc model implementation.
+
 
 +
* '''Description''': <code>gDoc</code> model implementation.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* [[GCube_Document_Model|'''Wiki Documentation''']]
+
* [[GCube_Document_Model|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''document model inclusive of metadata, annotations, parts, and alternatives''
−
:TODO
+
:* ''strongly-typed document construction''
 +
:* ''transparent bytestream resolution''
 +
:* ''annotation threading''
 +
:* ''new and proxy document modes''
 +
:* ''change tracking and staging for in-place updates without prior reads''
 +
:* ''XML binding facilities''
  
 
==gDoc Client API==
 
==gDoc Client API==
−
* '''Description''': gDoc client front-end to Tree Manager endpoints.
+
 
 +
* '''Description''': <code>gDoc</code>-based client proxies for [[#Tree Manager API|Tree Manager]] endpoints.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* [[GCube_Document_Library|'''Wiki Documentation''']]
+
* [[GCube Document Library (2.0)|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''creation of and access to <code>gDoc</code> document sources''
−
:TODO
+
:* ''bulk and streamed operations''
 +
:* ''caching, selection, and pruning for read operations''
 +
:* ''creation, discovery, and use of document views''
  
 
==View Manager API ==
 
==View Manager API ==
−
* '''Description''': View management for data sources accessible via Tree Manager endpoints.
+
* '''Description''': Views over data sources accessible via [[#Tree Manager API|Tree Manager]] endpoints.
 
* '''Type''': Web Service
 
* '''Type''': Web Service
 
* '''Protocol''': SOAP
 
* '''Protocol''': SOAP
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
−
* [[View_Manager|'''Wiki Documentation''']]
+
* [[View_Manager|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''view definition through arbitrary tree patterns''
−
:TODO
+
:* ''view characterisation through arbitrary properties''
 +
:* ''view publication in infrastructure''
 +
:* ''computation of dynamic view properties (cardinality, last update)''
  
 
==View Manager Client API==
 
==View Manager Client API==
−
* '''Description''': Java APIs for accessing the view-manager-service from within client runtimes.
+
* '''Description''': Client proxies for [[#View Manager API|View Manager]] endpoints.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
−
* [[View_Manager|'''Wiki Documentation''']]
+
* [[View_Manager|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''high-level proxy API''
−
:TODO
+
:* ''transparent endpoint discovery and caching''
 +
:* ''transparent fault tolerance over endpoint replicas''
 +
:* ''fluent proxy configuration API''
  
 
==Storage Manager API==
 
==Storage Manager API==
−
* '''Description''': TODO
+
* '''Description''': A client library for remote storage of files. The library allows clients to download, upload, remove, add, and list files. The file storage is based on MongoDB technology
 
* '''Type''': Java
 
* '''Type''': Java
−
* '''Protocol''': Local
+
* '''Protocol''': Custom
−
* [[Storage_Management_NEW|'''Wiki Documentation''']]
+
* [[Storage_Management_NEW|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:*''structured file storage: Clients can create folder hierarchies''
−
:TODO
+
:*''secure file storage: File access is authenticated, every files can have private, group, or public access rights''
 +
:*''scalable file storage: files are stored in chunks and chunks are distributed across clusters of servers based on the workload of individual servers;''
 +
:*''fault-tolerant file storage: files are asynchronously replicated across cluster servers for data recovery and redundancy''
  
 
==SPD API==
 
==SPD API==
−
* '''Description''': TODO
+
* '''Description''': Read access to pluggable sources of biodiversity data.
 
* '''Type''': Web Service
 
* '''Type''': Web Service
 
* '''Protocol''': SOAP
 
* '''Protocol''': SOAP
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
 
* '''Framework''': [https://gcore.wiki.gcube-system.org/gCube gCore]
−
* '''Wiki Documentation'''
+
* [http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases/org/gcube/data/spd/species-products-discovery-service/2.3.2-4.0.0-130321/species-products-discovery-service-2.3.2-4.0.0-130321-javadoc.jar|'''Documentation''']
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''access API tailored to biodiversity data sources''
−
:TODO
+
:* ''dynamically pluggable architecture of transformations (eg unified classification, synonyms presentation) from external sources of biodiversity data''
 +
:* ''plugins for key biodiversity data sources, including OBIS, GBIF and Catalogue of Life''
 +
:* ''dynamic clustering of discovered items''
 +
:* ''integration with workspace facilities''
 +
:* ''export of discovered items in multiple formats, including DarwinCore and DarwinCore Archive''
  
 
==SPD Plugin API==
 
==SPD Plugin API==
−
* '''Description''': TODO
+
* '''Description''': Development framework for [[#SPD API|SPD]] plugins.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
−
* '''Wiki Documentation'''
+
* [http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases/org/gcube/data/spd/spd-plugin-framework/3.0.0-4.0.0-126523/spd-plugin-framework-3.0.0-4.0.0-126523-javadoc.jar|'''Documentation''']
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''support for spd plugins implementation''
−
:TODO
+
:* ''partial interface implementations''
  
 
==SPD Client API==
 
==SPD Client API==
−
* '''Description''': TODO
+
* '''Description''': Client proxies for [[#SPD API|SPD]] endpoints.
 
* '''Type''': Java
 
* '''Type''': Java
 
* '''Protocol''': Local
 
* '''Protocol''': Local
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
 
* '''Framework''': [[Integration_and_Interoperability_Facilities_Framework:_Client_Libraries|CL]]
−
* '''Wiki Documentation'''
+
* [[Species_Product_Discovery:_client_library|'''Documentation''']]
 
* '''Key Features'''
 
* '''Key Features'''
−
:TODO
+
:* ''transparent endpoint discovery and caching''
−
:TODO
+
:* ''transparent fault tolerance over endpoint replicas''

Latest revision as of 18:38, 12 September 2016

Data Access and Storage APIs support both client and back-end access to a range of structured and unstructured data sources available in the Data e-Infrastructure. APIs include remote Web Service APIs, local client APIs, data model implementations, auxiliary APIs, and development frameworks for service plugins.

This document lists all the available APIs in these areas along with their key features, pointing to external documentation for details on their design and usage.

API Overview

Name Description Type Protocol Framework
Tree Manager Tree-based CRUD operations over pluggable sources of structured data WS SOAP gCore
Tree Manager Client Client proxies for Tree Manager endpoints Java Local CL
Tree Manager Plugin Development framework for Tree Manager plugins Java Local none
Trees Tree and tree pattern model implementations Java Local none
Streams Conversion, transformation, and publication of data streams Java Local none
gDoc gDoc model implementation Java Local none
gDoc Client gDoc-based client proxies for Tree Manager endpoints Java Local CL
View Manager Views over data sources accessible via Tree Manager endpoints WS SOAP gCore
View Manager Client Client proxies for View Manager endpoints Java Local CL
Storage Manager Access and storage of unstructured bytestreams, or files implementation Java Local none
Species Products Discovery (SPD) Read access to pluggable sources of biodiversity data WS SOAP gCore
SPD Plugin Development framework for SPD plugins Java Local CL
SPD Client Client proxies for SPD endpoints Java Local CL


The picture below offers an immediate view of the area and where each component is placed and which other interfaces:


DataManagementAPIs.png

Tree Manager API

  • Description: Tree-based CRUD operations over pluggable sources of structured data.
  • Type: Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Documentation
  • Key Features
  • lookup of whole trees as well as individual nodes
  • tree pattern matching on lookups and queries
  • in-place updates
  • streaming on all operations
  • native storage based on graph database technology

Tree Manager Client API

  • high-level proxy API
  • transparent endpoint discovery and caching
  • transparent fault tolerance over endpoint replicas
  • polling and notifications for asynchronous bindings
  • fluent proxy configuration API

Tree Manager Plugin API

  • support for a wide range of plugins: source-specific, source-generic, type-specific, type-generic, multi-type, etc
  • interface-based POJO framework, with no dependency on service implementation
  • transparent property publication and state persistence"
  • callback-based and event-based interaction with service"
  • full lifecycle support
  • partial interface implementations
  • derived implementation of node-based and streamed operations

Trees API

  • Description: Tree and tree pattern model implementations.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • untyped tree model with attributed nodes, labelled edges, and string-valued leaves
  • pattern language for characterisation of tree topology and leaf values
  • tree matching and pruning operations based on pattern language
  • tree difference and in-place update operations
  • fluent API for tree construction, tree navigation, and tree pattern construction
  • predefined XML bindings for trees and patterns
  • synthetic tree generation facilities for load testing

Streams API

  • Description: Conversion, transformation, and publication of data streams.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • high-level stream model based on Iterator-style interfaces
  • predefined implementations over standard Iterators and gRS2 resultsets
  • piping, folding, and unfolding stream operations
  • configurable error handling policies
  • stream lifecycle event notifications
  • publication as gRS2 resultset
  • fluent API for all stream operations

gDoc API

  • Description: gDoc model implementation.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • document model inclusive of metadata, annotations, parts, and alternatives
  • strongly-typed document construction
  • transparent bytestream resolution
  • annotation threading
  • new and proxy document modes
  • change tracking and staging for in-place updates without prior reads
  • XML binding facilities

gDoc Client API

  • creation of and access to gDoc document sources
  • bulk and streamed operations
  • caching, selection, and pruning for read operations
  • creation, discovery, and use of document views

View Manager API

  • Description: Views over data sources accessible via Tree Manager endpoints.
  • Type: Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Documentation
  • Key Features
  • view definition through arbitrary tree patterns
  • view characterisation through arbitrary properties
  • view publication in infrastructure
  • computation of dynamic view properties (cardinality, last update)

View Manager Client API

  • high-level proxy API
  • transparent endpoint discovery and caching
  • transparent fault tolerance over endpoint replicas
  • fluent proxy configuration API

Storage Manager API

  • Description: A client library for remote storage of files. The library allows clients to download, upload, remove, add, and list files. The file storage is based on MongoDB technology
  • Type: Java
  • Protocol: Custom
  • Documentation
  • Key Features
  • structured file storage: Clients can create folder hierarchies
  • secure file storage: File access is authenticated, every files can have private, group, or public access rights
  • scalable file storage: files are stored in chunks and chunks are distributed across clusters of servers based on the workload of individual servers;
  • fault-tolerant file storage: files are asynchronously replicated across cluster servers for data recovery and redundancy

SPD API

  • Description: Read access to pluggable sources of biodiversity data.
  • Type: Web Service
  • Protocol: SOAP
  • Framework: gCore
  • Documentation
  • Key Features
  • access API tailored to biodiversity data sources
  • dynamically pluggable architecture of transformations (eg unified classification, synonyms presentation) from external sources of biodiversity data
  • plugins for key biodiversity data sources, including OBIS, GBIF and Catalogue of Life
  • dynamic clustering of discovered items
  • integration with workspace facilities
  • export of discovered items in multiple formats, including DarwinCore and DarwinCore Archive

SPD Plugin API

  • Description: Development framework for SPD plugins.
  • Type: Java
  • Protocol: Local
  • Documentation
  • Key Features
  • support for spd plugins implementation
  • partial interface implementations

SPD Client API

  • Description: Client proxies for SPD endpoints.
  • Type: Java
  • Protocol: Local
  • Framework: CL
  • Documentation
  • Key Features
  • transparent endpoint discovery and caching
  • transparent fault tolerance over endpoint replicas
Retrieved from "https://gcube.wiki.gcube-system.org/index.php?title=Data_Access_and_Storage_APIs&oldid=26698"