Difference between revisions of "GCube Web Specifications and Standards Compliance"

Latest revision as of 17:13, 6 July 2016

This area collects the Standard Specifications supported by the gCube system APIs, as part of the WP11 activities and towards meeting the integration and interoperability objectives for promoting the openness of the e-Infrastructure to other neighbouring and external ones. The collection focuses on the widely used, HTTP-based Specifications and generic interchange protocols (data/content standards, metadata standards, Web interface standards, security standards, data sharing protocols, data transfer protocols) that service both disseminating and consuming system's needs. This analysis is conducted per functional category and addresses the use, need and relevance of the standards that fall under each gCube functional area.

Table of Protocols

• Functional areas: Data Consumption / Data Production / Computation Consumption / Infrastructure Management / Data Transfer

• Direction: Producer / Consumer

• Adoption Status: Completed / On going / Planned

OAI-PMH

Specification Description

The Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) is a well-established standard in the content management and library science worlds that is gaining in importance. It provides a low-barrier mechanism for repository interoperability and defines the following parties and software components:

• Data Providers are repositories that expose structured metadata via OAI-PMH. A 'Data Provider' such as an academic library runs a Repository that supports OAI-PMH as a means of exposing metadata information about resources, for instance academic publications.
• Service Providers then make OAI-PMH service requests to harvest that metadata. A 'Service Provider' uses Harvester software to harvest metadata from Data Providers. The harvested metadata can then be used to provide valued-added services, such as a website that allows browsing and searching through their catalog.

OAI-PMH is a set of six verbs or services that are invoked within HTTP. An implementation of OAI-PMH must support representing metadata in Dublin Core, but may also support additional representations.

gCube Use/Need/Relevance

Through OAI-PMH protocol, gCube infrastructure acts as a 'Data Provider' and disseminates the hosted metadata records in a standard fashion, thus allowing for interoperation with other data e-Infrastructures that run autonomously. Other infrasturctures can harvest the metadata descriptions of gCube content in archives so that their services can exploit the collections. The protocol provides an application-independent interoperability framework for metadata exchange between the online parties.

Functional Category

Data Consumption

Direction

• Producer

gCube Adoption Status

The protocol has already been integrated in the gCube system, from the 'Data Provider' perspective. The description of the adopted methodology towards the integration is described here.

Related gCube components

• aslHttp OAI_PMH: the http front end for the protocol
• applicationSupportLayer_OAI_PMH: business logic back-end component for the protocol

References

• Open Archives Initiative Protocol for Metadata Harvesting Home Page

OAI-ORE

Specification Description

Open Archives Initiative Object Reuse and Exchange (OAI-ORE) defines standards for the description and exchange of aggregations of Web resources. These aggregations, sometimes called compound digital objects, may combine distributed resources with multiple media types including text, images, data, and video. The goal of these standards is to expose the rich content in these aggregations to applications that support authoring, deposit, exchange, visualization, reuse, and preservation. Although a motivating use case for the work is the changing nature of scholarship and scholarly communication, and the need for cyberinfrastructure to support that scholarship, the intent of the effort is to develop standards that generalize across all web-based information.

In the physical world we create, use, and refer to aggregations of things all the time. We collect pictures in a photo album, read journals that are collections of articles, and burn CDs of our favorite songs. This practice of aggregating extends to the Web. We accumulate URL's in bookmarks or favorites lists in our browser, collect photos into sets in popular sites, browse over multiple page documents that are linked together through "prev" and "next" tags, and talk about Web sites as if they had some real existence beyond the set of pages of which they consist. Despite our frequent use of these aggregations, their existence on the Web is quite ephemeral. One reason for this is that there is no standard way to identify an aggregation. We often use the URI of one page of an aggregation to identify the whole aggregation. For example, we use the URI of the first page of a multi-page Web document to identify the whole document, or we use the URI of the HTML page that provides access to a collection of images to identify the entire set of images. But those URIs really just identify those specific pages, and not the union of pages that makes up the whole document, or the union of all images in a Flickr set, respectively. In essence, the problem is that there is no standard way to describe the constituents or boundary of an aggregation, and this is what OAI-ORE aims to provide.

gCube Use/Need/Relevance

gCube Content Model aims to provide high-level functionality for manipualtion of content over the Grid-based environments. Content in gCube is stored and organized following a graph-based data model, the Information Object Model, that allows finer control of content, by incorporating the possibility to annotate content with arbitrary properties an to relate different content unities via arbitrary relationships.

Starting from this model a document model has been built, in which complex documents, composed of various, eventually nested subparts, are represented as chains of Information Objects linked via appropriate relationships. For instance, an HTML document that includes a number of images may be modelled as a complex object that provides references to Information Objects (containing the images). In this respect, gCube documents are managed as compound objects comprising metadata, annotations, alternative representations and multiple parts. The notion of gCube documents is implemented and mangaged by the gCube Information Organisation Services family of subsystems that include storage services, access services, plugins and a number of distinguished clients that can be internal or external to the system.

The aggregated information that constructs a gCube document can be transfered through the solution provided by OAI-ORE, without the need for clients to rely on the API's of the individual system architectures and their definition of document boundaries. The gCube ORE Provider allows the dissemination of the digital objects stored in gCube repository as OAI-ORE Resource Maps.

Functional Category

Data Consumption

Direction

• Producer

gCube Adoption Status

The protocol has been recently integrated in the gCube System, from the producer perspective. The description of the adopted methodology towards the adoption and implementation is described here.

Components affected / relevant

• aslHttp ORE_Provider: the http front end for the protocol
• applicationSupportLayer_OAI_ORE: business logic back-end component for the protocol

References

• Open Archives Initiative Object Reuse and Exchange
• OAI-ORE Primer

OpenSearch

Specification Description

OpenSearch is a collection of technologies that allow publishing of search results in a format suitable for syndication and aggregation. It is a way for websites and search engines to publish search results in a standard and accessible format. OpenSearch helps search engines and search clients communicate by introducing a common set of formats to perform search requests and syndicate search results. The five basic pieces of information that a search client needs in order to communicate effectively with a search engine supporting the protocol are:

• local search engine location
• the query-grammar expected
• the request encoding
• the response encoding
• the record encoding

The OpenSearch protocols define what a description document looks like, but not how it is retrieved. The location of the description document is discovered by some means outside the protocol (a priori knowledge). The description document specifies the location of the local search engine, how to formulate the search URL, and the local search engine's language to which the queries submitted must comply.

gCube Use/Need/Relevance

As a producer, gCube can publish search results in a standard and accessible format, thus allowing metasesarch engines to know how to send a search to the gCube search engine and how to interpret the results. As a consumer, gCube can access external providers which publish their results through search engines conforming to the OpenSearch Specification. Therefore, it can act as a metasearch engine that combines results coming from a gCube search with the results from searching many other sites simultaneously, providing a high level of integration.

Functional Category

Data Consumption

Direction

• Producer: OpenSearch interface over gCube search
• Consumer: OpenSearch Framework accessing external OpenSearch providers

gCube Adoption Status

The protocol has been recently integrated in the gCube System, both from the producer and the consumer perspective. The description of the adopted methodology towards the adoption and implementation in the side of the producer, is described here. The consumer side functionality of the gCube OpenSearch implementation is concentrated in the OpenSearch framework, whose description and features analysis can be found here

Components affected / relevant

• aslHttp Information Retrieval - OpenSearch: the http front end for the protocol that allows the gCube search engine to act as an OpenSearch Provider
• OpenSearch Library: framework that includes a core library providing general-purpose OpenSearch functionality, and the OpenSearch Operator which utilizes functionality provided by the former
• OpenSearch Service: the web service responsible for the invocation of the OpenSearch Operator in the context of the provider to be queried

References

• OpenSearch

SRU

Specification Description

SRU is a standard XML-focused search protocol for Internet search queries, utilizing CQL (Contextual Query Language), a standard syntax for representing queries. As in OpenSearch, the five basic pieces of information provided by the mechanisms of the protocol to a client trying to communicate with the search engine, are: (1) local search engine location, (2) the query grammar expected, (3) the request encoding, (4) the response encoding, and (5) the record endoding. SRU expects that the content provider will have a description record that describes the search service. The protocols define what a description record looks like and specifies that it can be obtaines from the local search engine. The location of the local search engine is provided by means outside the protocol (a priori knowledge). SRU defines also how to formulate the search URL by defining it, and specifies a standard query grammarL CQ: (Common Query Language). This means that clients of the engine only have to write one translator for all the SRU local search engines but also that all SRU local search engines have to support the CQL query grammar.

gCube Use/Need/Relevance

The gCube Search engine supports CQL as its native query grammar, thus complying fully to the SRU requirements for query formulation. Providing and interface and the description mechanisms defined by the protocol would allow all SRU metasearch engines to access gCube results in a standard way and with a high integration level.

In the consuming side of the protocol, gCube would act as a metasearch engine for external search engines. An SRU provider integrated in the gCube system , would allow the dissemination of its results coming along with gCube results, within a single search in gCube system. The mechanism for the registration of the information that describes the SRU search engines in the gCube system, can be integrated with the one already implemented for OpenSearch providers, since the requirements for effective communication with external search engines are common in both protocols.

Functional Category

Data Consumption

Direction

• Producer: OpenSearch interface over gCube search
• Consumer: OpenSearch Framework accessing external OpenSearch providers

gCube Adoption Status

The protocol is planned to be integrated within the system, covering both consumer and producer sides and exploiting the already implemented mechanisms for OpenSearch providers subscription to the system.

Components affected / relevant

• aslHttp Information Retrieval - SRU: the http front end for the protocol that allows the gCube search engine to act as an SRU provider
• SRU - OpenSearch Convertor: the mechanism that will be converting the response of an 'explain' request in SRU protocol to the equivalent OpenSearch Description document, to register the provider information within the gCube System
• To be defined.

References

• Search Retrieval via URL

FTP/FTPS/SFTP

Specification Description

File Transfer Protocol (FTP) is a standard network protocol used to transfer files from one host or to another host over a TCP-based network, such as the Internet. FTP is built on a client-server architecture and uses separate control and data connections between the client and the server. FTP users may authenticate themselves using a clear-text sign-in protocol, normally in the form of a username and password, but can connect anonymously if the server is configured to allow it. For secure transmission that hides (encrypts) the username and password, and encrypts the content, FTP is often secured with SSL/TLS FTPS. SSH File Transfer Protocol SFTP is sometimes also used instead.

gCube Use/Need/Relevance

The access to servers in order to perform File transfer using standard protocols has to be provided in gCube in the context of the Data Transfer activity. In this particular case the usage of the FTP protocols family can be foreseen for both accessing external file repositories and as internal transfer protocol between nodes of the infrastructure.

Functional Category

Data Transfer

Direction

Consumer

gCube Adoption Status

The FTP protocol has been already integrated in the Data Transfer Agent Service. The secure versions of the protocol ( FTPS, SFTP) are also integrated but they requires the deployment of a secure infrastructure to be exploited ( on-going activity)

Components affected / relevant

• Data Transfer Agent Service: The component responsible for transferring data using standard protocols

References

• Specification
• gCube Wiki pointer

HTTP/HTTPS

Specification Description

HTTP functions as a request-response protocol in the client-server computing model. A web browser, for example, may be the client and an application running on a computer hosting a web site may be the server. The client submits an HTTP request message to the server. The server, which provides resources such as HTML files and other content, or performs other functions on behalf of the client, returns a response message to the client. The response contains completion status information about the request and may also contain requested content in its message body.

Hypertext Transfer Protocol Secure (HTTPS) is a widely used communications protocol for secure communication over a computer network, with especially wide deployment on the Internet. Technically, it is not a protocol in itself; rather, it is the result of simply layering the Hypertext Transfer Protocol (HTTP) on top of the SSL/TLS protocol, thus adding the security capabilities of SSL/TLS to standard HTTP communications.

gCube Use/Need/Relevance

The access to servers in order to perform File transfer using standard protocols has to be provided in gCube in the context of the Data Transfer activity. In this particular case the usage of the HTTP protocols family can be foreseen for accessing external repositories containing archives of Environmental data.

Functional Category

Data Transfer

Direction

Consumer

gCube Adoption Status

The HTTP protocol has been already integrated in the Data Transfer Agent Service. The secure version of the protocol ( HTTPS ) is also integrated but it requires the deployment of a secure infrastructure to be exploited ( on-going activity)

Components affected / relevant

• Data Transfer Agent Service: The component responsible for transferring data using standard protocols

References

• Specification
• gCube Wiki pointer

Web Services Data Access and Integration – The XML Realization (WS-DAIX) Specification

Specification Description

The WS-DAI (Web Service Data Access and Integration) family of specifications defines web service interfaces to data resources, such as relational or XML databases. The specifications have been developed by the Database Access and Integration Services Working Group of the Grid Forum, and can be used independently, or as part of a wider service-based grid architecture. The specifications include properties that can be used to describe a data service or the resource to which access is being provided, and define message patterns that support access to (query and update) data resources. The specifications include a data model-independent specification (WS-DAI), which is extended in two realizations to include model-dependent properties and operations in relational(WS-DAIR) and XML (WS-DAIX) specifications.

The XML realization extends the properties defined in WS-DAI, instantiates the templates for direct and indirect data access, and defines interfaces for accessing the results of requests directed at a XML data service. Key features include: properties and operations for manipulating collections of documents, direct and indirect access using both XQuery and XPath, and data modification using XUpdate.

gCube Use/Need/Relevance

gCube has a general purpose Information System whose handled information may be consumed by clients from outside the gCube infrastructure. WS-DAIX allows to consume such information in a standard way.

In addition, the Information System may act as a generic indexer of raw XML documents and components can use it to index and query their own data trough the exposed WS-DAIX operations.

Functional Category

Web interfaces

Direction

Producer and consumer

gCube Adoption Status

WS-DAIX's XMLCollectionAccess portType (feeding and retrieval of XML documents) has been fully implemented by the Information Collector Service. The same service exposes the XQueryExecute portType nearly compliant with the analogous portType defined by the standard.

The IS-Publisher library is both a producer and consumer of the XMLCollectionAccess portType, while the IS-Client library is a consumer of the XQueryExecute portType.

Components affected / relevant

• IS-InformationCollector: producer
• IS-Publisher: producer/consumer
• IS-Client: consumer

References

• Specification
• gCube Wiki pointer

WS-BaseNotification/WS-Topics

Specification Description

WS-BaseNotification is a standard developed by OASIS defining an event driven programming model between web services.

WS-BaseNotification enables publish and subscribe communication patterns to be exposed as WS-Topisc by using web services in an SOA environment. These patterns allow services to spread information to other web services, without prior knowledge of those web services. The standard promotes easy exchange of data (encapsulated in a WS-Topic) between suppliers and customers through use of web service operations and prevents vendor lock-in or adoption of proprietary standards.

gCube Use/Need/Relevance

In a highly dynamic environment such as the gCube Data Infrastructure, it is crucial for any actor to keep trace of changes occurring to the infrastructure itself. Instances of services, nodes, data sources and so on pop up and disappear continuously and their consumers must have a mechanism for receiving notifications about their lifetime. The IS-Notifier service (which also extends Base Notification to define a broker role, very close to the one defined by the WS-BrokeredNotification standard) implements the WS-BaseNotification standard and exposes WS-Topics on gCube resources' lifetime. Its subscribers are notified about any change on the topics through the facilities offered by the IS-Notification library.

Functional Category

Web interfaces

Direction

Producer and consumer

gCube Adoption Status

The notification mechanism has been part of the gCube System from its inception and exploited since many years.

Components affected / relevant

• IS-Notifier: producer
• IS-Notification: consumer

References

• Specification (WS-BaseNotification)
• Specification (WS-Topics)
• gCube Wiki pointer

WFS

Specification Description

Description and useful information about the Specification.

gCube Use/Need/Relevance

Describe the use/need/relevance of the specification in respect to the functional area of the system.

Functional Category

The functional category under which the services underlying the protocol fall.

Direction

The direction towards the system (Producer/consumer), along with any information to clarify the perspective of the interpretation as one or the other or both, if needed

gCube Adoption Status

Information about status of adoption of Specification within Our system. Whether the specification has already been integrated and supported within the system, or it is under implementation, or soon to be implemented.

Components affected / relevant

• component X: role

References

• Specification
• gCube Wiki pointer

WMS

Specification Description

Description and useful information about the Specification.

gCube Use/Need/Relevance

Describe the use/need/relevance of the specification in respect to the functional area of the system.

Functional Category

The functional category under which the services underlying the protocol fall.

Direction

The direction towards the system (Producer/consumer), along with any information to clarify the perspective of the interpretation as one or the other or both, if needed

gCube Adoption Status

Information about status of adoption of Specification within Our system. Whether the specification has already been integrated and supported within the system, or it is under implementation, or soon to be implemented.

Components affected / relevant

• component X: role

References

• Specification
• gCube Wiki pointer

SDMX

Specification Description

The Statistical Data and Metadata Exchange (SDMX) initiative sets standards that can facilitate the exchange of statistical data and metadata using modern information technology, with an emphasis on aggregated data. The standards specifically support a data-sharing process based on the use of central registry services. Registry services provide visibility into the data and metadata existing within a group of organizations. The data or metadata itself is not stored in a central registry – these services merely provide a useful set of metadata about the data (and additional metadata) in a known location, so that users/applications can easily locate and obtain whatever data and/or metadata is registered. The use of standards for all data, metadata, and the registry services themselves is ubiquitous, permitting a high level of automation within a data-sharing community.

gCube Use/Need/Relevance

An SDMX Registry implementation will be made available in order to allow the exchange of statistical data and metadata. The following SDMX Registry capabilities will be developed:

• Querying of structural metadata and data
• Maintenance of Structural Metadata
• Registration of Data Sources

Functional Category

Web Interfaces

Direction

Producer and Consumer

gCube Adoption Status

An implementation of SDMX Registry REST services is under development and has not yet been integrated within the system. SDMX specification version 2.0 will be taken into account for the development of the registry services.

Components affected / relevant

• SDMX Registry: the component responsible for acting as a central hub for the provisioning of statistical data and metadata between multiple organizations

References

• SDMX Initiative website

Protocol XX

Specification Description

Description and useful information about the Specification.

gCube Use/Need/Relevance

Describe the use/need/relevance of the specification in respect to the functional area of the system.

Functional Category

The functional category under which the services underlying the protocol fall.

Direction

The direction towards the system (Producer/consumer), along with any information to clarify the perspective of the interpretation as one or the other or both, if needed

gCube Adoption Status

Information about status of adoption of Specification within Our system. Whether the specification has already been integrated and supported within the system, or it is under implementation, or soon to be implemented.

Components affected / relevant

• component X: role

References

• Specification
• gCube Wiki pointer

Tentative Compliance

Add here specifications that are not there, neither the project commits yet into supporting them, along with the need and relevance.

• LDAP: Support integration of infrastructure structure with other systems (e.g. harvesting external infrastructure resources, or publishing D4Science infrastructure resources ).
• WSDM: Provide standard's compliant web API for infrastructure management.