Difference between revisions of "Data Transfer APIs"
From Gcube Wiki
Andrea.manzi (Talk | contribs) (→Data Transfer Agent Library) |
(→gRS2 Broker) |
||
(15 intermediate revisions by one other user not shown) | |||
Line 1: | Line 1: | ||
Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. | Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. | ||
− | The document outlines the design | + | The document outlines the design rationale for those APIs. |
==Overview== | ==Overview== | ||
Line 11: | Line 11: | ||
| align="center" style="background:#f0f0f0;"|'''Framework''' | | align="center" style="background:#f0f0f0;"|'''Framework''' | ||
|- | |- | ||
− | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Library Data Transfer Agent Library]||The CL for the Data Transfer Agent operations||Java||Local||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] | + | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Library Data Transfer Agent Library]||The CL proxy for the [[#Data_Transfer_Agent_Service |Data Transfer Agent Service]] operations||Java||Local||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] |
|- | |- | ||
| [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Service Data Transfer Agent Service]|| Structured/Unstructured transfer operations among data Source/Storages||gCube Web Service||SOAP||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_WSRF_API_Framework_Specification gCore] | | [https://gcube.wiki.gcube-system.org/gcube/index.php/Data_Transfer_APIs#Data_Transfer_Agent_Service Data Transfer Agent Service]|| Structured/Unstructured transfer operations among data Source/Storages||gCube Web Service||SOAP||[https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_WSRF_API_Framework_Specification gCore] | ||
Line 25: | Line 25: | ||
* '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] | * '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries CL] | ||
* '''Key Features''': | * '''Key Features''': | ||
− | + | :* ''synchronous/asynchronous transfers over data transfer service operations'' | |
− | :* '' | + | :* ''transfer monitoring'' |
− | :* '' | + | :* ''transfer outcomes to clients'' |
− | + | ||
− | :* '' | + | |
* [[Data_Transfer_Agent#Data_Transfer_Agent_Library|'''Documentation''']] | * [[Data_Transfer_Agent#Data_Transfer_Agent_Library|'''Documentation''']] | ||
==Data Transfer Agent Service== | ==Data Transfer Agent Service== | ||
− | * '''Description''': | + | * '''Description''': Structured/Unstructured transfer operations among data Source/Storages |
* '''Type''': gCube Web Service | * '''Type''': gCube Web Service | ||
* '''Protocol''': SOAP | * '''Protocol''': SOAP | ||
* '''Framework''': gCore | * '''Framework''': gCore | ||
* '''Key Features''': | * '''Key Features''': | ||
+ | :* ''transfer of local files from an external/internal client to a remote GHN'' | ||
+ | :* ''file transfer from a remote Data Source to a remote GHN using standard protocols ( HTTP, FTP, BitTorrent)'' | ||
+ | :* ''file transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard protocols ( HTTP, FTP, BitTorrent)'' | ||
+ | :* ''tree based data transfer from a remote Data Source to a remote tree-based data storage'' | ||
+ | :* ''transfer statistics publication trough messaging'' | ||
* [[Data_Transfer_Agent#Data_Transfer_Agent_Service|'''Documentation''']] | * [[Data_Transfer_Agent#Data_Transfer_Agent_Service|'''Documentation''']] | ||
Line 46: | Line 49: | ||
* '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_HTTP_API_Framework_Specification HTTP Front End] | * '''Framework''': [https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_HTTP_API_Framework_Specification HTTP Front End] | ||
* '''Key Features''': | * '''Key Features''': | ||
+ | :* ''results are exposed as an HTTP endpoint'' | ||
+ | :* ''results are in XML format'' | ||
+ | :* ''server response is compressed in order to achieve performance benefits'' | ||
+ | :* ''the records are retrieved iteratively - gradually'' | ||
+ | :* ''number of results can be customized (max parameter)'' | ||
+ | :* ''files are stored locally on the server (garbage collection)'' | ||
* Wiki Doc: | * Wiki Doc: |
Latest revision as of 19:15, 2 July 2012
Data Transfer APIs provide access to transfer mechanisms for large sets of multi-type datasets distributed across the system. The document outlines the design rationale for those APIs.
Contents
Overview
APIs
Name | Description | Type | Protocol | Framework |
Data Transfer Agent Library | The CL proxy for the Data Transfer Agent Service operations | Java | Local | CL |
Data Transfer Agent Service | Structured/Unstructured transfer operations among data Source/Storages | gCube Web Service | SOAP | gCore |
gRS2 Broker | This component exposes the gCube Resultset as an http endpoint | Java Servlet | REST | HTTP Front End |
Data Transfer Agent Library
- Description: The CL for the Data Transfer Agent operations
- Type: Java
- Protocol: Local
- Framework: CL
- Key Features:
- synchronous/asynchronous transfers over data transfer service operations
- transfer monitoring
- transfer outcomes to clients
Data Transfer Agent Service
- Description: Structured/Unstructured transfer operations among data Source/Storages
- Type: gCube Web Service
- Protocol: SOAP
- Framework: gCore
- Key Features:
- transfer of local files from an external/internal client to a remote GHN
- file transfer from a remote Data Source to a remote GHN using standard protocols ( HTTP, FTP, BitTorrent)
- file transfer from a remote Data Source to the gCube Storage Manager ( MongoDB) using standard protocols ( HTTP, FTP, BitTorrent)
- tree based data transfer from a remote Data Source to a remote tree-based data storage
- transfer statistics publication trough messaging
gRS2 Broker
- Description: This component exposes the gCube Resultset as an http endpoint
- Type: Java Servlet
- Protocol: REST
- Framework: HTTP Front End
- Key Features:
- results are exposed as an HTTP endpoint
- results are in XML format
- server response is compressed in order to achieve performance benefits
- the records are retrieved iteratively - gradually
- number of results can be customized (max parameter)
- files are stored locally on the server (garbage collection)
- Wiki Doc: