Difference between revisions of "Data Transfer Scheduler & Agent components"
From Gcube Wiki
Andrea.manzi (Talk | contribs) (→Key features) |
Andrea.manzi (Talk | contribs) (→Key features) |
||
Line 11: | Line 11: | ||
The components belonging to this class are responsible for: | The components belonging to this class are responsible for: | ||
− | ;reliable data transfer between Data | + | ;reliable data transfer between Infrastructure Data Sources and Data Storages |
− | :by exploiting the access interfaces provided by gCube and | + | :by exploiting the uniform access interfaces provided by gCube and standard transfer protocols |
;structured and unstructured Data Transfer | ;structured and unstructured Data Transfer | ||
− | :it guarantees both Tree based and | + | :it guarantees both Tree based and File based transfer to cover all possible iMarine use-cases |
− | ;transfers to local nodes | + | ;transfers to local nodes for data staging |
+ | :data staging for particular use cases can be enabled on each node of the infrastructure | ||
− | ;advanced transfer | + | ;advanced transfer scheduling and transfer optimization |
+ | :a dedicated gCube service responsible fot data transfer scheduling combined to transfer optimization at the level of protocols and Access interfaces. | ||
+ | |||
+ | ;transfer statistics availability | ||
+ | :transfers are traced by the system and make available to interested consumers. | ||
== Design == | == Design == |
Revision as of 11:22, 19 April 2012
Intro
Contents
Overview
This class of components manages transfer capabilities among gCube infrastructure nodes, in particular but not only it can handles data transfer between Data Sources and Data Storages exploiting the interfaces and the services implemented under the Data Access and Storage Facilities subsystem.
This document outlines the design rationale, key features, and high-level architecture, the options for their deployment and as well some use cases.
Key features
The components belonging to this class are responsible for:
- reliable data transfer between Infrastructure Data Sources and Data Storages
- by exploiting the uniform access interfaces provided by gCube and standard transfer protocols
- structured and unstructured Data Transfer
- it guarantees both Tree based and File based transfer to cover all possible iMarine use-cases
- transfers to local nodes for data staging
- data staging for particular use cases can be enabled on each node of the infrastructure
- advanced transfer scheduling and transfer optimization
- a dedicated gCube service responsible fot data transfer scheduling combined to transfer optimization at the level of protocols and Access interfaces.
- transfer statistics availability
- transfers are traced by the system and make available to interested consumers.
Design
Philosophy
Architecture
fugyre
Deployment
Large Deployment
figure
Small Deployment
figure