Data Transfer Facilities

THIS PAGE IS UNDER CONSTRUCTION

Overview

The implementation of a reliable data transfer mechanisms between the nodes of the iMarine Data e-Infrastructure is one of the main objectives when dealing with large set of multi-type datasets distributed across different repositories, like the one made available by the iMarine EA Community of Practice.

To promote an efficient and optimized consumption of these data resources, a number of components have been designed to meet the data transfer requirements.

This document outlines the design rationale and high-level architecture of such components.

Key Features

The components part of the subsystem provide the following main key features:

reliable data transfer between Infrastructure Data Sources and Data Storages

by exploiting the uniform access interfaces provided by gCube and standard transfer protocols

structured and unstructured Data Transfer

it guarantees both Tree based and File based transfer to cover all possible iMarine use-cases

transfers to local nodes for data staging

data staging for particular use cases can be enabled on each node of the infrastructure

advanced transfer scheduling and transfer optimization

a dedicated gCube service responsible fot data transfer scheduling combined to transfer optimization at the level of protocols and Access interfaces.

transfer statistics availability

transfers are traced by the system and make available to interested consumers.

Main Components

the Data Scheduler & Agent components

this family of components guarantees VO/VRE Administrators the possibility to transfer data among Data Sources and Data Storages. It can be exploited as well by any client or web services to implements data movement between infrastructure nodes by relying as well on the facilities offered by the Result Set components

the Result Set components