Motu Client Java

Usage

Configure your project

Maven coordinates:

  <dependency>
    <groupId>org.gcube.dataanalysis</groupId>
    <artifactId>motu-client</artifactId>
    <version>[1.0.0, 2.0.0)</version>
  </dependency>

Create a client

As a first step, you need to create a MotuClient object providing enough information to connect to the corresponding server:

  MotuClient client = new MotuClient("server endpoint");
  client.setUsername("username");
  client.setPassword("password");

Setting the preferred download size

You can optionally specify the size of chunks to be downloaded. If not provided, the client will use the maximum size allowed by the server (currently most servers allow between 1 and 2 GB).

  client.setPreferredDownloadSize(20*SizeUtils.MB);

Retrieving the product catalogue

A full catalogue containing all services published by the Motu server can be obtained with the code below.

Note: this might take some time to process as it recursively goes into nested server resources. If you only need some specific information, query for the specifically, as described in the following sections.

  // retrieve the catalogue (it might take some time)
  MotuCatalogue catalogue = client.getCatalogue();
 
  // get available services  Collection<ServiceMetadata> services = catalogue.getServices();

A service usually contain different products (datasets). Typically there are products for different time resolution and/or different variables.

  // get the list of products for a service
  Collection<ProductMetadataInfo> products = service.getProducts();

Product metadata can be obtained with:

  ProductMetadataInfo product = ...;
 
  // get timestamps for which there are data available
  List<Calendar> times = product.getAvailableTimeCodes(); 
  // the oldest timestamp in the dataset
  Calendar start = product.getFirstAvailableTimeCode();
 
  // the most recent timestamp in the dataset
  Calendar end = product.getLastAvailableTimeCode();
 
  // get the time resolution of the dataset (in hours)
  Long hours = product.getTimeResolution();
 
  // get a list of depths for which there are data available
  List<Double> depths = product.getAvailableDepths();
 
  // get the lowest depth value (i.e. closer to the surface)
  Double depth = product.getFirstAvailableDepth();
 
  // get the highest depth value (i.e. the deeper level)
  Double depth = product.getLastAvailableDepth();
 
  // get the set of variables in the dataset
  Collection<Variable> variables = product.getVariables();
 
  // get the dimensions of the dataset (e.g. lat, lon, depth)
  Collection<Axis> axes = product.getDataGeospatialCoverage();

Note: If you plan to deliver the catalogue in an interactive way (e.g. in a GUI), please consider caching to improve the user experience.

Getting the list of services

Instead of retrieving the full catalogue (it might be slow), you might want to get the list of available services, without any nested product information:

  // get a shallow list of available services
  Collection<ServiceMetadata> services = client.listServices();

Getting product metadata

Similarly, once you know the service and product name, you can retrieve product metadata directly:

  // get product metadata by service and product name
  ServiceMetadata service = ...;
  ProductMetadataInfo product = describeProduct(service, "productName");
   // or...
  ProductMetadataInfo product = describeProduct("serviceName", "productName");

Downloading products

Products can be downloaded using two approaches:

• the legacy Motu API (asynchronous, with limited download size);
• the enhanced gCube API (synchronous, without any limitation on the download size).

Building a download request

Once you've identified the product you want to download, a download request has to be prepared with references to the product, bounding box, time frame and variables.

  // build a download request
  DownloadRequest request = new DownloadRequest();
 
  // the service name is mandatory  request.setService("MEDSEA_ANALYSIS_FORECAST_PHYS_006_001-TDS");
 
  // product name is also mandatory
  request.setProduct("cmemsv02-med-ingv-cur-an-fc-d");
 
  // optionally, set the longitude range. If not set, the whole extent is downloaded
  request.setxRange(15d, 20d);
 
  // optionally, set the latitude range. If not set, the whole extent is downloaded
  request.setyRange(35d, 40d);
 
  // optionally, set the depth range. If not set, the whole depth extent is downloaded
  request.setzRange(1.4721018075942993, 5334.64794921875);
 
  // optionally, set the time window. If not set, all timestamps are downloaded
  request.settRange("2014-01-01 00:00:00", "2014-01-30 00:00:00");
 
  // this seems to be mandatory.
  request.setScriptVersion("1.4.00-20170410143941999");
 
  // process the request in asynchronous mode.
  request.setMode("status");
 
  // the output format (no alternatives here)
  request.setOutput("netcdf");
 
  // optionally set which variables to download. If not set, all of them are downloaded
  request.addVariable("vomecrty");
  request.addVariable("vozocrtx");

Getting the estimated download size

Motu has an operation to estimate the size of a download request:

  // get the estimated output size
  RequestSize requestSize = client.getSize(request);
 
  // here is the size (in bytes)
  Long actualSize = requestSize.getSizeInBytes();
 
  // here is also the maximum allowed download size
  Long maxSize = requestSize.getMaxAllowedSizeInBytes();

When using the legacy Motu download API, the size must be smaller than the one allowed by the server. If higher you'll need to shrink the request (either bounding box and/or time span and/or variables).

Using the legacy Motu API

Once a request has been built and the result size is within allowed limits, you can queue the download request:

  // queue the request and returns immediately (async call)
  StatusModeResponse submitStatus = client.queueProductDownload(reques);

Then, the status of remote processing can be checked with:

  // fetch the id of the request
  String requestId = submitStatus.getRequestId();
 
  // ask the server for an updated status
  StatusModeResponse status = client.checkStatus(requestId);

The status of the request is encoded in the 'status' property, with the same values returned by the Motu server. Possible values are:

 0: the request is being processed
 1: the file is ready for download
 2: an error occurred
 3: the request is pending

The status can be checked with the following methods:

  // check for specific status
  boolean r = status.isReady();
  boolean e = status.isError();
  boolean p = status.isInProgress();
 
  // or get the status code
  String code = status.getStatus();

When the request reaches the 'ready' status, the dataset is ready for download. The URL is also included in the status message:

  URI uri = status.getRemoteUri();

Note: the file is usually available on the remote server for a limited amount of time (TODO: quantify this).

Under the hood

TODO