OscarImporterSEPlugin
From Gcube Wiki
Revision as of 15:36, 6 July 2017 by Paolo.fabriani (Talk | contribs) (Created page with "The Oscar-Importer-SE-Plugin is a Smart Executor plugin which periodically (currently every week) builds a merged version of the OSCAR <ref>https://podaac.jpl.nasa.gov/dataset...")
The Oscar-Importer-SE-Plugin is a Smart Executor plugin which periodically (currently every week) builds a merged version of the OSCAR [1] dataset and uploads it to the infrastructures Thredds server.
Following user requirements, the importer is not using the original dataset, but a version [2] where coordinates have been rearranged into -180 to 180 instead of 20 to 420 like the other OSCAR datasets
Contents
Deployment
The plugin has to be deployed on a Smart Executor node with at least 25GB of free disk space in /tmp
On the smart executor node, copy the jar-with-dependencies in /home/gcube/tomcat/webapps/smart-executor/WEB-INF/lib
Managing the service
Load the plugin
- stop the container (
- start the container
Check
- check there are no exception in:
- SmartGears/ghn.log
- tomcat/logs/catalina.out
- tomcat/logs/localhost.log
- check that the profile are published in the IS; look at the infrastructure monitor (Service Endpoint -> VREManagement)
Starting the job
One shot
TODO
Scheduled
TODO
Under the hood
- To save disk space, the importer downloads small groups (currently 3) of yearly oscar files, merges them and proceeds iteratively.
- The merger uses /tmp/oscar-merger as working directory. All needed files (downloaded, merged, descriptors, temporary) are placed here. No other directory is used on the machine.
- Upon successful upload to Thredds, the work directory is cleaned with the exception of the last stable merged file (e.g. oscar-1992-2016.nc); this will speed-up next merge since only diff files (e.g. current year) will be downloaded and merged to it.
