OscarImporterSEPlugin
The Oscar-Importer-SE-Plugin is a Smart Executor plugin which periodically (currently every week) builds a merged version of the OSCAR [1] dataset and uploads it to the infrastructures Thredds server.
Following user requirements, the importer is not using the original dataset, but a version [2] where coordinates have been rearranged into -180 to 180 instead of 20 to 420 like the other OSCAR datasets
Contents
Deployment
The plugin has to be deployed on a Smart Executor node with at least 25GB of free disk space in /tmp. The disk space is required for files to be merged and for the merged file before uploading.
On the smart executor node, copy the jar-with-dependencies in /home/gcube/tomcat/webapps/smart-executor/WEB-INF/lib
Managing the service
The Smart executor needs to be restarted in order to discover and publish the plugin in the Information System.
stop the container
./stopContainer.sh
start the container
./startContainer.sh
Check
- check there are no exception in:
- SmartGears/ghn.log
- tomcat/logs/catalina.out
- tomcat/logs/localhost.log
- check that the profile are published in the IS; look at the infrastructure monitor (Service Endpoint -> VREManagement)
Starting the job
One shot
TODO
Scheduled
TODO
Under the hood
- To save disk space, the importer downloads small groups (currently 3) of yearly oscar files, merges them and proceeds iteratively.
- The merger uses /tmp/oscar-merger as working directory. All needed files (downloaded, merged, descriptors, temporary) are placed here. No other directory is used on the machine.
- Upon successful upload to Thredds, the work directory is cleaned with the exception of the last stable merged file (e.g. oscar-1992-2016.nc); this will speed-up next merge since only diff files (e.g. current year) will be downloaded and merged to it.
