Revision as of 13:50, 4 May 2016

The integration of the gCube system with FedCloud and, in general, with research/commercial cloud providers, aims to add elasticity to the D4Science infrastructure by allowing programmatic and dynamic creation, configuration and decommissioning of cloud resources on external infrastructures. The benefits of this integration facility for the gCube system are threefold:

• to exploit infrastructural resources in a pay-per-use model
• to scale in/out quickly in response to specific needs
• to reduce manual interventions of Infrastructure Managers.

Scenarios

• the VRE Manager, according to the demand of data analysis executions in the VRE, decides the number of SmartExecutor nodes needed in the VRE. Through the VRE administration portal, the VRE Manager chooses (from a list of templates) to create the required number of SmartExecutors. She/he chooses on which infrastructure the resources should be created and their characteristics in terms of computation and/or storage capacity. The gCube system, via the integration facility here described, will automatically create the new nodes, register them to the D4Science infrastructure and make them available in the VRE.
• the gCube system intelligently adapt the number of SmartExecutor nodes to the actual demand. In the VRE administration portal, the VRE Manager decides the minimum and maximum number of SmartExecutor expected for the VRE. Then, the gCube system via the integration facility here described, automatically scales-in/out the pool of SmartExecutors (by creating/destroying nodes) predicting the demand for SmartExecutors by analysing VRE accounting and monitoring data.

Requirements

It is possible to recognize the following functionalities that should be available in the integration facilities:

Functional requirements

• Registration and management of cloud providers, access credentials and cloud resources templates on a per-VRE basis. Credentials must be securely stored in the infrastructure;
• Cloud resource lifecycle management. Creation, configuration and decommissioning of cloud resources must be supported. The integration facility allows to use templates for the configuration of cloud resources (e.g. SmartExecutor Template, cpu and memory characteristics);
• Accountability and monitorability. Accounting and monitoring data on usage and workload of resources created on the cloud must be collected and made available in the infrastructure. This is both a requisite to comply with D4Science infrastructure policies and for the automatic scale-in/out;
• User interface. The user interface is integrated in the VRE administration portal since it already contains all the other VRE administration tools

Non-Functional requirements

The solution should have modular architecture in order to support the integration of new cloud providers in a simple and pluggable way.

Architecture

The architecture highlights a number of integration-specific components and their interaction among them as well as with existing components, either belonging to the D4Science infrastructure or to the EGI infrastructure. They are described in the subsections above.

Cloud Libraries

are third-party software providing language-specific APIs and data model to easily interact with clouds from within applications. Although they usually support a number of different clouds and cloud standards, there's no universal coverage for any of them; nor the the API and data model they expose is uniform across libraries.

Connectors

are built on top of cloud libraries in order to abstract the specifics of their APIs and data models and expose a uniform interface to the upper layers. Connectors do not interact with any other service nor are expected to persist any information.

Federated Hosting Node Manager (FHNM)

is the core part of the integration and is the place where all the business logic resides. It's the gateway for all the operations related to the management of external cloud infrastructures, via the most-appropriate connector/library; it manages connectors to the available clouds; it gathers accounting and status data and publishes them to the specific D4Science services.

Federated Hosting Node Manager Portlet

provides infrastructure and VRE administrators with a dashboard and a control panel to easily monitor and manage resources in external infrastructures. It enables administrators to register cloud infrastructures and credentials associated with them as well as virtual appliances and service profiles.

D4Science Information System

Collects, holds and provides all the information related to the D4Science infrastructure. In particular, for the purpose ot the integration of FedCloud, it holds the list of available cloud sites along with credentials to access them, the list of running cloud resources and their status and the list of virtual appliances available for instantiation.

D4Science monitoring system

Data collected is analysed to produce alarms and/or take countermeasures in case of problems with one or more resources

The Virtual Organization Membership Service (VOMS)

Authorization attributes to be embedded in X.509 proxy certificates, needed to access FedCloud sites.

EGI FedCloud sites

Host cloud resources instantiated through the components

AppDb

Maintains the set of available virtual appliances

Service Description

The solution has been developed by means of Java since it is the same technology of the gCube system and it made easier to integrate in the D4Science infrastructure and furthermore there is an high availability for third-party libraries implementing cloud standards.

A brief description of the service is visible in the following subsections.

Data Model

The datamodel has been designed with the aim of representing the information related to cloud providers, nodes, templates, accounting and monitoring in a homogenous way independently from the cloud provider exposed API and technology. Connectors and third-party Cloud Libraries ensured a proper translation between the internal datamodel and the specific cloud provider model.

A detailed class diagram of the adopted datamodel is available in the following picture.

FHN Manager

The FHNManager is a web service implemented in Java and running in a gCube SmartGears container. It exposes a REST API that allows to access data on nodes and cloud providers as well as to execute operations to create, destroy, start and stop nodes. The complete API specification is reported in Appendix II.

In order to account the usage of cloud provider resources by the D4Science infrastructure, the service relies on the gCube accounting framework provided by the infrastructure: new usage records have been defined to track changes in the lifecycle of the resources on the cloud. Through the connectors, the service also support the fetching of accounting data from the cloud provider (when available).

Concerning the monitoring data, the service implements the support to fetch it from the cloud provider (if available) as well as collect data directly from the resource via the gCube monitoring framework.

The current implementation of the service - and thus the portlet - does not support the registration of new cloud providers along with their associated credentials. However, since the registration entries for cloud providers are maintained in the D4Science Information System, VRE/Infrastructure managers can still use the editing functionalities provided by the Information system itself, although the editing capabilities provided are not tailored to the specific task.

Since the FHNManager is deployed in a SmartGears container, it is automatically registered to the D4Science infrastructure and it is automatically authorized to exploit infrastructure capabilities (Information, Accounting and Monitoring). In addition to properly generate and use X.509 proxy certificates, the host must also be equipped with VOMS clients (i.e. voms-proxy-init) and configured with trusted root certification authorities as distributed by EUGridPMA.

FHN Manager Portlet

The FHNManagerPortlet is a portlet offering facilities to exploit the FHNManager capabilities. Target users of this portlet are both infrastructure and VRE Managers, that will use it in order to monitor and control the virtual machines created via FHNManager service.

Built on top of gCube framework and according to Liferay specifications, the portlet uses these technology to discover and connect to the FHNManager service, gather information on the current status of the federated infrastructure, modify it through service's API and sharing information with other users through the portal's social facilities implemented by gCube.

Portlet main use cases are: CRUD operations on FHNManager data model entities start/stop of federated hosting nodes report generation and sharing

The portlet implementation is based on the GWT framework and Java 7.

The portlet is deployed in the D4Science portal, and exposed to the VREs configured to exploit FHNManager facilities. Access to the portlet is controlled by portal policies, enforced by means of the infrastructure capabilities involving the Information system, Accounting and Authorization systems.

Third-party dependencies

• jOCCI: java client library for clouds exposing the OCCI interface released under Apache License Version 2.0
• Jersey: java framework to to realize REST APIs licensed as CDDL Version 1.1 and GPL Version 2
• Liferay 6.0 CE: Enterprise portal framework released under LGPL 2.1
• GWT: Java framework for portlet developing released under Apache License Version 2.0

EGI FedCloud membership

D4Science-FedCloud integration is being tested in the “fedcloud.egi.eu” VO. Production-level operations will be moved to the “d4science.org” VO, registered on the “vomsmania.cnaf.infn.it” VOMS server and in production since December 2015.

Deployment

Components related to the D4Science-FedCloud integration are essentially assembled in two packages that are deployed independently:

• the Federated Hosting Node Manager, embedding cloud libraries and connectors, is deployed in a SmartGears container
• the FHNManagerPortlet is deployed in the gCube portal.

Source Code

The source code of the developed components are available on the gCube code repository.

Datamodel and Common: http://svn.research-infrastructures.eu/d4science/gcube/trunk/vo-management/fhn-manager-api/

Connector: http://svn.research-infrastructures.eu/d4science/gcube/trunk/vo-management/fhn-occi-connector/

FHNManager: http://svn.research-infrastructures.eu/d4science/gcube/trunk/vo-management/fhn-manager-service/

FHN Client Library: http://svn.research-infrastructures.eu/d4science/gcube/trunk/vo-management/fhn-manager-client/

FHNManagerPortlet: http://svn.research-infrastructures.eu/d4science/gcube/trunk/portlets/admin/fhn-manager-portlet/

gCube release and testing process

The software developed to enable the EGI Federated Cloud support in gCube follows the development process and guidelines defined for the whole gCube platform. The platform source code, kept on the project source code repository is continuously built upon each and every change using the ETICS tool. The components have been included in the last gCube release (ver. 3.11.0). Produced packages are available on the public gCube distribution site.

Further documentation

User documentation for the FHNManagerPortlet can be found on the gCube User Guide: https://wiki.gcube-system.org/gcube/User%27s_Guide

FHNManager, connectors and libraries’ API documentation can be found on the gCube Developer’s Guide: https://wiki.gcube-system.org/gcube/Developer%27s_Guide

Deployment and configuration details for the service and the portlet are available on the gCube Administrator’s Guide: https://wiki.gcube-system.org/gcube/Administrator%27s_Guide

License

The FHNManagerPortlet, the FHNManager and the connectors developed within the scope of this task are released under the terms of the European Union Public Licence (EUPL version 1.1), like the rest of the gCube software.