FHNManager

Overview

The integration of the gCube system with FedCloud and, in general, with research/commercial cloud providers, aims to add elasticity to the D4Science infrastructure by allowing programmatic and dynamic creation, configuration and decommissioning of cloud resources on external infrastructures. The benefits of this integration facility for the gCube system are threefold:

• to exploit infrastructural resources in a pay-per-use model
• to scale in/out quickly in response to specific needs
• to reduce manual interventions of Infrastructure Managers.

Key features

Registration and management of cloud providers

Access credentials and cloud resources templates on a per-VRE basis;

Credentials securely stored in the infrastructure;

Cloud resource lifecycle management

Creation, configuration and decommissioning of cloud resources must be supported;

The integration facility allows to use templates for the configuration of cloud resources (e.g. SmartExecutor Template, cpu and memory characteristics);

Accountability and monitorability

Accounting and monitoring data on usage and workload of resources created on the cloud are collected and made available in the infrastructure;

Requisite to comply with D4Science infrastructure policies and for the automatic scale-in/out;

User interface

Integrated in the VRE administration portal since it already contains all the other VRE administration tools;

Modular architecture

A plug-in based architecture for an easy integration of new cloud providers;

Use cases

It is possible to distinguish two possible scenarios:

• the VRE Manager decides the number of SmartExecutor nodes needed in the VRE. Through the VRE administration portal, the VRE Manager chooses (from a list of templates) to create the required number of SmartExecutors. She/he chooses on which infrastructure the resources should be created and their characteristics in terms of computation and/or storage capacity. The gCube system, via the integration facility here described, will automatically create the new nodes, register them to the D4Science infrastructure and make them available in the VRE.
• the gCube system intelligently adapt the number of SmartExecutor nodes to the actual demand. In the VRE administration portal, the VRE Manager decides the minimum and maximum number of SmartExecutor expected for the VRE. Then, the gCube system via the integration facility here described, automatically scales-in/out the pool of SmartExecutors (by creating/destroying nodes) predicting the demand for SmartExecutors by analysing VRE accounting and monitoring data.

Design

Architecture

The architecture highlights a number of integration-specific components and their interaction among them as well as with existing components, either belonging to the D4Science infrastructure or to the EGI infrastructure. They are described in the following:

• Cloud Libraries: Third-party software providing language-specific APIs and data model to easily interact with clouds from within applications. Although they usually support a number of different clouds and cloud standards, there's no universal coverage for any of them; nor the the API and data model they expose is uniform across libraries.

• Connectors: They are built on top of cloud libraries in order to abstract the specifics of their APIs and data models and expose a uniform interface to the upper layers. Connectors do not interact with any other service nor are expected to persist any information.

• Federated Hosting Node Manager (FHNM): is the core part of the integration and is the place where all the business logic resides. It's the gateway for all the operations related to the management of external cloud infrastructures, via the most-appropriate connector/library; it manages connectors to the available clouds; it gathers accounting and status data and publishes them to the specific D4Science services.

• Federated Hosting Node Manager Portlet: Such portlet provides infrastructure and VRE administrators with a dashboard and a control panel to easily monitor and manage resources in external infrastructures. It enables administrators to register cloud infrastructures and credentials associated with them as well as virtual appliances and service profiles.

• D4Science Information System: Collects, holds and provides all the information related to the D4Science infrastructure. In particular, for the purpose ot the integration of FedCloud, it holds the list of available cloud sites along with credentials to access them, the list of running cloud resources and their status and the list of virtual appliances available for instantiation.

• D4Science Monitoring system: Data collected is analysed to produce alarms and/or take countermeasures in case of problems with one or more resources.

• The Virtual Organization Membership Service (VOMS): VOMS manages the authorization attributes to be embedded in X.509 proxy certificates, needed to access FedCloud sites.

• EGI FedCloud sites: Host cloud resources instantiated through the components. The list of sites is available here.

• AppDb repository maintains the overall set of available virtual appliances (such as gCube Smart Executor).

The solution has been developed by means of Java since it is the same technology of the gCube system and it made easier to integrate in the D4Science infrastructure and furthermore, an high availability for third-party libraries implementing cloud standards is available.

A brief description of the service is visible in the following subsections.

Data Model

The datamodel has been designed with the aim of representing the information related to cloud providers, nodes, templates, accounting and monitoring in a homogenous way independently from the cloud provider exposed API and technology. Connectors and third-party Cloud Libraries ensured a proper translation between the internal datamodel and the specific cloud provider model.

A detailed class diagram of the adopted data model is shown in the following picture.

FHN Manager

The FHNManager is a web service implemented in Java and running in a gCube SmartGears container. It exposes a REST API that allows to access data on nodes and cloud providers as well as to execute operations to create, destroy, start and stop nodes.

In order to account the usage of cloud provider resources by the D4Science infrastructure, the service relies on the gCube accounting framework provided by the infrastructure: new usage records have been defined to track changes in the lifecycle of the resources on the cloud. Through the connectors, the service also support the fetching of accounting data from the cloud provider (when available).

Concerning the monitoring data, the service implements the support to fetch it from the cloud provider (if available) as well as collect data directly from the resource via the gCube monitoring framework.

The current implementation of the service - and thus the portlet - does not support the registration of new cloud providers along with their associated credentials. However, since the registration entries for cloud providers are maintained in the D4Science Information System, VRE/Infrastructure managers can still use the editing functionalities provided by the Information system itself, although the editing capabilities provided are not tailored to the specific task.

Since the FHNManager is deployed in a SmartGears container, it is automatically registered to the D4Science infrastructure and it is automatically authorized to exploit infrastructure capabilities (Information, Accounting and Monitoring). In addition to properly generate and use X.509 proxy certificates, the host must also be equipped with VOMS clients (i.e. voms-proxy-init) and configured with trusted root certification authorities as distributed by EUGridPMA.

Connectors

Please refer to FHN OCCI Connector Library page

FHN Manager Portlet

Please refer to FHNManager Portlet page

Third-party dependencies

• jOCCI: java client library for clouds exposing the OCCI interface released under Apache License Version 2.0
• Jersey: java framework to to realize REST APIs licensed as CDDL Version 1.1 and GPL Version 2
• Liferay 6.0 CE: Enterprise portal framework released under LGPL 2.1
• GWT: Java framework for portlet developing released under Apache License Version 2.0

EGI FedCloud membership

D4Science-FedCloud integration is being tested in the “fedcloud.egi.eu” VO. Production-level operations will be moved to the “d4science.org” VO, registered on the “vomsmania.cnaf.infn.it” VOMS server and in production since December 2015.

gCube release and testing process

The software developed to enable the EGI Federated Cloud support in gCube follows the development process and guidelines defined for the whole gCube platform. The platform source code, kept on the project source code repository is continuously built upon each and every change using the ETICS tool. The components have been included in the last gCube release (ver. 3.11.0). Produced packages are available on the public gCube distribution site.

Client Library

In order to test the functionalities proposed by the service and according to Client Libraries Framework guidelines, a rest client library has been developed.

Maven coordinates

<dependency>
  <groupId>org.gcube.resources.federation</groupId>
  <artifactId>fhn-manager-client</artifactId>
  <version>1.0.0-SNAPSHOT</version> 
</dependency>

Client Testing

The set of RestAPIs call can be performed by using the ClientTest.java class.

ScopeProvider.instance.set("/gcube/devsec"); //set the scope of interest here
 
FHNManager client = FHNManagerProxy.getService(new URL("http://"hostname":"port"/fhn-manager-service/rest")).build();
FHNManagerClient client = FHNManagerProxy.getService().build();
 
client.allServiceProfiles();
client.createNode(vmProviderId, serviceProfileId, resourceTemplateId);
client.findNodes(vmProviderId, serviceProfileId);
client.findResourceTemplate(vmProviderid);
client.findVMProviders(serviceProfileId);
client.getNodeById(NodeId);
client.getVMProviderbyId(vmProviderid);
client.startNode(NodeId);
client.stopNode(NodeId);
client.deleteNode(NodeId);

Alternatively, it is possible to test the functionalities (that later will be provided by way of the GUI) directly from browser by accessing to:

http://hostname:port/fhn-manager-service/rest/serviceprofiles
http://hostname:port/fhn-manager-service/rest/nodes/create?vmProviderId=""&serviceProfileId=""&resourceTemplateId=""  
http://hostname:port/fhn-manager-service/rest/nodes?vmProviderId=""&serviceProfileId=""                               
http://hostname:port/fhn-manager-service/rest/resourceTemplate?vmProviderid=""                                        
http://hostname:port/fhn-manager-service/rest/vmproviders?serviceProfileId=""                                        
http://hostname:port/fhn-manager-service/rest/nodes/nodeId                                                           
http://hostname:port/fhn-manager-service/rest/vmproviders/vmProviderid                                               
http://hostname:port/fhn-manager-service/rest/start?NodeId=""
http://hostname:port/fhn-manager-service/rest/nodes/stop?NodeId=""
http://hostname:port/fhn-manager-service/rest/delete?NodeId=""

An example of running service is available here: http://fedcloud.res.eng.it/fhn-manager-service/rest/*