Difference between revisions of "Process Management"

Revision as of 17:26, 30 November 2008

The Process Management class of services is dedicated to the management of gCube Compound Services (CS), i.e. complex services obtained by composing other simpler services through an appropriate workflow management language. The components in this family cover all aspects related to CS management. In particular, they allow to design via a dedicated user interface the services and perform various validation activities related to CS design, to manage CS definitions, to execute instances of the services and control in various ways the execution. Finally, a component in this family allows interacting with gLite jobs seeing them as another flavour of gCube service.

Reference Architecture

The Process Management components are:

• The Process Design and Monitoring Portlet - responsible for providing a user interface for viewing, editing and managing compound service definitions in a gCube based infrastructure, and for triggering the execution of existing/new processes;
• The BPEL Parser library- responsible for parsing a BPEL process definition and transforming it into its equivalent OSIRIS process definition;
• The CSValidator service - responsible for the validating the schema conformance of the new process definitions;
• The CSEngine service- dedicated to the orchestration of the distributed execution of a compound service;
• The CSResourceManager service - which provides common operations required by other Process Management components and by Process Optimization; in particular operations for handling (registering, un-registering) process resources, i.e. Process specifications and process instances, on the gCube Information System and to issue complex process-related queries against the gCube Information System;
• The gLiteJobWrapper service- that allows to invoke gLite jobs in the same way as any other gCube service, and therefore provides the means to integrate them into the (service-oriented) workflows that the process management class of service handles.

BPEL Parser Library

The BPEL Parser Library is responsible for parsing a process definition written in BPEL and transforming it into its equivalent OSIRIS process definition. The main operation provided by this library is:

• parseBPEL- that takes a String corresponding to a BPEL process definition and returns a String corresponding to the equivalent OSIRIS process definition

Resource Manager Service/Library

The CSResourceManager Service/library provides operations for storing (registering) and removing Process definitions and Running Process Instances, thereby acting as a proxy to the resource-related functionality of the gCube Information System. It also provides an abstraction layer for queries directed to the IS, in order to have a simple and tailored interface which can be used from all of the Process Management components and for the Process Optimization components: while all the “depending” components could, in theory, use the IS Client directly, the same (complicated) queries would have to be used in multiple places; therefore, the decision was basically to outsource the queries into a common location where they can be used in a uniform way, and duplication of effort is avoided.

Functions

All operations are implemented as static methods, called “lib<...>”, e.g. “libSaveSpecs”. This allows for also directly invoking the Java methods locally (from services installed on the same GHN), bypassing the overhead otherwise introduced by the GT4 infrastructure. All operations which are not static (“<...>”, e.g. “saveSpecs”) are available as WS operations and are merely wrapping the static methods.

• libSaveSpec() – which takes as input parameter a message containing a process specification and returns the Resource ID resulting from the publishing of such a resource in the IS;
• saveSpec() – the WS version of the above function;
• libSaveSpecs() – which takes as input parameter a message containing a list of process specification and returns the list of Resource ID resulting from the publishing of such resources in the IS;
• saveSpecs() – the WS version of the above function;
• libSaveInstance() – which takes as input parameter a message containing the process ID of the running instance it is running and returns the process instance ID resulting from the publishing of such resources in the IS;
• saveInstance() – the WS version of the above function;
• libUpdateInstance() – which takes as input parameter a message containing the process ID of the running instance it is running, the profile running instance ID and the new profile and returns the process running instance ID resulting from the publishing of such resources in the IS;
• updateInstance() – the WS version of the above function;
• libRemoveSpec() – which takes as input parameter a message containing the process ID and returns a Boolean value indicating whether the process has been successfully un-registered from the IS or not;
• removeSpec() – the WS version of the above function;
• libRemoveInstance() – which takes as input parameter a message containing the process running instance ID and returns a Boolean value indicating whether the process has been successfully un-registered from the IS or not;
• removeInstance() – the WS version of the above function;
• libGetProcessDefinition() – which takes as input parameter a message containing the process ID and returns the process definition of such a process;
• getProcessDefinition() – the WS version of the above function;
• libSearchForProcessDefinitionsByRegex() – which takes as input parameter a message containing a processing filtering condition and returns the list of process definition registered in the IS that matches the specified criteria;
• searchForProcessDefinitionsByRegex() – the WS version of the above function;
• libGetServicesOnGHN() – which takes as input parameter a message containing a GHN ID and returns the list of running instances currently hosted on the specified GHN by providing the GHN ID, the porttype univocally identifying the service, and the EPR the running instance can be reached;
• getServicesOnGHN() – the WS version of the above function;
• libGetCSInstances() – which takes as input parameter a message containing a process ID, the status of the running instance and returns the list of the current process running instances matching such criteria by providing the process ID, the process running instance ID and the operational status;
• getCSInstances() – the WS version of the above function;
• libGetRunningGCubeInstances() – which takes as input parameter a message containing a porttype univocally identifying a gCube service and, optionally, a gHN ID and returns an array of running instances matching such criteria by providing the gHN ID, the porttype and the EPR;
• getRunningGCubeInstances () – the WS version of the above function;
• libSearchForGCubeServicesByAnyText() – which takes as input parameter a message containing search criteria (specifying descriptive aspects like service class, service name, package name, entry name, porttype namespace and local name) and returns the list of the gCube service resources matching such criteria by providing the the service class, the service name, the package name, the entry name and the porttype;
• searchForGCubeServicesByAnyText() – the WS version of the above function;
• libSearchForGCubeServicesByFilters() – which takes as input parameter a message containing search criteria (specifying descriptive aspects like service class, service name, package name, entry name, porttype namespace and local name) and returns the list of the gCube service resources matching these criteria (logical AND) by providing the the service class, the service name, the package name, the entry name and the porttype;
• searchForGCubeServicesByFilters() – the WS version of the above function;
• libGetGHNInfo() – which takes as input parameter a message containing a GHN ID and returns the list of existing GHNs by providing their ID, hostname, IP address, the site name, the site country and other information maintained in the GHN profile like the RAM size;
• getGHNInfo() – the WS version of the above function;
• isEndpointReachable() – which takes as input parameter a message containing the EPR of a Running Instance and returns a Boolean value indicating whether the running instance has to be considered reachable or not by trying to establish a socket connection.

CS Validator Service

This service represents the point of reference for all activity related to validation. It performs syntax checks, semantic checks, and, of course, checks any transactional or other additional properties the process specification may have. The process validation is a necessary step if the process needs to be later executed and involves syntactical and semantic correctness checks. Optionally, it can also check any transactional or other additional properties that the process specification may have. The result of the validation process is a “validation signature” confirming that the process is correctly defined. This is a required step since the runtime engine will reject processes without the validation signature or those whose signature is corrupted. The implementation of the CSValidator service is based on the Apache Project’s XML Security Java libraries.

Functions

The main function supported by the CSValidator is:

• validate() – which takes as input parameter a message containing a process specification and a process schema against which the process must be validated and returns a signed process specification if the specification if valid;

CS Engine service

The CSEngine is the core service used for the orchestration of the distributed execution of a compound service. It is responsible for starting a process and executing process activities, and for routing the process execution to the next node. The CSEngine is based on a distributed process execution engine called OSIRIS. The functionality exposed by the service is in part implemented inside an internal library, the OSIRIS library. The library is meant for internal use by the CSEngine only and it’s not described here in detail. OSIRIS supports a custom graph based process description model, which can be partially mapped on the BPEL process description language. The main constructs of the model are the following activities:

• SOAPCall - this activity provides the functionality to send a given SOAP message to a web service defined by its endpoint and optionally its SOAP action (as defined in the WSDL for the operation). It corresponds more or less to the BPEL “invoke” activity. It takes as input arguments the EPR of the service to be invoked, the operation to be executed, the message to be passed to the service operation, a flag indicating alternative calling mechanisms, the number and list of additional headers to be passed in the SOAP call and returns the response message resulting from the service invocation;
• Assignment –this activity, which corresponds to the BPEL “assign” activity, provides the functionality to assign any number of process variables (or parts thereof) or expressions to other variables or parts thereof. It takes as input parameter the list of source of assignment and returns the list of results of the requested assignments;
• Wait – this activity, which corresponds to the BPEL “wait” activity, provides the functionality to wait for a specified amount of time, i.e. to pause process execution. Both types of parameters that are mentioned in the BPEL specification are supported, and exactly one of “For” or “Until” must be given. It takes as input parameters the specification of the time to wait.
• Empty – this activity corresponds to the BPEL “empty” activity. The semantics are those of a no-op, i.e. the activity does nothing and has no effect.
• Switch – this activity corresponds to the BPEL “switch” activity. i.e. its semantics are those of a conditional branch.
• While – this activity corresponds to the BPEL “while” activity, i.e. its semantics are those of a loop with an exit condition. OSIRIS process definitions are only allowed in the form of while activities.

Resources

The CSEngine is a stateful web service. Each running CSEngine service provides a WSRF resource for process instances it is involved in. The purpose of these resources is to allow for the monitoring of the execution of a process, as the execution proceeds, for the coordination of process joins (where parts of the process might be executed on different nodes of the infrastructure and need to “agree” on the node where the process join takes place), and for retrieving the result of a process which was run asynchronously. All resource properties that are available for the process instance have a dynamic QName (based on the process instance ID, and possibly information on which step in the process they refer to) for accessing them and are created at runtime. In addition, all resource properties are made available as WS-Notification Topics and registered in the ISNotifier.

Functions

The main functions supported by the CSEngine are:

• startCS() – which takes as input parameter a message containing the process ID, a Boolean indicating whether the process has to be executed asynchronously and a list * of parameters and returns (i) the process instance ID of the created instance if the process has to be executed asynchronously, or (ii) the list of the results resulting from the execution of the process if the process has to be executed synchronously;
• startAdHocCS() – which takes as input parameter a message containing a serialised version of an entire process definition, a Boolean indicating whether the process has to be executed asynchronously and a list of parameters and returns (i) the process instance ID of the created instance if the process has to be executed asynchronously, or (ii) the list of the results resulting from the execution of the process if the process has to be executed synchronously;
• getGenericCSOutput() – which returns the list of the results resulting from the execution of the process represented by the Resource it is invoked on;
• cleanupInstances() – which cleans up the resources created for process instances from the local CSEngine it is invoked in. Two flags on the input message may indicate to also propagate the cleanup to the other CSEngines and to remove the corresponding GCUBECSInstance(s) from the IS.