Gcube/Resource Management for the gCube Model/discussions

From Gcube Wiki
Jump to: navigation, search

04.05.2018

Participants: Lino, Manuele

Resource Management functionalities

  • Discarded functionality: Merging two contexts is not a requirement
  • New functionality: Cloning a context. We clone users, services, resources, but not data.
  • New functionality: Create a template definition from an existing context to recreate (or clone) it at a later time
    • Impact on the design: RM needs a new collection of resources Templates (in the REST sense) to operate on templates of contexts.
    • Cloning a context could be an operation on a Template resource.
  • The Resource Manager operates at Virtual Service level. A Virtual Service is a bag of Services that delivers a logical group of functionalities. A request to RM specifies which Virtual Service to add/remove/edit. The RM translates this request to the list of concrete services (by querying the Resource Registry) and then instructs the Resource Registry accordingly.
  • New functionality: When a Service is added or remove to a context (as part of a Virtual Service), certain configuration steps could be necessary (e.g. create a new user o space in a DBMS). See Actions in the following.

gCube Resource Model

  • Virtual Services: their definitions are created with a portlet
  • Often, the same Virtual Service differs from one VRE to another, only for 1 Service (typically a DBMS). In these cases, we would like to have a single Virtual Service definition, not many with just a slight difference
    • Possible solution: Associate also a Type to a service, define the Virtual Service to include a service of that Type, and let the RM discovers the service of that Type in the parent context of the target context.
    • TODO: Investigate if Type (or similar information) already exists in the gCube Resource Model
  • Actions Actions are steps to perform whenever there is a change in the participation of a service into a context. Actions are triggered by the Resource Manager when operating with the Service.
    • On the resource model we need something like Service ---triggers---> Actions
    • An action could be:
  1. a command line to execute
  2. a script to execute
  3. an invocation to a remote service with certain parameters
  • Possible Action definition:
Action
Trigger ADD REMOVE
Executable Bash Script Command line Remote call
Parameters List of params for the executable
  • TODO: Investigate if the Configuration Resource can be used to define Actions

Wiki documentation

  • gxRest should be linked in the Core Facilities section
  • the TOC of the development guide should be a real Table of Contents. It's not possible that to find (for instance) the FWS, one has to click on several links sparse on the pages (Manuele).

08.05.2018

Participants: Lino, Manuele

gCube Resource Model

Following up the needs emerged on 04.05.2018, we discuss changes on the gCube Resource Model.

Action

We started the discussion on how to model Actions.

  • Using Propagation Constraints to define Actions does not seem appropriate since their scope and role are completely different (used by the IS to propagate add/remove to/from contexts). The Configuration resource is also not suitable.
  • Action is a first-level resource to introduce in the gCube Resource Model. All the resources can have associated a list of actions to trigger.

Resource ---triggers---> Action

  • At this point, the first-level resources are:
    • Software
    • Service
    • Dataset
    • Actor
    • Action

Use cases:

  • Typically actions are related to the persistence of a service in a Software. E.g. each Tabman instance writes in a DB hosted on the same DBMS. The Virtual Service of Tabman should trigger an action with the database name (as formal parameter) to create the database.
  • Actions can potentially be considered also for deploying new services with an ansible script.

An Action is modeled with 1 or N Facets.

  • One would probably hold sensitive data for accessing databases, ansible repos and similar. Sensitive facets are decrypted with symmetric keys.

Actual Virtual Service

Actual Virtual Service is a new resource in the Resource Model. It models an instantiated Virtual Service, i.e. all it was needed to add the Virtual Service to a context.

Why do we need this new resource? We identify at least three (similar) use cases:

  1. When we delete a context and we want to save a Template of it for recreating it in the future
  2. When we clone a context
  3. When we need to create a context slightly similar to another (existing) one

All these cases requires to save the conditions on which one or more Virtual Services have been instantiate for a second instantiation.

An Actual Virtual Service keeps the following information in its facets:

  • a relation with the source Virtual Service from which it was instantiated
  • a list of service instances (e.g.: Tabman running on host:port), we can use e-service
  • a list of software
  • a list of actions with actual parameters

Note from Manuele: in spite of what we said during the discussion, an Actual Virtual Service should not be an extension/specialization of Virtual Services because it cannot have actions, it does not inherit their relations (but it does navigate them), and ultimately it cannot be used in place of a Virtual Service.

TODO: Luca to investigate the feasibility of these extensions. Make an implementation plan for gcube-resources lib.

Resource Management functionalities

Context Templates

  • What format has a Context Template definition? A list of Actual Virtual Services.
  • How do we create a Context Template?
  1. Starting from an existing context
  2. By editing an existing template and saving it as new template
  • Where do we store the Context Template? We need a persistent layer in the Resource Management. First idea is a DBMS, but investigation is needed (TODO). RM has to remain stateless.

Actions

Here we discuss the impact of the new Action resource on the Resource Management.

  • An Action is defined by the executable and a list of formal parameters.
  • The Resource Manager must be able to execute Actions. E.g. if the ansible script requires access to an ansible playbook stored somewhere, the resource manager must be granted access to the storage.
  • List of Actions and their actual parameters (values) are received along within the request to add/remove a Virtual Service to/from a context.

Creating a new context

  • After some discussion, we foresee two cases:
  1. The new context is created starting from a list of Virtual Services and related actions with their actual parameters.
  2. The new context is created starting from a previously stored Template (then, from the list of its Actual Virtual Services) and some editing of the actual parameters of the actions included in the template.

11.05.2018

Participants: Lino, Manuele

Actions: when and what executes them

  • Actions must be executed by the Resource Manager service
  • An additional service just for running Actions is an overhead and less-than-beneficial complex solution. Main cons:
    • it could not execute PRE Actions (i.e. Actions before a context-related event occurs in a Virtual Service)
    • it would rely on a subscription/notification mechanism with the Resource Registry, an approach that proven to be not effective in our past experiences (see IS-Notifier)
    • when the Resource Manager returns, the operation on the context must be fully functional (and therefore synchronization of steps is required)

15.05.2018

Participants: Lino, Manuele

gCube Resource Model

  • Proposal (TODO: to evaluate): There should be common root for e-service & hosting node. It can be named for instance Execution Unit. This because Services and Virtual Services are static definition of resources from with the Execution Units are derived.
  • Actual Virtual Services (TODO: try to come up with an alternative name) are not Services. They cannot be used wherever Services are placed. What does it mean to add an AVS to a context, for instance?
  • AVS is in a 1:1 relation with a context, not shared among contexts. Is a property with UUID of the context enough? TODO: to investigate with Luca
  • Requirement: An AVS must be in the same context of the VS it is instance of.
    • The VS is added to the context before the AVS is created and linked to it. This should cover the requirement.
  • TODO: ask Luca how he made this picture and the file to modify it.
  • TODO: Manuele to prepare some diagrams with changes to the gCube model to discuss at the next call

Resource Management functionalities

  • The parameters of a context creation are available only at creation time or at the time we add a new Virtual Service or Software. So, two options:
  1. either we store them somewhere and then use them to create the Actual Virtual Services when a new Template is being created at a later time
  2. or we create the Template and the Actual Virtual Services as soon as the context is created and Virtual Services/Software added.
Solution: to avoid duplication of storage and unnecessary complications, we decide to go for 2.
  • Where do we store a Context Template?
Proposal: a Context Template is a list of Actual Virtual Services. As we defined them on 08.05.2018, Actual Virtual Services are resources of the gCube Resource Model. As such, they can be stored in the Resource Registry. A simple property with a unique UUID mapped could link them in a Template. TODO: to investigate with Luca.

Proposal for changes to the gCube Model

Rationale behind the proposal:

  • Introduce the Action resource
  • Introduce the Actual Virtual Service resource
  • Reorganize resources around the Service resource
  • Remove some apparently redundant relations

Proposed Model

GCube Model May 22th 2018 03.png

Explanation of Main changes

  • The gCube resources are the same of the current gCube Resource Model. In addition, we have Actions and Actual Virtual Services.
  • Some relations have changed to model in a uniform way the 'active' resources, i.e. the ones with an Access Point Facet. They share several common relations. The Access Point Facet logically does not belong to the model. It was added to clarify this aspect, but it will be removed from the final version.
  • Some relations have been removed because they seemed redundant. With the assumption that if Resource2 and Resource3 extend Resource1, they inherit all its relations. The same relation can have different semantic depending if the source/target is R2 or R3. For instance:
CallsFor models a relation btw two Services. It can replace:
  • uses: E-Service uses E-Services
  • requires: Software requires Services
Runs models a relation btw E-Service and Software. In this proposal, it is replaced by CallsFor.
For the same reason, isPartOf btw Concrete Dataset and Dataset could be replaced by isCorrelatedTo bwt two Dataset (relation inherited by Concrete Dataset)

NOTE: This could potentially complicate the implementation with respect to explicitly name the relation instead of giving a different semantic according to the source/destination. If this is the case, the removed relations will be reintroduced.

  • The same logic applies to Facets. If R2 extends R1, it inherits all its facets.

Other open questions

  • Why some relations are passive and other active? For instance:
    • HostingNode isManagedBy Site (why not Site manages Hosting Nodes?)
    • RunningPlugin isActivatedBy e-Service (why not E-Service activates plugin)
  • Plugin vs Running Plugin. Since we don’t have dynamic deployment, why do we model Plugin and its instances as separate resources. We only have Running plugins, no? And they could be related to Software and E-Service. For this reason, the two resources do not have a relation in the proposed model.
  • If Running Plugin has its own Access Point, it should extend Service. A CallsFor relation with another Service of type E-Service could replace the isActivatedBy relation with E-Service.
  • Why the consistsOf relation between a Resource and the Access Point Facet does not have a name like the other has* relations?

29.5.2018

Participants: Luca, Manuele

Discussion on the proposed model

  • Preliminary remarks:
    • Resources do not have properties
    • Facets have properties
    • What is a Software? It's source code that can be downloaded from a version control system (VCS), compiled and activated. The VCS hosting the code is modeled as a Service.
  • We agree that Actual Virtual Services can be modeled without introducing a new type of resource:
    • Starting from the Virtual Service we can navigate its relations and find the resources added by propagation and rebuild it into another (cloned) context
  • After evaluating pros and cons of how to model Actions (as Resources, on the Relations as props), we agree that Actions will be modeled as Facets.
  • We can have a hierarchy of Action Facets:
    • Software Instantiation Facet to model the steps needed to instantiate a new E-Service from a Software
    • Actualization Facet to keep the values of the parameters used to instantiate the E-Service.
  • TODO: Model how Software can be instantiates by the Resource Manager

AddToContext: Relations and Facets

GCubeModel-relations.png

Open Issues

  • Is the Dataset resource a Service?
  • Can the Virtual Service be a separate resource than Service?
Retrieved from "https://gcube.wiki.gcube-system.org/index.php?title=Gcube/Resource_Management_for_the_gCube_Model/discussions&oldid=31335"