Overview

ETICS stands for e-Infrastructure for Testing, Integration and Configuration of Software. ETICS is an on-line collaborative service for managing software projects by managing their configuration, enforcing quality standards, building packages and testing them in environments as close as possible to real-world infrastructures. The ETICS tools can be used to run remote builds and tests on multiple platforms at the same time.

The main architectural components of ETICS (depicted in the figure above) are:

ETICS Web Portal is the main access point to ETICS. It allows to access (view and edit) the configuration of gCube components, submit builds and tests and check the reports;
ETICS Worker Nodes are the nodes were builds and tests take place. Users do not need to access to these nodes;
ETICS Command-Line Client is a set of command-line tools to access ETICS services. The Command-Line Client provides similar functionalities as the web application. In addition, it allows to run local builds on the developer's machine and is expecially suitable for scripting and batch executions;

How to create ETICS configurations for gCube components

This sections is meant to provide operative instructions about most relevant aspects of creation of ETICS configurations for Maven-based components.

Versioning

Version field of the ETICS configuration must match the version specified for the Maven project in the pom.xml, excepting for the "-SNAPSHOT" postfix. The match will be checked during the invocation of mvn to guarantee that ETICS versions are always in sync with Maven versions.

Build Commands

the compile target of their Build Commands must invoke Maven to build the components up to the deploy phase (e.g. mvn deploy). This allows the deployment of components into Maven repositories directly from ETICS builds

the install target of their Build Commands must copy the gCube profile of the component in the ${prefix} directory. This allows the registration of the profile with the Software Gateway when the component is released. It remains a good practice to copy also the outcome of the compilation (usually jar files) in ${prefix} in order to include them in packages generated by ETICS

Sample build commands for a maven-based component should look like the one in the picture below:

Example of maven-component's build commands

Note: please note that configuration's commands (mainly install target) could change slightly depending on the actual project structure.

Build Commands for gCube Services

gCube services components are usually multi-modules maven projects (see gCyve Service development guide for more information) containing a module for the stubs, one for the service and optionally one for test-suites. In ETICS a service must be modelled with one ETICS component for each Maven project's module.

For each ETICS component, build commands should contains commands to deploy only the correspondent Maven module, and, if needed, compile the other modules needed by the first.

Picture below shows the typical build commands for the stubs component. First the mvn deploy -N is invoked to deploy the project's parent pom. If not executed, stubs' compilation will fail because parent's pom will not be resolved. Then the mvn deploy -pl exampleservice-stubs is invoked to build and compile only the stubs module.

Finally, picture below shows the typical build commands for the service component. Since the service module of a gCube service needs the stubs to be compiled locally in order to generate the gar archive, before execute the mvn deploy -pl exampleservice-service it is needed to also build the stubs mvn install -pl exampleservice-stubs. Note that it is not needed to invoke target deploy for stubs, since they are deployed already by the stubs ETICS' component. The parent's pom will already exist in the local repository (deployed by the stubs' component) and, thereofore, it is not needed to compile it again.

Dependencies

The configuration must directly or indirectly depend on the ETICS configurations of all gCube components that are specified in the POM, including maven-builder. This guarantees that at build-time all dependencies requested by the pom.xml have been already compiled and installed on the Maven's local reposiotry.

Dependencies on third-party components available in Maven Central (or other Maven repository specified in the POM) do not need to be configured in ETICS (i.e. Maven repositories replace ETICS externals).

All dependencies must be declared of type dynamic and the actual version will be resolved at project level.

Example of dependencies for a Maven component:

Example of maven-component's dependency set

Environment

Unlike Ant-based components, Maven components do not use the CLASSPATH variable for compile-time dependency resolution, but rely on the Maven local repository. This means that the Environment section of Maven component's configurations will be empty (unless other environment variables has to be set for other purposes).

There exist an exception, though. If the component's artifact is meant to be used as compile-time dependency by others Ant-based components, the artifact produced by the Maven build needs to be exported in the CLASSPATH to allows the Ant-based component to find it. Only in this case the Environment section will contain an entry to add to CLASSPATH the artifact produced. It is suggested to use following convention:

CLASSPATH: ${moduleDir}/target/<artifactid>-${version}-[SNAPSHOT|${maven.system-version}].jar

where:

<artifactId> must be replaced with the name of the artifact produced by Maven compilation. By default it is the same of <artifactId> value in the pom
SNAPSHOT must be added only if you are setting up a configuration for org.gcube.HEAD otherwise
you should use the ${maven.system-version} property
${moduleDir} and ${version} are ETICS properties and will be resolved by ETICS itself

Valid examples of CLASSPATH:

CLASSPATH: ${moduleDir}/target/home-library-${version}-${maven.system-version}.jar
CLASSPATH: ${moduleDir}/target/home-library-${version}-SNAPSHOT.jar
CLASSPATH: ${moduleDir}/SoftwareGateway-stubs/target/softwaregateway-stubs-${version}-${maven.system-version}.jar
CLASSPATH: ${moduleDir}/SoftwareGateway-stubs/target/softwaregateway-stubs-${version}-SNAPSHOT.jar		
...

Tip: you can easily find exactly which artifact is created during an ETICS build by searching in the build report of the component for the line "Installing /home/gbuilder/catalogueoflife-spd-plugin/target/catalogue-of-life-spd-plugin-1.0.0-2.12.0.jar to..."

How to submit remote builds of gcube subsystems/components

A pre-requisite to submit a build is to have right to do that. Developers have rights to build their components, Subsystem Managers have rights to build their subsystem and the Release Manager have the rights to build the entire project.

In ETICS Portal, to access the Remote Build submitting form (see picture below) just right-click on the configuration you wish to build and then select "Remote Build..." from the contextual menu that will appear. The submission form is shown below:

The most common options to set for gCube builds are listed in the table below.

option	description	usage
verbose	useful to debug the build process	recommended
Ignore Locking	A locked configuration in ETICS has a "snapshot" of the gCube configuration in which it has been locked. When a locked configuration is built in a different version of gCube of the one in which has been locked, it will try to bring in the build all the configurations belonging to the "snapshot"; most likely this will result in multiple configurations for the same component that will break the build. The usage of "Ignore Locking" option will prevent configurations from the snapshots of locked configuration to be brought in the build.	mandatory
Propagate environment and ...	essential. It will instruct ETICS about the gCube version in which context (gCube's version) build the configuration.	mandatory
Checkout from Volatile area	every ETICS build stores the binary artifacts in a remote area. By setting this option the build can use, if they exist, binary artifacts from past builds instead of re-compiling it from source (it only applies to dependencies, not on the build target component).	recommended
Do not stop on errors	ETICS will try to complete the build also in case the build of one or more components fails.	recommended
Host selection	The platform against which the build will be performed. gCube is completely in java and, thus, it is completely cross-platform and, theoretically, any of available platforms can be chosen. However, currently only CentOS 6.10 is available for gCube builds.	mandatory

Once options have been set, just click on the submit icon (top-left of the submission form) and the build will start. You can keep trace of your builds in the 'Submissions tab of ETICS Portal.

ETICS Issues

The ETICS Issues section (reachable from the top-bar, on the left) lists the issues found in components during the build. Issues are related not only to the result of the builds but also to quality guidelines defined by projects. Some issues specific for gCube have been defined (e.g. compliance of README files) and are reported in this list.

As depicted in the figure below, the user can group (in the top-right of the page) the issue list by different attributes: module, priority, type and owner. A summary for each group is displayed at the top of the page reporting the type and subtype of the issue, the total number of issues in the group and the sum of the technical debt computed for the issues.

Under the groups summary, there is the list of all issues ordered by group. By clicking on the issue, the details are displayed (see figure below).

Issue Owners

For each issue, ETICS computes a list of owners. It is the list of users that have permissions on the component associated to the issue order by proximity: users that have permissions at component-level are listed first, then the users that have permission on the component's subystem and finally users that have permission at level of project.

Issue Lifecycle

ETICS runs a number of checkers during the remote builds and creates the discovered issues at the end of each build. An issue is automatically closed by ETICS when it runs again the build of the affected component and does not find the issue anymore. Closed issues appears with in Straight-through style for few days and then disappear completely.

Some checks run only during BTRT builds and, therefore, are updated only during BTRT builds.

ETICS Continuous Builds

Along the remote builds submitted by the users on demand (see here) ETICS has also the capability to trigger builds when specific events occurred.

There are two types of events monitored:

changes in the ETICS data model (e.g. the user modify build commands, the user modify a dependency);
changes in the VCS: new commits in the source code repository;

When one of these two events occurs, ETICS look for the ETICS Configuration(s) associated with the event and request a build for that configuration. The build submission is delayed of 5 minutes from the event. During this time, any other request of building the same configuration is ignored. This avoid multiple builds when several changes and/or several commits are done in a very short period of time.

Limitations

At the moment the user that made the change is not notified when the build is launched and/or finish.

Dismissed Modules

ETICS does not support the removal of modules (projects, subsystems and components) if they (or they children) have one or more locked configurations.

However, in big and long running projects some modules could be abandoned, replaced by new ones or dismissed. The number of dismissed modules can increase a lot and "pollute" the GUI with several unneeded modules and configurations.

For this reason, there is an option to hide dismissed modules in the ETICS GUI (only hide, they will remain in the database).

How to define dismissed modules

To flag a module as dismissed, it is enough to select the "Set Dismissed" action in the contextual menù in the project tree. The action will be enabled only if the user has the edit privileges on the module.

How to show/hide dismissed modules

To show/hide dismissed modules in the ETICS GUI there is a toggle option. It is located in the settings menù, in the toolbar as shown in the figure below.

The option is per-user and persisted server side. The default for users and for not authenticated users is to hide dismissed modules.

This option affects the ProjectTree (on the left-side of the GUI) and the list of submodules shown when editing the subconfigurations. In the latter case, please note that even if a module is flagged as dismissed, but one of its configurations is attached as subconfiguration, it will be shown in any case.

Under the hood

Dismissed modules are flagged by using the "#dismissed" tag in the description of the module. Any module that contains this tag in the description will be considered as dismissed. The option "Set Dismissed" is a shortcut to add such a tag in the description. If the tag is removed, the module will not be considered dismissed anymore.

ETICS ModelSync

This functionality allows to automatically update ETICS model and keep it in sync with changes in the source code.

In general terms, it works in the following way:

each ETICS Configuration is associated with a particular location on VCS (e.g. SVN folder, Git commit, ...) computed using the checkout commands and properties defined in ETICS by the users;
a specific module (namely ProjectParser) crawls the VCS location looking for Maven project descriptors (i.e. pom.xml files). The ProjectParser is general enough to allows also other project types (e.g. Ant, Python distuitls, Ivy, ...), but currently only parsing of Maven pom.xml files is supported;
The ProjectParser extract from the pom.xml the component information like coordinates, version, dependencies, description, etc, ...
The ModelSync sets the version of the ETICS Configuration to the version found in the pom.xml;
The ModelSync adds (or updates, if it already exists) the ".coords" property in the ETICS Configuration with the coordinates found in the pom.xml analysed. This realizes a mapping between ETICS Configurations and Maven coordinates. This mapping will be useful later when setting ETICS dependencies. Moreover, this mapping enables Maven-aware search operations like: "What ETICS Configuration provides a given maven coordinates?", "What artifact is produced by a given ETICS Configuration?";
The ETICS ModelSync adds the dependencies in the ETICS Configuration. For each dependency found in the pom.xml, a search is performed in ETICS to find the component that provides the dependency based on its coordinates. If one or more components are found, they are added as dependency in ETICS. This works not only for dependencies but also for the parent and dependencyManagement (e.g. boms) section in the pom.xml;
ETICS dependencies that are not found in the pom.xml are deleted;
It may happens that a given dependency found in the pom.xml is not added by the ETICS ModelSync because no ETICS Configurations that provides the dependency are found. This could be simply caused by the fact that the ProjectParser still did not analysed the pom.xml of the dependency component. In this case, ETICS remembers this "unresolved" dependency and as soon as the searched coordinates are found in an ETICS Configuration, the dependency is added;
In case multiple projects (i.e. pom.xml files) are detected in VCS location (e.g. multi-module projects like in case of gCube services), the build command is analysed to understand which Maven projects are actually built by the component;

This process is triggered automatically every time a change in ETICS or in the VCS is detected (e.g. the ETICS Configuration has been modified, a new commit in SVN occurred).

By default, this process is enabled for all configurations. However, its behaviour can be controlled through a set of ETICS properties that can be set in the ETICS Configurations:

reactor.ignore=true disables any action on the configuration (including automatic builds). It is usually automatically set when ETICS finds anomalies on the configuration (e.g. a configuration that points to the trunk)
model.sync.enabled=false disables completely the synchronization
model.sync.coordinates.enabled=false disables synchronization of the ".coords" property
model.sync.version.enabled=false disable synchronization of version
model.sync.dependencies.enabled=false disables the synchronization of dependencies
model.sync.dependencies.ignore=<etics-module-name-1>;<etics-module-name-2>;... a ";" separated list of ETICS module names to ignore while synchronizing dependencies. Dependencies on ETICS modules in this list are neither added nor deleted automatically;

In addition, the label #model.sync.ignorefordependencies added at module level (in the module description) excludes that module from being set as dependency in other modules. It is commonly used for dismissed modules when they have locked configurations that have the .coords property (and hence are considered when looking for modules that provides the given maven project).

Current limitations

currently only Maven projects are synchronised. For Ant components sync is not (and will be not) implemented;

How to use it

The rational behind this functionality is to automatically react to changes in the VCS and to update ETICS model accordingly. This is intended to be completely automatic without the interaction of users. The only way users can interact with the synchronisation mechanism is through setting of the control properties. When the synchronisation is enabled, users should not modify manually ETICS Configuration version and dependencies because they will be revert by the synchronisation mechanism.

What to do when a new component is created on ETICS? It is enough to set the ETICS checkout and build commands for the ETICS Configuration and the ModelSync will be able to automatically set (and keep aligned) the version and the dependencies.

External Links

Etics Web Portal

Etics Build Report

Etics Client Installation HowTo

ETICS

Contents