Integration and Interoperability Facilities Framework: Client Libraries Management Model

Client Libraries Management Model

In this section we focus in Client Libraries as gCube system components and list the steps needed to be managed as such. In particular, the

Management Model identifies best practices and/or tools in the areas of:

• building of client libraries, including interactive builds and continuous integration builds;
• profiling of client libraries as system components
• packaging of client libraries for distribution purposes
• testing of client libraries, including unit testing an integration testing

Building of Client Libraries

gCube Requirements

As gCube components, the Client Libraries must be delivered according to the packaging rules for gCube software. A Client Library in gCube is described by a 'Profile' document, named Service Profile. For each Service Profile a corresponding 'Software Archive' should be delivered. A Software Archive is a single TAR GZ file, which contains all the files declared on the Service Profile. Moreover, as gCube components, CLs take part in the continuous integration builds through ETICS, to ensure correct integration with other system components.

In the following sections, the requirements for managing local or ETICS builds are listed. We recommend the development of CLs as Maven-based gCube components and we assume Maven in illustrating how the requirements for both interactive and continuous integration builds can be met.

Local Builds

As illustrated bellow, system requirements for javadoc and sources can be accomodated by inheriting from maven-parent. This ensures compliance with project-wide requirements, from the enforcement of minimal Java and Maven versions to generation and packaging of Javadoc documentation and component sources.

Building Profile

Assuming the fooCL library, which is a Maven component in gCube class Samples, the Maven coordinates of one of its development versions are:

<groupId>org.gcube.samples</groupId>
<artifactId>fooCL</artifactId>
<version>1.0.0-SNAPSHOT</version>

Its POM specifies the following Maven parent:

<parent>
 <artifactId>maven-parent</artifactId>
 <groupId>org.gcube.tools</groupId>
 <version>1.0.0</version>
 <relativePath />
</parent>

The gCube Software profile of the library is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ID />
  <Type>Service</Type>
  <Profile>
    <Description>Embedded Domain-Specific Language for Stream Transformations</Description>
    <Class>Samples</Class>
    <Name>fooCL</Name>
     <Version>1.0.0</Version>
     <Packages>
       <Software>
         <Name>fooCL</Name>
         <Version>1.0.0-SNAPSHOT</Version>
         <MavenCoordinates>                       <groupId>org.gcube.samples</groupId>                       <artifactId>fooCL</artifactId>                       <version>1.0.0-SNAPSHOT</version>                  </MavenCoordinates>                  <Files>
           <File>fooCL-1.0.0-SNAPSHOT.jar</File>
         </Files>
       </Software>
     </Packages>
  </Profile>
</Resource>

Notice that:

• the profile includes a single package and this package corresponds to the main build artefact of the component;
• the whole profile and the package have usual gCube coordinates;
• the package name is aligned with the Maven artifactId of the component;
• the package version is aligned with the Maven version of the component;
• the package includes MavenCoordinates that can be directly copied and pasted from the POM;
• the package does not specify dependencies;
• the package points to the main artefact of the Maven build;

The alignment between profile and POM can be exploited to simplify the management of the profile across different component versions. In particular, fooCL uses POM variables top define its profile:

<?xml version="1.0" encoding="UTF-8"?>
<Resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ID />
  <Type>Service</Type>
  <Profile>
    <Description>${description}</Description>
     <Class>Samples</Class>
     <Name>${artifactId}</Name>
     <Version>1.0.0</Version>
     <Packages>
       <Software>
         <Name>${artifactId}</Name>
         <Version>${version}</Version>
         <MavenCoordinates>            <groupId>${groupId}</groupId>            <artifactId>${artifactId}</artifactId>             <version>${version}</version>         </MavenCoordinates>	 <Files>
            <File>${build.finalName}.jar</File>
         </Files>
       </Software>
     </Packages>
</Profile>
</Resource>

Note that the variables must be resolved (i.e. the profile is interpolated) before the profile can be used within the system. In particular, the main destination of the profile is the gCube Software Archive (SA) which packages the component for registration within the system.

Package as a Service Archive

CLs are responsible for generating their own SA as part of their Maven build. This requires dedicated logic in the POM but has the advantage that:

• SA validity can be verified locally;
• no SA configurations are required in ETICS;
• the required build logic can be easily reused across components;

CLs use the Maven Assembly Plugin to generate their own SA, with an approach that makes optimal use of Maven variable interpolation in static files such as README, svnpath.txt, MAINTAINERS, changelog.xml, etc. In particular, the Assembly plugin takes care of variable interpolation in the profile and other static files. The approach is best illustrated with a reference to the sources of the streams library.

The entry for the Assembly plugin in the POM is as follows:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-assembly-plugin</artifactId>
	<configuration>
		<descriptors>
			<descriptor>${distroDirectory}/descriptor.xml</descriptor>
		</descriptors>
	</configuration>
	<executions>
		<execution>
			<id>servicearchive</id>
			<phase>install</phase>
			<goals>
				<goal>single</goal>
			</goals>
		</execution>
	</executions>
</plugin>

Inside the artifact, there is a distro folder including the static files, the profile.xml and the descriptor.xml. The descriptor.xml is used for building the service archive and its content can be found here.

ETICS Builds

As mentioned above, no SA configurations are required in ETICS when placing the dedicated logic in the POM. ETICS however introduces a requirement for the CLs to produce an interpolated profile during integration builds, outside the context of the SA. The CL meets this requirement with the Maven Resources Plugin, through which the interpolated copy is prepared, as illustrated below:

<build>
<plugins>
 
	<plugin>
		<groupId>org.apache.maven.plugins</groupId>
		<artifactId>maven-resources-plugin</artifactId>
		<version>2.5</version>
		<executions>
			<execution>
				<id>copy-profile</id>
				<phase>install</phase>
				<goals>
					<goal>copy-resources</goal>
				</goals>
				<configuration>
					<outputDirectory>target</outputDirectory>
					<resources>
						<resource>
							<directory>${distroDirectory}</directory>
							<filtering>true</filtering>
							<includes>
								<include>profile.xml</include>
							</includes>
						</resource>
					</resources>
				</configuration>
			</execution>
		</executions>
	</plugin>
</plugins>
</build>

Then ETICS build must be configured to copy the profile in the right location, as described in the next section.

ETICS Configurations

Versioning

Version field of the ETICS configuration must match the version specified for the Maven project in the pom.xml, excepting for the "-SNAPSHOT" postfix. The match will be checked during the invocation of mvn to guarantee that ETICS versions are always in sync with Maven versions.

Build Commands

• the compile target of their Build Commands must invoke Maven to build the components up to the deploy phase (e.g. mvn deploy). This allows the deployment of components into Maven repositories directly from ETICS builds
• the install target of their Build Commands must copy the aformentioned gCube profile of the component in the ${prefix} directory. This allows the registration of the profile with the Software Gateway when the component is released. It remains a good practice to copy also the outcome of the compilation (usually jar files) in ${prefix} in order to include them in packages generated by ETICS

Sample build commands for a maven-based component should look like the one in the picture below:

Dependencies

The configuration must directly or indirectly depend on the ETICS configurations of all gCube components that are specified in the POM, including maven-parent. This guarantees that at build-time all dependencies requested by the pom.xml have been already compiled and installed on the Maven's local reposiotry.

Dependencies on third-party components available in Maven Central (or other Maven repository specified in the POM) do not need to be configured in ETICS (i.e. Maven repositories replace ETICS externals).

All dependencies must be declared of type dynamic and the actual version will be resolved at project level.

Example of dependencies for a Maven component:

For information about how to set the Environment for CLs that are used as compile-time dependencies by other Ant-based components, you can check here.

Testing of client libraries

Client Libraries can and should consider both unit and integration testing. With unit testing alone, the CL can simulate all the possible interactions with the service but integration testing is also needed to ensure compatibility with the latest version of the service. Therefore, the recommended approach for Client Libraries testing, complies to the following rules:

1) Enable Unit Testing by isolating 3rd party dependencies (i.e. the remote service) 2) Perform Integration Testing by to eventually test that 3rd party dependencies behave as assumed

In the following sections we point out the strenghts of each approach. The illustration of how to perform testing for each case using acquired FWK tools is described here(link).

Unit Testing

Unit Testing should be better used to ensure CL code handles all the interaction outcomes that may occur in production, particulary corner cases and failures. As opposed to cases where the operations' "success path" is being tested, case like bad outputs and failures are naturally and quickly dealt within unit testing. Moreover, unit testing must be also applied in cases where the services cannot produce outputs in my-container (e.g. because its external calls cannot be short-circuited, or because the CL does not know how to do it).

Integration Testing

Integration testing for Client Libraries can take the form of in-container testing with my-container. If testing of the Client Library is limited to unit testing, it's assumed that its mocks represent what the service will actually return. But if the service changes (e.g. changes its outputs), with integration testing, done locally or nightly, the CL makes sure that it's working with the latest version of the service. Therefore, integration testing is needed for regressong problems that will occure when the underlying service changes in non-compatible ways. However, my-container is a closed environment and supports integration testing up to a point; where that point is depends on how the service is designed. Therefore, there could be mainstream cases when testing functionalities cannot be integrated in my-container and is forced to renounce to unit testing.

To sum up, unit-testing best suits Client Library developers' needs when:

• Needing to mock all possible scenarios for the output of the operations and especially corner cases for bad results and failures
• Needing to test cases for operations that cannot produce output within my-container

While Integration Testing is needed when:

• Needing to test mainstream cases and anticipated interaction outcomes
• Needing to ensure that there are no interaction outcomes that have been overlooked due to changes of the underlying services