Difference between revisions of "Developing gCube Maven Components"

From Gcube Wiki
Jump to: navigation, search
(→‎Libraries)
(→‎Dependencies)
Line 33: Line 33:
  
 
== Dependencies ==
 
== Dependencies ==
 +
Service and libraries that depend on other gCube components (Maven-based or Ant-based) can find them in the gCube Maven Repositories: <code>gcube-snapshots</code> and <code>gcube-releases</code>.
 +
 
=== gCube Maven Repositories ===
 
=== gCube Maven Repositories ===
 +
As a one-off task, developers must configure these repositories in the <code>~/.m2/settings.xml</code> file of their local Maven installation, as follows:
 +
 +
<source lang="xml">
 +
<settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 +
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
 +
 +
<servers>
 +
<server>
 +
<id>gcube-snapshots</id>
 +
<username>gcube-user</username>
 +
<password>maven</password>
 +
</server>
 +
</servers>
 +
 +
 +
<profiles>
 +
<profile>
 +
<id>gcube</id>
 +
<repositories>
 +
<repository>
 +
<id>gcube-snapshots</id>
 +
<name>gCube Snapshots</name>
 +
<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-snapshots</url>
 +
<releases>
 +
<enabled>false</enabled>
 +
</releases>
 +
<snapshots>
 +
<enabled>true</enabled>
 +
</snapshots>
 +
</repository>
 +
<repository>
 +
<id>gcube-releases</id>
 +
<name>gCube Releases</name>
 +
<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases</url>
 +
<snapshots>
 +
<enabled>false</enabled>
 +
</snapshots>
 +
<releases>
 +
<enabled>true</enabled>
 +
</releases>
 +
</repository>
 +
<repository>
 +
<id>gcube-externals</id>
 +
<name>gCube Externals</name>
 +
<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-externals</url>
 +
<snapshots>
 +
<enabled>false</enabled>
 +
</snapshots>
 +
<releases>
 +
<enabled>true</enabled>
 +
</releases>
 +
</repository>
 +
</repositories>
 +
 +
<pluginRepositories>
 +
<pluginRepository>
 +
<id>gcube-snapshots</id>
 +
<name>gCube Snapshots</name>
 +
<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-snapshots</url>
 +
<releases>
 +
<enabled>false</enabled>
 +
</releases>
 +
<snapshots>
 +
<enabled>true</enabled>
 +
</snapshots>
 +
</pluginRepository>
 +
<pluginRepository>
 +
<id>gcube-releases</id>
 +
<name>gCube Releases</name>
 +
<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases</url>
 +
<snapshots>
 +
<enabled>false</enabled>
 +
</snapshots>
 +
<releases>
 +
<enabled>true</enabled>
 +
</releases>
 +
</pluginRepository>
 +
<pluginRepository>
 +
<id>gcube-externals</id>
 +
<name>gCube Externals</name>
 +
<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-externals</url>
 +
<snapshots>
 +
<enabled>false</enabled>
 +
</snapshots>
 +
<releases>
 +
<enabled>true</enabled>
 +
</releases>
 +
</pluginRepository>
 +
</pluginRepositories>
 +
 +
</profile>
 +
</profiles>
 +
 +
<activeProfiles>
 +
<activeProfile>gcube</activeProfile>
 +
</activeProfiles>
 +
</settings>
 +
</source>
 +
 +
 +
'''Note:Developers must not configure these repositories in their POMs''', only in <code>settings.xml</code>.
 +
 +
This is to:
 +
* keep their projects independent from future changes in repository locations;
 +
* avoid interference with ETICS builds, where access to these repositories may comprise correct system integration;
 +
 +
 
=== Third-party Repositories ===
 
=== Third-party Repositories ===
 +
Using the the <code>settings.xml</code> aforementioned, any dependency declared in pom.xml that can be found in one of gCube repositories or in Maven Central will be resolved at build time and deployment time without any further configuration. Frequently it may happens that a gCube component needs software stored on a public Maven repository not in on of those repositories. In this case, Maven provides an easy solution for declaring additional repositories for dependency resolution by adding a <repositories> section in the pom.xml of the project. E.g.:
 +
 +
<source lang="xml" highlight="3,4">
 +
<repositories>
 +
    <repository>
 +
    <id>[repository-id]</id>
 +
    <url>[repository URL]</url>
 +
    </repository>
 +
</repositories>
 +
</source>
 +
 +
Though technically possible to add any Maven repository in this way, in order to preserve the quality of gCube software it is required to check trustworthiness of its dependencies disallowing usage of certain external software. To enforce such a control over public Maven repositories, '''gCube components are allowed to use external repository not directly but only through a proxy repository''' (a.k.a. shadow repository) created on CNR's Nexus Repository Portal (http://maven.research-infrastructures.eu/nexus/).
 +
 +
List of all repositories in Nexus Portal is available at http://maven.research-infrastructures.eu/nexus/index.html#view-repositories
 +
 +
'''Note''': if a component uses a public Maven repository directly, it will work for building but would fail in deployment phase. In fact, Software Gateway is only able to download software accessible from Nexus.
 +
 +
==== How to request creation of a new proxy repository in Nexus ====
 +
Proxy Repositories on Nexus are created by the Release Manager. To request a new proxy a new ticket must be created.
 +
 +
Details of ticket are:
 +
#  open a ticket of type "Task" in iMarine TRAC (at https://issue.imarine.research-infrastructures.eu/)
 +
# ticket's '''description''' must report:
 +
#* URL of repository to add
 +
#* organization which hosts the repository
 +
#* which artifacts are needed from that repository
 +
# '''Type''' is ''Task''
 +
# '''Milestone''' field should be set to the gCube release in which the component is expected to be released
 +
# '''Component''' field must report the gCube component that needs the external repository
 +
# '''Blocking''' should list (if any) other tickets blocked by the addition of this Maven repository
 +
# '''Task Category''' is ''software''
 +
# assign the ticket to the Release Manager
 +
 +
An example of ticket for requesting a new repository mirror is reported in the figure below.
 +
[[Image:New_maven_mirror_repo.png|600px|center|Ticket for requesting a new Mirror Repository]]
 +
 +
Release Manager will process the ticket checking that no issues are foreseen from the usage of software stored in the repository. If request is accepted, a new proxy repository is created and the repository's URL will be communicated publicly on gCube's developers mailinglist (a.k.a. TCOM mailinglist).
 +
 +
Knowing the mirror repository URL, developer can use it adding following lines in pom.xml. E.g.
 +
 +
<source lang="xml" highlight="3,4">
 +
<repositories>
 +
    <repository>
 +
    <id>gcube-ext-osgeo</id>
 +
    <url>http://maven.research-infrastructures.eu/nexus/content/repositories/osgeo/</url>
 +
    </repository>
 +
</repositories>
 +
</source>
 +
 +
'''Note''': it is possible that software stored on proxy repositories is not included in search results on Nexus. This happens for those repositories that does not have indexes (or have indexes in a format that Nexus is not able to download). Anyway, once Nexus has downloaded artifacts from the original repository for the first time (e.g. because a compilation of a pom that uses the proxy repository), creates its own indexes and those artifacts will become available also for local search.
 +
 +
 +
=== Dependency Versions ===
 +
 +
ETICS builds ensure correct integration of gCube components, regardless of the versions their dependencies. It also ensures that components are released with fixed versions for their dependencies, i.e. ensures build reproducibility.
 +
 +
Accordingly, developers can be liberal in the choice of dependency versions.  i.e. rely on version ranges to:
 +
 +
* minimise the requirements for version management;
 +
* be readily informed of any mis-alignment with their dependencies;
 +
 +
As we start adopting Maven, the lower-bound of version ranges will be a SNAPSHOT version. For example, a dependency on the <code>streams</code> library can be specified as follows:
 +
 +
<source lang="xml">
 +
<dependency>
 +
    <groupId>org.gcube.data.access</groupId>
 +
    <artifactId>streams</artifactId>
 +
    <version>[1.0.0-SNAPSHOT,2.0.0)</version>
 +
</dependency>
 +
</source>
 +
 +
With this dependency, a component will be soon notified of  a non-retro compatible change that, contrary to conventions, <code>streams</code> may have introduced in, say, version <code>1.3.0</code> (typically the day after at the latest). The notification will be delivered in the IDE, assuming the availability of Maven integration (e.g. <code>m2e</code> in Eclipse). 
 +
The component is in fact free to open the range further if it wants to be aligned with its dependencies across major versions.
 +
 +
 
=== Common Dependencies ===
 
=== Common Dependencies ===
 +
 +
Many gCube components depend on <code>gcf</code>, which is now available as a Maven component since version <code>1.4</code>. At the time of writing, the component has not been released yet. A dependency on it is then specified as follows:
 +
 +
<source lang="xml">
 +
<dependency>
 +
    <groupId>org.gcube.core</groupId>
 +
    <artifactId>gcf</artifactId>
 +
    <version>[1.4.0-SNAPSHOT,2.0.0)</version>
 +
    <scope>provided</scope>
 +
</dependency>
 +
</source>
 +
 +
The scope <code>provided</code> indicates that these dependency will be satisfied in the development environment (the gHN) and should not be included in GAR and SA archives.
 +
 +
Some components depend on the libraries available in all gHNs, though they do not depend on <code>gcf</code> itself. The subset of these libraries that are required for compilation and integration testing have been coalesced in a single Jar, distributed as a Maven component called <code>ghn-core-runtime</code>. A dependency on this Jar can be specified as follows:
 +
 +
<source lang="xml">
 +
<dependency>
 +
    <groupId>org.gcube.distribution</groupId>
 +
    <artifactId>ghn-core-runtime</artifactId>
 +
    <version>1.0.0</version>
 +
    <scope>provided</scope>
 +
</dependency>
 +
</source>
 +
 +
Note: a component that depends on <code>gcf</code> indirectly depends on <code>gun-core-runtime</code>.
 +
 +
Another common dependency is on the gCube gRS2, an Ant component that the build infrastructure make available in Maven repositories. A dependency on this component can be specified as follows:
 +
 +
<source lang="xml">
 +
<dependency>
 +
    <groupId>org.gcube.execution</groupId>
 +
    <artifactId>grs2library</artifactId>
 +
    <version>[1.2.0-SNAPSHOT,2.0.0)</version>
 +
</dependency>
 +
</source>
  
 
= Libraries =
 
= Libraries =

Revision as of 17:01, 18 July 2012

Common

Maven Coordinates

Maven components' coordinates must follow these conventions:

  • their groupId must be of the following form:
org.gcube.<class>
where class identifies a functionally correlated set of components and helps a range of users (developers, administrator) to discover components in various UIs/reports used within the system and in tools. Logically, the class corresponds to the gCube class of the component's, as specified in its gCube Software Profile. However, more specific, or else cross-class groups may be agreed upon among developers. In all cases, the group should align with official Maven guidelines. By agreement, for example, components in gCube class DataAccess declare a groupId of org.gcube.data.access and those in gCube class DataTransfer declare a groupId of org.gcube.data.transfer. On the other hand, also by agreement, components in gCube class InformationSystem declare a groupId of org.gcube.informationsystem.
  • their artifactId must follow Maven conventions (e.g. avoid capital letters or punctuation other than hyphens).

Parent

All gCube components must inherit from maven-parent. This ensure and promotes compliance with project-wide requirements, from the enforcement of minimal Java and Maven versions to generation and packaging of Javadoc documentation and component sources. 

<parent>
 <artifactId>maven-parent</artifactId>
 <groupId>org.gcube.tools</groupId>
 <version>1.0.0</version>
 <relativePath />
</parent>

Service Archive (SA)

Whenever they are associated with a gCube Software Archive (SA), they must produce it as a secondary tar.gz artefact with classifier servicearchive. Note that the SA of Maven components does not need to include Javadoc documentation (as this documentation is produced and published directly as a Maven artefact);

  • for each deployable artefact they produce (primary or secondary), they must include the Maven coordinates of the arteact in the corresponding package section of the gCube Software Profile. Specifying package dependencies in the Software Profile, on the other hand, is no longer required;
  • whenever they are associated with a gCube Software Archive (SA), they must produce it as a secondary tar.gz artefact with classifier servicearchive. Note that the SA of Maven components does not need to include Javadoc documentation (as this documentation is produced and published directly as a Maven artefact);

Dependencies

Service and libraries that depend on other gCube components (Maven-based or Ant-based) can find them in the gCube Maven Repositories: gcube-snapshots and gcube-releases.

gCube Maven Repositories

As a one-off task, developers must configure these repositories in the ~/.m2/settings.xml file of their local Maven installation, as follows: 

<settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
 
	<servers>
		<server>
			<id>gcube-snapshots</id>
			<username>gcube-user</username>
			<password>maven</password>
		</server>
	</servers>
 
 
	<profiles>
		<profile>
			<id>gcube</id>
			<repositories>
				<repository>
					<id>gcube-snapshots</id>
					<name>gCube Snapshots</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-snapshots</url>
					<releases>
						<enabled>false</enabled>
					</releases>
					<snapshots>
						<enabled>true</enabled>
					</snapshots>
				</repository>
				<repository>
					<id>gcube-releases</id>
					<name>gCube Releases</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases</url>
					<snapshots>
						<enabled>false</enabled>
					</snapshots>
					<releases>
						<enabled>true</enabled>
					</releases>
				</repository>
				<repository>
					<id>gcube-externals</id>
					<name>gCube Externals</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-externals</url>
					<snapshots>
						<enabled>false</enabled>
					</snapshots>
					<releases>
						<enabled>true</enabled>
					</releases>
				</repository>
			</repositories>
 
			<pluginRepositories>
				<pluginRepository>
					<id>gcube-snapshots</id>
					<name>gCube Snapshots</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-snapshots</url>
					<releases>
						<enabled>false</enabled>
					</releases>
					<snapshots>
						<enabled>true</enabled>
					</snapshots>
				</pluginRepository>
				<pluginRepository>
					<id>gcube-releases</id>
					<name>gCube Releases</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-releases</url>
					<snapshots>
						<enabled>false</enabled>
					</snapshots>
					<releases>
						<enabled>true</enabled>
					</releases>
				</pluginRepository>
				<pluginRepository>
					<id>gcube-externals</id>
					<name>gCube Externals</name>
					<url>http://maven.research-infrastructures.eu/nexus/content/repositories/gcube-externals</url>
					<snapshots>
						<enabled>false</enabled>
					</snapshots>
					<releases>
						<enabled>true</enabled>
					</releases>
				</pluginRepository>
			</pluginRepositories>
 
		</profile>
	</profiles>
 
	<activeProfiles>
		<activeProfile>gcube</activeProfile>
	</activeProfiles>
</settings>


Note:Developers must not configure these repositories in their POMs, only in settings.xml.

This is to:

  • keep their projects independent from future changes in repository locations;
  • avoid interference with ETICS builds, where access to these repositories may comprise correct system integration;


Third-party Repositories

Using the the settings.xml aforementioned, any dependency declared in pom.xml that can be found in one of gCube repositories or in Maven Central will be resolved at build time and deployment time without any further configuration. Frequently it may happens that a gCube component needs software stored on a public Maven repository not in on of those repositories. In this case, Maven provides an easy solution for declaring additional repositories for dependency resolution by adding a <repositories> section in the pom.xml of the project. E.g.: 

<repositories>
    <repository>
    	<id>[repository-id]</id>    	<url>[repository URL]</url>    </repository>
</repositories>

Though technically possible to add any Maven repository in this way, in order to preserve the quality of gCube software it is required to check trustworthiness of its dependencies disallowing usage of certain external software. To enforce such a control over public Maven repositories, gCube components are allowed to use external repository not directly but only through a proxy repository (a.k.a. shadow repository) created on CNR's Nexus Repository Portal (http://maven.research-infrastructures.eu/nexus/).

List of all repositories in Nexus Portal is available at http://maven.research-infrastructures.eu/nexus/index.html#view-repositories

Note: if a component uses a public Maven repository directly, it will work for building but would fail in deployment phase. In fact, Software Gateway is only able to download software accessible from Nexus.

How to request creation of a new proxy repository in Nexus

Proxy Repositories on Nexus are created by the Release Manager. To request a new proxy a new ticket must be created.

Details of ticket are:

  1. open a ticket of type "Task" in iMarine TRAC (at https://issue.imarine.research-infrastructures.eu/)
  2. ticket's description must report:
    • URL of repository to add
    • organization which hosts the repository
    • which artifacts are needed from that repository
  3. Type is Task
  4. Milestone field should be set to the gCube release in which the component is expected to be released
  5. Component field must report the gCube component that needs the external repository
  6. Blocking should list (if any) other tickets blocked by the addition of this Maven repository
  7. Task Category is software
  8. assign the ticket to the Release Manager

An example of ticket for requesting a new repository mirror is reported in the figure below.

Ticket for requesting a new Mirror Repository

Release Manager will process the ticket checking that no issues are foreseen from the usage of software stored in the repository. If request is accepted, a new proxy repository is created and the repository's URL will be communicated publicly on gCube's developers mailinglist (a.k.a. TCOM mailinglist).

Knowing the mirror repository URL, developer can use it adding following lines in pom.xml. E.g. 

<repositories>
    <repository>
    	<id>gcube-ext-osgeo</id>    	<url>http://maven.research-infrastructures.eu/nexus/content/repositories/osgeo/</url>    </repository>
</repositories>

Note: it is possible that software stored on proxy repositories is not included in search results on Nexus. This happens for those repositories that does not have indexes (or have indexes in a format that Nexus is not able to download). Anyway, once Nexus has downloaded artifacts from the original repository for the first time (e.g. because a compilation of a pom that uses the proxy repository), creates its own indexes and those artifacts will become available also for local search.


Dependency Versions

ETICS builds ensure correct integration of gCube components, regardless of the versions their dependencies. It also ensures that components are released with fixed versions for their dependencies, i.e. ensures build reproducibility.

Accordingly, developers can be liberal in the choice of dependency versions. i.e. rely on version ranges to:

  • minimise the requirements for version management;
  • be readily informed of any mis-alignment with their dependencies;

As we start adopting Maven, the lower-bound of version ranges will be a SNAPSHOT version. For example, a dependency on the streams library can be specified as follows: 

<dependency>
    <groupId>org.gcube.data.access</groupId>
    <artifactId>streams</artifactId>
    <version>[1.0.0-SNAPSHOT,2.0.0)</version>
</dependency>

With this dependency, a component will be soon notified of a non-retro compatible change that, contrary to conventions, streams may have introduced in, say, version 1.3.0 (typically the day after at the latest). The notification will be delivered in the IDE, assuming the availability of Maven integration (e.g. m2e in Eclipse). The component is in fact free to open the range further if it wants to be aligned with its dependencies across major versions.


Common Dependencies

Many gCube components depend on gcf, which is now available as a Maven component since version 1.4. At the time of writing, the component has not been released yet. A dependency on it is then specified as follows: 

<dependency>
    <groupId>org.gcube.core</groupId>
    <artifactId>gcf</artifactId>
    <version>[1.4.0-SNAPSHOT,2.0.0)</version>
    <scope>provided</scope>
</dependency>

The scope provided indicates that these dependency will be satisfied in the development environment (the gHN) and should not be included in GAR and SA archives.

Some components depend on the libraries available in all gHNs, though they do not depend on gcf itself. The subset of these libraries that are required for compilation and integration testing have been coalesced in a single Jar, distributed as a Maven component called ghn-core-runtime. A dependency on this Jar can be specified as follows: 

<dependency>
    <groupId>org.gcube.distribution</groupId>
    <artifactId>ghn-core-runtime</artifactId>
    <version>1.0.0</version>
    <scope>provided</scope>
</dependency>

Note: a component that depends on gcf indirectly depends on gun-core-runtime.

Another common dependency is on the gCube gRS2, an Ant component that the build infrastructure make available in Maven repositories. A dependency on this component can be specified as follows: 

<dependency>
    <groupId>org.gcube.execution</groupId>
    <artifactId>grs2library</artifactId>
    <version>[1.2.0-SNAPSHOT,2.0.0)</version>
</dependency>

Libraries

The streams library is a Maven component in gCube class DataAccess. We will use it as example of a library component in Maven.

Profile

The gCube Software profile of the library is as follows: 

<?xml version="1.0" encoding="UTF-8"?>
<Resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ID />
  <Type>Service</Type>
  <Profile>
    <Description>Embedded Domain-Specific Language for Stream Transformations</Description>
    <Class>DataAccess</Class>
    <Name>streams</Name>
     <Version>1.0.0</Version>
     <Packages>
       <Software>
         <Name>streams</Name>
         <Version>1.0.0-SNAPSHOT</Version>
         <MavenCoordinates>           <groupId>org.gcube.data.access</groupId>           <artifactId>streams</artifactId>           <version>1.0.0-SNAPSHOT</version>         </MavenCoordinates>         <Files>
           <File>streams-1.0.0-SNAPSHOT.jar</File>
         </Files>
       </Software>
     </Packages>
  </Profile>
</Resource>

Notice that:

  • the profile includes a single package and this package corresponds to the main build artefact of the component;
  • the whole profile and the package have usual gCube coordinates;
  • the package name is aligned with the Maven artifactId of the component;
  • the package version is aligned with the Maven version of the component;
  • the package includes MavenCoordinates that can be directly copied and pasted from the POM;
  • the package does not specify dependencies;
  • the package points to the main artefact of the Maven build;


The alignment between profile and POM can be exploited to simplify the management of the profile across different component versions. In particular, streams uses POM variables top define its profile: 

<?xml version="1.0" encoding="UTF-8"?>
<Resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ID />
  <Type>Service</Type>
  <Profile>
    <Description>${description}</Description>
     <Class>DataAccess</Class>
     <Name>${artifactId}</Name>
     <Version>1.0.0</Version>
     <Packages>
       <Software>
         <Name>${artifactId}</Name>
         <Version>${version}</Version>
         <MavenCoordinates>
            <groupId>${groupId}</groupId>
            <artifactId>${artifactId}</artifactId>
             <version>${version}</version>
         </MavenCoordinates>
	 <Files>
            <File>${build.finalName}.jar</File>
         </Files>
       </Software>
     </Packages>
</Profile>
</Resource>

Note that the variables must be resolved (i.e. the profile is interpolated) before the profile can be used within the system. In particular, the main destination of the profile is the gCube Software Archive (SA) which packages the component for registration within the system.

Service Archive

streams is responsible for generating its own SA as part of its Maven build. This requires dedicated logic in the POM but has the advantage that:

  • SA validity can be verified locally;
  • no SA configurations are required in ETICS;
  • the required build logic can be easily reused across components;

streams uses the Maven Assembly Plugin to generate its own SA, with an approach that makes optimal use of Maven variable interpolation in static files such as README, svnpath.txt, MAINTAINERS, changelog.xml, etc. In particular, the Assembly plugin takes care of variable interpolation in the profile and other static files. The approach is best illustrated with a reference to the [sources] of the streams library.


As we discuss below, streams has the final requirement to produce an interpolated profile during ETICS builds, outside the context of the SA. streams meets this requirement with the Maven Resources Plugin. Again, consult the [sources] of the streams library to reuse this approach.

Services

Portlets

Retrieved from "https://gcube.wiki.gcube-system.org/index.php?title=Developing_gCube_Maven_Components&oldid=15879"