Difference between revisions of "Continuous Integration: Releases Jenkins Pipeline"

From Gcube Wiki
Jump to: navigation, search
(Jenkins Pipeline Configuration)
 
(67 intermediate revisions by the same user not shown)
Line 1: Line 1:
  
 
+
= gCubeBuilder Pipeline Project =
[https://jenkins.io/doc/book/pipeline/ Jenkins Pipeline] is a combination of plugins that support the integration and implementation of continuous delivery pipelines using Jenkins. A pipeline has an extensible automation server for creating simple or complex delivery pipelines "as code," via pipeline DSL (Domain-specific Language).
+
 
+
= gCubeRelease Pipeline Project =
+
  
 
In gCube we use a Pipeline to trigger the builds of [[Jenkins_Projects_(aka_Jobs)|jobs]] forming a gCube Release. The pipeline project is available at:
 
In gCube we use a Pipeline to trigger the builds of [[Jenkins_Projects_(aka_Jobs)|jobs]] forming a gCube Release. The pipeline project is available at:
https://jenkins.d4science.org/job/gCube-Release/
+
https://jenkins.d4science.org/job/Pipeline-gCubeBuilder/
  
 
== Parameters==
 
== Parameters==
[[File:Jenkins_pipeline_params.png|800px]]
+
 
 +
[[File:Jenkins_release_pipeline_params_resume.png|600px]]
 +
 
 +
=== Type ===
 +
The type of build selects:
 +
# the Maven profile(s) to activate during the pipeline's and the components' jobs
 +
# the rules applied to each job. For instance, if a RELEASE type is selected, the jobs with a snapshot version on the master branch will fail.
 +
 
 +
The most updated documentation for profiles and rules is available in the [https://code-repo.d4science.org/gCubeSystem/maven-parent maven parent] project.
 +
 
 +
=== gCube release version ===
 +
The number of the gCube release to build. Sample values: 4.14, 4.15, etc.
 +
 
 +
=== Cleanup gCube artifacts ===
 +
If checked, all the gcube artifacts from the local maven repository will be wiped out before the builds start.
 +
 
 +
=== Clean up local repo ===
 +
If checked, all the gCube artifacts from the local maven repository will be wiped out before the builds start.
 +
 
 +
=== Resume from ===
 +
 
 +
This optional parameter expects a previous jenkins build number. If set, the build will resume from a previous build identified by the number and all the jobs succeeded in the original build are skipped in the new build.
 +
 
 
== Triggers ==
 
== Triggers ==
 
No triggers are defined because the pipeline is expected to be manually launched by the [[Continuous_Integration:_Releases_Manager|Release Manager]]:
 
No triggers are defined because the pipeline is expected to be manually launched by the [[Continuous_Integration:_Releases_Manager|Release Manager]]:
Line 24: Line 43:
 
= Jenkins Pipeline Definition =
 
= Jenkins Pipeline Definition =
 
== Git Repository ==
 
== Git Repository ==
The definition of the gCube release pipeline is maintained in this Git Repository: https://code-repo.d4science.org/gCubePipelines/gCubeRelease
+
The definition of the gCube release pipeline is maintained in this Git Repository: https://code-repo.d4science.org/gCubeCI/gCubeBuilder
  
== Requirements==
+
== Requirements on Jenkins ==
 
* [https://jenkins.io| Jenkins] ver. 2.164.2 or newer
 
* [https://jenkins.io| Jenkins] ver. 2.164.2 or newer
 
* [https://plugins.jenkins.io/workflow-aggregator| Pipeline Plugin]
 
* [https://plugins.jenkins.io/workflow-aggregator| Pipeline Plugin]
 
* [https://plugins.jenkins.io/workflow-basic-steps| Pipeline: Basic Steps]
 
* [https://plugins.jenkins.io/workflow-basic-steps| Pipeline: Basic Steps]
 
* [https://plugins.jenkins.io/pipeline-maven| Pipeline: Maven]
 
* [https://plugins.jenkins.io/pipeline-maven| Pipeline: Maven]
 +
* [https://plugins.jenkins.io/kubernetes| Kubernetes Plugin] (for the YAML parser)
 +
* [https://plugins.jenkins.io/nodelabelparameter| NodeLabelParameter Plugins] (for the Label's job parameter)
 +
 
* Jenkins configured with a JDK named 'OpenJDK 8'
 
* Jenkins configured with a JDK named 'OpenJDK 8'
 
* One or more Jenkins agents labeled as 'pipeline-agent'
 
* One or more Jenkins agents labeled as 'pipeline-agent'
 +
* a global variable named MAVEN_CONFIG_FOLDER that points to the base folder to use for the maven local repositories
  
== Basic Structure ==
+
== Requirements on Downstream Jobs ==
[https://code-repo.d4science.org/gCubePipelines/gCubeRelease/src/branch/master/Jenkinsfile ''Jenkinsfile''] defines a [https://jenkins.io/doc/book/pipeline/#declarative-pipeline-fundamentals  Declarative Pipeline]. The Pipeline’s code defines the entire build process of a gCube Release.
+
 
 +
In order to customize the build environment, the pipeline requires that each downstream job is configured with the following 3 parameters:
 +
* gcube_settings (type: String Parameter): the filename of the settings.xml for maven
 +
* local_repo (type: String Parameter): the name of the folder to use as maven local repository
 +
* exec_label (type: Label): the label assigned to the nodes to use for the executions
 +
 
 +
Here's a screenshot for the parameters and their default values:
  
This is the stub of the gCubeRelease pipeline.
+
TBP
  
 +
Also, the build command of each job must be the following:
 
<pre>
 
<pre>
    
+
--settings $MAVEN_CONFIG_FOLDER/$gcube_settings -Dmaven.repo.local=$MAVEN_CONFIG_FOLDER/$local_repo dependency:tree deploy
   // manage options and settings
+
</pre>
 +
 
 +
== Basic Structure ==
 +
 
 +
The pipeline [https://code-repo.d4science.org/gCubeCI/gCubeBuilder/src/branch/master/Jenkinsfile ''code''] (written in the Groovy language) models the entire build process of a gCube Release.
 +
 
 +
This is the stub of the gCubeRelease pipeline available in Git.
 +
 
 +
<syntaxhighlight lang="Groovy">
 +
   import org.yaml.snakeyaml.Yaml
 +
 +
   // manage options and settings here (not shown)
 +
 
 +
  // load and parse the release file
 +
  String releaseURL = "<git releases repo url>/releases/gcube-${gCube_release_version}.yaml"
 +
  def text = releaseURL.toURL().getText()
 +
  def jsonConfig = new Yaml().load(text)
  
 
   // pipeline
 
   // pipeline
 
   pipeline {
 
   pipeline {
 
         // run only on agents with the label
 
         // run only on agents with the label
         agent { label 'pipeline-agent' }
+
         agent { label 'CD' }
  
 
         // expected input parameters
 
         // expected input parameters
 
         parameters {
 
         parameters {
            string(name: 'myInput', description: 'Some pipeline parameters')
+
          choice(name: 'Type',
 +
                choices: ['SNAPSHOT-DRY-RUN', 'SNAPSHOT', 'RELEASE-DRY-RUN', 'RELEASE-STAGING', 'RELEASE'],
 +
                description: 'The type of artifacts the build is expected to generate')
 +
 
 +
          string(name: 'gCube_release_version',
 +
                defaultValue: 'x.y.z',
 +
                description: 'The number of the gCube release to build. Sample values: 4.14.1, 4.15, etc.')
 
         }
 
         }
  
Line 61: Line 113:
 
                 }
 
                 }
 
             }
 
             }
             stage('build group 1') {
+
            // the maven-parent needs to be built (once) at each execution
 +
             stage('build maven-parent') {
 
                 steps {
 
                 steps {
              withMaven(..maven settings here..) {
+
                  echo build(job: 'maven-parent', wait: true,
                      build 'job name 1'
+
                        parameters: [[$class: 'StringParameterValue', name: 'gcube_settings', value: "${maven_settings_file}"],
                      build 'job name 2'
+
                                    [$class: 'StringParameterValue', name: 'local_repo', value: "${maven_local_repo_path}"],
                      build 'job name 3'
+
                                    [$class: 'LabelParameterValue', name: 'exec_label', label: "CD", nodeEligibility: [$class: 'AllNodeEligibility']]
                      build 'job name 4'
+
                        ]).result
 +
                  echo "Done with maven-parent"
 +
              }
 +
            }
 +
            stage('build components') {
 +
            steps {
 +
                script {
 +
                    jsonConfig.gCube_release.Components.each { group_name, component_list ->
 +
                        stage("build ${group_name} components") {
 +
                            buildComponents items: component_list?.collect { "${it.name}" },
 +
                                    "${maven_settings_file}", "${maven_local_repo_path}"
 +
                            echo "Done with ${group_name} components"
 +
                        }
 
                     }
 
                     }
                    echo 'Done with group 1'
 
 
                 }
 
                 }
            }
+
            }
            stage('build group 2') {
+
          }
                steps {
+
                    ...
+
                }
+
            }
+
 
         }
 
         }
 
// post-build actions
 
// post-build actions
Line 99: Line 159:
  
 
     }
 
     }
</pre>
+
 
 +
    def buildComponents(args, maven_settings_file, maven_local_repo_path) {
 +
    if (args.items) {
 +
        parallel args.items?.collectEntries { name ->
 +
            ["${name}": {
 +
                if (name && !"NONE".equalsIgnoreCase(name))
 +
                    build(job: name,
 +
                            parameters: [[$class: 'StringParameterValue', name: 'gcube_settings', value: "${maven_settings_file}"],
 +
                                        [$class: 'StringParameterValue', name: 'local_repo', value: "${maven_local_repo_path}"],
 +
                                        [$class: 'LabelParameterValue', name: 'exec_label', label: "CD", nodeEligibility: [$class: 'AllNodeEligibility']]
 +
                            ])
 +
                }
 +
            ]
 +
 
 +
        }
 +
    }
 +
  }
 +
</syntaxhighlight>
  
 
== Reference Documentation ==
 
== Reference Documentation ==
  
 
[https://jenkins.io/doc/book/pipeline/syntax Pipeline Syntax]
 
[https://jenkins.io/doc/book/pipeline/syntax Pipeline Syntax]
 +
 +
[https://jenkins.io/doc/pipeline/steps/pipeline-build-step/#pipeline-build-step Build Step]
 +
 +
= Jenkins Pipeline Configuration =
 +
Each gCube release requires a configuration file (called ''release file'') written in the YAML format.
 +
 +
The file must be placed in the ''/open'' folder of the [https://code-repo.d4science.org/gCubeCI/gCubeReleases Git releases repository] and named as ''gcube-<version>.yaml'' (e.g. gcube-4.14.5.yaml). The file must report the gcube release version and the list of '''Jenkins jobs''' to build grouped in logical groups.
 +
 +
This is a sample release file with 3 groups (SmartGears, Enabling, Data):
 +
<syntaxhighlight lang="YAML">
 +
 +
 +
gCube_release:
 +
  Version: 4.14.5
 +
  Components:
 +
    SmartGear:
 +
      - name: maven-parent
 +
        version: 1.1.0
 +
      - name: gcube-bom
 +
        version: 1.4.0
 +
      - name: maven-smartgears-bom
 +
        version: 1.1.0
 +
      - name: authorization-client
 +
        version: 1.1.0
 +
      - name: gxRest
 +
        version: 1.1.2
 +
 +
    Enabling:
 +
      - name: information-system-bom
 +
        version: 1.1.0
 +
      - name: information-system-model
 +
        version: 1.1.0
 +
      - name: resource-registry-api
 +
        version: 1.1.0
 +
      - name: resource-registry-client
 +
        version: 1.1.0
 +
 +
    Data:
 +
      - NONE
 +
 +
</syntaxhighlight>
 +
Do note that if a logical group is added, a correspondent stage must be added to the [[Continuous_Integration:_Releases_Jenkins_Pipeline#Jenkins_Pipeline_Definition| pipeline definition]].
 +
 +
Executing the release pipeline with the previous release file results in something like the following on the Jenkins interface:
 +
 +
[[File:JenkinsPipelineProgress.png|600px]]
 +
 +
= Build Flow =
 +
 +
There are two levels of execution in the gCube pipeline:
 +
 +
# ''stages'': stages are '''executed sequentially''', one after the other;
 +
# ''steps'' inside each state: steps belonging to the same stage are '''executed in parallel'''.
 +
 +
These two levels are mapped in the pipeline configuration as follows:
 +
 +
* stages correspond to the ''logical groups''
 +
* steps execute the list of jobs inside each group.
 +
 +
For instance, in the following pseudo-config:
 +
<pre>
 +
group A
 +
  |
 +
  |-job1
 +
  |-job2
 +
  |-job3
 +
  |-job4
 +
group B
 +
  |
 +
  |-job5
 +
  |-job6
 +
  |-job7
 +
 +
</pre>
 +
 +
* A is executed before B
 +
* job1,job2,job3,job4 are executed in parallel
 +
* job5,job6,job7 are executed in parallel if all the jobs in A succeed (do not fail or become unstable).
 +
 +
= Build Commits Report =
 +
If the pipeline execution succeeds, it sends to the release manager a build commits report.
 +
 +
The report includes the following information for each component released:
 +
* the group id
 +
* the artifact id
 +
* the artifact version
 +
* the SCM url
 +
* the commit released
 +
* the maven url where the component was pushed
 +
* the name of the file pushed
 +
* the type of packaging for the artifact
 +
 +
Here's an example of build commits report:
 +
 +
[[File:Jenkins-build-commits-report.png|800px]]
  
  

Latest revision as of 04:20, 12 July 2020

gCubeBuilder Pipeline Project

In gCube we use a Pipeline to trigger the builds of jobs forming a gCube Release. The pipeline project is available at: https://jenkins.d4science.org/job/Pipeline-gCubeBuilder/

Parameters

Jenkins release pipeline params resume.png

Type

The type of build selects:

  1. the Maven profile(s) to activate during the pipeline's and the components' jobs
  2. the rules applied to each job. For instance, if a RELEASE type is selected, the jobs with a snapshot version on the master branch will fail.

The most updated documentation for profiles and rules is available in the maven parent project.

gCube release version

The number of the gCube release to build. Sample values: 4.14, 4.15, etc.

Cleanup gCube artifacts

If checked, all the gcube artifacts from the local maven repository will be wiped out before the builds start.

Clean up local repo

If checked, all the gCube artifacts from the local maven repository will be wiped out before the builds start.

Resume from

This optional parameter expects a previous jenkins build number. If set, the build will resume from a previous build identified by the number and all the jobs succeeded in the original build are skipped in the new build.

Triggers

No triggers are defined because the pipeline is expected to be manually launched by the Release Manager:

Jenkins pipeline triggers.png

It can be changed according to the release needs and the availability of a sufficient number of dedicate agents in Jenkins.

Git

The pipeline definition is maintained in a Git repository. This section connects the project to the Git repository.

Jenkins pipeline git.png

Jenkins Pipeline Definition

Git Repository

The definition of the gCube release pipeline is maintained in this Git Repository: https://code-repo.d4science.org/gCubeCI/gCubeBuilder

Requirements on Jenkins

  • Jenkins configured with a JDK named 'OpenJDK 8'
  • One or more Jenkins agents labeled as 'pipeline-agent'
  • a global variable named MAVEN_CONFIG_FOLDER that points to the base folder to use for the maven local repositories

Requirements on Downstream Jobs

In order to customize the build environment, the pipeline requires that each downstream job is configured with the following 3 parameters:

  • gcube_settings (type: String Parameter): the filename of the settings.xml for maven
  • local_repo (type: String Parameter): the name of the folder to use as maven local repository
  • exec_label (type: Label): the label assigned to the nodes to use for the executions

Here's a screenshot for the parameters and their default values:

TBP

Also, the build command of each job must be the following:

--settings $MAVEN_CONFIG_FOLDER/$gcube_settings -Dmaven.repo.local=$MAVEN_CONFIG_FOLDER/$local_repo dependency:tree deploy

Basic Structure

The pipeline code (written in the Groovy language) models the entire build process of a gCube Release.

This is the stub of the gCubeRelease pipeline available in Git.

  import org.yaml.snakeyaml.Yaml
 
  // manage options and settings here (not shown)
 
  // load and parse the release file
  String releaseURL = "<git releases repo url>/releases/gcube-${gCube_release_version}.yaml"
  def text = releaseURL.toURL().getText()
  def jsonConfig = new Yaml().load(text)
 
  // pipeline
  pipeline {
        // run only on agents with the label
        agent { label 'CD' }
 
        // expected input parameters
        parameters {
          choice(name: 'Type',
                 choices: ['SNAPSHOT-DRY-RUN', 'SNAPSHOT', 'RELEASE-DRY-RUN', 'RELEASE-STAGING', 'RELEASE'],
                 description: 'The type of artifacts the build is expected to generate')
 
          string(name: 'gCube_release_version',
                 defaultValue: 'x.y.z',
                 description: 'The number of the gCube release to build. Sample values: 4.14.1, 4.15, etc.')
        }
 
 
        stages {
            stage('preliminary steps') {
               //prepare the environment for the builds
                steps {
                       //execute steps here if needed
                }
            }
            // the maven-parent needs to be built (once) at each execution
            stage('build maven-parent') {
                steps {
                  echo build(job: 'maven-parent', wait: true,
                        parameters: [[$class: 'StringParameterValue', name: 'gcube_settings', value: "${maven_settings_file}"],
                                     [$class: 'StringParameterValue', name: 'local_repo', value: "${maven_local_repo_path}"],
                                     [$class: 'LabelParameterValue', name: 'exec_label', label: "CD", nodeEligibility: [$class: 'AllNodeEligibility']]
                        ]).result
                   echo "Done with maven-parent"
               }
            }
            stage('build components') {
             steps {
                script {
                    jsonConfig.gCube_release.Components.each { group_name, component_list ->
                        stage("build ${group_name} components") {
                            buildComponents items: component_list?.collect { "${it.name}" },
                                    "${maven_settings_file}", "${maven_local_repo_path}"
                            echo "Done with ${group_name} components"
                        }
                    }
                }
             }
           }
        }
	// post-build actions
	post {
		always {
			echo 'This will always run'
		}
		success {
			echo 'This will run only if successful'
		}
		failure {
			echo 'This will run only if failed'
		}
		unstable {
			echo 'This will run only if the run was marked as unstable'
		}
		changed {
			echo 'This will run only if the state of the Pipeline has changed'
			echo 'For example, if the Pipeline was previously failing but is now successful'
		}
	}
 
    }
 
    def buildComponents(args, maven_settings_file, maven_local_repo_path) {
     if (args.items) {
        parallel args.items?.collectEntries { name ->
            ["${name}": {
                if (name && !"NONE".equalsIgnoreCase(name))
                    build(job: name,
                            parameters: [[$class: 'StringParameterValue', name: 'gcube_settings', value: "${maven_settings_file}"],
                                         [$class: 'StringParameterValue', name: 'local_repo', value: "${maven_local_repo_path}"],
                                         [$class: 'LabelParameterValue', name: 'exec_label', label: "CD", nodeEligibility: [$class: 'AllNodeEligibility']]
                            ])
                }
            ]
 
        }
    }
  }

Reference Documentation

Pipeline Syntax

Build Step

Jenkins Pipeline Configuration

Each gCube release requires a configuration file (called release file) written in the YAML format.

The file must be placed in the /open folder of the Git releases repository and named as gcube-<version>.yaml (e.g. gcube-4.14.5.yaml). The file must report the gcube release version and the list of Jenkins jobs to build grouped in logical groups.

This is a sample release file with 3 groups (SmartGears, Enabling, Data):

gCube_release:
  Version: 4.14.5
  Components:
    SmartGear:
      - name: maven-parent
        version: 1.1.0
      - name: gcube-bom
        version: 1.4.0
      - name: maven-smartgears-bom
        version: 1.1.0
      - name: authorization-client
        version: 1.1.0
      - name: gxRest
        version: 1.1.2

    Enabling:
      - name: information-system-bom
        version: 1.1.0
      - name: information-system-model
        version: 1.1.0
      - name: resource-registry-api
        version: 1.1.0
      - name: resource-registry-client
        version: 1.1.0

    Data:
      - NONE

Do note that if a logical group is added, a correspondent stage must be added to the pipeline definition.

Executing the release pipeline with the previous release file results in something like the following on the Jenkins interface:

JenkinsPipelineProgress.png

Build Flow

There are two levels of execution in the gCube pipeline:

  1. stages: stages are executed sequentially, one after the other;
  2. steps inside each state: steps belonging to the same stage are executed in parallel.

These two levels are mapped in the pipeline configuration as follows:

  • stages correspond to the logical groups
  • steps execute the list of jobs inside each group.

For instance, in the following pseudo-config:

 group A
   |
   |-job1
   |-job2
   |-job3
   |-job4
 group B
   |
   |-job5
   |-job6
   |-job7

  • A is executed before B
  • job1,job2,job3,job4 are executed in parallel
  • job5,job6,job7 are executed in parallel if all the jobs in A succeed (do not fail or become unstable).

Build Commits Report

If the pipeline execution succeeds, it sends to the release manager a build commits report.

The report includes the following information for each component released:

  • the group id
  • the artifact id
  • the artifact version
  • the SCM url
  • the commit released
  • the maven url where the component was pushed
  • the name of the file pushed
  • the type of packaging for the artifact

Here's an example of build commits report:

Jenkins-build-commits-report.png


Back to the CI guide.