Difference between revisions of "Continuous Integration: Releases Jenkins Pipeline"

From Gcube Wiki
Jump to: navigation, search
(Requirements)
(Requirements on Downstream Jobs)
Line 78: Line 78:
  
 
== Requirements on Downstream Jobs ==
 
== Requirements on Downstream Jobs ==
 +
 +
In order to customize the build environment, the pipeline requires that each downstream job is configure with the following 3 parameters:
 +
* gcube_settings (type: String Parameter): the filename of the settings.xml for maven
 +
* local_repo (type: String Parameter): the name of the folder to use as maven local repository
 +
* exec_label (type: Label): the label assigned to the nodes to use for the executions
 +
 +
Here's a screenshot for the parameters and their default values:
  
 
TBP
 
TBP
 +
 +
Also, the build command of each job must be the following:
 +
<pre>
 +
--settings $MAVEN_CONFIG_FOLDER/$gcube_settings -Dmaven.repo.local=$MAVEN_CONFIG_FOLDER/$local_repo dependency:tree deploy
 +
</pre>
  
 
== Basic Structure ==
 
== Basic Structure ==

Revision as of 21:09, 1 October 2019


Jenkins Pipeline is a combination of plugins that support the integration and implementation of continuous delivery pipelines using Jenkins. A pipeline has an extensible automation server for creating simple or complex delivery pipelines "as code," via pipeline DSL (Domain-specific Language).

Continuous Delivery and Jenkins

Jenkins has proven to be an expert in implementing continuous integration, continuous testing and continuous deployment to produce good quality software. When it comes to continuous delivery, Jenkins uses a feature called Jenkins pipeline. In order to understand why Jenkins pipeline was introduced, we have to understand what continuous delivery is and why it is important.

In simple words, continuous delivery is the capability to release a software at all times. It is a practice which ensures that the software is always in a production-ready state.

Jenkins Pipelines

What is a Jenkins pipeline?

A pipeline is a collection of jobs that brings the software from version control into the hands of the end users by using automation tools. It is a feature used to incorporate continuous delivery in our software development workflow.

Over the years, there have been multiple Jenkins pipeline releases including, Jenkins Build flow, Jenkins Build Pipeline plugin, Jenkins Workflow, etc. What are the key features of these plugins?

  • They represent multiple Jenkins jobs as one whole workflow in the form of a pipeline.
  • What do these pipelines do? These pipelines are a collection of Jenkins jobs which trigger each other in a specified sequence.

The maintenance cost for such a complex pipeline is huge and increases with the number of processes. It also becomes tedious to build and manage such a vast number of jobs. To overcome this issue, a new feature called Jenkins Pipeline Project was introduced.

The key feature of this pipeline is to define the entire deployment flow through code. What does this mean? It means that all the standard jobs defined by Jenkins are manually written as one whole script and they can be stored in a version control system. It basically follows the pipeline as code discipline. Instead of building several jobs for each phase, you can now code the entire workflow and put it in a Jenkinsfile. Below is a list of reasons why you should use the Jenkins Pipeline.

Jenkins Pipeline Advantages

  • It models simple to complex pipelines as code by using Groovy DSL (Domain Specific Language)
  • The code is stored in a text file called the Jenkinsfile which can be checked into a SCM (Source Code Management)
  • Improves user interface by incorporating user input within the pipeline
  • It is durable in terms of unplanned restart of the Jenkins master
  • It can restart from saved checkpoints
  • It supports complex pipelines by incorporating conditional loops, fork or join operations and allowing tasks to be performed in parallel
  • It can integrate with several other plugins

What is a Jenkinsfile?

A Jenkinsfile is a text file that stores the entire workflow as code and it can be checked into a SCM on your local system. How is this advantageous? This enables the developers to access, edit and check the code at all times.

The Jenkinsfile is written using the Groovy DSL and it can be created through a text/groovy editor or through the configuration page on the Jenkins instance. It is written based on two syntaxes, namely:

  • Declarative pipeline syntax
  • Scripted pipeline syntax

Declarative pipeline is a relatively new feature that supports the pipeline as code concept. It makes the pipeline code easier to read and write. This code is written in a Jenkinsfile which can be checked into a source control management system such as Git. gCube uses declarative pipelines to control the Continuous Delivery process.

gCubeRelease Pipeline Project

In gCube we use a Pipeline to trigger the builds of jobs forming a gCube Release. The pipeline project is available at: https://jenkins.d4science.org/job/gCube-Release/

Parameters

Jenkins pipeline params.png

Triggers

No triggers are defined because the pipeline is expected to be manually launched by the Release Manager:

Jenkins pipeline triggers.png

It can be changed according to the release needs and the availability of a sufficient number of dedicate agents in Jenkins.

Git

The pipeline definition is maintained in a Git repository. This section connects the project to the Git repository.

Jenkins pipeline git.png

Jenkins Pipeline Definition

Git Repository

The definition of the gCube release pipeline is maintained in this Git Repository: https://code-repo.d4science.org/gCubeCI/gCubeRelease

Requirements on Jenkins

  • Jenkins configured with a JDK named 'OpenJDK 8'
  • One or more Jenkins agents labeled as 'pipeline-agent'

Requirements on Downstream Jobs

In order to customize the build environment, the pipeline requires that each downstream job is configure with the following 3 parameters:

  • gcube_settings (type: String Parameter): the filename of the settings.xml for maven
  • local_repo (type: String Parameter): the name of the folder to use as maven local repository
  • exec_label (type: Label): the label assigned to the nodes to use for the executions

Here's a screenshot for the parameters and their default values:

TBP

Also, the build command of each job must be the following:

--settings $MAVEN_CONFIG_FOLDER/$gcube_settings -Dmaven.repo.local=$MAVEN_CONFIG_FOLDER/$local_repo dependency:tree deploy

Basic Structure

Jenkinsfile defines a Declarative Pipeline with the Groovy DSL. The Pipeline’s code models the entire build process of a gCube Release.

This is the stub of the gCubeRelease pipeline available in Git.

  import org.yaml.snakeyaml.Yaml
 
  // manage options and settings here (not shown)
 
  // load and parse the release file
  String releaseURL = "<git repo url>/releases/gcube-${gCube_release_version}.yaml"
  def text = releaseURL.toURL().getText()
  def jsonConfig = new Yaml().load(text)
 
  // pipeline
  pipeline {
        // run only on agents with the label
        agent { label 'pipeline-agent' }
 
        // expected input parameters
        parameters {
          choice(name: 'Type',
                 choices: ['SNAPSHOT-DRY-RUN', 'SNAPSHOT', 'RELEASE-DRY-RUN', 'RELEASE-STAGING', 'RELEASE'],
                 description: 'The type of artifacts the build is expected to generate')
 
          string(name: 'gCube_release_version',
                 defaultValue: 'x.y.z',
                 description: 'The number of the gCube release to build. Sample values: 4.14.1, 4.15, etc.')
        }
 
 
        stages {
            stage('preliminary steps') {
               //prepare the environment for the builds
                steps {
                       //execute steps here if needed
                }
            }
 
            stage('build <group1 name> components') {
                steps {
                  withMaven(jdk: "${maven_jdk}", mavenLocalRepo: "${maven_local_repo_path}", mavenSettingsFilePath: "${maven_settings_file}") {
                    buildComponents items: jsonConfig.gCube_release.Components.<group1 name>?.collect { "${it.name}" }
                  }
                  echo "Done with <group1 name> components"
               }
            }
 
            stage('build <group2 name> components') {
                steps {
                  withMaven(jdk: "${maven_jdk}", mavenLocalRepo: "${maven_local_repo_path}", mavenSettingsFilePath: "${maven_settings_file}") {
                    buildComponents items: jsonConfig.gCube_release.Components.<group2 name>?.collect { "${it.name}" }
                  }
                  echo "Done with <group2 name> components"
               }
            }
        }
	// post-build actions
	post {
		always {
			echo 'This will always run'
		}
		success {
			echo 'This will run only if successful'
		}
		failure {
			echo 'This will run only if failed'
		}
		unstable {
			echo 'This will run only if the run was marked as unstable'
		}
		changed {
			echo 'This will run only if the state of the Pipeline has changed'
			echo 'For example, if the Pipeline was previously failing but is now successful'
		}
	}
 
    }
 
    def buildComponents(args) {
      if (args.items) {
        parallel args.items?.collectEntries { name -> [ "${name}": {
            if (!"NONE".equalsIgnoreCase(name))
                build name
        }]}
      }
    }

Reference Documentation

Pipeline Syntax

Jenkins Pipeline Configuration

Each gCube release requires a configuration file (called release file) written in the YAML format.

The file must be placed in the /release folder of the Git repository and named as gcube-<version>.yaml (e.g. gcube-4.14.5.yaml). The file must report the gcube release version and the list of Jenkins jobs to build grouped in logical groups.

This is a sample release file with 3 groups (SmartGears, Enabling, Data):

gCube_release:
  Version: 4.14.5
  Components:
    SmartGear:
      - name: maven-parent
        version: 1.1.0
      - name: gcube-bom
        version: 1.4.0
      - name: maven-smartgears-bom
        version: 1.1.0
      - name: authorization-client
        version: 1.1.0
      - name: gxRest
        version: 1.1.2

    Enabling:
      - name: information-system-bom
        version: 1.1.0
      - name: information-system-model
        version: 1.1.0
      - name: resource-registry-api
        version: 1.1.0
      - name: resource-registry-client
        version: 1.1.0

    Data:
      - NONE

Do note that if a logical group is added, a correspondent stage must be added to the pipeline definition.

Executing the release pipeline with the previous release file results in something like the following on the Jenkins interface: JenkinsPipelineProgress.png

Build Flow

There are two levels of execution in the gCube pipeline:

  1. stages: stages are executed sequentially, one after the other;
  2. steps inside each state: steps belonging to the same stage are executed in parallel.

These two levels are mapped in the pipeline configuration as follows:

  • stages correspond to the logical groups
  • steps execute the list of jobs inside each group.

For instance, in the following pseudo-config:

 group A
   |
   |-job1
   |-job2
   |-job3
   |-job4
 group B
   |
   |-job5
   |-job6
   |-job7

  • A is executed before B
  • job1,job2,job3,job4 are executed in parallel
  • job5,job6,job7 are executed in parallel if all the jobs in A succeed (do not fail or become unstable).

Back to the CI guide.