Jenkins Part 5.1: Using the Job DSL for automatic Creation of Jenkins Jobs


Today, we will learn how to use the Jenkins Job DSL Plugin to create new Jenkins jobs at a push of a button. We will show how we can use Groovy scripts for defining a “Hello World” Jenkins freestyle project and create such a project by pushing the “Build now” button.

Why creating Jenkins Jobs via Groovy Scripts?

You might say: “Jenkins has such a nice Web interface. Why should I bother about creation of Jenkins Jobs via Groovy Scripts?”

Right: the web interface of Jenkins is simple to use and will lead to quick results for single projects. However, consider a situation with more than 100 projects with more than 400 Jenkins pipelines (e.g. build + test + deploy + release pipelines each). Automating the Jenkins job lifecycle can be a great time-saver here. Automating the Jenkins job lifecycle is best done with the Seed plugin:

From: https://blog.codecentric.de/en/2015/10/using-jenkins-job-dsl-for-job-lifecycle-management/
From: https://blog.codecentric.de/en/2015/10/using-jenkins-job-dsl-for-job-lifecycle-management/ showing a full Jenkins job lifecycle, which is the long term target for us. The current post is a first step towards this target.

The Seed plugin relies on the capabilities of the Job DSL plugin discussed in the current blog post. In one of the next blog posts, we will have a closer look to the Seed plugin.

 

Tools and Versions used

      • Vagrant 1.8.6
      • Virtualbox 5.0.20
      • Docker 1.12.1
      • Jenkins 2.32.2
      • Job DSL Plugin 1.58

Prerequisites:

      • Free DRAM for the a Docker Host VM >~ 4 GB.
      • Docker Host is available.
      • Tested with 2 vCPU (1 vCPU might work as well).

Step 1: Connect to Jenkins

Step 1.1: Start Jenkins in interactive Terminal Mode

Make sure that port 8080 is unused on the Docker host. If you were following all steps in part 1 of the series, you might need to stop cadvisor:

(dockerhost)$ sudo docker stop cadvisor

I assume that jenkins_home is already created, all popular plugins are installed and an Admin user has been created as shown in part 1 of the blog series. We start the Jenkins container with the jenkins_home Docker host volume mapped to /var/jenkins_home:

(dockerhost)$ cd <path_to_jenkins_home> # in my case: cd /vagrant/jenkins_home/
(dockerhost:jenkins_home)$ sudo docker run -it --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins
Running from: /usr/share/jenkins/jenkins.war
...
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

50c150e35a774cexxxxxxxxxxxxxxx

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
...
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Step 1.2: Open Jenkins in a Browser

Now we want to connect to the Jenkins portal. For that, open a browser and open the URL

<your_jenkins_host>:8080

In our case, Jenkins is running in a container and we have mapped the container-port 8080 to the local port 8080 of the Docker host. On the Docker host, we can open the URL.

localhost:8080

Note: In case of Vagrant with VirtualBox, per default, there is only a NAT-based interface and you need to create port-forwarding for any port you want to reach from outside (also the local machine you are working on is to be considered as outside). In this case, we need to add an entry in the port forwarding list of VirtualBox:
2016-11-30-19_22_22-regel-fur-port-weiterleitung

Note that this configuration is not permanent, unless you define the port mappings in the Vagrantfile as follows (see official Vagrant documentation):

config.vm.network "forwarded_port", guest: 8080, host: 8080

Step 1.3: Initialize Jenkins: Unlock Jenkins

Unlock Jenkins

insert the one-time password found in the log during startup or on /var/jenkins_home/secrets/initialAdminPassword

-> Continue

Step 1.4: Initialize Jenkins: Install Plugins

-> Install suggested plugins

Installing suggested plugins

Wait for the installation process to complete.

Step 1.4: Initialize Jenkins: Create Admin User

-> Create First Admin User

-> Save and Finish or “Continue as admin”

-> Start using Jenkins

Note: I recommend to log out and log in in order to test the login.

Step 2: Install the Job DSL Plugin

The Job DSL Plugin can be installed like any other Jenkins plugin:

-> Manage Jenkins
-> Manage Plugins
-> Plugins - Available

-> Filter job-dsl (with dash between job and dsl; wait for the filter to become active and do not press enter, otherwise you will get an error message)

-> Install without restart

Choose: Job DSL plugin

Job DSL: Installation Success

Step 3: Create Job DSL Jenkins Project

We create a Job DSL Job like follows:

-> Back to Dashboard

-> New Item

-> Enter Name: JobDSLJob

-> Freestyle Project

-> OK

Step 4: Configure Job DSL Project

-> Add build step: Process Job DSLs

-> if you have got a Github account, fork this open source Java Hello World software (originally created by of LableOrg) that will allow you to see, what happens with your Jenkins job, if you check in changed code. Moreover the hello world software allows you to perform JUnit 4 tests, run PowerMockito Mock services, run JUnit 4 Integration tests and calculate the code coverage using the tool Cobertura.

-> Use the provided DSL script

-> insert:

job('Job-DSL-Hello-World-Job') {
    scm {
        git('git://github.com/oveits/java-maven-junit-helloworld')
    }
    triggers {
        scm('H/15 * * * *')
    }
    steps {
        maven('-e clean test')
    }
}

here, exchange the username oveits by your own Github username.
2017-02-25-11_50_36-jobdsljob-config-jenkins

-> Save

Step 5: Prepare Maven Usage

Goto Jenkins -> Manage Jenkins -> Global Tool Configuration (available for Jenkins >2.0)

2016-12-09-11_34_55-manage-jenkins-jenkins

2016-12-09-11_35_26-global-tool-configuration-jenkins

Scroll down to Maven -> Add Maven

2017-01-02-14_32_46-global-tool-configuration-jenkins

-> choose Version (3.3.9 in my case)

-> Add a name (“Maven 3.3.9” in my case)

-> Save

Since we have checked “Install automatically” above, I expect that it will be installed automatically on first usage.

Step 6: Prepare Git Usage

As described in this StackOverflow Q&A, we need to add the Git username and email address, since Jenkins tries to tag and commit on the Git repo, which requires those configuration items to be set. For that, we perform:

-> 2017-02-25-18_04_32-job-dsl-hello-world-job-1-console-jenkins

-> Manage Jenkins

-> Configure System

-> scroll down to “Git plugin”

-> Git plugin; user.name = jenkins and user.email = admin@jenkins.org

Step 7: Create Jenkins Job from Code

Step 7.1 Build Project

-> Build Now

Step 7.2 (optional): Check Console Output

-> Build History: click on latest #nnn

-> Console Output

JobDSLJob: Console Output showing Added items: GeneratedJob{name='Job-DSL-Hello-World-Job'} and Finished: SUCCESS

->  Back to Project

Step 7.3: Review automatically built Project

-> 2017-02-25-11_48_23-update-center-jenkins

Jenkins Dashboard showing automatically created Job-DSL-Hello-World-Job

-> Job-DSL-Hello-World-Job

Project Job-DSL-Hello-World-Job showing build failure

This is showing a build failure, since I had not performed Step 5 and 6 before. In your case, it should be showing a success (in blue). If you are experiencing problems here, check out the Appendices below.

-> Configure

-> scroll down to Source Code Management

Source Code Management shows correct git URL

-> Scroll down to Build Triggers

Build Triggers showing Poll SCM every 15 minutes

-> Scroll down to Build

-> verify that “Maven 3.3.9” is chosen as defined in Step 5

-> enter “-e clean test” as Maven Goal

Maven Version "Maven 3.3.9" and Maven Goal "-e clean test"

-> Build Now

See, what happens by clicking on:

-> Build History

-> #nnn

-> Console Output

If everything went fine, we will see many downloads and a “BUILD SUCCESS”:

Job DSL Hello World Job: BUILD SUCCESS

Appendix A: Solve Git Problem: “tell me who you are”

Symptoms: Git Error: status code 128

In a new installation of Jenkins, Git does not seem to work out of the box. You can see this by choosing the Jenkins project Job-DSL-Hello-World-Job on the dashboard, then click “build now”, if the build was not already automatically triggered. Then:

-> Build History

-> Last Build (link works only, if Jenkins is running on localhost:8080 and you have chosen the same job name)

-> Console Output

There, we will see:

Caused by: hudson.plugins.git.GitException: Command "git tag -a -f -m Jenkins Build #1 jenkins-Job-DSL-Hello-World-Job-1" returned status code 128:
stdout: 
stderr: 
*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: empty ident name (for <jenkins@61915398735e.(none)>) not allowed

Resolution:

Step 1: Enter Git Username and Email

As described in this StackOverflow Q&A: we can resolve this issue by either suppressing the git tagging, or (I think this is better) by adding your username and email address to git:

-> 2017-02-25-18_04_32-job-dsl-hello-world-job-1-console-jenkins

-> Manage Jenkins

-> Configure System

-> scroll down to “Git plugin”

-> Git plugin; user.name = jenkins and user.email = admin@jenkins.org

Step 2: Re-run “Build Now” on the Project

To test the new configuration, we go to

-> the Job-DSL-Hello-World-Job and press

-> Build Now

Now, we should see a BUILD SUCCESS like follows:

-> Build History

-> #nnn

-> Console Output

If everything went fine, we will a “BUILD SUCCESS”:

Job DSL Hello World Job: BUILD SUCCESS

Appendix B: Maven Error: Cannot run program “mvn”

Symptoms:

When running a Maven Goal, the following error may appear on the Console log:

FATAL: command execution failed
java.io.IOException: Cannot run program "mvn" (in directory "/var/jenkins_home/workspace/Job-DSL-Hello-World-Job"): error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at hudson.Proc$LocalProc.(Proc.java:245)
	at hudson.Proc$LocalProc.(Proc.java:214)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:846)
	at hudson.Launcher$ProcStarter.start(Launcher.java:384)
	at hudson.Launcher$ProcStarter.join(Launcher.java:395)
	at hudson.tasks.Maven.perform(Maven.java:367)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
	at hudson.model.Build$BuildExecution.build(Build.java:205)
	at hudson.model.Build$BuildExecution.doRun(Build.java:162)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:404)
Caused by: java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 15 more
Build step 'Invoke top-level Maven targets' marked build as failure
Finished: FAILURE

Resolution:

Perform Step 5

2017-01-02-14_32_46-global-tool-configuration-jenkins

and

For Test, you can test a manual: choose the correct Maven version, when configuring a Maven build step like in Step 7:

Maven Version "Maven 3.3.9" and Maven Goal "-e clean test"

and verify that Build Now does not throw the Maven error anymore.

For our case, we need to correct the Job DSL like follows:

In the Script, we had defined the step:

    steps {
        maven('-e clean test')
    }

However, we need to define the Maven Installation like follows:

    steps {
        maven {
            mavenInstallation("Maven 3.3.9")
            goals('-e clean test')
        }
    }

Here, the mavenInstallation needs to specify the exact same name, as the one we have chosen in Step 5 above.

After correction, we will receive the correct Maven goal

-> JobDSL -> Build Now

Now, we can check the Maven configuration:

-> Jenkins Home leading to the Dashboard

-> Job-DSL-Hello-World-Job

-> Configure

After scrolling down, we will see the correct Maven Version:

Maven Version "Maven 3.3.9" and Maven Goal "-e clean test"

DONE

Appendix C: Updating Jenkins

Updating Jenkins (in my case: from 2.32.1 to 2.32.2) was as simple as following the steps below

Note: you might want to make a backup of your jenkins_home though. Just in case…

(dockerhost)$ cd <path_to_jenkins_home> # in my case: cd /vagrant/jenkins_home/
(dockerhost)$ docker pull jenkins # to update the jenkins image
(dockerhost)$ docker rm jenkins # to make shure the container named jenkins is removed
(dockerhost:jenkins_home)$ sudo docker run -d --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins

However, after that, some data was unreadable:

2017-02-24-20_04_36-manage-old-data-jenkins

I have clicked

-> Manage Jenkins
-> Manage
-> Discard Unreadable Data

to resolve the issue (hopefully…). At least, after that, the warning was gone.

Appendix D: Job DSL Syntax

The reference for the Job DSL syntax can be found on the Job DSL Plugin API pages. As an example, the syntax of Maven within a Freestyle project can be found on this page found via the path

> freeStyleJob > steps > maven:

maven {

  • // Allows direct manipulation of the generated XML.

    configure(Closure configureBlock)

  • // Specifies the goals to execute including other command line options.

    goals(String goals)

  • // Skip injecting build variables as properties into the Maven process.

    injectBuildVariables(boolean injectBuildVariables = true)

  • // Set to use isolated local Maven repositories.

    localRepository(javaposse.jobdsl.dsl.helpers.LocalRepositoryLocation location)

  • // Specifies the Maven installation for executing this step.

    mavenInstallation(String name)

  • // Specifies the JVM options needed when launching Maven as an external process.

    mavenOpts(String mavenOpts)

  • // Adds properties for the Maven build.

    properties(Map props)

  • // Adds a property for the Maven build.

    property(String key, String value)

  • // Specifies the managed global Maven settings to be used.

    providedGlobalSettings(String settingsIdOrName)

  • // Specifies the managed Maven settings to be used.

    providedSettings(String settingsIdOrName)

  • // Specifies the path to the root POM.

    rootPOM(String rootPOM)

}

A Maven example can be found on the same page:

job('example') {
    steps {
        maven('verify')
        maven('clean verify', 'module-a/pom.xml')
        maven {
            goals('clean')
            goals('verify')
            mavenOpts('-Xms256m')
            mavenOpts('-Xmx512m')
            localRepository(LocalRepositoryLocation.LOCAL_TO_WORKSPACE)
            properties(skipTests: true)
            mavenInstallation('Maven 3.1.1')
            providedSettings('central-mirror')
        }
    }
}

Summary

In this blog post, we have learned how to

  1. Start and initialize Jenkins via Docker
  2. Prepare the usage of Git and Maven
  3. Install the Job DSL Plugin
  4. Define a Jenkins Job via Groovy script
  5. Create a Jenkins Job by a push of  the “Build now” button
  6. Review and run the automatically created Jenkins job

We have seen that the usage of the Job DSL is no rocket science. The only topic, we had to take care, is, that Git and Maven need to be prepared for first usage on a Jenkins server.

More sophisticated Jenkins jobs can be created by following the documentation on the Job DSL Plugin API pages.

Now, we are ready to explore the Seed plugin, which relies on the Job DSL plugin and helps to fully automate the Jenkins job lifecycle. Our aim is, that one or more Jenkins job is automatically created, when a new project is created on the Git repository. If this is possible, we will see in one of my next blog posts.

Further Reading

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s