1

Jenkins Part 4.2: Code Quality Tests via Checkstyle


2017-01-19-23_30_11

Today, we will show how to use Checkstyle for improving the style of Java code. First, we will add Checkstyle to Gradle in order to create XML reports for a single build. Jenkins allows us to visualize the results of more than one test/build run into historic reports. After that, we will show, how a developer can use the Eclipse Checkstyle plugin in order to create better code:

2017-02-01-04_43_16-github-triggered-build-jenkins

This blog post series is divided into following parts:

    • Part 1: Installation and Configuration of Jenkins, loading Plugins
    • Part 2: Creating our first Jenkins job: GitHub download and Software build
    • Part 3: Periodic and automatically triggered Builds
    • Part 4.1: running automated tests: Functional Tests via Java JUnit
    • Part 4.2: running automated tests: Code Quality Test via Checkstyle (this post)
    • Part 4.3: running automated tests: Performance Tests with JMeter (work in progress)

What is Jenkins?

Jenkins is the leading open source automation server mostly used in continuous integration and continuous deployment pipelines. Jenkins provides hundreds of plugins to support building, deploying and automating any project.

 

Jenkins build, test and deployment pipeline
Jenkins build, test and deployment pipeline

A typical workflow is visualized above: a developer checks in the code changes into the repository. Jenkins will detect the change, build (compile) the software, test it and prepare to deploy it on a system. Depending on the configuration, the deployment is triggered by a human person, or automatically performed by Jenkins.

For more information, see the introduction found in part 1 of this blog series.

Checking Code with Checkstyle

In this post, we will show how to configure Jenkins for automated code checking as part of the Post-Build Tests:

2017-01-19-04_15_46

After this tutorial has been followed, we will have learned how to apply standard or custom checks on the code quality using Checkstyle in Eclipse and Jenkins.

Tools & Versions used

      • Vagrant 1.8.6
      • Virtualbox 5.0.20
      • Docker 1.12.1
      • Jenkins 2.32.1
        • Checkstyle Plug-in 3.47
      • Eclipse Kepler Service Release 2 (Build id: 20140224-0627)
        • Checkstyle Plug-in 7.2.0.201611082205

Prerequisites:

      • Free DRAM for the a Docker Host VM >~ 4 GB
      • Docker Host is available, Jenkins is installed and a build process is configured. For that, perform all steps in part 1 to part 3 of this blog series (new: you now can skip part 1, if you wish)
      • Tested with 2 vCPU (1 vCPU might work as well)

Step 1: Start Jenkins in interactive Terminal Mode

Make sure that port 8080 is unused on the Docker host. If you were following all steps in part 1 of the series, you might need to stop cadvisor:

(dockerhost)$ sudo docker stop cadvisor

I assume that jenkins_home is already created, all popular plugins are installed and an Admin user has been created as shown in part 1 of the blog series. We start the Jenkins container with the jenkins_home Docker host volume mapped to /var/jenkins_home:

(dockerhost)$ cd <path_to_jenkins_home> # in my case: cd /vagrant/jenkins_home/
(dockerhost:jenkins_home)$ sudo docker run -it --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins
Running from: /usr/share/jenkins/jenkins.war
...
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Step 2: Open Jenkins in a Browser

Now we want to connect to the Jenkins portal. For that, open a browser and open the URL

<your_jenkins_host>:8080

In our case, Jenkins is running in a container and we have mapped the container-port 8080 to the local port 8080 of the Docker host. On the Docker host, we can open the URL.

localhost:8080

Note: In case of Vagrant with VirtualBox, per default, there is only a NAT-based interface and you need to create port-forwarding for any port you want to reach from outside (also the local machine you are working on is to be considered as outside). In this case, we need to add an entry in the port forwarding list of VirtualBox:
2016-11-30-19_22_22-regel-fur-port-weiterleitung

We have created this entry in part 1 already, but I have seen that the entries were gone again, which seems to be a VirtualBox bug. I have added it again now.

Log in with the admin account we have created in the last session:

2016-12-09-10_24_00-jenkins

Step 3: Code Analysis: Checkstyle

With Gradle, we can invoke the Checkstyle plugin as follows:

Step 3.1: Prepare Gradle for performing Checkstyle

Add to build.gradle:

apply plugin: 'checkstyle'

tasks.withType(Checkstyle) {
 ignoreFailures = true
 reports {
 html.enabled = true
 }
}

We have set ignoreFailures to true, since we do not want the Gradle build to fail for now. We are just interested in the Checkstyle reports for now.

We can download an example Checkstyle configuration file from the Apache Camel repository, for example:

git clone <yourprojectURL>
mkdir -p <yourprojectDir>/config/checkstyle/
curl https://raw.githubusercontent.com/apache/camel/master/buildingtools/src/main/resources/camel-checkstyle.xml > <yourprojectDir>/config/checkstyle/checkstyle.xml

Step 3.2 (optional): Test Checkstyle locally

If you have no Git and/or no Gradle installed, you may want to skip this step and directly proceed to the next step, so Jenkins is performing this task for you.

We can locally invoke CheckStyle as follows:

gradle check

Step 3.3: Configure Jenkins to invoke Checkstyle

Adding Gradle Checkstyle tests to be performed before each build is as simple as performing Step 4.1 and then adding “check” as a goal to the list of Jenkins Build Gradle Tasks:

On Dashboard -> Click on Project name -> Configure -> Build, add “check” before the jar task:

2016-12-28-15_33_58-github-triggered-build-config-jenkins

Click Save.

Now we verify the setting by either checking changed code into the SW repository (now is a good time to commit and push the changes performed in Step 4.1) or by clicking “Build now” -> Click on Build Link in Build History -> Console Output in the Project home:

2016-12-28-15_39_37-github-triggered-build-725-console-jenkins2016-12-28-15_40_08-github-triggered-build-725-console-jenkins

We have received a very long list of CheckStyle Errors, but, as configured, the build does not fail nevertheless.

At the same time, CheckStyle Reports should be available on Jenkins now:

The Links specified in the output are only available on Jenkins, but since Jenkins is running as a Docker container on Vagrant VM residing in

D:\veits\Vagrant\ubuntu-trusty64-docker_openshift-installer\jenkins_home

I need to access the files on

file:///D:/veits/Vagrant/ubuntu-trusty64-docker_openshift-installer/jenkins_home/workspace/GitHub%20Triggered%20Build/build/reports/checkstyle/

2016-12-28-15_48_11-index-von-d__veits_vagrant_ubuntu-trusty64-docker_openshift-installer_jenkins_ho

And on main.html we find:

2016-12-28-15_49_04-main-html

Wow, it seems like I really need to clean the code…

Step 4: Visualize the CheckStyle Warnings and Errors to the Developer

Usually Jenkins is not running as a Docker container on the Developer’s PC or Notebook, so he has no access to the above report files. We need to publish the statistics via the Jenkins portal. For that, we need to install the CheckStyle Jenkins plugin:

Step 4.1 (optional): Install the “Static Analysis Utilities”

Note: I have not tried it out, but I believe that this step is not necessary, since the next step will automatically install all plugins the Checksytle plug-in depends on.

On Jenkins -> Manage Jenkins -> Manage Plugins -> Available

In the Filter field, type “Static Analysis U”

2016-12-28-22_44_53-update-center-jenkins

Check the checkbox of “Static Analysis Utilities” and Install without restart.

2016-12-28-22_47_06-update-center-jenkins

Step 4.2: Install Checkstyle Plugin

On Jenkins -> Manage Jenkins -> Manage Plugins -> Available

In the Filter field, type “Checkstyle ” (with white space at the end; this will limit the number of hits):

2016-12-28-22_56_26-update-center-jenkins

Check the checkbox of “Checkstyle Plug-in” and Install without restart.

2016-12-28-22_58_22-update-center-jenkins

Step 4.3: Configure Checkstyle Reporting in Jenkins

On Dashboard -> <your Project> -> Configure -> Post-build Actions -> Add post-build action, choose

Publish Checkstyle analysis results

Now add the path, where Gradle is placing its result xml files:

**/build/reports/checkstyle/*.xml

2016-12-28-23_10_57-github-triggered-build-config-jenkins

And click Save.

Step 4.4: Manually trigger a new Build

On the Project page, click “Build now”, then click on the build and then “Console output”:

2016-12-28-23_17_16-github-triggered-build-726-console-jenkins

We now can see [CHECKSTYLE] messages after the build, telling us, that the reports were collected. Now, where can we see them?

Step 4.5: Review Checkstyle Statistics

On the Project page, choose Status:

2016-12-28-23_20_31-github-triggered-build-726-jenkins-v2

and click on Checkstyle Warnings on the left, or the warnings link in the center of the page, and we get a graphical representation of the Checkstyle statistics:

2016-12-29-12_27_34-jenkins

When clicking on one of the File Links (MyRouteBuilder.java in this case), we can get an overview of the Warning types for this file:

2016-12-29-12_28_37-jenkins

We choose the category Indentation and get details on the warnings:

2016-12-29-09_03_58-jenkins

and after clicking on one of the links in the Warnings field, we see the java code causing the warning:

2016-12-29-09_05_56-jenkins

Okay, Camel’s Checkstyle configuration does not like my style of grouping each route’s first line with a smaller indent than the rest of the route:

2016-12-29-09_10_26-jenkins

And it does not seem to accept my style of putting the ; in a new line at the end of a route as seen by choosing the Whitespace category and then choosing an occurence:

2016-12-29-12_34_10-jenkins

I either need to change this style, or I need to adapte the checkstyle.xml configuration file to ignore those warnings.

Step 5: Improve Code Style

For the developer, it is very inconvenient to use the Jenkins Checkstyle messages from the console and match them with the code. We need something better than that: the Eclipse Checkstyle plugin.

Step 5.1: Install Eclipse Checkstyle Plugin via local Installation

Since the recommended installation via Marketplace did not work in my case (see Appendix A), I have followed some hints about a local installation found on StackOverflow:

Download Checkstyle from Sourceforge.

2016-12-30-13_54_36-add-repository

2016-12-30-13_55_14-install

In the next window, you are asked to specify some credentials we do not have. However, you can just ignore the window and click Cancel:

2016-12-30-14_01_54-login-required

->Cancel

Then the installation proceeds:

2016-12-30-14_04_17-install

2016-12-30-14_04_26-install

2016-12-30-14_04_33-installing-software

Now I had to klick OK on security warnings twice:

2016-12-29-19_55_50-security-warning

At the end, I had to restart Eclipse:

2016-12-30-19_09_15-software-updates

Now, the Checkstyle plugin is installed on Eclipse.

Step 5.2: Configure Project for Checkstyle Usage

The project in question must be enabled for Checkstyle usage by editing the Project Properties:

2017-01-07-23_14_44

Choosing the Checkstyle style. For now, let us choose the Google Checks in the drop-down list:

2017-01-07-23_18_41-properties-for-simple-restful-file-storage

Then confirm that the project is being re-built:

2017-01-07-23_18_50-rebuild-suggested

Now the code is more yellow than white, with many hints how to improve the code:

2017-01-07-23_28_00-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

However, the hints do not go away, if you correct the code. Do we need to rebuild again? Let us test:

2017-01-07-23_30_36-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

Google style does not like that there is no empty line before the package line (sorry, in German):

2017-01-07-23_29_57-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

So, let us add an empty line and save the file. However, the style warning does not change:

2017-01-07-23_31_53-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

Let us rebuild the project:

2017-01-07-23_33_05

Yes, after the re-build: the warning has disappeared:

2017-01-07-23_43_01-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

Step 5.3: Download and Create Custom Checkstyle Profile in Eclipse

In the Jenkins Checkstyle tests above, we have used following custom Checkstyle configuration file:

$ curl https://raw.githubusercontent.com/apache/camel/master/buildingtools/src/main/resources/camel-checkstyle.xml > <yourprojectDir>/config/checkstyle/checkstyle.xml

I.e. the Checkstyle file is found on <yourprojectDir>/config/checkstyle/checkstyle.xml

Correct:

2017-01-07-23_49_13-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

2017-01-07-23_52_04-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

2017-01-07-23_52_57-preferences

2017-01-07-23_55_39-check-configuration-properties

Step 5.4: Assign Custom Checkstyle Profile to the Project

To assign the new Checkstyle profile to the project, we change the project’s Checkstyle properties by

Project->Properties -> Checkstyle

2017-01-07-23_14_44

-> Choose new Checkstyle profile -> OK

2017-01-08-00_01_13-properties-for-simple-restful-file-storage

On the Rebuild suggested window -> Yes

2017-01-08-00_01_18-rebuild-suggested

This works fine:

2017-01-18-02_29_51-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles-v2

In the code, we can see the Checkstyle warnings. To get more information on the specific Checkstyle warning, the warning text can be retrieved via the mouse over function on the left of the code line, or on the markers tab on the lower pane of Eclipse.

Step 5.5: Improve Code Style

Step 5.5.1: Change Code

In order to test, how the developer can improve the code style, let us replace some of the tabs by four spaces here:

2017-01-18-02_48_39-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

Save the file now.

Step 5.5.2: Update Maven

Unfortunately, the Checkstyle warnings update process is a little cumbersome for custom Checkstyle profiles, it seems: we need to

  1. save the changed file,
  2. update Maven and
  3. rebuild the project.

Let us update Maven first:

right-click the project folder in the left pane -> Maven -> Update Project -> OK

2017-01-18-02_54_032017-01-18-02_58_21-update-maven-project

Then all Checkstyle markers are gone (although I have not changed all occurrences of a tab):

2017-01-18-02_59_32-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

Step 5.5.3 Rebuild the Project

To get the Checkstyle warnings back, we need to rebuild the project:

Project -> Build Project

2017-01-18-03_02_56

Now we can see that some of the Checkstyle warnings are gone:

2017-01-18-03_04_05-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

Next time, you check in the code to the Gir repository, you will see that the number of Checkstyle warnings we get from Jenkins via Gradle will decrease…

Step 6: Verify Jenkins Results

Since we have improved the source code, we expect the Jenkins Checkstyle warnings to decrease. We can verify this by doing the following:

-> save, commit and push the improved code -> log into Jenkins -> check out the build process that is triggered by the code push (or we can manually trigger the build process by clicking project -> Build now)

On the dashboard, we will see, that the Checkstyle statistics have (very) slightly improved:

2017-01-18-04_37_06-github-triggered-build-jenkins-v2

On the upper right edge of the figure, the number of warnings is slightly lower. The code quality is far from being perfect, but we now have all tools and plugins needed to improve the situation.

After changing all tabs by 4 spaces each, the number of Checkstyle violations goes down by ~50%. That is a good start.

2017-01-19-22_51_58-github-triggered-build-jenkins-v2

Perfect, we have learned how to use the Checkstyle plugin for Eclipse in order to produce better code. And the Jenkins Checkstyle plugin allows us to admire the progress we make.

😉

thumps_up_3

Appendix A: Problems with installing Checkstyle Eclipse Plugin via Marketplace

Note: this way of installation is recommended officially, but has failed in my case. If you hit the same problem, try the local installation as described in step 5.1 above.

To improve the style, it would be much too cumbersome to click through all 360 style warnings, edit the Java code, build the Code and check again. It is much better to give the programmer immediate feedback of the warning within the IDE. I am using Eclipse, so I need to install the Checkstyle Eclipse plugin as follows:

Choose Eclipse -> Help -> Eclipse Marketplace

2016-12-29-09_16_30-java-ee-simple-restful-file-storage_src_main_java_de_oveits_simplerestfulfiles

Search for “Checkstyle” and click install:

2016-12-29-09_19_00-eclipse-marketplace

And then “confirm”:

2016-12-29-09_20_01-eclipse-marketplace

What is that?

2016-12-29-09_21_18-proceed-with-installation_

I install it anyway. At this point, it is hanging quite a while:

2016-12-29-09_24_24-eclipse-marketplace

so, let me get a morning coffee…

After approximately two minutes, I can see it to proceed to 4 / 15. Good sign.

After the coffee, I still see 4 / 15. Not a good sign:

2016-12-29-09_44_41-eclipse-marketplace

Meanwhile I am researching the steps needed for performance testing…

After 2 hours or so: 6/15

This will take all day!

2016-12-29-11_31_35-eclipse-marketplace

Some hours later, I checked again, and I have seen the following:

2016-12-29-19_49_10-eclipse-marketplace

I have confirmed, confirmed the license:

2016-12-29-19_50_51-eclipse-marketplace

And have pressed Finish.

Then software gets installed:

2016-12-29-19_52_04-installing-software

I hope I will not break my good old Eclipse installation (it is installed locally, not in a virtual machine or container and it has ever worked better than any new version I have tested…).

After a two or three minutes:

2016-12-29-19_55_57-security-warning

I have confirmed with “OK”…

Then I had been asked to restart Eclipse and I have confirmed.

Problem: however, Checkstyle is still not installed:

Help -> Eclipse Marketplace

2016-12-30-13_20_09-eclipse-marketplace

Let us try again by clicking “Install”:

2016-12-30-13_24_45-eclipse-marketplace

2016-12-30-13_24_59-proceed-with-installation_

This does not work

Workaround

Instead of installing Checkstyle via the Eclipse marketplace, better install the Eclipse Checkstyle Plugin via download (see Step 5.1)

Summary

In this blog post we have performed following tasks:

  1. Started Jenkins in a Docker container
  2. Installed the Checkstyle Gradle Plugin to create Checkstyle plugins as XML files
  3. Installed the Checkstyle Jenkins Plugin to summarize the XML files into graphical historic reports
  4. Installed the Checkstyle Eclipse Plugin to be able to improve the code
  5. Installed custom Checkstyle policies
  6. Visualized the Code Improvement
  7. were happy

All steps apart from the installation of the Eclipse Checkstyle plugin were quite straightforward. For the Eclipse Checkstyle installation, we had to revert back to a local download and installation method described in step 5.1: the installation via Eclipse marketplace had failed. At the end, we could reduce the number of Checkstyle warnings by 50% without much effort.

Further Reading

2

Jenkins Part 4.1: Functional Java Tests via JUnit


2016-11-30-18_19_38

You also think that functional tests are one of the most important ingredients for delivering high quality software? You share my opinion that we should help the developer automating this task in order to get comparable results and to receive meaningful trend reports?

I will cover functional tests here. Instructions on how to perform code quality tests and performance tests are in draft status and will be covered in the next two blog posts.

Any questions and/or comments are highly welcome.

Introduction

As a developer you try hard to deliver high quality software.

You hate searching for this nasty bug that had been introduced unnoticed days ago. Or was it weeks ago? By whom? In which code?

Manual functional and performance testing after each commited code change quickly becomes a NO-GO as the number of features is rising constantly. In this blog post, we will show, how Jenkins can help you with both: delivering high quality software and minimizing the time needed to find the cause of a bug.

How about …

  1. creating automated tests for each functionality and performance at different levels (end to end, and unit tests)
  2. running the automated tests after each code change
  3. keeping track of the test results

… in order to avoid any bad surprises late in the game?

Okay: for 1., the developer needs to create automated functional and perfomance tests; I guess, there is no way around this. Better do this even before writing the actual code. For 2. and 3., however, automation tools like Jenkins step in and can be of great help. The developers checks in the code and Jenkins can do the rest for you.

In the current  blog post, we will show how to integrate automated JUnit functional tests into a Jenkins build pipeline. We will see that JUnit tests can be invoked easily via Gradle (Okay, Maven is more popular than Gradle, I guess, but I like Gradle because of some advantages I have discussed here; However, just give me a hint in a comment to this blog and I will prioritize the creation of a Maven version of this blog post). The Jenkins JUnit plug-in will be used to

  1. display reports on single build runs as well as
  2. display trend analysis graphs like the following one I have borrowed from here:
2016-12-30-18_41_45-jenkins-junit-project-home-jpg-826x707
Source: http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/

In this and the next two blog posts, we plan to cover following quality gate measures:

  • Part 4.1: Functional Tests (this blog post): we will use Java JUnit tests performed before building the executable JAR. Jenkins will report the test trend
  • Part 4.2: Code Quality Tests (coming soon): we will use the Checkstyle Gradle plugin for reporting to which degree the code adheres to the Apache Foundations formal rules
  • Part 4.3: Performance Tests (planned): we will use JMeter for testing and reporting the performance trend performed after the Java build using external performance testers like JMeter

Older blogs of this series:

This blog post series about Jenkins build pipelines is divided into following parts:

    • Part 1: Installation and Configuration of Jenkins, loading Plugins
    • Part 2: Creating our first Jenkins job: GitHub download and Software build
    • Part 3: Periodic and automatically triggered Builds

What is Jenkins?

Jenkins is the leading open source automation server mostly used in continuous integration and continuous deployment pipelines. Jenkins provides hundreds of plugins to support building, deploying and automating any project.

2016-12-30-21_04_46

A typical workflow is visualized above: a developer checks in the code changes into the repository. Jenkins will detect the change, build (compile) the software, test it and prepare to deploy it on a system. Depending on the configuration, the deployment is triggered by a human person, or automatically performed by Jenkins. After each step, the developer is informed depending on the priorites defined.

For more information, see the introduction found in part 1 of this blog series.

Automated Functional Testing based on JUnit

In this blog post, we will show how we need to configure Gradle and Jenkins for automated JUnit testing and reporting. In order to build a quality gate, we will reverse the original order and perform the JUnit tests before we build the executable JAR file (we do not want to create JAR files that are not functional):

2016-12-28-12_50_23

Tools used

      • Vagrant 1.8.6
      • Virtualbox 5.0.20
      • Docker 1.12.1
      • Jenkins 2.19.3
        • JUnit Plug-in 1.19

Prerequisites:

      • Free DRAM for the a Docker Host VM >~ 4 GB
      • Docker Host is available, Jenkins is installed and a build process is configured. For that, perform all steps in part 1 to part 3 of this blog series
      • Tested with 2 vCPU (1 vCPU might work as well)

Step 1: Start Jenkins in interactive Terminal Mode

Make sure that port 8080 is unused on the Docker host. If you were following all steps in part 1 of the series, you might need to stop cadvisor:

(dockerhost)$ sudo docker stop cadvisor

I assume that jenkins_home is already created, all popular plugins are installed and an Admin user has been created as shown in part 1 of the blog series. We start the Jenkins container with the jenkins_home Docker host volume mapped to /var/jenkins_home:

(dockerhost)$ cd <path_to_jenkins_home> # in my case: cd /vagrant/jenkins_home/
(dockerhost:jenkins_home)$ sudo docker run -it --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins
Running from: /usr/share/jenkins/jenkins.war
...
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Step 2: Open Jenkins in a Browser

Now we want to connect to the Jenkins portal. For that, open a browser and open the URL

<your_jenkins_host>:8080

In our case, Jenkins is running in a container and we have mapped the container-port 8080 to the local port 8080 of the Docker host. On the Docker host, we can open the URL.

localhost:8080

Note: In case of Vagrant with VirtualBox, per default, there is only a NAT-based interface and you need to create port-forwarding for any port you want to reach from outside (also the local machine you are working on is to be considered as outside). In this case, we need to add an entry in the port forwarding list of VirtualBox:
2016-11-30-19_22_22-regel-fur-port-weiterleitung

We have created this entry in part 1 already, but I have seen that the entries were gone again, which seems to be a VirtualBox bug. I have added it again now.

Log in with the admin account we have created in the last session:

2016-12-09-10_24_00-jenkins

Step 3: Pre-Build JUnit Tests invoked by Gradle

In this step, we will invoke Gradle Tests before building the JAR. For that, we should verify locally that the Gradle tests are successful and then define a test Gradle task in the build process.

Step 3.1 (optional): Verify that Gradle Tests are successful

You can skip this test and directly let Jenkins do this for you. This may come handy, if you have not installed Git and/or Gradle locally.

Prerequisites

  • Your Java project has successful JUnit tests defined
  • Git is installed
  • The Project is cloned to a local directory
  • Gradle is installed

In order to test, whether the JUnit tests are successful, we can test those on a system with the project cloned (git, java and gradle must be installed):

(basesystem)$ gradle test
Starting a Gradle Daemon (subsequent builds will be faster)
Parallel execution is an incubating feature.
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava
warning: [options] bootstrap class path not set in conjunction with -source 1.6
1 warning
:processTestResources
:testClasses
:test

BUILD SUCCESSFUL

Total time: 29.9 secs

With that, we have verified that the command “gradle test”succeeds.

Note that the JUnit test must be designed in a way that they are independent of whether or not the JAR file is run in parallel. No simple way of running the executable JAR file in parallel to the execution of the JUnit tests seems to exist. In my case, I had to alter the JUnit tests to fulfill this prerequisite.

Step 3.2: Add Gradle test Task to Jenkins

As long as JUnit tests are defined in src/test of the project, adding Gradle tests to Jenkiny is as simple as adding “test” as a task to the list of Jenkins Build Gradle Tasks as follows:

On Dashboard -> Click on Project name -> Configure -> Build, add “task ” before the jar task:

2016-12-28-08_50_11-github-triggered-build-config-jenkins

Click Save.

If you have made local code changes on the project, now is the best time to commit and push them to the Git repository. If you have followed the steps in part 3, then this will automatically trigger a build process, so you do not need to click on “Build now” in that case. Otherwise, click on “Build now” on the Jenkins project page (e.g. Dashboard -> click on project name -> “Build now”).

Now we observe the result by clicking on the build process, then -> “Console Output”:

2016-12-28-09_45_34-github-triggered-build-724-console-jenkins

Don’t be confused by the blinking red ball on the upper left of the Console Output page: we see a BUILD SUCCESSFUL message and if we re-enter the same page, the ball is turned to static blue, indicating a successful build.

Step 4: Add JUnit Test Result Reporting to Jenkins

Now we will show how to add the JUnit test reports to the Jenkins build process.

Step 4.1: Install Jenkins JUnit Plugin

For Jenkins JUnit reporting, we need to install the JUnit Plug-in. For that, goto -> Jenkins Dashboard -> Manage Jenkins -> Manage Plugins -> Available -> Enter “JUnit Plugin” to the Find field -> Install

Note: If you do not find the plugin on the Available tab, search for it in the “Installed” tab.

You can install the plugin without reloading Jenkins.

Step 4.2: Configure Jenkins to collect and display the JUnit Test Results

In this step, we will configure Jenkins, so it will display the test results for individual builds as well as trend reporting. For that, navigate to:

Jenkins -> (choose Project) -> Configure -> Post-build Actions -> Publish JUnit test results report

2016-12-30-14_15_25-github-triggered-build-config-jenkins

Add

**/build/test-results/test/TEST-*.xml

to the “Test report XMLs” field, since this is the path, where Gradle is placing its JUnit test result reports (I have found the info here).

2016-12-30-14_18_51-github-triggered-build-config-jenkins

Now click Save.

Step 4.3: Verify JUnit individual Test Reporting

To test the Jenkins JUnit reporting feature, we trigger a clean build by adding “clean” to the Gradle tasks on Project -> Configure -> Build:

2016-12-30-17_43_59-github-triggered-build-config-jenkins

and clicking Save.

Then trigger a new build by clicking on Project -> Build now.

Then click on the Build Process, and then on Console output:

2016-12-30-17_48_38-github-triggered-build-731-console-jenkins

…scrolling down…

2016-12-30-17_50_01-github-triggered-build-731-console-jenkins

Do not be confused that the build process never seems to finish. Just click the Back to Project link:

Back to Project

On the Status page, we see that there were no failed tests:

2016-12-30-17_55_34-github-triggered-build-731-jenkins-v2

When we click on the Tests Result link on the left (or on the lower middle part on the Status page), we will see more details:

2016-12-30-17_58_25-github-triggered-build-731-test-results-jenkins-v2

We can see that we have had four tests (Create/Read/Update/Delete a file) and 100% of them were successful.

Step 4.3: Verify JUnit Test Trend Reporting

On the project’s Status page, a Test Trend graph is automatically added, as soon as there are two or more tests available. For that, click on “Build Now” on the left for a second time and click ENABLE AUTO REFRESH on the upper right. After the second build is complete, the (hopefully) blue Test Result Trend graph is showing up on the project status page:

2016-12-30-18_12_21-github-triggered-build-jenkins

The new blue graph shows that we had 4 successful tests in the last two builds.

Note: disregard the red Checkstyle Trend graph for now. This is something we will cover in the next blog post.

Step 5: Verify failed Test Reporting

Per default, Gradle build will fail, if one of the JUnit tests has failed, so it is building a strict quality gate. Will the test result be collected and reported nevertheless?

Let us test this now by breaking one of the JUnit tests by purpose. We have added an assert message that is expected to fail in one of the tests:

2016-12-30-19_27_46-java-ee-simple-restful-file-storage_src_test_java_de_oveits_simplerestfulfiles-v2

Now we commit and push the change to the SW repository:

$ git clone <Repository-URL>
$ cd <Repository Dir>
<perform the code changes here...>
$ git diff src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
diff --git a/src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java b/src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
index 684d30f..10200d5 100644
--- a/src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
+++ b/src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
@@ -115,6 +115,9 @@ public class SimpleRestfulFileStorageTests extends CamelSpringTestSupport {
 // mock expectations need to be specified before sending the message:
 mock.expectedBodiesReceived("File ttt created: href=http://localhost:2005/files/ttt");
 mock.expectedMessageCount(1);
+ ^M
+ // In order to break this test for Jenkins test reporting, we temporarily add a requirement that will fail:^M
+ mock.expectedMessageCount(2);^M

 template.sendBodyAndHeaders("direct:recipientList", body, headers);

$ git add src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
$ git commit -m "Breaking a JUnit test by purpose for Jenkins reporting tests"
[jenkinstest 33655b9] Breaking a JUnit test by purpose for Jenkins reporting tests
 1 file changed, 4 insertions(+), 1 deletion(-)

olive@LAPTOP-P5GHOHB7 /d/veits/eclipseWorkspaceRecent/simple-restful-file-storage (jenkinstest)
$ git push
Counting objects: 9, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (9/9), 744 bytes | 0 bytes/s, done.
Total 9 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To https://github.com/oveits/simple-restful-file-storage.git
 edb49f7..33655b9 jenkinstest -> jenkinstest

This will automatically trigger a new build (if you have followed part 3 of this series; otherwise just press “Build Now” on Jenkin’s project page).

We can see on the dashboard, that the build has failed:

2016-12-30-19_36_12-dashboard-jenkins

This was expected. Now let us click on the Project name, and we will see, what happened:

2016-12-30-19_37_40-github-triggered-build-jenkins

Perfect, that is exactly, what I wanted to achieve: On the Test Result Trend, we can see that we have performed 4 tests, one of which has failed.

Let us fix the failed test by commenting out (or removing) the wrong code again:

2016-12-30-19_40_01-java-ee-simple-restful-file-storage_src_test_java_de_oveits_simplerestfulfiles-v2

After

$ git add <file>
$ git commit -m "Fixed JUnit test again to test Jenkins JUnit trend report"
$ git push

The next build should be successful again and we can see in the trends graph that the failed test is fixed again:

2016-12-30-19_47_48-github-triggered-build-jenkins

thumps_up_3

Summary

In this blog post, we have shown

  1. How to add Java functional tests to the Jenkins build pipeline based on Gradle JUnit Plugin
  2. How to install the JUnit plug-in to Jenkins for report collection
  3. How to display JUnit test results for individual builds on the Jenkins portal
  4. How to display JUnit trend analysis on the Jenkins portal

The only challenge I have encountered is, that I had to re-write my JUnit tests in a way that they were successful when run stand-alone. Before they were successful only, if the executable JAR file was started manually before running the JUnit tests. This was resolved in a way specific to the framework used (Apache Camel in this case).

Coming Soon: Code Analysis Trend Analysis via Jenkins Checkstyle plugin

Further Reading

7

Jenkins Part 3.1: periodic vs triggered Builds


2016-11-30-18_19_38

Today, we will make sure that Jenkins will detect a code change in the software repository without manual intervention. We will show two methods to do so:

  1. Periodic Builds via Schedulers: Jenkins periodically asks the software repository for any code changes
  2. Triggered Builds via Webhooks: Jenkins is triggered by the software repository to perform the build task

We will see that the triggering build processes is more challenging to set up, but has quite some advantages in terms of economics and handling, once it is set up properly. See also the Summary at the end of this post.

This blog post series is divided into following parts:

    • Part 1: Installation and Configuration of Jenkins, loading Plugins
    • Part 2: Creating our first Jenkins job: GitHub download and Software build
    • Part 3 (this blog): Periodic and automatically triggered Builds
    • Part 4 (planned): running automated tests

What is Jenkins?

Jenkins is the leading open source automation server mostly used in continuous integration and continuous deployment pipelines. Jenkins provides hundreds of plugins to support building, deploying and automating any project.

 

Jenkins build, test and deployment pipeline
Jenkins build, test and deployment pipeline

A typical workflow is visualized above: a developer checks in the code changes into the repository. Jenkins will detect the change, build (compile) the software, test it and prepare to deploy it on a system. Depending on the configuration, the deployment is triggered by a human person, or automatically performed by Jenkins.

For more information, see the introduction found in part 1 of this blog series.

Automatic Jenkins Workflow: Periodic Polling

In this chapter, we will show how we need to configure Jenkins for automatic polling of the Software repository and start the build process, if code changes are detected.

2016-12-09-10_12_31

Tools used

      • Vagrant 1.8.6
      • Virtualbox 5.0.20
      • Docker 1.12.1
      • Jenkins 2.19.3

Prerequisites:

      • Free DRAM for the a Docker Host VM >~ 4 GB
      • Docker Host is available, Jenkins is installed and a build process is configured. For that, perform all steps in part 1 and part 2 of this blog series
      • Tested with 2 vCPU (1 vCPU might work as well)

Step 1: Start Jenkins in interactive Terminal Mode

Make sure that port 8080 is unused on the Docker host. If you were following all steps in part 1 of the series, you might need to stop cadvisor:

(dockerhost)$ sudo docker stop cadvisor

I assume that jenkins_home is already created, all popular plugins are installed and an Admin user has been created as shown in part 1 of the blog series. We start the Jenkins container with the jenkins_home Docker host volume mapped to /var/jenkins_home:

(dockerhost)$ cd <path_to_jenkins_home> # in my case: cd /vagrant/jenkins_home/
(dockerhost:jenkins_home)$ sudo docker run -it --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins
Running from: /usr/share/jenkins/jenkins.war
...
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Step 2: Open Jenkins in a Browser

Now we want to connect to the Jenkins portal. For that, open a browser and open the URL

<your_jenkins_host>:8080

In our case, Jenkins is running in a container and we have mapped the container-port 8080 to the local port 8080 of the Docker host. On the Docker host, we can open the URL.

localhost:8080

Note: In case of Vagrant with VirtualBox, per default, there is only a NAT-based interface and you need to create port-forwarding for any port you want to reach from outside (also the local machine you are working on is to be considered as outside). In this case, we need to add an entry in the port forwarding list of VirtualBox:
2016-11-30-19_22_22-regel-fur-port-weiterleitung

We have created this entry in part 1 already, but I have seen that the entries were gone again, which seems to be a VirtualBox bug. I have added it again now.

Log in with the admin account we have created in the last session:

2016-12-09-10_24_00-jenkins

Step 3: Configure Project for periodic Polling of SW Repository

Step 3.1: Goto Build Trigger Configuration

On the Jenkins Dashboard, find the hidden triangle right of the project name,

2016-12-09-18_17_35-dashboard-jenkins

In the drop-down list, choose “Configure”

2016-12-09-18_18_06-dashboard-jenkins

(also possible: on the Dashboard, click on the project name and then “Configure”).

Step 3.2: Configure a Schedule

We scroll down to “Build Triggers” and check “Build periodically” and specify that it will be done every 10 minutes (H/10 * * * *). I do not recommend to use lower values than that since I have seen that even my monster notebook with i7-6700HQ and 64GB RAM is quite a bit stressed by the build those many build processes.

2016-12-22-23_54_06-github-triggered-build-config-jenkins

Note that this is a very short polling period for our test purposes only; we do not want to wait very long after a code change is detected.

Note also: you can click the Blue Question Markright of the Schedule text box to get help with the scheduler syntax.

Step 3.2: Save

Click Save

Step 4: Change the content of the Software Repository

Now we expect that a change of the SW repository is detected latest 2 minutes after new code is checked in. Let us do so now: In this case, I have changed the content of README.md and commited the change:

(local repository)$ git add README.md
(local repository)$ git commit -m "changed README"
(local repository)$ git push

Within 2 minutes, I see a new job #24 running on the lower left:

2016-12-09-18_35_13-dashboard-jenkins

It seems that the page needs to be reloaded by refreshing the browser, so the dashboard displays the #24 build process as “Last Success”:

The build process was very quick, since we have not changed any relevant source code. The console log can be reached via the Jenkins -> Project Link -> Build History -> click on build number -> Console:

2016-12-11-21_55_22-github-triggered-build-687-console-jenkins

As you can see, after some hours, the git repository is downloaded even if there was no code change at all. However, Gradle will detect that the JAR file is up-to-date and it will not re-build the JAR file, unless there is a code change.

The disadvantage of a scheduled build process with high frequency is that the number of builds in the build history is increasing quickly:

2016-12-11-22_02_24-github-triggered-build-jenkins

Note: The build history is spammed by many successful builds with no code change, and it is not easy to find the interesting build among all those many unnecessary builds. Let us try to improve the situation by replacing periodic, scheduled builds by triggered builds:

Step 5: Triggered Builds

In Step 4, we have seen that periodic builds should not be performed in a very short timeframe, because:

  1. the Jenkins server is stressed quite a bit by configuring a too low build frequency
  2. the build history is polluted by information of many irrelevant build processes with no changed code.

Therefore, it is much better to create a triggered build. The target is to trigger a build process every time the developer is checking in new code to the software repository:

2016-12-21-15_12_25

In this way, a periodic build is not necessary, or can be done much less frequently.

What do we need to do?

  1. Make sure that the Jenkins server is reachable from the SW repository
  2. Configure the SW repository with a web hook for informing Jenkins upon each code change
  3. Configure Jenkins for triggered build

Let us start:

Step 5.1 Configure Jenkins for triggered Build

On the Jenkins Dashboard, click on the project:

2016-12-22-18_56_44-dashboard-jenkins

and then “Configure” on the left pane:

2016-12-22-18_58_28-github-triggered-build-jenkins

Scroll down to Build Triggers and check the “Trigger build remotels (e..g. , from scripts)” checkbox and choose an individual secret token (do not use the one you see here):

2016-12-22-19_03_16-github-triggered-build-config-jenkins

You will be provided with the build trigger URL, which is in my case:

JENKINS_URL/job/GitHub%20Triggered%20Build/build?token=TOKEN_NAME

And the JENKINS_URL is the URL needed to be contacted by the Git Repository. Save the URL above for later use.

Now click Save.

Step 5.2 Test Trigger URL locally

Now we can test the trigger URL locally on the Docker Host as follows (as found on this StackOverflow Q&A):

We need to retrieve a so-called Jenkins-Crumb:

(dockerhost)$ CRUMB=$(curl -s 'http://admin:your_admin_password@localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
(dockerhost)$ echo $CRUMB
Jenkins-Crumb:CCCCCCCCCCCCCCCCCCCCCCCCCC

Please make a note of the returned Jenkins-Crumb, since we will need this value in the next step.

Then we can use the Jenkins-Crumb as header in the build trigger request:

(dockerhost)$ curl -H $CRUMB 'http://admin:your_admin_password@localhost:8080/job/GitHub%20Triggered%20Build/build?token=hdsghewriohwziowrhwsn'

This should trigger a new build on Jenkins:

2016-12-22-21_56_44-dashboard-jenkins-v2

By clicking on the build and then the “Console Output”, we see a successful build with no changed data:

2016-12-22-22_01_36-github-triggered-build-712-console-jenkins

Step 5.3 Make sure that the Jenkins Server is reachable from the SW repository

We are running the Jenkins server as a Docker container within a Vagrant VM as host. In step 2 we have made sure that the Docker container is reachable from the local network by exposing the Docker ports and by configuring port forwarding in VirtualBox. However, the Docker container is not yet reachable from the Git Repository, since the Router will block all requests, as long as no pot forwarding is configured on the router:

2016-12-21-16_02_27

Let us fix that now:

In my case, the (sorry, German) input mask of the router looks like follows:

2016-12-22-19_30_22-fritzbox-7490

I am mapping outside port 8080 to the internal machine running the Docker Host VM.

Now, the routing should work. We will test this in the next step.

2016-12-22-23_59_49

Step 5.4: Add Webhook to Git SW Repository

Now we need to add a Webhook to the Git repository. In my case, the repository is located at https://github.com/oveits/simple-restful-file-storage. On that page, goto

Settings -> Webhooks-> Add webhook -> Confirm password

Then copy&paste the URL of Step 5.1 into the Payload URL with following changes:

  • Change JENKINS_URL by the IP address or DNS name your router is reachable from the Internet.
  • Choose a port that you intend to open for this service (e.g. 8080) in the next step.
  • Add admin:your_admin_password@ before the JENKINS_URL; use your own username and password here
  • append &Jenkins-Crumb=CCCCCCCCCCCCCCCC to the URL with the value of the Jenkins-Crumb we have retrieved in the previous step

Example with the items to change in red:

http://admin:your_admin_password@your_public_ip_or_name:8080/job/GitHub%20Triggered%20Build/build?token=TTTTTTTTTTTTTTTTT&Jenkins-Crumb=CCCCCCCCCCCCCCCCCCCC

2016-12-22-22_08_39-webhook-http___veits-no-ip-biz_8080_job_github%20triggered%20build_build

 

For the other fields, keep the defaults and klick Add Webhook.

If everything works fine, we already should see a successful delivery of the trigger on the lower end of the Github page:

2016-12-22-22_17_57-webhook-http___veits-no-ip-biz_8080_job_github%20triggered%20build_build-v2

If it was not successful, you can see more details by clicking on the request:

2016-12-22-22_20_52-webhook-http___veits-no-ip-biz_8080_job_github%20triggered%20build_build

Step 6: Test triggered Build upon Code Push

This is the final step of this tutorial: we now will test that a build is triggered each time a user pushes new code to the repository.

Step 6.1: Install Git locally

If Git is not installed locally, so do it now.

Step 6.2: Download the Project Repository

We now clone the project by issuing the command

$ git clone https://github.com/oveits/simple-restful-file-storage

Step 6.3: Change Code

You can perform a minor change the content of the README.md in order to test the triggered build.

Step 6.4: Push Code to the Repository

With the commands

$ git commit -am "Minor change of README.md to trigger a Jenkins build"
$ git push

We push the changed code to the SW repository.

If everything works correctly, we will immediately see, that Git has triggered Jenkins to perform a build by reloading the Jenkins Dashboard (32 sec ago, in this screenshot):

2016-12-22-22_50_14-dashboard-jenkins-v2

We can check the build by clicking on the Last Success build and then “Console Output”:

2016-12-22-22_53_08-github-triggered-build-713-console-jenkins

Gradle was clever enough to detect that no relevant code had been changed, so everything is still up to date.

With this procedure we have made sure that the Software repository will trigger a new build process on each and every code change. Moreover, the Jenkins server is not polluted with unnecessary builds anymore, since we have switched off periodic builds.
thumps_up_3

Summary

In this blog post we have performed following tasks:

  1. Started Jenkins in a Docker container
  2. Configured and tested periodic builds
  3. Configured and tested triggered builds
  4. Made sure that the Git Software repository is triggering such a build at every code change

As in the other parts of this series, we have run Jenkins interactively in a Docker container. See below a discussion of the advantages of periodic and triggered builds:

Periodic Builds vs Triggered Builds

When we compare periodic builds with triggered builds, we see following advantages/disadvantages:

Complexity of Setup: periodic builds are much easier to set up. They only need to be configured on Jenkins. Triggered builds requires setup steps on the Jenkins Server, the Software Repository and intermediate Firewalls, if the Jenkins Server is located in a private network.

Economics: Triggered builds are more economic in terms of Jenkins Server load. The build processes run only, when needed.

Handling: Triggered builds have important handling advantages compared to triggered builds: firstly, each and every code change can be tested helping the programmer to get near immediate feedback for every code change. Secondly, the build log is not polluted by hundreds of irrelevant builds.

In my opinion, a clear winner is: triggered builds. Those may be combined with periodic clean builds at certain milestones.

2016-12-22-23_39_41

 

References

 

5

Jenkins Part 2: automated Code Download and Build (Gradle + Maven)


2016-11-30-18_19_38

NEW (2017-01-02): you now can immediately start with part 2 (this post) without going through the steps of part 1. A corresponding pre-installed Docker image is provided.

NEW (2017-01-05): I have added the Maven build path with a fat executable JAR file.

In this blog post, we will perform our first automated job within Jenkins, the most popular open source tool for Continuous Integration and Continuous Deployment. Like in part 1 of this blog series, we will start Jenkins in a Docker container and define and run a first task:

  • download source code from GitHub
  • Create a lean executable JAR file via Gradle
  • Create a fat  executable JAR file via Maven

At the end of this session, we will have learned how to download GitHub code and build a Java program (executable JAR file) on a push of a button.

This blog post series is divided into following parts:

    • Part 1: Installation and Configuration of Jenkins, loading Plugins
    • Part 2 (this blog): Creating our first Jenkins job: GitHub download and Software build
    • Part 3: Periodic and automatically triggered Builds
    • Part 4 (planned): running automated tests

What is Jenkins?

Jenkins is the leading open source automation server mostly used in continuous integration and continuous deployment pipelines. Jenkins provides hundreds of plugins to support building, deploying and automating any project.

 

Jenkins build, test and deployment pipeline
Jenkins build, test and deployment pipeline

A typical workflow is visualized above: a developer checks in the code changes into the repository. Jenkins will detect the change, build (compile) the software, test it and prepare to deploy it on a system. Depending on the configuration, the deployment is triggered by a human person, or automatically performed by Jenkins.

For more information, see the introduction found in part 1 of this blog series.

Our first Jenkins Job

In this hello world, we will perform the first part of the typical build pipeline shown in the previous chapter, applied to a java hello world program:

  1. Upon a push of a button, download code from the GIT repository
  2. Build an executable jar from the java code

2016-12-09-15_29_08

Tools used

      • Vagrant 1.8.6
      • Virtualbox 5.0.20
      • Docker 1.12.1
      • Jenkins 2.32.1

Prerequisites:

      • Free DRAM overall >~ 4 GB
      • A Docker Host is available. Perform Step 1 in Part 1 of this blog series, if you are in need of a Docker host.

Step 1: Start Jenkins in interactive Terminal Mode

Step 1.1: Make sure the TCP Port is unused

Make sure that port 8080 is unused on the Docker host. If you were following all steps in part 1 of the series, you might need to stop cadvisor:

(dockerhost)$ sudo docker stop cadvisor

Alternatively, you can alter the ports option below from -p8080:8080 to -p9090:8080, as an example.

Step 1.2: Alternative (A): If you have followed all steps in part 1 of this blog series, start Jenkins from the official Jenkins image

I assume that you have followed all steps in part 1 of the blog series. In this case, you have created a jenkins_home directory on you Docker host. All popular plugins are installed and an Admin user has been created. If the Jenkins container is not started already, we start it with the jenkins_home Docker host volume mapped to /var/jenkins_home (as we have done in part 1 of this blog series):

(dockerhost)$ cd <path_to_jenkins_home>
(dockerhost:jenkins_home)$ sudo docker run -it --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins
Running from: /usr/share/jenkins/jenkins.war
...
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Step 1.2: Alternative (B): You prefer to start from a pre-installed Jenkins Image

If you have not followed the steps in part 1, or if you prefer to start from a cleanly installed image, you also can start Jenkins from a cleanly installed Docker image like follows:

(dockerhost)$ sudo docker run -it --name jenkins -p8080:8080 -p50000:50000 oveits/jenkins_tutorial:part2_step1
Running from: /usr/share/jenkins/jenkins.war
...
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Note that in alternative (B), all data is kept within /var/jenkins_home_local directory within the created container. For the case you want to save your work thereafter, I have not used the docker run remove option --rm. This will give you the chance to stop and start the same container later. In addition, you can create your own image from the stopped container in order to retain the work.

Step 2: Open Jenkins in a Browser

Now we want to connect to the Jenkins portal. For that, open a browser and open the URL

<your_jenkins_host>:8080

In our case, Jenkins is running in a container and we have mapped the container-port 8080 to the local port 8080 of the Docker host. On the Docker host, we can open the URL.

localhost:8080

Note: In case of Vagrant with VirtualBox, per default, there is only a NAT-based interface and you need to create port-forwarding for any port you want to reach from outside (also the local machine you are working on is to be considered as outside). In this case, we need to add an entry in the port forwarding list of VirtualBox:
2016-11-30-19_22_22-regel-fur-port-weiterleitung

We have created this entry in part 1 already, but I have seen that the entries were gone again, which seems to be a VirtualBox bug. I have added it again now.

Log in with the admin account we have created in the last session:

2016-12-09-10_24_00-jenkins

Step 3: Alternative (a): Prepare Gradle Usage

If you later prefer to use Gradle instead of (or in addition to) Maven, you need to prepare its first usage. For Maven preparation, see Step 3, Alternative (b) below.

On this wiki page about the Gradle plugin we find that we need to configure Gradle first:

Goto Jenkins -> Manage Jenkins -> Global Tool Configuration (available for Jenkins >2.0)

2016-12-09-11_34_55-manage-jenkins-jenkins

2016-12-09-11_35_26-global-tool-configuration-jenkins

Scroll down to Gradle -> Add Gradle

2017-01-02-14_27_26-global-tool-configuration-jenkins

-> choose Version (Gradle 3.2.1 in my case)

-> Add a name (“Gradle 3.2.1” in my case)

-> Save

Since we have checked “Install automatically” above, I expect that it will be installed automatically on first usage.

Step 3: Alternative (b): Prepare Maven Usage

If you later prefer to use Maven instead of (or in addition to) Gradle, you need to prepare its first usage. For Gradle preparation, see Step 3, Alternative (a) above.

Goto Jenkins -> Manage Jenkins -> Global Tool Configuration (available for Jenkins >2.0)

2016-12-09-11_34_55-manage-jenkins-jenkins

2016-12-09-11_35_26-global-tool-configuration-jenkins

Scroll down to Maven -> Add Maven

2017-01-02-14_32_46-global-tool-configuration-jenkins

-> choose Version (3.3.9 in my case)

-> Add a name (“Maven 3.3.9” in my case)

-> Save

Since we have checked “Install automatically” above, I expect that it will be installed automatically on first usage.

Step 4: Create a Job (Freestyle Project)

Step 4.1 Enter Name and Project Type

Either click on “create new jobs” or on New Item.

Now enter an Item name and click on Freestyle Project and OK:

2016-12-09-10_55_56-new-item-jenkins

Step 4.2: Specify GitHub Project

Check “GitHub project” and add the HTTPS GitHub URL. I have used a small Apache Camel project of mine that provides a simple restful file storage:

https://github.com/oveits/simple-restful-file-storage

2016-12-09-11_02_22-github-triggered-build-config-jenkins

Step 4.3 Configure Source Code Management

Under Source Code Management, we choose “Git” and specify the GitHub repository a second time. If it is public, we do not enter the credentials for now:

2016-12-09-14_53_46-github-triggered-build-config-jenkins

-> Click Apply

Note also that I have chosen a branch different from the master branch (“jenkinstest”). I have created this new branch in order to keep the master branch clean from any changes that might be needed to test Jenkins.

Step 4.4 Configure Build Triggers (postponed to part 3 of this blog post series)

For now, we will test only manual “build now” triggers, so we do not need to specify any build triggers. Build triggers will be tested in the next blog post.

Step 4.5: Alternative (a): Configure Gradle Build

Prerequisite: For creation of an executable JAR, the file build.gradle in the project root directory must be prepared. See e.g.

Here, we show how to build the project via Gradle. If you prefer to build via Maven, seee Step 4.5: Alternative (b): Configure Maven Build below.

Here, we scroll down to “Build” -> click 2017-01-02-14_54_07-github-triggered-build-config-jenkins -> “Invoke Gradle script”

2016-12-09-11_11_47-github-triggered-build-config-jenkins

Choose the Gradle version we have prepared in Step 3 and add the task “jar”:

2017-01-02-14_56_34-github-triggered-build-config-jenkins

Note that keeping the (Default) Gradle version will not work, as long as this Default has not been defined. See Appendix A for details.

The Gradle task “jar” will create our executable JAR file.

-> Click Save at the bottom left.

Step 4.5: Alternative (b): Configure Maven Build

Here, we show how to build the project via Maven. If you prefer to build via Gradle, seee Step 4.5: Alternative (a): Configure Graven Build above.

Here, we scroll down to “Build” -> click 2017-01-02-14_54_07-github-triggered-build-config-jenkins -> “Invoke Maven script”

2017-01-03-13_52_27-github-triggered-build-config-jenkins

Choose the Maven version we have prepared in Step 3 and specify the goal “package”:

2017-01-03-14_00_05-github-triggered-build-config-jenkins

Note that keeping the (Default) Maven version will not work, as long as this Default has not been defined.

The Maven goal “package”will build our JAR file.

-> Click Save at the bottom left.

Step 5: Test manually triggered Build

We can trigger a build manually via Jenkins -> drop-down right of “GitHub Triggered Build” -> Build Now.

2016-12-09-11_46_03-dashboard-jenkins

Click on #1 of the build history:

2017-01-02-14_59_15-github-triggered-build-jenkins-v2

then on Console Output:

2017-01-02-15_02_46-github-triggered-build-1-jenkins-v2

 

We can observe the console output e.g. by clicking on the build link in the build history, then clicking Console:

Output for step 5 in case of Gradle:

2017-01-02-15_04_25-github-triggered-build-1-console-jenkins

This may take a while (~11 min in my case with a 100Mbps Internet connection):

2017-01-02-15_48_17-github-triggered-build-1-console-jenkins

thumps_up_3

This was the first successful Jenkins triggered Git download and Gradle build.

Output for step 5 in case of Maven:

2017-01-03-14_05_03-github-triggered-build-4-console-jenkins

This may take a while (~8 min in my case with a 100Mbps Internet connection):

2017-01-03-14_42_22-github-triggered-build-4-console-jenkins

We can see in the output, that the JAR file was placed to

/var/jenkins_home_local/workspace/GitHub Triggered Build/target/camel-spring4-0.0.1-SNAPSHOT.jar

thumps_up_3

This was the first successful Jenkins triggered Git download and Maven build.

Step 5.2 (Optional): Measure Time Consumption for Gradle clean Build

Let us test again, whether the build is quicker the second time:

-> Back to Project

-> Configure

-> Add “clean” Gradle task before “jar” Gradle task:

2017-01-02-16_00_27-github-triggered-build-config-jenkins

-> Save

-> Build Now

-> Build Histrory -> current build

-> Console Output

Clean Build - Console Ouptut

This is showing that a clean build takes only ~6.4 sec, if all SW is downloaded already.

Step 6 (Gradle): Retrieve and start executable JAR File

For Maven, scroll down to Step 6 (Maven).

Let us see, where the executable jar file can be found:

For that, let us enter a bash session on the same Docker container:

(dockerhost)$ docker exec -it jenkins bash
jenkins@(container):/$ cd /var/jenkins_home_local/workspace/GitHub\ Triggered\ Build/build/libs/

In case you have started Jenkins with the jenkins image (Step 1.2, alternative (A)), the project will be found on

(container):$ cd /var/jenkins_home

In case you have started Jenkins with the oveits/jenkins_tutorial image (Step 1.2, alternative (B)), the project will be found on

(container):$ cd /var/jenkins_home_local

Then enter the Project. In my case “GitHub Triggered Build”

(container)$ cd 'GitHub Triggered Build'

The jar is found on the path defined in build.gradle file (default: build/libs).

(container)$ cd build/libs
(container)$ ls
GitHub Triggered Build-0.0.1-SNAPSHOT.jar   META-INF   lib   log4j.properties   properties   templates

Now let us start the executable file:

$ java -jar 'GitHub Triggered Build-0.0.1-SNAPSHOT.jar'
[                          main] MainSupport                    INFO  Apache Camel 2.16.0 starting
0 [main] INFO org.apache.camel.main.MainSupport  - Apache Camel 2.16.0 starting
[                          main] DefaultTypeConverter           INFO  Loaded 196 type converters
1706 [main] INFO org.apache.camel.impl.converter.DefaultTypeConverter  - Loaded 196 type converters
...
2762 [main] INFO org.apache.camel.spring.SpringCamelContext  - Total 15 routes, of which 15 is started.
[                          main] SpringCamelContext             INFO  Apache Camel 2.16.0 (CamelContext: camel-1) started in 1.046 seconds
2765 [main] INFO org.apache.camel.spring.SpringCamelContext  - Apache Camel 2.16.0 (CamelContext: camel-1) started in 1.046 seconds

Yes. perfect, it seems to work.
thumps_up_3

You can stop the Apache Camel process by pressing <CTRL>-C in the console.

Step 6 (Maven): Retrieve and start executable JAR File

For Gradle, scroll up to Step 6 (Gradle).

In case of Maven, the location of the created JAR file can be seen at the end of the build console output:

[INFO] Building jar: /var/jenkins_home_local/workspace/GitHub Triggered Build/target/camel-spring4-0.0.1-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:15 min
[INFO] Finished at: 2017-01-03T13:13:06+00:00
[INFO] Final Memory: 37M/263M
[INFO] ------------------------------------------------------------------------
Finished: SUCCESS

Let us test the executable JAR:

(dockerhost)$ docker exec -it jenkins bash
jenkins@(container):/$ java -jar '/var/jenkins_home_local/workspace/GitHub Triggered Build/target/camel-spring4-0.0.1-SNAPSHOT.jar'
no main manifest attribute, in /var/jenkins_home_local/workspace/GitHub Triggered Build/target/camel-spring4-0.0.1-SNAPSHOT.jar

Okay, the jar is not executable yet. Let us change the POM file to create an executable fat JAR as described on Mkyong’s page:

$ git clone <repository-URL>
$ cd <repository-Dir>
$ vi pom.xml

Add the following text to the plugins-part of pom.xml:

cloning the git repository, adding the text below to the plugins part, adding pom.xml to git, commit the git change and push the change:

      <!-- Maven Assembly Plugin -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>2.4.1</version>
        <configuration>
          <!-- get all project dependencies -->
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
          <!-- MainClass in mainfest make a executable jar -->
          <archive>
            <manifest>
            <mainClass>de.oveits.simplerestfulfilestorage.MainApp</mainClass>
            </manifest>
          </archive>

        </configuration>
        <executions>
          <execution>
          <id>make-assembly</id>
          <!-- bind to the packaging phase -->
          <phase>package</phase>
          <goals>
            <goal>single</goal>
          </goals>
          </execution>
        </executions>
      </plugin>

For other projects, you will need to adapt the blue part above.

Then:

$ git add pom.xml
$ git commit -m "Maven creates fat executable JAR file now"
$ git push

Now again, let us build the project:

-> Build Now

-> 2017-01-05-01_15_51-github-triggered-build-jenkins

-> Console Output

Now there are many downloads, and it takes a while:

2017-01-05-01_18_33-github-triggered-build-6-console-jenkins

After ~2.5 minutes, it is ready:

2017-01-05-01_19_25-github-triggered-build-6-console-jenkins

And we can find and run the new fat JAR file on the Docker container:

(dockerhost)$ docker exec -it jenkins bash
(container) $ ls -ltr '/var/jenkins_home_local/workspace/GitHub Triggered Build/target'
total 57680
drwxr-xr-x 3 jenkins jenkins     4096 Jan  3 13:12 generated-sources
drwxr-xr-x 6 jenkins jenkins     4096 Jan  3 13:12 classes
drwxr-xr-x 3 jenkins jenkins     4096 Jan  3 13:12 generated-test-sources
drwxr-xr-x 3 jenkins jenkins     4096 Jan  3 13:12 test-classes
drwxr-xr-x 2 jenkins jenkins     4096 Jan  3 13:13 maven-archiver
-rw-r--r-- 1 jenkins jenkins    44657 Jan  4 23:58 camel-spring4-0.0.1-SNAPSHOT.jar
drwxr-xr-x 2 jenkins jenkins     4096 Jan  5 00:00 archive-tmp
-rw-r--r-- 1 jenkins jenkins 58988354 Jan  5 00:00 camel-spring4-0.0.1-SNAPSHOT-jar-with-dependencies.jar

Here, we can see, that a large JAR file with all dependencies has been created. Now let us try to execute it:

(container) $ java -jar '/var/jenkins_home_local/workspace/GitHub Triggered Build/target/camel-spring4-0.0.1-SNAPSHOT-jar-with-dependencies.jar'
17/01/05 00:07:50 INFO main.MainSupport: Apache Camel 2.16.0 starting
0 [main] INFO org.apache.camel.main.MainSupport  - Apache Camel 2.16.0 starting
...
17/01/05 00:07:52 INFO spring.SpringCamelContext: Total 15 routes, of which 15 is started.
2420 [main] INFO org.apache.camel.spring.SpringCamelContext  - Total 15 routes, of which 15 is started.
17/01/05 00:07:52 INFO spring.SpringCamelContext: Apache Camel 2.16.0 (CamelContext: camel-1) started in 0.876 seconds
2422 [main] INFO org.apache.camel.spring.SpringCamelContext  - Apache Camel 2.16.0 (CamelContext: camel-1) started in 0.876 seconds

Yes. perfect, it seems to work.

thumps_up_3

You can stop the Apache Camel process by pressing <CTRL>-C in the console.

Step 7: How to get the JAR file from the Jenkins Container to the Docker Host in case you have mapped the jenkins_home

In case, you have taken the alternative (A) way or starting Jenkins with your own jenkins_home directory on the Docker host on step 1.2, you can retrieve the JAR file from the project file without copying the file from the container to the Docker host. In my case, the project folder is located on

<jenkins_home>/workspace/GitHub Triggered Build

2016-12-09-15_49_11-github-triggered-build

And from there, the default location for Gradle to place the created JAR file is on ‘build/libs’ as discussed here:
2016-12-09-15_50_27-libs

Step 7: Alternative (B) Retrieve JAR File in case jenkins_home is located on the Container only

In case, you have taken the alternative (B) way or starting Jenkins with the jenkins_home directory on the Docker container on step 1.2, you need to copy the JAR file from the container to another location. The easiest way to do so is to copy it via

(dockerhost) $ docker cp <containerId>:/file/path/within/container /host/path/target

The container ID can be seen via docker ps:

(dockerhost) $ docker ps | grep jenkins
9159bedefbee        oveits/jenkins_tutorial:part2_step1   "/start.sh"              9 hours ago         Up 9 hours          0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   jenkins

Now we can copy the jar file to the Docker host via:

(dockerhost) $ docker cp '9159bedefbee:/var/jenkins_home_local/workspace/GitHub Triggered Build/build/libs/GitHub Triggered Build-0.0.1-SNAPSHOT.jar' '/vagrant/GitHub Triggered Build-0.0.1-SNAPSHOT.jar'

Since our Docker host is a Vagrant virtual machine, we have chosen a destination on the /vagrant folder, since this folder is synchronized with the Vagrant host machine per default. This way, we can get access to the JAR file on the host machine without further ado:
2017-01-03-00_17_19-ubuntu-trusty64-docker_openshift-installer

Appendix A: Error Message “Cannot run gradle”

Problem:

If the Gradle plugin is installed, but not configured according to step 3 above, you will get following build error:

2016-12-09-11_19_14-github-triggered-build-1-console-jenkins

Resolution:

Perform step 3: Prepare Gradle Usage (alternative (a), see above)

Appendix B: How I have created the Image for Part 2 (oveits/jenkins_tutorial:part2_step1)

Note: This appendix is for reference only and describes, how I have created the Docker images for usage in Part 2 (Part 3, …). You do not need to follow those steps.

The images like jenkins_tutorial:part2 are designed for users, who want to skip steps performed in part 1 and directly start with step 1 in part 2. Similarly with images like jenkins_tutorial:part3_step1jenkins_tutorial:part4_step1, …

My target is to create an Image that has the jenkins_home directory stored in the container. The directory /var/jenkins_home cannot be saved in the container, since it is defined as external volume (if I run a docker commit, all changes in /var/jenkins_home are ignored, even if it is not mapped to a Docker host volume).

Since I do not know how to remove the VOLUME label from /var/jenkins_home in the official jenkins image, the only solution I see is to use a different Jenkins home directory within the container/image.

Step B.1 Start Container from official Jenkins Image

sudo docker run -it --name jenkins_tutorial -p8080:8080 -p50000:50000 -eJENKINS_HOME="/var/jenkins_home_local" --entrypoint=bash jenkins

Step B.2: Create new home Directory

On another terminal determine the container ID (with docker ps).

Assuming that running container has ID 84ec9c83c1ce, I log into the container as root (see this Stackoverflow page), even though we do not know the root password:

(dockerhost)$ sudo docker exec -u 0 -it 84ec9c83c1ce bash

Inside the container, we now can create the new home directory and assign it to the user “jenkins”

(container)# mkdir /var/jenkins_home_local
(container)# chown jenkins:jenkins /var/jenkins_home_local

Step B.3: Create new Entrypoint Startup Script

Now we can create a new entrypoint startup script as follows:

(container)# echo '#!/bin/bash' > /start.sh
(container)# echo 'export JENKINS_HOME=/var/jenkins_home_local' >> /start.sh
(container)# echo '/bin/tini -- /usr/local/bin/jenkins.sh' >> /start.sh
(container)# chmod +x /start.sh
(container)# exit

Step B.4: Save Container into an Image

With the following command, we can save the container as a Docker image:

(dockerhost)$ sudo docker commit e82d5277431d jenkins_local

Step B.5: Change Entrypoint

We now change the entrypoint to run the start.sh script we have created above. For that, we run a new container with the entrypoint and save it as image again.

(dockerhost)$ sudo docker run -it --name jenkins_tutorial -p8080:8080 -p50000:50000 -eJENKINS_HOME="/var/jenkins_home_local" --entrypoint="/start.sh" jenkins_local

Step B.6: Save Container as Docker Image

We now can save the container as a Docker image with the new entrypoint. For that, we can issue a docker ps command to find the container ID.

Assuming that the new container ID is 5e82d8360b3e, we can save the container as an image with the following command. With the docker push commands, we save the image to Docker Hub (here as version 0.4 and as an implicit latest tag):

(dockerhost)$ docker commit 5e82d8360b3e oveits/jenkins_tutorial:part1_step1_v0.4
(dockerhost)$ docker tag jenkins_local oveits/jenkins_tutorial:part1_step1
(dockerhost)$ docker push oveits/jenkins_tutorial:part1_step1_v0.4
(dockerhost)$ docker push oveits/jenkins_tutorial:part1_step1

Step B.7: Create Image for Part 2 (Part 3, …, Part X)

For creating the image oveits/jenkins_tutorial:part1_step1 for usage on part2, we

  1. start image oveits/jenkins_tutorial:part1_step1
  2. perform all steps described on part 1
  3. save (commit) the changes to image oveits/jenkins_tutorial:part2_step1

TODO: the steps of Appendix B should be automated by using a Dockerfile and saving start.sh and the Dockerfile to Github.

Appendix C: docker exec produces error “unable to find user jenkins: no matching entries in passwd file”

Symptoms:

The images I have created in Appendix B can be started and will run inside as the user “jenkins”. However, if the container is running and you want to create a new parallel bash session like so:

(dockerhost)$ docker exec -it 9159bedefbee bash
unable to find user jenkins: no matching entries in passwd file

Here, I was assuming that the Docker container ID is 9159bedefbee.

The error message is misleading, since I can show that the passwd file contains a jenkins user by entering as root:

(dockerhost)$ docker exec -u 0 -it 9159bedefbee bash
root@9159bedefbee:/# grep jenkins /etc/passwd
jenkins:x:1000:1000::/var/jenkins_home:/bin/bash

Workaround:

You can run an exec session as user “jenkins” by specifying the user ID 1000:

(dockerhost)$ docker exec -u 1000 -it 9159bedefbee bash
jenkins@9159bedefbee:/$

Summary

In this blog post we have performed following tasks:

  1. Started Jenkins in a Docker container
  2. Created a project
  3. Configured the project with Git and Gradle information
  4. Manually triggered a build
    1. Jenkins has downloaded the Git repository,
    2. started Gradle build and performed the tasks we have configured in the project configuration
  5. Searched for the executable JAR file on the Jenkins server and started the JAR file successfully from command line

In order to avoid any compatibility issues with the java version on the host, we have run Jenkins in a Docker container. In order to better see what happens under the hood, we have chosen to run the Docker container in interactive terminal mode. While it was a little bit confusing that the GitHub repository had to be specified twice, everything was working fine, once the Source code management was configured correctly

Further Reading:

 

10

Jenkins Part 1: Installation the Docker Way


2016-11-30-18_19_38
In this blog post, we will deploy and get started with Jenkins, the most popular open source tool for Continuous Integration and Continuous Deployment. As a modern way of installing, we install a Docker host and deploy a Jenkins Docker container on this host. Then we will log in and install commonly used plugins, before we poke around and prepare for the next step, i.e. creating an automated build process for creating, testing and deploying software.

This blog post series is divided into following parts:

    • Part 1 (this blog): Installation and Configuration of Jenkins, loading Plugins
    • Part 2: Creating our first Jenkins Job: GitHub download and Software build
    • Part 3: Periodic and automatically triggered Builds
    • Part 4 (planned): running automated Tests

What is Jenkins?

Jenkins is the leading open source automation server mostly used in continuous integration and continuous deployment pipelines. Jenkins provides hundreds of plugins to support building, deploying and automating any project.

Waterfall Process

In former times, software version life cycle was often counted in many months or years: a set of required features for a new version was defined, and then all of those features were first designed, then implemented, verified, delivered and maintained. In this software development process called waterfall process, each of the phase could take weeks or even months.

2016-12-01-20_37_59
Waterfall Process

Agile Process

Nowadays, most new software is created in an agile process. Unlike in the waterfall process, the time and other resources for a process cycle, the so-called sprint, is fixed: often to two or four weeks. A quite small set of features is implemented during each sprint, but all phases from design over implementation, test to delivery are accomplished in a single sprint. The great advantage of the agile methodology is that you can deliver a fully tested software version every two or four weeks. This is giving your customer the chance to see the progress and the developers can get early feedback, which makes sure they do not run into a wrong direction.

Agile Methodology
Source: https://crowdsourcedtesting.com/resources/wp-content/uploads/2016/07/agile-methodolody_695x260.jpg found in this article.

Jenkins for Build, Test and Deployment Automation

Since all processes, including software build, test and deployment, are performed every two or four weeks, this is an ideal playground for automation tools like Jenkins: it does not make sense to occupy a person with those tedious repeated tasks, if it can also be done by a computer.

Here comes Jenkins into play: After the developer commits a code change to the repository, Jenkins will detect this change and will trigger the build and test process. The results are immediately sent to the developer. Since the steps are automated now, the developer can trigger the process after each small, incremental change. He gets early feedback, if he has broken the software unintentionally, which makes troubleshooting much easier.

Jenkins build, test and deployment pipeline
Jenkins build, test and deployment pipeline

If all tests were successful, Jenkins can automatically deliver and deploy the software to a staging system or even to a production system.

The whole process is called continuous integration (build and test automation with feedback), continuous delivery (deployment can be done “automated” with a push of a button) or continuous deployment (deployment is performed automatically with no manual intervention).

Installing Jenkins the Docker Way

2016-12-02-10_40_22

Tools used

      • Vagrant 1.8.6
      • Virtualbox 5.0.20
      • Docker 1.12.1
      • Jenkins 2.19.3

Prerequisites:

      • Free DRAM overall >~ 1 GB, better 2 GB (Vagrant, VirtualBox and Jenkins container)
      • If Docker host is available already: free DRAM of host before starting the Jenkins container >~ 600MB, better 1 GB

Step 1: Install a Docker Host via Vagrant and Connect to the Host via SSH

If you are using an existing docker host, you can skip this step. Make sure that your host has enough memory.

We will run Jenkins in a Docker container in order to allow for maximum interoperability. This way, we always can use the latest Jenkins version without the need to control the java version used.

If you are new to Docker, you might want to read this blog post.

Installing Docker on Windows and Mac can be a real challenge, but no worries: we will show an easy way here, that is much quicker than the one described in Docker’s official documentation:

Prerequisites of this step:

      • I recommend to have direct access to the Internet: via Firewall, but without HTTP proxy. However, if you cannot get rid of your HTTP proxy, read this blog post.
      • Administration rights on you computer.

Steps to install a Docker Host VirtualBox VM:

Download and install Virtualbox (if the installation fails with error message “Setup Wizard ended prematurely”, see Appendix A of this blog post: Virtualbox Installation Workaround below)

1. Download and Install Vagrant (requires a reboot)

2. Download Vagrant Box containing an Ubuntu-based Docker Host and create a VirtualBox VM like follows:

basesystem# mkdir ubuntu-trusty64-docker ; cd ubuntu-trusty64-docker
basesystem# vagrant init williamyeh/ubuntu-trusty64-docker
basesystem# vagrant up
basesystem# vagrant ssh

Now you are logged into the Docker host and we are ready for the next step: to create the Ansible Docker image.

Note: I have experienced problems with the vi editor when running vagrant ssh in a Windows terminal. In case of Windows, consider to follow Appendix C of this blog post and to use putty instead.

Step 2 (optional): Download Jenkins Image

This extra download step is optional, since the Docker image will be downloaded automatically in step 3, if it is not already found on the system:

(dockerhost)$ sudo docker pull jenkins
Using default tag: latest
latest: Pulling from library/jenkins
Digest: sha256:8820149b54bfc5d05146b82150b5fdab583eef3e0499fb4ed630f77647a42942
Status: Image is up to date for jenkins:latest

The version of the downloaded Jenkins image can be checked with following command:

(dockerhost)$ sudo docker run -it --rm jenkins --version
2.19.3

We are using version 2.9.13 currently. If you want to make sure that you use the exact same version as I have used in this blog, you can use the imagename jenkins:2.19.3 in all docker commands instead of jenkins only.

Note: The content of the jenkins image can be reviewed on this link. There, we find that the image has an entrypoint /bin/tini -- /usr/local/bin/jenkins.sh, which we could override with the --entrypoint bash option, if we wanted to start a bash shell in the jenkins image. However, in Step 3, we will keep the entrypoint for now.

Step 3: Start Jenkins in interactive Terminal Mode

In this step, we will run Jenkins interactively (with -it switch instead of -d switch) to better see, what is happening. But first, we check that the port we will use is free:

(dockerhost)$ sudo docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                            NAMES
0ec82b4ca2fd        google/cadvisor:latest   "/usr/bin/cadvisor -l"   2 days ago          Up 2 days           0.0.0.0:8080->8080/tcp                           cadvisor
...

Since we see that one of the standard ports of Jenkins (8080, 50000) is already occupied and I do not want to confuse the readers of this blog post by mapping the port to another host port, I just stop the cadvisor container for this “hello world”:

(dockerhost)$ sudo docker stop cadvisor
cadvisor

Jenkins will be in need of a persistent storage. For that, we create a new folder on the Docker host:

(dockerhost)$ mkdir jenkins_home; cd jenkins_home

Note: The content of the jenkins image can be reviewed on this link. There, we find that the image has an entrypoint /bin/tini -- /usr/local/bin/jenkins.sh, which we could override with the --entrypoint bash option, if we wanted to start a bash shell in the jenkins image.

We start the Jenkins container with the jenkins_home Docker host volume mapped to /var/jenkins_home:

(dockerhost)$ sudo docker run -it --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Nov 30, 2016 6:12:14 PM Main deleteWinstoneTempContents
WARNING: Failed to delete the temporary Winstone file /tmp/winstone/jenkins.war
Nov 30, 2016 6:12:14 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Logging initialized @347ms
Nov 30, 2016 6:12:14 PM winstone.Logger logInternal
INFO: Beginning extraction from war file
Nov 30, 2016 6:12:14 PM org.eclipse.jetty.util.log.JavaUtilLog warn
WARNING: Empty contextPath
Nov 30, 2016 6:12:14 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: jetty-9.2.z-SNAPSHOT
Nov 30, 2016 6:12:16 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Nov 30, 2016 6:12:17 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started w.@7674f035{/,file:/var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
Nov 30, 2016 6:12:17 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started ServerConnector@548d708a{HTTP/1.1}{0.0.0.0:8080}
Nov 30, 2016 6:12:17 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: Started @3258ms
Nov 30, 2016 6:12:17 PM winstone.Logger logInternal
INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled
Nov 30, 2016 6:12:17 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Nov 30, 2016 6:12:17 PM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Nov 30, 2016 6:12:19 PM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Nov 30, 2016 6:12:19 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Nov 30, 2016 6:12:19 PM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
Nov 30, 2016 6:12:20 PM jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
Nov 30, 2016 6:12:20 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Download metadata
Nov 30, 2016 6:12:20 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 97 ms
Nov 30, 2016 6:12:20 PM org.jenkinsci.main.modules.sshd.SSHD start
INFO: Started SSHD at port 44955
Nov 30, 2016 6:12:21 PM jenkins.util.groovy.GroovyHookScript execute
INFO: Executing /var/jenkins_home/init.groovy.d/tcp-slave-agent-port.groovy
Nov 30, 2016 6:12:22 PM jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
Nov 30, 2016 6:12:22 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@453fc3cf: display name [Root WebApplicationContext]; startup date [Wed Nov 30 18:12:22 UTC 2016]; root of context hierarchy
Nov 30, 2016 6:12:22 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@453fc3cf]: org.springframework.beans.factory.support.DefaultListableBeanFactory@79a53f4b
Nov 30, 2016 6:12:22 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@79a53f4b: defining beans [authenticationManager]; root of factory hierarchy
Nov 30, 2016 6:12:22 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@7ea44b7: display name [Root WebApplicationContext]; startup date [Wed Nov 30 18:12:22 UTC 2016]; root of context hierarchy
Nov 30, 2016 6:12:22 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@7ea44b7]: org.springframework.beans.factory.support.DefaultListableBeanFactory@12544046
Nov 30, 2016 6:12:22 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@12544046: defining beans [filter,legacy]; root of factory hierarchy
Nov 30, 2016 6:12:22 PM jenkins.install.SetupWizard init
INFO:

*************************************************************
*************************************************************
*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

0c4a8413a47943ac935a4902e3b8167e

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************

Nov 30, 2016 6:12:27 PM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
Nov 30, 2016 6:12:27 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Step 4: Open Jenkins in a Browser

Now we want to connect to the Jenkins portal. For that, open a browser and open the URL

<your_jenkins_host>:8080

In our case, Jenkins is running in a container and we have mapped the container-port 8080 to the local port 8080 of the Docker host. On the Docker host, we can open the URL.

localhost:8080

Note: In case of Vagrant with VirtualBox, per default, there is only a NAT-based interface and you need to create port-forwarding for any port you want to reach from outside (also the local machine you are working on is to be considered as outside). In this case, we need to add an entry in the port forwarding list of VirtualBox:
2016-11-30-19_22_22-regel-fur-port-weiterleitung

The Jenkins login screen will open:

2016-11-30-19_36_42-jenkins-jenkins

The admin password can be retrieved from the startup log, we have seen above (0c4a8413a47943ac935a4902e3b8167e), or we can find it by typing

(dockerhost: .../jenkins_home)$ cat secrets/initialAdminPassword
0c4a8413a47943ac935a4902e3b8167e

on the mapped jenkins_home folder on the Docker host.

Step 5: Install Plugins

Let us install the suggested plugins:

2016-11-30-19_37_43-setupwizard-jenkins

This may take a while to finish:

2016-11-30-19_39_38-setupwizard-jenkins

Step 6: Create an Admin User and log in

Then we reach a page, where we can create an Admin user:

2016-11-30-19_48_25-setupwizard-jenkins

Let us do so and save and finish.

2016-11-30-19_48_38-setupwizard-jenkins

Note: After this step, I have deleted the Jenkins container and started a new container attached to the same Jenkins Home directory. After that, all configuration and plugins were still available and we can delete containers after usage without loosing relevant information.

I have had a dinner break at this point. Maybe this is the reason I got following message when clicking the “Start using Jenkins” button?

2016-11-30-20_42_21-setupwizard-jenkins

What ever. After clicking “retry”, we reach the login page:

2016-11-30-20_44_53-jenkins

Coming Soon: Create a New Job

In the next, upcoming blog post, we will create our first Jenkins job. I plan to trigger the Maven and/or Gradle build of a Java executable file upon detection of a code change.

2016-11-30-20_47_12-dashboard-jenkins

Summary

In this blog post we have performed following tasks:

      1. installed a Docker host using Vagrant and VirtualBox
      2. downloaded and installed Jenkins the Docker way
      3. installed often used Plugins
      4. created an Admin account

In order to avoid any compatibility issues with the java version on the host, we have run Jenkins in a Docker container. In order to better see what happens under the hood, we have chosen to run the Docker container in interactive terminal mode. We have had an intermediate error “Unable to connect to Jenkins”, but this error was unreproducible and gone after the next click.

Further Reading: