0

Jenkins Part 5.1: Using the Job DSL for automatic Creation of Jenkins Jobs


Today, we will learn how to use the Jenkins Job DSL Plugin to create new Jenkins jobs at a push of a button. We will show how we can use Groovy scripts for defining a “Hello World” Jenkins freestyle project and create such a project by pushing the “Build now” button.

Why creating Jenkins Jobs via Groovy Scripts?

You might say: “Jenkins has such a nice Web interface. Why should I bother about creation of Jenkins Jobs via Groovy Scripts?”

Right: the web interface of Jenkins is simple to use and will lead to quick results for single projects. However, consider a situation with more than 100 projects with more than 400 Jenkins pipelines (e.g. build + test + deploy + release pipelines each). Automating the Jenkins job lifecycle can be a great time-saver here. Automating the Jenkins job lifecycle is best done with the Seed plugin:

From: https://blog.codecentric.de/en/2015/10/using-jenkins-job-dsl-for-job-lifecycle-management/
From: https://blog.codecentric.de/en/2015/10/using-jenkins-job-dsl-for-job-lifecycle-management/ showing a full Jenkins job lifecycle, which is the long term target for us. The current post is a first step towards this target.

The Seed plugin relies on the capabilities of the Job DSL plugin discussed in the current blog post. In one of the next blog posts, we will have a closer look to the Seed plugin.

 

Tools and Versions used

      • Vagrant 1.8.6
      • Virtualbox 5.0.20
      • Docker 1.12.1
      • Jenkins 2.32.2
      • Job DSL Plugin 1.58

Prerequisites:

      • Free DRAM for the a Docker Host VM >~ 4 GB.
      • Docker Host is available.
      • Tested with 2 vCPU (1 vCPU might work as well).

Step 1: Connect to Jenkins

Step 1.1: Start Jenkins in interactive Terminal Mode

Make sure that port 8080 is unused on the Docker host. If you were following all steps in part 1 of the series, you might need to stop cadvisor:

(dockerhost)$ sudo docker stop cadvisor

I assume that jenkins_home is already created, all popular plugins are installed and an Admin user has been created as shown in part 1 of the blog series. We start the Jenkins container with the jenkins_home Docker host volume mapped to /var/jenkins_home:

(dockerhost)$ cd <path_to_jenkins_home> # in my case: cd /vagrant/jenkins_home/
(dockerhost:jenkins_home)$ sudo docker run -it --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins
Running from: /usr/share/jenkins/jenkins.war
...
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

50c150e35a774cexxxxxxxxxxxxxxx

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
...
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Step 1.2: Open Jenkins in a Browser

Now we want to connect to the Jenkins portal. For that, open a browser and open the URL

<your_jenkins_host>:8080

In our case, Jenkins is running in a container and we have mapped the container-port 8080 to the local port 8080 of the Docker host. On the Docker host, we can open the URL.

localhost:8080

Note: In case of Vagrant with VirtualBox, per default, there is only a NAT-based interface and you need to create port-forwarding for any port you want to reach from outside (also the local machine you are working on is to be considered as outside). In this case, we need to add an entry in the port forwarding list of VirtualBox:
2016-11-30-19_22_22-regel-fur-port-weiterleitung

Note that this configuration is not permanent, unless you define the port mappings in the Vagrantfile as follows (see official Vagrant documentation):

config.vm.network "forwarded_port", guest: 8080, host: 8080

Step 1.3: Initialize Jenkins: Unlock Jenkins

Unlock Jenkins

insert the one-time password found in the log during startup or on /var/jenkins_home/secrets/initialAdminPassword

-> Continue

Step 1.4: Initialize Jenkins: Install Plugins

-> Install suggested plugins

Installing suggested plugins

Wait for the installation process to complete.

Step 1.4: Initialize Jenkins: Create Admin User

-> Create First Admin User

-> Save and Finish or “Continue as admin”

-> Start using Jenkins

Note: I recommend to log out and log in in order to test the login.

Step 2: Install the Job DSL Plugin

The Job DSL Plugin can be installed like any other Jenkins plugin:

-> Manage Jenkins
-> Manage Plugins
-> Plugins - Available

-> Filter job-dsl (with dash between job and dsl; wait for the filter to become active and do not press enter, otherwise you will get an error message)

-> Install without restart

Choose: Job DSL plugin

Job DSL: Installation Success

Step 3: Create Job DSL Jenkins Project

We create a Job DSL Job like follows:

-> Back to Dashboard

-> New Item

-> Enter Name: JobDSLJob

-> Freestyle Project

-> OK

Step 4: Configure Job DSL Project

-> Add build step: Process Job DSLs

-> if you have got a Github account, fork this open source Java Hello World software (originally created by of LableOrg) that will allow you to see, what happens with your Jenkins job, if you check in changed code. Moreover the hello world software allows you to perform JUnit 4 tests, run PowerMockito Mock services, run JUnit 4 Integration tests and calculate the code coverage using the tool Cobertura.

-> Use the provided DSL script

-> insert:

job('Job-DSL-Hello-World-Job') {
    scm {
        git('git://github.com/oveits/java-maven-junit-helloworld')
    }
    triggers {
        scm('H/15 * * * *')
    }
    steps {
        maven('-e clean test')
    }
}

here, exchange the username oveits by your own Github username.
2017-02-25-11_50_36-jobdsljob-config-jenkins

-> Save

Step 5: Prepare Maven Usage

Goto Jenkins -> Manage Jenkins -> Global Tool Configuration (available for Jenkins >2.0)

2016-12-09-11_34_55-manage-jenkins-jenkins

2016-12-09-11_35_26-global-tool-configuration-jenkins

Scroll down to Maven -> Add Maven

2017-01-02-14_32_46-global-tool-configuration-jenkins

-> choose Version (3.3.9 in my case)

-> Add a name (“Maven 3.3.9” in my case)

-> Save

Since we have checked “Install automatically” above, I expect that it will be installed automatically on first usage.

Step 6: Prepare Git Usage

As described in this StackOverflow Q&A, we need to add the Git username and email address, since Jenkins tries to tag and commit on the Git repo, which requires those configuration items to be set. For that, we perform:

-> 2017-02-25-18_04_32-job-dsl-hello-world-job-1-console-jenkins

-> Manage Jenkins

-> Configure System

-> scroll down to “Git plugin”

-> Git plugin; user.name = jenkins and user.email = admin@jenkins.org

Step 7: Create Jenkins Job from Code

Step 7.1 Build Project

-> Build Now

Step 7.2 (optional): Check Console Output

-> Build History: click on latest #nnn

-> Console Output

JobDSLJob: Console Output showing Added items: GeneratedJob{name='Job-DSL-Hello-World-Job'} and Finished: SUCCESS

->  Back to Project

Step 7.3: Review automatically built Project

-> 2017-02-25-11_48_23-update-center-jenkins

Jenkins Dashboard showing automatically created Job-DSL-Hello-World-Job

-> Job-DSL-Hello-World-Job

Project Job-DSL-Hello-World-Job showing build failure

This is showing a build failure, since I had not performed Step 5 and 6 before. In your case, it should be showing a success (in blue). If you are experiencing problems here, check out the Appendices below.

-> Configure

-> scroll down to Source Code Management

Source Code Management shows correct git URL

-> Scroll down to Build Triggers

Build Triggers showing Poll SCM every 15 minutes

-> Scroll down to Build

-> verify that “Maven 3.3.9” is chosen as defined in Step 5

-> enter “-e clean test” as Maven Goal

Maven Version "Maven 3.3.9" and Maven Goal "-e clean test"

-> Build Now

See, what happens by clicking on:

-> Build History

-> #nnn

-> Console Output

If everything went fine, we will see many downloads and a “BUILD SUCCESS”:

Job DSL Hello World Job: BUILD SUCCESS

Appendix A: Solve Git Problem: “tell me who you are”

Symptoms: Git Error: status code 128

In a new installation of Jenkins, Git does not seem to work out of the box. You can see this by choosing the Jenkins project Job-DSL-Hello-World-Job on the dashboard, then click “build now”, if the build was not already automatically triggered. Then:

-> Build History

-> Last Build (link works only, if Jenkins is running on localhost:8080 and you have chosen the same job name)

-> Console Output

There, we will see:

Caused by: hudson.plugins.git.GitException: Command "git tag -a -f -m Jenkins Build #1 jenkins-Job-DSL-Hello-World-Job-1" returned status code 128:
stdout: 
stderr: 
*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  git config --global user.name "Your Name"

to set your account's default identity.
Omit --global to set the identity only in this repository.

fatal: empty ident name (for <jenkins@61915398735e.(none)>) not allowed

Resolution:

Step 1: Enter Git Username and Email

As described in this StackOverflow Q&A: we can resolve this issue by either suppressing the git tagging, or (I think this is better) by adding your username and email address to git:

-> 2017-02-25-18_04_32-job-dsl-hello-world-job-1-console-jenkins

-> Manage Jenkins

-> Configure System

-> scroll down to “Git plugin”

-> Git plugin; user.name = jenkins and user.email = admin@jenkins.org

Step 2: Re-run “Build Now” on the Project

To test the new configuration, we go to

-> the Job-DSL-Hello-World-Job and press

-> Build Now

Now, we should see a BUILD SUCCESS like follows:

-> Build History

-> #nnn

-> Console Output

If everything went fine, we will a “BUILD SUCCESS”:

Job DSL Hello World Job: BUILD SUCCESS

Appendix B: Maven Error: Cannot run program “mvn”

Symptoms:

When running a Maven Goal, the following error may appear on the Console log:

FATAL: command execution failed
java.io.IOException: Cannot run program "mvn" (in directory "/var/jenkins_home/workspace/Job-DSL-Hello-World-Job"): error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at hudson.Proc$LocalProc.(Proc.java:245)
	at hudson.Proc$LocalProc.(Proc.java:214)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:846)
	at hudson.Launcher$ProcStarter.start(Launcher.java:384)
	at hudson.Launcher$ProcStarter.join(Launcher.java:395)
	at hudson.tasks.Maven.perform(Maven.java:367)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
	at hudson.model.Build$BuildExecution.build(Build.java:205)
	at hudson.model.Build$BuildExecution.doRun(Build.java:162)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:404)
Caused by: java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 15 more
Build step 'Invoke top-level Maven targets' marked build as failure
Finished: FAILURE

Resolution:

Perform Step 5

2017-01-02-14_32_46-global-tool-configuration-jenkins

and

For Test, you can test a manual: choose the correct Maven version, when configuring a Maven build step like in Step 7:

Maven Version "Maven 3.3.9" and Maven Goal "-e clean test"

and verify that Build Now does not throw the Maven error anymore.

For our case, we need to correct the Job DSL like follows:

In the Script, we had defined the step:

    steps {
        maven('-e clean test')
    }

However, we need to define the Maven Installation like follows:

    steps {
        maven {
            mavenInstallation("Maven 3.3.9")
            goals('-e clean test')
        }
    }

Here, the mavenInstallation needs to specify the exact same name, as the one we have chosen in Step 5 above.

After correction, we will receive the correct Maven goal

-> JobDSL -> Build Now

Now, we can check the Maven configuration:

-> Jenkins Home leading to the Dashboard

-> Job-DSL-Hello-World-Job

-> Configure

After scrolling down, we will see the correct Maven Version:

Maven Version "Maven 3.3.9" and Maven Goal "-e clean test"

DONE

Appendix C: Updating Jenkins

Updating Jenkins (in my case: from 2.32.1 to 2.32.2) was as simple as following the steps below

Note: you might want to make a backup of your jenkins_home though. Just in case…

(dockerhost)$ cd <path_to_jenkins_home> # in my case: cd /vagrant/jenkins_home/
(dockerhost)$ docker pull jenkins # to update the jenkins image
(dockerhost)$ docker rm jenkins # to make shure the container named jenkins is removed
(dockerhost:jenkins_home)$ sudo docker run -d --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins

However, after that, some data was unreadable:

2017-02-24-20_04_36-manage-old-data-jenkins

I have clicked

-> Manage Jenkins
-> Manage
-> Discard Unreadable Data

to resolve the issue (hopefully…). At least, after that, the warning was gone.

Appendix D: Job DSL Syntax

The reference for the Job DSL syntax can be found on the Job DSL Plugin API pages. As an example, the syntax of Maven within a Freestyle project can be found on this page found via the path

> freeStyleJob > steps > maven:

maven {

  • // Allows direct manipulation of the generated XML.

    configure(Closure configureBlock)

  • // Specifies the goals to execute including other command line options.

    goals(String goals)

  • // Skip injecting build variables as properties into the Maven process.

    injectBuildVariables(boolean injectBuildVariables = true)

  • // Set to use isolated local Maven repositories.

    localRepository(javaposse.jobdsl.dsl.helpers.LocalRepositoryLocation location)

  • // Specifies the Maven installation for executing this step.

    mavenInstallation(String name)

  • // Specifies the JVM options needed when launching Maven as an external process.

    mavenOpts(String mavenOpts)

  • // Adds properties for the Maven build.

    properties(Map props)

  • // Adds a property for the Maven build.

    property(String key, String value)

  • // Specifies the managed global Maven settings to be used.

    providedGlobalSettings(String settingsIdOrName)

  • // Specifies the managed Maven settings to be used.

    providedSettings(String settingsIdOrName)

  • // Specifies the path to the root POM.

    rootPOM(String rootPOM)

}

A Maven example can be found on the same page:

job('example') {
    steps {
        maven('verify')
        maven('clean verify', 'module-a/pom.xml')
        maven {
            goals('clean')
            goals('verify')
            mavenOpts('-Xms256m')
            mavenOpts('-Xmx512m')
            localRepository(LocalRepositoryLocation.LOCAL_TO_WORKSPACE)
            properties(skipTests: true)
            mavenInstallation('Maven 3.1.1')
            providedSettings('central-mirror')
        }
    }
}

Summary

In this blog post, we have learned how to

  1. Start and initialize Jenkins via Docker
  2. Prepare the usage of Git and Maven
  3. Install the Job DSL Plugin
  4. Define a Jenkins Job via Groovy script
  5. Create a Jenkins Job by a push of  the “Build now” button
  6. Review and run the automatically created Jenkins job

We have seen that the usage of the Job DSL is no rocket science. The only topic, we had to take care, is, that Git and Maven need to be prepared for first usage on a Jenkins server.

More sophisticated Jenkins jobs can be created by following the documentation on the Job DSL Plugin API pages.

Now, we are ready to explore the Seed plugin, which relies on the Job DSL plugin and helps to fully automate the Jenkins job lifecycle. Our aim is, that one or more Jenkins job is automatically created, when a new project is created on the Git repository. If this is possible, we will see in one of my next blog posts.

Further Reading

0

Upgrading Rails to 4.2.x and Ruby to 2.2.x — an end to end Example


In this post, I have recorded my experiences when upgrading my hobby Web Application, a ProvisioningEngine front end from Rails 4.1.x to Rails 4.2.x and ruby from 2.1.x to 2.2.x. The upgrade went relatively smoothly, apart from an additional gem needed (‘test-unit’) and a test failure due to changes in the Rails Url helpers.

v1: 2016-01-26: original post
v2: 2016-02-13: added a caveats section at the end (just before the summary) and added two rspec caveats with respect to the exception handling

Upgrading Rails and Ruby

Prerequisites

  • In this post, I assume that rbenv is installed. If you need to install rbenv, find the instructions here. Other options are possible (e.g. RVM), but they are out of scope for this blog post.

Preparation steps:

  • Better do not upgrade unless you have good coverage of automated tests of your application which all pass.
    • In my case, I have a small app with about ~200 rspec tests and an in-built mock that allows me to perform the tests independently of the backend systems I integrate with. Perform the tests after each major step.
  • Create a backup of your system.

Upgrade steps:

  • (recommended) Update rails to latest minor version and perform the automated tests. In my case I have updated rails from 4.1.4 to 4.1.14, and check that the tests are still O.K.
  • (optional, depending on the current ruby version and your target rails version) Upgrade ruby to the latest stable version and perform the automated tests. In my case, I have updated ruby from 2.1.3 to 2.2.4. Since our target rails version 4.2.x supports older versions of ruby, this step was optional. However, it will be mandatory later on, when we upgrade rails to 5.0, so I have performed it now.
  • (mandatory) Now we can upgrade rails to 4.2.x (you can find all rails versions here). I have decided to specify version >~ 4.2.4 in the Gemfile and we will see that the latest available 4.2.x version is fetched (4.2.5.1). Also, it is not clear yet, whether this version is supported by travisCI.
  • (mandatory) resolve issues and handle deprecation warnings

Cleanup steps:

  • (mandatory) save and commit changes to the SW repository
  • (mandatory) integration tests and merge to master

Step 0: take backup and perform automated tests:

Perform your automated tests and confirm that they are successful. In my case (I am using rspec), this means:

$ bundle exec rspec -f d spec/requests/provisioningobjects_spec.rb
...
Finished in 40.04 seconds
196 examples, 0 failures

I am a lazy guy, so I have created a VMware Snapshot in stead of performing a backup, which helps me to go back within two minutes. Once the upgrade has proven to be successful, I will remove the snapshot in order to retain the full performance. In addition, I have made sure the SW is committed and pushed to my GitHub repository.

Step 1: update rails to the latest stable minor version:

If you are using git, I recommend to create a new branch, e.g.:

$ git checkout -b upgrade_branch

You may need to save your work before with git add/git commit or save away current changes for later use with git stash.

In the application’s root folder, edit the Gemfile and change the rails version (in my case from 4.1.4 to 4.1.14):

# Gemfile
gem 'rails', '4.1.14'

Now repeat the automated tests and check the results. If successful, commit and save your work e.g. with git:

git add .; git commit -am "updated rails 4.1.4 -> 4.1.14"; git push

Step 2: update ruby

Step 2.1 update rbenv, so you get the full list of ruby versions:

cd ~/.rbenv/
git pull
rbenv install --list

Step 2.2: install new ruby version

Install an available ruby version:

rbenv install 2.2.4
ruby -v

The last command will still show the old version. We still need to tell rbenv to use the new version. Since I am running a productive rails server on the host, I do not want to change the global setting, so I am specifying an app-local version, which will be written into .ruby-version:

rbenv versions
rbenv local 2.2.4
rbenv versions
ruby -v

Now ruby is updated.

Step 2.3: re-install rails and other gems

For this version of ruby, rails and the other gems are not yet installed. Therefore the following error message is expected:

$ rails -v
rbenv: rails: command not found

The `rails' command exists in these Ruby versions:
 2.1.3

Since I have a working project with a Gemfile, we can install bundle first and install rails by performing a bundle update within the project’s root folder:

gem install bundle
bundle update

The latter command will install all gems specified in the Gemfile, inclulding the rails gem. In my case, I had following output:

provisioningengine@ProvisioningEngine:~/ProvisioningEnginev0.5.15_testfolder$ gem install bundle
Fetching: bundler-1.11.2.gem (100%)
Successfully installed bundler-1.11.2
Fetching: bundle-0.0.1.gem (100%)
Successfully installed bundle-0.0.1
Parsing documentation for bundler-1.11.2
Installing ri documentation for bundler-1.11.2
Parsing documentation for bundle-0.0.1
Installing ri documentation for bundle-0.0.1
Done installing documentation for bundler, bundle after 6 seconds
2 gems installed
provisioningengine@ProvisioningEngine:~/ProvisioningEnginev0.5.15_testfolder$ bundle install
Fetching gem metadata from https://rubygems.org/...........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Installing rake 10.5.0
Installing i18n 0.7.0
Installing json 1.8.3 with native extensions
Installing minitest 5.8.4
Installing thread_safe 0.3.5
Installing builder 3.2.2
Installing erubis 2.7.0
Installing rack 1.5.5
Installing mime-types 2.99
Installing arel 5.0.1.20140414130214
Installing addressable 2.4.0
Installing execjs 2.6.0
Installing sass 3.2.19
Installing mini_portile2 2.0.0
Installing ffi 1.9.10 with native extensions
Installing choice 0.2.0
Installing coffee-script-source 1.10.0
Installing thor 0.19.1
Installing cookiejar 0.3.0
Installing daemons 1.2.3
Installing tilt 1.4.1
Installing diff-lcs 1.2.5
Installing eventmachine 1.0.9.1 with native extensions
Installing http_parser.rb 0.6.0 with native extensions
Installing hike 1.2.3
Installing multi_json 1.11.2
Installing pg 0.15.1 with native extensions
Using bundler 1.11.2
Installing ruby-graphviz 1.2.2
Installing rails_serve_static_assets 0.0.4
Installing rails_stdout_logging 0.0.4
Installing rspec-core 2.13.1
Installing rspec-mocks 2.13.1
Installing rubyzip 0.9.9
Installing websocket 1.0.7
Installing spork 1.0.0rc4
Installing sqlite3 1.3.11 with native extensions
Installing rdoc 4.2.1
Installing tzinfo 1.2.2
Installing rack-test 0.6.3
Installing rack-protection 1.5.3
Installing mail 2.6.3
Installing autoprefixer-rails 6.3.1
Installing uglifier 2.7.2
Installing nokogiri 1.6.7.2 with native extensions
Installing childprocess 0.5.9
Installing coffee-script 2.4.1
Installing figaro 1.1.1
Installing rspec-expectations 2.13.0
Installing em-socksify 0.3.1
Installing sprockets 2.12.4
Installing rails_12factor 0.0.2
Installing sdoc 0.4.1
Installing activesupport 4.1.14
Installing sinatra 1.4.7
Installing bootstrap-sass 3.3.5
Installing xpath 2.0.0
Installing selenium-webdriver 2.35.1
Installing em-http-request 1.1.3
Installing actionview 4.1.14
Installing activemodel 4.1.14
Installing delayed_job 4.1.1
Installing factory_girl 4.2.0
Installing jbuilder 2.4.0
Installing capybara 2.1.0
Installing actionpack 4.1.14
Installing activerecord 4.1.14
Installing actionmailer 4.1.14
Installing railties 4.1.14
Installing kaminari 0.16.3
Installing sprockets-rails 2.3.3
Installing delayed_job_active_record 4.1.0
Installing delayed_job_web 1.2.5
Installing rails-erd 1.4.5
Installing seed_dump 3.2.4
Installing coffee-rails 4.0.1
Installing factory_girl_rails 4.2.0
Installing jquery-rails 3.1.4
Installing respond-rails 1.0.1
Installing rspec-rails 2.13.1
Installing rails 4.1.14
Installing sass-rails 4.0.5
Installing turbolinks 2.5.3
Installing spork-rails 4.0.0
Installing jquery-turbolinks 2.1.0
Bundle complete! 30 Gemfile dependencies, 85 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
  Post-install message from ruby-graphviz:

Depending on your version of ruby, you may need to install ruby rdoc/ri data:

<= 1.8.6 : unsupported
 = 1.8.7 : gem install rdoc-data; rdoc-data --install
 = 1.9.1 : gem install rdoc-data; rdoc-data --install
>= 1.9.2 : nothing to do! Yay!

Post-install message from capybara:
IMPORTANT! Some of the defaults have changed in Capybara 2.1. If you're experiencing failures,
please revert to the old behaviour by setting:

    Capybara.configure do |config|
      config.match = :one
      config.exact_options = true
      config.ignore_hidden_elements = true
      config.visible_text_only = true
    end

If you're migrating from Capybara 1.x, try:

    Capybara.configure do |config|
      config.match = :prefer_exact
      config.ignore_hidden_elements = false
    end

Details here: http://www.elabs.se/blog/60-introducing-capybara-2-1

Step 2.4: check ruby installation

The I tried to perform the automated tests, but there are some missing dependencies:

$ bundle exec rspec -f d spec/requests/provisioningobjects_spec.rb
/home/provisioningengine/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/activesupport-4.1.14/lib/active_support/dependencies.rb:247:in `require': cannot load such file -- test/unit/assertions (LoadError)
        from /home/provisioningengine/.rbenv/versions/2.2.4/lib/ruby/gems/2.2.0/gems/activesupport-4.1.14/lib/active_support/dependencies.rb:247:in `block in require'

8-((

What the heck is going on here?

Following the hints in the issue descriptions here and here, I have added the gem ‘test-unit’ to the test group of the Gemfile:

# Gemfile
group :test do
  # ...
  gem 'minitest'
  gem 'test-unit'
end

and performed

bundle install

This has done the job:

$ bundle exec rspec -f d spec/requests/provisioningobjects_spec.rb
...(lots of output and debugs omitted here)...
Finished in 40.04 seconds
196 examples, 0 failures

The procedure might also work without the gem ‘minitest’. In my case it was already installed, so I have not tried without.

Now is a good time to commit the change in git:

$ git add.; git commit -a

And I have specified:

Updated ruby to 2.2.4, re-installed all gems; rails v4.1.14

all 196 tests successful in simulation mode

Step 3: upgrade rails to v 4.2.x

In the Gemfile, I have exchanged the rails line by:

# Gemfile
gem 'rails', '~> 4.2.4'

Then:

$ bundle update
Fetching gem metadata from https://rubygems.org/...........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies............................................................................
Using rake 10.5.0
Using i18n 0.7.0
Using json 1.8.3
Using minitest 5.8.4
Using thread_safe 0.3.5
Using builder 3.2.2
Using erubis 2.7.0
Using mini_portile2 2.0.0
Installing rack 1.6.4 (was 1.5.5)
Using mime-types 2.99
Installing arel 6.0.3 (was 5.0.1.20140414130214)
Using addressable 2.4.0
Using execjs 2.6.0
Using sass 3.2.19
Using bundler 1.11.2
Using ffi 1.9.10
Using choice 0.2.0
Using coffee-script-source 1.10.0
Using thor 0.19.1
Using cookiejar 0.3.0
Using daemons 1.2.3
Using tilt 1.4.1
Using diff-lcs 1.2.5
Using eventmachine 1.0.9.1
Using http_parser.rb 0.6.0
Using hike 1.2.3
Using multi_json 1.11.2
Using pg 0.15.1
Installing power_assert 0.2.7 (was 0.2.2)
Using ruby-graphviz 1.2.2
Using rails_serve_static_assets 0.0.4
Using rails_stdout_logging 0.0.4
Using rspec-core 2.13.1
Using rspec-mocks 2.13.1
Using rubyzip 0.9.9
Using websocket 1.0.7
Using spork 1.0.0rc4
Using sqlite3 1.3.11
Using rdoc 4.2.1
Using tzinfo 1.2.2
Using nokogiri 1.6.7.2
Using rack-test 0.6.3
Using rack-protection 1.5.3
Using mail 2.6.3
Using autoprefixer-rails 6.3.1
Using uglifier 2.7.2
Using childprocess 0.5.9
Using coffee-script 2.4.1
Using figaro 1.1.1
Using rspec-expectations 2.13.0
Using em-socksify 0.3.1
Using sprockets 2.12.4
Installing test-unit 3.1.7 (was 3.0.8)
Using rails_12factor 0.0.2
Using sdoc 0.4.1
Installing activesupport 4.2.5.1 (was 4.1.14)
Installing loofah 2.0.3
Using xpath 2.0.0
Using sinatra 1.4.7
Using bootstrap-sass 3.3.5
Using selenium-webdriver 2.35.1
Using em-http-request 1.1.3
Installing rails-deprecated_sanitizer 1.0.3
Installing globalid 0.3.6
Installing activemodel 4.2.5.1 (was 4.1.14)
Using delayed_job 4.1.1
Using factory_girl 4.2.0
Using jbuilder 2.4.0
Installing rails-html-sanitizer 1.0.3
Using capybara 2.1.0
Installing rails-dom-testing 1.0.7
Installing activejob 4.2.5.1
Installing activerecord 4.2.5.1 (was 4.1.14)
Installing actionview 4.2.5.1 (was 4.1.14)
Using delayed_job_active_record 4.1.0
Using delayed_job_web 1.2.5
Using rails-erd 1.4.5
Using seed_dump 3.2.4
Installing actionpack 4.2.5.1 (was 4.1.14)
Installing actionmailer 4.2.5.1 (was 4.1.14)
Installing railties 4.2.5.1 (was 4.1.14)
Using kaminari 0.16.3
Using sprockets-rails 2.3.3
Using coffee-rails 4.0.1
Using factory_girl_rails 4.2.0
Installing jquery-rails 4.1.0 (was 3.1.4)
Using respond-rails 1.0.1
Using rspec-rails 2.13.1
Installing rails 4.2.5.1 (was 4.1.14)
Using sass-rails 4.0.5
Using turbolinks 2.5.3
Using spork-rails 4.0.0
Using jquery-turbolinks 2.1.0
Bundle updated!

We can see in the log that rails has been upgraded to the latest available 4.2 version at time of writing (4.2.5.1). Now let us test again. This time we get lots of DEPRECATION WARNINGs and an error:

bundle exec rspec -f d spec/requests/provisioningobjects_spec.rb
...
DEPRECATION WARNING: Calling URL helpers with string keys controller, action is deprecated. Use symbols instead. (called from _app_views_customers__sidebar_html_erb__545434719134816655_70018526941280 at /home/provisioningengine/ProvisioningEnginev0.5.15_testfolder/app/views/customers/_sidebar.html.erb:8)
 should have link to 'New Customer' (FAILED - 1)
Failures:

  1) On target solution 'Environment2_V7R1' index should have link to 'New Customer'
     Failure/Error: expect(page).to have_link( "New #{obj}", href: new_provisioningobject_path(obj) + '?per_page=all') #unless obj == "Site"
       expected #has_link?("New Customer", {:href=>"/customers/new?per_page=all"}) to return true, got false
     # ./spec/requests/provisioningobjects_spec.rb:1246:in `block (5 levels) in <top (required)>'

Finished in 17.13 seconds
68 examples, 1 failure

Failed examples:

rspec ./spec/requests/provisioningobjects_spec.rb:1241 # On target solution 'Environment2_V7R1' index should have link to 'New Customer'

Step 4.1: resolving the introduced Url-Helper issue

Okay, there is more work to be done. Maybe I should have tried with ‘bundle install’ first, not with ‘bundle update’, which was updating many gems at once? We could go back with a git stash and try again with bundle install. However, let us quickly see, if there is a quick solution:

The DEPRECATION WARNING is related with the problem. It points to

/home/provisioningengine/ProvisioningEnginev0.5.15_testfolder/app/views/customers/_sidebar.html.erb:8

and there we find:

<li><%= link_to 'New Customer', new_customer_path(@params) %></li>

That is causing the errored test:

expected #has_link?("New Customer" ...

So let us try to understand the DEPRECATION WARNING. By guessing from the error message, I have replaced the @params assignment in

class ProvisioningobjectsController < ApplicationController
def index
  @params = params

by

  @params = {"per_page"=>"all", "controller"=>"customers", "action"=>"index", "target_id"=>nil}

after I had seen that a puts @params.inspect had returned {“per_page”=>...}. Surprisingly, after this change the test was successful with the the DEPRECATION WARNING still present. The reasons seems to be that params is an ActionController::Parameters while the literal {...} is a Hash.

The remaining warning was removed by replacing the keys by symbols:

  @params = {:per_page=>"all", :controller=>"customers", :action=>"index", :target_id=>nil}

Since I do not want to care about the actual parameters, I replaced this by the more general code:

class ProvisioningobjectsController < ApplicationController
def index
  @params = {}
  params.each do |key,value|
     @params[key.parameterize.underscore.to_sym] = value
  end

This will convert any ActionController::Parameters parameters by a Hash with symbol keys.

Even more surprisingly, now all 196 test cases are successful:

$ bundle exec rspec -f d spec/requests/provisioningobjects_spec.rb
...
Finished in 42.56 seconds
196 examples, 0 failures

Step 4.2: resolving DEPRECATION WARNING: You are passing an instance of ActiveRecord::Base to `find`

During the test, we have seen other deprecation warnings like follows:

DEPRECATION WARNING: You are passing an instance of ActiveRecord::Base to `find`. Please pass the id of the object by calling `.id`. (called from validate at /home/provisioningengine/ProvisioningEnginev0.5.15_testfolder/app/models/user.rb:15)

Let us resolve this as well. The waring is quite explicit and points to a line containing:

@site = Site.find(record.site)

This can easily be resolved by exchanging it by:

@site = Site.find(record.site.id)

Done and works.

Step 4.3: resolving DEPRECATION WARNING: renamed configuration option

Now let us tackle the last remaining deprecation warning:

DEPRECATION WARNING: The configuration option `config.serve_static_assets` has been renamed to `config.serve_static_files` to clarify its role (it merely enables serving everything in the `public` folder and is unrelated to the asset pipeline). The `serve_static_assets` alias will be removed in Rails 5.0. Please migrate your configuration files accordingly. (called from block in <top (required)> at /home/provisioningengine/ProvisioningEnginev0.5.15_testfolder/config/environments/test.rb:16)

We just need to exchange the line 16 in config/environments/test.rb

config.serve_static_assets  = true

by

config.serve_static_files  = true

and we are done with all deprecation warnings.

Step 5: cleanup: save changes, additional test with travisCI and remove snapshot

Finally, this is a good time to commit the change to git and to merge the branch to the original branch (“development” branch) and push it to GitHub. In my case, this has triggered an additional test on travisCI (if you have no access, try travisCI home).

2016.01.26-23_59_14-hc_001

Success.

Once real live tests with real provisioned legacy end systems have proven to be successful as well, we will merge the development branch to the master branch and remove the VMware snapshot we had taken before we started the upgrade.

Caveats

  • In the new rspec version (rspec-core: 3.4.2), the exception handling seems to have changed: if there is an exception/abort in the rspec script or in the called code, the rspec just stops, but it does not indicate an error. A short google session did not lead to any results. I need to find a solution or a workaround for this.
  • rspec: begin/rescue block does not work as (I had) expected: if you write a begin/rescue block like
begin
  something_raising_an_exception
rescue
  some_other_code
end

then the rescue is ignored, the abort message is printed and the rspec test suite is stopped.

Summary

For my hobby Web Application, a ProvisioningEngine front end,  I could upgrade Rails and Ruby relatively smoothly within an afternoon+evening session (including the documentation on my blog). There only was a missing dependency ‘test-unit’ and a changed behavior of the url helpers, which has forced me to convert the Url helper parameter from ActionController::Parameters class ) to Hash {...} class with the same content. In addition, the Hash keys had to be converted from "String" to :symbol in order to get rid of the corresponding DEPRECATION WARNING. After that, all ~200 rspec tests were successful.