3

Docker Java Performance Tests


In this blog we will show the surprising result that Docker on Ubuntu seems to outperform Docker on CoreOS by ~30%, when tested with a java templating web service.

But first let us discuss the reason on why I have started the tests in the first place: on my blog Docker Web Performance Tests we had found the surprising result that a Rails Web Application on Docker on a CoreOS VM has about 50% better performance, than Rails directly on Windows hardware. Will we find the same result for java?

java on Windows Hardware vs. Ubuntu VirtualBox VM (beware: apples vs. bananas!)

The answer is no: the chosen java application, an Apache Camel templating web service has a factor ~4 better performance on Windows Hardware than on an Ubuntu VirtualBox machine (beware: apples vs. bananas!).

Now comparing apples with apples (all on VirtualBox VMs):

java on Ubuntu vs. Docker/Ubuntu vs. Docker/CoreOS vs. Docker/boot2docker (apples vs. apples)

The java performance on a the chosen Vagrant deploy-able Ubuntu image has almost the same performance on Docker as on native Ubuntu (only 5% performance degradation). With that good result, the Ubuntu Docker alternative has outperformed the other Docker alternatives by more than 30% and has impressed with java startup times that were more than 3 times as fast as on CoreOS.

Note: At the time of writing the blog post, this dockerization vs. virtualization vs. native comparison paper was unknown to me: it follows a more comprehensive scientific approach by testing many different performance aspects separately. I like it a lot: check it out!

This is only a small proof of concept (POC) of a hobby developer and the POC can only give hints about which road to take. Even though now the more scientific paper is available, I am still fine with having invested the time of testing and publishing, since my test results are focused on the applications I develop (Rails and java Apache Camel) and the infrastructure alternatives I have on my laptop: native Windows vs. Ubuntu VirtualBox VM vs. CoreOS VirtualBox VM vs. boot2docker Virtualbox VM. The scientific paper was only testingUbuntu 13.10 and no CoreOS I was interested in.

Document Versions

v1 original post
v2: updated CoreOS Vagrant image from v713.3.0 to v766.4.0 with no performance improvement
v3: giving CoreOS a last chance: updated to alpha v845.0.0 with no performance improvement
v4: changed introduction and re-organized Tldr; and added a link to a more scientific paper.

Tldr;

When comparing the java performance on a native Ubuntu VM with several Docker host VM alternatives (UbuntuCoreOS and boot2docker), we can see that the chosen Vagrant deploy-able Ubuntu image has almost the same performance on Docker as on native Ubuntu. With that, the Ubuntu Docker alternative has outperformed the other Docker alternatives by more than 30% and has impressed with java startup times that were more than 3 times as fast as the one of the second best candidate: CoreOS.

As expected, java performance has shown to be substantially (~4 times) higher on Windows Hardware than on VirtualBox Ubuntu VM with same number of cores. Note that this last comparison is like comparing apples with bananas: Windows on Hardware vs. Ubuntu Linux on VirtualBox virtual machine with less DRAM.

Test Scenarios

In the current blog, we perform performance measurements of a java Apache Camel application for following scenarios:

  1. application run natively on Windows 7 SP1 on the host system (8 GB DRAM, with ~2.2 GB free before start of the application; 2 core CPU i5-2520M@2.5 GHz)
  2. application run within a VirtualBox Ubuntu 14.04 LTS (64 bit) VM with 1.5 GB DRAM and 1 vCPU on the same Windows host system
  3. application run within a docker container on the above Ubuntu VM (vagrant box version 1.8.1)
  4. application run within a docker container on a CoreOS (Vagrant CoreOS stable 717.3.0 and v766.4.0) Virtualbox VM with same resources as the Ubuntu VM (1.5 GB DRAM and 1 vCPU) run on the Windows system
  5. application run within a docker container on a boot2docker Virtualbox VM with same resources as in the old blog, i.e. (2 GB DRAM and 2 vCPU), run on the same Windows host.
    Note that boot2docker is deprecated by now, but since we have performed the Rails Docker Web Performance Tests on boot2docker, we keep it as a reference.

Considering the results we have seen with the Rails web application, we expect that 3. and 4. have higher performance than 1.

With 2. and 3. we can easily extract the difference that comes from the additional docker layer. For the option 4., we expect the same or a higher performance than in 3, since CoreOS is optimized for running Docker containers.

As in the other performance test blog, we use Apache Bench as the measurement tool.

Test Results

Java Apache Camel Startup Time

  1. application run natively on Windows on the host system
    1. NEED TO TEST AGAIN WITH STDOUT REDIRECTION INTO FILE!!!
      2 cores: Apache Camel 2.12.2 (CamelContext: camel-1) started in 3.5 +-0.6 seconds
  2. application run within a VirtualBox Ubuntu VM with 1.5 GB DRAM and 1 vCPU
    1. 1 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 5.0 seconds
    2. 2 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 3.7 seconds
  3. application run within a docker container on the above Ubuntu VM
    1. 1 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 5.2 seconds
    2. 2 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 4.0 seconds
  4. application run within a docker container on a CoreOS VM with same resources as the Ubuntu VM
    1. 1 vCPU, v713.3.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in 15.2 seconds
    2. 2 vCPU
      1. v713.3.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in 13.4 seconds
      2. v766.4.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in 14.5 +- 1.5 seconds
      3. v845.0.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in [15.0, 13.2, 19.2,13.2] seconds (4 tests)
  5. application run within a docker container on a boot2docker VM with same resources as in the old blog, i.e. (2 GB DRAM and 2 vCPU)
    1. 2 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 24.1 +- 0,5 seconds (4 Tests)

Windows: has the quickest Apache Camel startup (95 routes) with ~3.5 sec +- 0.6 sec.

Ubuntu with or without Docker: with 2 vCPUs, the startup is slightly slower. Only with 1 vCPU, the startup takes ~40% more time on Ubuntu (both, native and in Docker).

CoreOS: Apache Camel has a 3.8 and 3,4 times longer (!) startup time on CoreOS Docker than on Windows and on Ubuntu, respectively. CoreOS is more lightweight than the Ubuntu image, and is supposed to be optimized for Docker. Therefore, this is a surprising result.

boot2docker: has the worst Apache Camel bootup time: it takes almost a 7 times longer (!!) than the same process on Windows. boot2docker is deprecated by now. This performance results is another reason not to use it anymore.

Java Apache Camel Throughput

Test script:

# test:
./ab -n 10000 -c 100 "http://<IP>:<Port>/ProvisioningEngine?action=Add%20Customer&CustomerName=ttt&offlineMode=offlineMode"

Shortly before we run the actual test, we train the application by issuing the same command with -n 100 and -c 100.

  1. application run natively on Windows
    1. 2 i5-2520M CPU cores: Requests per second:    46.3 [#/sec] (mean)
  2. application run within a VirtualBox Ubuntu VM with 1.5 GB DRAM and 1 vCPU (on an i5 host)
    1. 1 vCPU: Requests per second:    14.3 [#/sec] (mean)
    2. 2 vCPU: Requests per second:    17.9 [#/sec] (mean) (i.e. ~25% higher than 1 vCPU)
  3. application run within a docker container on the above Ubuntu VM
    1. 1 vCPU: Requests per second:    12.5 [#/sec] (mean)
    2. 2 vCPU: Requests per second:    17.0 [#/sec] (mean) (i.e. ~35% higher than 1 vCPU)
  4. application run within a docker container on a CoreOS VM with same resources as the Ubuntu VM (1.5 GB DRAM and 1 or 2 vCPU)
    1. 1 vCPU, v713.3.0:: Requests per second:    8.82 [#/sec] (mean)
    2. 2 vCPU
      1. v713.3.0: Requests per second:    13.0 [#/sec] (mean) (i.e. ~48% higher than 1 vCPU)
      2. v766.4.0: Requests per second:    11.8 [#/sec] (mean)
      3. v845.0.0: Requests per second:    13.3 [#/sec] (mean)
  5. application run within a docker container on a boot2docker VM with same resources as in the old blog, i.e. (2 GB DRAM and 2 vCPU)
    1. 1 vCPU: not tested
    2. 2 vCPU: We get the error “Test aborted after 10 failures”, a memory allocation failure, even though we have 2 GB DRAM instead of 1.5 GB!

Note that each requests is creating tons of output on the terminal (deep tracing). However, in call cases, I have redirected the stream into a log file (using the “>” operator).

Note also that the comparison between Windows and Linux variants is not fair, since Windows was tested on hardware with 2 real CPU cores, while the Linux variants was tested on Virtualbox VMs with one or two vCPUs (Ubuntu, Docker on Ubuntu and CoreOS) or 2 vCPUs (boot2docker). However, considering you have a Windows notebook and you want to decide, whether to perform your development and tests on native Windows vs. Linux VM vs. on Docker, this is what you get:

  • Windows: on the host system with 2 real CPU cores is ~4 times higher throughput performance than any of the Linux VM variants on 1 vCPU (comparing apples with bananas, but still relevant for my decision)
  • Ubuntu vs Docker on Ubuntu: Native Linux has a ~5 to 15% higher throughput performance than Docker on Linux
  • CoreOS: Interestingly, the Ubuntu Docker VM image has a ~40% higher throughput performance than the optimized CoreOS image. Because of this negative result for CoreOS, I have updated the v713.3.0 image to the latest versions available: stable-v766.4.0 and alpha-v845.0.0. This had no substantial impact on the (still bad) performance.
  • boot2docker: The boot2docker image has memory allocation problems. Since boot2docker is deprecated, I did not investigate this further.

Summary

We have compared the performance of a java Apache Camel application on following platforms: on a Windows 7 laptop, on an Ubuntu Virtualbox image and in a Docker container, where Docker was tested on Ubuntu, CoreOS and the official, but deprecated boot2docker image.

Note that Windows has been tested on hardware, while the Linux variants have been tested on Virtualbox VMs. This is not a fair test between Windows and Linux, but it still helps me to decide, whether I better keep performing my java development directly on my Windows laptop, or whether I can move all java development to a Virtualbox Linux VM on my Windows laptop. For Rails, I had found the surprising result, that the performance was better on the VMs than directly on the hardware. Not for java, though:

2015.10.28-10_22_08-hc_001

We have seen that the performance on Windows 7 without virtualization is ~4 times higher than on any of the Virtualbox VMs, which will account for the expected performance degradation effect of software virtualization. This is no comparison between java on Windows and java on Ubuntu. It is just a comparison of the options I have on a Windows laptop: directly work on Windows or work on the VMs. The CoreOS image has a ~30% lower performance than the Ubuntu image, which is surprising, since it is much more lightweight than the Ubuntu image.

The application’s startup times are quite good for Ubuntu and for Docker on Ubuntu. Surprisingly, both, CoreOS as well as boot2docker fall back substantially with respect to startup times (factor 3 and 6 compared to Ubuntu).

The deprecated boot2docker image has major memory allocation problems and should not be used.

Recommendation

All in all, development is best to be performed directly on the Windows laptop, with Ubuntu or Docker on Ubuntu being a good alternative because of low application startup times. CoreOS does not seem to be a good alternative, since in the development phase, I often need to restart the application, which takes ~13 sec on CoreOS instead of ~3.5 to 4 sec on Windows hardware or on the Ubuntu VM’s Docker.

Performance tests are best to be done on Windows hardware (or on HW-virtualized VMs, which I have not performance-tested yet). However, because of the deployment advantages, I would like to deliver my application as a Docker image, so I need to perform tests on Docker as well. For those Docker tests, the Ubuntu-trusty64-docker Virtualbox image of William Yeh has shown the best performance results.

If I need to test the behavior of the application on clustered Docker hosts, CoreOS and Kubernetes are the only cluster alternatives I have experience with, and I have not done more than a small installation POC in case of Kubernetes.  However, considering the low CoreOS performance, I will need to investigate its alternatives, I guess: e.g. Docker Swarm, Kubernetes, or others (ping me, if you have suggestions).

My Path towards Continuous Integration of my Java Application

I am planning to do the following:

  • I will continue to develop my java application on native Windows and I will continue to push the code to Github often.
  • I have linked Github with TravisCI, so the code is automatically tested in the cloud. This way, I do not need to wait for test results. The test results are sent via email and I can react soon for any newly introduced bugs.
  • I will link TravisCI with Docker Hub, so a Docker image will be created automatically.
  • Locally, I will use the Docker on Ubuntu, if I need to troubleshoot the Docker build process, or I need to manually test, whether the Docker image really works.
  • If it comes to productive deployment including Docker host clustering, I had made good experience with CoreOS, even though clustering behind a HTTP proxy is a challenge; see here. However, the performance is sub-optimal, and I need to evaluate, which alternatives I have (e.g. test new CoreOS versions? Docker Swarm? Google Kubernetes?).

Watch out for my next blog post(s)…

;-))

0

Java Build Automation Part 1: Create lean executable jar using Maven


Have you ever tried to create an executable jar file, while keeping all configuration files outside of the jar file, accessible to operations folks? Here is a step by step guide.

JAR files are java archive files and as such, are not designed to be manipulated by operations folks (“ops”). However, if a developer creates an executable jar, all needed files, including configuration and template files are packed into the jar file as a default. In this post, I will show, how to make configuration and template files accessible again.

Update/Comment (2016-11-20): I now have created a blog, which (in my opinion) shows a better way of creating a lean executable JAR file using gradle: see here.

Automate the creation of the executable jar

Up to now, I have used Eclipse’s export function to create executable jars for both, the main program and the JUnit test program.

2015.10.19-20_26_44-hc_001   2015.10.19-20_28_30-hc_001

However, this mouse-click procedure has forced me to manually extract the configuration files from the jar file, so the administrator can change the configuration files without compiling the source code again. Since I want to come a little step closer to an automated build of a Docker image for my application, I need to automate the creation of the executable jar. A short Internet research has pointed into the direction of three possible options to do so (please tell me, if you now better ones!):

  1. using jar commands
  2. using ant
  3. using maven

About 1.: jar

jar commands are similar to tar commands and give you full flexibility to control, which folders&files are put in the jar file, choose the folder structure within the jar, and choose, which files&folders are placed outside of the jar. However, I prefer to use a higher level tool like ant or maven, since they are opinionated about the structure. This helps me to choose a structure which is close to best practice.

About 2.: ant

The ant tool has the advantage that an ant script can be created using eclipse export.

2015-10-11_204351_capture_004

However, when looking into the resulting ant script, I am missing a possibility to control, whether of not the resources are packed into the jar file (tell me, if you know, how to do it!).

2015.10.19-20_34_27-hc_001

Moreover, each and every dependable jar file is explicitly copied by the ant script. In case I will add a dependency to the POM file later, the ant script needs to be adapted. This is not so optimal. In any case, I have decided to look at the next option: the maven script.

About 3.: maven

On http://www.mkyong.com/maven/how-to-create-a-jar-file-with-maven/ I have found a very good step by step documentation on how to create a executable jar. It was quite easy to apply the documentation on my Apache Camel java program. It also shows how to place the log4j properties file outside of the jar. However, I have more files than the log4j properties file, which I want to make accessible to the administrator.

Executable jar via Maven

Additions to the pom.xml file:

<!-- Make this jar runnable/executable -->
<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <!-- documentation: https://maven.apache.org/plugins/maven-jar-plugin/jar-mojo.html -->
  <artifactId>maven-jar-plugin</artifactId>
  <configuration>
  <!-- custom output directory -->
  <outputDirectory>${project.build.directory}/jarDir</outputDirectory>
  <!-- custom Name of the generated JAR (default: ${project.build.finalName} e.g. de.oveits.provisioningengine_recent-0.5.2.21_stable). Note, that .jar is always appended -->
  <finalName>OpenScapeConnector</finalName>
   <archive>
    <manifest>
     <addClasspath>true</addClasspath>
     <mainClass>de.oveits.provisioningengine.MainApp</mainClass>
     <classpathPrefix>lib/</classpathPrefix>
    </manifest>
   </archive>
  </configuration>
</plugin>

In order to avoid a “fat” jar, i.e. a large jar, which has all dependent jars included, we use the “copy-dependencies” plugin:

<!-- Copy project dependency -->
<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-dependency-plugin</artifactId>
 <version>2.5.1</version>
 <executions>
  <execution>
   <id>copy-dependencies</id>
   <phase>package</phase>
   <goals>
    <goal>copy-dependencies</goal>
   </goals>
   <configuration>
    <!-- exclude junit, we need runtime dependency only -->
    <includeScope>runtime</includeScope>
    <!-- custom output directory -->
    <outputDirectory>${project.build.directory}/jarDir/lib/</outputDirectory> 
  </configuration> 
</execution> 
</executions> 
</plugin>

Note: Eclipse might complain about the copy-dependencies plugin not being compatible with m2e of the eclipse IDE. In this case, you can either accept one of the offered solutions, or you can manually add the following code to the POM file:

<pluginManagement>
 <plugins>
 <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
  <plugin>
  <groupId>org.eclipse.m2e</groupId>
  <artifactId>lifecycle-mapping</artifactId>
  <version>1.0.0</version>
  <configuration>
  <lifecycleMappingMetadata>
  <pluginExecutions>
  <pluginExecution>
  <pluginExecutionFilter>
  <groupId>
  org.apache.maven.plugins
  </groupId>
  <artifactId>
   maven-dependency-plugin
  </artifactId>
  <versionRange>
  [2.5.1,)
  </versionRange>
  <goals>
  <goal>copy-dependencies</goal>
  </goals>
  </pluginExecutionFilter>
  <action>
  <ignore></ignore>
  </action>
  </pluginExecution>
  </pluginExecutions>
  </lifecycleMappingMetadata>
  </configuration>
  </plugin>
 </plugins>
</pluginManagement>

With this, the plugin is ignored with m2e, therefore avoiding the conflict.

Then, we can create the jar file by:

mvn package -Dmaven.test.skip=true

In my case, I have skipped the JUnit tests by specifying the option “-Dmaven.test.skip=true”, because I have not implemented simulator/mock services yet and the test with the real target systems take >~30 minutes.

This step will create a jar file in the <outputDirectory> as defined in the POM file (here: target/jarDir) with the name <finalName> + jar  (here: “OpenScapeConnector.jar”).

The dependent jars are put in <outputDirectory> of the maven-dependency-plugin. The <classpathPrefix> must be the relative path from the jar file to the dependent libraries with trailing “/” (here: lib/). Otherwise, the dependent jars will not be found.

2015.10.19-17_49_08-hc_001

The program then can be started with:

java -jar target/jarDir/OpenScapeConnector.jar

Here, OpenScapeConnector.jar is the <finalName> and target/jarDir is the <outputDirectory> as defined in the POM.

Note that I have kept the log4j properties in the jar file for now, which is a small deviation from the description on Mkyong’s Post. I will take care of this later below.

Extract an remove Configuration Folders from the jar File

We have created an executable jar, but this jar still contains configuration files which are not accessible easily by the operators/administrators. In my case, “cfg” and “properties” contain configuration files, “templates” contains many velocity templates and the “wiretap” folder is the destination folder for traces. All of those folders need to be easily accessible by operations folks:

2015-10-19_180722_capture_001

I have not found any cool maven plugin that could be of any help with this task (apart from excluding the files in the jar and copy them per Linux cp commands to the target folder), so I had decided to create a jar first, and then use standard jar and zip commands to extract and delete the configuration files from the jar:

Moving out and deleting of the folders/files can be done via shell script like follows:

# extract the files and directories myFile1 myFile2 myDir1 myDir2 myDir3 from the jar:
extractList="myFile1 myFile2 myDir1 myDir2 myDir3"
jar xvf myJarFile.jar $extractList

# find and remove the files and folders from the jar. 
find $extractList -exec zip -d $jarFile {} \;

The find command recursively finds all extracted directories. The zip delete command can be used to delete the file/directory from the jar file, since jar files have the same format as zip files.

Now let us try to start the jar file:

java -jar target/jarDir/OpenScapeConnector.jar
...
Exception in thread "main" org.apache.camel.RuntimeCamelException: org.apache.camel.FailedTo CreateRouteException:...not found in classpath

The jar file does not work. What happened? Now, that we have moved some folders out of the jar file, java does not know, where to find them. All files and folders are located outside of the jar in the same directory as the jar file, so we have to add the current directory “.” to the classpath in the Manifest file.

First, I have tried to solve that issue via maven-jar-plugin by adding

<additionalClasspathElements><additionalClasspathElement>.</additionalClasspathElement></additionalClasspathElements>

to the POM file. However, this element does not seem to work within this maven-jar-plugin: it had no effect on the resulting Manifest file (did you get it work?). Therefore, I have decided to use low level manipulations via jar, awk and zip commands instead. This is not very complex either: we need to

  1. extract file from jar using jar command
  2. manipulate file using awk and gsub
  3. update file in jar using zip command

like follows:

#!/bin/sh
jarDir=target/jarDir
jarFile=OpenScapeConnector.jar

cd $jarDir

# 1) extract MANIFEST.MF file:
jar xvf $jarFile META-INF/MANIFEST.MF

# 2) fix MANIFEST.MF: 
awk '{gsub(/Class-Path: /, "Class-Path: . "); print $0}' META-INF/MANIFEST.MF > MANIFEST.MF.temp; mv MANIFEST.MF.temp META-INF/MANIFEST.MF 

# 3) update MANIFEST.MF in jar file: 
zip -u OpenScapeConnector.jar META-INF/MANIFEST.MF

With the jar xvf command, we have extracted the Manifest file from the jar file. With the awk command, we have added the current directory ‘.’ as she first entry to of the Class-Path:

2015.10.19-19_07_34-hc_001

The zip command has updated the jar file with the manipulated Manifest file.

Now we try again to start the application:

java -jar target/jarDir/OpenScapeConnector.jar
...
6140 [main] INFO org.apache.camel.spring.SpringCamelContext - Total 94 routes, of which 94 is started.
[ main] SpringCamelContext INFO Apache Camel 2.12.2 (CamelContext: camel-1) started in 2.781 seconds
6141 [main] INFO org.apache.camel.spring.SpringCamelContext - Apache Camel 2.12.2 (CamelContext: camel-1) started in 2.781 seconds

Bingo! The java program has started with no problems.

Verification

Now let us test, that a change in one of the extracted files has the desired effect. E.g. we can change the log4j settings:

vi target/jarDir/log4j.properties

and change

log4j.logger.org.apache.camel=INFO

to

log4j.logger.org.apache.camel=DEBUG

Now at the startup of the java application, many DEBUGs are shown:

ent.event.EventComponent@5b8dbd69
[ main] DefaultManagementAgent DEBUG ...
[ main] SpringCamelContext INFO Total 94 routes, of which 94 is started.
9322 [main] INFO org.apache.camel.spring.SpringCamelContext - Total 94 routes, of which 94 is started.
[ main] SpringCamelContext INFO Apache Camel 2.12.2 (CamelContext: camel-1) started in 5.905 seconds

and the startup of the Apache Camel routes take twice as long. This is expected and shows that the administrator now can change the configuration files, and after a restart of the program, the changes are active. No re-creation of the jar file is necessary.

The same holds, if I manipulate one of the configuration files. In my web application, the IP address of the target is saved in cfg/WebPortal.cfg; i.e. one of the files, I have extracted from the jar file. Before changing the file, I see on the Web Portal page:

2015.10.19-19_51_10-hc_001

In order to verify that the configuration files in the cfg folder outside of the jar file are “active”, I have changed the OSVIP variable from 10.152.0.10 to 1.1.1.1 in cfg/WebPortal.cfg:

2015.10.19-20_01_00-hc_001

After a restart of the java application and a reload of the page, we see that the web portal content has changed:

2015.10.19-19_56_23-hc_001.

The Web Portal content has been adapted without the need to re-compile the jar file: Bingo! Our goal has been achieved.

Summary

We have shown how to automate the creation of an executable jar file using maven. In order to create a “lean” jar, the dependent jars were extracted and placed outside the jar using a maven plugin. In order to give administrators the chance to change configuration files without re-compiling the jar from source, we have extracted those files using jar and zip commands. Finally, the search path (classpath) had to be manipulated within the jar file, using jar, awk and zip commands.

2015.10.20-14_52_48-hc_001

Note that I am still left with the manual task of the test JUnit jar creation. The maven-jar-plugin does not seem to be designed to be usable for both, the main application and its corresponding JUnit test application, see https://maven.apache.org/plugins/maven-jar-plugin/examples/create-test-jar.html for details. The “easy way” described there did not work for me (“cannot find main class”, when I try to start the jar). So, either I need to find, what I am doing wrong, or I need to follow the “preferred way”, i.e. to move the test code to a separate project with its own POM file and apply the same procedure to the test main app as I did for the main application.