0

JHipster: Exploring and Improving the Spring Boot REST API


In our last blog post, we have created a simple blog application using JHipster, which is a Yeoman based code generator for Angular and Spring Boot. This time, we will explore and improve the Spring Boot REST API that is generated by JHipster automatically.

In the last blog, we had created following model; a simple blog:

In this blog post, we will concentrate on the “Blog” objects. We will try to create, read, update and delete such objects. We will find out that any user has all rights to perform those function, and we will make sure that only the blog owner can update and delete the objects.

Step 1: Consulting Swagger

A good thing about JHipster is, that it not only creates a Spring Boot application with the desired objects, but it also auto-generates a swagger documentation page about the API. This will help us find out how to handle the API.

Another good thing about JHipster is, that the REST API is secure by default in the sense that only logged in users will receive positive answers.

If we try to retrieve a blog (http://localhost:9000/api/blogs/4 in this case), we will get the answer, that we are not authorized:

{
    "timestamp": "2017-08-23T19:41:41.175+0000",
    "status": 401,
    "error": "Unauthorized",
    "message": "Access Denied",
    "path": "/api/blogs/4"
}

That is good in one sense, but we need to find out how to authenticate, before we can perform anything on the API. For that, we consult the swagger API documentation on http://localhost:/9000/#/docs. We find an entry for the user-jwt-controller and it tells us that we

According to the documentation, we need to POST a body of the format

{
  "password": "user",
  "rememberMe": true,
  "username": "user"
}

According to swagger, this has to be sent to the URL http://localhost:9060/api/authenticate. However, this did not work in my case. In my case, I had to send the POST to http://localhost:9000/api/authenticate instead. This might be a reason, why swagger’s Try it out! Button did not work.  I got an error telling me that there was no response from the server:

Also the curl command had to be reworked a little bit: the port had to be changed from 9060 to 9000 and the Backslashes had to be removed. But then it worked fine:

(anyhost)$ curl -X POST --header 'Content-Type: application/json' --header 'Accept: */*'  -d '{
   "password": "user",
   "rememberMe": true,
   "username": "user"
 }' 'http://localhost:9000/api/authenticate'
{
  "id_token" : "eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJ1c2VyIiwiYXV0aCI6IlJPTEVfVVNFUiIsImV4cCI6MTUwNTkxOTg3NH0.n4B1RrmokbjuSQqWjOxNcUWgUXoqZODLb-nzEN-Km-7zjx7sGElWL8xgSWhc3DMrTHQDauT81ZZKF4IrmNh71A"
}

The curl command can be issued from the container or the docker host. Or it can be issued on the Vagrant host (a Windows 10 machine in my case), that is hosting the docker host VM, if port 9000 is mapped from Vagrant host to the VM:

In the latter case, we can use the graphical POSTman instead of a curl command, if we wish to:

Step 2: Create/Get Tokens

In the previous step, swagger has helped us to retrieve a token. Let us now write the result into an environment variable like follows:

(anyhost)$ USER_TOKEN=$(curl -k -D - -X POST --header 'Content-Type: application/json' --header 'Accept: */*' -d '{ "password": "user", "rememberMe": true, "username": "user" }' 'http://localhost:9000/api/authenticate' | grep id_token | awk -F '["]' '{print $4;}')
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
100 258 0 196 100 62 164 51 0:00:01 0:00:01 --:--:-- 164
(anyhost)$ echo $USER_TOKEN
eyJhbGciOiJIUzUxMiJ9.eyJzdWIiOiJ1c2VyIiwiYXV0aCI6IlJPTEVfVVNFUiIsImV4cCI6MTUwNjExMDU5NX0.MozFvp72L23PyAPsHg2tLfaxmqIRQXhT0DGrlRwAZDXISaceANqJIeOkaXbZXwDNPGW-3H_n3bzAwitvCeZE8g
Similarily, we can write the admin's token into a different environment variable:
(anyhost)$ ADMIN_TOKEN=$(curl -k -D - -X POST --header 'Content-Type: application/json' --header 'Accept: */*' -d '{ "password": "admin", "rememberMe": true, "username": "admin" }' 'http://localhost:9000/api/authenticate' | grep id_token | awk -F '["]' '{print $4;}')
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   276    0   212  100    64   1532    462 --:--:-- --:--:-- --:--:--  1536

Step 3: Create a Blog

Step 3.1: Explore createBlog on Swagger

Now let us use the token to create a blog. We consult swagger again. we find a createBlog function on

POST /api/blogs

with the example body

{
  "handle": "string",
  "id": 0,
  "name": "string",
  "user": {
    "activated": true,
    "email": "string",
    "firstName": "string",
    "id": 0,
    "imageUrl": "string",
    "langKey": "string",
    "lastName": "string",
    "login": "string",
    "resetDate": "2017-08-21T13:49:44.421Z"
  }
}

However, it does not make sense to specify all user parameters in the request. The user is not a blog private variable, it is only a pointer to an existing user. And the user is fully specified by the user ID (ID=4 for user named “user”). Therefore, let us try to use a minimized version as follows:

{
  "handle": "usersBlogCreatedByApi",
  "name": "user's blog created by API",
  "user": {
    "id": 4
  }
}

We also have omitted the blog id 0, since the ID will be assigned automatically.

Step 3.2: Find the user ID

How did I know that the user named “user” has the ID 4? I did not. I just issued the following curl command

(anyhost)$ curl -X GET --header 'Accept: */*' --header "Authorization: Bearer $USER_TOKEN" 'http://localhost:9000/api/users'

[ {
  "id" : 1,
  "login" : "system",
  "firstName" : "System",
  "lastName" : "System",
  "email" : "system@localhost",
  "imageUrl" : "",
  "activated" : true,
  "langKey" : "en",
  "createdBy" : "system",
  "createdDate" : "2017-08-15T11:47:06.180Z",
  "lastModifiedBy" : "system",
  "lastModifiedDate" : null,
  "authorities" : [ "ROLE_USER", "ROLE_ADMIN" ]
}, {
  "id" : 3,
  "login" : "admin",
  "firstName" : "Administrator",
  "lastName" : "Administrator",
  "email" : "admin@localhost",
  "imageUrl" : "",
  "activated" : true,
  "langKey" : "en",
  "createdBy" : "system",
  "createdDate" : "2017-08-15T11:47:06.180Z",
  "lastModifiedBy" : "system",
  "lastModifiedDate" : null,
  "authorities" : [ "ROLE_USER", "ROLE_ADMIN" ]
}, {
  "id" : 4,
  "login" : "user",
  "firstName" : "User",
  "lastName" : "User",
  "email" : "user@localhost",
  "imageUrl" : "",
  "activated" : true,
  "langKey" : "en",
  "createdBy" : "system",
  "createdDate" : "2017-08-15T11:47:06.180Z",
  "lastModifiedBy" : "system",
  "lastModifiedDate" : null,
  "authorities" : [ "ROLE_USER" ]
} ]

And there it is: the user named “user” has the ID=4.

Step 3.3: Create the Blog

Now let us create the blog:

(anyhost)$ curl -D - -X POST --header 'Content-Type: application/json' --header 'Accept: */*' --header "Authorization: Bearer $USER_TOKEN" -d '{
    "handle": "users_blog_via_api_using_user_token",
    "name": "user'\''s blog created by the API using the user'\''s token",
    "user": {
      "id": 4
    }
  }' 'http://localhost:9000/api/blogs'

HTTP/1.1 201 Created
x-powered-by: Express
x-blogapp-alert: blogApp.blog.created
expires: 0
cache-control: no-cache, no-store, max-age=0, must-revalidate
x-xss-protection: 1; mode=block
pragma: no-cache
location: /api/blogs/19
date: Wed, 23 Aug 2017 20:25:11 GMT
connection: close
x-content-type-options: nosniff
content-type: application/json;charset=UTF-8
x-application-context: blog:swagger,dev:8080
x-blogapp-params: 19
transfer-encoding: chunked

{
  "id" : 19,
  "name" : "user's blog created by the API using the user's token",
  "handle" : "users_blog_via_api_using_user_token",
  "user" : {
    "id" : 4,
    "login" : null,
    "firstName" : null,
    "lastName" : null,
    "email" : null,
    "activated" : false,
    "langKey" : null,
    "imageUrl" : null,
    "resetDate" : null
  }
}

Note that I have escaped the single quotes by ending the quote with a ', then write an escaped quote \' and start the quote again with a single '. Together a single quote within single quotes is escaped as '\''.

The blog can be seen on the JHipster UI on http://localhost:9000/#/blog:

Step 4: Update the Blog

Step 4.1: Consult Swagger

Before we delete the blog, let us update the blog. For that, let us consult swagger again:

The updateBlog function is similar to the createBlog function. However, this time we will not omit the blog ID, since this is the way to tell the API, which entity is to be updated. As with the createBlog function, we can omit all user’s variables apart from the user ID. This way, we can move the blog to the ownership of the admin, if we wish. Let us do that:

curl -D - -X PUT --header 'Content-Type: application/json' --header 'Accept: */*' --header "Authorization: Bearer $USER_TOKEN" -d '{ 
 "id": 19,
 "handle": "users_blog_via_api_using_user_token", 
 "name": "modified user'\''s blog created by the API using the user'\''s token and assigned to the admin now", 
 "user": { "id": 3 } 
}' 'http://localhost:9000/api/blogs'


HTTP/1.1 200 OK
x-powered-by: Express
x-blogapp-alert: blogApp.blog.updated
expires: 0
cache-control: no-cache, no-store, max-age=0, must-revalidate
x-xss-protection: 1; mode=block
pragma: no-cache
date: Wed, 23 Aug 2017 20:43:46 GMT
connection: close
x-content-type-options: nosniff
content-type: application/json;charset=UTF-8
x-application-context: blog:swagger,dev:8080
x-blogapp-params: 19
transfer-encoding: chunked

{
  "id" : 19,
  "name" : "modified user's blog created by the API using the user's token and assigned to the admin now",
  "handle" : "users_blog_via_api_using_user_token",
  "user" : {
    "id" : 3,
    "login" : "admin",
    "firstName" : "Administrator",
    "lastName" : "Administrator",
    "email" : "admin@localhost",
    "activated" : true,
    "langKey" : "en",
    "imageUrl" : "",
    "resetDate" : null
  }
}

With this request, we have assinged the blog to the admin and we have changed the name of the blog.

Note that you most probably need to change the blog ID of the request: use the ID you have received in the createBlog response.

Note that it is not possible to change only the name of the blog without specifying the handle or the user. The handle is mandatory also for PUT (Update) requests, and the user is cleared, if it is omitted.

Step 5: Delete the Blog

Now that we have created and updated a blog, let us delete the blog. We will see that the user can delete a blog that is owned by the admin. This is something we will improve later on.

Step 5.1: Consult Swagger

Again, let us have a look to Swagger on http://localhost:9000/#/docs:

Step 5.2: Delete Blog

For deletion of blog with ID=19, we just need to send a HTTP DELETE to /api/blogs/19:

(anyhost)$ curl -D - -X DELETE --header 'Content-Type: application/json' --header 'Accept: */*' --header "Authorization: Bearer $USER_TOKEN" 'http://localhost:9000/api/blogs/19'
HTTP/1.1 200 OK
x-powered-by: Express
x-blogapp-alert: blogApp.blog.deleted
expires: 0
cache-control: no-cache, no-store, max-age=0, must-revalidate
x-xss-protection: 1; mode=block
pragma: no-cache
date: Wed, 23 Aug 2017 20:53:21 GMT
connection: close
x-content-type-options: nosniff
content-length: 0
x-application-context: blog:swagger,dev:8080
x-blogapp-params: 19

We receive a 200 OK message. Yes, the blog was destroyed as expected, as can be verified on the UI again.

However, in a real-world scenario, the admin would be pissed, if a normal user was allowed to delete a blog owned by the admin, I guess. Let us improve the situation.

Step 6: Improving the API

Step 6.1: Improve the Answer when trying to delete an non-existing entity

Before improving the API, let us, see, the answer of the system, if we try to delete a blog that does not exist anymore:

(anyhost)$ curl -D - -X DELETE --header 'Content-Type: application/json' --header 'Accept: */*' --header "Authorization: Bearer $USER_TOKEN" 'http://localhost:9000/api/blogs/19'

HTTP/1.1 500 Internal Server Error
x-powered-by: Express
expires: 0
cache-control: no-cache, no-store, max-age=0, must-revalidate
x-xss-protection: 1; mode=block
pragma: no-cache
date: Mon, 21 Aug 2017 18:27:13 GMT
connection: close
x-content-type-options: nosniff
content-type: application/json;charset=UTF-8
x-application-context: blog:swagger,dev:8080
transfer-encoding: chunked

{
  "message" : "error.internalServerError",
  "description" : "Internal server error",
  "fieldErrors" : null
}

A HTTP 500 Server Error! Not good. It seems like the JHipster implementation is sub-optimal. We should receive a 404 Not Found Error instead.

A clue for the reason can be seen in the log of the server:

2017-08-21 18:27:13.426 DEBUG 4253 --- [  XNIO-2 task-8] org.jhipster.web.rest.BlogResource       : REST request to delete Blog : 9
Hibernate: select blog0_.id as id1_0_0_, blog0_.handle as handle2_0_0_, blog0_.name as name3_0_0_, blog0_.user_id as user_id4_0_0_, user1_.id as id1_6_1_, user1_.created_by as created_2_6_1_, user1_.created_date as created_3_6_1_, user1_.last_modified_by as last_mod4_6_1_, user1_.last_modified_date as last_mod5_6_1_, user1_.activated as activate6_6_1_, user1_.activation_key as activati7_6_1_, user1_.email as email8_6_1_, user1_.first_name as first_na9_6_1_, user1_.image_url as image_u10_6_1_, user1_.lang_key as lang_ke11_6_1_, user1_.last_name as last_na12_6_1_, user1_.login as login13_6_1_, user1_.password_hash as passwor14_6_1_, user1_.reset_date as reset_d15_6_1_, user1_.reset_key as reset_k16_6_1_ from blog blog0_ left outer join jhi_user user1_ on blog0_.user_id=user1_.id where blog0_.id=?
2017-08-21 18:27:13.431 ERROR 4253 --- [  XNIO-2 task-8] org.jhipster.aop.logging.LoggingAspect   : Exception in org.jhipster.web.rest.BlogResource.deleteBlog() with cause = 'NULL' and exception = 'No class org.jhipster.domain.Blog entity with id 9 exists!'

org.springframework.dao.EmptyResultDataAccessException: No class org.jhipster.domain.Blog entity with id 9 exists!

If we peek into org.jhipster.web.rest.BlogResource, the reason becomes clear:

@DeleteMapping("/blogs/{id}")
    @Timed
    public ResponseEntity deleteBlog(@PathVariable Long id) {
        log.debug("REST request to delete Blog : {}", id);
        blogRepository.delete(id); 
        return ResponseEntity.ok().headers(HeaderUtil.createEntityDeletionAlert(ENTITY_NAME, id.toString())).build();      
    }

The deleteBlog function calls blogRepository.delete(id) without checking, whether a blog with the corresponding id exists. Let us improve the situation:

    @DeleteMapping("/blogs/{id}")
    @Timed
    public ResponseEntity deleteBlog(@PathVariable Long id) {
        log.debug("REST request to delete Blog : {}", id);
        
        Blog blog = blogRepository.findOne(id);
      
        if(blog != null) { 
        	blogRepository.delete(id); 
        	return ResponseEntity.ok().headers(HeaderUtil.createEntityDeletionAlert(ENTITY_NAME, id.toString())).build();
        } else {
        	return ResponseEntity.notFound().headers(HeaderUtil.createEntityDeletionAlert(ENTITY_NAME, id.toString())).build();
        }
        
        
    }

This has done the job:

(anyhost)$ curl -D - -X DELETE --header 'Content-Type: application/json' --header 'Accept: */*' --header "Authorization: Bearer $USER_TOKEN" 'http://localhost:90
00/api/blogs/9'

HTTP/1.1 404 Not Found
x-powered-by: Express
x-blogapp-alert: blogApp.blog.deleted
expires: 0
cache-control: no-cache, no-store, max-age=0, must-revalidate
x-xss-protection: 1; mode=block
pragma: no-cache
date: Mon, 21 Aug 2017 18:36:03 GMT
connection: close
x-content-type-options: nosniff
content-length: 0
x-application-context: blog:swagger,dev:8080
x-blogapp-params: 9

Let us update swagger correspondingly:

Step 6.2: Make sure only Owner can DELETE an blog object

In the moment, every user that is logged in can delete a blog. Let us make sure only the user of the blog can delete the object.

For that, let us try the following:

    @DeleteMapping("/blogs/{id}")
    @Timed
    public ResponseEntity deleteBlog(@PathVariable Long id) {
        log.debug("REST request to delete Blog : {}", id);
        
        Blog blog = blogRepository.findOne(id);
        
        
        if(blog != null) {
        	if (!blog.getUser().getLogin().equals(SecurityUtils.getCurrentUserLogin())) { 
        		// The user is not allowed to delete this blog, if it is not owned by this user:
            	        log.debug("Found blog, but user is not allowed to delete it");
            	        return ResponseEntity.status(403).headers(HeaderUtil.createEntityDeletionAlert(ENTITY_NAME, id.toString())).build();
        	} else {
	        	blogRepository.delete(id); 
	        	return ResponseEntity.ok().headers(HeaderUtil.createEntityDeletionAlert(ENTITY_NAME, id.toString())).build();
        	}
        } else {
        	return ResponseEntity.notFound().headers(HeaderUtil.createEntityDeletionAlert(ENTITY_NAME, id.toString())).build();
        }
        
        
    }

If we now create a blog for user “admin”:

$ curl -D - -X POST --header 'Content-Type: application/json' --header 'Accept: */*' --header "Authorization: Bearer $ADMIN_TOKEN" -d '{
   "id": 9, "handle": "users_blog_via_api_using_admin_token",
   "name": "modified user'\''s blog created by the API using the admin'\''s token",
   "user": {
     "id": 3
   }
 }' 'http://localhost:9000/api/blogs'

HTTP/1.1 200 OK
x-powered-by: Express
x-blogapp-alert: blogApp.blog.updated
expires: 0
cache-control: no-cache, no-store, max-age=0, must-revalidate
x-xss-protection: 1; mode=block
pragma: no-cache
date: Mon, 21 Aug 2017 19:30:40 GMT
connection: close
x-content-type-options: nosniff
content-type: application/json;charset=UTF-8
x-application-context: blog:swagger,dev:8080
x-blogapp-params: 9
transfer-encoding: chunked

{
  "id" : 18,
  "name" : "modified user's blog created by the API using the admin's token",
  "handle" : "users_blog_via_api_using_admin_token",
  "user" : {
    "id" : 3,
    "login" : "admin",
    "firstName" : "Administrator",
    "lastName" : "Administrator",
    "email" : "admin@localhost",
    "activated" : true,
    "langKey" : "en",
    "imageUrl" : "",
    "resetDate" : null
  }
}

Now we try to delete it with as user “user”:

$ curl -D - -X DELETE --header 'Content-Type: application/json' --header 'Accept: */*' --header "Authorization: Bearer $USER_TOKEN" 'http://localhost:90
00/api/blogs/18'

HTTP/1.1 403 Forbidden
x-powered-by: Express
x-blogapp-alert: blogApp.blog.deleted
expires: 0
cache-control: no-cache, no-store, max-age=0, must-revalidate
x-xss-protection: 1; mode=block
pragma: no-cache
date: Mon, 21 Aug 2017 19:33:20 GMT
connection: close
x-content-type-options: nosniff
content-length: 0
x-application-context: blog:swagger,dev:8080
x-blogapp-params: 18

That works as expected.

Now let us delete it as user admin:

$ curl -D - -X DELETE --header 'Content-Type: application/json' --header 'Accept: */*' --header "Authorization: Bearer $ADMIN_TOKEN" 'http://localhost:9
000/api/blogs/18'

HTTP/1.1 200 OK
x-powered-by: Express
x-blogapp-alert: blogApp.blog.deleted
expires: 0
cache-control: no-cache, no-store, max-age=0, must-revalidate
x-xss-protection: 1; mode=block
pragma: no-cache
date: Mon, 21 Aug 2017 19:39:34 GMT
connection: close
x-content-type-options: nosniff
content-length: 0
x-application-context: blog:swagger,dev:8080
x-blogapp-params: 18

Perfect, this is doing the job.

Excellent! Thump up!

 

Summary

We have explored the API of a Spring Boot application created by JHipster. We have seen how we can use Swagger to find out, how to use the API. We have created, updated and deleted an object. After seeing that the JHipster implementation of the API is sub-optimal, we have improved the API:

  • we have made sure the API does not throw a HTTP 500 Server Error, if someone tries to delete a non-existing entity. Instead, we will return a 404 Not Found Error.
  • We have show, how we can make sure that only the owner can delete an entity.

Next Steps

  • We have (not yet) applied the improvements on other functions like create (automatically assign the object to the logged in user as owner), update.
  • In the moment, only logged-in users can read the entities. We could show, how to make an entity visible to the world instead, i.e. how to allow anonymous access.

 

 

 

0

Hello Java Hipster: Angular 4 and Spring Boot


In this blog post, Java Hipster will help us creating a mini blog application based on Angular 4 and Spring Boot. Angular is a popular framework for creating reactive single page applications, while Spring Boot is a robust java-based backend framework that helps you create database access and RESTful APIs.

We will closely follow the JHipster introduction on YouTube and explore administrative functions like user management, logging and API management (with swagger). After creating an importing a simple model for a blog, we will manipulate the blog and its entries via the GUI.

For your convenience, we will run everything in a Docker container, so you do not need to install any of the required software packages (apart from Docker itself). As a little goody, we will outline how to add some additional security functions: we will restrict delete access to authorized users. That is an extension to the many good examples shown in the JHipster introduction on YouTube video.

Tools and Versions used

  • Vagrant 1.8.6
  • Virtualbox 5.0.20 r106931
  • jhipster Docker image v4.6.2

Why JHipster?

JHipster is a Yeoman-based code generator for a complete Web application based on Angular and Spring Boot, two popular frameworks for front-end and backend systems.

  • Angular is a popular single page frontend framework
  • Spring Boot is a high-performance backend framework

mixed with CSS Bootstrap styles, and an easy-to use web-based modeling tool that creates text-based models that can be used to generate the corresponding database and REST entities/functions, JHipster, JHipster is a quick way to get projects started.

I have not checked out yet all other features mentioned on their home page, like

  • microservice support with JHipster Registry, Netflix OSS, Elastic stack and Docker

We will build our first application within a Docker container.

Step 0: Install a Docker Host

This time (again), our application will need more than 750 MB RAM. Therefore, we cannot use my beloved Katacoda as our Docker playground, which is limited to this amount of DRAM. Instead, you need to get access to a Docker host with, >~ 2 GB RAM. A nice way of installing an Ubuntu Docker host via Vagrant is described here (search for the term “Install a Docker Host”).

Prerequisites of this step:

  • I recommend to have direct access to the Internet: via Firewall, but without HTTP proxy. However, if you cannot get rid of your HTTP proxy, read this blog post.
  • Administration rights on you computer.

Steps to install a Docker Host VirtualBox VM:

Download and install Virtualbox (if the installation fails with error message “Oracle VM Virtualbox x.x.x Setup Wizard ended prematurely” see Appendix A of this blog post: Virtualbox Installation Workaround below)

1. Download and Install Vagrant (requires a reboot)

2. Download Vagrant Box containing an Ubuntu-based Docker Host and create a VirtualBox VM like follows:

(basesystem)# mkdir ubuntu-trusty64-docker ; cd ubuntu-trusty64-docker
(basesystem)# vagrant init williamyeh/ubuntu-trusty64-docker
(basesystem)# vagrant up
(basesystem)# vagrant ssh

Now you are logged into the Docker host and we are ready for the next step: to download the Docker image and to start the application in a container.

Step 1: Start JHipster Docker Container

(dockerhost)$ docker run -it -p 8080:8080 -p 3001:3001 -p 9000:9000 -p 9060:9060 -v $(pwd):/localdir jhipster/jhipster:v4.6.2 bash
(containter)# cd /localdir

Note: You also can omit the version tag (:v4.6.2) and work with the latest image. However, v4.6.2 is the version tested in this blog post.

Note that you might have to view (docker ps) and stop (docker stop) any container that is running on port 8080 first, since I have experienced problems, when mapping the port 8080 to another port than 8080 (I need to try again). If you have followed step 0, you will need to issue the command

(dockerhost) sudo docker stop cadvisor

Step 2: Create, Start and Login to App

Step 2.1: Create App

(container)# mkdir blog; cd blog
(container)# yo jhipster

I have chosen

  • monolithic
  • base name: blog
  • default java package name: org.jhipster <– should use another name, probably
  • authentication: JWT
  • database type: SQL
  • production database: MySQL
  • development database: H2 with disk-based persistence
  • Hibernate 2nd level cache: Yes, with ehcache (local cache, for a single node)
  • Other Technologies (choose none, just press enter)
  • Framework: Angular 2
  • libSass: Yes
  • Internationalization: Yes
  • Chosen English and German
  • Testing Frameworks (bBesides JUnit and Karma:
    • Gatling
    • Protractor

Wait for ~2-4 minutes

Step 2.2 Start App

In order to be able to explore an (empty) JHipster app, we can start the application as follows:

Start Spring Boot Application:

In the project root directory, type:

(container)# ./mvnw

Only as a reference: Start Webpack development server (was not needed in my case!)

container)# yarn start

Step 2.3 (optional): Connect to (default) App

In a local browser, connect to localhost:8080:

Sign in as admin with password “admin”.

Step 2.4 (optional): Explore (default) App

The Entities are empty yet, but unser Administration we can see the user management, metrics etc:

We have not yet created and imported a model, so this is still the default JHipster app. It comes with a User Model, though, with three users created per default:

  • admin / admin
  • user / user
  • system / system?

Under Administration -> Metrics, we can see JVM Metrics as well as HTTP statistics:

Under Administration -> Configuration, we see the Spring properties, even though they seem to be read-only:

However, the log configuration can be edited in the way that a log can be chosen e.g. to show DEBUG instead of WARN messages only

Under Administration -> API we find a swagger page that helps you to test your REST services:

Okay, I am cheating a little here: you will not be able to see the blog-resource yet, since we will create it not before the next two steps. You will only see the account, the profile info, the user-jwt controller and the user resource. It can be used to test the interface. E.g. you can read all users by clicking the button

Try it out!

of users-resource GET /api/users and we will receive something like follows:

[
  {
    "id": 1,
    "login": "system",
    "firstName": "System",
    "lastName": "System",
    "email": "system@localhost",
    "imageUrl": "",
    "activated": true,
    "langKey": "en",
    "createdBy": "system",
    "createdDate": "2017-08-15T11:47:06.180Z",
    "lastModifiedBy": "system",
    "lastModifiedDate": null,
    "authorities": [
      "ROLE_USER",
      "ROLE_ADMIN"
    ]
  },
  {
    "id": 3,
    "login": "admin",
    "firstName": "Administrator",
    "lastName": "Administrator",
    "email": "admin@localhost",
    "imageUrl": "",
    "activated": true,
    "langKey": "en",
    "createdBy": "system",
    "createdDate": "2017-08-15T11:47:06.180Z",
    "lastModifiedBy": "system",
    "lastModifiedDate": null,
    "authorities": [
      "ROLE_USER",
      "ROLE_ADMIN"
    ]
  },
  {
    "id": 4,
    "login": "user",
    "firstName": "User",
    "lastName": "User",
    "email": "user@localhost",
    "imageUrl": "",
    "activated": true,
    "langKey": "en",
    "createdBy": "system",
    "createdDate": "2017-08-15T11:47:06.180Z",
    "lastModifiedBy": "system",
    "lastModifiedDate": null,
    "authorities": [
      "ROLE_USER"
    ]
  }
]

We also can choose the German language, since we have chosen English AND German:

Step 3: Create and Download Your Model

You can create and download the model from JDL Studio and access it from your application. If you have followed the instructions above, you need to make sure the file is visible from the JHipster container. For that, you need to place it to the folder you have mapped in step 1 by using the -v docker run flag.

It might be more convenient to cut&paste the content into a file /localdir/blog/model.jh within the container:

entity Blog {
name String required minlength(3),
handle String required minlength(2)
}

entity Entry {
title String required,
content TextBlob required,
date ZonedDateTime required
}

entity Tag {
name String required minlength(2)
}

relationship ManyToOne {
Blog{user(login)} to User,
Entry{blog(name)} to Blog
}

relationship ManyToMany {
Entry{tag(name)} to Tag{entry}
}

paginate Entry, Tag with infinite-scroll

The graphical representation of the model looks like follows:

The modes is re-using the User model that is present in JHipster per default. Now, each blog is mapped to a user. Each blog can have many blog entries and each entry can have many tags, while each tag can be attached to many entries.

In the next step, we will import this model.

Step 4: Import Model

To import the model, we stop the currently running mvnh process (if so), and import the model as follows:

(container)# yo jhispter:import-jdl ./model.jh
Overwrite liquibase/master.xml?: a

Step 5: Start Spring Boot App

After 5 minutes or so, we can start the application again:

(container)# ./mvnw

Note: yarn start alone will not do the trick. See https://stackoverflow.com/questions/43291477/jhipster-cannot-login-after-starting-with-yarn-start-webpack

Step 6: Connect and Login as admin

In a Browser, we need to navigate to localhost:8080 and login as admin with password admin or user with password user. It does not make a difference for now.

We will notice, that now the model entities are available:

Step 7: Create a Blog

Let us choose “Blog” and Create a new Blog:

Click Save.

Step 8: Create a Blog Entry

Now let us Create a new Entry on Entities -> Entry:

We can see that the Blog field is not mandatory (otherwise the mouse will show that the Save button cannot be pressed). However, let us choose the Admin’s Blog we have just created:

Note, that I have put only part of the content in headline style.

 

The ID is 2 instead of 1, since I have played around already and I have created and deleted a blog entry already.

Step 9: Add Multi User Support

In this step we will make sure that a user can only edit and delete any entity that belongs to the logged in user.

Step 9.1: Log out and log in as User “user”

We now log out and log in as user “user” with password “user”. You will notice that you can see and edit or delete any entity that belongs to the admin. This is a no-go for any multi-user application and needs to be fixed.

Step 9.2: Add a User’s Blog

Let us first create a User’s Blog and a Blog entity like follows:

We will see both Blog entries:

(since you have not yet fixed the HTML Display, the content will look different in your case, you will see the escaped HTML instead like

However the Date is missing. Oups, when checking with my screenshot above, I have entered a wrong date far in the future. And there is a bad side-effect: when I try to edit or delete the entry, the application does not react.

For the records:

  • we need to make sure that the user input is verified
  • we need to find a way to delete this entry

For now, let us create a new entry:

This is an entry that can be edited and deleted, if needed.

Note: It seems to take quite long, until the entry is loaded. Also, it will be “Loading…” forever, if you click on View of the upper entry and use the “back” button on that page. Maybe this is the case because of the bad entry #2?

If you are reloading the page, it does display, though and the new entry can be edited, if needed:

Step 9.3: Test write access to admin’s entries

Now click Edit of the admin’s entry and you will see that the user is allowed to edit admin’s entries.

We need to take two measures:

  • the Edit and Delete buttons should not be visible (or grayed out and not functional) on entries that do not belong to to the logged in user.
  • the user should not be able to circumvent security by directly calling the Edit or Delete function for entities of other users.

Step 9.4: Start Application in develop Mode

In order to view the results of any file changes immediately, you need to issue the command

(container)# yarn start

and connect to port 9000 instead of 8080:

Log in as user “user” again.

Step 9.5: Remove foreign Blogs from Blog Table View

For now, as user named “user” we can see both blogs we had created:

A quick&dirty way to remove the Edit and Delete buttons from the user’s view is to remove the admin’s blogs from the view, as shown in the JHipster introduction on YouTube:

Let us search for the function getAllBlogs in the java File BlogResource (found on the container as src/main/java/org/jhipster/web/rest/BlogResource.java)

and we change

findAll()

by

findByUserIsCurrentUser()

which is available on JHipster on any model per default.

After restarting ./mvnw, the admin’s blog will be removed from the blog table:

Step 9.6: Restrict Access for Single Blog View

In the previous step, we have removed the admin’s blog from the user’s blog table view. However, the admins’s blog can still be accessed by the user named “user”, if he knows (or guesses) the ID:

This is a topic that is not mentioned in the JHipster introduction on YouTube. So, let that fix too:

On java/org/jhipster/repository/BlogRepository.java, we can reset the return value to “null”, if the found blog does not belong to the logged in user. For that we add:

// file: java/org/jhipster/repository/BlogRepository.java
import import org.jhipster.security.SecurityUtils;
...
    @GetMapping("/blogs/{id}")
    @Timed
    public ResponseEntity getBlog(@PathVariable Long id) {
        log.debug("REST request to get Blog : {}", id);
        Blog blog = blogRepository.findOne(id);
        
        // The user is not allowed to access this blog, if it is not owned by this user:
        if (!blog.getUser().getLogin().equals(SecurityUtils.getCurrentUserLogin())) { blog = null; }

        return ResponseUtil.wrapOrNotFound(Optional.ofNullable(blog));
    }

For that we import the SecurityUtils and compare the user login of the found blog with the user login of the user that is logged in. If it is not equal, we just reset the found value to null.

With that, the blog is not accessible anymore, after we have restarted ./mvnw:

Excellent! Thump up!

This is, what we want to achieve: blog/3 is not visible because it is not owned by the logged in user.

However, let us double-check, that general access to the blogs is not broken. For that, let us access the user’s blog, /blog/4 in my case:

Yes, that’s it.

Note: since we have not done anything on the DELETE function, any user, who knows how to access the API, will still be able to DELETE or UPDATE the entry. This will be covered in the next blog post, where we will have a closer look to the API created by Spring Boot. To cut is short, following code change (in red) will be shown there:

    @DeleteMapping("/blogs/{id}")
    @Timed
    public ResponseEntity deleteBlog(@PathVariable Long id) {
        log.debug("REST request to delete Blog : {}", id);
        
        Blog blog = blogRepository.findOne(id);     
        
        if(blog != null) {
        	if (!blog.getUser().getLogin().equals(SecurityUtils.getCurrentUserLogin())) { 
        		// The user is not allowed to delete this blog, if it is not owned by this user:
            	        log.debug("Found blog, but user is not allowed to delete it");
            	        return ResponseEntity.status(403).headers(HeaderUtil.createEntityDeletionAlert(ENTITY_NAME, id.toString())).build();
        	} else {
	        	blogRepository.delete(id); 
	        	return ResponseEntity.ok().headers(HeaderUtil.createEntityDeletionAlert(ENTITY_NAME, id.toString())).build();
        	}
        } else {
        	return ResponseEntity.notFound().headers(HeaderUtil.createEntityDeletionAlert(ENTITY_NAME, id.toString())).build();
        }
        
        
    }

Note: A similar code change is needed for the PUT (Update) function.

Step 10: Fix the HTML Display

We can see on Entities -> Entry -> View that the HTML display is not correct, since it is escaped:

Let us fix that now. For that, navigate to src/main/webapp/app/entities/entry/entry-detail.component.html (you can do that inside the container via vi, or, since the content is mapped to the Docker host, we also can use an IDE like Visual Studio Code from outside of the container:


There, we can change

<span>{{entry.content}}</span>

by

<span [innerHTML]="entry.content">Loading...</span>

Unlike what is shown in the JHipster introduction, the application running on localhost:8080 does not seem to recognize, when a file is changed. Even stopping and restarting mvnw did not change anything:

Instead, I have run

(container)# yarn start

within the container and I have connected to the Webpack in development mode on port 9000:

This did the trick.

Excellent! Thump up!

Now, I can change the files from within the container (by opening an extra session into the Docker container via docker exec -it <containerid> bash, and webpack will re-transpile the code within seconds (but I need to refresh the Browser, it seems).

Note: after running the application some days on my notebook, I have logged in as admin again (without reloading the application) and the change now was visible on port 8080 as well:

I guess, this problem needs some more investigation…

 

Summary

In this blog post, we have created a little blog application using the Java Hipster code generator. For that, we have

  • installed a Docker host, if needed
  • started a Docker JHipster container on the Docker host
  • run and explored the default administration functions of the JHipster application like
    • user management,
    • API exploration (via swagger) and
    • logging
  • edited a model for our application using the JDL Studio Web Page
  • imported the model file of a blog application (blogs, entries, tags) into JHipster
  • explored the new functions of the application
  • tweaked the Spring Boot Read and Delete function, so that only the owner is able to see and delete a blog (this is an addition to what you will find on the JHipster introduction on YouTube.)
  • tweaked the view of the blog within an Angular template to display HTML content correctly

We have seen, how easy it is to import an arbitrary model into the default JHipster application and to create and display the entities defined in the model. In our case, we have created a simple blog application with blogs, entries and tags.

Coming Soon

  • I already have started a Blog Post, where I am exploring the REST API that is automatically generated by JHipster. We will learn, how to use swagger to find the correct curl commands, how to authenticate the service and how to tweak the REST interface, so  only the owner of a blog entity is allowed to delete the entry. Follow this blog, if you are interested in the blog post.
2

Jenkins Part 4.1: Functional Java Tests via JUnit


2016-11-30-18_19_38

You also think that functional tests are one of the most important ingredients for delivering high quality software? You share my opinion that we should help the developer automating this task in order to get comparable results and to receive meaningful trend reports?

I will cover functional tests here. Instructions on how to perform code quality tests and performance tests are in draft status and will be covered in the next two blog posts.

Any questions and/or comments are highly welcome.

Introduction

As a developer you try hard to deliver high quality software.

You hate searching for this nasty bug that had been introduced unnoticed days ago. Or was it weeks ago? By whom? In which code?

Manual functional and performance testing after each commited code change quickly becomes a NO-GO as the number of features is rising constantly. In this blog post, we will show, how Jenkins can help you with both: delivering high quality software and minimizing the time needed to find the cause of a bug.

How about …

  1. creating automated tests for each functionality and performance at different levels (end to end, and unit tests)
  2. running the automated tests after each code change
  3. keeping track of the test results

… in order to avoid any bad surprises late in the game?

Okay: for 1., the developer needs to create automated functional and perfomance tests; I guess, there is no way around this. Better do this even before writing the actual code. For 2. and 3., however, automation tools like Jenkins step in and can be of great help. The developers checks in the code and Jenkins can do the rest for you.

In the current  blog post, we will show how to integrate automated JUnit functional tests into a Jenkins build pipeline. We will see that JUnit tests can be invoked easily via Gradle (Okay, Maven is more popular than Gradle, I guess, but I like Gradle because of some advantages I have discussed here; However, just give me a hint in a comment to this blog and I will prioritize the creation of a Maven version of this blog post). The Jenkins JUnit plug-in will be used to

  1. display reports on single build runs as well as
  2. display trend analysis graphs like the following one I have borrowed from here:
2016-12-30-18_41_45-jenkins-junit-project-home-jpg-826x707
Source: http://nelsonwells.net/2012/09/how-jenkins-ci-parses-and-displays-junit-output/

In this and the next two blog posts, we plan to cover following quality gate measures:

  • Part 4.1: Functional Tests (this blog post): we will use Java JUnit tests performed before building the executable JAR. Jenkins will report the test trend
  • Part 4.2: Code Quality Tests (coming soon): we will use the Checkstyle Gradle plugin for reporting to which degree the code adheres to the Apache Foundations formal rules
  • Part 4.3: Performance Tests (planned): we will use JMeter for testing and reporting the performance trend performed after the Java build using external performance testers like JMeter

Older blogs of this series:

This blog post series about Jenkins build pipelines is divided into following parts:

    • Part 1: Installation and Configuration of Jenkins, loading Plugins
    • Part 2: Creating our first Jenkins job: GitHub download and Software build
    • Part 3: Periodic and automatically triggered Builds

What is Jenkins?

Jenkins is the leading open source automation server mostly used in continuous integration and continuous deployment pipelines. Jenkins provides hundreds of plugins to support building, deploying and automating any project.

2016-12-30-21_04_46

A typical workflow is visualized above: a developer checks in the code changes into the repository. Jenkins will detect the change, build (compile) the software, test it and prepare to deploy it on a system. Depending on the configuration, the deployment is triggered by a human person, or automatically performed by Jenkins. After each step, the developer is informed depending on the priorites defined.

For more information, see the introduction found in part 1 of this blog series.

Automated Functional Testing based on JUnit

In this blog post, we will show how we need to configure Gradle and Jenkins for automated JUnit testing and reporting. In order to build a quality gate, we will reverse the original order and perform the JUnit tests before we build the executable JAR file (we do not want to create JAR files that are not functional):

2016-12-28-12_50_23

Tools used

      • Vagrant 1.8.6
      • Virtualbox 5.0.20
      • Docker 1.12.1
      • Jenkins 2.19.3
        • JUnit Plug-in 1.19

Prerequisites:

      • Free DRAM for the a Docker Host VM >~ 4 GB
      • Docker Host is available, Jenkins is installed and a build process is configured. For that, perform all steps in part 1 to part 3 of this blog series
      • Tested with 2 vCPU (1 vCPU might work as well)

Step 1: Start Jenkins in interactive Terminal Mode

Make sure that port 8080 is unused on the Docker host. If you were following all steps in part 1 of the series, you might need to stop cadvisor:

(dockerhost)$ sudo docker stop cadvisor

I assume that jenkins_home is already created, all popular plugins are installed and an Admin user has been created as shown in part 1 of the blog series. We start the Jenkins container with the jenkins_home Docker host volume mapped to /var/jenkins_home:

(dockerhost)$ cd <path_to_jenkins_home> # in my case: cd /vagrant/jenkins_home/
(dockerhost:jenkins_home)$ sudo docker run -it --rm --name jenkins -p8080:8080 -p50000:50000 -v`pwd`:/var/jenkins_home jenkins
Running from: /usr/share/jenkins/jenkins.war
...
--> setting agent port for jnlp
--> setting agent port for jnlp... done

Step 2: Open Jenkins in a Browser

Now we want to connect to the Jenkins portal. For that, open a browser and open the URL

<your_jenkins_host>:8080

In our case, Jenkins is running in a container and we have mapped the container-port 8080 to the local port 8080 of the Docker host. On the Docker host, we can open the URL.

localhost:8080

Note: In case of Vagrant with VirtualBox, per default, there is only a NAT-based interface and you need to create port-forwarding for any port you want to reach from outside (also the local machine you are working on is to be considered as outside). In this case, we need to add an entry in the port forwarding list of VirtualBox:
2016-11-30-19_22_22-regel-fur-port-weiterleitung

We have created this entry in part 1 already, but I have seen that the entries were gone again, which seems to be a VirtualBox bug. I have added it again now.

Log in with the admin account we have created in the last session:

2016-12-09-10_24_00-jenkins

Step 3: Pre-Build JUnit Tests invoked by Gradle

In this step, we will invoke Gradle Tests before building the JAR. For that, we should verify locally that the Gradle tests are successful and then define a test Gradle task in the build process.

Step 3.1 (optional): Verify that Gradle Tests are successful

You can skip this test and directly let Jenkins do this for you. This may come handy, if you have not installed Git and/or Gradle locally.

Prerequisites

  • Your Java project has successful JUnit tests defined
  • Git is installed
  • The Project is cloned to a local directory
  • Gradle is installed

In order to test, whether the JUnit tests are successful, we can test those on a system with the project cloned (git, java and gradle must be installed):

(basesystem)$ gradle test
Starting a Gradle Daemon (subsequent builds will be faster)
Parallel execution is an incubating feature.
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava
warning: [options] bootstrap class path not set in conjunction with -source 1.6
1 warning
:processTestResources
:testClasses
:test

BUILD SUCCESSFUL

Total time: 29.9 secs

With that, we have verified that the command “gradle test”succeeds.

Note that the JUnit test must be designed in a way that they are independent of whether or not the JAR file is run in parallel. No simple way of running the executable JAR file in parallel to the execution of the JUnit tests seems to exist. In my case, I had to alter the JUnit tests to fulfill this prerequisite.

Step 3.2: Add Gradle test Task to Jenkins

As long as JUnit tests are defined in src/test of the project, adding Gradle tests to Jenkiny is as simple as adding “test” as a task to the list of Jenkins Build Gradle Tasks as follows:

On Dashboard -> Click on Project name -> Configure -> Build, add “task ” before the jar task:

2016-12-28-08_50_11-github-triggered-build-config-jenkins

Click Save.

If you have made local code changes on the project, now is the best time to commit and push them to the Git repository. If you have followed the steps in part 3, then this will automatically trigger a build process, so you do not need to click on “Build now” in that case. Otherwise, click on “Build now” on the Jenkins project page (e.g. Dashboard -> click on project name -> “Build now”).

Now we observe the result by clicking on the build process, then -> “Console Output”:

2016-12-28-09_45_34-github-triggered-build-724-console-jenkins

Don’t be confused by the blinking red ball on the upper left of the Console Output page: we see a BUILD SUCCESSFUL message and if we re-enter the same page, the ball is turned to static blue, indicating a successful build.

Step 4: Add JUnit Test Result Reporting to Jenkins

Now we will show how to add the JUnit test reports to the Jenkins build process.

Step 4.1: Install Jenkins JUnit Plugin

For Jenkins JUnit reporting, we need to install the JUnit Plug-in. For that, goto -> Jenkins Dashboard -> Manage Jenkins -> Manage Plugins -> Available -> Enter “JUnit Plugin” to the Find field -> Install

Note: If you do not find the plugin on the Available tab, search for it in the “Installed” tab.

You can install the plugin without reloading Jenkins.

Step 4.2: Configure Jenkins to collect and display the JUnit Test Results

In this step, we will configure Jenkins, so it will display the test results for individual builds as well as trend reporting. For that, navigate to:

Jenkins -> (choose Project) -> Configure -> Post-build Actions -> Publish JUnit test results report

2016-12-30-14_15_25-github-triggered-build-config-jenkins

Add

**/build/test-results/test/TEST-*.xml

to the “Test report XMLs” field, since this is the path, where Gradle is placing its JUnit test result reports (I have found the info here).

2016-12-30-14_18_51-github-triggered-build-config-jenkins

Now click Save.

Step 4.3: Verify JUnit individual Test Reporting

To test the Jenkins JUnit reporting feature, we trigger a clean build by adding “clean” to the Gradle tasks on Project -> Configure -> Build:

2016-12-30-17_43_59-github-triggered-build-config-jenkins

and clicking Save.

Then trigger a new build by clicking on Project -> Build now.

Then click on the Build Process, and then on Console output:

2016-12-30-17_48_38-github-triggered-build-731-console-jenkins

…scrolling down…

2016-12-30-17_50_01-github-triggered-build-731-console-jenkins

Do not be confused that the build process never seems to finish. Just click the Back to Project link:

Back to Project

On the Status page, we see that there were no failed tests:

2016-12-30-17_55_34-github-triggered-build-731-jenkins-v2

When we click on the Tests Result link on the left (or on the lower middle part on the Status page), we will see more details:

2016-12-30-17_58_25-github-triggered-build-731-test-results-jenkins-v2

We can see that we have had four tests (Create/Read/Update/Delete a file) and 100% of them were successful.

Step 4.3: Verify JUnit Test Trend Reporting

On the project’s Status page, a Test Trend graph is automatically added, as soon as there are two or more tests available. For that, click on “Build Now” on the left for a second time and click ENABLE AUTO REFRESH on the upper right. After the second build is complete, the (hopefully) blue Test Result Trend graph is showing up on the project status page:

2016-12-30-18_12_21-github-triggered-build-jenkins

The new blue graph shows that we had 4 successful tests in the last two builds.

Note: disregard the red Checkstyle Trend graph for now. This is something we will cover in the next blog post.

Step 5: Verify failed Test Reporting

Per default, Gradle build will fail, if one of the JUnit tests has failed, so it is building a strict quality gate. Will the test result be collected and reported nevertheless?

Let us test this now by breaking one of the JUnit tests by purpose. We have added an assert message that is expected to fail in one of the tests:

2016-12-30-19_27_46-java-ee-simple-restful-file-storage_src_test_java_de_oveits_simplerestfulfiles-v2

Now we commit and push the change to the SW repository:

$ git clone <Repository-URL>
$ cd <Repository Dir>
<perform the code changes here...>
$ git diff src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
diff --git a/src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java b/src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
index 684d30f..10200d5 100644
--- a/src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
+++ b/src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
@@ -115,6 +115,9 @@ public class SimpleRestfulFileStorageTests extends CamelSpringTestSupport {
 // mock expectations need to be specified before sending the message:
 mock.expectedBodiesReceived("File ttt created: href=http://localhost:2005/files/ttt");
 mock.expectedMessageCount(1);
+ ^M
+ // In order to break this test for Jenkins test reporting, we temporarily add a requirement that will fail:^M
+ mock.expectedMessageCount(2);^M

 template.sendBodyAndHeaders("direct:recipientList", body, headers);

$ git add src/test/java/de/oveits/simplerestfulfilestorage/SimpleRestfulFileStorageTests.java
$ git commit -m "Breaking a JUnit test by purpose for Jenkins reporting tests"
[jenkinstest 33655b9] Breaking a JUnit test by purpose for Jenkins reporting tests
 1 file changed, 4 insertions(+), 1 deletion(-)

olive@LAPTOP-P5GHOHB7 /d/veits/eclipseWorkspaceRecent/simple-restful-file-storage (jenkinstest)
$ git push
Counting objects: 9, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (9/9), 744 bytes | 0 bytes/s, done.
Total 9 (delta 4), reused 0 (delta 0)
remote: Resolving deltas: 100% (4/4), completed with 4 local objects.
To https://github.com/oveits/simple-restful-file-storage.git
 edb49f7..33655b9 jenkinstest -> jenkinstest

This will automatically trigger a new build (if you have followed part 3 of this series; otherwise just press “Build Now” on Jenkin’s project page).

We can see on the dashboard, that the build has failed:

2016-12-30-19_36_12-dashboard-jenkins

This was expected. Now let us click on the Project name, and we will see, what happened:

2016-12-30-19_37_40-github-triggered-build-jenkins

Perfect, that is exactly, what I wanted to achieve: On the Test Result Trend, we can see that we have performed 4 tests, one of which has failed.

Let us fix the failed test by commenting out (or removing) the wrong code again:

2016-12-30-19_40_01-java-ee-simple-restful-file-storage_src_test_java_de_oveits_simplerestfulfiles-v2

After

$ git add <file>
$ git commit -m "Fixed JUnit test again to test Jenkins JUnit trend report"
$ git push

The next build should be successful again and we can see in the trends graph that the failed test is fixed again:

2016-12-30-19_47_48-github-triggered-build-jenkins

thumps_up_3

Summary

In this blog post, we have shown

  1. How to add Java functional tests to the Jenkins build pipeline based on Gradle JUnit Plugin
  2. How to install the JUnit plug-in to Jenkins for report collection
  3. How to display JUnit test results for individual builds on the Jenkins portal
  4. How to display JUnit trend analysis on the Jenkins portal

The only challenge I have encountered is, that I had to re-write my JUnit tests in a way that they were successful when run stand-alone. Before they were successful only, if the executable JAR file was started manually before running the JUnit tests. This was resolved in a way specific to the framework used (Apache Camel in this case).

Coming Soon: Code Analysis Trend Analysis via Jenkins Checkstyle plugin

Further Reading

3

Java Build Automation Part 2: Create executable jar using Gradle


Original title: How to build a lean JAR File with Gradle

2016-11-14-19_15_52

In this step by step guide, we will show that Gradle is a good alternative to Maven for packaging java code into executable jar files. In order to keep the executable jar files “lean”, we will keep the dependent jar files outside of the jar in a separate folder.

Tools Used

  1. Maven 3.3.9
  2. JDK 1.8.0_101
  3. log4j 1.2.17 (downloaded automatically)
  4. Joda-time 2.5 (downloaded automatically)
  5. Git-2.8.4 with GNU bash 4.3.42(5)

Why using Gradle for a Maven Project?

In this blog post, we will show how Gradle can be used to create a executable/runnable jar. The task has been accomplished on this popular Mkyong blog post by using Maven. Why would we want to do the same task using Gradle?

By working with both, Maven and Gradle, I have found that:

  • Gradle allows me to move any resource file to outside of the jar without the need of any additional Linux script or alike;
  • Gradle allows me to easily create an executable/runnable jar for the JUnit tests, even if those are not separated into a separate project.

Moreover, while Maven is descriptive, Gradle is procedural in nature. With Maven, you describe the goal and you rely on Maven and its plugins to perform the steps you had in mind. Whereas with Gradle, you have explicit control on each step of the build process. Gradle is easy to understand for programmers and it gives them fine-grained control over the build process.

The Goal: a lean, executable JAR File

In the following step by step guide, we will create a lean executable jar file with all dependent libraries and resources.

Step 1 Download Hello World Maven Project of Mkyong

Download this hello world Maven project you can find on this popular HowTo page from Mkyong:

curl -OJ http://www.mkyong.com/wp-content/uploads/2012/11/maven-create-a-jar.zip
unzip maven-create-a-jar.zip
cd dateUtils

Logs:

$ curl -OJ http://www.mkyong.com/wp-content/uploads/2012/11/maven-create-a-jar.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7439  100  7439    0     0  23722      0 --:--:-- --:--:-- --:--:-- 24963

olive@LAPTOP-P5GHOHB7  /d/veits/eclipseWorkspaceRecent/MkYong/ttt
$ unzip maven-create-a-jar.zip
Archive:  maven-create-a-jar.zip
   creating: dateUtils/
  inflating: dateUtils/.classpath
  inflating: dateUtils/.DS_Store
   creating: __MACOSX/
   creating: __MACOSX/dateUtils/
  inflating: __MACOSX/dateUtils/._.DS_Store
  inflating: dateUtils/.project
   creating: dateUtils/.settings/
  inflating: dateUtils/.settings/org.eclipse.jdt.core.prefs
  inflating: dateUtils/log4j.properties
  inflating: dateUtils/pom.xml
   creating: dateUtils/src/
   creating: dateUtils/src/main/
   creating: dateUtils/src/main/java/
   creating: dateUtils/src/main/java/com/
   creating: dateUtils/src/main/java/com/mkyong/
   creating: dateUtils/src/main/java/com/mkyong/core/
   creating: dateUtils/src/main/java/com/mkyong/core/utils/
  inflating: dateUtils/src/main/java/com/mkyong/core/utils/App.java
   creating: dateUtils/src/main/resources/
  inflating: dateUtils/src/main/resources/log4j.properties
   creating: dateUtils/src/test/
   creating: dateUtils/src/test/java/
   creating: dateUtils/src/test/java/com/
   creating: dateUtils/src/test/java/com/mkyong/
   creating: dateUtils/src/test/java/com/mkyong/core/
   creating: dateUtils/src/test/java/com/mkyong/core/utils/
  inflating: dateUtils/src/test/java/com/mkyong/core/utils/AppTest.java
olive@LAPTOP-P5GHOHB7  /d/veits/eclipseWorkspaceRecent/MkYong/ttt
$ cd dateUtils/

olive@LAPTOP-P5GHOHB7  /d/veits/eclipseWorkspaceRecent/MkYong/ttt/dateUtils
$ 

Step 2 (optional): Create GIT Repository

In order to see, which files have been changed by which step, we can create a local GIT repository like follows

git init
# echo "Converting Maven to Gradle" > Readme.txt
git add .
git commit -m "first commit"

After each step, you then can perform the last two commands with a different message, so you can always go back to a previous step, if you need to do so. If you have made changes in a step that you have not committed yet, you can go back easily to the last clean commit state by issuing the command

# go back to status of last commit:
git stash -u

Warning: this will delete any new files you have created since the last commit.

Step 3 (required): Initialize Gradle

gradle init

This will automatically create a file build.gradle file from the Maven POM file with following content:

apply plugin: 'java'
apply plugin: 'maven'

group = 'com.mkyong.core.utils'
version = '1.0-SNAPSHOT'

description = """dateUtils"""

sourceCompatibility = 1.7
targetCompatibility = 1.7

repositories {

     maven { url "http://repo.maven.apache.org/maven2" }
}
dependencies {
    compile group: 'joda-time', name: 'joda-time', version:'2.5'
    compile group: 'log4j', name: 'log4j', version:'1.2.17'
    testCompile group: 'junit', name: 'junit', version:'4.11'
}

Step 4 (required): Gather Data

Since we are starting from a Maven project, which is prepared to create a runnable JAR via Maven already, we can extract the needed data from the POM.xml file:

MAINCLASS=`grep '<mainClass' pom.xml | cut -f2 -d">" | cut -f1 -d"<"`

Note: In cases with non-existing maven plugin, you need to set the MAINCLASS manually, e.g.

MAINCLASS=com.mkyong.core.utils.App

We also can define, where the dependency jars will be copied to later:

DEPENDENCY_JARS=dependency-jars

Logs:

$ MAINCLASS=`grep '<mainClass' pom.xml | cut -f2 -d">" | cut -f1 -d"<"`
$ echo $MAINCLASS
com.mkyong.core.utils.App
$ DEPENDENCY_JARS=dependency-jars
echo $DEPENDENCY_JARS
dependency-jars

Step 5 (required): Prepare to copy dependent Jars

Here, we will add instructions to the build.gradle file, which dependency JAR files are to be copied into a directory accessible by the executable jar.

We will need to copy the jars, we depend on, to a folder the runnable jar will access later on. See e.g. this StackOverflow question on this topic.

cat << END >> build.gradle

// copy dependency jars to build/libs/$DEPENDENCY_JARS 
task copyJarsToLib (type: Copy) {
    def toDir = "build/libs/$DEPENDENCY_JARS"

    // create directories, if not already done:
    file(toDir).mkdirs()

    // copy jars to lib folder:
    from configurations.compile
    into toDir
}
END

Step 6 (required): Prepare the Creation of an executable JAR File

In this step, we define in the build.gradle file, how to create an executable jar file.

cat << END >> build.gradle
jar {
    // exclude log properties (recommended)
    exclude ("log4j.properties")

    // make jar executable: see http://stackoverflow.com/questions/21721119/creating-runnable-jar-with-gradle
    manifest {
        attributes (
            'Main-Class': '$MAINCLASS',
            // add classpath to Manifest; see http://stackoverflow.com/questions/30087427/add-classpath-in-manifest-file-of-jar-in-gradle
            "Class-Path": '. dependency-jars/' + configurations.compile.collect { it.getName() }.join(' dependency-jars/')
            )
    }
}
END

Step 7 (required): Define build Dependencies

Up to now, a task copyJarsToLib was defined, but this task will not be executed, unless we tell Gradle to do so. In this step, we will specify that each time, a Jar is created, the copyJarsToLib task is to be performed beforehand. This can be done by telling Gradle that the jar goal depends on the copyJarsToLib task like follows:

cat << END >> build.gradle

// always call copyJarsToLib when building jars:
jar.dependsOn copyJarsToLib
END

Step 8 (required): Build Project

Meanwhile, the build.gradle file should have following content:

apply plugin: 'java'
apply plugin: 'maven'

group = 'com.mkyong.core.utils'
version = '1.0-SNAPSHOT'

description = """dateUtils"""

sourceCompatibility = 1.7
targetCompatibility = 1.7

repositories {

     maven { url "http://repo.maven.apache.org/maven2" }
}
dependencies {
    compile group: 'joda-time', name: 'joda-time', version:'2.5'
    compile group: 'log4j', name: 'log4j', version:'1.2.17'
    testCompile group: 'junit', name: 'junit', version:'4.11'
}

// copy dependency jars to build/libs/dependency-jars
task copyJarsToLib (type: Copy) {
    def toDir = "build/libs/dependency-jars"

    // create directories, if not already done:
    file(toDir).mkdirs()

    // copy jars to lib folder:
    from configurations.compile
    into toDir
}

jar {
    // exclude log properties (recommended)
    exclude ("log4j.properties")

    // make jar executable: see http://stackoverflow.com/questions/21721119/creating-runnable-jar-with-gradle
    manifest {
        attributes (
            'Main-Class': 'com.mkyong.core.utils.App',
            // add classpath to Manifest; see http://stackoverflow.com/questions/30087427/add-classpath-in-manifest-file-of-jar-in-gradle
            "Class-Path": '. dependency-jars/' + configurations.compile.collect { it.getName() }.join(' dependency-jars/')
            )
    }
}

// always call copyJarsToLib when building jars:
jar.dependsOn copyJarsToLib

Now is the time to create the runnable jar file:

gradle build

Note: Be patient at this step: it can appear to be hanging for several minutes, if it is run the first time, while it is working in the background.

This will create the runnable jar on build/libs/dateUtils-1.0-SNAPSHOT.jar and will copy the dependency jars to build/libs/dependency-jars/

Logs:

$ gradle build
:compileJava
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
:processResources
:classes
:copyJarsToLib
:jar
:assemble
:compileTestJava
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
:processTestResources UP-TO-DATE
:testClasses
:test
:check
:build

BUILD SUCCESSFUL

Total time: 3.183 secs

$ ls build/libs/
dateUtils-1.0-SNAPSHOT.jar dependency-jars

$ ls build/libs/dependency-jars/
joda-time-2.5.jar log4j-1.2.17.jar

Step 9: Execute the JAR file

It is best practice to exclude the log4j.properties file from the runnable jar file, and place it outside of the jar file, since we want to be able to change logging levels at runtime. This is, why we had excluded the properties file in step 6. In order to avoid an error “No appenders could be found for logger”, we need not specify the location of the log4j.properties properly on the command-line.

Step 9.1 Execute JAR file on Linux

On a Linux system, we run the command like follows:

java -jar -Dlog4j.configuration=file:full_path_to_log4j.properties build/libs/dateUtils-1.0-SNAPSHOT.jar

Example:

$ java -jar -Dlog4j.configuration=file:/usr/home/me/dateUtils/log4j.properties build/libs/dateUtils-1.0-SNAPSHOT.jar
11:47:33,018 DEBUG App:18 - getLocalCurrentDate() is executed!
2016-11-14

Note: if the log4j.properties file is on the current directory on a Linux machine, we also can create a batch file run.sh with the content

#!/usr/bin/env bash
java -jar -Dlog4j.configuration=file:`pwd`/log4j.properties build/libs/dateUtils-1.0-SNAPSHOT.jar

and run it via bash run.sh

Step 9.1 Execute JAR file on Windows

In case of Windows in a CMD shell all paths need to be in Windows style:

java -jar -Dlog4j.configuration=file:D:\veits\eclipseWorkspaceRecent\MkYong\dateUtils\log4j.properties build\libs\dateUtils-1.0-SNAPSHOT.jar
11:45:30,007 DEBUG App:18 - getLocalCurrentDate() is executed!
2016-11-14

If we run the command on a Windows GNU bash shell, the syntax is kind of mixed: the path to the jar file is in Linux style while the path to the log properties file needs to be in Windows style (this is, how the Windows java.exe file expects the input of this option):

$ java -jar -Dlog4j.configuration=file:'D:\veits\eclipseWorkspaceRecent\MkYong\dateUtils\log4j.properties' build/libs/dateUtils-1.0-SNAPSHOT.jar
11:45:30,007 DEBUG App:18 - getLocalCurrentDate() is executed!
2016-11-14

Inverted commas have been used in order to avoid the necessity of escaped backslashes like D:\\veits\\eclipseWorkspaceRecent\\… needed on a Windows system.

Note: if the log4j.properties file is on the current directory on a Windows machine, we also can create a batch file run.bat with the content

java -jar -Dlog4j.configuration=file:%cd%\log4j.properties build\libs\dateUtils-1.0-SNAPSHOT.jar

To run the bat file on GNU bash on Windows, just type ./run.bat

Yepp, that is it: the hello world executable file is printing the date to the console, just as it did in Mkyong’s blog post, where the executable file was created using Maven.

simpleicons_interface_folder-download-symbol-svg

Download the source code from GIT.

Note: in the source code, you also will find a file named prepare_build.gradle.sh, which can be run on a bash shell and will replace the manual steps 4 to 7.

References

Next Steps

  • create an even leaner jar with resource files kept outside of the executable jar. This opens the opportunity to changing resource files at runtime.
  • create an executable jar file that will run the JUnit tests.

 

3

Docker Java Performance Tests


In this blog we will show the surprising result that Docker on Ubuntu seems to outperform Docker on CoreOS by ~30%, when tested with a java templating web service.

But first let us discuss the reason on why I have started the tests in the first place: on my blog Docker Web Performance Tests we had found the surprising result that a Rails Web Application on Docker on a CoreOS VM has about 50% better performance, than Rails directly on Windows hardware. Will we find the same result for java?

java on Windows Hardware vs. Ubuntu VirtualBox VM (beware: apples vs. bananas!)

The answer is no: the chosen java application, an Apache Camel templating web service has a factor ~4 better performance on Windows Hardware than on an Ubuntu VirtualBox machine (beware: apples vs. bananas!).

Now comparing apples with apples (all on VirtualBox VMs):

java on Ubuntu vs. Docker/Ubuntu vs. Docker/CoreOS vs. Docker/boot2docker (apples vs. apples)

The java performance on a the chosen Vagrant deploy-able Ubuntu image has almost the same performance on Docker as on native Ubuntu (only 5% performance degradation). With that good result, the Ubuntu Docker alternative has outperformed the other Docker alternatives by more than 30% and has impressed with java startup times that were more than 3 times as fast as on CoreOS.

Note: At the time of writing the blog post, this dockerization vs. virtualization vs. native comparison paper was unknown to me: it follows a more comprehensive scientific approach by testing many different performance aspects separately. I like it a lot: check it out!

This is only a small proof of concept (POC) of a hobby developer and the POC can only give hints about which road to take. Even though now the more scientific paper is available, I am still fine with having invested the time of testing and publishing, since my test results are focused on the applications I develop (Rails and java Apache Camel) and the infrastructure alternatives I have on my laptop: native Windows vs. Ubuntu VirtualBox VM vs. CoreOS VirtualBox VM vs. boot2docker Virtualbox VM. The scientific paper was only testingUbuntu 13.10 and no CoreOS I was interested in.

Document Versions

v1 original post
v2: updated CoreOS Vagrant image from v713.3.0 to v766.4.0 with no performance improvement
v3: giving CoreOS a last chance: updated to alpha v845.0.0 with no performance improvement
v4: changed introduction and re-organized Tldr; and added a link to a more scientific paper.

Tldr;

When comparing the java performance on a native Ubuntu VM with several Docker host VM alternatives (UbuntuCoreOS and boot2docker), we can see that the chosen Vagrant deploy-able Ubuntu image has almost the same performance on Docker as on native Ubuntu. With that, the Ubuntu Docker alternative has outperformed the other Docker alternatives by more than 30% and has impressed with java startup times that were more than 3 times as fast as the one of the second best candidate: CoreOS.

As expected, java performance has shown to be substantially (~4 times) higher on Windows Hardware than on VirtualBox Ubuntu VM with same number of cores. Note that this last comparison is like comparing apples with bananas: Windows on Hardware vs. Ubuntu Linux on VirtualBox virtual machine with less DRAM.

Test Scenarios

In the current blog, we perform performance measurements of a java Apache Camel application for following scenarios:

  1. application run natively on Windows 7 SP1 on the host system (8 GB DRAM, with ~2.2 GB free before start of the application; 2 core CPU i5-2520M@2.5 GHz)
  2. application run within a VirtualBox Ubuntu 14.04 LTS (64 bit) VM with 1.5 GB DRAM and 1 vCPU on the same Windows host system
  3. application run within a docker container on the above Ubuntu VM (vagrant box version 1.8.1)
  4. application run within a docker container on a CoreOS (Vagrant CoreOS stable 717.3.0 and v766.4.0) Virtualbox VM with same resources as the Ubuntu VM (1.5 GB DRAM and 1 vCPU) run on the Windows system
  5. application run within a docker container on a boot2docker Virtualbox VM with same resources as in the old blog, i.e. (2 GB DRAM and 2 vCPU), run on the same Windows host.
    Note that boot2docker is deprecated by now, but since we have performed the Rails Docker Web Performance Tests on boot2docker, we keep it as a reference.

Considering the results we have seen with the Rails web application, we expect that 3. and 4. have higher performance than 1.

With 2. and 3. we can easily extract the difference that comes from the additional docker layer. For the option 4., we expect the same or a higher performance than in 3, since CoreOS is optimized for running Docker containers.

As in the other performance test blog, we use Apache Bench as the measurement tool.

Test Results

Java Apache Camel Startup Time

  1. application run natively on Windows on the host system
    1. NEED TO TEST AGAIN WITH STDOUT REDIRECTION INTO FILE!!!
      2 cores: Apache Camel 2.12.2 (CamelContext: camel-1) started in 3.5 +-0.6 seconds
  2. application run within a VirtualBox Ubuntu VM with 1.5 GB DRAM and 1 vCPU
    1. 1 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 5.0 seconds
    2. 2 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 3.7 seconds
  3. application run within a docker container on the above Ubuntu VM
    1. 1 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 5.2 seconds
    2. 2 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 4.0 seconds
  4. application run within a docker container on a CoreOS VM with same resources as the Ubuntu VM
    1. 1 vCPU, v713.3.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in 15.2 seconds
    2. 2 vCPU
      1. v713.3.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in 13.4 seconds
      2. v766.4.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in 14.5 +- 1.5 seconds
      3. v845.0.0: Apache Camel 2.12.2 (CamelContext: camel-1) started in [15.0, 13.2, 19.2,13.2] seconds (4 tests)
  5. application run within a docker container on a boot2docker VM with same resources as in the old blog, i.e. (2 GB DRAM and 2 vCPU)
    1. 2 vCPU: Apache Camel 2.12.2 (CamelContext: camel-1) started in 24.1 +- 0,5 seconds (4 Tests)

Windows: has the quickest Apache Camel startup (95 routes) with ~3.5 sec +- 0.6 sec.

Ubuntu with or without Docker: with 2 vCPUs, the startup is slightly slower. Only with 1 vCPU, the startup takes ~40% more time on Ubuntu (both, native and in Docker).

CoreOS: Apache Camel has a 3.8 and 3,4 times longer (!) startup time on CoreOS Docker than on Windows and on Ubuntu, respectively. CoreOS is more lightweight than the Ubuntu image, and is supposed to be optimized for Docker. Therefore, this is a surprising result.

boot2docker: has the worst Apache Camel bootup time: it takes almost a 7 times longer (!!) than the same process on Windows. boot2docker is deprecated by now. This performance results is another reason not to use it anymore.

Java Apache Camel Throughput

Test script:

# test:
./ab -n 10000 -c 100 "http://<IP>:<Port>/ProvisioningEngine?action=Add%20Customer&CustomerName=ttt&offlineMode=offlineMode"

Shortly before we run the actual test, we train the application by issuing the same command with -n 100 and -c 100.

  1. application run natively on Windows
    1. 2 i5-2520M CPU cores: Requests per second:    46.3 [#/sec] (mean)
  2. application run within a VirtualBox Ubuntu VM with 1.5 GB DRAM and 1 vCPU (on an i5 host)
    1. 1 vCPU: Requests per second:    14.3 [#/sec] (mean)
    2. 2 vCPU: Requests per second:    17.9 [#/sec] (mean) (i.e. ~25% higher than 1 vCPU)
  3. application run within a docker container on the above Ubuntu VM
    1. 1 vCPU: Requests per second:    12.5 [#/sec] (mean)
    2. 2 vCPU: Requests per second:    17.0 [#/sec] (mean) (i.e. ~35% higher than 1 vCPU)
  4. application run within a docker container on a CoreOS VM with same resources as the Ubuntu VM (1.5 GB DRAM and 1 or 2 vCPU)
    1. 1 vCPU, v713.3.0:: Requests per second:    8.82 [#/sec] (mean)
    2. 2 vCPU
      1. v713.3.0: Requests per second:    13.0 [#/sec] (mean) (i.e. ~48% higher than 1 vCPU)
      2. v766.4.0: Requests per second:    11.8 [#/sec] (mean)
      3. v845.0.0: Requests per second:    13.3 [#/sec] (mean)
  5. application run within a docker container on a boot2docker VM with same resources as in the old blog, i.e. (2 GB DRAM and 2 vCPU)
    1. 1 vCPU: not tested
    2. 2 vCPU: We get the error “Test aborted after 10 failures”, a memory allocation failure, even though we have 2 GB DRAM instead of 1.5 GB!

Note that each requests is creating tons of output on the terminal (deep tracing). However, in call cases, I have redirected the stream into a log file (using the “>” operator).

Note also that the comparison between Windows and Linux variants is not fair, since Windows was tested on hardware with 2 real CPU cores, while the Linux variants was tested on Virtualbox VMs with one or two vCPUs (Ubuntu, Docker on Ubuntu and CoreOS) or 2 vCPUs (boot2docker). However, considering you have a Windows notebook and you want to decide, whether to perform your development and tests on native Windows vs. Linux VM vs. on Docker, this is what you get:

  • Windows: on the host system with 2 real CPU cores is ~4 times higher throughput performance than any of the Linux VM variants on 1 vCPU (comparing apples with bananas, but still relevant for my decision)
  • Ubuntu vs Docker on Ubuntu: Native Linux has a ~5 to 15% higher throughput performance than Docker on Linux
  • CoreOS: Interestingly, the Ubuntu Docker VM image has a ~40% higher throughput performance than the optimized CoreOS image. Because of this negative result for CoreOS, I have updated the v713.3.0 image to the latest versions available: stable-v766.4.0 and alpha-v845.0.0. This had no substantial impact on the (still bad) performance.
  • boot2docker: The boot2docker image has memory allocation problems. Since boot2docker is deprecated, I did not investigate this further.

Summary

We have compared the performance of a java Apache Camel application on following platforms: on a Windows 7 laptop, on an Ubuntu Virtualbox image and in a Docker container, where Docker was tested on Ubuntu, CoreOS and the official, but deprecated boot2docker image.

Note that Windows has been tested on hardware, while the Linux variants have been tested on Virtualbox VMs. This is not a fair test between Windows and Linux, but it still helps me to decide, whether I better keep performing my java development directly on my Windows laptop, or whether I can move all java development to a Virtualbox Linux VM on my Windows laptop. For Rails, I had found the surprising result, that the performance was better on the VMs than directly on the hardware. Not for java, though:

2015.10.28-10_22_08-hc_001

We have seen that the performance on Windows 7 without virtualization is ~4 times higher than on any of the Virtualbox VMs, which will account for the expected performance degradation effect of software virtualization. This is no comparison between java on Windows and java on Ubuntu. It is just a comparison of the options I have on a Windows laptop: directly work on Windows or work on the VMs. The CoreOS image has a ~30% lower performance than the Ubuntu image, which is surprising, since it is much more lightweight than the Ubuntu image.

The application’s startup times are quite good for Ubuntu and for Docker on Ubuntu. Surprisingly, both, CoreOS as well as boot2docker fall back substantially with respect to startup times (factor 3 and 6 compared to Ubuntu).

The deprecated boot2docker image has major memory allocation problems and should not be used.

Recommendation

All in all, development is best to be performed directly on the Windows laptop, with Ubuntu or Docker on Ubuntu being a good alternative because of low application startup times. CoreOS does not seem to be a good alternative, since in the development phase, I often need to restart the application, which takes ~13 sec on CoreOS instead of ~3.5 to 4 sec on Windows hardware or on the Ubuntu VM’s Docker.

Performance tests are best to be done on Windows hardware (or on HW-virtualized VMs, which I have not performance-tested yet). However, because of the deployment advantages, I would like to deliver my application as a Docker image, so I need to perform tests on Docker as well. For those Docker tests, the Ubuntu-trusty64-docker Virtualbox image of William Yeh has shown the best performance results.

If I need to test the behavior of the application on clustered Docker hosts, CoreOS and Kubernetes are the only cluster alternatives I have experience with, and I have not done more than a small installation POC in case of Kubernetes.  However, considering the low CoreOS performance, I will need to investigate its alternatives, I guess: e.g. Docker Swarm, Kubernetes, or others (ping me, if you have suggestions).

My Path towards Continuous Integration of my Java Application

I am planning to do the following:

  • I will continue to develop my java application on native Windows and I will continue to push the code to Github often.
  • I have linked Github with TravisCI, so the code is automatically tested in the cloud. This way, I do not need to wait for test results. The test results are sent via email and I can react soon for any newly introduced bugs.
  • I will link TravisCI with Docker Hub, so a Docker image will be created automatically.
  • Locally, I will use the Docker on Ubuntu, if I need to troubleshoot the Docker build process, or I need to manually test, whether the Docker image really works.
  • If it comes to productive deployment including Docker host clustering, I had made good experience with CoreOS, even though clustering behind a HTTP proxy is a challenge; see here. However, the performance is sub-optimal, and I need to evaluate, which alternatives I have (e.g. test new CoreOS versions? Docker Swarm? Google Kubernetes?).

Watch out for my next blog post(s)…

;-))

0

Java Build Automation Part 1: Create lean executable jar using Maven


Have you ever tried to create an executable jar file, while keeping all configuration files outside of the jar file, accessible to operations folks? Here is a step by step guide.

JAR files are java archive files and as such, are not designed to be manipulated by operations folks (“ops”). However, if a developer creates an executable jar, all needed files, including configuration and template files are packed into the jar file as a default. In this post, I will show, how to make configuration and template files accessible again.

Update/Comment (2016-11-20): I now have created a blog, which (in my opinion) shows a better way of creating a lean executable JAR file using gradle: see here.

Automate the creation of the executable jar

Up to now, I have used Eclipse’s export function to create executable jars for both, the main program and the JUnit test program.

2015.10.19-20_26_44-hc_001   2015.10.19-20_28_30-hc_001

However, this mouse-click procedure has forced me to manually extract the configuration files from the jar file, so the administrator can change the configuration files without compiling the source code again. Since I want to come a little step closer to an automated build of a Docker image for my application, I need to automate the creation of the executable jar. A short Internet research has pointed into the direction of three possible options to do so (please tell me, if you now better ones!):

  1. using jar commands
  2. using ant
  3. using maven

About 1.: jar

jar commands are similar to tar commands and give you full flexibility to control, which folders&files are put in the jar file, choose the folder structure within the jar, and choose, which files&folders are placed outside of the jar. However, I prefer to use a higher level tool like ant or maven, since they are opinionated about the structure. This helps me to choose a structure which is close to best practice.

About 2.: ant

The ant tool has the advantage that an ant script can be created using eclipse export.

2015-10-11_204351_capture_004

However, when looking into the resulting ant script, I am missing a possibility to control, whether of not the resources are packed into the jar file (tell me, if you know, how to do it!).

2015.10.19-20_34_27-hc_001

Moreover, each and every dependable jar file is explicitly copied by the ant script. In case I will add a dependency to the POM file later, the ant script needs to be adapted. This is not so optimal. In any case, I have decided to look at the next option: the maven script.

About 3.: maven

On http://www.mkyong.com/maven/how-to-create-a-jar-file-with-maven/ I have found a very good step by step documentation on how to create a executable jar. It was quite easy to apply the documentation on my Apache Camel java program. It also shows how to place the log4j properties file outside of the jar. However, I have more files than the log4j properties file, which I want to make accessible to the administrator.

Executable jar via Maven

Additions to the pom.xml file:

<!-- Make this jar runnable/executable -->
<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <!-- documentation: https://maven.apache.org/plugins/maven-jar-plugin/jar-mojo.html -->
  <artifactId>maven-jar-plugin</artifactId>
  <configuration>
  <!-- custom output directory -->
  <outputDirectory>${project.build.directory}/jarDir</outputDirectory>
  <!-- custom Name of the generated JAR (default: ${project.build.finalName} e.g. de.oveits.provisioningengine_recent-0.5.2.21_stable). Note, that .jar is always appended -->
  <finalName>OpenScapeConnector</finalName>
   <archive>
    <manifest>
     <addClasspath>true</addClasspath>
     <mainClass>de.oveits.provisioningengine.MainApp</mainClass>
     <classpathPrefix>lib/</classpathPrefix>
    </manifest>
   </archive>
  </configuration>
</plugin>

In order to avoid a “fat” jar, i.e. a large jar, which has all dependent jars included, we use the “copy-dependencies” plugin:

<!-- Copy project dependency -->
<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-dependency-plugin</artifactId>
 <version>2.5.1</version>
 <executions>
  <execution>
   <id>copy-dependencies</id>
   <phase>package</phase>
   <goals>
    <goal>copy-dependencies</goal>
   </goals>
   <configuration>
    <!-- exclude junit, we need runtime dependency only -->
    <includeScope>runtime</includeScope>
    <!-- custom output directory -->
    <outputDirectory>${project.build.directory}/jarDir/lib/</outputDirectory> 
  </configuration> 
</execution> 
</executions> 
</plugin>

Note: Eclipse might complain about the copy-dependencies plugin not being compatible with m2e of the eclipse IDE. In this case, you can either accept one of the offered solutions, or you can manually add the following code to the POM file:

<pluginManagement>
 <plugins>
 <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
  <plugin>
  <groupId>org.eclipse.m2e</groupId>
  <artifactId>lifecycle-mapping</artifactId>
  <version>1.0.0</version>
  <configuration>
  <lifecycleMappingMetadata>
  <pluginExecutions>
  <pluginExecution>
  <pluginExecutionFilter>
  <groupId>
  org.apache.maven.plugins
  </groupId>
  <artifactId>
   maven-dependency-plugin
  </artifactId>
  <versionRange>
  [2.5.1,)
  </versionRange>
  <goals>
  <goal>copy-dependencies</goal>
  </goals>
  </pluginExecutionFilter>
  <action>
  <ignore></ignore>
  </action>
  </pluginExecution>
  </pluginExecutions>
  </lifecycleMappingMetadata>
  </configuration>
  </plugin>
 </plugins>
</pluginManagement>

With this, the plugin is ignored with m2e, therefore avoiding the conflict.

Then, we can create the jar file by:

mvn package -Dmaven.test.skip=true

In my case, I have skipped the JUnit tests by specifying the option “-Dmaven.test.skip=true”, because I have not implemented simulator/mock services yet and the test with the real target systems take >~30 minutes.

This step will create a jar file in the <outputDirectory> as defined in the POM file (here: target/jarDir) with the name <finalName> + jar  (here: “OpenScapeConnector.jar”).

The dependent jars are put in <outputDirectory> of the maven-dependency-plugin. The <classpathPrefix> must be the relative path from the jar file to the dependent libraries with trailing “/” (here: lib/). Otherwise, the dependent jars will not be found.

2015.10.19-17_49_08-hc_001

The program then can be started with:

java -jar target/jarDir/OpenScapeConnector.jar

Here, OpenScapeConnector.jar is the <finalName> and target/jarDir is the <outputDirectory> as defined in the POM.

Note that I have kept the log4j properties in the jar file for now, which is a small deviation from the description on Mkyong’s Post. I will take care of this later below.

Extract an remove Configuration Folders from the jar File

We have created an executable jar, but this jar still contains configuration files which are not accessible easily by the operators/administrators. In my case, “cfg” and “properties” contain configuration files, “templates” contains many velocity templates and the “wiretap” folder is the destination folder for traces. All of those folders need to be easily accessible by operations folks:

2015-10-19_180722_capture_001

I have not found any cool maven plugin that could be of any help with this task (apart from excluding the files in the jar and copy them per Linux cp commands to the target folder), so I had decided to create a jar first, and then use standard jar and zip commands to extract and delete the configuration files from the jar:

Moving out and deleting of the folders/files can be done via shell script like follows:

# extract the files and directories myFile1 myFile2 myDir1 myDir2 myDir3 from the jar:
extractList="myFile1 myFile2 myDir1 myDir2 myDir3"
jar xvf myJarFile.jar $extractList

# find and remove the files and folders from the jar. 
find $extractList -exec zip -d $jarFile {} \;

The find command recursively finds all extracted directories. The zip delete command can be used to delete the file/directory from the jar file, since jar files have the same format as zip files.

Now let us try to start the jar file:

java -jar target/jarDir/OpenScapeConnector.jar
...
Exception in thread "main" org.apache.camel.RuntimeCamelException: org.apache.camel.FailedTo CreateRouteException:...not found in classpath

The jar file does not work. What happened? Now, that we have moved some folders out of the jar file, java does not know, where to find them. All files and folders are located outside of the jar in the same directory as the jar file, so we have to add the current directory “.” to the classpath in the Manifest file.

First, I have tried to solve that issue via maven-jar-plugin by adding

<additionalClasspathElements><additionalClasspathElement>.</additionalClasspathElement></additionalClasspathElements>

to the POM file. However, this element does not seem to work within this maven-jar-plugin: it had no effect on the resulting Manifest file (did you get it work?). Therefore, I have decided to use low level manipulations via jar, awk and zip commands instead. This is not very complex either: we need to

  1. extract file from jar using jar command
  2. manipulate file using awk and gsub
  3. update file in jar using zip command

like follows:

#!/bin/sh
jarDir=target/jarDir
jarFile=OpenScapeConnector.jar

cd $jarDir

# 1) extract MANIFEST.MF file:
jar xvf $jarFile META-INF/MANIFEST.MF

# 2) fix MANIFEST.MF: 
awk '{gsub(/Class-Path: /, "Class-Path: . "); print $0}' META-INF/MANIFEST.MF > MANIFEST.MF.temp; mv MANIFEST.MF.temp META-INF/MANIFEST.MF 

# 3) update MANIFEST.MF in jar file: 
zip -u OpenScapeConnector.jar META-INF/MANIFEST.MF

With the jar xvf command, we have extracted the Manifest file from the jar file. With the awk command, we have added the current directory ‘.’ as she first entry to of the Class-Path:

2015.10.19-19_07_34-hc_001

The zip command has updated the jar file with the manipulated Manifest file.

Now we try again to start the application:

java -jar target/jarDir/OpenScapeConnector.jar
...
6140 [main] INFO org.apache.camel.spring.SpringCamelContext - Total 94 routes, of which 94 is started.
[ main] SpringCamelContext INFO Apache Camel 2.12.2 (CamelContext: camel-1) started in 2.781 seconds
6141 [main] INFO org.apache.camel.spring.SpringCamelContext - Apache Camel 2.12.2 (CamelContext: camel-1) started in 2.781 seconds

Bingo! The java program has started with no problems.

Verification

Now let us test, that a change in one of the extracted files has the desired effect. E.g. we can change the log4j settings:

vi target/jarDir/log4j.properties

and change

log4j.logger.org.apache.camel=INFO

to

log4j.logger.org.apache.camel=DEBUG

Now at the startup of the java application, many DEBUGs are shown:

ent.event.EventComponent@5b8dbd69
[ main] DefaultManagementAgent DEBUG ...
[ main] SpringCamelContext INFO Total 94 routes, of which 94 is started.
9322 [main] INFO org.apache.camel.spring.SpringCamelContext - Total 94 routes, of which 94 is started.
[ main] SpringCamelContext INFO Apache Camel 2.12.2 (CamelContext: camel-1) started in 5.905 seconds

and the startup of the Apache Camel routes take twice as long. This is expected and shows that the administrator now can change the configuration files, and after a restart of the program, the changes are active. No re-creation of the jar file is necessary.

The same holds, if I manipulate one of the configuration files. In my web application, the IP address of the target is saved in cfg/WebPortal.cfg; i.e. one of the files, I have extracted from the jar file. Before changing the file, I see on the Web Portal page:

2015.10.19-19_51_10-hc_001

In order to verify that the configuration files in the cfg folder outside of the jar file are “active”, I have changed the OSVIP variable from 10.152.0.10 to 1.1.1.1 in cfg/WebPortal.cfg:

2015.10.19-20_01_00-hc_001

After a restart of the java application and a reload of the page, we see that the web portal content has changed:

2015.10.19-19_56_23-hc_001.

The Web Portal content has been adapted without the need to re-compile the jar file: Bingo! Our goal has been achieved.

Summary

We have shown how to automate the creation of an executable jar file using maven. In order to create a “lean” jar, the dependent jars were extracted and placed outside the jar using a maven plugin. In order to give administrators the chance to change configuration files without re-compiling the jar from source, we have extracted those files using jar and zip commands. Finally, the search path (classpath) had to be manipulated within the jar file, using jar, awk and zip commands.

2015.10.20-14_52_48-hc_001

Note that I am still left with the manual task of the test JUnit jar creation. The maven-jar-plugin does not seem to be designed to be usable for both, the main application and its corresponding JUnit test application, see https://maven.apache.org/plugins/maven-jar-plugin/examples/create-test-jar.html for details. The “easy way” described there did not work for me (“cannot find main class”, when I try to start the jar). So, either I need to find, what I am doing wrong, or I need to follow the “preferred way”, i.e. to move the test code to a separate project with its own POM file and apply the same procedure to the test main app as I did for the main application.