Thoughts on the RebelLabs Great App Server Debate

It was with great trepidation that I sat down to read the RebelLabs report into developer friendly application servers. I’ve spent over 2 years pouring my heart and soul into making the Liberty profile the best runtime for developers and here was the first independent report comparing them.

Overall I thought the report was reasonably fair. Given I’m generally happy with how Liberty was rated in this report why am I writing this post? Well I was very disappointed in the performance numbers. I’m disappointed for a number of reasons; the first is no one likes to be told their baby is slow, the second is that I had been running Jenkins on my laptop for a few weeks before the report came out and I’d had a much better experience than RebelLabs. For context I’ve repeated the RebelLabs numbers here:

1 Empty Server startup time 2s
2 Server startup time with Jenkins installed 4s
3 Server Restart time with Jenkins installed 6s
4 Jenkins app deploy time 2s
5 Application Initialization 21s
6 Startup with app & invoke Jenkins 25s
7 Startup, deploy & invoke Jenkins 25s

As you can see the app initialization time shown is pretty poor and surprised me so I arranged to head up to the RebelLabs office in the UK to talk to Simon Maple about the details. My goal was to find out how the numbers were generated so I could compare apples with apples and get to the bottom of why we had different experiences. Simon was very helpful which led me to write this blog post. In the RebelLabs report the following terms are used, and I’ve provided a description of what they mean based on my meeting with Simon.

  • Startup/restart time – how long the server starts (or restarts) with zero, or one app deployed
  • Deploy time – how long between copying the application into the runtime and it being reported as available
  • Application initialization – how long it took to respond to the first request

The tables presented below use the same rows as the RebelLabs report. Note that my rows 6 and 7 are generated by summing the results of previous rows, I’ve indicated in brackets which rows are used for the derived value.

Test Setup
Tests were run on a Mac Book Pro laptop. Spec: 16GB RAM (1333MHz DDR3), 2.2GHz Intel Core i7 CPU with 4 hyper-threaded cores, 500GB Hitachi HTS725050A9A362 HDD, OSX 10.7.5. I used the same Java 7u21 build that RebelLabs used.

I was using the latest release of Liberty profile 8.5.5.0, I now know that Rebel Labs were using 8.5.0.1.

This isn’t a scientific test (but then neither was RebelLabs’), but I wrote an ant script to get the numbers, ran each test 3 times and averaged the result from the best 2 rounded to the nearest half second.

Just to note these tests were run with IBM Notes, Word, Firefox, Safari, Twitter, 1Password, Emacs , Lotus Symphony, IBM Sametime, Lotus Mobility Client and iTerm (with 2 terminals) running. You wanted real numbers and a real developer runs more than the app server right?

Liberty results

1 Empty Server startup time 2.5s
2 Server startup time with Jenkins installed 4s
3 Server Restart time with Jenkins installed 4s
4 Jenkins app deploy time 2s
5 Application Initialization 9s
6 Startup with app & invoke Jenkins (2 + 5) 13s
7 Startup, deploy & invoke Jenkins (1 + 4 + 5) 13.5s

As you can see the Liberty numbers I’ve got are much better than the ones RebelLabs got. The big difference is that I am running 8.5.5, which is the latest release. RebelLabs were running 8.5.0.1. We put a lot of improvements into 8.5.0.2 and 8.5.5 in this area and as you can see the results are significantly better. That said even with 8.5.0.1 RebelLabs shouldn’t have got the numbers they got, but since that was released in October last year and we have had two releases since then I’m not going to lose sleep worrying about it.

I also decided to run the numbers against Tomcat 7.0.39 to have some form of baseline from the RebelLabs test. After all I’m on different hardware.

1 Empty Server startup time 6s
2 Server startup time with Jenkins installed 10s
3 Server Restart time with Jenkins installed 6s
4 Jenkins app deploy time 4s
5 Application Initialization 5s
6 Startup with app & invoke Jenkins (2 + 5) 11s
7 Startup, deploy & invoke Jenkins (1 + 4 + 5) 15s

The RebelLabs report does note that we have a very fast startup time, and a larger initialization time and asks what we are doing to achieve this. One of the Liberty design principals is deferred initialization. This means we do the least amount of initialization up front we can, which results in fast startup times, but can lengthen the time taken to make the first request. This can be disabled in the server.xml by adding:

<webContainer deferServletLoad=”false” />

This can make a significant difference depending on the application and it explains why I had been getting a different set of numbers from RebelLabs. They had been using the default configuration, which defers initialization of Servlets, but I had overridden this to eagerly initialize Servlets. So I decided to see what happens when I configure my tests to run with eager initialization.

Liberty deferServletLoad Results

1 Empty Server startup time 2.5s
2 Server startup time with Jenkins installed 6s
3 Server Restart time with Jenkins installed 6s
4 Jenkins app deploy time 4.5s
5 Application Initialization 6s
6 Startup with app & invoke Jenkins (2 + 5) 12s
7 Startup, deploy & invoke Jenkins (1 + 4 + 5) 13s

As you can see this increases the startup time and shortens the initialization time quite a bit. Although you see a small improvement in startup time it is really just moving stuff around, it doesn’t necessarily make the runtime that much faster.

Conclusion

From my study the Liberty performance is significantly better than the RebelLabs experience. Perhaps the biggest change is I’m using the most recent release of the Liberty profile. We put a significant amount of attention to performance and application initialization in the 8.5.0.2 and 8.5.5 release and I’m pretty sure that is reflected here. I also believe that had RebelLabs configured eager Servlet initialization they would have seen much better numbers. I accept that this isn’t the default configuration though and it is reasonable for them to have used the default configuration.

Before I finish there is is one final thing that Liberty profile does that I think is quite cool. The Liberty profile runs servlet initialization asynchronously. What that means is that we will report the application as being available and initialize the Servlets in the background. In reality this gives you a nice fast startup time, but mitigates a lot of the setup cost of the first request. To simulate this I reran my test (with deferServletLoad set to false) and added a five second pause before making the first request.

Liberty (5 second pause before request)

1 Empty Server startup time 2.5s
2 Server startup time with Jenkins installed 6s
3 Server Restart time with Jenkins installed 6s
4 Jenkins app deploy time 4.5s
5 Application Initialization 2s
6 Startup with app & invoke Jenkins (2 + 5) 8s
7 Startup, deploy & invoke Jenkins (1 + 4 + 5) 9s

As you can see this results in an apparent 2 second initialization time. A 5 second pause might seem like a long time, but when doing real world development a Human is involved and we tend not to react to stimuli in the way computers do. There will always be some length of pause, whether you were distracted by Twitter or email, or paused for a sip of coffee. The point here is if we are looking at real-world performance automatically generated stats aren’t necessarily showing you the performance, as you would experience it.

I believe that were RebelLabs to rerun their numbers on our most recent release they would come up with a different conclusion than the one they did a month ago. It also demonstrates how seriously we take performance and all the work we have done to improve things over the last 9 months.

Appendix – How to reproduce these numbers

If you want to try this out for yourself you can. You just need to download the three ant scripts below and execute the tests using ant -Dserver.runtime=[liberty|tomcat]. You will need jenkins, tomcat and liberty installed the same directory as the ant scripts. The targets to run are:

  1. startEmptyServer
  2. startupWithJenkins
  3. restartWithJenkins
  4. deployJenkins
  5. appInit

The ant scripts are:

  • Mike Croft

    Those numbers were a bit surprising for me too. I haven’t actually done anything more than install it yet, so I can’t comment on my experience, but it’s good to hear a lot of positive things about Liberty!

    What’s the word on Java EE 7 support?

    • Alasdair

      As you have probably spotted the latest Liberty profile does not have support for Java EE 7, but I’m not allowed to talk about future product plans outside of an NDA. So for now there is no word I can share. If you are interested in Java EE 7, or specific bits of Java EE 7 such as WebSockets then you can go to the Request for Enhancements tool and vote for existing RFEs or submit new ones.

  • Roman Kharkovski

    I tested latest versions of all these products, plus WebSphere and WebLogic and posted my findings here: http://whywebsphere.com/2013/11/13/developer-point-of-view-on-the-app-server-debate/

  • Pingback: Developer point of view on the App Server debate « Why WebSphere? Blog