Optimizing Play 2 for Database-Driven Apps

Update: There is now a detailed doc in Play’s documentation about configuring Play’s thread pools.

Last week Matt Raible and I presented the Play vs. Grails Smackdown at ÜberConf. The goal of the session was to compare Play 2 + Java with Grails by creating the same app with each framework. We used a number of criteria for the comparison including some benchmarks. You can read more about the results in Matt’s session recap blog. After presenting the results we learned that it’s pretty important to optimize Play 2′s Akka threading system in these types of applications. Play 2 is optimized out-of-the-box for HTTP requests which don’t contain blocking calls (i.e. asynchronous). Most database-driven apps in Java use synchronous calls via JDBC so Play 2 needs a bit of extra configuration to tune Akka for these types of requests.

Let me illustrate this with a simple example. In order to simulate a real-world scenario where the database blocks for a realistic amount of time I will run this benchmark on Heroku instead of locally. I’ll be using the “java-ebean” branch from my “play2bars” example app and the shared Heroku PostgreSQL database.

To run Apache Bench and send 10,000 requests with 100 concurrent connections to the JSON service in the app, I ran the following command from an EC2 server located in the same AWS region as my app on Heroku:

ab -n 10000 -c 100 http://falling-dusk-7291.herokuapp.com/bars

The results of the first, un-optimized run were:

Concurrency Level:      100
Time taken for tests:   10.272 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1560000 bytes
HTML transferred:       510000 bytes
Requests per second:    973.53 [#/sec] (mean)
Time per request:       102.719 [ms] (mean)

At 973 requests per second this isn’t too bad but in some test runs I was getting errors like:

[error] play - Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)

This meant that Akka was blocked for too long so I tuned the Akka settings in Play to better handle synchronous code and re-ran the tests:

Concurrency Level:      100
Time taken for tests:   7.274 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1560000 bytes
HTML transferred:       510000 bytes
Requests per second:    1374.70 [#/sec] (mean)
Time per request:       72.743 [ms] (mean)

This time I got 1,374 requests per second! The simple optimization is to give Akka more threads to work with and longer timeouts so when threads gets blocked by synchronous database calls there are still more to use for request handling. Here is the Play configuration I used:

play {
 
    akka {
 
        actor {
 
            deployment {
 
                /actions {
                    router = round-robin
                    nr-of-instances = 100
                }
 
                /promises {
                    router = round-robin
                    nr-of-instances = 100
                }
 
            }
 
            retrieveBodyParserTimeout = 5 seconds
 
            actions-dispatcher = {
                fork-join-executor {
                    parallelism-factor = 100
                    parallelism-max = 100
                }
            }
 
            promises-dispatcher = {
                fork-join-executor {
                    parallelism-factor = 100
                    parallelism-max = 100
                }
            }
 
        }
 
    }
 
}

The number of threads Akka will use is determined by the number of CPUs multiplied by the “parallelism-factor” setting, up to the “parallelism-max” setting. The default “parallelism-factor” is “1″ meaning the number of threads is equal to the number of CPUs with a default “parallelism-max” of “24″. The Play documentation provides more details about these settings.

The full config files are at available in the play2bars project.

As with all benchmarks, take this with a grain of salt, run your own benchmarks, and do your own tuning for your environment. Let me know if you have any questions.

UPDATE: Sadek Drobi has posted a very detailed article about Async, Reactive, Threads, Futures, ExecutionContexts in Play.

This entry was posted in Play Framework. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.
  • duc tung vu

    Hi James,
    what do you think about performance of Play 1 vs Play 2. In Play 1.x Akka was not a integral part of the framework as with Play 2.x. I know, the Groovy-based template engine of Play 1 was bad, but we have the awesome Japid Engine.

    Play 1.x + Japid vs Play 2 What do you think?

    And another question i want to ask is, why must  Akka be a integrated part of the Framwork when i just want to use it per Java API, what for advantages does one have by having Akka in Framework’s core?

    Thank you!

    • http://www.jamesward.com James Ward

      Good question.  I’m not really sure about the performance differences because I haven’t run any benchmarks on Play 1.  But Akka is great and having it at the core of Play 2 is a great choice, especially for asynchronous apps.

      • lionel

        hello i have somme trooble with th configuration is not taken into account

        in my app i have done

        public static Result sleep(java.lang.Integer time) throws InterruptedException {
        Logger logger = LoggerFactory.getLogger(Autre.class);
        Calendar cal = Calendar.getInstance();
        long startat=cal.getTimeInMillis();
        logger.debug(startat+”befor”);
        for (int i=0;i<time;i++){
        logger.debug(startat + " top:"+i);
        Thread.currentThread().sleep(1000);
        }
        logger.debug("after");
        return ok("end sleep :"+time);
        }

        i see my request serialized … one by one

        i run play2 with stage mode

        and i use a akka-syn.conf in conf file with
        include "akka-sync.conf" at th end of the application.conf

        i d'on't know what i do wrong
        thanks lionel

        • http://www.jamesward.com James Ward

          Looks like the code formatting got a big screwy. Do you have a project somewhere that I can take a look at?

    • Viktor Klang

      Akka is designed to be used from both Java and Scala, so why would that be an issue?

  • Chris Webb

    It sounds like this tuning for db driven apps, a very common scenario, should be saved as a template in the Play2 distro

  • http://twitter.com/develsas develsas

    Hi James, great post, as usual

    I think this template should be, at least, in the documentation… It’s a very comon scenario

    I also think there should be an akka profile or something, so that we may set a db-app setting with just one line in the application.conf…

    • http://www.jamesward.com James Ward

      I agree. :)

  • http://twitter.com/jhulten Jeffrey Hulten

    Is it possible to isolate DB type activities into a pool of actors that get a larger parallelism factor and maximum?

    • http://www.jamesward.com James Ward

      It seems like that should be possible but I’m not sure how to do it.

    • http://twitter.com/pkolloch Peter Kolloch

      Actually, you probably should at least not execute them on a ForkJoinThreadPool. ForkJoinThreadPools are generally not suited for longer running/blocking tasks. Tasks might get blocked by a completely unrelated task since work stealing is used.

      So either use a “normal” e.g. fixed size thread pool for everything or delegate longer running tasks to another pool with akka.dispatch.Future. akka.dispatch.Future expects an implicit ExecutionContext which you can use to decide which thread pool to execute on.

  • Ryan Means

    What version of Play were you using? I’m trying your config example in Play 2.0.3, and we still receive the error. For our test, we have a couple of really long blocking calls that take between 10 – 20 seconds each (report building). If I spin up about 8 or more of those requests and then immediately try to open any other page I will receive the actor ask timeout error immediately. It’s like the system didn’t even try as the other 8 requests had just started within the last 2 seconds.

    • http://www.jamesward.com James Ward

      I did this with Play 2.0.2 but it should work the same way with 2.0.3. Are you running in production mode or development mode?

      • Ryan Means

        Interesting. I didn’t catch that – running in Prod mode worked. Any idea why? Do the akka settings not pick up in dev mode?

        • http://www.jamesward.com James Ward

          I think that in dev mod Play only uses 1 thread. Not sure why.

          • http://twitter.com/pkolloch Peter Kolloch

            In dev mode play also rechecks if any files changed etc.

            Dev mode is really not suitable for load testing.

  • Ryan Means

    James, thanks for your follow-ups to my previous comment – you got us up and going with a tuned instance. However, I noticed some other issues going on with static content and blocking style apps. You might want to check out this playframework google group discusion that I had with one of the framework authors – James Roper (my comment comes in on October 5th): https://groups.google.com/forum/?fromgroups=#!topic/play-framework/p1aq1s3PhJg. It would probably be good to update your example to mention this blocking issue with the static content. Thanks for the good work!

    • http://www.jamesward.com James Ward

      Thanks Ryan for the heads up. I’ll post a response on the Play Google Group.

  • http://www.facebook.com/thejustinmann Justin Mann

    How many DB connections are you using? In my understanding, every thread that uses the DB needs its own DB connection. Assuming 4 CPUs times 24, would mean 96 threads or 96 DB connections.

    • http://www.jamesward.com James Ward

      The connections are pooled across all of the request handling threads. The pool size is configurable but I believe the default is 10 connections. I believe that database code will block until it gets a connection from the pool or that operations times out.

  • Gadille Lionel

    Hello,

    I come back with my configuration trouble and link on the project

    I have extract code to show my purpose

    https://bitbucket.org/gadillel/testplay2/src

    and the conf used
    https://bitbucket.org/gadillel/testplay2/src/be56818410c89d9fce590ec6c192951d0a12bbd3/conf/application.conf?at=default

    I launch 10time this query to exercise the conf
    http://localhost:9000/Async/sleep?time=30
    and look to the log file for the messages

    I see that 5 request are process concurrently (cpu 4core +1) it’s the default
    But i don’t find how to change this value
    I need that because i use java and i have some query with not local database with ~500ms by database access and about 4query (2s)
    Also i freeze the application and i don’t want to put all this query on aka promise system is too heavy

    Did you now how to change this pool size?

    thanks lionel

    • http://www.jamesward.com James Ward

      You config looks like to increase the thread pool size. But I can’t try your code locally. We should try to simplify this down to just something that blocks a thread. (No db stuff)

  • Gadille Lionel

    Note for the test the database is never use
    I would split the project if you prefer

    I have read a lot today

    I read on google group that if you try using same resource with same navigator the request are queuing

    here i found a bug on play
    request are serialized on ubuntu chrome navigator
    but not for ubuntu firefox (request are parallelized with top to 5)
    … it make me lost a lot of time with (i have not pay attention of navigator change)

    now i will try with automatically concurrent test
    because i need different agent
    and more reproducibility

    i will come back with the test done with my tools

    • http://www.jamesward.com James Ward

      If you can simplify the test app as much as possible, that would be helpful. You should also be testing with a tool like Apache Bench, rather than with a browser.

  • Gadille Lionel

    hello

    I have made more test around burst charge
    First i have place a more realistic burst (i imagine a better result)
    My 300 hundred request come on 6 000ms (6s)*

    the result was worth
    09:37:28.412 [main] DEBUG testController.TestAsync – * End in: 31s *
    09:37:28.412 [main] DEBUG testController.TestAsync – * nbr ok: 218 *
    09:37:28.412 [main] DEBUG testController.TestAsync – * nbr Error:82 *
    09:37:28.412 [main] DEBUG testController.TestAsync – * code: ,500 *
    09:37:28.412 [main] DEBUG testController.TestAsync – *************************************

    that is interesting is that

    All the 0to99 first request are ok (conf have a lot of 100)

    The error are between 100 ~ … here the request order are meld

    but all the error are grouped …

    [debug] c.Application – after95
    [debug] c.Application – after96
    [debug] c.Application – after97
    [debug] c.Application – after98
    [debug] c.Application – after99
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [error] play – Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)
    [debug] c.Application – after107
    [debug] c.Application – after115
    [debug] c.Application – after110

    if i switch the
    /actions {
    router = round-robin
    nr-of-instances = 100
    }

    ################## youpi

    i find why i still bugging for me
    i fact i was doing my test whith play ~run
    but is ok with play start

    Thanks , lionel

    • http://www.jamesward.com James Ward

      That seems like the correct behavior. “play start” uses your configured thread pool. But “play ~run” just uses one thread. If you need more help with this, then lets chat over email. james.ward at typesafe dot com

  • http://www.chrononaut.org/ David Moles

    Hi, James –

    Long time Java developer with zero Play / Akka experience here. We’re looking at using Play 2 to layer some RESTful services over an existing giant mess of business logic, clients for other RESTful services, clients for traditional (XML/WSDL) web service clients, direct database access, and probably some weirder stuff I’m not remembering right now.

    What I’m wondering is, assuming you had another mechanism (say something Java 6 concurrency-based) for making those calls asynchronous, can you push the blocking problem down to that layer and leave the Akka configuration more or less alone? Or would Akka itself (in a many-threads long-timeouts config) be a better tool for implementing that mechanism in the first place?

    • http://www.jamesward.com James Ward

      Hi David,

      Thanks for the good question. You can definitely wrap Futures or Actors around anything you have that is blocking. The blocking piece will still use a thread. But by putting Futures or Actors around it you will have a nice way to manage the blocking, configure the thread pool / execution context, and optionally distribute the app via Akka Clustering.

      I hope that helps.

      -James

      • http://www.chrononaut.org/ David Moles

        Thanks! I suspect that once I dig into Akka it may turn out to be easier than rolling my own Java-side callback solution.

        • http://www.jamesward.com James Ward

          Yes it certainly will! :)

  • DunkelheitJP

    Hi James, good block. I’m new in Play Framework and I’m developing a metaheuristic algorithm. the execution time is very long. After an hour I get the following error on the web
    [AskTimeoutException: Timed out] .
    In this website is my problem :

    http://stackoverflow.com/questions/19601580/asktimeoutexception-timed-out-play-framework-2-1

    I hope you can help me. thank you very much.

  • Sathish Reddy

    Hi James,

    I had already have Ruby on rails app with me which does not support the asynchronous feature.So I want to integrate the Play with Ruby on rails in order to get the asynchronous feature.Is there any possibility of integration and also give me any sample asynchronous application using play 2.x as I was new to this?

    Any help would be greatly appreciated,thank you very much

    • http://www.jamesward.com James Ward

      This might be possible with JRuby but other than that it probably will be hard since Play runs on the JVM and Ruby does not.

      • Sathish Reddy

        Thanks James for the reply.

        Please give me a sample application of Play 2.x which has asynchronous mechanism and also if possible provide me a link how do we achieve asynchronous mechanism using JRuby



  • View James Ward's profile on LinkedIn