Ajax and Flex Data Loading Benchmarks

For close to a year I’ve been working (in my infrequent spare time) on an application that shows differences in data loading for RIAs (Rich Internet Applications), comparing Ajax methods, Ajax frameworks, and various Flex methods. The results are pretty surprising. The screenshot below is from a test run I did with the server running locally. (Note for the screenshot below: All tests except Dojo were 5000 rows, while the Dojo test was 500 rows.)

To run the benchmarks yourself, go to:
http://www.jamesward.org/census/

I have tried to be as fair as possible with these tests. Still, I encourage you to use these results only as an initial guide. You should always do your own benchmarks that more closely resemble your actual use case.

You can find more details explaining the tests and the results inside the “Guide Me” mode of the application, so I won’t go into depth here. Also, the code is Open Source (GPL) on SourceForge. If you feel that a test should be done differently or you find other things that you think should be changed, get the code, fix it, and send me a patch file. Please don’t just tell me I did something wrong. In the spirit of Open Source, help me fix it. I want this application to be as fair and accurate as possible.

While these results may be eye-opening to some, once you understand some fundamental differences between Flex and Ajax, they end up making a lot of sense. Flex applications run within the Mozilla Tamarin VM inside Flash 9. Tamarin does JIT compilation which makes code execution 10 to 1000 times faster than interpreted JavaScript in the browser. Also Tamarin can be very efficient when dealing with typed objects.

In addition, Flex supports a compact binary object serialization protocol, called AMF3, which is extremely fast. There are numerous Open Source implementations using various backend technologies for AMF3, as well as a commercial / supported Adobe product, called Flex Data Services. If you control both sides of the wire in your application, then there is rarely a good reason to serialize and deserialize text. Objects are blazing fast as you can tell by this benchmark. AMF3 is also typically much more efficient across the wire (even without compression) and consumes much less client side memory. JIT compilation and binary object serialization are the primary reasons why Flex AMF3 is so fast, even in the test with the full 20,000 rows. And, it’s not just faster for loading – it also speeds client side sorting and filtering. (Try client-side sorting in the Dojo benchmark and the Flex AMF benchmark.)

A quick note on compression: I did not include compression support in this version because its value really depends on many application-specific factors. For instance, if most of your users have slow connections then compression can help speed things up, but if most have fast connections the added latency can actually make things slower. However, I do want to add a Gzip option to this benchmark for the next release.

This is really a version 0.1 release. There are many improvements I have on the to-do list. And I’d love the community’s help. Here are some things I want to do for the 0.2 release:
– Add Gzip option on tests
– Add a Laszlo test
– Implement more Ajax frameworks that have DataGrids and that support paging and sorting
– Add a server side sorting option
– Add Ajax client side memory usage metrics (don’t know if this is possible)
– Improve the UI for use on projectors (I use this demo in every presentation I give)
– Integrate feedback I get from the initial version
– Maybe add a WPF/E test (which I won’t be able to do because I run Linux)

I sincerely hope that you find this demo useful. I built it as a tool to help people learn more about the various methods of data loading in RIAs. I did not in any way build this to attack Ajax, Dojo, and SOAP. Please help me improve this application so that we all continue to learn more about the technologies available to us.

This entry was posted in Ajax, Flex, Open Source. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.
  • Jordan

    Thanks for the update, I’ll be happy to see the next version. I did figure out what I was missing… in the new versions of LCDS, the file is called lcds.war (and there’s also lcds-samples.war) instead of flex.war (and samples.war). Changing the build file to reference lcds.war fixed one problem, but I was still receiving compile errors. It seems that flex-messaging.jar was renamed to flex-messaging-core.jar, so I changed that on (or near) line 173 in the build file for the javac task. Also, there is no longer a license.properties file in the lcds root, but creating a blank file seemed to work.

    I just thought I would document some of this stuff in case others are still trying to build the old version. I have it building now,

    –Jordan

  • http://www.jamesward.org James Ward

    Thanks Jordan!

    -James

  • Pingback: Flex AMF3 « GS journals - flash, flex, movies, music, and lots more…

  • Pingback: » Zend AMF at Adobe MAX ‘08 > Wade Arnold

  • Thomas

    @James Ward
    Hi,
    I am really interested in this new version since I too meet various problems trying to build and run census on a local machine. Do you have some kind of target date?

    Thanks

  • http://www.jamesward.com James Ward

    @Thomas

    I’m still working on it. Now that we are in the holiday season it probably won’t happen until the beginning of 2009. Sorry.

    -James

  • Thomas

    @James Ward
    No reason to be sorry. I’m just looking forward to seeing it.
    Thanks for answering.

  • Joey

    Hi James,
    Can you clarify the following statement from your piece above…”If you control both sides of the wire in your application, then there is rarely a good reason to serialize and deserialize text.”

    Are you saying that the best way to transfer data (text), when you control a webservices and Flex, is to query the data source, build the object server-side, and then transfer the object or an array of objects to Flex, which is then used to build the client-side objects in Flex for databinding? If so, when would you not use this method?

  • http://www.jamesward.com James Ward

    Hi Joey,

    I’m saying that text is an inefficient serialization technology. When possible use binary serialization technologies like AMF.

    -James

  • Pingback: Uxebu.com - JavaScript addicts » dojo.beer(2) and Webmontag Munich

  • Vlad G

    Hi James!

    If you are interested, try also a modification to Flex (AS3) JSON parser for improved strings parsing speed. See following cookbook entry, solution 2:
    http://www.adobe.com/cfusion/communityengine/index.cfm?event=showdetails&productId=2&postId=13226&loc=en_US

    Thanks for this great comparison tool!

  • Jason

    Hi James!

    Can you take a look at the Data you are using for your benchmark tool? It appears not to be bring back any data. I would love to show this to some of my colleagues.

    Thanks!

  • http://www.jamesward.com James Ward

    Hi Jason,

    Firefox 3 broke something with the iFrames. So clicking on the “Output” pane will launch the test.

    Let me know if that helps.

    -James

  • Pingback: Building UIs on Top of Very Large Database Tables « Business Intelligence, Flex and China

  • Pingback: Zend Con 2009 session files | Wade Arnold

  • Pingback: Learning Flex – Lesson 18, Accessing Server Side Objects « M@ Blog

  • Pingback: RPC in Flash, AMF « Sergein: Flex, AS3, Algorithms

  • Leonardo Leon

    Hi,

    I’m using ZendAmf with Flex and it’s working well but I have some questions.

    1. If I use an HTTP Analyzer, I can see that using HTTPService and AMF is the same Content-Length received. How could I benchmark this two approaches?

    2. Using HTTP Analyzer with http://www.jamesward.org/census/ you can see that EX4 5000 rows uses 48 K while AMF uses 86K bytes received.

    3. When AMF send the result, isn’t it in binary mode? I have a php class that return a simpleXML and I can see it with HTTP Analyzer in text mode not binary, Why?

    Thanks

    • http://www.jamesward.com James Ward

      Hi Leonardo,

      1. What are you connecting to? What data are you returning? The content-length is usually different between AMF and XML.
      2. Hmm… My guess is that one of them is using gzip. Try my new Census2: http://www.jamesward.com/census2
      3. With AMF the HTTP response body is binary. XML is not AMF. And wrapping XML in AMF is not a good idea.

      • Leonardo Leon

        Hi, James

        Thanks for you response,

        Well, basically I’m testing and I want to understand the behavior behind AMF.

        I want to see the advantage of using AMF over HTTPService. In my case I have HTTPService returning XML (Oracle EPG). I made a php class wrapper to wrap XML into AMF just to play around and see what happen in terms of benchmark.

        I did use your http://www.jamesward.com/census2 but I don’t understand what I see in the HTTP Analyzer, because with AMF the content-lenght is higher.

        Thanks again

        • http://www.jamesward.com James Ward

          Wrapping XML in AMF actually increases the size and does not do anything to take advantage of the benefits of AMF. AMF is a binary serialization format. XML is a text serialization format.

          I’m not sure what you mean by the content-length being higher for AMF. With Census the AMF transfer sizes are much lower than the XML transfer sizes.

          • Leonardo Leon

            With content-lenght I mean

            This is with AMF: 86,33K
            – 00:00:00.000 iexplore.exe[23208] (Count=427, Sent=354,25 K, Received=1,88 M, ElapsedTime=1172,520 s)
            9 False +01:47.422 s 1,763 s POST 200 86,33 K application/x-amf http://www.jamesward.com/census/messagebroker/amf?id=flex_amf3&gzip=true

            HTTP/1.1 200 OK
            Date: Sat, 02 Apr 2011 14:40:11 GMT
            Server: Apache-Coyote/1.1
            X-Powered-By: Servlet 2.5; JBoss-5.0/JBossWeb-2.1
            Cache-Control: no-cache
            Expires: Sat, 25 Dec 1999 00:00:00 GMT
            Pragma: no-cache
            Content-Type: application/x-amf
            Content-Length: 85972
            Via: 1.1 http://www.jamesward.com
            Keep-Alive: timeout=15, max=97
            Connection: Keep-Alive

            And this is with XML: 48,02K
            – 00:00:00.000 iexplore.exe[23208] (Count=427, Sent=354,25 K, Received=1,88 M, ElapsedTime=1172,520 s)
            2 False +13.977 s 1,811 s GET 200 48,02 K text/xml http://www.jamesward.com/census/servlet/CensusServiceServlet?id=flex_xml_e4x&command=getXML&gzip=true&rows=5000

            I noted that it is gziped.

            In my test I use this php code: return simplexml_load_string($response);

            I thought that if a php class return back a XML object using AMF it was going to be compressed thus less Kbytes. But in my case it is the same using HTTPService and AMF.

            James, I’m doing this just to understand the underhood of AMF, I know that I must use Value Objets to return data.

          • http://www.jamesward.com James Ward

            It appears that the AMF is not actually being gzipped. That is probably a bug in Census. With 5000 rows and gzip the AMF is 28K. The same 5000 rows with XML and gzip is the 48K that you see.

            In Census2 the reported transfer sizes are accurate and gzip works correctly.

            One big reason that AMF is smaller than XML even without gzip is because AMF only puts the object metadata into the stream once. Whereas in XML it is constantly repeated. In the case of Census, the string “classOfWorker” is in the stream 10,000 times for 5,000 rows because there is an open and close tag for each time that property is sent. In AMF the string “classOfWorker” is only put into the stream once. If you want to learn more about AMF check out the spec:
            http://opensource.adobe.com/wiki/download/attachments/1114283/amf3_spec_05_05_08.pdf?version=1

          • Leonardo Leon

            Thanks for your anwers.

            How can I be sure that AMF is being gzipped, because I have the same issue.

          • http://www.jamesward.com James Ward

            AMF doesn’t do any gzipping on it’s own. You must gzip it in your web server. Check out:
            http://wadearnold.com/blog/flash/gzip-compression-is-not-part-of-amf

          • Leonardo Leon

            Thank you very much, I’m very grateful for your time.

  • http://sebastien-arbogast.com Sebastien Arbogast

    I’d be curious to see how it compares to http://msgpack.org/

  • Pingback: Andrew Trice » Blog Archive » AMF vs. JSON in AIR Mobile Applications

  • Jj

    alert(’Testing For XSS Hole’)

    • http://www.jamesward.com James Ward

      Really?

  • http://twitter.com/makhansingh Navneet Karnani

    I looked at Census2 and thought that adding a Flex JSON evaluation would be a great value add, just like the AJAX JSON, only done using Flex. 

    • http://www.jamesward.com James Ward

      I used to have it in there but it was so embarrassingly slow I took it out.  It locked up the test with anything more than 1000 rows or something.  But I think Flash Player might have native JSON support now, which would certainly be faster.

  • VonVictor Valentino Rosenchild

    Thank

  • VonVictor Valentino Rosenchild

    I want to thank for creating your blog and sharing so much valuable information. I am currently teaching myself various programming languages and at the moment I am focused on Clojure, Common Lisp, F# and R. I know that you posted this back in 2007 but that is alright, because it is still very valuable.

  • Dennis Flanagan

    Thanks, James. Still relevant! Question came up regarding JSON and … Thanks!



  • View James Ward's profile on LinkedIn