Ajax and Flex Data Loading Benchmarks
Monday, 30 April 2007
For close to a year I’ve been working (in my infrequent spare time) on an application that shows differences in data loading for RIAs (Rich Internet Applications), comparing Ajax methods, Ajax frameworks, and various Flex methods. The results are pretty surprising. The screenshot below is from a test run I did with the server running locally. (Note for the screenshot below: All tests except Dojo were 5000 rows, while the Dojo test was 500 rows.)
To run the benchmarks yourself, go to:
http://www.jamesward.org/census/
I have tried to be as fair as possible with these tests. Still, I encourage you to use these results only as an initial guide. You should always do your own benchmarks that more closely resemble your actual use case.
You can find more details explaining the tests and the results inside the “Guide Me” mode of the application, so I won’t go into depth here. Also, the code is Open Source (GPL) on SourceForge. If you feel that a test should be done differently or you find other things that you think should be changed, get the code, fix it, and send me a patch file. Please don’t just tell me I did something wrong. In the spirit of Open Source, help me fix it. I want this application to be as fair and accurate as possible.
While these results may be eye-opening to some, once you understand some fundamental differences between Flex and Ajax, they end up making a lot of sense. Flex applications run within the Mozilla Tamarin VM inside Flash 9. Tamarin does JIT compilation which makes code execution 10 to 1000 times faster than interpreted JavaScript in the browser. Also Tamarin can be very efficient when dealing with typed objects.
In addition, Flex supports a compact binary object serialization protocol, called AMF3, which is extremely fast. There are numerous Open Source implementations using various backend technologies for AMF3, as well as a commercial / supported Adobe product, called Flex Data Services. If you control both sides of the wire in your application, then there is rarely a good reason to serialize and deserialize text. Objects are blazing fast as you can tell by this benchmark. AMF3 is also typically much more efficient across the wire (even without compression) and consumes much less client side memory. JIT compilation and binary object serialization are the primary reasons why Flex AMF3 is so fast, even in the test with the full 20,000 rows. And, it’s not just faster for loading – it also speeds client side sorting and filtering. (Try client-side sorting in the Dojo benchmark and the Flex AMF benchmark.)
A quick note on compression: I did not include compression support in this version because its value really depends on many application-specific factors. For instance, if most of your users have slow connections then compression can help speed things up, but if most have fast connections the added latency can actually make things slower. However, I do want to add a Gzip option to this benchmark for the next release.
This is really a version 0.1 release. There are many improvements I have on the to-do list. And I’d love the community’s help. Here are some things I want to do for the 0.2 release:
– Add Gzip option on tests
– Add a Laszlo test
– Implement more Ajax frameworks that have DataGrids and that support paging and sorting
– Add a server side sorting option
– Add Ajax client side memory usage metrics (don’t know if this is possible)
– Improve the UI for use on projectors (I use this demo in every presentation I give)
– Integrate feedback I get from the initial version
– Maybe add a WPF/E test (which I won’t be able to do because I run Linux)
I sincerely hope that you find this demo useful. I built it as a tool to help people learn more about the various methods of data loading in RIAs. I did not in any way build this to attack Ajax, Dojo, and SOAP. Please help me improve this application so that we all continue to learn more about the technologies available to us.
No. 1 — April 30th, 2007 at 10:54 am
Quite interesting :)
You have in you html file (begining of file) of this blog “9a4″.
No. 2 — April 30th, 2007 at 10:56 am
Sorry.. there is something generating.. Some numbers with every refresh of website… The same in div class=”column1″. Fe. 55f ul.
No. 3 — April 30th, 2007 at 11:02 am
Very strange. I wonder where those are coming from. Maybe a Wordpress plugin? I’ll have to dig in and see if I can figure that one out. Thanks for the heads up.
-James
No. 4 — April 30th, 2007 at 11:04 am
Hey James, just wanted to let you know I keep getting an error when trying to view your app in the feed demon window. Works great in IE or Gecko based browsers.
No. 5 — April 30th, 2007 at 11:13 am
Thanks Don. I’ll look further into that.
No. 6 — May 1st, 2007 at 9:55 am
That error problem is happening all over the web (as seen via my FF anyway).
ZDNet has been killin me since their redesign.
Thx for the benchmarks.
No. 7 — May 1st, 2007 at 6:13 pm
Hey James, would love to contribute to version 2.
Particular to showing results For Flex ZLib compressed XML E4X and how it competes against Flex AM3.
Bjorn
No. 8 — May 1st, 2007 at 9:58 pm
Nice stuff! We gotta get some Silverlight in there eventually.
I can see the XML is slow crowd gettin’ juiced off this for years to come ;)
No. 9 — May 2nd, 2007 at 12:10 am
Hi Bjorn,
That would be great if you can help add that! Here are the basic instructions…
1) Get the code from SourceForge http://www.sf.net/projects/flexapps (cvs module is “census”)
2) Get LCDS from http://labs.adobe.com
3) Get JBoss 4.0.4
4) Set the JBOSS_HOME and LCDS_HOME env properties
5) Copy the default server in JBoss to census
6) Start JBoss
7) Run “ant deploy-war” in the dir where you checked out census to
As far as the UI goes, it would be nice to have an options area on the Navigator with a checkbox for “Enable GZip”. This way GZip could be enabled for any test. For the benchmarks we can just set a request parameter or HTTP header that would get checked in another servlet filter (that does the gzipping).
Got it? Let me know if you have any questions / problems. james<at>jamesward<dot>org
Thanks!
-James
No. 10 — May 2nd, 2007 at 12:33 am
[...] Ajax and Flex Data Loading Benchmarks Exciting work from Adobe’s James Ward. There’s even an OSS project to run it on your own. Hopefully folks will add in things like Silverlight and other RIA/DHTML stacks. (tags: adobe redmonkclients benchmarks numbers ria dojo ajax flex xml) [...]
No. 11 — May 2nd, 2007 at 9:17 am
[...] AJAX and Flex Data Loading Benchmarks Flex apps run within the Tamarin VM inside Flash 9. Tamarin does JIT compilation which makes code execution 10-1000x faster than interpreted JavaScript in the browser. Also Tamarin can be very efficient when dealing with typed objects. (tags: AJAX Flex Dojo Benchmarks) [...]
No. 12 — May 2nd, 2007 at 2:08 pm
Hi James,
How are you ?
Good job, I will analyze and write down it into our magazine, my website as well.
Maybe next time, you can analyze WPF vs. Apollo ;-)
-Bing.
No. 13 — May 4th, 2007 at 7:28 pm
[...] James Ward’s Blog » Blog Archive » Ajax and Flex Data Loading Benchmarks A good performance comparison between Ajax and Flex using different data formats. Shows that Flex can be 10 to 1000 times faster. James is looking for more Ajax toolkits and options in the test, it’s completely open source so let’s help him out. (tags: adobe ajax dojo flex javascript performance ria xml grid) [...]
No. 14 — May 5th, 2007 at 6:18 pm
How about a Flex JSON test?
No. 15 — May 5th, 2007 at 10:19 pm
Hi Nick,
I originally had a Flex JSON test. It is still in cvs if you want to play with it. But I removed it because there were some problems with an early version of the JSON library I was using. I’ll add it back to the list for version 2.
-James
No. 16 — May 8th, 2007 at 2:14 pm
Hey James, this came in useful for a nice Ajax performance screenshot for my JavaOne presentation :)
I will try to grab the code and look into getting ain XSLT option n there as well since it should be pretty fast in IE but maybe not Firefox.
-dave
No. 17 — May 9th, 2007 at 1:39 am
[...] : Ajax and Flex Data Loading Benchmarks Fuente : James Ward’s [...]
No. 18 — May 10th, 2007 at 10:29 pm
Hi, James
I cannot retrieve the code for testing, from http://www.sf.net/projects/flexapps (cvs module is “census”). It denies the anonymous access (at least empty password does not work). Can you reopen it? Thanks!
Sheng
No. 19 — May 11th, 2007 at 10:57 am
Hi Sheng,
Anonymous CVS is open. Perhaps you can’t access port 2401 on flexapps.cvs.sourceforge.net due to a firewall. That would prevent you from being able to get to the files. Sorry. I will see if there is any way I can migrate everything to SVN.
-James
No. 20 — May 19th, 2007 at 1:30 pm
Hi James,
very nince tool!
I hava a question though…
The flex dataservices always have a server exec time of 1 or 0 ms. even if i retrieve pages of 1000 records. Is it that fast or is it a bug in your computations?
Really would like to know because we are probably going to use the dataservives in our project which has high performance requirements.
Thanks
Arnoud
No. 21 — May 19th, 2007 at 4:41 pm
Hi Arnoud,
That must be a bug. Does bandwidth come though? Both values are pushed from the server (or pulled if you can’t make the rtmp connection).
-James
No. 22 — May 21st, 2007 at 4:28 am
Hi James
the bandwidth is always 216 bytes independent of the page size.
Actually it looks just like your screendump :)
Arnoud
No. 23 — May 21st, 2007 at 6:21 am
Arnoud, It sounds like you have found a bug. The bandwidth and server exec time should vary depending on the benchmark and number of rows. Can you let me know what OS, browser, version of Flash, etc you are using? Thanks.
-James
No. 24 — May 26th, 2007 at 4:29 pm
[...] James Ward – RIA Cowboy » Blog Archive » Ajax and Flex Data Loading Benchmarks (tags: flex ria flash) [...]
No. 25 — May 28th, 2007 at 5:15 pm
Hi James,
I’m hearing from Alex Russell that the version of Dojo you’re using for comparison here is 0.4.1, is that right? There was a change made post 0.4.1 that implemented hashtable lookups for the store underlying the Dojo FilteringTable; we were kind of wondering if you’d be willing to pick up the latest version of Dojo and try it? Not that I expect the results to be too different; the reality is that I hit an essentially hard limit at about 450 rows of data before every browser decides it has enough…in this particular case the main issue seems to be rendering in a table element :( Either way, I never designed FilteringTable to be handling anything more than about 200 rows of data (for the aforementioned reason)…
Thanks!
No. 26 — May 28th, 2007 at 8:28 pm
Hi Tom,
I’ve updated to 0.4.3 and it’s certainly faster! About twice as fast as far as I can tell. So that’s great! Nice work!
And just so you know, I really don’t mean to bash on the FilteringTable or Dojo. I think both are great! It’s just that they aren’t well suited for dealing with large data sets on the client. (At least until Tamarin makes it into Firefox.)
So keep up the great work!
-James
No. 27 — May 28th, 2007 at 8:43 pm
Hi James–
Oh, don’t worry, we know :) Large datasets on the client using HTML tables get very unwieldy very fast…we’re looking at a paging solution in the near future that only renders a small amount of HTML table rows at a time, hopefully that will make a huge difference in performance.
–Tom
No. 28 — May 28th, 2007 at 8:51 pm
Thanks Tom. Rico does virtual renderers and paging which does greatly improve perceived performance. Let me know when you guys have it working and I’ll add it to the benchmark suite. Thanks again for the heads up and all the hard work on making this stuff better!
-James
No. 29 — May 29th, 2007 at 7:36 am
[...] Ward has created a nice little application for benchmarking Rich Internet Applications. And Flex sure is the winner, when it comes to separating content and layout. Serverside-generated [...]
No. 30 — May 29th, 2007 at 6:33 pm
[...] While AJAX has some minor advantages over the Flash platform (requires no plugin…and….um, I’m sure there are others), I usually find myself really having to dig to find any significant advantages in AJAX development. James Ward has put together a nice little Flex 2 app that gracefully shows performance comparisons between various data transfer methods. AJAX falls far short in these comparisons. View the application here: RIA Data Loading Benchmarks [...]
No. 31 — June 3rd, 2007 at 5:44 am
>Arnoud, It sounds like you have found a bug. The bandwidth and server exec time should
>vary depending on the benchmark and number of rows. Can you let me know what OS, >browser,
>version of Flash, etc you are using? Thanks.
>-James
Hi James,
A bit delayed but here are the details: flash 9.0.28.0 / windows xp home sp2, / firefox 2.0.0.3
Arnoud
No. 32 — June 5th, 2007 at 6:09 am
[...] times. Looks like Flex is pretty quick. The graphic below says it all, but you can find more here and use a live Flex app to demonstrate the [...]
No. 33 — June 17th, 2007 at 7:59 am
[...] results show why Flex is going to be so huge… check them out here and check out the rest of Jamse’s blog, he has a lot of good things to say. Posted by [...]
No. 34 — July 19th, 2007 at 9:25 am
[...] Of course AMF is teh bestest). [...]
No. 35 — August 28th, 2007 at 6:00 am
When I run the app the Flex AMF3 test shows up as 90% “transfer time” and just a tiny amout “server exec time” and “render time”, and while it is still faster than any of the other methods it is the slowest when only looking at only transfer time.
This can’t be right. Since AMF3 data is smaller in size than XML and JSON the time to transfer this between server and client should be the same or less, not more. Moreover, in my experience AMF3 coding is a lot more processing intensive on the server side than XML or JSON generation is.
How do you measure server exec time vs. transfer time? Could it be that the AMF3 encoding process is registered as transfer time for some reason?
No. 36 — August 28th, 2007 at 7:25 am
Hi Theo,
This has bothered me too. What I was told by the engineers is that there is actually some “parsing” or decoding of the AMF taking place before I can get an event raised that the response has returned from the server. However the server exec time should be accurate. I record the server exec time on the server through the servlet filter. When the request first comes into the server I start the clock and when the response is ready to be sent I stop the clock. That should be nearly 100% accurate. But the transfer time could be off if I can’t find out exactly when the response has been fully received by the client. Yet if I can’t get close enough to that event it doesn’t effect the total time measurement. I hope that helps. Feel free to check out the code on sourceforge and see if there is any way you can think of to make everything more accurate.
-James
No. 37 — August 28th, 2007 at 8:19 am
I wouldn’t be surprised if some of the transfer time is in fact the Flash Player doing some decoding, but doing it before it notifies the Flex application that the data is available. If that time is included in the transfer time I’m not surprised that it looks longer than it should.
I should have said in the last comment: good work, the cencus application is intuitive and also looks nice. Thanks.
No. 38 — December 19th, 2007 at 11:48 am
I also came to these comments to remark about the processing time of amf not registering, but I see the concern is already here =) There is a way you can measure this, in fact. Do your request with type AMF, that takes time T1 of ‘bandwidth + processing’. Do another request that loads the data as binary using sockets so that flash will not know the binary data is in AMF format and do any parsing on it. It will load this from the server in time T2. T3=T1-T2 is then the time that it took the client to process the AMF data.
No. 39 — January 4th, 2008 at 4:27 pm
[...] If you want a great examination of real-world capabilities, check out James Ward’s Ajax and Flex Data Loading Benchmark application, which tests data transfer benchmarks for a variety of technologies. Proof of Flex’s rising [...]
No. 40 — February 3rd, 2008 at 5:49 am
[...] para criação de aplicativos RIA. Você pode ver mais detalhes dos criterios usandos por ele neste link. del.icio.us | digg! | furl | reddit | rojo | yahoo! | rec6 | dzone | Bloglines | Eu [...]
No. 41 — March 11th, 2008 at 1:06 pm
[...] RIA Flex benchmarking [...]
No. 42 — April 28th, 2008 at 1:22 pm
Would it be possible to add a comparison using Curl (www.curl.com)? I am not familiar with Flex but would like to see the results.
No. 43 — April 28th, 2008 at 1:31 pm
Hi Friedger,
I’m actually working on that with the Curl developers. Should have something in the next few weeks.
-James
No. 44 — June 14th, 2008 at 3:46 pm
I could not find the census app module in sourceforge. All I could see is photographer_db, anvil, and flexpaint. Could you make the census app available for download ?
Thanks!
No. 45 — July 20th, 2008 at 2:15 pm
Hi Gunny,
It’s only in CVS.
-James
No. 46 — August 21st, 2008 at 3:09 am
[...] AMF??????????????????Ajax and Flex Data Loading Benchmarks | James Ward – RIA Cowboy??????????????????? [...]
No. 47 — August 29th, 2008 at 11:43 am
When running the app on your server (http://www.jamesward.org/census/), the clock just keeps spinning, no results ever show. Not sure what’s going on.
(using ff3.0.1, no firewall restrictions)
–Jordan
No. 48 — August 29th, 2008 at 7:58 pm
Hi Jordan,
Sorry for the problems. Unfortunately FF 3 has changed something about how it deals with IFrames. So to run the test you have to click on the results panel. This opens the IFrame and begins the test. I’m working on a new version which will resolve this problem.
-James
No. 49 — September 3rd, 2008 at 8:09 am
Thanks, that worked. Now I’m trying to get the Census app to build and run on my local machine… but I don’t seem to have a flex.war file any where on my system, and after much scouring via Google, I have no idea where that file may be (its a dependency in the build). Is it possible that Flex 3 just doesn’t have it? I don’t see any way to build the Flex sdk into that war file either. I do have the Flex sdk (a few versions) and Flex Builder, but no flex.war file.
Also, your instructions up near the top need to include installation of a laszlo server (and the env variable).
Thanks,
Jordan
No. 50 — September 3rd, 2008 at 3:14 pm
Hi Jordan,
I’m working on a new version that will be much easier to download and run locally. Until that is ready (at least a few weeks) you can try to get things compiling but it’s probably pretty painful. The flex.war file is part of LiveCycle Data Services. The next version will use BlazeDS (which also has a flex.war file).
-James
No. 51 — September 4th, 2008 at 7:40 am
Thanks for the update, I’ll be happy to see the next version. I did figure out what I was missing… in the new versions of LCDS, the file is called lcds.war (and there’s also lcds-samples.war) instead of flex.war (and samples.war). Changing the build file to reference lcds.war fixed one problem, but I was still receiving compile errors. It seems that flex-messaging.jar was renamed to flex-messaging-core.jar, so I changed that on (or near) line 173 in the build file for the javac task. Also, there is no longer a license.properties file in the lcds root, but creating a blank file seemed to work.
I just thought I would document some of this stuff in case others are still trying to build the old version. I have it building now,
–Jordan
No. 52 — September 5th, 2008 at 10:34 am
Thanks Jordan!
-James
No. 53 — October 30th, 2008 at 2:04 am
[...] http://www.jamesward.com/wordpress/2007/04/30/ajax-and-flex-data-loading-benchmarks/ [...]
No. 54 — November 20th, 2008 at 1:27 pm
[...] hats! He was looking pretty metro sexual for being the RIA cowboy! We are going to try and get census updated to compare Zend Amf vs java. Ohh that’s going to trigger some work on [...]
No. 55 — November 27th, 2008 at 7:10 am
@James Ward
Hi,
I am really interested in this new version since I too meet various problems trying to build and run census on a local machine. Do you have some kind of target date?
Thanks
No. 56 — November 27th, 2008 at 9:25 am
@Thomas
I’m still working on it. Now that we are in the holiday season it probably won’t happen until the beginning of 2009. Sorry.
-James
No. 57 — November 28th, 2008 at 1:52 am
@James Ward
No reason to be sorry. I’m just looking forward to seeing it.
Thanks for answering.
No. 58 — November 28th, 2008 at 10:14 pm
Hi James,
Can you clarify the following statement from your piece above…”If you control both sides of the wire in your application, then there is rarely a good reason to serialize and deserialize text.”
Are you saying that the best way to transfer data (text), when you control a webservices and Flex, is to query the data source, build the object server-side, and then transfer the object or an array of objects to Flex, which is then used to build the client-side objects in Flex for databinding? If so, when would you not use this method?
No. 59 — November 28th, 2008 at 10:56 pm
Hi Joey,
I’m saying that text is an inefficient serialization technology. When possible use binary serialization technologies like AMF.
-James
No. 60 — December 10th, 2008 at 3:48 pm
[...] Mihai showed some interesting slides about a comparison James Ward made with his Census application. He shows the speed improvements of AMF in favour of JSON communication when transferring 5000 rows and more. This is a very special use case, I rather see the latest usability and interaction patterns stearing into the direction of loading just the data on demand that are really required. In most cases this big number of data is also hard for users to grasp. There is charting and some visual presentations that might require this load of data, but the optimization on the server will probably serve a better user experience here. An interesting read is Jared Jurkiewicz blog article about comparing these numbers with Dojo today, as the comparison of James Ward was made in April 2007. [...]
No. 61 — February 24th, 2009 at 11:11 am
Hi James!
If you are interested, try also a modification to Flex (AS3) JSON parser for improved strings parsing speed. See following cookbook entry, solution 2:
http://www.adobe.com/cfusion/communityengine/index.cfm?event=showdetails&productId=2&postId=13226&loc=en_US
Thanks for this great comparison tool!
No. 62 — April 9th, 2009 at 7:42 am
Hi James!
Can you take a look at the Data you are using for your benchmark tool? It appears not to be bring back any data. I would love to show this to some of my colleagues.
Thanks!
No. 63 — April 10th, 2009 at 6:53 am
Hi Jason,
Firefox 3 broke something with the iFrames. So clicking on the “Output” pane will launch the test.
Let me know if that helps.
-James