Bursting Bubbles

Bubblemark is a popular benchmark for some of the RIA technologies including Flex, Adobe AIR, Ajax (DHTML), Java Swing, Java FX, Silverlight, etc. I’ve been trying for a while to create a new Flex version of Bubblemark to show just how fast Flash Player and Adobe AIR are. But I’ve come to a few realizations… First, you can make benchmarks say whatever you want them to say.

When trying to optimize Bubblemark I found a few interesting things. First was that IE (and some versions of Firefox) limit the frame rate of Flash Player (and possibly other plugins). This means that while the Flash Player VM might be able to actually achieve 200+ frames per second the actual visual result might be only 60 fps. And maybe this is for good reason. Why do you need a visual frame rate faster than the refresh rate on a monitor? You don’t. And especially not for RIAs.

Graphic rendering performance is certainly relevant for RIAs but that is only one factor which affects overall application performance. As Chet Haase points out, Bubblemark lumps a number of different factors – calculation speed, rendering performance, and timer resolution – into a single “frames per second” metric. That leads to my second realization.

The frame rate of bouncing bubbles isn’t very relevant to RIAs. As Josh Marinacci of Sun says – “there aren’t any real benchmarks yet for rich internet applications.” I think that my Census RIA benchmark is more pertinent to RIAs than bouncing bubbles. Census benchmarks how quickly an application can get data from a server, parse that data, and render the data in a datagrid. Sorting and filtering are also important benchmarks that I’m working on integrating into Census. Today the Census app has various Flex and Ajax tests but I’m also currently working on adding a Silverlight test, an Ext JS test, and an updated Dojo test. I’d love it if someone would create a Java FX version as well.

Just for kicks let’s go back to Bubblemark and see what I was able to come up with (with help from Chet Haase). I created two versions of the new Flex Bubblemark application. You can run each version in the browser (Flash Player) or on the desktop (Adobe AIR). The results will vary between the web and desktop versions due to the browser throttling. The first application will only move the bubbles once per frame. This is similar to the original Flex Bubblemark application. If the browser is only letting Flash Player run at 60 fps then the maximum fps will be 60. Adobe AIR seems to limit the fps to 250 but I haven’t yet confirmed that. With both of these versions CPU utilization should be pretty low since they are being artificially throttled. The second set of applications moves the bubbles as many times as possible per frame. The appearance of these might be choppy since the bubbles only get rendered after being moved potentially hundreds of times. The second set of benchmarks is more real world if you are interested in VM processing speed as opposed to resolution of the timing mechanism. However a third set of tests which is probably the most useful and that I haven’t yet written would calculate how many bubbles can be moved once per frame and maintain 60 fps. That kind of metric is probably relevant to RIAs that need to do a lot on every frame. But most RIAs I’ve seen are more concerned with the speed of data processing. Here are the new Flex Bubblemark applications:

Single Move Per Frame: Web | Desktop (requires Adobe AIR 1.0)
Many Moves Per Frame: Web | Desktop (requires Adobe AIR 1.0)

(Flex Bubblemark Source Code)

Benchmarks are only useful when you apply them to your scenario. Is the performance good enough for what you need? Are your users able to run what you build? There are thousands of RIAs which run in Flash Player and now hundreds of Adobe AIR applications. For the millions of users of these applications the frame rate is fast enough and the VM performance is superb. For the developers of these applications the Flex tooling enabled them to efficiently build these applications. Not only do the runtimes and the tools work on Windows, Mac, and Linux – the runtimes and core development tools are free. There is more to choosing a platform than the speed at which a runtime can bounce objects around the screen. Sorry if I’ve burst any bubbles.

This entry was posted in Adobe AIR, Flash Player, Flex. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.
  • http://neilmiddleton.com Neil Middleton

    Completely agree with you James, although there is one point that I need to make.
    Recently I have been looking at lots of Flex demos, and what you can do with the UI. A lot of these demos contain draggable panels and the like. One thing I have noticed is that sometimes (probably 50% of the demos), when dragging a panel (and it doesn’t have to be huge), there is a noticeable drop in frame-rate, making the whole thing feel a little tacky/sticky. Now, I’m not sure that this is not purely constrained to draggables, or the machines I have tried them on, but there does appear to be something there.

    For an example I found this: http://www.iloveflex.com/dragpanel/ which appears to be a little sticky on my 24″ iMac.

    After saying all this, I do feel that the performance of the Flash Player is more than good enough for 99% of use cases out there.

  • http://www.jamesward.org James Ward

    Hi Neil,

    Performance of those types of things could probably be improved by caching the thing being dragged as a bitmap. Usually the results are the same and performance is much better. I think that Flex’s built-in move effect does this by default.

    -James

  • http://graphics-geek.blogspot.com Chet Haase

    Neil,

    I’ve also had luck making dragged-object movement smoother by calling event.updateAfterEvent() in the mouse handling routine. This forces Flash to render the changes immediately instead of waiting for the next frame update. This is not typically necessary for regular animations, but really helps perceived performance when you can sync the drag motion to the mouse motion. Not sure that’s where your problems came from, but it seemed worth mentioning.

    Chet.

  • http://www.sakri.net/blog sakri

    Hi James,

    I actually ran into an uncomfortable situation last year while preparing to “show off” as3 performance over as2. I built two nearly identical “bouncing ball” tests in flash8 and 9, I wasn’t concerned about FPS, rather focusing on the number of balls, and to my disappointment, they both behaved more or less the same. If interested you can see the samples and source here: http://www.sakri.net/blog/2007/10/22/your-balls-werent-that-great-anyway/

    Like you said, the problems are with screen refresh… I guess if the math had been more complex, as3 would have been the victor… Still, this discovery was not cool just an hour before my presentation :D

    Cheers,

    Sakri

  • http://www.craftymind.com Sean Christmann

    Your example points out just how easily this test can be gamed by any vendor. You can go even further by blitting the balls directly to bitmapdata. Lee Thomason has discussed how drawing to bitmap runs independent of the player framerate and most likely the browser framerate. You might be interested in a post I just added over at http://www.craftymind.com/2008/04/11/why-bubblemark-is-a-poor-ui-benchmark/ that discussed other issues with the test and provides a possible alternative.

  • http://www.jamesward.org James Ward

    Thanks Sean. Perhaps you, Chet, and I can work together to create a more meaningful benchmark for RIAs. Maybe we could add some form of chart to my Census benchmark and track how long it takes to render some visualization of the 20,000 items of Census data. A column chart of the number of people in particular age groups for instance. Let me know what you think.

    -James

  • Pingback: Sébastien Letélié and Cyril Balit weblog - Java, Javascript, Eclipse, RCP, RIA, SOA, AJAX, WPF, Swing, SWT, JFace, Web Services, REST

  • Pingback: Sébastien Letélié and Cyril Balit weblog - Java, Javascript, Eclipse, RCP, RIA, SOA, AJAX, WPF, Swing, SWT, JFace, Web Services, REST

  • Pingback: Mark Finkle’s Weblog » Buublemark

  • Pingback: Craftymind » Blog Archive » Updated ‘Elastic Racetrack’ for Flash 9 and AVM2

  • Pingback: coderkind.com » Blog Archive » Flash performance in browsers and within AIR

  • http://kong.arcanecoder.com Jacob

    I’m glad to see another blog post on how the bubblemark stats are inaccurate. Before I found this on and the one at Craftymind, I put together my own version: http://kong.arcanecoder.com/misc/bubblemark.html

    I get a full 120 fps easily in FF for 128 spheres. Thats real fps too, with an ENTER_FRAME event; I’m not emulating a virtual fps or pushing multiple calculations per frame.

  • HSCharles

    I have a flash website
    i’m looking for the script who shows google adsense with flash.
    can you give me the link?

  • http://www.jamesward.org James Ward

    Hi HSCharles,

    I’m not sure.

    -James

  • Tony

    James, a few points:

    – Your Flex implementation rocks! I wish I will be that good one day.

    – Will the IE/Firefox limit on FPS still exists on AIR applications?

    – Can you recommend me a good book to become a serious Flex/AIR developer? I’m a traditional Java/Ruby server-side development guy, with some basic knowledge of JS and AJAX.

    Thank you!

  • http://www.jamesward.com James Ward

    Thanks Tony.

    AIR applications are still limited but not as low. I vaguely remember it being 250 fps.

    You can check out First Steps in Flex (by Bruce Eckel and me) or the numerous Flex books on Amazon.

    I hope that helps.

    -James

  • Julien Nicoulaud

    Really interesting, thank you.



  • View James Ward's profile on LinkedIn