Data Paging in Flex 4

I know — you’ve heard it from me before — AMF rocks! With AMF you can load massive amounts of data into your Flex (or JavaScript) apps very quickly. This can often obviate the need for paging data. But what if you have lots, and lots, and lots of data? Well then you should use data paging. And here is how…

There is a new collection wrapper class in Flex 4 called “AsyncListView“. The UI data controls in Flex 4 know how to handle an AsyncListView as a dataProvider. The purpose of the AsyncListView is to give you a callback when the underlying list throws an ItemPendingError. The ItemPendingError indicates that an item that the list thinks it has isn’t really there yet. This allows you to then load the data and update the list. In order to throw an ItemPendingError you need to keep track of which items haven’t been loaded and then, when an item is requested that isn’t really there, throw the ItemPendingError. Here is some code from my PagedList implementation:

public function getItemAt(index:int, prefetch:int = 0):Object
{
    if (fetchedItems[index] == undefined)
    {
      throw new ItemPendingError("itemPending");
    }
 
    return _list.getItemAt(index, prefetch);
}

In my main application I just create a PagedList, set its length (which should really be done by a remote call instead of manually), and then assign the instance of PagedList to the list property on my instance of AsyncListView. With MXML this looks like:

<local:PagedList id="items" length="100000"/>
<s:AsyncListView id="asyncListView" list="{items}"
    createPendingItemFunction="handleCreatePendingItemFunction"/>

When the ItemPendingError is thrown, the handleCreatePendingItemFunction is called. Now I just figure out what page of data is needed, make sure that there isn’t already a pending request for that page, and then make the request. When the response comes back I simply update the underlying collection. Here is the code that does that:

private function handleCreatePendingItemFunction(index:int, ipe:ItemPendingError):Object
{
  var page:uint = Math.floor(index / pageSize);
  if (fetchedPages[page] == undefined)
  {
    var numItemsToFetch:uint = pageSize;
    var startIndex:uint = pageSize * page;
    var endIndex:uint = startIndex + pageSize - 1;
    if (endIndex > items.length)
    {
      numItemsToFetch = items.length - startIndex;
    }
    var asyncToken:AsyncToken = ro.getElements(startIndex, numItemsToFetch);
    asyncToken.addResponder(new AsyncResponder(function result(event:ResultEvent, token:Object = null):void {
      for (var i:uint = 0; i < event.result.length; i++)
      {
        items.setItemAt(event.result[i], token + i);
      }
    }, function fault(event:FaultEvent, token:Object = null):void {
    }, startIndex));
    fetchedPages[page] = true;
  }
  return null;
}

In this example I’m using RemoteObject (AMF) but this could be anything (HTTPService, WebService, etc.) as long as there is a method on the back end that lets me set the starting location and the page size.

Here is a simple demo of data paging using the new Spark DataGrid in Flex Hero. The page size is 100 and there are 100,000 total items.

Fork or view the code for this example on github.com.

Of course another option for doing data paging more automatically is with LiveCycle Data Services. Check out a great example of Data Paging with LCDS in Tour de Flex to learn more about that.

A big thanks to Mike Labriola for helping me figure this out! Please let me know if you have any questions about this.

This entry was posted in Flex and tagged . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.
  • HERLoct_HENT

    Thanks for your great example of AsyncListView. Is it possible to do it in Helo Datagrid and/or Spark List/DropDownList??? I have tried some AsyncListView samples on Spark List but the view isn’t refreshed until some user interaction like clicking on the item. It works well in Helo Datagrid and Helo List though.

    The sourcecode that i’ve tried : http://blog.vihinen.net/2008/11/lazily-loading-list-items-in-flex.html

    Thanks before

    • akhand

      Some of the rows remains null on the datagrid dataprovider although the data on PagedList is set to the value. What would I be doing wrong?

  • Carrera PHP

    Hi James,

    The Mouse Wheel event for the Spark DataGrid is not working. I tried selecting a row inside the grid and scroll using the mouse wheel but it didn’t work.

    Carrera PHP

    • http://www.jamesward.com James Ward

      This is a known bug on Mac. But I believe it was fixed in recent versions of Flash Player and Safari. What browser / version of FP are you using?

      • carrera PHP

        Am using FP10.1 in Google Chrome on Windows XP SP3. Sorry, i forgot to mention in my last comment “Thanks for the post, This is the first blog post i am seeing spark datagrid in action.”

  • Richard Rodseth

    Nice example. It would be great to see one with sorting. Presumably the ideal would be to have sorting happen client-side if all items are present locally, but hit the server if they aren’t?

    • http://www.jamesward.com James Ward

      That would be great! I’ll try to work on that. But in the mean time, feel free to give it a try yourself. :)

      • Richard Rodseth

        Oh, and another nice one is “Select All and operate on the selection”.

  • http://www.portalscreen.com Francisco Castro

    An example very clear and helpful. One question, what happens if the data included in the list are graphic objects? Is there a problem with the rendering of flex?, Thanks in advance. Greetings

    • http://www.jamesward.com James Ward

      Should work fine.

  • Pingback: Handling with large amounts of data in Flex part 1: loading and parsing « Claudiu Ursica

  • Jo

    When i try to remove an item (pagedList.removeItemAt(…)) from the pagedList i always get RangeErrors from PagedList/getItemAt() function. (Stacktrace: http://pastie.org/1370418)

    This is what i’m doing:
    – User selects an item and clicks remove button
    – a service deletes the item in the database
    – in the result event handler i do the following

    var i:int = model.pagedList.getItemIndex(deletedItem);
    if (i > -1) {
    model.pagedList.removeItemAt(i);
    }

    Do i have to decrement the pagedList length by myself? Do i need to call asnyListView.removeItemAt or pagedList.removeItemAt?

    Thanks!

    • http://www.jamesward.com James Ward

      Yes, you need to update the pagedList’s length. You can do this in PagedList’s removeItemAt method. But don’t call the PagedList’s length setter or it will reinitialize the underlying ArrayList.

  • Pingback: Daniel Harfleet

  • http://flexblog.faratasystems.com/ Valery Silaev

    James,

    Thank you for detailed example.

    I have a question to you and other readers: did anyone try AsyncListView with Spark List (rather than DataGrid)?

    From my experiments it follows that List, even with default single-field renderer, is noticeably slower than DataGrid… I mean, you literally can see how it’s slow when you scroll with thumb by large deltas. It’s very surprising result if we consider the extra-amount of work performed by DataGrid to display several columns. Either I’m doing something wrong, or VerticalLayout/HorizontalLayout are not optimized well.

    • http://www.jamesward.com James Ward

      Wow! Just tried a List and it definitely has some performance issues. I filed a bug:
      https://bugs.adobe.com/jira/browse/SDK-29819

      Please go vote for it.

      Thanks!

      • http://flexblog.faratasystems.com/ Valery Silaev

        Adobe Flex team is amaizing! Already verified, fixed and approved for inclusion in next SDK drop!

        • http://www.jamesward.com James Ward

          Well… It’s not really fixed. They just made it possible to fix through extension and overriding a method that was previously private.

  • Chris

    Thanks for posting this. I was able to get this working perfectly with an advanced data grid using a your IList impl for the data provider. Unfortunately, it looks like I’m going to be required to use HierarchicalData as the data provider and I have not been able to find a way to get lazy load to work via the AsyncListView when used in combination with HierarchicalData as the data provider. Any thoughts? Can these two things play nice together?

    • http://www.jamesward.com James Ward

      I’ve heard that should work but I’ve never tried it. Sorry. Let me know if you figure it out!

      • Mark

        Thanks for the great tutorial! I am trying to get this approach to work with HierarchicalData. It appears that the createPendingItemFunction is being called after data is initially loaded, unlike in your example where createPendingItemFunction is called when new data is requested. I was wondering if you might have any insight into how to get this example to work using HierarchicalData.

        • http://www.jamesward.com James Ward

          Hmm… Interesting. I’m not really sure why HierarchicalData would work differently. Perhaps it’s a bug? Want to file it? http://bugs.adobe.com/flex

  • Pingback: 1 million row DataGrid: ZendAMF, Flex data-paging, MySql « Bishop on Development

  • Drew Pierce

    Thx James for this excellent example.

    1) what is the meaning of just the return{} stmt in the source code for fcn handleCreatePendingItemFunction when above you show a different fcn with a dozen or so commands and a call inside of it to ro.getElements ?

    2) could you give an example of the mxml declaration for calling a wsdl fcn. i can’t seem to put it together manually. I guess it would call your

    http://www.jamesward.com/census2-tests/services/CensusSOAPService?wsdl

    most of my wsdl calls are baked inside of super classes using Data / Connect to Data Service / Webservice Wsdl

    thx

    • http://www.jamesward.com James Ward

      The return statement in handleCreatePendingItemFunction is the same as:
      return new Object();

      It just gives the DataGrid a placeholder object to use while the pending item is being fetched.

      For my SOAP endpoint it would be pretty similar:
      <s:WebService id=”ro” wsdl=”http://www.jamesward.com/census2-tests/services/CensusSOAPService?wsdl”/>

      • Maurice

        You could also return {id: “Loading…” } instead of {} to get a better visual feedback of loading items.

  • Tom Chiverton

    I converted this to use an MX DataGrid so as not to require the very latest Player, and I noticed that when sorting the DataGrid, it retrieves all rows (bad ;-). You can get around this, but requires mx_internal to set the sort indicator correctly.

    <mx:DataGrid dataProvider="{asyncListView}" width="100%" height="100%"
    				 headerRelease="table_headerReleaseHandler(event)">
    var asyncToken:AsyncToken = ro.getElements(token.text, start, count,orderby);
    protected function table_headerReleaseHandler(event:DataGridEvent):void
    {
    	var o:int=pagedList.length;
    	asyncListView.removeAll();
    	pagedList.removeAll();
    	table.selectedIndex=0;
    	pagedList.length=o;
     
    	_remoteOrderByDirection= (_remoteOrderByDirection != "asc")?"asc":"desc";
    	orderby = event.dataField+" "+_remoteOrderByDirection;
     
    	event.preventDefault();//default is to load all rows. let's not. PagedList will take over and just load ones we need.
     
    	table.mx_internal::sortIndex=event.columnIndex;
    	table.mx_internal::sortDirection=_remoteOrderByDirection.toUpperCase();
    }
    private var orderby:String = "&";
    private var _remoteOrderByDirection:String ;

    Aside: *do not* just take the “orderby” string and shove it into the SQL. Injection attacks kill kittens !

    • Maurice

      Hi, I have tried the code above, also on an mx:DataGrid, with little adaption to connect to my own remote service.
      Clicking the header does trigger the external sorting as expected. Thzt’s fine.

      However, the header does not display the sort arrow corresponding to the current sort.
      Ihave done some investigation and found out that DataGrid.updateSortIndexAndDirection() recomputes sortIndex and sortDirection based on the sort field of the DataGrid’s dataProvider.
      Since it’s not set, then sortIndex is set back to -1, although it was set to event.columnIndex in the code above.

      What do you think? did you have the same behavior ?

      • Maurice

        FYI, I have implementated a subclass of mx:DataGrid to overcome this limitation.
        It overrides collectionChangeHandler() to force sortIndex and sortDirection after they have been cleared by DataGrid.updateSortIndexAndDirection()
        It also handles internally the column header click and dispatches a new “externalSort” event [SortEvent( sortField, sortDirection)] that can be handled to do the external service call.

        This makes the code more readable.

        • Maurice

          If you are interested, I can clean up the code and post a link here

          • Tom Chiverton

            That would help other people out I’m sure.

  • Maurice

    what is token.text? the variable does not exist

    • Tom Chiverton

      Token.text is passed into the AsyncResponder as ‘startIndex’.

  • Maurice

    Hi James, very nice example; I tried it with mx:DataGrid and I worked fine.
    I have one question though: what do we need to create a Vector the size of the whole data and set every item to undefined (cf. PagedList.length) ?
    This could be a memory issue if we have dozen millions of items, and will likely display only hundreds of them.
    I will try to change the implementation to use a sparse vector (ie Array) and set items in the Array only when they are requested? Do you think it is going to work?

    • http://www.jamesward.com James Ward

      I don’t think it necessarily needs to be initialized for the whole data set. But I haven’t tried.

  • Maurice

    Hi Again,
    I have tried the DataPaging project standalone and it worked fine. But when I included the PagedList class into my own project (usnig Flex SDK 4.1, SpiceFactory Parsley 2.3.2 and Cairngorm 3), I am getting this very strange error (messages are translated from French) :

    [compc] Loading configuration file C:\Program Files\Adobe\Adobe Flash Builder 4 Plug-in\sdks\4.1.0\frameworks\flex-config.xml
    [compc] Unexpected multiname type: 16
    [compc] Unexpected multiname type: 16
    [compc] D:\Fx\EclipseWorkspace\BBrFxCore\bin-release\FxCore.swc(PagedList)
    [compc] Error 1046: Type was not found or was not a compile-time constant .

    When I remove the PagedList class, it compiles succesfully.

    Any idea?

    • Tom Chiverton

      When I get that, doing a clean compile clears it (assuming you’re not just missing an include).

      • Maurice

        Yes , doing a clean compile removes the error. It just that it happens again an again (every 2 or 3 compiles) and I have to clean every time.
        Could it be a bug in the compc compiler?

        • Tom Chiverton

          All I can think of is it’s related to one of the lines where James does :
          const newLength:int = value;
          and similar, where 99% of the code I see uses ‘var’.

          Smells like a compiler issue, tbh. Did you try Flex 4.5 ?

          • Maurice

            Definitely a compiler issue.
            The error disappeared in FB 4.5.

          • Maurice

            Just after I wrote that, the error also appeared in FB4.5. Damn!

    • Maurice

      I forgot something: my project is also using Efflex_v0.04.swc and there was a naming conflict with some classes in this lib. When I remove the lib, compiles is OK.

      • Tom Chiverton

        Unless that .swc includes the same class (in which case take it out of the project source) it shouldn’t conflict as long as you’ve got the correct XMLNS. Without seeing your minimal source code it’s kinda hard to know.

    • http://flexdiary.blogspot.com Amy

      I get a different compiler error type not found or is not a compile time constant “.”, and there are orange squiggly lines under anything referring to a property or method of the data Vector. This goes usually away if I clean the project, but returns every 3rd or 4th compile. It may be related to one of the following:

      I’m using Flex 4.1 SDK with FB 4.5.
      I have put the PagingList in a Library Project, in a com.jamesWard.pagingData package.

      Thoughts?

      Amy

      • Tom Chiverton

        I can make this go away by doing both of :
        1) change all the const to var (methods and variables)
        2) rather than using s:Declaration, create a private variable to hold the instance, and use the BindingUtils to set it up

        I’m 75% sure only 1) is required but have no idea why, and don’t have time at the moment to track down exactly which line it is that trips the compiler up.

        • http://flexdiary.blogspot.com Amy

          Thanks, Tom.

          I can now get rid of my “proof of concept” test project that used bindings, and my real project just puts the PagedList into the AsyncListView once enough information is known for the createPendingItemFunction to work (what I’m doing is just in time parsing of XML rather than going to a server, so the main thing I’m using the PagedList for is just to tell the View how many items there are to show the scrollbars or not). So I’ll try that part first.

          If it doesn’t work, then I’ll fall back on editing the Class (since it’s not my Class, it feels somehow invasive to do so) :-)

          Thanks!

          Amy

        • http://flexdiary.blogspot.com Amy

          OK, I tried both of these, and neither worked. What has worked so far (fingers crossed) is simply changing the data type of that variable to Array.

  • Maurice

    one suggestion:
    PagedList.storeItemsAt() just fails if the items are out of range.
    However, if some of the items are still within range, it would be preferable to update just those items and optionally throw an error.

    What do you think?

    • Tom Chiverton

      If you have items arriving off the end of the “total length” number, something is very wrong !

      • Maurice

        I don’t agree. This is the scenario (that is specific to my app, but no unusual):
        I send an rpc call to get a range of items that will return both the page of items and the total length of the list.
        The list of items dynamically evolve over time. So the total length at time T may be different from total length at time T+10.

        So what I do in the result handler of the rpc call is the following:
        pagedList.length = virtualSize ;
        pagedList.storeItemsAt( pageOfItems , startIndex);

        Now, if the user scrolls to the very end of the list, I request the last page of items, but could get a new total length that is slightly lower than the previous one. In this case, I should partly fill the list with only the items that have been retrieved.

        pagedList.length = virtualSize ;
        var partLength = virtualSize % pageSize ;
        subPageOfItems = pageOfItems.slice(0,partLength ) ;
        pagedList.storeItemsAt( subPageOfItems , startIndex);

        So my request was to include the slice statement in storeItems (..).

        but really it’s not a big deal.

        • Maurice

          Sorry, forget about this reply. There was an error in my reasoning.

        • Tom Chiverton

          Oh, isn’t it easier to do :
          …. in the result handler….
          //as now
          var v:Vector. = new Vector.();
          for each (var i:Object in value.data)
          {
          v.push(i);
          }
          //new
          if ( value.start + v.length > _listData.length ){
          _listData.length= value.start + v.length;
          }
          //as now
          _listData.storeItemsAt(v, value.data.typesAndCols.start);

    • http://www.jamesward.com James Ward

      Another option would be to use a Dictionary for storage of the retrieved items.

  • Pingback: Flex 4.5 MySQL Query Browser

  • Jay

    Hi James,
    Thanks for a great example. I was wondering how do I trigger this in AS. In your example DataGrid is an mxml object so I assume that loadItems function is called on completion in my case DataGrid is not a mxml object but, added to the stage using AS, so how and at what point do I trigger loadItems()

    Hope this makes sense

    Thanks
    Jay

    • Tom Chiverton

      As long as you set up your DataGrid’s dataProvider to be bound to an AsyncListView it doesn’t matter if you use MXML or AS. BindingUtils is the class you want if you are not familiar with this.

  • Jarno

    How does this could be implemented if there are two different rpc call, of which returns total count in back end and other rpc call, of which retrieves the real data? As Maurice said, pagedList lenght could be change if i need to search data with search criteria.

    • Tom Chiverton

      Works fine.
      ‘Just’ return the length with your first page of results, and update the length of the AsyncList when you see the extra part of the result structure.

  • Hadrien

    Hi James,

    Just a little question about your solution, i’m new with AsyncListView and i would like to know why you don’t use a classic ArrayCollection instead of PagedList ?

    Thanks
    Hadrien

  • chris f

    Sorry to resurrect an old posting, but I like the approach. I was wondering if you’ve had this issue.
    When adding my very first object to the PagedList, I get this error:
    RangeError: Index ’0′ specified is out of bounds.

    I’m using an HTTP service, from which I retrieve Objects that are JSON encoded. Once I get that array, I try to push them , one by one, into the internalList.

    Did I miss something super important?

    • chris f

      my XML is this:
      (didn’t show up above, i’ll try to figure this one out)

      utils:PagedList id=”internalPacketList” pageSize=”1000″ length=”100000″ loadItemsFunction=”loadItems”

  • http://flexriadev.blogspot.com Raghupathi Reddy

    Hi James,

    Thanks…

    I am having an issue if i use this with robotlegs framework.

    If i use this PagedList.as file & try to save some other as file, it is throwing the below error.

    Error 1131: Classes must not be nested.
    abc bytecode decoding failed on PagedList.as file.

    If i clean the project, then these errors will not be there.

    Please help me asap.

    • Tom Chiverton

      While I’ve no particular need or want to help you ‘asap’, the fix is outlined above already.

      • Raghupathi Reddy

        Thanks Tom. I got the solution.

  • buddy

    Hi James,

    Not a right question for this blog, but u may help me. Very simple for u. what if i m using BlazeDS callresponder and async services and value objects, can i use it? what is the difference between using Remote Object and using BlazeDS, using those services?

    • Tom Chiverton

      BlazeDS provides the server-side connection from Flex to Java (or ColdFusion) services. If you want to use a REST service in PHP or something, go right ahead, this code is agnostic.

  • Jamie

    James,
    Looking for some help here. I’ve been working with your data paging example for a while (it’s awesome btw) and suddenly in stopped working for me. After hunting around for a while I realized I had recently upgraded to flash player 11. I uninstalled and downgraded to 10.3 and sure enough, works great.

    It seems that in flash player 11 the loadItemsFunction on the PagedList class is not firing. Do you have any advice on how to resolve this?

    I’m in the processes of upgraded my entire application for my company to Flex 4.5 (from Flex 3) solely for the purpose of being able to use live date paging so I just want to make sure I can resolve this before moving forward.

    Any advice would be greatly appreciated.

    Thanks,
    ~Jamie

    • Tom Chiverton

      We’re using a slightly modified version (the const Vector of items replaced as above) with Flex 4.0 and it’s fine with the latest (debug) v11 Player, if that’s any help.

      • Jamie

        Thanks Tom, I’ll check it out.

      • Joshua

        Hello Tom,

        I have the same issue. Could you please post your modified version or email it to me.

        Thank you

        • Tom Chiverton

          Change it to read like this rather than Vector:
          .
          .
          .
          /**
          * @private
          * The IList’s items.
          */
          private const data:Array = new Array();
          .
          .
          .

          • Joshua

            Thanks Tom,

            It works!

            p.s. Flex 4.6 & Air 3.1

  • Sondang

    Hi.. james
    I have question.. i m have project trading application… using as3 socket and datagrid.
    with huge data save to arrayCollection bind to datagrid.
    Every millisecond/second Data is updated .
    my app is very slow…
    how to Async refresh datagrid data or using AsyncListView can help ?
    Or you have solution ??
    Thanks for help…

  • pbesi

    Thank you James, great example!
    Anyway now there are some problems with the flash player 11. More precisaly, when you create the asyncToken related to the remote method, it seems nothing happens, debugging the java code, it seems the remote method doesn’t start. Working with flash player 10.x everything works fine. Please, do you have any idea? Thank you

  • paul

    Hi James,

    Is it possible to perform lazy loading using FLEX 3? I have a FLEX application that we wish to expand and allow users to return database queries to the screen. We would like to keep this all within the current FLEX framework – but Im struggling to see an example anywhere.

    Thanks

  • Luke Davidson

    This thread might be dead, but there’s something I really can’t understand.
    In your PagedList class, in the addItemAt() method, you call splice to add the item to the vector.  Why is index the first and second parameter?  Wouldn’t that remove index number of items?  If you’re really going to add an item, and increase the length by 1, wouldn’t you want the second parameter of splice to be 0?

    It might not matter much as the splice method seems to have been depreciated from the flex vector class, but I’d really like to know what’s going on here.  It seems to be an important gap in my understanding of flex.

  • http://www.facebook.com/marius.pena Marius Pena

    has anyone ecountered this error ?
    Error: invalidIndex
        at spark.layouts.supportClasses::LinearLayoutVector/remove()[E:dev4.yframeworksprojectssparksrcsparklayoutssupportClassesLinearLayoutVector.as:541]

    it happens when pagedList.length > pagedList.pageSize
    PS: I setup pagedList.length prior to making the call for paged items.

  • Dennis Flanagan

    James,

    Thank you for this posting!  I know I am two years late, but this was highly educational.

  • Markzolotoy

    How can I run it against data provided in the example?

  • Jessewk

    2 bugs and a comment:

    1. Luke Davidson is correct in his comment above: addItemAt() should use data.splice(index, 0, item)

    2. The setter for length overwrites the last loaded item with undefined when extending the length.  newIndex:int = Math.max(oldLength – 1, 0) should be newIndex:int = Math.max(oldLength, 0)

    Comment: The data const/var should be declared as a protected so that it can be accessed by extending classes. I had to extend PagedList to add some functionality and fix the length bug above and had to overwrite your code. I think editing files that were not produced in-house is generally bad form.

  • bizops

    1. I tried this example with  Flash Player 11.2 and it doesn’t work.  It works very well with Flash Player 10.
    Has anyone tried with latest flash player version?

    2.  I want it to use this to show Hierarchical data in Advanced Datagrid, any pointers to do it smartly?

    • http://www.facebook.com/shavenzov Denis Shavenzov

      Try to replace all undefined values to null in PageList.as. In my case it works.



  • View James Ward's profile on LinkedIn