Building JavaScript Client Apps with gulp

Yesterday I started playing around with gulp which is a build tool for JavaScript applications. My goal was to have a build for a fully client-side application built with AngularJS and Bootstrap. In modern JavaScript apps we now need a build tool to compile various pieces, manage libraries, do production transformations (like minification), and to provide a nice development cycle when working on a project. I created a Gulp Starter project that has the following:

  • Very simple example AngularJS & Bootstrap app
  • Asset pipeline for LESS, JavaScript, images, and HTML that does basic compilation and syntax checking in development mode and minification for production mode
  • Development server with LiveReload integration
  • Production server used to test the production assets
  • Transparent Bower integration (for managing client-side dependencies)

Check out a seven minute walk-through of the Gulp Starter project:



Now to get started check out the README in the Gulp Starter GitHub project.

I’m new to this world so I’d love any feedback you have. Thanks!

Official Support for RequireJS in WebJars

WebJars Locator now has direct support for RequireJS! For a while the WebJars Play helper library has had support for RequireJS which was great but it had a few issues and wasn’t useful outside of Play.

There is now a org.webjars.RequireJS.getSetupJavaScript(String webjarUrlPrefix) method that produces the RequireJS configuration for the WebJars you are using. If you are using only the jQuery WebJar then that JavaScript configuration looks like:

var webjars = {
    versions: { 'requirejs': '2.1.10', 'jquery': '2.1.0' },
    path: function(webjarid, path) {
        return '/webjars/' + webjarid + '/' + webjars.versions[webjarid] + '/' + path;
    }
};
 
var require = {
    callback: function() {
        // no-op webjars requirejs plugin loader for backwards compatibility
        define('webjars', function () {
            return { load: function (name, req, onload, config) { onload(); } }
        });
 
        // all of the webjar configs from their webjars-requirejs.js files
 
        // webjar config for jquery
        requirejs.config({
            paths: { "jquery": webjars.path("jquery", "jquery") },
            shim: { "jquery": { "exports": "$" } }
        });
    }
}

This sets up the paths for RequireJS modules in WebJars and pulls in any other custom configuration from the WebJar’s RequireJS config. To use this setup JavaScript with RequireJS you can add something like the following to your web page:

<script>
// the following includes the setup JavaScript directly in the page
// (in this case using Play's server-side templating)
@Html(org.webjars.RequireJS.getSetupJavaScript("/webjars/"))
 
// alternatively you can create a controller that returns the JavaScript
// and then use a src="blah" to load the setup JavaScript
</script>
 
<!-- The following can use the WebJars Locator to avoid hard-coding the full path -->
<script data-main="/assets/index" src="/webjars/requirejs/2.1.10/require.min.js"></script>

The WebJars Play library now has a new helper that wraps all that up into a nice function:

@Html(org.webjars.play.RequireJS.setup("index"))

That loads the setup JavaScript and loads RequireJS with the right data-main. Check out a full example of this with AngularJS in the Play Angular Seed Activator template.

Now using RequireJS is just normal syntax (no more webjars! prefix), like:

require(["jquery"], function(jquery) {
    console.log(jquery)
});

All of this was added very recently so make sure you are using at least webjars-locator version 0.12 and webjars-play version 2.2.1-2. Also this change may cause some incompatibilities in the WebJar RequireJS config files (webjars-requirejs.js) so if you experience strange behavior please file an issue on the WebJar.

Let me know how it goes!

OSCON Workshop: 6 Minute Apps! Build Your First Modern Web App

Next week at OSCON 2013 in Portland I’ll be leading a hands-on workshop on Monday at 1:30pm where we’ll build a modern web app with Play Framework and Scala. Here are the details:

6 Minute Apps! Build Your First Modern Web App

The web application landscape is rapidly shifting back to a Client/Server architecture. This time around, the Client is JavaScript, HTML, and CSS in the browser. The tools and deployment techniques for these types of applications are abundant and fragmented.

This session will teach you how to pull together jQuery, Bootstrap, and some CoffeeScript to build the Client. The Server could be anything that talks HTTP, but this session will use the Play Framework with Java and Scala to build JSON REST services.

Bring your laptop so you can follow along and walk away with a fully functional modern web application!

Hope to see you there!

Dynamically Rendering GitHub Files in Web Pages

You know how you can easily embed GitHub Gists into a webpage? I’ve always wanted that for any file on GitHub. I post a lot of code on my blog and it’s always tedious and error-prone having to copy and paste the code. I’d rather be able to dynamically render a specific version of a file hosted on GitHub. So I created a little JavaScript jQuery plugin called github-files that pulls blobs from GitHub. Those blobs can then be rendered client-side and optionally syntax highlighted.

To use github-files, get the git sha for a file:

git rev-parse HEAD:src/main/javascript/github-files.js

Then you can grab the file from GitHub in JavaScript:

$.getGithubFile("user", "repo", "sha",
    function(contents) {
        console.log(contents)
    }
)

Here’s the source for github-files.js, fetched dynamically using github-files.js and rendered with highlight.js:


This uses GitHub’s JSONP API to pull the file in. It’s pretty simple and could use a few more features but it’s a start! Matt Raible and I are using this for our upcoming ÜberConf presentation to render code in our reveal.js preso. Hopefully this is useful for others. Let me know what you think.

Getting Started with Node.js on The Cloud

In my new job at salesforce.com I’m incredibly exited about getting into Heroku, a Platform as a Service provider / Cloud Application Platform. In a future blog post I’ll provide more details on what Heroku is and how it works. But if you are like me the first thing you want to do when learning a new technology is to take it for a test drive. I decided to take my Heroku test drive using the recently announced Node.js support. I’m new to Node.js, but at least I know JavaScript. Heroku also offers Ruby / Rails support but I don’t know Ruby – yet. So let me walk you through the steps I took (and that you can follow) to get started with Node.js on the Heroku Cloud.

Step 1) Sign up for Heroku

Step 2) Install the Heroku command line client

Step 3) Login to Heroku via the command line:

heroku auth:login

Step 4) Install git

Step 5) Install Node.js

Step 6) Create a Node.js app

I started by building a very simple “hello, world” Node.js app. In a new project directory I created two new files. First is the package.json file which specifies the app metadata and dependencies:

{
  "name": "hellonode",
  "version": "0.0.1",
  "dependencies": {
    "express": "2.5.11"
  },
  "engines": {
    "node": "0.8.4",
    "npm": "1.1.45"
  }
}

Then the actual app itself contained in a file named web.js:

var express = require('express');
 
var app = express.createServer(express.logger());
 
app.get('/', function(request, response) {
  response.send('hello, world');
});
 
var port = process.env.PORT || 3000;
console.log("Listening on " + port);
 
app.listen(port);

This app simply maps requests to “/” to a function that sends a simple string back in the response. You will notice that the port to listen on will first try to see if it has been specified through an environment variable and then fallback to port 3000. This is important because Heroku can tell our app to run on a different port just by giving it an environment variable.

Step 7) Install the app dependencies with npm:

npm install .

This uses the package.json file to figure out what dependencies the app needs and then copies them into a “node_modules” directory.

Step 8) Try to run the app locally:

node web.js

You should see “Listening on 3000″ to indicate that the Node.js app is running! Try to open it in your browser:
http://localhost:3000/

Hopefully you will see “hello, world”.

Step 9) Heroku uses a “Procfile” to determine how to actually run your app. Here I will just use a Procfile to tell Heroku what to run in the “web” process. But the Procfile is really the foundation for telling Heroku how to run your stuff. I won’t go into detail here since Adam Wiggins has done a great blog post about the purpose and use of a Procfile. Create a file named “Procfile” in the project directory with the following contents:

web: node web.js

This will instruct Heroku to run the web app using the node command and the web.js file as the main app. Heroku can also run workers (non-web apps) but for now we will just deal with web processes.

Step 10) In order to send the app to Heroku the files must be in a local git repository. To create the local git repo, run the following inside of your project directory:

git init

Now add the three files you’ve created to the git repo:

git add package.json Procfile web.js

Note: Make sure you don’t add the node_modules directory to the git repo! You can have git ignore it by creating a .gitignore file containing just “node_modules”.

Commit the files to the local repo:

git commit -m "initial commit"

Step 11) Create an app on Heroku:

heroku create

A default / random app name is automatically assigned to your app.

Step 12) Now you can push your app to Heroku! Just run:

git push heroku master

You should see something like:

Counting objects: 8, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 729 bytes, done.
Total 6 (delta 2), reused 0 (delta 0)
 
-----> Heroku receiving push
-----> Node.js app detected
-----> Resolving engine versions
       Using Node.js version: 0.8.4
       Using npm version: 1.1.41
-----> Fetching Node.js binaries
-----> Vendoring node into slug
-----> Installing dependencies with npm
       npm WARN package.json hellonode@0.0.1 No README.md file found!
       npm http GET https://registry.npmjs.org/express/2.5.11
       npm http 200 https://registry.npmjs.org/express/2.5.11
       npm http GET https://registry.npmjs.org/express/-/express-2.5.11.tgz
       npm http 200 https://registry.npmjs.org/express/-/express-2.5.11.tgz
       npm http GET https://registry.npmjs.org/connect
       npm http GET https://registry.npmjs.org/mime/1.2.4
       npm http GET https://registry.npmjs.org/qs
       npm http GET https://registry.npmjs.org/mkdirp/0.3.0
       npm http 200 https://registry.npmjs.org/qs
       npm http 200 https://registry.npmjs.org/mkdirp/0.3.0
       npm http 200 https://registry.npmjs.org/mime/1.2.4
       npm http 200 https://registry.npmjs.org/connect
       npm http GET https://registry.npmjs.org/qs/-/qs-0.4.2.tgz
       npm http GET https://registry.npmjs.org/connect/-/connect-1.9.2.tgz
       npm http GET https://registry.npmjs.org/mkdirp/-/mkdirp-0.3.0.tgz
       npm http GET https://registry.npmjs.org/mime/-/mime-1.2.4.tgz
       npm http 200 https://registry.npmjs.org/qs/-/qs-0.4.2.tgz
       npm http 200 https://registry.npmjs.org/mime/-/mime-1.2.4.tgz
       npm http 200 https://registry.npmjs.org/connect/-/connect-1.9.2.tgz
       npm http 200 https://registry.npmjs.org/mkdirp/-/mkdirp-0.3.0.tgz
       npm WARN package.json connect@1.9.2 No README.md file found!
       npm http GET https://registry.npmjs.org/formidable
       npm http 200 https://registry.npmjs.org/formidable
       npm http GET https://registry.npmjs.org/formidable/-/formidable-1.0.11.tgz
       npm http 200 https://registry.npmjs.org/formidable/-/formidable-1.0.11.tgz
       express@2.5.11 node_modules/express
       ├── qs@0.4.2
       ├── mime@1.2.4
       ├── mkdirp@0.3.0
       └── connect@1.9.2 (formidable@1.0.11)
       npm WARN package.json hellonode@0.0.1 No README.md file found!
       npm WARN package.json connect@1.9.2 No README.md file found!
       express@2.5.11 /tmp/build_h2gd35fmkgzm/node_modules/express
       connect@1.9.2 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/connect
       qs@0.4.2 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/qs
       mime@1.2.4 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/mime
       formidable@1.0.11 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/connect/node_modules/formidable
       mkdirp@0.3.0 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/mkdirp
       Dependencies installed
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size is 4.0MB
-----> Launching... done, v5
       http://fast-plateau-8288.herokuapp.com deployed to Heroku
 
To git@heroku.com:fast-plateau-8288.git
   26b4efc..6e333a2  master -> master

Now you should be able to connect to your app in the browser! An easy way to open the app in your browser is to run:

heroku open

To see your app logs (provisioning, management, scaling, and system out messages) run:

heroku logs

To see your app processes run:

heroku ps

And best of all, if you want to add more Dynos* just run:

heroku scale web=2

* Dynos are the isolated containers that run your web and other processes. They are managed by the Heroku Dyno Manifold. Learn more about Dynos.

That increases the number of Dynos running the app from one to two. Automatically Heroku will distribute the load across those two Dynos, detect dead Dynos, restart them, etc! That is seriously easy app scalability!

There is much more to Heroku and I’ll be continuing to write about it here. But in the meantime, check out all of the great docs in the Heroku Dev Center. And please let me know if you have any questions or problems. Thanks!

amf.js – A Pure JavaScript AMF Implementation

I just finished the first version of a new pure JavaScript library for AMF. I’ve wanted to do this for a while but didn’t think it was possible since JavaScript doesn’t have a ByteArray. But then I came across this: “High Performance JavaScript Port of ActionScript’s ByteArray“. That became the basis for amf.js. Before I get into the gory details of how this works, check out some developer eye candy:
http://www.jamesward.com/demos/JSAMF/censusTest.html

Ok, hopefully that worked for you. I’ve tested this in the latest Chrome, Firefox, Safari, and IE and they all seem to work. It should also work on your iPad, iPhone, or Android device.

Now for those gory details… AMF is a protocol initially created in Flash Player as a way to serialize data for storage on disk or transfer over a network. Typically in web apps we use text-based serialization protocols (like JSON or RESTful XML) for data transfer. But there are some advantages to using binary protocols – primarily much better performance. There are two versions of the AMF protocol, AMF0 and AMF3. Both are publicly documented by Adobe and numerous server-side implementations of AMF exist. AMF is just a serialization technology, not a transport. So you can put AMF encoded data into any transport (like HTTP / HTTPS). Typically Flash Player is the client that reads / writes AMF data.

I recently had a conversation with Stephan Janssen who runs Parleys.com (an amazing Flex app), which started me on this fun project. The Parleys.com PC-profile web client and the Adobe AIR desktop client both use BlazeDS and AMF as the primary serialization protocol for moving data between client and server. This is a great choice for those clients because it makes the apps snappy. But for the HTML5 client Stephan wants to reuse his AMF endpoints. This is where amf.js comes in.

Flash Player has a ByteArray API that can be used for a lot of amazing things. One of those things is to read and write AMF. If you have an object in Flash Player and you create a new ByteArray and then call “byteArray.writeObject(myObject)” you will get a ByteArray with the AMF representation of that object. Likewise if you get some AMF and you call “byteArray.readObject()” you get the object(s) from the AMF. In Flex there are high level APIs (like RemoteObject, Consumer, etc.) that use this native AMF support in Flash Player.

To create a pure JavaScript AMF library the first thing that is needed is a pure JavaScript ByteArray library since JavaScript doesn’t natively have one. I used one from adamia.com since it was similar to the ByteArray in Flash Player, seemed fast, and seemed to parse floats correctly. This ByteArray has some of the basic functions like readByte, readFloat, etc. But what about that cool readObject function? Well, that has to be built from scratch. And it should support both AMF0 and AMF3.

Using the AMF specs and code from BlazeDS & pyamf as a reference I was able to add the other functions to the ByteArray. But there was a problem. Using XMLHttpRequest as the method of getting the AMF was not working right. Some bytes were incorrect. It turns out XMLHttpRequest uses UTF-8 and that screws up some of the bytes above 128. I tried other charsets and each one would change some range of bytes. That is not good because I need the bytes to be exactly what the server sent. Then I came across this gem:

//XHR binary charset opt by Marcus Granado 2006 [http://mgran.blogspot.com]
req.overrideMimeType('text/plain; charset=x-user-defined');

Using the “x-user-defined” charset left the bytes alone. Perfect! Except that IE doesn’t support the req.overrideMimeType function. But IE does actually have a real ByteArray available in req.responseBody via VBScript. For now in IE I just change the ByteArray into a string (like req.responseText in the other browsers) although a lot of optimization could be done to just use the VBScript ByteArray directly.

Right now amf.js is just a basic JavaScript library for reading AMF data. It doesn’t support using a BlazeDS MessageBrokerServlet yet because I need to be able to assemble a AMF object in JavaScript and send that in the HTTP request to the servlet. But it works fine with a custom servlet that uses BlazeDS’s AMF library to just write AMF into the HTTP response. It should also work with pyamf, AMFPHP, and other AMF server libraries.

To use amf.js start by dumping some AMF into an HTTP response. In Java with BlazeDS I did this:

public void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException
{
    response.setHeader("Content-Type", "application/x-amf;charset=x-user-defined");
    ServletOutputStream out = response.getOutputStream();
    ActionMessage requestMessage = new ActionMessage(MessageIOConstants.AMF3);
    MessageBody amfMessage = new MessageBody();
    amfMessage.setData(someSerializableObject);
    requestMessage.addBody(amfMessage);
    AmfMessageSerializer amfMessageSerializer = new AmfMessageSerializer();
    amfMessageSerializer.initialize(SerializationContext.getSerializationContext(), out, new AmfTrace());
    amfMessageSerializer.writeMessage(requestMessage);
    out.close();
}

In a HTML web page add the amf.js script:

<script type="text/javascript" src="amf.js"></script>

In JavaScript make a XHR request for some data:

var url = "TestServlet";
var req;
 
function getAMF()
{
    if (window.ActiveXObject)
    {
        req = new ActiveXObject("Microsoft.XMLHTTP");
    }
    else if (window.XMLHttpRequest)
    {
        req = new XMLHttpRequest();
        //XHR binary charset opt by Marcus Granado 2006 [http://mgran.blogspot.com]
        req.overrideMimeType('text/plain; charset=x-user-defined');
    }
    req.onreadystatechange = processReqChange;
    req.open("GET", url, true);
    req.send(null);
}

And when the response comes back decode the AMF:

function processReqChange()
{
    if (req.readyState == 4)
    {
        if (req.status == 200)
        {
            var o = decodeAMF(req.responseText).messages[0].body;
        }
        else
        {
            alert("There was a problem retrieving the data:\n" + req.statusText);
        }
    }
}

For details on how to support IE, check out the source code for censusTest.html.

While amf.js works in my tests there is more work to be done. I need to add the write functions so that AMF can be sent to the server. Then supporting BlazeDS’s MessageBrokerServlet should be pretty straightforward. I’d also like to create pure JavaScript implementations of Flex’s RemoteObject, Consumer, and Producer APIs. Also, I need people to test amf.js with their AMF to make sure that I’ve implemented things correctly. All of the code is on github.com so go ahead and fork it! Let me know what you think.