DeveloperWeek 2013: Modern Web Apps With Scala and Play

Tomorrow (February 5, 2013) I will be presenting about Modern Web Apps With Scala and Play at DeveloperWeek in San Francisco. Here is the session description:

The web application architecture is rapidly evolving to accommodate mobile, more interactive experiences, integrated real-time, and service composition. This session will teach you how to build modern web applications using Play Framework and Scala. You will learn the end-to-end architecture including: non-blocking service composition, RESTful JSON endpoints, single page UIs, and real-time push.

Hope to see you there!

Getting Started with Play 2, Scala, and Squeryl

My friend Ryan Knight and I co-authored an article which has just been published on Artima: Getting Started with Play 2, Scala, and Squeryl. This article will help you get started building a Play 2 application from scratch that uses Scala and Squeryl for ORM. The article also covers how use ScalaTest, JSON, jQuery, CoffeeScript, and deployment on Heroku. Give it a try and let me know how it goes!

Tutorial: Play Framework 2 with Scala, Anorm, JSON, CoffeeScript, jQuery & Heroku

Play Framework 2 RC2 has been released and it is quickly becoming a mature and productive way to build modern web apps. Lets walk through building a quick app with Play 2, Scala, Anorm, JSON, CoffeeScript, and jQuery. Once the app works locally we will deploy it on the cloud with Heroku. (Note: This is the Play 2 + Scala version of my Play 1 + Java tutorial.) You can grab the completed source from GitHub.

Step 1) Download and install Play 2 RC2

Step 2) Create a new application:

play new foobar

When prompted select to use Scala as the language.

Step 3) In the newly created “foobar” directory generate the IDE config files if you’d like to use one. For IntelliJ, run:

play idea

Note: This generates an iml file which is not directly importable as a project. Instead you need to create a new project without a module and then import the module from the generated iml file. If you need help with this, follow instructions 8 – 10 in my Play 1 + Scala IntelliJ article.

For Eclipse, run:

play eclipsify

Step 4) Start the Play server:

play run

Test that it works by visiting: http://localhost:9000

Step 5) Play 2 with Scala doesn’t provide an ORM by default. Instead the default RDBMS persistence provider is Anorm (Anorm is Not an Object Relational Mapper). This simple application will just have one persistence object: a Bar with a primary key and a name. Anorm requires a SQL schema creation / destruction script since it doesn’t do auto schema creation. Create a new file named “conf/evolutions/default/1.sql” containing:

# --- First database schema
 
# --- !Ups
 
CREATE TABLE bar (
  id                        SERIAL PRIMARY KEY,
  name                      VARCHAR(255) NOT NULL
);
 
# --- !Downs
 
DROP TABLE IF EXISTS bar;

Anorm can use a Scala “case class” as a Value Object and a singleton object as the persistence / CRUD interface. Create the Bar case class and object in a file named “app/models/Bar.scala” with the following contents:

package models
 
import play.api.db._
import play.api.Play.current
 
import anorm._
import anorm.SqlParser._
 
case class Bar(id: Pk[Long], name: String)
 
object Bar {
 
  val simple = {
    get[Pk[Long]]("id") ~
    get[String]("name") map {
      case id~name => Bar(id, name)
    }
  }
 
  def findAll(): Seq[Bar] = {
    DB.withConnection { implicit connection =>
      SQL("select * from bar").as(Bar.simple *)
    }
  }
 
  def create(bar: Bar): Unit = {
    DB.withConnection { implicit connection =>
      SQL("insert into bar(name) values ({name})").on(
        'name -> bar.name
      ).executeUpdate()
    }
  }
 
}

The “simple” var provides a basic row parser that maps a database row values to the Bar case class. The “findAll” and “create” static functions just do the regular data access stuff. Notice how the “findAll” function uses the “simple” row parser to turn each row into a Bar.

Step 6) Configure the default data source to use an in-memory h2 database by adding the following values to the “conf/application.conf” file:

db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:mem:play"

Step 7) Create an Application controller function that will translate a HTTP POST into a Bar and then save it to the database by updating the “app/controllers/Application.scala” file to be:

package controllers
 
import play.api.data.Form
import play.api.data.Forms.{single, nonEmptyText}
import play.api.mvc.{Action, Controller}
import anorm.NotAssigned
 
import models.Bar
 
 
object Application extends Controller {
 
  val barForm = Form(
    single("name" -> nonEmptyText)
  )
 
  def index = Action {
    Ok(views.html.index(barForm))
  }
 
  def addBar() = Action { implicit request =>
    barForm.bindFromRequest.fold(
      errors => BadRequest,
      {
        case (name) =>
          Bar.create(Bar(NotAssigned, name))
          Redirect(routes.Application.index())
      }
    )
  }
 
}

The “barForm” maps request parameters to a Form object and can apply validations to the incoming data. The “addBar” static function handles a request and attempts to map the request parameters to the “barForm”. If it fails then the controller returns a BadRequest. If it succeeds then the Bar’s name is used to construct a new “Bar” which is saved into the database and a redirect to the index page is sent. The index function has been changed to pass the “barForm” to the template which will be updated in Step 9.

Step 8) Create a new route to map POST requests to the “/addBar” URL to the “Application.addBar” function by adding the following line to the “conf/routes” file:

POST    /addBar                     controllers.Application.addBar

Step 9) Update the “app/views/index.scala.html” template to take the Form parameter, extend the “main” template, and then use the Play 2 form helper to render the form on the web page:

@(form: play.api.data.Form[String])
 
@main("Welcome to Play 2.0") {
 
    @helper.form(action = routes.Application.addBar) {
        @helper.inputText(form("name"))
        <input type="submit"/>
    }
 
}

In your browser load http://localhost:9000, apply the database evolution, and test the form. If everything works when you submit the form you should just be redirected back to the index page.

Step 10) Create a JSON service to get all of the bars by adding a new function to the “app/controllers/Application.scala” file:

  import com.codahale.jerkson.Json
 
  def listBars() = Action {
    val bars = Bar.findAll()
 
    val json = Json.generate(bars)
 
    Ok(json).as("application/json")
  }

This new function returns the list of “Bar” objects from “Bar.findAll()” as serialized JSON.

Add a new GET request handler for requests to “/listBars” by adding the following to the “conf/routes” file:

GET     /listBars                   controllers.Application.listBars

Try it out by opening http://localhost:9000/listBars in your browser. You should see the Bar you created in Step 9.

Step 11) Create a new CoffeeScript file that will use jQuery to fetch the JSON serialized Bars and add each one to the page by creating a new “app/assets/javascripts/index.coffee” file containing:

$ ->
  $.get "/listBars", (data) ->
    $.each data, (index, item) ->
      $("#bars").append "<li>Bar " + item.name + "</li>"

This CoffeeScript uses jQuery to make a GET request to “/listBars”, iterate through the returned data, and add each item to the “bars” element on the page (which will be added in Step 12).

Step 12) Update the “app/views/index.scala.html” template to use the index.js script that is automatically compiled from the CoffeeScript source by adding the following to the “main” block:

    <script src="@routes.Assets.at("javascripts/index.js")" type="text/javascript"></script>
 
    <ul id="bars"></ul>

In your browser load http://localhost:9000 and verify that the bars are being displayed on the page and that adding new bars works as expected.

Great! You just built a Play 2 app with Scala, Anorm, JSON, CoffeeScript, and jQuery! All of the source code for this example is on GitHub. Now lets deploy it on the cloud with Heroku.

Step 1) Create an account on Heroku.com, install the Heroku Toolbelt, install git, and then login to Heroku from the command line:

heroku login

If this is the first time you’ve done this then new SSH keys for git will be created and associated with your Heroku account.

Step 2) Each application on Heroku has a Postgres database for testing. To use that database when running on Heroku we need to configure it. There are a few ways to do that with Play 2 but the easiest will be to override the database settings with the startup command line – which we will setup in Step 3. Before we do that we need to specify the Postgres JDBC driver as a project dependency. In the “project/Build.scala” file set the “appDependencies” like so:

    val appDependencies = Seq(
      "postgresql" % "postgresql" % "9.1-901-1.jdbc4"
    )

Step 3) To tell Heroku what process to run we need to create a file named “Procfile” (case sensitive) in the project’s root directory containing:

web: target/start -Dhttp.port=$PORT -DapplyEvolutions.default=true -Ddb.default.driver=org.postgresql.Driver -Ddb.default.url=$DATABASE_URL

Play 2 uses the Scala Build Tool (SBT) to build the project. When the project is deployed to Heroku the “sbt stage” command is run to compile the project. That process generates a “target/start” script that sets the Java classpath and starts the Play server. Heroku tells our application what HTTP port to listen on using the “PORT” environment variable so the “http.port” Java property is set accordingly. Also the default way to provide database (and other resource) connection strings to an application on Heroku is through environment variables. The DATABASE_URL environment variable will contain the database host, name, username, and password. So the “db.default.url” property is set with that value. Also the driver is set to the Postgres JDBC driver class.

Step 4) Heroku uses git as a means to uploading applications. Whether or not you use git for your SCM tool you can use git as the tool to upload an app to Heroku. In the root directory of your project create git repo, add the files to it, and then commit the files:

git init
git add Procfile app conf project public
git commit -m init

Note: Instead of doing a selective “git add” you can update the “.gitignore” file.

Step 5) Now we will provision a new application on Heroku using the Heroku CLI. Each application you create gets 750 free “dyno” hours per month. So as a developer you can use Heroku for free and only pay when you need to scale beyond one dyno. On the command line create a new application using the “cedar” stack:

heroku create -s cedar

This creates an HTTP endpoint and a git endpoint for your application. You can also use a custom name and point your own domain names at the application.

Step 6) The application is ready to be deployed to the cloud. From a command line do a “git push” to the master branch on Heroku:

git push heroku master

Once the files have been received by Heroku, Play Framework’s precompiler will be run, Heroku will assemble a “slug file”, and then the “slug” will be deployed onto a dyno.

Step 7) You can now open the application in your browser by navigating to the domain outputted following the “heroku create” or by simply running:

heroku open

You’ve deployed a Play 2 app with Scala, Anorm, JSON, CoffeeScript, and jQuery on the cloud with Heroku! For more details on how to use Heroku, check out the Heroku Dev Center. Let me know if you have any question.

Next Gen Web Apps with Scala, BlueEyes, and MongoDB

Web application architecture is in the midst of a big paradigm shift. Since the inception of the web we’ve been treating the browser like a thin client. Apps just dump markup to the browser which is then rendered. Every interaction requires a request back to the server which then returns more logic-less markup to the browser. In this model our web applications are server applications. There are certainly advantages to this model – especially when the markup consumers don’t have the capabilities to do anything more (or have inconsistent capabilities).

Web browser modernization has paved the way to move from the old “browser as a thin client” model to more of a Client/Server model where the client-side of the application actually runs in the browser. There are a lot of names for this new model, Ajax being a step along the way, and perhaps HTML5 being the full realization of Client/Server web apps.

There are hundreds of new libraries, tools, programming languages, and services that support the creation of these next generation web (and mobile) apps. One thing that is certainly lacking is one cohesive, end-to-end solution. I have no doubt that we will be seeing those crop up soon, but in the mean time lets pick a few libraries and build a couple simple apps to get a better understanding of what this new model looks like.

I’ve decided to try this with a stack consisting of Scala for my back-end code, BlueEyes for handling data access and serialization with transport over HTTP, MongoDB for data storage, and JavaScript with jQuery for the client-side. I’m going to keep this very simple and not pull in anything else. In future articles I’ll expand on this foundation.

So lets dive into some code. All of the code is on GitHub. If you want to get a local copy just do:

git clone git://github.com/jamesward/helloblueeyes.git

This project uses the Scala Build Tool (SBT) for downloading dependencies, compiling, and packaging. Follow the SBT setup instructions if you want to be able to compile the code on your machine. If you want to use IntelliJ or Eclipse then run one of the following commands:

sbt gen-idea
sbt eclipse

Lets start with some BlueEyes basics. BlueEyes is a lightweight web framework built on Netty (a really high performance, NIO-based HTTP library). BlueEyes makes it easy to define a tree of HTTP request handling patterns. I chose BlueEyes because it’s really lightweight and utilizes some nice features of the Scala language to make the code very concise. It’s kinda like a DSL for HTTP request handling.

To get started with BlueEyes we first need to create an AppServer that provides a way to start the server and a place to plug services into. Here is the code for the AppServer:

package net.interdoodle.example
 
object AppServer extends EnvBlueEyesServer with HelloHtmlServices with HelloJsonServices with HelloStartupShutdownServices with HelloMongoServices

My AppServer is an object which means it’s a singleton (that makes sense because I can’t have more than one of these in a single instance). The AppServer extends EnvBlueEyesServer which is a wrapper around the base BlueEyesServer that provides a way to get the HTTP port via an environment variable (this is necessary for running on Heroku – which we will get to later). The AppServer composes in a bunch of Services that actually setup HTTP request pattern trees.

Starting with the simplest one, HelloHtmlServices we have:

trait HelloHtmlServices extends BlueEyesServiceBuilder with BijectionsChunkString {
  val helloHtml = service("helloHtml", "0.1") { context =>
      request {
        path("/") {
          produce(text/html) {
            get { request =>
              val content = <html>
                <body>Hello, world!</body>
              </html>
              val response = HttpResponse(content = Some(content.buildString(true)))
              Future.sync(response)
          }
        }
      }
    }
  }
}

Using the BlueEyesServiceBuilder the HelloHtmlServices trait creates a service that consists of a big pattern match that goes something like this: For a request to path “/”, produce an HTML response when there is a HTTP GET request. Notice that the response is actually wrapped in a Future. This is an important part of the next generation web app architecture. Because the underlying HTTP server is NIO-based (non-blocking), the actual HTTP handling threads can be very efficiently managed so that a thread is only in use when actual IO is happening. BlueEyes supports this by using Future wrapped responses, allowing the HTTP thread to be unblocked by back-end processing.

If you want to try this out locally run the following to compile the app and create a Unix script that makes it easy to start the AppServer:

sbt stage

Now start the AppServer process:

target/start net.interdoodle.example.AppServer

Verify that HelloHtmlServices is working by opening the following URL in your browser:
http://localhost:8080/

Now lets walk through a few steps to deploy this application on the cloud with Heroku.
Step 1) Sign up for a Heroku account
Step 2) Install the Heroku Toolbelt
Step 3) Login to Heroku from the command line:

heroku login

Step 4) Create a new application on Heroku using the Cedar stack:

heroku create -s cedar

Step 5) Deploy the app on Heroku:

git push heroku master

Step 6) Verify that the app works in your browser:

heroku open

Great! We have an app built with Scala and BlueEyes running on the cloud. Now for the next piece of the architecture. The UI of the application will be written in JavaScript and will run on the client-side. But we need to have a way to transfer data between the client and the server. JSON has become the regular way to do this because it is parsed easily and quickly on the client. So a BlueEyes service will generate some JSON data that can then be consumed by a JavaScript application running on the client. But how do we get that JavaScript application running on the client? This is the next piece of the architecture.

Ideally the JSON services are served separately from the client-side of the application (where the client-side includes all HTML, JavaScript, CSS, images, and other static assets). It is also ideal to serve the client-side from a Content Delivery Network (CDN) so that it’s edge cached and loads super fast. Dynamic assets (the JSON services) can’t be edge cached so there is a logical (and functional) separation between the client-side and the server-side. But there is a problem with this approach.

The CDN and the JSON services hosts each have different DNS names and browsers don’t allow cross-origin requests (actually they don’t allow access to the responses). There are a few different ways to deal with this (JSONP, iframe hackery, etc) but the approach I’m taking is simple. The JSON services server will serve an entry point / thin shim web page. That shim then loads the rest of the client-side from the CDN and since the origin of the web page is the same as the JSON services, there is no need to make a cross-origin request. (Side note: There is a new way to do cross-origin requests cleanly and natively in the browser, but it’s not ubiquitously supported yet.)

Here is HelloJsonServices that illustrates these patterns:

trait HelloJsonServices extends BlueEyesServiceBuilder with BijectionsChunkJson with BijectionsChunkString {
  val helloJson:HttpService[ByteChunk] = service("helloJson", "0.1") { context:HttpServiceContext =>
    request {
      path("/json") {
        contentType(application/json) {
          get { request: HttpRequest[JValue] =>
            val jstring = JString("Hello World!")
            val jfield = JField("result", jstring)
            val jobject = JObject(jfield :: Nil)
            val response = HttpResponse[JValue](content = Some(jobject))
            Future.sync(response)
          }
        } ~
        produce(text/html) {
          get { request: HttpRequest[ByteChunk] =>
            val contentUrl = System.getenv("CONTENT_URL")
 
            val content = <html xmlns="http://www.w3.org/1999/xhtml">
              <head>
                <script type="text/javascript" src={contentUrl + "jquery-1.7.min.js"}></script>
                <script type="text/javascript" src={contentUrl + "hello_json.js"}></script>
              </head>
              <body>
              </body>
            </html>
            Future.sync(HttpResponse[String](content = Some(content.buildString(true))))
          }
        }
      }
    }
  }
}

This service handles requests to “/json” and if the requested content type is JSON then a simple JSON object is returned. If the requested content is HTML then the shim web page is returned. Inside the shim web page an environment variable, CONTENT_URL, is used to specify the URLs to load the JavaScript for the application.

The first JavaScript is the minified jQuery library. This abstracts some browser differences, makes doing the JSON request and modifying the webpage very simple. Here is the main client application hello_json.js:

$(function() {
    $.ajax("/json", {
        contentType: "application/json",
        success: function(data) {
            $("body").append(data.result);
        }
    });
});

This application has a function handler for when the page has loaded. That function makes an ajax request to “/json” with the “application/json” content type. Then on a successful response it just appends the result into the web page.

To run this locally first set the CONTENT_URL environment variable to the CDN where I’ve put the JavaScript files:

export CONTENT_URL=http://cdn-helloblueeyes.interdoodle.net/

Now start (or restart) the server and open:
http://localhost:8080/json

To test the JSON service locally run:

curl --header "Content-Type:application/json" http://localhost:8080/json

You can also use a local static file server for testing, just run net.interdoodle.example.httpstaticfileserver.HttpStaticFileServer and set CONTENT_URL accordingly.

If you want to try this on Heroku then set CONTENT_URL for your application:

heroku config:add CONTENT_URL=http://cdn-helloblueeyes.interdoodle.net/

This will set the environment variable and restart the application. Then navigate to the your Heroku application’s “/json” URL in your browser. Now both the server-side and client-side are cloudified!

Side Note: I’m using Amazon CloudFront as the CDN for this example. But any CDN would work fine. There are various ways to upload files to Amazon CloudFront (actually they get uploaded to Amazon S3) but one thing I’m looking into is a SBT plugin that will do this.

Now lets add some simple data access into the mix. I’ve chosen MongoDB because it has native support for JSON and we can take advantage of Hammersmith – a non-blocking MongoDB driver. In this example I’m not using Hammersmith since support for it hasn’t been officially added to BlueEyes (but at least there is the opportunity to easily switch to Hammersmith in the future).

Here is the code for HelloMongoServices:

trait HelloMongoServices extends BlueEyesServiceBuilder with MongoQueryBuilder with BijectionsChunkJson with BijectionsChunkString {
 
  val helloMongo = service("helloMongo", "0.1") {
    logging { log => context =>
      startup {
        // use MONGOLAB_URI in form: mongodb://username:password@host:port/database
        val mongolabUri = Properties.envOrElse("MONGOLAB_URI", "mongodb://127.0.0.1:27017/hello")
        val mongoURI = new MongoURI(mongolabUri)
 
        HelloConfig(new EnvMongo(mongoURI, context.config.configMap("mongo"))).future
      } ->
      request { helloConfig: HelloConfig =>
        path("/mongo") {
          contentType(application/json) {
            get { request: HttpRequest[JValue] =>
              helloConfig.database(selectAll.from("bars")) map { records =>
                HttpResponse[JValue](content = Some(JArray(records.toList)))
              }
            }
          } ~
          contentType(application/json) {
            post { request: HttpRequest[JValue] =>
              request.content map { jv: JValue =>
                helloConfig.database(insert(jv --> classOf[JObject]).into("bars"))
                Future.sync(HttpResponse[JValue](content = request.content))
              } getOrElse {
                Future.sync(HttpResponse[JValue](status = HttpStatus(BadRequest)))
              }
            }
          } ~
          produce(text/html) {
            get { request: HttpRequest[ByteChunk] =>
              val contentUrl = System.getenv("CONTENT_URL")
 
              val content = <html xmlns="http://www.w3.org/1999/xhtml">
                              <head>
                                <script type="text/javascript" src={contentUrl + "jquery-1.7.min.js"}></script>
                                <script type="text/javascript" src={contentUrl + "hello_mongo.js"}></script>
                              </head>
                              <body>
                              </body>
                            </html>
              Future.sync(HttpResponse[String](content = Some(content.buildString(true))))
            }
          }
        }
      } ->
      shutdown { helloConfig: HelloConfig =>
        Future.sync(())
      }
    }
  }
}

This service has a startup hook that creates a MongoDB connection based on the MONGOLAB_URI environment variable or a default value. Then there are three request handlers for the “/mongo” path. The first handles JSON HTTP GET requests by fetching some data from MongoDB and returning it. A JSON HTTP POST handler inserts the JSON that was sent into MongoDB. Then there is a HTML HTTP GET request handler that returns the web page shim that loads the JavaScript from the CDN.

Here is the client-side of the application hello_mongo.js:

$(function() {
 
    $("body").append('<h4>Bars:</h4>');
    $("body").append('<ul id="bars"></ul>');
    $("body").append('<input id="bar"/>');
    $("body").append('<button id="submit">GO!</button>');
 
 
    $("#submit").click(addbar);
    $("#bar").keyup(function(key) {
        if (key.which == 13) {
            addbar();
        }
    });
 
    loadbars();
 
});
 
function loadbars() {
    $.ajax("/mongo", {
        contentType: "application/json",
        success: function(data) {
            $("#bars").children().remove();
            $.each(data, function(index, item) {
                $("#bars").append("<li>" + item.name + "</li>");
            });
        }
    });
}
 
function addbar() {
    $.ajax({
        url: "/mongo",
        type: 'post',
        dataType: 'json',
        success: loadbars,
        data: '{"name": "' + $("#bar").val() + '"}',
        contentType: 'application/json'
    });
}

This JavaScript application starts by adding some elements to the page for displaying and entering “bars”. The loadbars() function makes a ajax GET request to the JSON service and then appends the result into the element on the page with the id of bars. The addbar() method makes a POST request to the JSON service, passing serialized JSON containing the name of the “bar” to the JSON service, then on success it calls loadbars().

If you want to run this locally then you will need a MongoDB server setup and running locally. Make sure CONTENT_URL is set correctly and if your MongoDB settings differ from the default, then set the MONGOLAB_URI environment variable. Start the AppServer process and then open:
http://localhost:8080/mongo

You should be able to add new “bars” and see them listed.

To run on Heroku you will need to add the MongoLab add-on. When the MongoLab add-on is added the MONGOLAB_URI environment variable on your Heroku application will be automatically set with the connection information for the newly provisioned MongoDB system. To add the MongoLab add-on just run:

heroku addons:add mongolab

Now open your Heroku app’s “/mongo” URL in your browser. Wahoo! You have a client/server browser app running on the cloud! Hopefully that has helped you to get a glimpse of where web application architecture is going. This is just part of the picture and I have a few other articles planned that will cover some of the other pieces. So keep watching here and let me know what you think.

Acknowledgments: Mike Slinn, a friend of mine that does Scala and Akka consulting, helped me understand BlueEyes and helped write some of the code for this project. Mike is also the author of the recently released book Composable Futures with Akka 2.0 with Java and Scala Code Examples.