Refactoring to Microservices

Right now there is a ton of hype and pushback around Microservices. Most of the current debate revolves around when Microservices make sense with smart people arguing all across the spectrum. As with all architectural topics the right answer is “it depends” so you should never blindly chose Microservices without understanding your goals and how they align with Microservices.

Using the open source WebJars project as an example I’d like to walk through a process of deciding where to use Microservices and then refactor part of the webjars.org app to a Microservice. First a little background on WebJars… WebJars are JavaScript & CSS libraries packaged into Jar files and published on Maven Central for easy consumption by JVM build tools. The webjars.org site is a Play Framework + Scala app that provides search, publishing, and file service for the jsDelivr CDN.

Here is my checklist for determining whether a piece of functionality should be broken out into a separate Microservice:

  1. The piece of functionality does NOT have shared mutable state.
    When using a Microservice a copy of the data will be shared. Mutating that copy will likely not propagate those changes back to the original source and all of the other possible copies of the data. While shared mutable state is common in many OO apps, this makes it very hard to switch to Microservices. Functional Programming on the other hand encourages immutable data which makes it much easier to switch to Microservices where copies can be mutated but it is clear that those mutations do not act on the original or other copies.
  2. The piece of functionality has independent operational or computational needs.
    If SLAs, scaling, or deployment needs vary between different pieces of functionality then Microservices might make sense. For example, if one piece of a system requires five nines but rarely changes while another piece does not have an SLA requirement and changes multiple times a day, Microservices make sense. Likewise you shouldn’t need to scale up every part of a system just because one piece of functionality has significant computation needs.
  3. The piece of functionality has cross-platform clients.
    While sharing code across platforms (e.g. JVM, Ruby, Node.js, etc) is sometimes possible, it is often easier and more maintainable to just expose the needed piece of functionality as a Microservice so that any platform can use it. For example, webjars.org uses a bower-as-a-service Microservice that runs in Node.js because it uses the Bower NPM package. The webjars.org app is a cross-platform (JVM) client to the Node.js Microservice.

The whole webjars.org app is functional and uses immutable data so there isn’t any shared mutable state that would make it hard to break pieces of functionality out into Microservices. In Play Framework a controller is really just a stateless function that takes a request and returns a response. This means that any of the web endpoints can be easily moved without impacting the system.

One possible candidate for a Microservice in webjars.org is a utility that converts SemVer-style version ranges to Maven-style version ranges. The SemVer.convertSemVerToMaven() function is not side-effecting so it could easily become a Microservice. But at this time the utility does not have independent operational or computation needs and it also does not have any other clients than the webjars.org app. If the functionality was needed outside of webjars.org then it could easily be turned into a library but a Microservice would definitely be overkill.

Another candidate for a Microservice in webjars.org is a web endpoint that serves a file from a WebJar. The Application.file controller function is stateless and does not use shared mutable state so it could easily become a Microservice. This function is what provides the content for WebJars on the jsDelivr CDN. When a request for a WebJar file on jsDelivr is received, if the CDN does not have the asset it gets it from webjars.org. For example:
https://cdn.jsdelivr.net/webjars/org.webjars/jquery/2.1.0/jquery.js
Is backed by:
https://webjars.herokuapp.com/files/org.webjars/jquery/2.1.0/jquery.js

The operational and computational needs of this piece of functionality are pretty different from the rest of the webjars.org app. Let’s compare the needs:

The webjars.org File Service Rest of webjars.org
SLA If it goes down then many production sites break No production uptime requirements
Scaling Most load is handled by the CDN but sometimes load spikes when caches are stale or invalidated Very light load
Deployment Rarely changes Changes a few times a week

So this seems like a great candidate for a Microservice! Here are the steps I used to break out this functionality into a Microservice.

Step 1) Create a new Play + Scala app

I used Typesafe Activator to create a new Play Framework + Scala app:

activator new webjars-file-service play-scala

Here is the commit from that starting place:
https://github.com/webjars/webjars-file-service/commit/4ef3567a8ea901c31cc1060af67ed80331b9b6a4

Step 2) Clean up the build and copy the code into the new project

I copy and pasted the parts of the code that I wanted to move to the Microservice into the new project. Here is the full change set:
https://github.com/webjars/webjars-file-service/commit/5cc050f0dab16e8cb14296d811a965df9616ca0b

There was very minimal refactoring between the original source and the new Microservice. Everything worked great locally so it was time to deploy the Microservice.

Step 3) Create a new Heroku app and setup GitHub auto-deployment

I created a new app on Heroku:

heroku create webjars-file-service

Instead of doing the usual git push heroku master I setup auto-deployment so that whenever I push to GitHub, Heroku deploys the changes. Check out a screencast of how to do that:
https://www.youtube.com/watch?v=QUvxrzINj5Q

Now that the webjars-file-service is deployed let’s try it out:
https://webjars-file-service.herokuapp.com/files/org.webjars/jquery/2.1.0/jquery.js

Everything is working great so lets switch webjars.org over to the new Microservice.

Step 4) Make the webjars.org app use the new Microservice

To make webjars.org use the new Microservice I removed the actual logic but I didn’t want to break any clients that were using the endpoints. To do this I added redirects for the actual file service functionality and for the file listing functionality I added a utility that wraps the new webjars-file-service Microservice. Along the way I had to do a small refactor of some Memcache-related functionality. Here is the full change set:
https://github.com/webjars/webjars/commit/5b7cd4b8fa410cd6e4ef9824a69d28790f42e9bd

After pushing the changes to GitHub, Codeship verified that the tests passed, and Heroku deployed webjars.org.

This whole process only took a few hours and so far everything has been working great! Because the file service functionality in webjars.org did not have shared mutable state it was incredibly easy to move to a Microservice which enables me to handle it’s unique operational and computational needs.

The decision to move something to a Microservice is always full of “it depends” factors. Microservices are certainly not a silver bullet especially when dealing with code bases that have shared mutable state. Like any tool, Microservice can be a powerful way to help you, or they can be the chainsaw you use to cut down the tree that falls on you. Handle with care!

NPM Packages in Maven Central with NPM WebJars

A few months ago I launched Bower WebJars which provides a way for anyone to deploy Bower packages into Maven Central through WebJars. Since then 539 packages have been deployed! Today I’ve added NPM WebJars which is built on the same foundation as Bower WebJars but for NPM packages.

Give it a try and let me know how it goes. If you are curious about the changes to make this happen, check out the pull request.

Auto-Deploy GitHub Repos to Heroku

My favorite new feature on Heroku is the GitHub Integration which enables auto-deployment of GitHub repos. Whenever a change is made on GitHub the app can be automatically redeployed on Heroku. You can even tell Heroku to wait until the CI tests pass before doing the deployment. I now use this on almost all of my Heroku apps because it allows me to move faster and do less thinking (which I’m fond of).

For apps like jamesward.com I just enable deployment straight to production. But for apps that need a less risky setup I have a full Continuous Delivery pipeline that looks like this:

  1. Push to GitHub
  2. CI Validates the build
  3. Heroku deploys changes to staging
  4. Manual testing / validation of staging
  5. Using Heroku Pipelines, promote staging to production

I’m loving the flexibility and simplicity of this new feature! Check out a quick screencast to see how to setup and use Heroku GitHub auto-deployment:

Notice that none of this required a command line! How cool is that?!?

Reactive Postgres with Play Framework & ScalikeJDBC

Lately I’ve built a few apps that have relational data. Instead of trying to shoehorn that data into a NoSQL model I decided to use the awesome Heroku Postgres service but I didn’t want to lose out on the Reactiveness that most of the NoSQL data stores support. I discovered ScalikeJDBC-Async which uses postgresql-async, a Reactive (non-blocking), JDBC-ish, Postgres driver. With those libraries I was able to keep my data relational and my app Reactive all the way down. Lets walk through how to do it in a Play Framework app. (TL;DR: Jump to the the full source.)

If you want to start from scratch, create a new Play app from the Play Scala Seed.

The minimum dependencies needed in the build.sbt file are:

libraryDependencies ++= Seq(
  "org.postgresql"       %  "postgresql"                    % "9.3-1102-jdbc41",
  "com.github.tototoshi" %% "play-flyway"                   % "1.2.0",
 
  "com.github.mauricio"  %% "postgresql-async"              % "0.2.16",
  "org.scalikejdbc"      %% "scalikejdbc-async"             % "0.5.5",
  "org.scalikejdbc"      %% "scalikejdbc-async-play-plugin" % "0.5.5"
)

The play-flyway library handles schema evolutions using Flyway. It is a great alternative to Play’s JDBC module because it just does evolutions and does one-way evolutions (i.e. no downs). But because play-flyway doesn’t use the postgresql-async driver, it needs the standard postgresql JDBC driver as well.

The scalikejdbc-async-play-plugin library manages the lifecycle of the connection pool used by scalikejdbc-async in a Play app.

To use play-flyway and scalikejdbc-async-play-plugin a conf/play.plugins file must tell Play about the plugins:

776:com.github.tototoshi.play2.flyway.Plugin
777:scalikejdbc.async.PlayPlugin

A first evolution script in conf/db/migration/default/V1__create_tables.sql will create a table named bar that will hold a list of bars for our little sample app:

DROP TABLE IF EXISTS bar;
CREATE TABLE bar (
  id SERIAL PRIMARY KEY,
  name VARCHAR NOT NULL
);

You will of course need a Postgres database to proceed. You can either install one locally or create a free one on the Heroku Postres cloud service. Then update the conf/application.conf file to point to the database:

db.default.driver="org.postgresql.Driver"
db.default.url="postgres://admin:admin@localhost:5432/test"
db.default.url=${?DATABASE_URL}

The last line above overrides the database connection url if there is a DATABASE_URL environment variable set (which is the case if your app is running on Heroku).

To run this app locally you can start the Play app by starting the Activator UI or from the command line with:

activator ~run

When you first open your app in the browser, the play-flyway plugin should detect that evolutions needs to be applied and ask you to apply them. Once applied you will be ready to create a simple database object and a few reactive request handlers.

Here is a Bar database object named app/models/Bar.scala that uses scalikejdbc-async for reactive creation and querying of Bars:

package models
 
import play.api.libs.json.Json
import scalikejdbc.WrappedResultSet
import scalikejdbc._
import scalikejdbc.async._
import scalikejdbc.async.FutureImplicits._
 
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
 
case class Bar(id: Long, name: String)
 
object Bar extends SQLSyntaxSupport[Bar] {
 
  implicit val jsonFormat = Json.format[Bar]
 
  override val columnNames = Seq("id", "name")
 
  lazy val b = Bar.syntax
 
  def db(b: SyntaxProvider[Bar])(rs: WrappedResultSet): Bar = db(b.resultName)(rs)
 
  def db(b: ResultName[Bar])(rs: WrappedResultSet): Bar = Bar(
    rs.long(b.id),
    rs.string(b.name)
  )
 
  def create(name: String)(implicit session: AsyncDBSession = AsyncDB.sharedSession): Future[Bar] = {
    val sql = withSQL(insert.into(Bar).namedValues(column.name -> name).returningId)
    sql.updateAndReturnGeneratedKey().map(id => Bar(id, name))
  }
 
  def findAll(implicit session: AsyncDBSession = AsyncDB.sharedSession): Future[List[Bar]] = {
    withSQL(select.from[Bar](Bar as b)).map(Bar.db(b))
  }
 
}

The db functions perform the mapping from SQL results to the Bar case class.

The create function takes a Bar name and returns a Future[Bar] by doing a non-blocking insert using the ScalikeJDBC Query DSL. When the insert has completed the primary key is returned and a new Bar instance is created and returned.

The findAll method uses the ScalikeJDBC Query DSL to select all of the Bars from the database, returning a Future[List[Bar]]].

Now that we have a reactive database object, lets expose these through reactive request handlers. First setup the routes in the conf/routes file:

GET        /bars                   controllers.Application.getBars
POST       /bars                   controllers.Application.createBar

Define the controller functions in the app/controllers/Application.scala file:

def getBars = Action.async {
  Bar.findAll.map { bars =>
    Ok(Json.toJson(bars))
  }
}
 
def createBar = Action.async(parse.urlFormEncoded) { request =>
  Bar.create(request.body("name").head).map { bar =>
    Redirect(routes.Application.index())
  }
}

Both functions use Action.async which holds a function that takes a request and returns a response (Result) in the future. By returning a Future[Result] Play is able to make requests to the controller function non-blocking. The getBars controller function calls the Bar.findAll and then transforms the Future[List[Bar]] into a Future[Result], the 200 response containing the JSON serialized list of bars. The createBar controller function parses the request, creates the Bar, and then transforms the Future[Bar] into a Future[Result] once the Bar has been created.

From the non-blocking perspective, here is what a request to the getBars controller function looks like:

  1. Web request made to /bars
  2. Thread allocated to web request
  3. Database request made for the SQL select
  4. Thread allocated to the database request
  5. Web request thread is deallocated (but the connection remains open)
  6. Database request thread is deallocated (but the connection remains open)
  7. Database response handler reallocates a thread
  8. SQL result is transformed to List[Bar]
  9. Database response thread is deallocated
  10. Web response handler reallocates a thread
  11. Web response is created from the list of bars
  12. Web response thread is deallocated

So everything is now reactive all the way down because there is a moment where the web request is waiting on the database to respond but no threads are allocated to the request.

Try it yourself with curl:

$ curl -X POST -d "name=foo" http://localhost:9000/bars
$ curl http://localhost:9000/bars
[{"id":1,"name":"foo"}]

Grab the the full source and let me know if you have any questions. Thanks!

Scaling the WebJars Project with On-Demand Bower Packages

WebJars has been my hobby project for almost 3 years and thanks to tons of help from the community the project now has almost 900 JavaScript libraries in Maven Central. In February 2015 there were over 500,000 downloads of WebJars! Up until now all of the WebJars have been manually created, deployed, and maintained. Today I’m happy to launch Bower WebJars, an on-demand service that allows users to deploy new WebJars from Bower packages.

If a WebJar doesn’t exist for the library you need, but it does exist in Bower, then select the Bower package & version and deploy it! Under the covers the Bower package is converted to a Maven package (including transitive dependency definitions), deployed to BinTray, and then synced to Maven Central. Check out the code.

Bower WebJars should work for most Bower packages but I’ve noticed a few packages that do not follow standard dependency versioning practices or do not have standard OSS licenses. So if you have any problems file an issue and we’ll see if there is something we need to fix on our side or if we can work with the package provider to resolve the issue.

We’ve started with packaging Bower libraries but it would be logical to also do this for NPM. We are discussing teaming up with the npmaven folks to do this. So stay tuned.

A huge thanks to Fred Simon (Founder at JFrog) and the rest of the folks behind BinTray. At the WTF Conference a few weeks ago, Fred helped me get a handle on the BinTray REST APIs which really kick-started this project. Since then the BinTray folks have been ridiculously helpful and instrumental in getting the Bower WebJar stuff working seamlessly.

I hope the new Bower WebJars are useful for you. Let me know what you think.

Salesforce Canvas Quick Start for Java Developers

Salesforce provides a variety of different ways to integrate external apps into the Salesforce UI. Canvas is an iframe-based approach for loading externally hosted UIs into pages on Salesforce. The nice thing about Canvas versus a plain iframe is that Canvas has a JavaScript bridge which enables secure communication between the external iframe and Salesforce. This communication happens in the context of the Salesforce user and doesn’t require the typical OAuth handshake. Because Canvas apps live outside of Salesforce they can be built with any language and run anywhere, including Heroku.

I’ve put together a quick start Canvas app that uses Java and Play Framework to get you going in minutes. You can either run this app on Heroku or on your local machine. The fewest steps to get everything setup is with Heroku so I’ll cover that first.

Deploying a Canvas App on Heroku

Heroku is an app delivery platform that works with a variety of back-end programming languages and frameworks. I’ve created a Canvas-ready Java app using Play Framework and set it up for instant deployment. To get started, signup for a free Heroku account (if needed), then launch your own copy of the salesforce-canvas-seed app.

Once the app has been deployed, open / view the app and follow the instructions to complete the setup process. You will create a new Connected App on Salesforce that is used to identify the external app. Once you’ve completed the instructions you will now have a fully functional Canvas app, running on Heroku, and written in Java.

Running a Canvas App Locally

Cloud deployment with Heroku is quick and easy but if you want to make changes to your app you should setup a local development environment. To do that, download the salesforce-canvas-seed Activator bundle, extract the zip, and from a command line in the project’s root directory, run (on Windows, omit the ./):

./activator -Dhttps.port=9443 ~run

This will start the app with HTTPS enabled. Connect to the app: https://localhost:9443
Your browser may give you a warning about the certificate not being valid, which you should ignore / approve the connection. Then you will see instructions for setting up a new Connected App on Salesforce which will be used for local development. After following those instructions you will have a Canvas app running locally. Now you can begin making changes to the app.

The app/assets/javascripts/index.js file contains the client-side application while the app/views/index.scala.html file contains the HTML content for the application. The app you’ve just setup should just be displaying the logged in user’s name. That is happening using some jQuery and the Canvas SDK. The HTML file contains:

<h1>Hello <span id='username'></span></h1>

The index.js file contains:

Sfdc.canvas(function() {
  // Save the token
  Sfdc.canvas.oauth.token(window.signedRequestJson.oauthToken);
  Sfdc.canvas.byId('username').innerHTML = window.signedRequestJson.context.user.fullName;
});

This JavaScript sets up the OAuth token for the Canvas SDK. Then the contents of the username tag are replaced with the current user’s name.

That should be everything you need to get started building against the Canvas SDK. For more information on that check out the Canvas Docs.

Let me know how it goes!

Introducing Force WebJars: Add JavaScript Libs to Salesforce With a Click

The typical method of adding JavaScript and CSS libraries (e.g. jQuery, Bootstrap, and AngularJS) to Salesforce environments is to locate a library’s download, download it, then upload it to Salesforce, then figure out the structure of the files so that you can use them from Visualforce. Using WebJars as a basis, I’ve created an easy way to add libraries to Salesforce, called Force WebJars.

Here is a quick demo:

Give it a try and let me know how it goes!

BTW: The source is on GitHub.

Goodbye Java Posse Roundup – Hello WTF 2015

For the past 8 (maybe 9?) years the absolute best yearly conference I’ve attended has been The Java Posse Roundup in Crested Butte, Colorado. At most conferences my favorite parts are the conversations at the bar and writing code with other attendees. The Java Posse Roundup has always been a conference just of those best parts.

Even though The Java Posse is no-more, the Java Posse Roundup will live on as the Winter Tech Forum. I’m sure that many of the same developers will come since it has been a yearly pilgrimage for many. But hopefully the new name helps bring an even more diverse audience this year.

This year I’m looking forward to doing more with IoT and Salesforce. A few years ago Matt Raible and I built a shot ski so maybe this year we can build a smart shot ski that connects to an auto-reordering system. :)

Hope to see you there!

Introducing Gulp Launcher

Many developers already have the Node.js toolchain installed on their machines but when I lead workshops there are always a few who don’t. The process of installing Node build toolchains can take quite a bit of time for new users (especially on Windows). To simplify the process of getting the gulp toolchain setup, Bruce Eckel and I created gulp launcher. With a fresh system you can run gulp with only one download and one command:

$ gulp
Downloading jq 1.3 for x86_64 MAC
Downloading Node 0.10.33 for x86_64 MAC
... npm install output ...
[09:13:25] Using gulpfile ~/projects/gulp-starter/gulpfile.js
[09:13:25] Starting 'default'...
[09:13:25] Finished 'default' after 7.28 μs

On Windows the user can just double click on gulp.exe and it will run the default task for the gulp build.

If you want to try it out either download gulp (Linux, Mac, and Cygwin) or gulp.exe (Windows) from the latest release. Or if you want a ready-to-go starter gulp project, grab the gulp-starter project.

I’ve setup automated tests for the gulp launcher on Travis CI and AppVeyor but there isn’t a lot of real-world testing yet. So if you discover any problems please file an issue. Let me know how it goes. Thanks!