The Magic Behind Heroku’s “git push” Deployment

In my spare time I help out with a little app called Greg’s Toolkit that was built before I knew about Heroku. The app runs on EC2 and deploying new versions of the app is pretty tedious. Here is the deployment instructions copied directly from the project’s wiki:

./gradlew war
scp  ./gft_server/build/libs/gft_server.war api.gregstoolkit.com:
ssh api.gregstoolkit.com
cd /opt/apache-tomcat-7.0.21
sudo -u www-data bin/shutdown.sh
cd webapps/ROOT
sudo rm -r *
sudo -u www-data jar -xvf ~/gft_server.war
sudo -u www-data sed -i 's/dev/prod/' WEB-INF/web.xml
cd ../..
sudo -u www-data bin/startup.sh

That certainly isn’t as cumbersome as some deployment methods but it is time consuming, error-prone, and causes downtime.

If you’ve used Heroku one of the most magical things you first come across is Instant Deployment. With Heroku you just upload your code to Heroku and that kicks off a process that results in the new version of the app being deployed. Heroku uses the Git tool as a way for developers to upload an app’s source to Heroku. So to deploy a new version of an app on Heroku (from the command line), you just run:

git push heroku master

It’s magic! But what actually happens on Heroku when you “push” a Git repository to Heroku? Anything you want! A “Buildpack” translates whatever you push to Heroku into something that can be run on Heroku. Heroku provides a number of out-of-the-box Buildpacks like the Maven Buildpack (for Java projects), the NPM Buildpack (for Node.js projects), and many others.

Now here is the really cool thing… The Buildpack system on Heroku is totally open and pluggable. There is even a documented API for the Buildpack system. So if you want to modify the process the takes your code from the “git push” to a runnable application, you can either: customize an existing Buildpack, create an entirely new Buildpack from scratch, or use a third party Buildpack.

Lets try a really simple example using a third party Buildpack. The Null Buildpack literally does nothing on a “git push”. Whatever you push gets deployed. Lets try it out. I’m going to use Python since it includes a simple web server in the base system and doesn’t need to be compiled. Follow along:

  1. Install Git and the Heroku Toolbelt
  2. Login to Heroku from the command line:
    heroku login
  3. In a new directory create a new file named “web.py” containing:
    import os
    from wsgiref.simple_server import make_server
     
    port = os.environ.get("PORT", "5000")
     
    def hello_world_app(environ, start_response):
        status = '200 OK'
        headers = [('Content-type', 'text/plain')]
        start_response(status, headers)
        return ["hello, world"]
     
    httpd = make_server('', int(port), hello_world_app)
    print "Serving HTTP on port " + port + "..."
     
    httpd.serve_forever()
  4. If you want to test the app locally run:
    python web.py

    Then in your browser go to http://localhost:5000 and you should see “hello, world”.

  5. To run this simple app on Heroku you need to let Heroku know what command to run on the system. To do this create a file named “Procfile” (in the same directory as “web.py”) containing:
    web: python web.py

    This tells Heroku that for the process named “web” run “python web.py” on the system. The managed systems that run apps on Heroku are called “Dynos“. You can allocate as many Dynos as needed to each process in your app.

  6. Since Heroku uses Git for file upload and download you need to create a new Git repository, add your files to it, and commit them:
    git init
    git add Procfile web.py
    git commit -m init
  7. Now provision (i.e. create) a new app on Heroku but instead of using the default set of Buildpacks, explicitly tell Heroku to use the Null Buildpack:
    heroku create --buildpack https://github.com/ryandotsmith/null-buildpack.git

    This provisions the app on Heroku and sets a new Git remote named “heroku” in the configuration for your Git repository. The “heroku” remote is just a name that points to the Git URL for your app on Heroku.

  8. Now push your app to the “heroku” Git remote repository:
    git push heroku master

Once the files are uploaded, Heroku’s Slug Compiler uses the specified Buildpack to perform the build, converting source code into something runnable. Everything that is needed to run the app (with the exception of the base system) goes into a “slug”. Then the slug is copied to a Dyno and the process named “web” is run. Here is the output from my “git push”:

Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 522 bytes, done.
Total 4 (delta 0), reused 0 (delta 0)
 
-----> Heroku receiving push
-----> Fetching custom buildpack... done
-----> Null app detected
-----> Nothing to do.
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size is 4K
-----> Launching... done, v4
       http://blazing-waterfall-8915.herokuapp.com deployed to Heroku
 
To git@heroku.com:blazing-waterfall-8915.git
 * [new branch]      master -> master

Check out the app in your browser by running:

heroku open

That’s it! Even though it feels magical, it is really a pretty simple (and totally customizable) process.

Lets recap what happens when you do a “git push heroku master”:

  1. The Git repository is uploaded to Heroku.
  2. The Slug Compiler uses either a custom specified Buildpack or a default Buildpack to transform the source that was pushed into something runnable. This process can be as simple as doing nothing or as complex is resolving dependencies, compiling source, packaging assets, running tests, etc. All files in the project directory when the Buildpack finishes are put into the slug.
  3. The slug is copied to new Dyno(s). (Usually the default is 1 Dyno for the “web” process.)
  4. The defined processes are started on the Dynos.

Pretty simple! Here is another cool feature of Heroku’s Buildpack system… If the Buildpack returns an error then the process stops. So with compiled languages you get some automatic “testing” of the code before it is deployed.

To recap: A Buildpack translates whatever you push to Heroku into something that can be run on Heroku. There are tons and tons of Buildpacks out there (mostly on GitHub). Next time you need to deploy something out of the ordinary, see if there is a Buildpack for it already. If not, then why not create your own? It’s actually really simple. My first custom Buildpack simply converts Markdown files to HTML so that I can instantly deploy Markdown on the web. It is under 40 lines of code and was pretty simple to write. If I can create a Buildpack then you probably can too! :)

Play 2 Scala Console on Heroku

I’ve been working on a Play 2 application that I’ll be using for the Grails vs Play Smackdown at ƜberConf next week. The app is running in production on Heroku but since I don’t have an admin UI yet, I needed a quick and easy way to create a new entity. I could have gone straight to the database but thought it would be better to run Play 2’s Scala Console on Heroku and then just run some arbitrary Scala code. The Scala Console in Play 2 is really just the Scala REPL in SBT but it allows you to interact with a Play application.

I’ll walk you through how to use the Play 2 Scala Console on Heroku. If you’d like to follow along, grab the play2bars app by running:

git clone https://github.com/jamesward/play2bars.git
cd play2bars
git checkout java-ebean

To run the console locally, run:

play -DapplyEvolutions.default=true console

You will see something like:

[info] Loading project definition from /home/jamesw/Desktop/play2bars/project
[info] Set current project to play2bars-java (in build file:/home/jamesw/Desktop/play2bars/)
[info] Updating {file:/home/jamesw/Desktop/play2bars/}play2bars-java...
[info] Resolving org.hibernate.javax.persistence#hibernate-jpa-2.0-api;1.0.1.Fin                                                                                [info] Done updating.                                                        
[info] Compiling 4 Scala sources and 3 Java sources to /home/jamesw/Desktop/play2bars/target/scala-2.9.1/classes...
[warn] Note: /home/jamesw/Desktop/play2bars/app/models/Bar.java uses unchecked or unsafe operations.
[warn] Note: Recompile with -Xlint:unchecked for details.
[info] Starting scala interpreter...
[info] 
Welcome to Scala version 2.9.1.final (OpenJDK 64-Bit Server VM, Java 1.7.0_03).
Type in expressions to have them evaluated.
Type :help for more information.
 
scala>

Start a Play app from the Scala console:

new play.core.StaticApplication(new java.io.File("."))

You will see something like:

[info] play - database [default] connected at jdbc:h2:mem:play
[info] play - Application started (Prod)
res0: play.core.StaticApplication = play.core.StaticApplication@2e338c56

Now you can interact with the Play application. Lets create a new “Bar” entity:

var bar = new models.Bar()
bar.name = "foo bar"
bar.save

And now you can query the “Bar” entities:

import scala.collection.JavaConversions._
models.Bar.find.all.foreach(bar => println(bar.name))

So that’s pretty cool, right?

If you want to run this example on Heroku, then install the Heroku Toolbelt and run:

heroku create -s cedar

And push the application to Heroku:

git push heroku java-ebean:master

Once the application is built and deployed on Heroku, check to make sure it works:

heroku open

To run the Scala console on Heroku, run:

heroku run bash
java -DapplyEvolutions.default=true -Ddb.default.driver=org.postgresql.Driver -Ddb.default.url=$DATABASE_URL -jar ~/.sbt_home/bin/sbt-launch-0.11.3-2.jar

SBT will resolve some dependencies and then you will be in the SBT console. From there run:

set fullClasspath in Compile += Attributed.blank(file("target/staged/*"))
console

Now you will be in the Scala REPL for your Play app on Heroku. So you can run things like:

new play.core.StaticApplication(new java.io.File("."))
var bar = new models.Bar()
bar.name = "foo bar"
bar.save

Reload your app on Heroku in the browser and you should now see the new Bar listed on the page!

Let me know if you have any questions.

Webinar: Social Enterprise Java Apps on Heroku

Tomorrow, May 30th, I will be co-hosting a webinar about Social Enterprise Java Apps on Heroku. The webinar will be at both 2:00 p.m. GMT and 10:00 a.m. PDT. Register at: http://www.developerforce.com/events/webinars/2012-05-30/registration.php?d=70130000000sW7g

Here is the description:

In this webinar you will learn how to build Social Enterprise applications using Salesforce.com, Heroku, and Java. Through live coding and demonstrations you will learn how to instantly deploy and scale Java apps on the cloud with Heroku. You will also learn how to integrate those applications with Salesforce.com and Force.com through REST.

Hope to see you there!

Graphs in the Cloud: Spring + Neo4j on Heroku

Last week I hosted a webinar about running Java apps on Heroku that use the Spring Framework and the Neo4j graph database. Here is the recording of that webinar:

In the webinar I began by deploying a copy of the Spring MVC + Hibernate template app from heroku.com/java on Heroku. Then I made a few modifications to the app to switch the persistence from Hibernate / JPA to Neo4j. You can get the full source code on GitHub.

Here is a quick recap of what I did to switch the template app to use Neo4j:

  1. Added the Neo4j Heroku Add-on:
    heroku addons:add neo4j
  2. Added the Spring Data Neo4j dependencies (optionally you can remove the unused JPA dependencies) to the “pom.xml” Maven build descriptor:
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-data-neo4j-rest</artifactId>
        <version>2.0.1.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>org.hibernate</groupId>
        <artifactId>hibernate-validator</artifactId>
        <version>4.2.0.Final</version>
    </dependency>
  3. Modified the “src/main/java/com/example/service/PersonService.java” interface to use the Neo4j GraphRepository:
    package com.example.service;
     
    import com.example.model.Person;
    import org.springframework.data.neo4j.repository.GraphRepository;
     
    public interface PersonService extends GraphRepository<Person> {
     
     
    }
  4. Removed the unneeded “src/main/java/com/example/service/PersonServiceImpl.java” DAO.
  5. Modified the “src/main/java/com/example/model/Person.java” POJO to be a @NodeEntity (instead of JPA Entity) and switched the “id” primary key property to be a Long annotated as a @GraphId:
    package com.example.model;
     
    import org.springframework.data.neo4j.annotation.GraphId;
    import org.springframework.data.neo4j.annotation.NodeEntity;
     
    @NodeEntity
    public class Person {
     
        @GraphId
        private Long id;
     
        // the rest is omitted
  6. Modified the “src/main/java/com/example/controller/PersonController.java” Spring MVC controller to use the new “PersonService”, take a Long parameter in the “deletePerson” method, and make the “deletePerson” and “addPerson” methods transactional:
    package com.example.controller;
     
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.stereotype.Controller;
    import org.springframework.transaction.annotation.Transactional;
    import org.springframework.web.bind.annotation.ModelAttribute;
    import org.springframework.web.bind.annotation.PathVariable;
    import org.springframework.web.bind.annotation.RequestMapping;
    import org.springframework.web.bind.annotation.RequestMethod;
     
    import com.example.model.Person;
    import com.example.service.PersonService;
     
    import java.util.Map;
     
    @Controller
    public class PersonController {
     
        @Autowired
        private PersonService personService;
     
        @RequestMapping("/")
        public String listPeople(Map<String, Object> map) {
            map.put("person", new Person());
            map.put("peopleList", personService.findAll().iterator());
            return "people";
        }
     
        @RequestMapping(value = "/add", method = RequestMethod.POST)
        @Transactional
        public String addPerson(@ModelAttribute("person") Person person) {
            personService.save(person);
            return "redirect:/people/";
        }
     
        @RequestMapping("/delete/{personId}")
        @Transactional
        public String deletePerson(@PathVariable("personId") Long personId) {
            personService.delete(personId);
            return "redirect:/people/";
        }
    }
  7. Then I modified the “src/main/resources/applicationContext.xml” Spring config file to use a file for local Neo4j storage in the “default” profile and then in the “prod” profile the “NEO4J_REST_URL”, “NEO4J_LOGIN”, and “NEO4J_PASSWORD” environment variables are used to connect to the Neo4j Heroku add-on service:
    <?xml  version="1.0" encoding="UTF-8"?>
    <beans xmlns="http://www.springframework.org/schema/beans"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xmlns:context="http://www.springframework.org/schema/context"
           xmlns:tx="http://www.springframework.org/schema/tx"
           xmlns:mvc="http://www.springframework.org/schema/mvc"
           xmlns:neo4j="http://www.springframework.org/schema/data/neo4j"
           xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                               http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
                               http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd
                               http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc.xsd
                               http://www.springframework.org/schema/data/neo4j http://www.springframework.org/schema/data/neo4j/spring-neo4j-2.0.xsd">
     
        <context:annotation-config />
        <context:spring-configured />
        <context:component-scan base-package="com.example" />
     
        <neo4j:repositories base-package="com.example.service"/>
     
        <mvc:annotation-driven/>
     
        <bean id="jspViewResolver" class="org.springframework.web.servlet.view.InternalResourceViewResolver">
            <property name="viewClass" value="org.springframework.web.servlet.view.JstlView" />
            <property name="prefix" value="/WEB-INF/jsp/" />
            <property name="suffix" value=".jsp" />
        </bean>
     
        <tx:annotation-driven />
     
        <beans profile="default">
            <neo4j:config storeDirectory="target/neo4j-db"/>
        </beans>
     
        <beans profile="prod">
            <bean class="org.springframework.data.neo4j.rest.SpringRestGraphDatabase" id="graphDatabaseService">
                <constructor-arg index="0" value="#{systemEnvironment['NEO4J_REST_URL']}"/>
                <constructor-arg index="1" value="#{systemEnvironment['NEO4J_LOGIN']}"/>
                <constructor-arg index="2" value="#{systemEnvironment['NEO4J_PASSWORD']}"/>
            </bean>
     
            <neo4j:config graphDatabaseService="graphDatabaseService"/>
        </beans>
     
    </beans>
  8. After testing my changes locally (which actually didn’t work in my webinar due to a problem with Eclipse) I committed my changes to the git repo and pushed them to Heroku:
    git push heroku master

If you want to just skip to a working example on the cloud, simply follow the instructions in the project README.

Hopefully that helps you get started with Neo4j and Java applications on the cloud!

BTW: If you watched the webinar, you probably noticed that my Maven and Eclipse were misbehaving. Turns out that M2E didn’t read my Maven config and all I had to do was right-click on the project, select Maven, and then Update Project Configuration. That got everything back in sync. My excuse for not being able to figure that out during the demo… I usually use IntelliJ IDEA. :)

Heroku, Java, Play and Neo4j Presos: Denver JUG, Atlanta JUG, London Flash UG & Webinar

Over the next couple weeks I’ll be doing two Java User Group presentations, a Flash Platform User Group presentation and one Webinar. Hope to see you at one of these events:

Play Framework 2 & HTML5 on Heroku at Philly ETE and Devoxx Paris

Over the next few weeks I’ll be doing a few presentations about Java, Scala, Play Framework 2, HTML5, and Heroku:

Hope to see you there!

Java on Heroku at NH JUG and DevNexus 2012

This week I’ll will be doing a few presentations about running Java and Play apps on Heroku:

Hope to see you there!