Combining Reactive Streams, Heroku Kafka, and Play Framework

Heroku recently announced early access to the new Heroku Kafka service and while I’ve heard great things about Apache Kafka I hadn’t played with it because I’m too lazy to set that kind of stuff up on my own. Now that I can setup a Kafka cluster just by provisioning a Heroku Addon I figured it was time to give it a try.

If you aren’t familiar with Kafka it is kinda a next generation messaging system. It uses pub-sub, scales horizontally, and has built-in message durability and delivery guarantees. Originally Kafka was built at LinkedIn but is now being used by pretty much every progressive enterprise that needs to move massive amounts of data through transformation pipelines.

While learning Kafka I wanted to build something really simple: an event producer that just sends random numbers to a Kafka topic and a event consumer that receives those random numbers and sends them to a browser via a WebSocket. I decided to use Play Framework and the Akka Streams implementation of Reactive Streams.

In Reactive Streams there is the pretty standard “Source” and “Sink” where an event producer is a Source and a consumer is a Sink. A “Flow” is a pairing between a Source and a Sink with an optional transformation. In my example there are two apps, each with a Flow. A worker process will send random numbers to Kafka so its Source will be periodically generated random numbers and its Sink will be Kafka. In the web process the Source is Kafka and the Sink is a WebSocket that will push the random numbers to the browser.

Here is the worker app with some necessary config omitted (check out the full source):

object RandomNumbers extends App {
 
  val tickSource = Source.tick(Duration.Zero, 500.milliseconds, Unit).map(_ => Random.nextInt().toString)
 
  kafka.sink("RandomNumbers").map { kafkaSink =>
    tickSource
      .map(new ProducerRecord[String, String]("RandomNumbers", _))
      .to(kafkaSink)
      .run()(app.materializer)
  }
 
}

The tickSource is a Source that generates a new random Int every 500 milliseconds. That Source is connected to a Kafka Sink with a Flow that transforms an Int into a ProducerRecord (for Kafka). This uses the Reactive Kafka library which is a Reactive Streams API for working with Kafka.

On the web app side, Play Framework has builtin support for using Reactive Streams with WebSockets so all we need is a controller method that creates a Source from a Kafka topic and hooks that to a WebSocket Flow (full source):

def ws = WebSocket.acceptOrResult[Any, String] { _ =>
  kafka.source(Set("RandomNumbers")) match {
    case Failure(e) =>
      Future.successful(Left(InternalServerError("Could not connect to Kafka")))
    case Success(source) =>
      val flow = Flow.fromSinkAndSource(Sink.ignore, source.map(_.value))
      Future.successful(Right(flow))
  }
}

Notice that the Flow has a Sink.ignore which just says to ignore any incoming messages on the WebSocket (those sent from the browser). Play takes care of all the underlying stuff and then whenever Kafka gets a message on the “RandomNumbers” topic, it will be sent out via the WebSocket.

And it all works!
hello-play-kafka

Check out the full source for instructions on how to get this example setup on your machine and on Heroku. Let me know how it goes!

Building a Mock HVAC for Smart Thermostat Demos

Recently I needed to create a mock HVAC system so that I could have a portable smart thermostat for various demos. I searched around but couldn’t find any such thing. So with some sleuthing and the help of my friend Bruce Eckel I was able to build a simple system that powers a smart thermostat and simulates a heating system. This post will document how to do this in case anyone else ever needs such a thing.

Modern HVAC systems typically provide 24 volt AC power to thermostats so the first thing you will need is a 120 VAC (wall power) to 24 VAC transformer. I used this one:

Then you’ll need something that can simulate a single stage heating system. A smart thermostat uses 24 volt AC to flip on and off a relay (electronic component that uses one power current to toggle another, usually stronger, current). I could have used an actual relay for this but decided to go with something easier, a red LED light that runs on 24 volt AC:

For my smart thermostat I used an ecobee3 (due to their REST APIs) but a Nest should work as well:

Now to wire it together pick a terminal on the transformer to be the “common” and run two wires, one to the LED and one to the “C” terminal on the smart thermostat. Then run a wire from the other side of the LED to the “W1” terminal on the smart thermostat. And finally connect the other terminal on the transformer to the “Rh” terminal on the smart thermostat. Like this:
Ecobee3_Demo_Wiring

Here is what it looks like wired together:
wiring

Once the smart thermostat is powered on and setup, the LED should turn on when the “heater” is on:
on

And turn off when the heater is off:
off

I hope that helps someone out. Let me know if you have any questions!

The 6 Minute Cloud/Local Dev Roundtrip with Spring Boot

Great developer experiences allow you go from nothing to something amazing in under ten minutes. So I’m always trying to see how much I can minimize getting started experiences. My latest attempt is to deploy a Spring Boot app on Heroku, download the source to a developer’s machine, setup & run the app locally, make & test changes, and then redeploy those changes — all in under ten minutes (assuming a fast internet connection). Here is that experience in about six minutes:

To try it yourself, start at the hello-springboot GitHub repo. Let me know how it goes!

Pulling Go Code Colorado Data into Salesforce

This weekend I’m at the Go Code Colorado Challenge Weekend event in Durango. The purpose of Go Code Colorado 2016 is for teams to build something useful for businesses using one or more of the Colorado Public Datasets. Some teams are using Salesforce for the back-office / business process side of the app they are building. So I decided to see if I could pull a Colorado Public Dataset into Salesforce. Turns out it’s super easy! Just follow these steps:

  1. Sign up for a Salesforce Developer Edition
  2. Create a new External Data Source with the following field values:

    External Data Source = Colorado Public Data
    Name = Colorado_Public_Data
    Type = Lightning Connect: OData 2.0
    URL = https://data.colorado.gov/OData.svc
    Special Compatibility = Socrata

    new_external_data_source

  3. Save the new External Data Source and then hit “Validate and Sync” to fetch the metadata for the services.
  4. Select one or more tables from the list. A good table to test with is the “Occupational Employment Statistics” dataset.
    validate_and_sync
    Sync the table and you should see a new “External Object” in the list of External Objects.
    external_objects
  5. The data is now available in Salesforce. An easy way to see the dataset is to create a tab in the Salesforce UI. On the Custom Tabs Setup page create a new Custom Object Tab for the “Occupational Employment Statistics” object and select a Tab Style:
    new_custom_object_tab
    Complete the creation of the tab (select Next, Next, Save).
  6. Select the “Occupational Employment Statistics” tab (which might be in a drop-down menu depending on the width of your browser:
    custom_tabs
    Next to the View – All selector, hit “Go!” to fetch the data from the Colorado Public Data source. You’ll now see the records:
    records
    Note: The columns displayed in this view can be customized in the External Object’s Search Layout.
    Selecting a record’s ID will display the record details:
    record_details

That’s it! Now you can build all sorts of business processes and other employee-facing interactions around the public data.

Good luck to all of the Go Code Colorado teams!

Quick Force Java – Getting Started with Salesforce REST in Java

Recently I blogged about a toolchain that quickly gets you going with the Salesforce REST APIs. I believe developers should be able to get started with new technologies without having to install tons of stuff and struggle for days. That blog used Quick Force Node for those who want to use JavaScript / Node.js. I’ve had a number of requests for a Java version of this toolchain so I created Quick Force Java.

Check out a screencast that shows how to start with nothing, deploy a Salesforce REST app on Heroku, setup OAuth, setup a local dev environment, make & test changes to the app, and then deploy those changes back to the cloud (all in under 12 minutes):

Try out Quick Force Java and let me know how it goes!

Salesforce REST APIs – From Zero to Cloud to Local Dev in Minutes

When getting acquainted with new technologies I believe that users shouldn’t have to spend more than 15 minutes getting something simple up and running. I wanted to apply this idea to building an app on the Salesforce REST APIs so I built Quick Force (Node). In about 12 minutes you can deploy a Node.js app on Heroku that uses the Salesforce REST APIs, setup OAuth, then pull the app down to your local machine, make and test changes, and then redeploy those changes. Check out a video walkthrough:

Ok, now give it a try yourself by following the instructions in Quick Force (Node)!

I hope this will be the quickest and easiest way you’ve gotten started with the Salesforce REST APIs. Let me know how it goes!

FYI: This *should* work on Windows but I haven’t tested it there yet. So if you have any problems please let me know.

Winter Tech Forum 2016 – My Favorite Developer Conference!

I’ve been to a TON of developer conferences and by a landslide my favorite is the Winter Tech Forum (which used to be the Java Posse Roundup). Here is why… Learning for me is experiential.

Typical eyes-forward conferences are like being a passenger on a sail boat. I can watch what is happening but I could definitely not become the captain based on my experience as a passenger. This is what makes WTF different; every attendee is a captain (or maybe a skipper if you are new). The whole conference is the experiences that the attendees want to have. Sometimes that means we write code together, explore new technologies, or discuss ideas. Those experiences have made a significant impact on my technical skills. We also eat together and play together which has helped me build some amazing relationships.

This might sound a little crazy until you actually experience it. Which I highly encourage you to do! This year’s WTF is Feb 29 – March 4 in Crested Butte, CO and will be followed by a new Developer Retreat event that also looks to be awesome. I hope to see you there!

Dreamforce 2015 Video: Tour of Heroku + Salesforce Integration Methods

This year at Dreamforce I presented a session that walked through a few of the ways to integrate Heroku apps with Salesforce. Here is the session description:

Combining customer-facing apps on Heroku with employee-facing apps on Salesforce enables a whole new generation of connected and intelligent experiences. There are four primary ways to do this integration: Heroku Connect, Canvas, Apex / Process Callouts, and the Salesforce REST APIs. Using code and architectural examples, we’ll walk through these different methods. You will walk away knowing when you should use each and how to use them.

Check out the video recording of the session.

To dive into these methods here are the “Further Learning” resources for each method:

I hope this is helpful. Let me know if you have any questions.

Smoothing the Cloud & Local Roundtrip Developer Experience

Getting started with new technologies is usually a huge pain. Often I stumble around for hours trying to get an app’s toolchain setup correctly. Instructions are usually like:

Things get worse when I lead workshops for hundreds of enterprise developers where many are on Windows machines and not very comfortable with cmd.exe.

Experiencing this pain over-and-over is what led me to create Typesafe Activator as a smooth way to get started with Play Framework, Akka, and Scala. Developers have been thrilled with how easy taking their first step with Activator is but I never finished polishing the experience of the second step: App Deployment.

Over the past few months I’ve been working on a set of tools that make the roundtrip between deployment and local development super smooth with zero-CLI and zero-install. Check out a demo:

Here is a summary of the “from scratch” experience:

  1. Deploy the Click, Deploy, Develop app on the cloud
  2. Download the app’s source
  3. Run gulp from a file explorer to download Node, the app’s dependencies, and Atom and then launch the Node / Express server and the Atom code editor
  4. Open the local app in a browser: http://localhost:5000
  5. Make a change in Atom to the app.js file
  6. Test the changes locally
  7. Login to Heroku via Atom
  8. Deploy the changes via Atom

That is one smooth roundtrip!

For more detailed docs on this flow, checkout the Click, Deploy, Develop project.

Great dev experience starts with the simplest thing that can possibly work and has layered escape hatches to more complexity.

That kind of developer experience (DX) is something I’ve tried to do with this toolchain. It builds on top of tools that can be used directly by advanced users. Underneath the smooth DX is just a normal Node.js / Express app, a Gulp build, and the Atom code editor. Here are the pieces that I’ve built to polish the DX, creating the zero-CLI and zero-install experience:

I hope that others find this useful for helping to give new users a great roundtrip developer experience. Let me know what you think.

Note: Currently gulp-atom-downloader does not support Linux because there isn’t a standalone zip download of Atom for Linux. Hopefully we can get that resolved soon.