Quick & Easy ETL from Salesforce to MySQL with Workflow & Heroku

While sometimes unfortunate it is often necessary to have data silos that share data. The Extract, Transform, and Load (ETL) pattern has been around for a long time to address this need and there are tons of solutions out there. If you just need a quick and easy way to copy new & updated records in Salesforce to an external data source, a simple Heroku app and Salesforce Workflow might be the quickest and easiest solution. I’ve put together a sample Node.js application for this: https://github.com/jamesward/salesforce-etl-mysql

Check out a demo:

This sample app uses a Salesforce Workflow that sends created and updated Contact records to an app on Heroku, which inserts or updates those records in a MySQL database. A simple transform JavaScript function makes it easy to customize this app for your own use case. To setup this app for your own uses, check out the instructions in the GitHub repo: https://github.com/jamesward/salesforce-etl-mysql

Let me know how it goes!

Salesforce REST APIs – From Zero to Cloud to Local Dev in Minutes

When getting acquainted with new technologies I believe that users shouldn’t have to spend more than 15 minutes getting something simple up and running. I wanted to apply this idea to building an app on the Salesforce REST APIs so I built Quick Force (Node). In about 12 minutes you can deploy a Node.js app on Heroku that uses the Salesforce REST APIs, setup OAuth, then pull the app down to your local machine, make and test changes, and then redeploy those changes. Check out a video walkthrough:

Ok, now give it a try yourself by following the instructions in Quick Force (Node)!

I hope this will be the quickest and easiest way you’ve gotten started with the Salesforce REST APIs. Let me know how it goes!

FYI: This *should* work on Windows but I haven’t tested it there yet. So if you have any problems please let me know.

NPM Packages in Maven Central with NPM WebJars

A few months ago I launched Bower WebJars which provides a way for anyone to deploy Bower packages into Maven Central through WebJars. Since then 539 packages have been deployed! Today I’ve added NPM WebJars which is built on the same foundation as Bower WebJars but for NPM packages.

Give it a try and let me know how it goes. If you are curious about the changes to make this happen, check out the pull request.

Introducing Gulp Launcher

Many developers already have the Node.js toolchain installed on their machines but when I lead workshops there are always a few who don’t. The process of installing Node build toolchains can take quite a bit of time for new users (especially on Windows). To simplify the process of getting the gulp toolchain setup, Bruce Eckel and I created gulp launcher. With a fresh system you can run gulp with only one download and one command:

$ gulp
Downloading jq 1.3 for x86_64 MAC
Downloading Node 0.10.33 for x86_64 MAC
... npm install output ...
[09:13:25] Using gulpfile ~/projects/gulp-starter/gulpfile.js
[09:13:25] Starting 'default'...
[09:13:25] Finished 'default' after 7.28 μs

On Windows the user can just double click on gulp.exe and it will run the default task for the gulp build.

If you want to try it out either download gulp (Linux, Mac, and Cygwin) or gulp.exe (Windows) from the latest release. Or if you want a ready-to-go starter gulp project, grab the gulp-starter project.

I’ve setup automated tests for the gulp launcher on Travis CI and AppVeyor but there isn’t a lot of real-world testing yet. So if you discover any problems please file an issue. Let me know how it goes. Thanks!

Getting Started with Node.js on The Cloud

In my new job at salesforce.com I’m incredibly exited about getting into Heroku, a Platform as a Service provider / Cloud Application Platform. In a future blog post I’ll provide more details on what Heroku is and how it works. But if you are like me the first thing you want to do when learning a new technology is to take it for a test drive. I decided to take my Heroku test drive using the recently announced Node.js support. I’m new to Node.js, but at least I know JavaScript. Heroku also offers Ruby / Rails support but I don’t know Ruby – yet. So let me walk you through the steps I took (and that you can follow) to get started with Node.js on the Heroku Cloud.

Step 1) Sign up for Heroku

Step 2) Install the Heroku command line client

Step 3) Login to Heroku via the command line:

heroku auth:login

Step 4) Install git

Step 5) Install Node.js

Step 6) Create a Node.js app

I started by building a very simple “hello, world” Node.js app. In a new project directory I created two new files. First is the package.json file which specifies the app metadata and dependencies:

{
  "name": "hellonode",
  "version": "0.0.1",
  "dependencies": {
    "express": "2.5.11"
  },
  "engines": {
    "node": "0.8.4",
    "npm": "1.1.45"
  }
}

Then the actual app itself contained in a file named web.js:

var express = require('express');
 
var app = express.createServer(express.logger());
 
app.get('/', function(request, response) {
  response.send('hello, world');
});
 
var port = process.env.PORT || 3000;
console.log("Listening on " + port);
 
app.listen(port);

This app simply maps requests to “/” to a function that sends a simple string back in the response. You will notice that the port to listen on will first try to see if it has been specified through an environment variable and then fallback to port 3000. This is important because Heroku can tell our app to run on a different port just by giving it an environment variable.

Step 7) Install the app dependencies with npm:

npm install .

This uses the package.json file to figure out what dependencies the app needs and then copies them into a “node_modules” directory.

Step 8) Try to run the app locally:

node web.js

You should see “Listening on 3000” to indicate that the Node.js app is running! Try to open it in your browser:
http://localhost:3000/

Hopefully you will see “hello, world”.

Step 9) Heroku uses a “Procfile” to determine how to actually run your app. Here I will just use a Procfile to tell Heroku what to run in the “web” process. But the Procfile is really the foundation for telling Heroku how to run your stuff. I won’t go into detail here since Adam Wiggins has done a great blog post about the purpose and use of a Procfile. Create a file named “Procfile” in the project directory with the following contents:

web: node web.js

This will instruct Heroku to run the web app using the node command and the web.js file as the main app. Heroku can also run workers (non-web apps) but for now we will just deal with web processes.

Step 10) In order to send the app to Heroku the files must be in a local git repository. To create the local git repo, run the following inside of your project directory:

git init

Now add the three files you’ve created to the git repo:

git add package.json Procfile web.js

Note: Make sure you don’t add the node_modules directory to the git repo! You can have git ignore it by creating a .gitignore file containing just “node_modules”.

Commit the files to the local repo:

git commit -m "initial commit"

Step 11) Create an app on Heroku:

heroku create

A default / random app name is automatically assigned to your app.

Step 12) Now you can push your app to Heroku! Just run:

git push heroku master

You should see something like:

Counting objects: 8, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 729 bytes, done.
Total 6 (delta 2), reused 0 (delta 0)
 
-----> Heroku receiving push
-----> Node.js app detected
-----> Resolving engine versions
       Using Node.js version: 0.8.4
       Using npm version: 1.1.41
-----> Fetching Node.js binaries
-----> Vendoring node into slug
-----> Installing dependencies with npm
       npm WARN package.json hellonode@0.0.1 No README.md file found!
       npm http GET https://registry.npmjs.org/express/2.5.11
       npm http 200 https://registry.npmjs.org/express/2.5.11
       npm http GET https://registry.npmjs.org/express/-/express-2.5.11.tgz
       npm http 200 https://registry.npmjs.org/express/-/express-2.5.11.tgz
       npm http GET https://registry.npmjs.org/connect
       npm http GET https://registry.npmjs.org/mime/1.2.4
       npm http GET https://registry.npmjs.org/qs
       npm http GET https://registry.npmjs.org/mkdirp/0.3.0
       npm http 200 https://registry.npmjs.org/qs
       npm http 200 https://registry.npmjs.org/mkdirp/0.3.0
       npm http 200 https://registry.npmjs.org/mime/1.2.4
       npm http 200 https://registry.npmjs.org/connect
       npm http GET https://registry.npmjs.org/qs/-/qs-0.4.2.tgz
       npm http GET https://registry.npmjs.org/connect/-/connect-1.9.2.tgz
       npm http GET https://registry.npmjs.org/mkdirp/-/mkdirp-0.3.0.tgz
       npm http GET https://registry.npmjs.org/mime/-/mime-1.2.4.tgz
       npm http 200 https://registry.npmjs.org/qs/-/qs-0.4.2.tgz
       npm http 200 https://registry.npmjs.org/mime/-/mime-1.2.4.tgz
       npm http 200 https://registry.npmjs.org/connect/-/connect-1.9.2.tgz
       npm http 200 https://registry.npmjs.org/mkdirp/-/mkdirp-0.3.0.tgz
       npm WARN package.json connect@1.9.2 No README.md file found!
       npm http GET https://registry.npmjs.org/formidable
       npm http 200 https://registry.npmjs.org/formidable
       npm http GET https://registry.npmjs.org/formidable/-/formidable-1.0.11.tgz
       npm http 200 https://registry.npmjs.org/formidable/-/formidable-1.0.11.tgz
       express@2.5.11 node_modules/express
       ├── qs@0.4.2
       ├── mime@1.2.4
       ├── mkdirp@0.3.0
       └── connect@1.9.2 (formidable@1.0.11)
       npm WARN package.json hellonode@0.0.1 No README.md file found!
       npm WARN package.json connect@1.9.2 No README.md file found!
       express@2.5.11 /tmp/build_h2gd35fmkgzm/node_modules/express
       connect@1.9.2 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/connect
       qs@0.4.2 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/qs
       mime@1.2.4 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/mime
       formidable@1.0.11 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/connect/node_modules/formidable
       mkdirp@0.3.0 /tmp/build_h2gd35fmkgzm/node_modules/express/node_modules/mkdirp
       Dependencies installed
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size is 4.0MB
-----> Launching... done, v5
       http://fast-plateau-8288.herokuapp.com deployed to Heroku
 
To git@heroku.com:fast-plateau-8288.git
   26b4efc..6e333a2  master -> master

Now you should be able to connect to your app in the browser! An easy way to open the app in your browser is to run:

heroku open

To see your app logs (provisioning, management, scaling, and system out messages) run:

heroku logs

To see your app processes run:

heroku ps

And best of all, if you want to add more Dynos* just run:

heroku scale web=2

* Dynos are the isolated containers that run your web and other processes. They are managed by the Heroku Dyno Manifold. Learn more about Dynos.

That increases the number of Dynos running the app from one to two. Automatically Heroku will distribute the load across those two Dynos, detect dead Dynos, restart them, etc! That is seriously easy app scalability!

There is much more to Heroku and I’ll be continuing to write about it here. But in the meantime, check out all of the great docs in the Heroku Dev Center. And please let me know if you have any questions or problems. Thanks!