Heroku at JavaZone 2012

I’m really looking forward to being back at JavaZone in Oslo this September! I’ll be speaking about Running Java, Play! and Scala Apps on the Cloud. This session will teach you how to get started deploying Java and Scala apps on Heroku. I’ll be around for the whole conference, so if anyone wants to grab a drink, deploy some apps, or write some code with me, then let me know. See you there!

Dreamforce 2012: Java Apps on Heroku & Force.com

Dreamforce 2012 is going to be a fantastic event! And not just because Red Hot Chili Peppers, Colin Powell, and Richard Branson will be there. I’ll also be there talking about Building Java Apps on Heroku and Force.com. Here is the description for my session:

In this session you will learn how to build Social Enterprise applications using Salesforce, Heroku, and Java. Through live coding and demonstrations you will learn how to instantly deploy and scale Java apps on the cloud with Heroku. You will also learn how to integrate those applications with Salesforce and Force.com through REST.

Sounds pretty exciting when put next the celebrities, right? Hope to see you there!

Getting Started with Clojure on Heroku

Last week I introduced the Den of Clojure to Heroku. I really enjoyed learning more about Clojure and experiencing super simple Clojure deployment on Heroku. For those who haven’t yet deployed Clojure on Heroku, lets walk through 8 quick steps to get you started:

  1. Install the Heroku Toolbelt and Leiningen
  2. Login to Heroku from the command line:
    heroku login

    If this is your first time logging into Heroku from the command line then you will be led through some steps to associate an SSH key with your Heroku account.

  3. Create a new Leiningen build definition by creating a file named project.clj containing:
    (defproject hello-clojure-noir "0.1.0-SNAPSHOT"
      :main web
      :dependencies [[org.clojure/clojure "1.4.0"]
                     [noir "1.2.1"]])

    As you can see from the dependencies, this simple app uses the Noir web framework.

  4. Create a simple Noir app by creating a file named src/web.clj containing:
    (ns web
      (:use noir.core)
      (:require [noir.server :as server]))
    (defpage "/" [] "hello, world")
    (server/start (Integer/parseInt (or (System/getenv "PORT") "8080")))

    This very basic web app returns “hello, world” for requests to “/”. It starts the server using either the port defined by an environment variable named “PORT” or a default of 8080.

  5. Test this app locally by running:
    lein run

    Then visit http://localhost:8080 in your browser and verify that you see “hello, world”.

  6. To upload this application to Heroku you will first need to create a Git repository, add the files to it, and commit them:
    git init
    git add project.clj src
    git commit -m init
  7. Now create a new application on Heroku:
    heroku create

    This creates an HTTP and a Git endpoint for your application. The Git endpoint will be added to your Git configuration as a “remote” named “heroku”.

  8. Upload your Git repository to the Git repository for your application on Heroku:
    git push heroku master

    This will kick off the Leiningen build process on Heroku. The build will download the dependencies for the app then compile the app and put everything into a “slug” that will be deployed onto a Dyno. Once the process is complete you can open the HTTP endpoint for your app in your browser:

    heroku open

    You should now see “hello, world” coming from the Cloud!

The source for this example is on GitHub.

To learn more about Clojure on Heroku, check out the Heroku Dev Center. Let me know how it goes!

Edge Caching With Play 2, Heroku, and CloudFront

Web applications are primarily comprised of data, services, and the User Interface (UI). The UI is comprised of HTML, CSS, images, and probably JavaScript. In the traditional web architecture all of the UI assets are static files except the HTML which is dynamically generated by the server. In the modern web architecture the entire UI is static files that consume RESTful / JSON services. The static files for the UI must be downloaded to the client so the less time it takes for them to be downloaded, the better the overall performance of the application.

Bits are moved around the world through beams of light (fiber optics). Unfortunately the speed of light just isn’t fast enough when it comes to transferring data. It takes 134ms for light to travel once around the world. Think of all the light and bits that had to move from the Eastern United States (where this article is being served from) to you. Some of the time you spent waiting for the content to load was purely in the moving of bits over long distances. There is an obvious reason why physically moving bits closer to the consumer has massive performance benefits.

It would be amazing if a web application could just live right down the street from every user of the application. But doing so would cause there to be many copies of the data, services and UI for an application. The data usually needs to be consistent across all of the users of the application. Maintaining a geo-distributed and consistent data set is a really hard thing to do without massive data synchronization overhead. The services are just the gateway to the data so they need to be near the data. But the UI, all of those static assets, can easily live in many places; ideally located near the consumers.

This is exactly what a Content Delivery Network (CDN) does. Also known as “edge caching”, a CDN takes copies of static files and replicates them to servers around the world so that whenever someone downloads a static file the bits don’t have to transfer across large distances.

It has often been a big hassle to geo-distibute / edge cache the static assets in web applications. The typical setups for utilizing a CDN involve complex deployment procedures and brittle architectures. But the value of edge caching the static assets is immense for any size web application. The most effective way to improve the overall performance of just about every web application is to use a CDN.

Lets walk through how you can use the Amazon CloudFront CDN, Heroku, and Play 2 to transparently edge cache static assets. Feel free to follow along.

Create a Play 2 App

Create a new Play 2 application:

  1. Install Play 2
  2. Create a new app from the command line:
    play new play2-cloudfront
  3. Confirm the app name and select 3 for an empty project:
    jamesw@T420s:~/Desktop$ play new play2-cloudfront
           _            _ 
     _ __ | | __ _ _  _| |
    | '_ \| |/ _' | || |_|
    |  __/|_|\____|\__ (_)
    |_|            |__/ 
    play! 2.0.3, http://www.playframework.org
    The new application will be created in /home/jamesw/Desktop/play2-cloudfront
    What is the application name? 
    > play2-cloudfront
    Which template do you want to use for this new application? 
      1 - Create a simple Scala application
      2 - Create a simple Java application
      3 - Create an empty project
    > 3
    OK, application play2-cloudfront is created.
    Have fun!

For this example we are going to load jQuery from a WebJar so you can remove the play2-cloudfront/public/javascripts/jquery-1.7.1.min.js file. Now update the play2-cloudfront/project/Build.scala file to include a dependency on the jQuery WebJar and the WebJars repository:

import sbt._
import Keys._
import PlayProject._
object ApplicationBuild extends Build {
    val appName         = "play2-cloudfront"
    val appVersion      = "1.0-SNAPSHOT"
    val appDependencies = Seq(
      "com.jquery" % "jquery" % "1.7.2-1"
    val main = PlayProject(appName, appVersion, appDependencies, mainLang = JAVA).settings(
      resolvers += "webjars" at "http://webjars.github.com/m2"

If you want to use IntelliJ IDEA or Eclipse then you can generate the project files. From the command line within the play2-cloudfront directory, run either:

play idea


play eclipsify

You can now start the Play server from the command line within the play2-cloudfront directory:

play ~run

Verify that the Play server is running by opening the local documentation in your browser: http://localhost:9000/@documentation

Play has a simple static asset controller that serves files from the classpath (any Jar dependency or source directory). However, the Assets Controller doesn’t provide a mechanism in its URL resolver (Play 2’s reverse routing) to change the URL of the asset. We will need this functionality later since loading assets from CloudFront requires using a different, non-relative, domain name. To solve this we will create a new RemoteAssets controller that wraps the Assets controller and optionally adds a domain prefix in front of the resolved URLs.

Create a new file named app/controllers/RemoteAssets.scala that contains:

package controllers
import play.api.mvc._
import play.api.Play
import play.api.Play.current
import org.joda.time.format.{DateTimeFormat, DateTimeFormatter}
import org.joda.time.DateTimeZone
import scala.Some
object RemoteAssets extends Controller {
  private val timeZoneCode = "GMT"
  private val df: DateTimeFormatter =
    DateTimeFormat.forPattern("EEE, dd MMM yyyy HH:mm:ss '"+timeZoneCode+"'").withLocale(java.util.Locale.ENGLISH).withZone(DateTimeZone.forID(timeZoneCode))
  type ResultWithHeaders = Result { def withHeaders(headers: (String, String)*): Result }
  def getAsset(path: String, file: String): Action[AnyContent] = Action { request =>
    val action = Assets.at(path, file)
    val result = action.apply(request)
    val resultWithHeaders = result.asInstanceOf[ResultWithHeaders]
    resultWithHeaders.withHeaders(DATE -> df.print({new java.util.Date}.getTime))
  def getUrl(file: String) = {
    Play.configuration.getString("contenturl") match {
      case Some(contentUrl) => contentUrl + controllers.routes.RemoteAssets.getAsset(file).url
      case None => controllers.routes.RemoteAssets.getAsset(file)

This Scala class has a getAsset method that takes path and file parameters and returns the actual asset in the response. A Date header is also added to the response headers. The getUrl method takes a file parameter and returns a URL to the file. That URL will be prefixed by a contentUrl if one is provided in the application’s configuration. To setup the configuration so that a contentUrl can be optionally provided, add the following to the conf/application.conf file:


If an environment variable named CONTENT_URL is provided then the contenturl configuration parameter is set.

Now lets create and use some static content. First lets write a little CoffeeScript that will use jQuery to fade an image in. This will help to illustrate how even compiled and minimized assets can be loaded from the CDN. Create a new file named app/assets/javascripts/index.coffee containing:

$ ->

This simple script simply fades in all of the images on the page when the page has loaded.

Also update the public/stylesheets/main.css file to give the web page a new background color:

body {
    background-color: #ddddff;

Now create a new server-side template that will load the stylesheet, jQuery (from the WebJar), the index.coffee script, and the public/images/favicon.png image. Create a new file named app/views/index.scala.html containing:

<!DOCTYPE html>
    <title>Play 2 with CloudFront</title>
    <link type='text/css' rel='stylesheet' href='@RemoteAssets.getUrl("stylesheets/main.css")'/>
    <script type='text/javascript' src='@RemoteAssets.getUrl("jquery.min.js")'></script>
    <script type='text/javascript' src='@RemoteAssets.getUrl("javascripts/index.min.js")'></script>
    <img src='@RemoteAssets.getUrl("images/favicon.png")' style="display: none;"/>

Notice how the getUrl method in the RemoteAssets controller is used to get a URL for each asset. Now we need a simple controller that will render the index template. Create a new file named app/controllers/Application.java containing:

package controllers;
import play.mvc.Controller;
import play.mvc.Result;
import views.html.index;
public class Application extends Controller {
  public static Result index() {
    return ok(index.render());

This controller has a single method named index that just returns the rendered index template with a 200 HTTP status.

The last thing to do is to create a mapping between HTTP request verbs & paths and the controller that serves the request. Edit the conf/routes file and add the following:

# Home page
GET     /                           controllers.Application.index()
# Map static resources from the /public folder to the /assets URL path
GET     /assets/*file               controllers.RemoteAssets.getAsset(path="/public", file)

Now GET requests to / will be handled by the controllers.Application.index method and GET requests to /assets/ will be handled by the controllers.RemoteAssets.getAsset method.

This simple little application is now ready for local testing. Open it in your browser: http://localhost:9000

You should see the Play logo fade in on top of a purple-ish background. This page required a total of five HTTP requests:

Local HTTP Requests

Lets take this application and deploy it on the cloud with Heroku and then we will setup CloudFront to serve the static assets.

Deploy on Heroku

Heroku is a Cloud Application Platform that can run many different types of apps. To deploy this application on Heroku:

  1. Signup for a Heroku Account
    Note: These instructions will not use Heroku beyond the free tier.
  2. Install Git (Or use Git from your IDE)
  3. Install the Heroku Toolbelt
  4. Login to Heroku from the command line:
    heroku login

    If this is your first time using the Heroku Toolbelt then you will be led through the steps to associate an SSH key with your Heroku account. This SSH key will be used to authenticate your uploads via Git.

  5. From the command line in your play2-cloudfront directory, create a new Git repository, add your files to it, and commit them:
    git init
    git add app conf project public
    git commit -m init
  6. From the command line in your play2-cloudfront directory, provision a new application on Heroku:
    heroku create

    This will create a new application with corresponding HTTP and Git endpoints, like:

    Creating peaceful-retreat-3158... done, stack is cedar
    http://peaceful-retreat-3158.herokuapp.com/ | git@heroku.com:peaceful-retreat-3158.git
    Git remote heroku added
  7. To deploy your application on Heroku simply upload your Git repository to Heroku:
    git push heroku master

    This will push the master branch of your Git repository to the Git remote named heroku which in my case points to the git@heroku.com:peaceful-retreat-3158.git URL. When Heroku receives the files it will run the project build (SBT for Play 2 projects), then deploy and run the application. When the application is running you can access it in your browser:

    heroku open

This time all five requests go to Heroku:

Local HTTP Requests

The requests take quite a bit longer than locally, in-part because the bits have a much larger distance to travel. All of the requests except the index page (because it’s dynamic) can be served from a CDN. Now lets setup CloudFront to serve the static assets.

Serve Static Assets with CloudFront

CloudFront has a very simple way to load static assets into it’s CDN. When a request comes into CloudFront, if the asset is not on the CDN or has expired, then CloudFront can get the asset from an “origin server”. The application you just deployed on Heroku will now be the origin server for the static assets. To setup a new CloudFront “Distribution”:

  1. Signup for a AWS Account
    Note: CloudFront does not have a free tier. So following these instructions will cost you a tiny bit.
  2. Open the CloudFront Management Console
  3. Select Create Distribution
  4. Leave Download selected as the delivery method and select Continue
  5. In the Origin Domain Name field enter the domain name for your application on Heroku. In my case it is: peaceful-retreat-3158.herokuapp.com
  6. Keep the other default values as-is and select Continue

    CloudFront Setup

  7. Do the same for the next two steps (keep the defaults)
  8. Select Create Distribution

It will now take about ten minutes for AWS to create the CloudFront distribution. You can monitor the status in the AWS Console. While you wait, take note of the domain name provided for your distribution. Mine is: d7471vfo50fqt.cloudfront.net

You can test the status of distribution by making a request for the favicon.png file. In my case the full URL is: http://d7471vfo50fqt.cloudfront.net/assets/images/favicon.png

The first time that request goes through, CloudFront will make a request back to the app on Heroku and then load the asset into the CDN. If you examine the HTTP response headers on that request you will see:

X-Cache: Miss from cloudfront

That indicates that the resource was not on the CDN. A subsequent request should contain the following response header:

X-Cache: Hit from cloudfront

That indicates that the resource was served from the CDN and there was no need to go back to the origin server.

Now that the static assets are loadable via CloudFront lets tell the app on Heroku and the RemoteAssets controller to point to them. Just set the CONTENT_URL environment variable on your application by running the following from the command line (make sure you replace the URL value with the one for the distribution you just created):

heroku config:add CONTENT_URL="http://d7471vfo50fqt.cloudfront.net"

Now test out your application on Heroku in your browser:

heroku open

You should now see all four static asset requests going to CloudFront:

Heroku and CloudFront - Miss

But as you can see they assets didn’t load very quickly because the first request is a Miss from cloudfront. Reload the page (clear your cache to avoid 304s) and you should see much faster responses:

Heroku and CloudFront - Hit

And now your static assets are being edge cached!

Learn More

Using a CDN is step one of significantly speeding up your web applications but there is certainly more that you can do. By default Play sets the expiration time of static assets to 1 hour (via the Cache-Control response header):

Cache-Control: max-age=3600

You can change that value by modifying the application.conf file (more details). Often times you will also want to use far future expires and use naming conventions to instruct the client to fetch a new version of a static asset.

Grab the source for this example.

Check out the live demo.

Let me know if you have any questions.

UPDATE: Check out the Play 2 CloudFront Module!

Integrating Java Spring Apps on Heroku with Force.com REST APIs

Recently I co-presented a webinar about how to integrate Java Spring Apps on Heroku with the Force.com / Salesforce.com REST APIs. Check out the recording:

I’ve also created an in-depth walk through of the code example and step-by-step instructions for setting up and deploying the example Java Spring app on Heroku. Check it out and let me know how it goes.

The Magic Behind Heroku’s “git push” Deployment

In my spare time I help out with a little app called Greg’s Toolkit that was built before I knew about Heroku. The app runs on EC2 and deploying new versions of the app is pretty tedious. Here is the deployment instructions copied directly from the project’s wiki:

./gradlew war
scp  ./gft_server/build/libs/gft_server.war api.gregstoolkit.com:
ssh api.gregstoolkit.com
cd /opt/apache-tomcat-7.0.21
sudo -u www-data bin/shutdown.sh
cd webapps/ROOT
sudo rm -r *
sudo -u www-data jar -xvf ~/gft_server.war
sudo -u www-data sed -i 's/dev/prod/' WEB-INF/web.xml
cd ../..
sudo -u www-data bin/startup.sh

That certainly isn’t as cumbersome as some deployment methods but it is time consuming, error-prone, and causes downtime.

If you’ve used Heroku one of the most magical things you first come across is Instant Deployment. With Heroku you just upload your code to Heroku and that kicks off a process that results in the new version of the app being deployed. Heroku uses the Git tool as a way for developers to upload an app’s source to Heroku. So to deploy a new version of an app on Heroku (from the command line), you just run:

git push heroku master

It’s magic! But what actually happens on Heroku when you “push” a Git repository to Heroku? Anything you want! A “Buildpack” translates whatever you push to Heroku into something that can be run on Heroku. Heroku provides a number of out-of-the-box Buildpacks like the Maven Buildpack (for Java projects), the NPM Buildpack (for Node.js projects), and many others.

Now here is the really cool thing… The Buildpack system on Heroku is totally open and pluggable. There is even a documented API for the Buildpack system. So if you want to modify the process the takes your code from the “git push” to a runnable application, you can either: customize an existing Buildpack, create an entirely new Buildpack from scratch, or use a third party Buildpack.

Lets try a really simple example using a third party Buildpack. The Null Buildpack literally does nothing on a “git push”. Whatever you push gets deployed. Lets try it out. I’m going to use Python since it includes a simple web server in the base system and doesn’t need to be compiled. Follow along:

  1. Install Git and the Heroku Toolbelt
  2. Login to Heroku from the command line:
    heroku login
  3. In a new directory create a new file named “web.py” containing:
    import os
    from wsgiref.simple_server import make_server
    port = os.environ.get("PORT", "5000")
    def hello_world_app(environ, start_response):
        status = '200 OK'
        headers = [('Content-type', 'text/plain')]
        start_response(status, headers)
        return ["hello, world"]
    httpd = make_server('', int(port), hello_world_app)
    print "Serving HTTP on port " + port + "..."
  4. If you want to test the app locally run:
    python web.py

    Then in your browser go to http://localhost:5000 and you should see “hello, world”.

  5. To run this simple app on Heroku you need to let Heroku know what command to run on the system. To do this create a file named “Procfile” (in the same directory as “web.py”) containing:
    web: python web.py

    This tells Heroku that for the process named “web” run “python web.py” on the system. The managed systems that run apps on Heroku are called “Dynos“. You can allocate as many Dynos as needed to each process in your app.

  6. Since Heroku uses Git for file upload and download you need to create a new Git repository, add your files to it, and commit them:
    git init
    git add Procfile web.py
    git commit -m init
  7. Now provision (i.e. create) a new app on Heroku but instead of using the default set of Buildpacks, explicitly tell Heroku to use the Null Buildpack:
    heroku create --buildpack https://github.com/ryandotsmith/null-buildpack.git

    This provisions the app on Heroku and sets a new Git remote named “heroku” in the configuration for your Git repository. The “heroku” remote is just a name that points to the Git URL for your app on Heroku.

  8. Now push your app to the “heroku” Git remote repository:
    git push heroku master

Once the files are uploaded, Heroku’s Slug Compiler uses the specified Buildpack to perform the build, converting source code into something runnable. Everything that is needed to run the app (with the exception of the base system) goes into a “slug”. Then the slug is copied to a Dyno and the process named “web” is run. Here is the output from my “git push”:

Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 522 bytes, done.
Total 4 (delta 0), reused 0 (delta 0)
-----> Heroku receiving push
-----> Fetching custom buildpack... done
-----> Null app detected
-----> Nothing to do.
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size is 4K
-----> Launching... done, v4
       http://blazing-waterfall-8915.herokuapp.com deployed to Heroku
To git@heroku.com:blazing-waterfall-8915.git
 * [new branch]      master -> master

Check out the app in your browser by running:

heroku open

That’s it! Even though it feels magical, it is really a pretty simple (and totally customizable) process.

Lets recap what happens when you do a “git push heroku master”:

  1. The Git repository is uploaded to Heroku.
  2. The Slug Compiler uses either a custom specified Buildpack or a default Buildpack to transform the source that was pushed into something runnable. This process can be as simple as doing nothing or as complex is resolving dependencies, compiling source, packaging assets, running tests, etc. All files in the project directory when the Buildpack finishes are put into the slug.
  3. The slug is copied to new Dyno(s). (Usually the default is 1 Dyno for the “web” process.)
  4. The defined processes are started on the Dynos.

Pretty simple! Here is another cool feature of Heroku’s Buildpack system… If the Buildpack returns an error then the process stops. So with compiled languages you get some automatic “testing” of the code before it is deployed.

To recap: A Buildpack translates whatever you push to Heroku into something that can be run on Heroku. There are tons and tons of Buildpacks out there (mostly on GitHub). Next time you need to deploy something out of the ordinary, see if there is a Buildpack for it already. If not, then why not create your own? It’s actually really simple. My first custom Buildpack simply converts Markdown files to HTML so that I can instantly deploy Markdown on the web. It is under 40 lines of code and was pretty simple to write. If I can create a Buildpack then you probably can too! :)

Play 2 Scala Console on Heroku

I’ve been working on a Play 2 application that I’ll be using for the Grails vs Play Smackdown at ƜberConf next week. The app is running in production on Heroku but since I don’t have an admin UI yet, I needed a quick and easy way to create a new entity. I could have gone straight to the database but thought it would be better to run Play 2’s Scala Console on Heroku and then just run some arbitrary Scala code. The Scala Console in Play 2 is really just the Scala REPL in SBT but it allows you to interact with a Play application.

I’ll walk you through how to use the Play 2 Scala Console on Heroku. If you’d like to follow along, grab the play2bars app by running:

git clone https://github.com/jamesward/play2bars.git
cd play2bars
git checkout java-ebean

To run the console locally, run:

play -DapplyEvolutions.default=true console

You will see something like:

[info] Loading project definition from /home/jamesw/Desktop/play2bars/project
[info] Set current project to play2bars-java (in build file:/home/jamesw/Desktop/play2bars/)
[info] Updating {file:/home/jamesw/Desktop/play2bars/}play2bars-java...
[info] Resolving org.hibernate.javax.persistence#hibernate-jpa-2.0-api;1.0.1.Fin                                                                                [info] Done updating.                                                        
[info] Compiling 4 Scala sources and 3 Java sources to /home/jamesw/Desktop/play2bars/target/scala-2.9.1/classes...
[warn] Note: /home/jamesw/Desktop/play2bars/app/models/Bar.java uses unchecked or unsafe operations.
[warn] Note: Recompile with -Xlint:unchecked for details.
[info] Starting scala interpreter...
Welcome to Scala version 2.9.1.final (OpenJDK 64-Bit Server VM, Java 1.7.0_03).
Type in expressions to have them evaluated.
Type :help for more information.

Start a Play app from the Scala console:

new play.core.StaticApplication(new java.io.File("."))

You will see something like:

[info] play - database [default] connected at jdbc:h2:mem:play
[info] play - Application started (Prod)
res0: play.core.StaticApplication = play.core.StaticApplication@2e338c56

Now you can interact with the Play application. Lets create a new “Bar” entity:

var bar = new models.Bar()
bar.name = "foo bar"

And now you can query the “Bar” entities:

import scala.collection.JavaConversions._
models.Bar.find.all.foreach(bar => println(bar.name))

So that’s pretty cool, right?

If you want to run this example on Heroku, then install the Heroku Toolbelt and run:

heroku create -s cedar

And push the application to Heroku:

git push heroku java-ebean:master

Once the application is built and deployed on Heroku, check to make sure it works:

heroku open

To run the Scala console on Heroku, run:

heroku run bash
java -DapplyEvolutions.default=true -Ddb.default.driver=org.postgresql.Driver -Ddb.default.url=$DATABASE_URL -jar ~/.sbt_home/bin/sbt-launch-0.11.3-2.jar

SBT will resolve some dependencies and then you will be in the SBT console. From there run:

set fullClasspath in Compile += Attributed.blank(file("target/staged/*"))

Now you will be in the Scala REPL for your Play app on Heroku. So you can run things like:

new play.core.StaticApplication(new java.io.File("."))
var bar = new models.Bar()
bar.name = "foo bar"

Reload your app on Heroku in the browser and you should now see the new Bar listed on the page!

Let me know if you have any questions.