Sending Play Framework File Uploads to Amazon S3

UPDATE: I’ve released a S3 Play Module based on this project.

A couple of questions [1, 2] on StackOverflow.com led me to look into how we can send file uploads in a Play Framework application to Amazon S3 instead of the local disk. For applications running on Heroku this is especially important because the local disk is not persistent. Persistent disk storage makes it hard to scale apps. Instead of using the file system, it’s better to use an external service which is independent of the web tier.

While at JavaZone I sat down with Peter Hilton and Nicolas Leroux to come up with a way to handle this. It only took us 30 minutes to get something working – start to finish – including setup time. This is what is so compelling about Play Framework. I’ve built many Java web apps and it always seems like I spend too much time setting up builds, IDEs, and plumbing. With Play we were setup and working on the actual app in less than a minute. After getting everything working locally it took another minute to actually run it on the cloud with Heroku. The combination of Play Framework and Heroku is a developer’s dream for fast-paced development and deployment.

All of the code for the sample application is on github:
https://github.com/jamesward/plays3upload

The basics of what we did was this:

public static void doUpload(String comment, File attachment)
{
    AWSCredentials awsCredentials = new BasicAWSCredentials(System.getenv("AWS_ACCESS_KEY"), System.getenv("AWS_SECRET_KEY"));
    AmazonS3 s3Client = new AmazonS3Client(awsCredentials);
    s3Client.createBucket(BUCKET_NAME);
    String s3Key = UUID.randomUUID().toString();
    s3Client.putObject(BUCKET_NAME, s3Key, attachment);
    Document doc = new Document(comment, s3Key, attachment.getName());
    doc.save();
    listUploads();
}

This uses a JPA Entity to persist the metadata about the file upload (for some reason we named it ‘Document’) and a reference to the file’s key in S3. But there was a sexier way, so my co-worker Tim Kral added a new S3Blob type that could be used directly in the JPA Entity. Tim also cleaned up the configuration to make it more Play Framework friendly. So lets walk through the entire app so you can see the pieces.

The app/models/Document.java JPA Entity has three fields – the file being of type S3Blob:

package models;
 
import javax.persistence.Entity;
 
import play.db.jpa.Model;
import s3.storage.S3Blob;
 
@Entity
public class Document extends Model
{
    public String fileName;
    public S3Blob file;
    public String comment;
}

The S3Blob is now doing all of the work to talk to the Amazon S3 APIs to persist and fetch the actual file.

Configuration of S3 is done by adding a plugin to the conf/play.plugins file:

0: s3.storage.S3Plugin

The S3Plugin handles reading the AWS credentials from the conf/application.conf file, setting up the S3Client, and creating the S3 Bucket – if necessary.

In the conf/application.conf file, environment variables are mapped to the configuration parameters in the Play application:

aws.access.key=${AWS_ACCESS_KEY}
aws.secret.key=${AWS_SECRET_KEY}
s3.bucket=${S3_BUCKET}

The values could be entered into the conf file directly but I used environment variables so they would be easier to change when running on Heroku.

The Amazon AWS API must be added to the conf/dependencies.yml file:

require:
    - play
    - com.amazonaws -> aws-java-sdk 1.2.7

The sample application has a new controller in app/controllers/Files.java that can display the upload form, handle the file upload, display the list of uploads, and handle the file download:

package controllers;
 
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.util.List;
 
import models.Document;
import play.libs.MimeTypes;
import play.mvc.Controller;
import s3.storage.S3Blob;
 
public class Files extends Controller
{
 
  public static void uploadForm()
  {
    render();
  }
 
  public static void doUpload(File file, String comment) throws FileNotFoundException
  {
    final Document doc = new Document();
    doc.fileName = file.getName();
    doc.comment = comment;
    doc.file = new S3Blob();
    doc.file.set(new FileInputStream(file), MimeTypes.getContentType(file.getName()));
 
    doc.save();
    listUploads();
  }
 
  public static void listUploads()
  {
    List<Document> docs = Document.findAll();
    render(docs);
  }
 
  public static void downloadFile(long id)
  {
    final Document doc = Document.findById(id);
    notFoundIfNull(doc);
    response.setContentTypeIfNotSet(doc.file.type());
    renderBinary(doc.file.get(), doc.fileName);
  }
 
}

The uploadForm() method just causes the app/views/Files/uploadForm.html page to be displayed.

The doUpload() method handles the file upload and creates a new Document object that stores the file in S3 and the comment in a database. After storing the file and comment it runs the listUploads() method. Of-course a database must be configured in the conf/application.conf file. For running on Heroku the database is provided and just needs to be configured with the following values:

db=${DATABASE_URL}
jpa.dialect=org.hibernate.dialect.PostgreSQLDialect
jpa.ddl=update

The listUploads() method fetches all Document objects out of the database and then displays the apps/views/files/listUploads.html page.

If a user selects a file from the list then the downloadFile() method is called which finds the file in S3 and sends it back to the client as a binary stream. An alternative to this would be to get the file directly from Amazon using either the S3 generatePresignedUrl() method or via CloudFront.

Finally in the conf/routes file, requests to “/” have been mapped to the Files.uploadForm() method:

GET     /                                       Files.uploadForm

That’s it! Now we have an easy way to persist file uploads in an external system!

Running the Play! app on Heroku

If you’d like to run this example on Heroku, here is what you need to do:

Install the heroku command line client on Linux, Mac, or Windows.

Login to Heroku via the command line:

heroku auth:login

Clone the git repo:

git clone git@github.com:jamesward/plays3upload.git

Move to the project dir:

cd plays3upload

Create the app on Heroku:

heroku create -s cedar

Set the AWS environment vars on Heroku:

heroku config:add AWS_ACCESS_KEY="YOUR_AWS_ACCESS_KEY" AWS_SECRET_KEY="YOUR_AWS_SECRET_KEY" S3_BUCKET="AN_AWS_UNIQUE_BUCKET_ID"

Upload the app to Heroku:

git push heroku master

Open the app in the browser:

heroku open

Let me know if you have any questions or problems. And thanks to Peter, Nicolas, and Tim for helping with this!

This entry was posted in Heroku, Java, Play Framework. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.
  • http://www.cloudberrylab.com Girlincloud

    I am wondering if you are interested in reviewing CloudBerry Explorer freeware that helps to manage S3 and CloudFront http://www.cloudberrylab.com/s3

    Girlincloud,
    CloudBerry Lab team

  • http://www.vanderveer.be Roderik van der Veer

    Hey,

    thanks for this post, just what i was looking for for our now play project on heroku :)
    Do you know if there is any work done to create a module out of this? It would make it a lot cleaner to apply to projects.

    • http://www.jamesward.com James Ward

      Thanks! I was thinking about creating an official module for this but haven’t had the chance. Let me know if you want to work together on that.

      • http://www.vanderveer.be Roderik van der Veer

        Will give it a try, we’ve just started exploring Play! so some further study on how to do this is needed :)

      • http://www.vanderveer.be Roderik van der Veer

        Allright, i gave it a whirl: https://github.com/roderik/S3-Blobs-module-for-Play

        First attempt at a module before i even made a site with Play (apart from the blog example), so please be gentle :) Not sure how the whole route thing and modules “works” within a real project but at least it’s a start.

        • http://www.jamesward.com James Ward

          Awesome! I’ve forked it and will take a look. Thanks!

  • http://twitter.com/GokhanOrun Gökhan ÖRÜN

    Hi, 

    first thanks for share. I have questions ;

    1. How can I change the file name and destination folder on s3 while uploding
    2.How can I make file (image) visible to everyone, I’ll plan to use s3 for my user’s profile photos.

    best regards.

  • Pingback: Just Released the S3Blobs Play Framework Module for Amazon S3()

  • brainztorm1 brainztorm1

    Hi James,
    Thanks for this module !

    i have a question though,i would like to use the generatePresignedUrl method you talk about but in the module code (S3Blob.java), the static field “static AmazonS3 s3Client” is not public so i cannot do a
    s3Client.generatePresignedUrl(…) (in order to get an image url for my view).

    I could easily change that and re-compile the module but how Heroku handle custom play module ?I’ve seen some alternatives on the web
    (http://www.proquestit.com/heroku-moves-your-developers-to-best-practise) but as a perfectionist i would like your advices on that ;)

    Thanks again.
    François.

  • http://profile.yahoo.com/IOI3J7BHDPDP6GEZHQKLYA32Y4 chnoor

    Hi, thanks for share, I’m new in using play, when I wanted to run the test example (plays3upload) the error is raised  (com.amazonaws.services.s3.AmazonS3 cannot be resolved) any help? Thanks..

    • http://www.jamesward.com James Ward

      Are you using this code or the Play 1 S3Blobs module?  http://www.playframework.org/modules/s3blobs

      • Chnoorhawramy

         I tried both, I am usin play-1.2.4, does I need to do any thing else except installing s3blob? I need to upload a pdf file to my project and then download it again through the project, any help will be greatly appreciate

        • http://www.jamesward.com James Ward

          It should just work.  What is the problem you are having?

          • Chnoorhawramy

             but it does not, I do’nt now why, here is the compilation error :
            The file /app/play/modules/s3blobs/S3Blob.java could not be compiled.
            Error raised is : com.amazonaws.services.s3.AmazonS3 cannot be resolved

          • http://www.jamesward.com James Ward

            You shouldn’t need that class if you are using the module. Can you start from scratch and follow the instructions here: http://www.playframework.org/modules/s3blobs-0.1/home

          • Chnoorhawramy

            I tried again, this time I got

            The file /app/controllers/Files.java could not be compiled.
            Error raised is : play.modules.s3blobs.S3Blob cannot be resolved

            really sorry for any inconvenience

          • Chnoorhawramy

            Hi again, what I have to run is plays3upload file inside modules-> s3blobs-0.1->samples-and-tests ?? or it is not???  thanks

          • http://www.jamesward.com James Ward

            Sorry.  My bad.  That sample has some issues.  Try this one:
            https://github.com/jamesward/plays3upload

            And make sure you run:
            play deps –sync

          • Chnoorhawramy

            Hi,
            thanks James, but really still I have the same problem (play.modules.s3blobs.S3Blob cannot be resolved) , why is that happen? :(

          • http://www.jamesward.com James Ward

            Did you run:
            play deps –sync

            Then you should have a directory called modules containing a file named “s3blobs-0.1″.

          • Chnoorhawramy

            many  thanks james, it does work

  • Ming-der Wang

    Hi James, could you tell us how to remove a file from S3 when you remove its respective Document instance for local database. Thanks in advance.

    • http://www.jamesward.com James Ward

      AFAIK there isn’t a way to do that.  You have to manually do the delete on the S3Object.

      • Ashok

        If  the container of your S3 blob  extends the Model class, then its being persisted using hibernate. You can set up en entity listener in hibernate (by config or annotation) which can callback your code to delete the S3 blob.  

  • Alexander Hanschke

    Hi James,

    while porting the module to Play2 I just figured that AmazonS3#getObjectMetadata(bucket, key) throws an exception, if the specified resource does not exist. I used this method the same way as you did in S3Blob#exists(), but it will either be set, or the aforementioned exception will be thrown. Couldn’t find the issue section for the module, so I posted here – feel free to move it :)

    Cheers,
    Alex

    • http://www.jamesward.com James Ward

      Cool!  Glad to hear you are working on this.  I can’t remember what the behavior is, and maybe my library had a bug.  Here is their API doc: http://docs.amazonwebservices.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html#getObjectMetadata(java.lang.String, java.lang.String)

      Where is your code at?  Maybe I can take a look.

      • Alexander Hanschke

        Right now it’s not a module itself but tightly integrated with an application I’m working on. I’m just interested in letting users upload an image, which is immediately stored in a S3 bucket. I keep a public URL to the resource, so I can embed the image in my view.

        You can see the approach in this Gist: https://gist.github.com/2897560
        The client is created in the Global object upon the start of the application.

  • Ashok

    Hi James,
    I am using your S3 module and works great . My app is deployed to heroku. The issue i am facing is that when i want work locally, it still tries to connect to the S3 and sometimes I dont have an internet connection when I am traveling. I can avoid uploading when working locally but the issue is that this module is invoked on app startup and the entire applciation fails.  Is there a workaround ? 

    • http://www.jamesward.com James Ward

      You would need some kind of mock / local S3 library.  I’ve heard that one exists for Ruby but I’m not sure about Java.

      • ManishS

        Is it possible to just enable S3 module only for production. In the development mode, it should act as normal Play Blob and not S3Blob.

        • http://www.jamesward.com James Ward

          That is probably possible but not trivial.

  • T Lasseter

    Thought this might be a PlayFramework issue, but can’t find any StackOverflow comments on it.

    In trying to setup plays3upload, I’m getting the following link problem:

    java.lang.UnsatisfiedLinkError: C:UsersTom LasseterAppDataLocalTempjline_0_11_3.dll: Can’t load IA 32-bit .dll on a AMD 64-bit platform

    I’m running on a Windows 7 64-bit system (and need to for my application).  Are there a 64-bit dlls that I should use or alternatively how do I get around this problem?  

    • http://www.jamesward.com James Ward

      That is really odd.  I’m not sure what would cause that.  Sorry.

  • grud

    Thank you for this module. I use it in several play apps.

    Recently, I started using a Play 1.x build from play-master because I need some jpa/hibernate stuff that is coming in 1.3. The Play 1.3 build will use hibernate-3.6 …Or at least that appears to be the game plan.

    Unfortunately, hibernate-3.6 BREAKS S3Blob because, as the errors says, “The type S3Blob must implement the inherited abstract method Usertype.nullSafeGet(ResultSet, String[], SessionImplementor, Object)”

    I’ve tried fiddling around with some different ways of fixing this, but I haven’t been successful. I’m not very proficient with java. Any ideas about how to add nullSafeGet in such a way that hibernate 3.6 will accept it?

    • http://www.jamesward.com James Ward

      This module isn’t compatible with the latest Play 1.x stuff. Sorry. Hopefully someone will contribute a fix for this.

  • pacoalface

    Hello!! this project works properly on my local computer, but when I deployed the app in Heroku, Heroku shows a connection error with S3, the error is “Connection not Obtained From this manager” I have introduced all my credentials of s3. Please Help me. thank you very much!!

    • http://www.jamesward.com James Ward

      Perhaps the config vars aren’t set correctly on Heroku? You an check them with “heroku config”.

      • pacoalface

        Thanks for your reply. The error was caused by the aws sdk version. I solved this problem adding the next lines in dependencies.yml:

        require:
        – play 1.2.5
        – play -> secure
        – com.amazonaws -> aws-java-sdk 1.3.33
        – org.apache.httpcomponents -> httpcore 4.2.3
        – org.apache.httpcomponents -> httpclient 4.2.3
        – play -> s3blobs 0.2

        Now I have another problem, again in local all works fine, but in heroku I can upload images to S3 but when I try get this photo I get the next error in renderBinary()

        Internal Server Error (500) for request GET /Fotos/nick/pacoalface
        Execution exception (In /app/controllers/Fotos.java around line 107)
        play.exceptions.JavaExecutionException: com.amazonaws.services.s3.model.S3Object.getObjectContent()Ljava/io/InputStream;
        at play.mvc.ActionInvoker.invoke(ActionInvoker.java:237)
        at Invocation.HTTP Request(Play!)
        Caused by: java.lang.NoSuchMethodError: com.amazonaws.services.s3.model.S3Object.getObjectContent()Ljava/io/InputStream;
        at play.modules.s3blobs.S3Blob.get(S3Blob.java:39)
        at controllers.Fotos.descargar(Fotos.java:107)
        NoSuchMethodError occured : com.amazonaws.services.s3.model.S3Object.getObjectContent()Ljava/io/InputStream;
        at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:557)
        at play.mvc.ActionInvoker.invoke(ActionInvoker.java:508)
        at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:484)

        Thank you!!

        • http://www.jamesward.com James Ward

          Looks like a version mismatch with the AWS library. Maybe try 1.2.15?



  • View James Ward's profile on LinkedIn