Android file upload to Amazon S3 with progress bar

Programming for Android devices can be a lot of fun but every now and you're faced with a task which seems simple at first glance but gets you hitting a few walls before you finally find a satisfying solution. This time the requirement for me was to upload a file from an android device to a bucket in the Amazon Simple Storage Service (S3). The progress of the file upload

should be accurately visualized by a linear progress bar. Sounds simple right?

Well here's the path I followed. Luckily, this story has a happy ending.

Before I got started I needed to read up on how to upload a file to Amazon S3 in the first place. This is pretty well documented in the Amazon developer documentation and their getting started docs. You basically need to first sign up for Amazon S3, create a bucket and then perform an Http multipart POST. This POST should go to http://your-bucket-name.s3.amazonaws.com/. Inside your bucket, you are free to create subdirectories by including them in the so-called object key. For example, you can POST a file to images/car.jpg and the image becomes available at http://your-bucket-name.s3.amazonaws.com/images/car.jpg

So far so good. Actually, things are a little more complicated than just POSTing a file to a certain URL because of policy files and the like but we leave that out of this discussion for now. How do we perform a Http POST from an Android application? We could use the WebView, create an HTML page and POST a form, but then we would have no control over the file upload progress. Next idea: to use the built-in HttpClient in the package org.apache.commons.httpclient. We soon discover that this HttpClient does not support multipart file upload out of the box. A bit weird, uploading a file through a Http POST seems like a quite regular requirement to me but by default it's not included in the java fork presented to you by Google in the android libraries.

After a bit of searching a simple solution presents itself: to include a separate Apache library HttpMime which does contain the multipart file upload as described here. I put together some code to test it and all seemed well until I started receiving Http error codes from Amazon. As it turns out, the HttpClient does not specify the Content-Length header in the POST request. This is a hard requirement imposed by Amazon S3 as described here. So we hit a dead end.

HttpClient is really a convenience class, hiding the low level complexity of manually managing an HttpUrlConnection. So if HttpClient doesn't do the job for us, we will have to dig one step deeper and work directly with an HttpUrlConnection. It means we will have to step by step construct the multipart request with its boundaries and headers. It's a dirty job but certainly not impossible. A clean example of what this request should look like is readily available in the Amazon docs.

This all works like a charm; the file is uploaded to the Amazon bucket. But wait, we forgot one piece of the requirement: to display a progress bar. No problem because Android contains the cool ProgressBar class. We create an Activity, define a ProgressBar in the layout XML, subclass AsyncTask where we will perform the upload asynchronously and write a while loop where we send chunks of say 4096 bytes to HttpUrlConnection and after every chunk we publish the progress to the progress bar and that's it! Yes, but of course not quite. It turns out we've run into an issue 3164 of the pre-froyo Android platform. Thanks to this bug all content in the file upload is buffered and only gets sent to the server at the connection.flush() in one big chunk at the time. Of course this takes forever with my T-Mobile contract. The progress bar indicates that the file upload has almost finished (because it got updated after each 4096 byte chunk) but then the waiting starts. I experimented with setting connection.setChunkedStreamingMode to true but this is not accepted by Amazon S3 because in that case it violates the requirement we saw before to mandatorily specify the Content-Length up front.

Almost about to give up I got inspired by the movie Inception where criminals invade each other's dreams up to three levels deep. Amazing stuff. Time to sink one level deeper into the HttpUrlConnection by working directly onto a java.net.Socket. This proved to be the final solution. We open a Socket onto your-bucket-name.s3.amazonaws.com to port 80 and write the multipart POST directly into this socket connection. In this case we must manually pass the required Http headers which we could previously specify through the HttpUrlConnection object. It's now possible to send a chunk of 4096 bytes, update the progress bar and see the real progress. After the final chunk all data has really been sent to the server and the state of the progress bar correctly reflects the progress of the upload: Done!

Now for those of you interested in the details, here we go.

First the definition of the ProgressBar in one of your layout XMLs:

<ProgressBar
  android:id="@+id/progressBarUpload"
  android:layout_width="150dip"
  android:layout_height="15dip"
  android:layout_centerInParent="true"
  android:layout_marginBottom="15dip"
  android:layout_marginLeft="10dip"
  android:layout_marginRight="10dip"
  style="?android:attr/progressBarStyleHorizontal"/>

Note the style attribute, this is the way to explain to Android that you want a horizontal progress bar instead of a spinning image. Now let's inflate the ProgressBar inside our Activity:

progressBar = (ProgressBar) findViewById(R.id.progressBarUpload);
progressBar.setMax(100);
progressBar.setProgress(0);

Then, when we're ready to launch the upload task:

new PutOrderFilesTask(orderParams, getApplicationContext(), progressBar, uploadFilesHandler)
   .execute("your-bucket-name.s3.amazonaws.com");

which is a subclass of AsyncTask defined like this:

public class PutOrderFilesTask extends AsyncTask<String, Long, Integer> {

and we should override the method doInBackground:

@Override
protected Integer doInBackground(String... unused) {
   Map params = new HashMap();
   Uri uri = Uri.parse("the-uri-to-your-file");
   params.put("AWSAccessKeyId", "our-aws-access-key");
   params.put("Content-Type", "image/jpeg");
   params.put("policy", "some-policy-defined-by-yourself");
   params.put("Filename", "photo.jpg");
   params.put("key", "images/photo.jpg");
   params.put("acl", "private");
   params.put("signature", "some-signature-defined-by-yourself");
   params.put("success_action_status", "201");

   try {
      HttpRequest.postSocket("your-bucket-name.s3.amazonaws.com", params,
         context.getContentResolver().openInputStream(uri)
         fileSize, this, 10, 70, "photo.jpg", "image/jpeg");
   } catch (Exception e) {
      return -1;
   }
   return 1;
}

The HttpRequest class contains all the low level details of actually performing the upload:

   public class HttpRequest {
 private static final String boundary = "-----------------------******";
    private static final String newLine = "\r\n";
    private static final int maxBufferSize = 4096;

    private static final String header = 
     "POST / HTTP/1.1\n" +
  "Host: %s\n" +
  "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.10) Gecko/20071115 Firefox/2.0.0.10\n" +
  "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\n" +
  "Accept-Language: en-us,en;q=0.5\n" +
  "Accept-Encoding: gzip,deflate\n" +
  "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\n" +
  "Keep-Alive: 300\n" +
  "Connection: keep-alive\n" +
  "Content-Type: multipart/form-data; boundary=" + boundary + "\n" +
  "Content-Length: %s\n\n";

 public static void postSocket(String sUrl, Map params, InputStream stream, long streamLength, 
      PutOrderFilesTask task, int startProgress, int endProgress, String fileName, String contentType) {
      OutputStream writer = null;
      BufferedReader reader = null;
      Socket socket = null;
            try {
                int bytesAvailable;
                int bufferSize;
                int bytesRead;
                int totalProgress = endProgress - startProgress;

                task.myPublishProgress(new Long(startProgress));
                
                String openingPart = writeContent(params, fileName, contentType);
                String closingPart = newLine + "--" + boundary + "--" + newLine;
                long totalLength = openingPart.length() + closingPart.length() + streamLength;

                // strip off the leading http:// otherwise the Socket will not work
                String socketUrl = sUrl;
                if (socketUrl.startsWith("http://")) {
                 socketUrl = socketUrl.substring("http://".length());
                }
                
                socket = new Socket(socketUrl, 80);
                socket.setKeepAlive(true);
             writer = socket.getOutputStream();
             reader = new BufferedReader(new InputStreamReader(socket.getInputStream()));

             writer.write(String.format(header, socketUrl, Long.toString(totalLength)).getBytes());
                writer.write(openingPart.getBytes());

                bytesAvailable = stream.available();
                bufferSize = Math.min(bytesAvailable, maxBufferSize);
                byte[] buffer = new byte[bufferSize];
                bytesRead = stream.read(buffer, 0, bufferSize);
                int readSoFar = bytesRead;
                task.myPublishProgress(new Long(startProgress + Math.round(totalProgress * readSoFar / streamLength)));
                while (bytesRead > 0) {
                    writer.write(buffer, 0, bufferSize);
                    bytesAvailable = stream.available();
                    bufferSize = Math.min(bytesAvailable, maxBufferSize);
                    bytesRead = stream.read(buffer, 0, bufferSize);
                    readSoFar += bytesRead;
                    task.myPublishProgress(new Long(startProgress + Math.round(totalProgress * readSoFar / streamLength)));
                }
                stream.close();
                writer.write(closingPart.getBytes());
                Log.d(Cards.LOG_TAG, closingPart);
                writer.flush();
                
                // read the response
                String s = reader.readLine();
                // do something with response s
            } catch (Exception e) {
             throw new HttpRequestException(e);
            } finally {
             if (writer != null) { try { writer.close(); writer = null;} catch (Exception ignore) {}}
             if (reader != null) { try { reader.close(); reader = null;} catch (Exception ignore) {}}
             if (socket != null) { try {socket.close(); socket = null;} catch (Exception ignore) {}}
            }
            task.myPublishProgress(new Long(endProgress));
        }

 /**
     * Populate the multipart request parameters into one large stringbuffer which will later allow us to 
     * calculate the content-length header which is mandatotry when putting objects in an S3
     * bucket
     * 
     * @param params
     * @param fileName the name of the file to be uploaded
     * @param contentType the content type of the file to be uploaded
     * @return
     */
 private static String writeContent(Map params, String fileName, String contentType) {
  
  StringBuffer buf = new StringBuffer();

  Set keys = params.keySet();
        for (String key : keys) {
            String val = params.get(key);
            buf.append("--")
             .append(boundary)
             .append(newLine);
            buf.append("Content-Disposition: form-data; name=\"")
             .append(key)
             .append("\"")
             .append(newLine)
             .append(newLine)
             .append(val)
             .append(newLine);
        }

        buf.append("--")
         .append(boundary)
         .append(newLine);
        buf.append("Content-Disposition: form-data; name=\"file\"; filename=\"")
         .append(fileName)
         .append("\"")
         .append(newLine);
        buf.append("Content-Type: ")
         .append(contentType)
         .append(newLine)
         .append(newLine);
        
        return buf.toString();
 }
}