I am currently working on a new startup called ShootProof. ShootProof utilizes many of Amazon web services. We recently have been hearing sporadic feedback from our beta testers that sometimes their uploads are slower than they think they should be. Currently the way we are accepting uploads is that we send each file up via XMLHttpRequest to our EC2 instances, doing some quick inspection of the file and then store it in an upload bucket. A few moments later a re-sizer batch job comes along and does resizing/watermarking/other stuff to the photo and moves it into place.

After we started to investigate why some beta testers were sometimes getting slower than ideal upload speed we decided to test out the ability to do out uploads directly to S3. Amazon S3 support HTTP POST uploads which is great as it takes us out of the middle of all of that traffic. Essentially what this means is that users of ShootProof should never be limited upload-wise by our EC2 instance. Also we will not need to constantly spool up and down EC2 instances to handle load spikes. After each upload is completely sent to S3 we will fire off a small notification call that will let us know we have a new photo we need to take care of. Upload traffic to our EC2 instances will drop by at least 99%. Also to be sure that we never miss a new photo that is placed into our S3 upload bucket we will monitor the contents up the upload bucket to ensure that they match what we are expecting. All photos that are uploaded into S3 by the user are marked to have an ACL of private so that they are essentially being put into a dropbox.

Below is a table that shows the tests that we did to come to our conclusion to post directly to S3. The file that was used for this test is a 13.1MB JPEG. All uploads were done using a internet connection that is a full 10Mbit up.

XMLHttpRequest Post (EC2 -> S3) HTTPS S3 Post HTTP S3 Post
13.2 20.6 9.5
14 19.1 9.6
13 cialis pas cher en pharmacie.7 22.7 10.3
14.3 24.9 9.3
13.7 15.3 9.4
13.9 18.4 9.2
24.1 24.2 9.7
13.7 17.5 9.4
13.8 17.3 9.9
15.2 17.4 9.4
14.96 sec 20.04 sec 9.57 sec Average

3 responses


Do you want to comment?

Comments RSS and TrackBack Identifier URI ?

Hi Robert. In your tests were you still able to use the XMLHttpRequest to transfer directly to S3? Cheers.

November 26, 2010 2:21 am

Hi Robert,
Thanks for the great speed test data!
Love ShootProof superb site and excellent idea.
What scripting language did you use to code? (if you don’t mind me asking)
I’m working on a project that requires uploading json files to S3 is there a way to do this directly from the client/browser without having to post through my EC2 instance?
Thanks
:-)

July 1, 2011 11:28 pm

Nelson,

You can see an example that Amazon has about POSTing directly you a bucket here. http://s3.amazonaws.com/doc/s3-example-code/post/post_sample.html

Robert

July 6, 2011 8:55 pm

Comment now!
















Trackbacks