Tuesday, December 2, 2014

Uploading Large Files to S3

I've downloaded and tried out quite a few AWS tools since I started using Amazon's web services in 2007. Originally, S3 had a 5 GB file size limitation which was increased to 5 TB four years ago. Unfortunately, all the S3 tools I've downloaded are limited to 5 GB. The challenge appears to be that any object larger than 5 GB requires the use of the multipart upload API and none of my tools can handle that.

Today, I needed to upload a 7.35 GB file. I was almost ready to give up after trying a few of my desktop tools along with my plug-ins for Firefox and Chrome. Then I got the idea to simply log into AWS with Safari and use their web based management console (duh). That, along with my speedy fast home Internet connection, did the trick. It was such a simple solution that I almost overlooked it.

Update: One bit of funkiness is that the tools which can't handle files over 5 GB can't properly report an object's size, although they do seem to be able to successfully download the files.

No comments: