Amazon S3 is storage for the Internet. It is designed to make web-scale computing easier for developers.

Its a no frills service and does exactly what it promises — makes it easy for developers so that they can concentrate on features and leave the scaling to Amazon. If you are new to Amazon S3, heres a good starting guide for you.

S3 is like a sharp sword, you must know how to play with it lest you can hurt yourself. Thats exactly what happened to me. One of the many applications that we are developing on MySpace, Sketch Me, required us to store (and serve) huge amount of image data (user sketches). And due to the viral nature of the application, the load almost tripled every month. S3 was a clear choice. It saved us time, money and headache. Our current stats (with caching) are:

• Total files stored: 205GB
• Bandwidth per month: 2TB
• GET Requests per month: 112m

Clearly I would not like to waste time setting up image serving servers that can handle such load and I am more than happy to outsource it to Amazon S3.

You would be surprised before caching when our total images were just 5GB, the no of requests were 263m ($363.91) (almost double to what it is now with 205GB of images)

So if we take the total request to be directly proportional to number of images, with that rate the actual requests should be approx 4.5 billion or $15,000 :O

How did I tame the beast?

At first look the pricing of Amazon’s S3 services seems quite cheap. Wait until you get your first bill and you will see have cents add up to huge $$$.

Storage

• $0.150 per GB – first 50 TB / month of storage used

Data Transfer

• $0.100 per GB – all data transfer in
• $0.170 per GB – first 10 TB / month data transfer out

Requests

• $0.01 per 1,000 PUT, POST, or LIST requests
• $0.01 per 10,000 GET and all other requests

So after getting the first bill for a few hundred dollars, I sat down thinking how to bring that down. When I was digging the HTTP headers, I found out that by default S3 doesn’t have any cache request headers set. So even when the visitor has the requested the file from S3 before and has it in his browser cache, the browser will send a HTTP GET request to S3 just to verify if the file has changed. S3 returns a 304 Not Modified header if the file has not changed and file wont be downloaded. You may think, S3 saved me a few GB of bandwidth cost. But each of this requests cost you ($0.01 per 10,000 GET) which is generally the bulk of the S3 bill.

Since photos our users upload almost never change. Asking S3 every time if the file has changed on S3 is certainly not required. You can stop browser sending this extra request for the same user by setting appropriate Cache-Control headers or Expires headers for the files. We can set Cache-Control max-age=864000 which will tell browser to not request the same file until next 10 days (3600*24*10 sec)

Fortunately S3 allows us to do that, but there is no simple and easy way to do that. So I decided write a small script (see below) to achieve this.

After Domestication

After setting Cache headers, my bill got down drastically. The traffic doubled and number of images  almost tripled (5GB – 15GB) within a month while the number of requests was reduced 3 times. So that is 9 times reduction in cost. Ideally, your bandwidth cost should be more than your requests cost.

If you own a high traffic blog or website, you can also store your javascripts or css files on S3 by using far fetched expires headers and using versioning (changing file name when contents change) so that the browser knows when the file has changed.

For Example:

<link href="http://s3.amazonaws.com/lalit/style.css?v=3" ... />
after change in stylesheet, change your code to,
<link href="http://s3.amazonaws.com/lalit/style.css?v=4" ... />

Domestication

The popular Firefox extension S3Fox doesn’t allow us to do that. So I decided write a small script using the S3 PHP REST Library.

You can download the code here (zip 6k).

Update: Fixed a bug in code (mime type calculation was failing on some php configurations)

To use the script, you have to upload it to your server (running PHP). You need to edit the upload.php file to specify your AWS access key and Secret, your S3 bucket name.

/* One time settings. */
$awsAccessKey	= '---';	// your AWS Key
$awsSecretKey	= '---';	// your AWS Secret
$bucket_name	= '---';	// S3 Bucket name
$age		= 3600*24*10;	// Cache age 10 days	
 
/* File Data */
$s3_dir_name	= 'dir1/dir2/';	// Directory on s3 where you want to upload file
				// example http://s3.amazonawas.com/bucket_name/dir1/dir2/filename.ext
 
$upload_file	= 'filename.ext';// name of the file you want to upload.
				// keep it in the same dir as this file.

After you have saved the config info, every time you need to upload a file, you have to specify the file name and the dir name in upload.php and callit from the browser http://yoursite.com/s3/upload.php. (I know it sucks, I promise will make it better soon)

If you want to escape all this trouble, there is a paid tool Bucket Explorer which helps you to upload files with custom headers. I havn’t used it but it looks great and works on Win/Linux/Mac.

I hope your next S3 bill will come down

Demo

After a few hours of javascript hacking, this is what I came up with.

How it works

It will fetch random (Creative Commons Licensed) sideshows from slideshare.net using the API. The application uses the SlideShare.net API and Prototype/Scriptacluous to create some pretty visual effects. The SlideShare Karaoke Randomizer was used for the first time on 11th July at the Creative Commons Salon in San Francisco!

Here’s a great writeup for how to run your own PPT karaoke.
More info on SlideShare.net blog here and here

Download

The code has been released under the CC-GPL license at Google Code project hosting.

Now we can store and retrieve JavaScript Objects, Arrays, Boolean, String, Number values using cookies, just like storing Java Objects in session on the server side.

Works with Firefox 1.5, 2.0, IE 6.0 and Opera 9.10.

Example

(Execute this example)

jar = new CookieJar({
expires:3600,   // seconds
path: '/'
});
 
dog = {name: 'Jacky', breed: 'Alsatian', age:5};
 
jar.put('mydog', dog);
mydog = jar.get('mydog');
 
alert("My dog's name is " + mydog.name);
alert("He is " + mydog.age + " years old");
alert("He is an " + mydog.breed);

Download

You can view the source code here.
This code is released under CC Attribution-ShareAlike 2.5

This code is released under Apache Software License.

API

CookieJar(options)
Constructor. Takes in a object as options.

options = {
expires: '',  // time in seconds (defualt: 3600)
path: '',     // cookie path
domain: '',   // cookie domain
secure: ''    // secure ?
}

boolean put(string name, mixed value)
Puts a particular cookie in the cookie jar. The cookie is associated with the name. Returns false if cannot add cookie (Ex: max cookie size exceeded!). Returns true on success.

mixed get(string name)
Gets a particular cookie from the cookie jar. Returns null if not found.

boolean remove(string name)
Removes a particular cookie from the cookie jar. Returns true on success, false otherwise.

void empty()
Empties the Cookie Jar.

array getKeys()
Gets array of all the cookie names.

object getPack()
Gets all the cookies as a single JavaScript object (Package) with name value pairs.

Change Log

v 0.5 (26-Jan-09)

1. Changed the License to Apache Software License 2.0

v 0.4 (11-Aug-07)

1. Removed a extra comma in options (was breaking in IE and Opera). (Thanks Jason)
2. Removed the parameter name from the initialize function
3. Changed the way expires date was being calculated. (Thanks David)

v 0.3 (22-Jun-07)

1. Removed dependancy on json.js (http://www.json.org/json.js)
2. empty() function only deletes the cookies set by CookieJar. Leaves alone other cookies like session_id etc.

v 0.2 (12-Apr-07)

1. Released for public use.

Related

• Ajaxian review
• MooTools port
• jQuery port

How does it work?

This code works on the simple principle that each character appears differently in different fonts. So different fonts will take different width and height for the same string of characters of same font-size.

We try to create a string with a specified font-face. If the font-face is not available, it takes up the font-face of the parent element. We then compare the width of the string with the specified font-face and width of the string with the font-face of the parent element, if they are different, then the font exists, otherwise not.

The string which we will use to generate the widths can be anything. But I guess we use ‘m’ or ‘w’ because these two characters take up the maximum width. And we use a trailing ‘l’ so that the same width fonts-faces can get separated based on the width of l character.

I have tested it on Firefox 2, 3, 3.5 IE 6, IE 7, Opera 9, Opera 10.

^Update Now works on Opera 9.10. It required to execute the script after its completely loaded. I guess, Opera cannot calculate the offset width until all the parents are completely loaded.

^{Update (10-Jul-09)} Fixed the Tests failing on Firefox 3 issue. changed comparison font to serif from sans-serif, as in FF3.0 font-face of child element didn’t fallback to font-face of parent element if the font-face of child element is missing.

Download

Get the JavaScript code here.

Released under CC Attribution-ShareAlike 2.5

You can also see Lemmi’s version of the script as Prototype plug-in here.

Demo

You can test any Font here:

// Usage
var detective = new Detector();
alert(detective.test('font name'));

Fonts on your computer

This table below shows which fonts are present on your system. (I have listed some of the most common and some uncommon fonts.)

Johan Sundström has presented this data in a more interesting and cool layout here. Thanks Johan!

Download: Snake (122 kb, with source code)

Download: Fifteen (zip 13kb)

Screenshot

If you want to know more about this epic, you must see this excellent movie or this YouTube video.

Download: Ramayana (zip 422kb)

Screenshots