require_once('lib/sdb.php');
 
define('AWS_ACCESS_KEY_ID', 'XXXXXXXXXXXXXXX');
define('AWS_SECRET_ACCESS_KEY', 'YYYYYYYYYYYYYYY');
 
// machine specific information
$amiId = file_get_contents('http://169.254.169.254/latest/meta-data/ami-id');
$machineData = array(
    'ami_id' => array('value' => $amiId),
    'ip' => array('value' => file_get_contents('http://169.254.169.254/latest/meta-data/local-ipv4')),
    'hostname' => array('value' => file_get_contents('http://169.254.169.254/latest/meta-data/local-hostname')),
    'availability_zone' => array('value' => file_get_contents('http://169.254.169.254/latest/meta-data/placement/availability-zone')),
    'last_updated' => array('value' => time())
);
 
 
$sdb = new SimpleDB(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY);
$domain = 'simple_db_domain_name';
 
$expectedMachineData = null;
 
$objectInfo = $sdb->select($domain, "select * from `" . $domain . "` where `ami_id` = '" . $amiId . "'");
 
if (count($objectInfo)) {
    $attributes = $objectInfo[0]['Attributes'];
 
    $tmpMachineData = array();
 
    foreach ($machineData as $key => $data) {
        $data['replace'] = 'true';
        $tmpMachineData[$key] = $data;
    }   
 
    $machineData = $tmpMachineData;
 
    $expectedMachineData = array(
        'last_updated' => array(
            'value' => $attributes['last_updated']
        )   
    );  
}
 
// put the object into simple db
$sdb->putAttributes($domain, $amiId, $machineData, $expectedMachineData);
 
// find any old records to clean up
$recordsToDelete = $sdb->select($domain, "select * from `" . $domain . "` where `last_updated` <= '" . (time() - 300) . "'");
 
foreach ($recordsToDelete as $toDelete) {
    // if this is the same machine as the one this script is running on let's skip the
    // delete and just run the proceeding update
    if ($toDelete['Name'] == $amiId) {
        continue;
    }   
 
    // let's delete the object
    $sdb->deleteAttributes($domain, $toDelete['Name']);
}
 
// find any active servers
$activeMachines = $sdb->select($domain, "select * from `" . $domain . "` where `last_updated` > '" . (time() - 300) . "'");
 
$ips = array();
 
// make the list of all ips
foreach ($activeMachines as $activeMachine) {
    if (trim($activeMachine['Attributes']['ip']) == '') {
        continue;
    }   
 
    $ips[] = trim($activeMachine['Attributes']['ip']);
}
 
$memcache = new Memcache();
$memcache->addServer('127.0.0.1', '11212');
$memcache->set('cluster-ip-list', $ips, 0);

It is also a good idea to make a separate user in AWS for these scripts so that you can get a new access key and secret that can be locked down to just that domain in SimpleDB with only the needed permissions.

1. Ensure that your apache config includes:

NameVirtualHost *:443

2. Your vhosts:

]]> http://www.robertswarthout.com/2010/10/wildcard-ssl-one-ip-multiple-vhosts/feed/ 0 http://www.robertswarthout.com/2010/10/amazon-s3-upload-speeds/?utm_source=rss&utm_medium=rss&utm_campaign=amazon-s3-upload-speeds http://www.robertswarthout.com/2010/10/amazon-s3-upload-speeds/#comments Wed, 06 Oct 2010 13:42:33 +0000 Robert Swarthout http://www.robertswarthout.com/?p=120 I am currently working on a new startup called ShootProof. ShootProof utilizes many of Amazon web services. We recently have been hearing sporadic feedback from our beta testers that sometimes their uploads are slower than they think they should be. Currently the way we are accepting uploads is that we send each file up via XMLHttpRequest to our EC2 instances, doing some quick inspection of the file and then store it in an upload bucket. A few moments later a re-sizer batch job comes along and does resizing/watermarking/other stuff to the photo and moves it into place.

After we started to investigate why some beta testers were sometimes getting slower than ideal upload speed we decided to test out the ability to do out uploads directly to S3. Amazon S3 support HTTP POST uploads which is great as it takes us out of the middle of all of that traffic. Essentially what this means is that users of ShootProof should never be limited upload-wise by our EC2 instance. Also we will not need to constantly spool up and down EC2 instances to handle load spikes. After each upload is completely sent to S3 we will fire off a small notification call that will let us know we have a new photo we need to take care of. Upload traffic to our EC2 instances will drop by at least 99%. Also to be sure that we never miss a new photo that is placed into our S3 upload bucket we will monitor the contents up the upload bucket to ensure that they match what we are expecting. All photos that are uploaded into S3 by the user are marked to have an ACL of private so that they are essentially being put into a dropbox.

Below is a table that shows the tests that we did to come to our conclusion to post directly to S3. The file that was used for this test is a 13.1MB JPEG. All uploads were done using a internet connection that is a full 10Mbit up.

I typically will not write about anything non-technical but I will make an exception this time. Why-oh-why is it like pulling teeth when dealing with the iPhone App Developer Program support. I get the feeling that when I call and wait my turn in the phone queue, all I am getting to talk to is a receptionist who is going to “escalate the matter”. When asked how long it will take to get a response back, I am told that it will take a week to a week and a half. This is a VERY simple matter. All I want to know is how many downloads my app has had. Shouldn’t this automatically, programmatically for that matter, be turned on in the iTunesConnect portal by default when an app goes live? If that was the case, I could be writing a blog post about how awesome the developer program is, but no, I am stuck here writing in frustration.

Having dealt with Apple support for a Macbook Pro a couple of times, the iPhone Developer support seems like a bunch of amateurs who have no idea what’s going on. As an AAPL share holder, I am glad that the general public does not have to deal with the iPhone Developer support, as it would turn people away in droves.

If anyone with power to help in this situation is reading this post please feel free to contact me at rswarthout [at] gmail [dot] com.

You’re deep in the jungle when suddenly a silly little monkey starts laughing at you. Better show him who’s boss and slap that monkey! Similar to the classic arcade game “Whack A Mole” this game is both funny and easy to play. Exciting features test your speed and coordination and you try to slap every monkey you see. Extra points if you catch him while he is stealing bananas. Guaranteed to be more fun than a… barrel full of monkeys.

Features:

• Multi-touch game play
• Endless Levels
• 3 independent difficulty settings
• Sound Control
• Top Score Board

Here are a few screen shots of what this will look like:

Be sure to let me know what you think about this game.

Run #1:

11409 fetches, 20 max parallel, 45636 bytes, in 600.011 seconds
4 mean bytes/connection
19.0146 fetches/sec, 76.0586 bytes/sec
msecs/connect: 0.211857 mean, 39.668 max, 0.029 min
msecs/first-response: 1050.65 mean, 2662.53 max, 323.514 min
HTTP response codes: code 200 — 11409

Run #2:

11141 fetches, 20 max parallel, 44564 bytes, in 600.005 seconds
4 mean bytes/connection
18.5682 fetches/sec, 74.2727 bytes/sec
msecs/connect: 0.206869 mean, 30.293 max, 0 min
msecs/first-response: 1076.17 mean, 2317.13 max, 350.16 min
HTTP response codes: code 200 — 11141

Run #3:

11183 fetches, 20 max parallel, 44732 bytes, in 600 seconds
4 mean bytes/connection
18.6383 fetches/sec, 74.5533 bytes/sec
msecs/connect: 0.201182 mean, 24.691 max, 0.03 min
msecs/first-response: 1071.81 mean, 2448.18 max, 366.686 min
HTTP response codes: code 200 — 11183

Results Analysis

• Average requests per second: 18.7043
• Average requests completed: 11,244

These results can be compared to the previous post on this topic.

I used MySQL 5.0.45, PHP 5.2.4 for this benchmarking. You can download the same sample MySQL database that is used for this set of benchmarking by visiting http://dev.mysql.com/doc/#sampledb. In the list of example databases, choose the one named ‘sakila’.

For each connection type I will run three tests to see compensate for variations in tests. I will be using http_load to do the benchmarking of the connection file. To view each of the connection files, download the file bundle at the end of this post.

Below is the benchmark command with the options used for each run.

./http_load -verbose -parallel 20 -seconds 600 urls.txt

The urls.txt file contains the following:

http://127.0.0.1/mysql_connection_bench_{CONNECT_TYPE}.php

Between each benchmark run I will restart both Apache and MySQL.

Results for the standard MySQL functions in PHP using mysql_connect.

Run #1:

11024 fetches, 20 max parallel, 44096 bytes, in 600.007 seconds
4 mean bytes/connection
18.3731 fetches/sec, 73.4925 bytes/sec
msecs/connect: 0.22087 mean, 48.992 max, 0.03 min
msecs/first-response: 1087.74 mean, 2880.95 max, 355.012 min
HTTP response codes: code 200 — 11024

Run #2:

10895 fetches, 20 max parallel, 43580 bytes, in 600.004 seconds
4 mean bytes/connection
18.1582 fetches/sec, 72.6328 bytes/sec
msecs/connect: 0.219867 mean, 48.58 max, 0.031 min
msecs/first-response: 1100.1 mean, 2623.47 max, 369.341 min
HTTP response codes: code 200 — 10895

Run #3:

11134 fetches, 20 max parallel, 44536 bytes, in 600.003 seconds
4 mean bytes/connection
18.5566 fetches/sec, 74.2264 bytes/sec
msecs/connect: 0.217796 mean, 42.036 max, 0.03 min
msecs/first-response: 1076.59 mean, 3079.73 max, 270.734 min
HTTP response codes: code 200 — 11134

Results for the standard MySQL functions in PHP using mysql_pconnect.

Run #1:

10982 fetches, 20 max parallel, 43928 bytes, in 600.025 seconds
4 mean bytes/connection
18.3026 fetches/sec, 73.2103 bytes/sec
msecs/connect: 0.216663 mean, 69.795 max, 0 min
msecs/first-response: 1091.52 mean, 3166.82 max, 254.36 min
HTTP response codes: code 200 — 10982

Run #2:

11002 fetches, 20 max parallel, 44008 bytes, in 600.001 seconds
4 mean bytes/connection
18.3366 fetches/sec, 73.3465 bytes/sec
msecs/connect: 0.20555 mean, 47.176 max, 0.031 min
msecs/first-response: 1089.51 mean, 2953.82 max, 414.826 min
HTTP response codes: code 200 — 11002

Run #3:

11196 fetches, 20 max parallel, 44784 bytes, in 600 seconds
4 mean bytes/connection
18.66 fetches/sec, 74.6399 bytes/sec
msecs/connect: 0.2013 mean, 22.478 max, 0.03 min
msecs/first-response: 1070.5 mean, 2506.02 max, 448.161 min
HTTP response codes: code 200 — 11196

Results for non-persistent connections with PDO_MYSQL.

Run #1:

10995 fetches, 20 max parallel, 43980 bytes, in 600.008 seconds
4 mean bytes/connection
18.3248 fetches/sec, 73.299 bytes/sec
msecs/connect: 0.203731 mean, 17.309 max, 0.029 min
msecs/first-response: 1090.44 mean, 2852.34 max, 358.45 min
HTTP response codes: code 200 — 10995

Run #2:

10971 fetches, 20 max parallel, 43884 bytes, in 600.002 seconds
4 mean bytes/connection
18.285 fetches/sec, 73.1398 bytes/sec
msecs/connect: 0.21357 mean, 32.036 max, 0 min
msecs/first-response: 1092.36 mean, 3515.71 max, 456.688 min
HTTP response codes: code 200 — 10971

Run #3:

11161 fetches, 20 max parallel, 44644 bytes, in 600.001 seconds
4 mean bytes/connection
18.6016 fetches/sec, 74.4065 bytes/sec
msecs/connect: 0.20571 mean, 31.679 max, 0.031 min
msecs/first-response: 1073.96 mean, 2539.06 max, 351.236 min
HTTP response codes: code 200 — 11161

Results for persistent connections with PDO_MYSQL.

Run #1:

10347 fetches, 20 max parallel, 41388 bytes, in 600.006 seconds
4 mean bytes/connection
17.2448 fetches/sec, 68.9793 bytes/sec
msecs/connect: 0.253049 mean, 96.354 max, 0 min
msecs/first-response: 1158.52 mean, 2596.78 max, 282.647 min
HTTP response codes: code 200 — 10347

Run #2:

11106 fetches, 20 max parallel, 44424 bytes, in 600.008 seconds
4 mean bytes/connection
18.5098 fetches/sec, 74.0391 bytes/sec
msecs/connect: 0.226898 mean, 51.29 max, 0.03 min
msecs/first-response: 1079.29 mean, 2393.4 max, 311.85 min
HTTP response codes: code 200 — 11106

Run #3:

10978 fetches, 20 max parallel, 43912 bytes, in 600.029 seconds
4 mean bytes/connection
18.2958 fetches/sec, 73.1831 bytes/sec
msecs/connect: 0.210895 mean, 30.339 max, 0 min
msecs/first-response: 1091.98 mean, 3455.6 max, 373.751 min
HTTP response codes: code 200 — 10978

Results Analysis

• mysql_connect
  • Average requests per second: 18.3626
  • Average requests completed: 11,017
• mysql_pconnect
  • Average requests per second: 18.4330
  • Average requests completed: 11,060
• PDO_MySQL non-persistent
  • Average requests per second: 18.4038
  • Average requests completed: 11,042
• PDO_MySQL persistent
  • Average requests per second: 18.0168
  • Average requests completed: 10,810

Basically what the numbers above shows us is that in an isolated environment it makes no difference which connection type you are going to use. My past experience in live heavily trafficked environment says that using non-persistent connections is better than using persistent connections. Where this difference comes shows itself the most is when an apache thread will hang and still hold the persistent connection to mysql preventing other apache processes from connecting. By tweaking your apache and mysql settings you can achieve very good performance with non-persistent connections.

Source files for this benchmarking test.