Tracking EC2 instance internal IPs in a central location … SimpleDB

So us code poets over at ShootProof have been hard at work on doing an upgrade to our server architecture and one of the issues that we ran across was how to always know about which app servers are active and should be considered as part of the memcached pool. The goal was to have it totally automated such that we can bring machines up and down and never have to think about updating a list of IPs. The solution we came up with is to have a cron running on all of the app servers that updates a record in SimpleDB. This record cron runs each minute and gathers some information. Mainly it is just updating a timestamp in its SimpleDB record saying that it is alive. Near the end of the script it runs a query against SimpleDB to get a list of all of the active servers. It takes that list, builds an array of IP addresses and puts that in the local memcache instance. The website application then will be able to do a lookup to get the list of IPs to build the cluster at run time. Below is a copy of the script, feel free to pick it apart in the comments.

require_once('lib/sdb.php');
 
define('AWS_ACCESS_KEY_ID', 'XXXXXXXXXXXXXXX');
define('AWS_SECRET_ACCESS_KEY', 'YYYYYYYYYYYYYYY');
 
// machine specific information
$amiId = file_get_contents('http://169.254.169.254/latest/meta-data/ami-id');
$machineData = array(
    'ami_id' => array('value' => $amiId),
    'ip' => array('value' => file_get_contents('http://169.254.169.254/latest/meta-data/local-ipv4')),
    'hostname' => array('value' => file_get_contents('http://169.254.169.254/latest/meta-data/local-hostname')),
    'availability_zone' => array('value' => file_get_contents('http://169.254.169.254/latest/meta-data/placement/availability-zone')),
    'last_updated' => array('value' => time())
);
 
 
$sdb = new SimpleDB(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY);
$domain = 'simple_db_domain_name';
 
$expectedMachineData = null;
 
$objectInfo = $sdb->select($domain, "select * from `" . $domain . "` where `ami_id` = '" . $amiId . "'");
 
if (count($objectInfo)) {
    $attributes = $objectInfo[0]['Attributes'];
 
    $tmpMachineData = array();
 
    foreach ($machineData as $key => $data) {
        $data['replace'] = 'true';
        $tmpMachineData[$key] = $data;
    }   
 
    $machineData = $tmpMachineData;
 
    $expectedMachineData = array(
        'last_updated' => array(
            'value' => $attributes['last_updated']
        )   
    );  
}
 
// put the object into simple db
$sdb->putAttributes($domain, $amiId, $machineData, $expectedMachineData);
 
// find any old records to clean up
$recordsToDelete = $sdb->select($domain, "select * from `" . $domain . "` where `last_updated` <= '" . (time() - 300) . "'");
 
foreach ($recordsToDelete as $toDelete) {
    // if this is the same machine as the one this script is running on let's skip the
    // delete and just run the proceeding update
    if ($toDelete['Name'] == $amiId) {
        continue;
    }   
 
    // let's delete the object
    $sdb->deleteAttributes($domain, $toDelete['Name']);
}
 
// find any active servers
$activeMachines = $sdb->select($domain, "select * from `" . $domain . "` where `last_updated` > '" . (time() - 300) . "'");
 
$ips = array();
 
// make the list of all ips
foreach ($activeMachines as $activeMachine) {
    if (trim($activeMachine['Attributes']['ip']) == '') {
        continue;
    }   
 
    $ips[] = trim($activeMachine['Attributes']['ip']);
}
 
$memcache = new Memcache();
$memcache->addServer('127.0.0.1', '11212');
$memcache->set('cluster-ip-list', $ips, 0);

It is also a good idea to make a separate user in AWS for these scripts so that you can get a new access key and secret that can be locked down to just that domain in SimpleDB with only the needed permissions.

AWS, EC2, PHP, ShootProof, SimpleDB

July 6th, 2011

1 response

One response


Do you want to comment?

Comments RSS and TrackBack Identifier URI ?

Alan Jay

Very interesting idea. Thanks.

Being someone who loves to find these ideas but then looks at the code elements and realises that for someone who is a bit of a novice that example needs a bit more work to actually run it on a server.

As someone who just thought SimpleDB would be a good way to do this but knows nothing about how its frustrating to get 90% of the answer but thanks (really) now to just work out the bits that I should probably already know.

Regards
Alan

December 28, 2011 7:09 am

Comment now!
















Trackbacks