CloudMess

Got this from Infoworld who got it from Microsoft at the Mix09 conference that Microsoft’s supporting PHP on Azure. Whoa. Supported open-source environment on Azure?!? They’re talking about their FastCGI environment running other stacks, too. Ruby was mentioned.

My mind is officially blown.

I just came across this blog post from Tim Bray, which gives some good insider-perspective on what Sun’s got building for a cloud offering. I’m intrigued:

• It’s not a hosted-application-cloud, it’s a real, honest-to-goodness IT virtual datacenter cloud a-la Amazon EC2.
• They’re developing an open api to control the thing. More on that later.
• The API is so open, you can join the project.

He’s also mentioned there’s a storage component, a computing component, powered by the Q-Layer technology that Sun acquired in January. Here’s a great YouTube clip of an interview with one of the Q-Layer principals. This is cool for the network admins in the crowd: a drag-and-drop browser based interface that allows you to build your virtual infrastructure graphically, similar to 3Tera.

What’s most interesting here is that, according to Tim, Sun’s REALLY getting the point here: open designs, open APIs. Creative Commons license on the API. This allows other virtual infrastructure providers to use the API for portability, so that you can build a cotrtol interface to manage multiple cloud infrastructures. The point, according to Tim, is “Zero Barrier to Exit.” No one wants vendor lock-in as a customer. Amazon has been somewhat aggressive in protecting their API IP, in the one case that someone has white-boxed it: Eucalyptus. With a common API, the portability barriers diminish, so that you’ll find most cloud-based ‘mission critical’ infrastructures spanning different offerings. Vendor lock-in means only one company gets that slice of the whole pie, where open barriers mean that one customer will likely pick two or more providers to minimize points of failure. That’s a truly positive development that will help the industry as a whole. I just signed up, and I’m looking forward to seeing how I can contribute.

Sometimes you need a system backup. Other times, you need to launch your box ten times. Maybe you’re working on your new web cluster and need to build an image for your web server role. There’s tons of reasons, but if you’re using Amazon EC2, there will come a time when you need a custom server image. The simplest way to boot an instance off a public AMI, make the changes you need, and then roll up a disk image and throw it up to S3. Here’s the easiest way:

First, make sure you’ve got your /mnt partition built. Most of the public AMIs don’t launch with your big data drive formatted and available, and your disk image will be as big as the system itself. You may fill your drive image! Nothing hurts like a full drive. Nothing.

Second, make sure you have the EC2 AMI Tools installed. If not, go ahead and get them on your box. You can download them here.

I’m also assuming you have followed Amazon’s suggestions in setting up your shell profile so that you have your authentication to AWS in environment variables. Note: if you do this, don’t make your image public! Your credentials will then be in your system for everyone to see. Bad juju, so don’t do it. If you need to make your image public, then make sure you turn off your shell’s history, and type in the credentials on the command line.

Okay, so here we go:

ec2-bundle-vol -c $EC2_CERT -k $EC2_PRIVATE_KEY -u $AWS_ACCOUNT_ID -s 10240 -d /mnt
ec2-upload-bundle -b yourbucket/yourimagename_`date +%Y-%m-%d_%H:%M:%S` -m /mnt/image.manifest.xml -a $AMAZON_ACCESS_KEY_ID -s $AMAZON_SECRET_ACCESS_KEY

We’ll need a little explanation here. I’ll go step-by-step:

ec2-bundle-image makes a series of files that, when pieced together, make a disk image. This overcomes the S3 5gig file limitation, and makes this thing easier to upload to S3. We tell it to make a 1Gig image (The biggest it allows) and throw the files in /mnt. If you’ve followed along, your /mnt partition should be big enough to hold all of this. There are other, advanced options that allow you to manage ramdisks, mount points, kernel images, and other things that you most likely won’t need. And if you do, read the docs. They’re pretty good.

ec2-upload-bundle takes the manifest from your image.manifest.xml file that was created by ec2-bundle-image, and shoves it up to your S3 bucket named yourbucket and in a file named yourimagename_datestamp. I recommend changing those bucket and image names to something meaningful to you, and feel free to remove the timestamp if you don’t want it. It will loop through all the files in the manifest and get it all up to S3.

S3, however, isn’t super reliable for these kinds of things. And ec2-upload-bundle is dumb enough to just crap out. All is not lost, however, as you’ll see which parts of your image that have been uploaded, and you’ll know the next one. Amazon was kind enough to add the --part flag to start uploading at any arbitrary part. So just run your command again, add –part 23 or whatever the next part is, and go from there.

Once your image is up on S3, we need to tell EC2 it’s there so you can use it. The tool to do that is built into the API tools package, which you’ll need to launch your images anyway. Once you’ve got your environment set up right, you can just type:

ec2-register yourbucket/yourimagename/image.manifest.xml

Of course, you’ll need to change the names to the same as ran in the upload bundle command, but AWS will come back with an image ID that you then can use to launch your new private image.