CloudMess

Sören Bleikertz has been poking around EC2 instances and found some nice ways of seeing what’s under the hood. Check it out at his blog.

and I missed a lot.

First: a confession. I’m a sporadic blogger at best, so you won’t see me posting early and often here.

Meat: I missed manifestogate. I was following it via twitter (I’m @keithhudgins) that I caught from John Willis, and picked up Reuven Cohen who is, unbeknownst to me, one of the net-centric, non-corporate community organizers in the cloud world. This fiasco is what happens when corporate interests get involved in community efforts and find those efforts contrary to their goals. The Cloud Community Manifesto has some good goals behind it, but I’d rather see the businesses involved put some code and API’s where their wallets are. I’ll pontificate more about that in another post.

Side: Ilya Grigorik had some thoughts about a nice analogy for cloud and new-style virtual resource platform architecture: the assembly line model. Really good stuff, you should take a look.

Side: Google’s announced Java support for AppEngine. Kinda cool, this will get them some traction from the enterprise crowd. This also means, if you know anything about the JRuby stack (I don’t), you can run Rails apps at Google.

Dessert: I just found Elastic Server by CohesiveFT. I’m impressed, we’ll be exploring this more in the future.

Got this from Infoworld who got it from Microsoft at the Mix09 conference that Microsoft’s supporting PHP on Azure. Whoa. Supported open-source environment on Azure?!? They’re talking about their FastCGI environment running other stacks, too. Ruby was mentioned.

My mind is officially blown.

I just came across this blog post from Tim Bray, which gives some good insider-perspective on what Sun’s got building for a cloud offering. I’m intrigued:

• It’s not a hosted-application-cloud, it’s a real, honest-to-goodness IT virtual datacenter cloud a-la Amazon EC2.
• They’re developing an open api to control the thing. More on that later.
• The API is so open, you can join the project.

He’s also mentioned there’s a storage component, a computing component, powered by the Q-Layer technology that Sun acquired in January. Here’s a great YouTube clip of an interview with one of the Q-Layer principals. This is cool for the network admins in the crowd: a drag-and-drop browser based interface that allows you to build your virtual infrastructure graphically, similar to 3Tera.

What’s most interesting here is that, according to Tim, Sun’s REALLY getting the point here: open designs, open APIs. Creative Commons license on the API. This allows other virtual infrastructure providers to use the API for portability, so that you can build a cotrtol interface to manage multiple cloud infrastructures. The point, according to Tim, is “Zero Barrier to Exit.” No one wants vendor lock-in as a customer. Amazon has been somewhat aggressive in protecting their API IP, in the one case that someone has white-boxed it: Eucalyptus. With a common API, the portability barriers diminish, so that you’ll find most cloud-based ‘mission critical’ infrastructures spanning different offerings. Vendor lock-in means only one company gets that slice of the whole pie, where open barriers mean that one customer will likely pick two or more providers to minimize points of failure. That’s a truly positive development that will help the industry as a whole. I just signed up, and I’m looking forward to seeing how I can contribute.

Sometimes you need a system backup. Other times, you need to launch your box ten times. Maybe you’re working on your new web cluster and need to build an image for your web server role. There’s tons of reasons, but if you’re using Amazon EC2, there will come a time when you need a custom server image. The simplest way to boot an instance off a public AMI, make the changes you need, and then roll up a disk image and throw it up to S3. Here’s the easiest way:

First, make sure you’ve got your /mnt partition built. Most of the public AMIs don’t launch with your big data drive formatted and available, and your disk image will be as big as the system itself. You may fill your drive image! Nothing hurts like a full drive. Nothing.

Second, make sure you have the EC2 AMI Tools installed. If not, go ahead and get them on your box. You can download them here.

I’m also assuming you have followed Amazon’s suggestions in setting up your shell profile so that you have your authentication to AWS in environment variables. Note: if you do this, don’t make your image public! Your credentials will then be in your system for everyone to see. Bad juju, so don’t do it. If you need to make your image public, then make sure you turn off your shell’s history, and type in the credentials on the command line.

Okay, so here we go:

ec2-bundle-vol -c $EC2_CERT -k $EC2_PRIVATE_KEY -u $AWS_ACCOUNT_ID -s 10240 -d /mnt
ec2-upload-bundle -b yourbucket/yourimagename_`date +%Y-%m-%d_%H:%M:%S` -m /mnt/image.manifest.xml -a $AMAZON_ACCESS_KEY_ID -s $AMAZON_SECRET_ACCESS_KEY

We’ll need a little explanation here. I’ll go step-by-step:

ec2-bundle-image makes a series of files that, when pieced together, make a disk image. This overcomes the S3 5gig file limitation, and makes this thing easier to upload to S3. We tell it to make a 1Gig image (The biggest it allows) and throw the files in /mnt. If you’ve followed along, your /mnt partition should be big enough to hold all of this. There are other, advanced options that allow you to manage ramdisks, mount points, kernel images, and other things that you most likely won’t need. And if you do, read the docs. They’re pretty good.

ec2-upload-bundle takes the manifest from your image.manifest.xml file that was created by ec2-bundle-image, and shoves it up to your S3 bucket named yourbucket and in a file named yourimagename_datestamp. I recommend changing those bucket and image names to something meaningful to you, and feel free to remove the timestamp if you don’t want it. It will loop through all the files in the manifest and get it all up to S3.

S3, however, isn’t super reliable for these kinds of things. And ec2-upload-bundle is dumb enough to just crap out. All is not lost, however, as you’ll see which parts of your image that have been uploaded, and you’ll know the next one. Amazon was kind enough to add the --part flag to start uploading at any arbitrary part. So just run your command again, add –part 23 or whatever the next part is, and go from there.

Once your image is up on S3, we need to tell EC2 it’s there so you can use it. The tool to do that is built into the API tools package, which you’ll need to launch your images anyway. Once you’ve got your environment set up right, you can just type:

ec2-register yourbucket/yourimagename/image.manifest.xml

Of course, you’ll need to change the names to the same as ran in the upload bundle command, but AWS will come back with an image ID that you then can use to launch your new private image.

I knew I was missing something when I was listing out some hosting providers for developer-focused cloud app hosting, so here goes:

Morph AppSpace is a paid hosting service with some cloudlike features - you can pay by the ‘cube’ or hosting unit - kinda like an Amazon EC2 Compute Unit - and it’s free to get started, which is nice for startups. They support Ruby, Rails, PHP, and Java (although I don’t know what kind of java hosting they provide, I’d assume it’s Tomcat-centric) and through the Java support, Grails if you’re into that kind of thing.

The cloudlike features are the ability to scale by adding new ‘cubes’ and the built-in monitoring that allow you make informed decisions on how much more money to spend on your scaling. It doesn’t dynamically scale, but it gives you the ability to add more resources to your app as you need them.

Like Google AppEngine (although, again, on Google’s side this is changing), you can’t run scheduled tasks, so you’ll have to work around this if you’re planning on hosting with Morph.

So there you go, one more tool for your belt.

I’m running off to the Atlanta Cloud Meetup in a few minutes, so I don’t have a lot of time to post, but here’s a few links that I’ve come across on some cloud updates:

• Google has updated the AppEngine SDK. I’m not a big AppEngine programmer, so hopefully someone will chime in on what’s cool and fresh on this one.
• Amazon just opened up Flexible Payment Services and are offering a free getting-started promo. There’s getting to be some nice service-based payment systems these days.
• Christopher Hoff’s put together a far better picture of cloud-type services than I’ve been doing with words. Take a look.

There are two types of cloud infrastructure offerings that are important for small business and startups: developer-centric hosting services, which take care of a lot of the day-to-day system tasks for you, or the infrastructure-focused options, which give you flexibility at the cost of having to do a lot of custom configuration. There are also service-tier systems that support the more flexible options, but I’m going to focus on an overview of the major cloud-based easy-to-get-into services that I know of.

All of these options require you to build your app with the platform in mind. Some knowledge of the limitations and capabilities of the systems will go a long way in smoothing the deployment of your new app. Building to these environments requires a certain amount of trust, however, and porting your app to a different platform once you’re up and running may be a chore if you don’t take steps early on.

Google AppEngine

Google has App Engine, a python-centric platform with a rich API that allows developers to build fairly robust applications quickly, deploy them, and just let them run. There’s no system administration involved whatsoever, which allows very small teams to build robust tools quickly without worrying about scaling, configuring servers, single points of failure or any of that rigamarole.

What you don’t get is some of the standard tools that most web developers take for granted: instead of a SQL-compliant database, you get BigTable. It’s capable, but different. Scheduled tasks currently don’t work, either. You can build an api-based, authenticated action and trigger it remotely to work around that, but watch for fault tolerance if you go that route. The biggest drawback is that the offering is free, but capped. Meaning, if you consume your allotted resources for the month, you’re done until the calendar ticks by. Google hasn’t yet opened payments for AppEngine, but it looks like the payment system is just around the corner.

Mosso

Mosso’s CloudSites product is more flexible: you get your choice of Windows or Linux, and you deploy your app or web site onto their servers. You get a full hosting environment, witn no shell access - all control is done through their web panel. It’s a big benefit if you’re not command-line centric. On the Linux side, you get PHP, Perl, Python, MySQL; Windows users can choose from different versions of the .NET framework, and use IIS and SQL Server. So you can build your application using your favorite framework that fits within those parameters, just remember that you have very little system control. If you’re using CPAN modules, you’ll have to use what they have available or build them into your app directly. You can’t install anything else. However, they offer 500GB of bandwidth out of the gate, with a slightly expensive $.25/GB once you go over. At $100, it’s not a bad option. Check their Does It Fit? page to see if what you’re building fits in their parameters.

EngineYard Solo

EngineYard is a fairly new player in the cloud game. They got started in 2006 as a Ruby on Rails hosting company, and had some really bad hiccups geting started (sorry guys, but I was a customer then and remember the whole cluster taking over an hour to reboot). They are, however, really damn smart, and determined to live on the bleeding edge. They use a custom-build, highly tuned Gentoo Linux server image that’s optimized to host Rails apps, and they do it very well. They’re driving a lot of thought and development in the systems administration world.

Recently, they just opened up Solo, in partnership with Amazon. They’re hosting and provisioning server instances using their custom stack on EC2, and make it pretty easy to get a Rails app off the ground. They charge a premium: $125/month, as opposed to a raw EC2 box at $70 or so, but what you get is a prebuilt box with a lot of good, smart defaults in the setup that’s built to reliably host a Rails app. If you’re not a sysadmin, or can’t affor the time or cash to get that work done for you, this looks to be a good route to go.

“What is The Cloud?”

Mighty big question, let me tell you. Cloud computing is the biggest buzzword to hit the IT industry since Local-Area-Networking. I’m serious. And it’s got the potential to be as big of a deal. A cloud-managed infrastructure is what every major IT installation will be based on by 2015. There will be legacy boxes floating around, but when  you can suck up your legacy box into a cloud infrastructure, liberating it from the iron it’s running on, why wouldn’t you?

Let’s get back on topic: what is The Cloud? And why is everyone calling it “The Cloud?” As in, “Let’s run that in The Cloud.” Or, “We’re hosting it in The Cloud.” As if there’s only one. Seriously, there’s more than one. And that’s a good thing. Every pundit and talking head in the tech industry is talking about it, but I honestly think that maybe 30% of them know what they’re talking about - and even they can’t agree.

Here’s the simple, real-deal definition:

The Cloud (n): A marketing buzzword that is essentially meaningless, tagging some service or offering to a nebulous idea of Resource-Based Computing.

Remember old-school network diagrams? When you connect your network to an outside source, the traditional symbol is a cloud. Meaning, ’some network or bunch of stuff out there, but we don’t know and don’t care for the scope of this diagram.’ So you wave your hands and draw a cloud. More recently, that cloud on your network diagram means The Internet. This is where the cloud term came from, folks. Some cloudy thing beyond the scope of the document. That’s what marketers are using when they say “The Cloud.”

That Resource-Based Computing term I just used is closer to what people really mean when they talk about The Cloud. Every major IT broker and their brother offer their own cloud-like solution. Microsoft is getting closer to Azure. Amazon has EC2 (which I’ll be blogging about extensively). Sun is probably going to be bought by someone, but they’ve got the Grid Engine. Google has AppEngine. And they’re all different. They all work differently. So what draws them together?

Resource-Based Computing. It’s the idea that computing is a resource that you can bill for, rather than buy outright. It’s rooted in modern operating systems, with the ultimate hardware abstraction layer - the whole damn computer. You build your app to fit the resource you want to tap, depending on your level of skill or commitment to the nuts and bolts of the machine. The more abstract, the less control you have over your environment, but the less you have to manage day-to-day. Like everything else, it’s a tradeoff.

Over the next week, I’m going to look deeper at the different cloud computing options, and where they fit on a strategic level. As we go, and as I experiment, I’ll post technical nitty-gritty details where I can.