News of last night’s problems with the Virginia Data Centers power system taking down sites like Netflix and Pinterestshows that sometimes not programming for fail over

Image via CrunchBase

or data center failure is a pretty foolish thing to do. Especially with costs somewhat reasonable per gigabyte in Amazon’s S3 system.

Anyone who does not program for fail over in any system takes foolish risks with their computing systems and their companies.

So synching S3 buckets is fairly easy, and there is no reason not to do it, you can cut your own custom script that will determine all the contents in an S3 bucket and make sure that even though the data centers are somewhat isolated ensure the contents are the same in each S3 bucket. You can use a freeware/minimal cost software like S3CMD from S3 tools to help you out if you need a baseline system command set to do this. Or you can manually do an object check between S3 buckets using an EC2 instance as a master controller for the S3 buckets. The EC2 instances should be able to talk to each other using the public interface without having to get an IP lease from AWS to do so.

You should also be load balancing between regions, doing a simple health check and using cloud watch to ensure that once the parameters are met for an outage for the world to be notified that one data center is down. Load balancing is a necessary process if you want to program for fail over.

Elastic Load Balancing that comes as part of AWS is a good way to ensure that regions are healthy, and that an outage in one region will not influence or impact the good working conditions of the company that is using AWS. On the ELB page on Amazon they state:

Elastic Load Balancing can detect the health of Amazon EC2 instances. When it detects unhealthy load-balanced Amazon EC2 instances, it no longer routes traffic to those Amazon EC2 instances and spreads the load across the remaining healthy Amazon EC2 instances.

Using Elastic Load Balancing, you can distribute incoming traffic across your Amazon EC2 instances in a single Availability Zone or multiple Availability Zones. Elastic Load Balancing automatically scales its request handling capacity in response to incoming application traffic.

You have to use S3, a master EC2 in each availability region to perform the automation for each region, Elastic Load Balancing, and software like S3CMD to make all this work. You will probably spend the better part of an afternoon setting this up, but failure to take into account an outage in any region is simply poor cloud computing or normal computing practice. There will always be disasters regardless of how stable any cloud computing solution seems, and you must architect for failover for every system you design and develop regardless of the platform.

Netflix, Pinterest and others who were down because they were only in the Virginia data center need to program for failover, and if they lost business or customers because of it, then they need to go back and reevaluate their AWS dependencies and work out a decent failover process.

Related articles

• More Problems for Amazon EC2 Cloud(datacenterknowledge.com)

  ---

• Amazon EC2 outage lessons(agilesysadmin.net)

  ---

It is not often that you read a book that simply makes sense, and Tim Wu

Image via Wikipedia

strings together a series of events evocative of the old “connections” TV show. Tim paints a picture of the difficulties that companies have cannibalizing current product lines and products in favor of the next big thing. By using AT&T and other companies throughout his book, you can see how difficult it would be for a manager to back the unproven “next big thing” when the current product set is still making tons of money and is very profitable. The picture that he portrays shows that companies are naturally disinclined to support technologies, processes, and ideas that are disruptive to profitable products.

We can see this with Microsoft struggling with Cloud Computing, and the Music and Movie industries having such a difficult time moving to a viable digital model in the internet era. The music industry has railed against every change in technology since the player piano and sheet music saying that it would decimate sales, only to find a working economic model that would allow them to be successful selling sheet music and player piano roles. This book will leave anyone transfixed, and has information for people who are familiar with disruptive technologies as a product line as well as general people who have an interest in how disruptive ideas eventually work their way into the mainstream, even it takes 40 some odd years as it did with the voice message machine that we take for granted today.

This book is truly 5 of 5 stars, I sat down and read it in one sitting unable to put it down. Well worth reading for all levels of interest from managers to employees, nonprofessionals to experts there is something here for everyone. Tim writes an eloquent book about technology, corporate and business model disruption, and why companies have a hard time working with disruptive ideas that might not be the next big thing. Moreover Tim points out why a disruptive idea might be more suited to smaller companies who can afford high risks, rather than a tried and true blue company that has different ideas of business lines and profit models.

Related articles

• Embrace Disruption (virtualnotes.wordpress.com)
• The App Internet – Disruptive Technology For Google Search (staygolinks.com)

Image via Wikipedia

In case you have been wondering what has been keeping me busy, the InfoSec Institute out of Chicago has requested that I write down everything I know about Cloud Computing, and this is turning into an interesting project.

As I am nearing my deadline, and finishing up the book on Advanced Cloud Computing for ISI, I am realizing just how little people really know about setting up and maintaining a cloud computing infrastructure. While people get ideas of virtualization, when you look at virtualized NIC cards, or Virtualized Memory Caches, a lot of folks have no idea what to do with them. Amazon provides some amazingly cool stuff to work with, but dealing with atomic components like GPU’s and CPU’s for a render farm and what works best is a “best guess” for people in how to use them.

The research part of this has been very interesting, because there is such a lack of data on some of the Amazon Cloud Computing components. Amazon of course has an excellent set of white papers on the subject, but little if any real examples in the real world how someone used cloud computing to do something very cool using various components of Amazon’s cloud computing resources.

Looking at things like Amazon’s High Performance computing stack (which would make an awesome render farm for a video game or video company), I would like to see more real world examples of how it has been used, and used to save money or time. Using atomic components for EBS (Elastic Block Storage) like putting an additional NIC card on the EBS system. Or the dangers of using EBS rather than S3, EBS is meant to be temporary while S3 is meant to be more permanent.

Then there are the security implications to S3 that I think some folks forgot about or didn’t realize what they were working with so you could programmatically drunkards walk through someone’s S3 volume without them even knowing about what you were doing. Logging in S3 in some of the real world examples I have seen drew blank stares because they didn’t realize you could log the events on one S3 bucket to another S3 bucket.

That is why I am having so much fun with this book, there are a lot of very cool things you can do, but there are a lot of interesting ways of mucking things up so that you do not get what you were expecting. The state of cloud computing architecture is still fairly primitive even if we are years into the idea, tools are obscure or shoehorned into working in a cloud environment rather than being built from the ground up to be compatible with the cloud environment. And yes that statement alone should get vendors angry with me wanting to assert my statement. But selling a network based IDS/IPS system for the cloud is an interesting prospect given that a company does not have direct access to the network, and cannot really put an IDS/IPS inline to the network even when using a Virtual Private Cloud, the best place for an inline IDS/IPS is still going to be at the company at the entry point for the VPN, not in the Amazon web space.

In all there are a lot of cool things you can do, but there are also a lot of changes in how we think about what we are doing in the cloud. I hope you will be interested in the book when it comes out, because I think it is going to pretty much so rock, and raise interesting questions, and offer interesting solutions to a wide range of issues with cloud computing.

Related articles

• Why is cloud computing significant for college faculty? (speedofcreativity.org)