Elastic Computing - a brief history

The basis of my cloud computing analogy is that computing capacity like the development of the early electrical grid is becoming the fundamental basis for our information based society. Therefore we need a universal system for the interchange of computing capacity like we have for power. Well it appears that no matter how much I complain, the only way to truly create an analogy these days is to write an old fashion book (on paper no less).

Being I am no fan of paper, or books for that matter. (Sorry Amazon) I thought I'd take a brief moment to discuss how I came up with the idea for our Elastic Computing Platform as well as well as the electrical analogy which I've been telling people for years. To do this, I need to take you back to how I first conceived the idea of "elastic computing" almost 5 years ago.

At the time I had just started Enomaly, and like all good bootstrapped entrepreneurs I had absolutely no idea what I wanted to do for with our business. Previously as a freelance open source developer I knew there was a strong demand for implementing open source technologies within an enterprise context. But in fairness that wasn't where my passion was. My real passion was in the development of new and exciting forms of emerging internet technology. So on a rainy drive one evening back from visiting a business partner in Montreal, myself and Lars Forsberg (my other business partner) discussed various potential ideas. One of the earliest ideas we discussed was the creation of a shared computing exchange where compute capacity could be exchanged similar to that of a commodities exchange. This platform would provide the ability for IT departments to tap into virtual computing resources beyond the confines of their existing data center on a utility basis. We envisioned telecoms & hosting providers signing up the service. Problem was there was no demand for such a service at the time, and worse yet there was no software to enable such an exchange of compute capacity.

We then took it upon ourselves to create a website around the idea, which was initially called "distributed Potential" & "distributed exchange". We built the websites, and to our surprise we had zero interest in such a service.

Undaunted we then spotted a actual customer need based on the open source projects we had been deploying at the time. For the most part these projects involved the development of open source content management systems and portal systems for large "fortune 500" type organizations. These were usually built in linux, apache, mysql, php and python (LAMPP) . One the major challenges in those early LAMPP deployments was in there inability to easily scale. The majority of the php system at that time (circa 2003) were built for low volume / lower usage sites. We then took the tools we had be developing as part of our customer deployments and released it as an open source project called Enomalism in late 2005.

Back then virtualization was just starting to catch on and we immediately saw an opportunity to use the new Xen hyper-visor as a mechanism to adjust the application environment on the fly based on the "somewhat" real time demands placed on the overall environment. Typically this meant adjusting RAM, storage, networking and creating replicated LVM snap shots wherever possible. We also utilized openLDAP & ssh based authentication providing for a method of utility and metered access to the virtualized resource pool (now called cloud) . We also saw an opportunity in using a URI based webservice instead of the more common SOAP approach of the time. This approach later became known as RESTful web services. We described this method for the automatic scaling of an application by tying an application server (apache) directly to the hypervisor as "elastic computing", well before anyone else that I am aware of. This more recently has become known as "cloud computing" or "infrastructure as a service".

Although our first attempts at "elastic computing" were for the most part a financial disappointment they did act as an opportunity generator, opening doors to projects and companies we may never had access to otherwise. Among those was in July of 2006 when Amazon came knocking. At the time they had been working on a top secret project described as a "grid utility". For the first time our idea for tapping into resources beyond the confines of your data center were starting to take shape. Over the next few months we took this opportunity to learn as much as we could about the benefits as well as the hurdles to creating this type of large scale compute utility. When Amazon EC2 finally launched into a private beta, (one we could publicly speak about) we were amazed at the amount of interest our up until then unknown products suddenly received. Among the opportunities that presented was the chance to work with Intel on several next generation virtualization and media projects (Intel remains our biggest customer). We had found our product and it was in the cloud.