Tuesday, July 29, 2008

Midori - Microsoft's Cloud OS

There have been several reports flying around today about microsofts new "secret" cloud OS project, code named Midori. This project appears to be an attempt to address the next logical step in operating systems design where a desktop is no longer centric to a users computing experience. Midori looks to be an Internet-centric OS, based on the idea of connected systems, that largely eliminates the dependencies between local applications and the hardware they run on that exist with a typical operating system today.

Some interesting aspects of Midori include; “The ability to distribute applications across nodes, Midori will introduce a higher-level application model that abstracts the details of physical machines and processors. The model will be consistent for both the distributed and local concurrency layers, and it is internally at Microsoft known as Asynchronous Promise Architecture.” (Sounds like virtualization to me)

There is also some innovative interface advancements planned at the presentation layer, Microsoft is making a clean break from the existing Windows GUI model, where applications must update their display on one and only one thread at a time, and the associated problems that affect OS stability and make it more difficult to write multithreaded applications. (Guess they're looking at Linux)

I should note that Microsoft officials have been trying to downplay the significance of Midori by constituting it as nothing but a research project. It seems to be closely related to another microsoft project code named Singularity. Singularity is a Microsoft Research project started in 2003 to build a highly-dependable operating system in which a microkernel, device drivers, and applications are all written in managed code.

HP, Intel, Yahoo partner in cloud computing

According to a press release issued today, Hewlett-Packard, Intel and Yahoo have partnered with three research groups to create an open source test bed for research in cloud-computing, the concept of using globally distributed computers as a platform for delivering services.

As many of you know we've been working with Intel for the last 18 months on several "cloud" projects and our reoccurring problem has been how do you test a cloud platform designed to manage upwards of 100,000 servers without access to a real world test bed. This is a good first step, but with less then 1000 servers it still doesn't adequately address the problem.

I wonder how others developing large scale cloud platforms are handling scalability testing?

Monday, July 28, 2008

Vmware, Welcome to the cloud

I'd like to personally welcome VMware to the cloud wars.

Today VMware announced its stand-alone ESXi hypervisor will be available at no cost. ESXi 3.5 update 2, available today, meets the criteria for mass cloud deployments: (1) ease of use and (2) maturity and stability now having been ‘battle tested’ for six months with customers. The leading server manufacturers have all embedded VMware ESXi, including Dell, Fujitsu-Siemens, HP, IBM, and NEC.

This is the first major step for VMware in the cloud computing scene. Historically VMware has focused on server consolidation and test / dev and has priced thier products accordingly. By making the underlying hyper visor free they effectivily enable cloud providers with the ability to utilize ESXi in their massive cloud server deployments while also helping keep their costs competitive with companies like Amazon who utilize Xen.

You can download ESXI from: http://www.vmware.com/products/esxi/

Sunday, July 27, 2008

Amazon's S3 Gossip Protocol

At 9:41am PDT on July 20th something strange started happening with Amazon's Simple Storage Service (S3). The service used by hundreds of thousands around the globe and millions more through end user web applications was no longer responding. Later it was determined that servers within Amazon S3 were having problems communicating with each other. In responding to this incident Amazon for the first time shed some light on its innerworkings including the use of a gossip protocol which quickly spreads server state information throughout the S3 system.

According to their public statement, This gossip protocol allows Amazon S3 to quickly route around failed or unreachable servers, among other things. When one server connects to another as part of processing a customer's request, it starts by gossiping about the system state. Only after gossip is completed will the server send along the information related to the customer request. On that day Amazon S3 began to see a large number of servers that were spending almost all of their time gossiping and a disproportionate amount of servers that had failed while gossiping. In order to fix the problem they needed to preform a full system restart.

This brings up an interesting question about the use of federated network protocols within cloud services. At eNomaly we have been big fans of use of XMPP for federated communications within our Enomalism cloud platform for multi cloud communications (Wide Area Cloud). XMPP is interesting because it natively solves a number of federation problems within a tried and tested framework. One of the biggest benefits to the use of a gossip protocol lies in the the robust spread of information and the exponential nature of it's sharing of information within a large number of machines.

One such example provided by wikipedia is in a network with 25,000 machines, it's usage can find the best match after about 30 rounds of gossip: 15 to spread the search string and 15 more to discover the best match. A gossip exchange could occur as often as once every tenth of a second without imposing undue load, hence this form of network search could search a big data center in about 3 seconds.

I wonder what are others doing to address federation issues within large scale cloud deployments? And how can we avoid the full system reboot in a worst case scenario?

Cloud Storage Discussion Group Created

This is quick note to let everyone know we've created a cloud storage discussion group. The Cloud Storage Group was formed in order to provide a common ground for the discussion and advancement of cloud based storage.

You may signup for the group at:

Or by sending an email to [email protected]

Looking forward to seeing you join the discussion.

Saturday, July 26, 2008

The Rise of The Dark Cloud

For nearly as long as the internet has been around there have been private subnetworks called the darknets. These private, covert and often secret networks were typically formed as decentralized groups of people engaged in the sharing of information, computing resources and communications typically for illegal activities.

Recently there has been a resurgence in interest of the darknet ranging from the more unsavory such as P2P filesharing and botnets as well as more mainstream usages such as inter-government information sharing, bandwidth alliances or even offensive military botnets. All of these activities are pointing to a growing interest in the form of covert computing I call "dark cloud computing" whereby a private computing alliance is formed. In this alliance members are able to pool together computing resources to address the ever expanding need for capacity.

According to my favorite source of quick disinformation, The term Darknet was originally coined in the 1970s to designate networks which were isolated from ARPANET (which evolved into the Internet) for security purposes. Some darknets were able to receive data from ARPANET but had addresses which did not appear in the network lists and would not answer pings or other inquiries. More recently the term has been associated with the use of dark fiber networks, private file sharing networks and distributed criminal botnets.

The botnet is quickly becoming the tool of choice for governments around the globe. Recently Col. Charles W. Williamson III. staff judge advocate, Air Force Intelligence, Surveillance and Reconnaissance Agency, writes in Armed Forces Journal for the need of botnets within the US DoD. In his report he writes " The world has abandoned a fortress mentality in the real world, and we need to move beyond it in cyberspace. America needs a network that can project power by building an af.mil robot network (botnet) that can direct such massive amounts of traffic to target computers that they can no longer communicate and become no more useful to our adversaries than hunks of metal and plastic. America needs the ability to carpet bomb in cyberspace to create the deterrent we lack."

I highly doubt the US is alone in this thinking. The world is more then ever driven by information and botnet usages are not just limited to governments but to enterprises as well. In our modern information driven economy the distinction between corporation and governmental organization has been increasingly blurred. Corporate entities are quickly realizing they need the same network protections. By covertly pooling resources in the form of a dark cloud or cloud alliance, members are able to counter or block network threats in a private, anonymous and quarantined fashion. This type distributed network environment may act as an early warning and threat avoidance system. An anonymous cloud computing alliance would enable a network of decentralized nodes capable of neutralizing potential threats through a series of counter measures.

My question is: Are we on the brink of seeing the rise of private corporate darknets aka dark clouds? And if so, what are the legal ramifications, and do they out weight the need to protect ourselves from criminals who can and will use these tactics against us?

Friday, July 25, 2008

Ebay to build a cloud?

Just got an interesting piece of information from our Google Cloud Computing Group about an ebay job posting. In the posting they are appear to be looking for a "Director, Cloud Computing Engineering".

Here is the job description:
The Systems & Architecture team is chartered with providing superior reusable infrastructure, best practices, services, tools and shared components enabling world-class development and operational management of eBay applications. As part of the Systems team, this position will lead engineering teams to develop innovative solutions that meet needs with respect to functionality, performance, scalability, reliability, realistic implementation schedules, and adherence to development goals and principles.
This position is responsible for leading the strategy and implementation of initiatives, as well as the day to day engineering team management for the Cloud Computing initiative.
It is interesting they have a cloud computing initiative, I wonder what it will look like?

VCs Have Their Heads in the Clouds

Interesting article over at businessweek.com about venture capital and cloud computing. The article states that so far this year companies providing cloud services or building services on top of the cloud have raked in more than $70 million in VC funding. That's an amazing number, given industry is just emerging and the opportunities are still for the mos part unknown.

Enomaly was also mentioned, "With the exception of Enomaly and Coghead, all of these startups have scored venture funding this year."

The only problem with the article was I was misquoted. The article stated that I was getting 40 VC inquires a week. The actual number is 40 or so all toll.

It's great to see so much interest in cloud computing and this can only help upstarts get the funding they need to become the next Google!

Read the whole article here >

Wednesday, July 23, 2008

IEEE Keynote & Network Centric Warfare

Had one of those surreal evenings tonight. The keynote at the IEEE summit in Washington went off like clockwork. At this point I could probably do a cloud computing keynote speech in my sleep. It is interesting giving a practical overview to a room full of academics who typically focus on the theoretical rather then practical uses of technology.

After the speech I went for drinks with several of the organizers of the event including a high ranking DoD (Department of Defense) official involved in the management of their technical infrastructure. The conversation ranged from the discussion of "network centric warfare", to Zen, to life after death. In midst of the conversation the official ordered a bottle of Shiraz wine, when we received the bottle its label read "Peace" with a big peace sign on the front. Again extremely ironic given my counterpart and the topic of our conversation.

One interesting piece of news I did get was confirmation that the DoD is indeed looking at creating their own compute cloud and it's usage ranges from run of the mill IT system management to counter network defenses and the word du jour, network-centric operations (NCO). According to wikipedia NCO seeks to translate an information advantage, enabled in part by information technology, into a competitive warfighting advantage through the robust networking of well informed geographically dispersed forces. It appears that cloud computing is an ideal tool to enable this. Who knew?

Monday, July 21, 2008

Cut Amazon a break!

My phone has been ringing off the hook today from reporters looking for me to say something bad about the Amazon outage. Bluntly, it ain't going to happen. Amazon is a pioneer blazing a path in an unproven and emerging sector of technology. Amazon has embarked on an ambitious service few if any, have ever attempted before. Things undoubtedly can and will go wrong. The fact is I can count on one hand the amount of times Amazon has gone down over the last two and half years. The trouble with being first is your first.

My faith in Amazon is as strong now as it was when I first started working with them. Shit happens and I doubt anyone else could do any better. So cut them a break already.

The Cloud is the Computer

It's been more then 25 years since Sun Co-founder John Gage first said the phrase, "The Network is the Computer." As cloud computing has begun to signify a shift in computing this statement has become more relevant then ever. For me, "The Cloud is the Computer". As many readers of my blog know I am a big proponent to a future where all your applications and storage are consumed over a network connection. In my vision of our computing future, the computer starts to look more like a thin client or mobile device and less like a traditional desktop.

So this brings me to a new company called CherryPal that today announced they are to begin offering a $249 paperback-sized box containing an underpowered processor and a token amount of memory . The idea is you just stick in any keyboard or mouse with a USB connector and any standard monitor (VGA), and you’re up and running.

According to their press release the "The CherryPal" set to begin shipping at the end of this month, is a true cloud computer, meaning almost all if its applications and data are stored online." For those who do nothing more then check their email, access productivity apps and surf the web the Cherrypal might be a perfect low cost alternative to the traditional desktop.

With Amazon's recent S3 outage ongoing a big question still remains: Are mainstream consumers ready to offload control of their digital life to a series remote cloud services providers that can and do go down?

More details here > http://cherrypal.com/

Sunday, July 20, 2008

Amazon S3 & SQ3 is down 2 hours and counting

This is a quick note to let everyone know Amazon s3 and SQS are down and have been down for a least two as hours as of 7pm EST.

For more details please visit

Saturday, July 19, 2008

The Global Cloud Exchange

Back in 2005 Sun Microsystems announced the company planned to build the world's first online compute exchange. More then three years later there has been no other mention of this supposed compute exchange. In the original press released they described the offering as a plan to introduce a new electronic trading environment that will allow customers to bid on CPU usage cycles. They went on to say that being able to dynamically bid for open compute cycles will provide companies across the globe with unprecedented flexibility in planning for the purchase and use of compute power. This is a new paradigm in computing where companies can access an unlimited number of CPUs as they need them. Today as cloud computing begins to take off and regional cloud utilities start to come online the idea of a cloud exchange is again beginning to be discussed.

Back in April at the Interop conference several attendees mentioned they wanted to see the creation of such an exchange platform. The reasoning was that as new regional clouds come online having a uniform point of entry to a world wide cloud ecosystem will make this type of transition more efficient.

Right now most clouds have there own set of APIs, interfaces and unique technologies. An open compute exchange may provide a centralized point where cloud consumers and providers would be able to make decisions based upon which cloud resources they may want to utilize as well as a clearing house for providers with excess capacity. Variables may include metering based on actual use of the resources in CPU hours, gigabits (Gbs) consumed, load, network I/o, peak vs off peak time frames, geographical location, SLA's, and quality of service rules could be just some of the metrics that determine the price of a cloud providers resources.

One usage example might be in terms of a green or eco-centric point of view. Let's say Cloud A uses cheaper coal based energy source and Cloud B uses a more expensive Hydro source. Although more expensive, choosing Cloud B may help offset an enterprises carbon credits and therefor actually be a bit cheaper from a carbon point of view.

Another example may be based on geographical cloud computing. Let's say a UK cloud and a North American cloud. Rather scaling based on system load, a cloud user may want to monitor application response times based on geographical location and scale according to an end users experience. By have the option to access compute capacity through an exchange, cloud consumers who are running global network services would no longer have to signup for several cloud services. This would also effectively render edge based CDN services like Akamai irrelevant.

What are your thoughts?

Enomalism 2.1 Released

Enomaly, Inc. is pleased to announce the latest beta release of the Enomalism Elastic Computing Platform, Enomalism v2.1 beta 2

Latest Enomalism Updates include:

New Features

  • Machine import feature - Existing running KVM/Xen VM's will be automatically imported into Enomalism
  • Machine auto-start feature - Enomalism now tracks the last running state of a VM and will restore that state on startup (VM's will boot up after power failure)

Bug Fixes

  • Fixed user create/update issues
  • Fixed user language setting issue for admin and regular users
  • Removed non-functioning hypervisor install targets from VM Creator
  • Internationalization enhancements
For installation Documentation and core distribution download, please visit

Direct Downloads are available on sourceforge at

You can chat with the developers, and get answers to difficult problems on IRC @ irc.freenode.net #enomalism

If you are interested in lending a hand, please visit the Enomalism forums at

Friday, July 18, 2008

CloudWars: Pentagon Seeking to adopt cloud computing

I've heard rumors about the Pentagon's IT unit seeking to adopt cloud computing for a few months. I knew something was up when I started seeing a surprising amount of download traffic on our Enomalism project from .mil and gov hosts. Today it seems they have gone public, and yes they are looking very closely at cloud computing.

Quoting from the New York Times article, "John Garing, a retired Air Force colonel who now is DISA's CIO, said he and his staff have met with their counterparts at businesses such as Google Inc., Amazon.com Inc., United Parcel Service of America Inc. and travel-reservation systems operator Sabre Holdings Corp. to talk about how the companies use technology. A pending trip to FedEx Corp. is scheduled for next week, he added."

He goes on to say ", Garing is convinced that cloud-based IT services will be the future of military data processing. Cloud computing is "going to be the way it has to be," he said. "We have to get to this standard environment that is provisionable and scalable."

The real reasoning seems fairly simple to me, they don't want to out gunned in the new computing arms race. In the late 20th century cold wars were fought using the placement of real weapons. In the 21st century they will be fought with information and compute capacity. The cloud (a.k.a a botnet) seems to be the weapon of choice for countries like Russia, China and some say even the US in this newest form of one-ups-manship.

Lets say for example China is building a massive data under or near the three gorges damn. The likelihood of that data center underneath being bombed and killing 20 million people is fairly slim. Putting massive data centers near nuclear reactors seems like another smart location for governments. Cheap limitless power is always nice and bombing a critical piece of infrastructure tends to make for a rather difficult situation to get out of.

I'll be keynoting the IEEE summit next week in Washington DC, so any spooks that want to chat about cloud computing, feel free to ping me.

Full article is available here> http://www.nytimes.com/idg/IDG_852573C400693880002574890080F9EF.html?ref=technology

Wednesday, July 16, 2008

The offline cloud

I've spent last couple days in jolly old London England engaged in a variety of meetings as well as to attend the first ever CloudCamp London which took place earlier this evening. The event was again a total success with more then 250+ attending. The beer and wine was flowing and the audience was a great mix of experts and novices alike. One of the more ironic parts of the evening was when we lost the internet connectivity for our Crypt on the Green venue. (Located in the basement of an old church) Given the event was for cloud computing the irony was not lost on anyone.

The network going down did bring up an interesting topic of discussion. What happens with cloud apps when the internet goes down? When I use my blackberry on a plane I can still check my previously downloaded email but at the same time I can't twitter my latest updates or check our CRM, SVN or any other hosted services. There are a number of projects attempting to address this offline cloud problem. Most noticeably Google Gears.

My question is; are we moving toward a future where the desktop is nothing more then a local cache? Or will we soon reach a point where technologies like Wi-max, 3G and wifi on planes make the internet a ubiquitous part of our everyday life where local storage isn't needed at all?

I guess only time will tell.

Saturday, July 12, 2008

RumorMill: Microsoft to Rollout EC2 for Windows in October 2008

I've been away the last few weeks hanging out at my cottage in Ontario. Before I left for the the laid back country life, I did get an interesting piece of gossip. It seems that Microsoft is working on a project code named "Red Dog" which is said to be an "EC2 For Windows" cloud offering. The details are sketchy, but the word on the street is that it will launch in October during Microsoft's PDC2008 developers conference in Los Angeles. Original reports back in April indicated that the service would look more like the Google App Engine or a Mosso. But from what I'm being told it's going to look more like an EC2 environment.

A few companies have attempted to offer on demand Windows clouds including Terremark's "The Enterprise Cloud" (VMware based) and Gogrid. But both have yet to release an API for programmatic access, so there usefulness in a truly "elastic computing" environment is questionable. The folks at Gogrid have indicated they plan on offering an API in the near future which should certainly help in getting further user adoption. It also appears that just about every major telecom or data center provider is either thinking about or already in the midst of developing their own cloud offerings. From what I'm seeing, the market for "cloud builders" and "cloud enablers" appears to be booming.

While I'm on the topic of rumors, I also got wind of a Chinese cloud project rumored to be currently in development "under a hydro dam" which when complete may exceed one million servers. (If you're involved in this project ping me, We'd love to help out)

These rumors come from multiple credible sources, but then again they are just "rumors" so don't blame me if they turn out to be erroneous. None the less a Microsoft cloud could pose a serious threat for any companies looking to build their own windows based cloud offerings.

#DigitalNibbles Podcast Sponsored by Intel

If you would like to be a guest on the show, please get in touch.