Cloud Chaos & Decentralization

In the late nineties and early part of this decade there was a marketing push around the concept of "centralization". Companies like IBM, Oracle and Sun focused on creating hardware and software platforms with single points of deployment and administration in the vain attempt to make it easier manage your infrastructure. It quickly became apparent that for all its marketing hype centralization has created more problems then it has solved.

In nature, most things are not centralized, they are almost always decentralized. Centralization is a human construct used to create structure to an unstructured world. Whether an ant hill or a human body, the Sun or a Galaxy, decentralization and chaos is all around us. Some may see decentralization as anarchy or chaos but in the chaos comes the ability for systems whose states can evolve and adapt over time. These adaptive systems can exhibit dynamics that are highly sensitive to initial conditions and may adjust to demands placed on them.

To build scalable cloud platforms the use of decentralized architectures and systems maybe our best option. The cloud must run like a decentralized organism, one without a single person or organization managing it. Like the Internet it should allow 99 percent of its day-to-day operations to be coordinated without a central authority. The Internet is in itself the best example of a scalable decentralized system and should serve as our model.

The general concept of decentralization is to remove the central structure of a network so that each object can communicate as an equal to any other object. The main benefits to decentralization are applications deployed in this fashion tend to be more adaptive and fault tolerant, because a single point of failure is eliminated. On the flip side, they are also harder to shut down and can be slower. For a wide variety of applications decentralization appears to be an ideal model for an adaptive computing environment.

For me, cloud computing is a metaphor for Internet based computing and therefore should be the basis for any cloud reference architectures. In the case of the creation of cloud computing platforms we need to look at decentralization as a way of autonomously coordinating a global network of unprecedented scale and complexity with little or no human management. Through the chaos of decentralization will emerge our best hope for truly scalable cloud environments.
----
This has been a random thought brought to you on a random night.