Thirty years ago the technology world moved in two distinct architectural directions for resiliency; enterprise computing and networking/internet proceeded toward a loosely coupled resiliency model whereas storage focused on a tightly coupled model.

Both were viable options and, at the time, the directional choices were logical. Today, however, enterprise storage has become the weak link in modern IT largely because these architectural tenants have not been challenged. In fact, the storage attributes that drove the use of tightly coupled architectures no longer exist and I believe that enterprise storage should move to a loosely coupled foundation.

For those not familiar with tight and loose coupling let me provide real-world examples. Tightly coupled systems are used when simple forms of replication or redundancy are not practical or are too expensive. Airplanes are a good example of tightly coupled systems. If you are flying at 30,000 feet and something breaks you can’t simply switch planes; you really, really want to keep that plane flying. Systems must work together in very coordinated ways and yet still be resilient to failures. Planes use sophisticated software and systems and rely on meticulous maintenance to achieve their reliability. This is all clearly necessary but it does not come cheap.

For automobile travel we have loosely coupled systems. Our cars have many single points of failure and if we want “resiliency” we buy a second car, or rent one, etc. to deal with issues.  It may seem expensive but having two cars is very economical vs. trying to create a single car that NEVER breaks down.

Building tightly coupled systems may be hard but it’s very often necessary. In the case of storage, the implementation and use of RAID technology drove the requirement for tight coupling. RAID provided cost optimization for the highest-cost element at the time (storage) by creating tightly coupled groups of disk drives in conjunction with sophisticated resiliency algorithms to protect data against device failure and minimize storage consumption. This solution also required sophisticated redundant controllers, shared memory busses and things like mirrored cache. Most importantly, it required expensive and complex software that necessitated extensive testing to verify even minor changes.

This architectural choice made lots of sense in 1985 when 1 GB of storage cost $100,000 and storage capacity made up the bulk of system cost. Today, with storage approaching $0.01/GB the situation is quite different. Today, raw storage is no longer the largest cost of “storing” data. In fact, it can almost be considered free.

While the price of storage is going to zero, the overall cost of building traditional enterprise storage systems continues to climb due to the rising complexity of the software and continued need to develop customized hardware.  A foundational change in needed to put storage back in line with modern IT.

The transition to loose coupling for storage is well underway in many modern technology companies. For these companies it was actually more economical to hire teams, design, and build their own custom storage software platform vs. purchasing off-the-shelf arrays! These solutions generally use commodity hardware and simple replication for data protection and resilience. Our new company, Formation, will be the first storage company to bring this technology to the commercial market.

The benefits for loose coupling are numerous.  Loosely coupled systems are easier to engineer, can have more features, and can generally use more commodity components. Given the price of storage today, they can also be far more economical even though they typically replicate data rather than use RAID for data protection. For very low cost (deep archive) solutions, a technology called erasure coding can be used to further reduce storage cost while still maintaining loose coupling.

One concern I often get from enterprise IT leaders is that they “need the availability and reliability of an enterprise array.”  This is where things really get interesting because, done right, a loosely coupled system can not only be faster and more flexible but it can also be far more robust and reliable.  I will save the math for another blog but suffice it to say that loose coupling wins in reliability. Hands down.

Just about every storage player is declaring that they are “software defined” or that they use “commodity hardware” as even classic arrays mostly use x86 architectures these days. The real key to transformational features, reliability, and scale at dramatically lower costs is in the fundamental architecture.

I often get asked about Formation’s motto “Demand Incompatibility” as it appears to defy logic for “good business.” At Formation we understand that achieving breakthrough results means making foundational changes. One of these specific changes is moving to a loosely coupled system model.  The simple fact is that transformational results nearly always require disruptive change.




Read More