The Value of Long-Range Vision – Data Storage


(An example of how improper data storage can turn a growing and successful business in a new industry into a poorly growing and eventually shrinking business. This story is from real events, but people's names and organizations have been changed or anonymized.)

Acme Inc was formed with a vision of data collection that would assist companies in understanding their customer's buying behavior. They put together a working version of their product, launched before others had similar products, got companies signed up for their service, and began building up a solid book of business.

However, as the years went by, it became steadily harder for companies to extract out the information they needed. The Acme website, formerly capable of providing detailed reports on demand, was slowing to a crawl. Without easy and reliable access to the reports that Acme was providing, many of Acme's clients began looking at competitors who might be able to provide a more consistent result.

When Acme asked its technical team to investigate and fix the problem, they discovered that the original version of the way their consumer data had been organized was set up so that generating the reports involved going through the entire set of information on all consumer actions going back to the founding of the company. Early on, this data set was relatively small, because Acme didn't have years of history or a large pool of clients, but as the company grew what had been a small problem was now so large that there was no clear solution. To reorganize the information would mean an unacceptable long disruption in service. Data could be moved into a longer-term archive or data warehouse, but this would also involve a service disruption, and would make it impossible for Acme's clients to see information about consumer activity before the cutoff to move the data into the warehouse.

Slowly but steadily, Acme became unable to provide the service it had advertised and sold. Instead of revenue growth and expanding client base, they are now facing the very real risk of going out of business as their clients head to their competitors. They cannot fix their problem without major changes to their systems, and the costs piled up:

  • $300,000 worth of staff time.
  • $30,000 on hardware to keep the existing systems functioning "well enough".
  • $180,000 annually in lost revenue from major clients who have since left.
  • $50,000 annually in the new products that could have been developed but weren't because of the organizational focus on the data storage problem.

At Erie Eyrie, whenever we are handling user information, we assume that the information we have to track will continue to grow indefinitely. We expect to have to adjust how it is stored, how it is searched, how it is collated, and how it is reported on, in order to keep that data accessible and useful.

Currently unrated


Oupa 1 year, 9 months ago

I had a similar experience with a multinational insurance company. They had spent millions developing a new application for underwriters. It was tested. The business people had signed off and it was rolled out to thousands of locations. At first everything was good. People were happy. And then the response times became slower and slower and slower. Clients were on hold for 3 or 4 minutes.

That's when I was called in. Since I did not have time to delve into the millions of lines of code, I monitored the SQL requests going from the app to the data server. The unacceptable response was all from one query. The query caused the server to do a sequential search of a table that had quickly grown into hundreds of millions of rows. Fortunately it was easy to be a hero. I was able to build an index for the tables and all was good.

Link | Reply
Currently unrated

Comment deleted 1 year, 2 months ago

Comment deleted 1 year, 2 months ago

Comment deleted 1 year, 1 month ago

New Comment


required (not published)