My Big Data Obsession

My own obsession with Big Data began while I was training for Ironman France. It was late summer of 2011 and after barely clawing my way through a sprint distance triathlon, I desperately wanted to get back in shape.

I needed a big goal to help motivate me and I wanted to do something social as well. Training for a longer distance triathlon with members of the San Francisco Triathlon Club--without whom I never would have been able to complete the Ironman--seemed like just thing.

For those who aren’t familiar with it, an Ironman is a grueling 2.4 mile swim, 112 mile bike ride, and full 26.2 mile marathon, back-to-back. It’s made tougher by weather--over 90 degrees in the case of Ironman France--and the need for sustained nutrition.

According to some estimates, athletes can burn as many as 12,000 calories during an Ironman, in contrast to the 2,500 to 3,000 calories we burn on a typical day. Over the course of my nine months of training, I would log thousands of laps, hills, and runs on my way to finishing the Ironman.

And I wasn’t alone. To date, fitness buffs and athletes alike have logged almost 2.8 billion miles to Garmin Connect, a web site hosted by the well-known maker of GPS devices.

This combination of low-cost “sensor” devices and web based data logging and analysis epitomizes Big Data. It once used to cost hundreds of thousands or millions of dollars to capture data points, store them, and analyze them, not to mention the cost of data analysts and engineers to work with the data and convert it into insights. Now anyone can log basic data and gain useful knowledge for just a few hundred dollars or less.

But training for the Ironman was just one half of my Big Data equation. The other half was the public offerings last year of companies like Facebook and LinkedIn--which at their core are Big Data companies--and Splunk, a Big Data company targeting businesses.

This combination of the potential impact of Big Data on my own life plus the sense that these IPOs foretold a sea change in the technology landscape intrigued me. Just as I was finishing the Ironman, I embarked on another journey, to understand the full landscape of Big Data. Much of the knowledge I have gained in the process I plan to share in this column and in my upcoming book, Big Data Demystified: How Big Data Is Changing The Way We Live, Love and Learn.

We’ll look at how companies like Amazon, Facebook, Google, and Netflix are leveraging Big Data. We’ll also look at Big Data problems that continue to stump today’s best and brightest: why we still can’t predict earthquakes despite having immense amounts of data on the subject, and why the picture you choose for your online dating profile may not matter as much as you think it does. From talking with thought leaders in the space to exploring the impact of Big Data on our businesses and our lives, I look forward to dialoging with you about this exciting topic.

Disclosure: Mr. Feinleib holds shares in Facebook, Splunk, and other technology companies.