Becoming intimate with Big Data
About a year ago, I had a chance to have a discussion with one of the smartest person I’ve ever met, currently a board member of our company. This man has not only built his fortune out of nothing by being able to identify trends in the market and position his companies accordingly, he is also a genuine human being that forces admiration. But I digress. During this conversation, he mentioned that one of the things that helped him succeed was his capacity to understand the intrinsic values that define a generation. As an example, he mentioned that his generation, during the 90s was all about financial success. The following generation, the 2000s kids was all about fame (big brother anyone?). Then he told me that he was yet to figure out what my generation was all about. Since then I have been able to understand what makes my generation tick. After about a year of poking around, I think that I found the answer: my generation is the selfish generation. We are all selfish and think about our individuality. Look around, it’s selfies, freedom above all, my Facebook or my privacy, my right for an opinion, my right for an outlet to express my idea. I’m including myself in this of course, I am writing a blog after all. What’s interesting about this realization is to understand the consequences it has on the market, and specifically in a domain in which I have at least a bit of expertise: Big Data.
Big Data is driven by the individual
In a recent report from Forrester (link), companies were asked “Which use cases are driving the demand for continuous global data availability at your organization?”. The most common use case representing 52% of the answers received was 360-degree view of the business, product. This means that more than half of the big data drivers are coming from the consolidation of data to represent an individual unit of business. Make no mistake, in many cases, the product is you. What drives big data is the intimate knowledge of the individual. This makes perfect sense if you agree with the premise of my first paragraph: big data, and the market in general wants to cater to the selfish generation, and therefore is implementing solutions to know each individual personally.
This report is only one of numerous examples corroborating what I’m trying to explain here. We see machine learning algorithms and data scientists arguing about what algorithm is the best to target individual with the right add. IoT is tracking and personalizing every aspects of our lives. Anecdotally, I even witnessed the re-naming of a data analytics team in a large company to “Your Data”.
What does this mean for your Big Data implementation
First you need to consider that in order to be able to keep a relevant edge on your competition, you must be able to have access to a solution to individualize your data collection. I have expressed this opinion quite a bit, but I believe that ultimately individualization of data is a use case that requires its own solution. There is no magic end to end consolidation platform that will do everything. You need to consider a big data individualization platform, as opposed to a big data generic platform that you then try to morph in order to cater to your individualization needs. Once implemented, this data individualization platform can be leveraged to implement further features like real-time provisioning, data virtualization, personalized analytics or real customer centric support, but your platform must be intimate with your unit of business first.