The amount of Data being generated by individuals, corporations and governments everyday is daunting and in many ways scary. If so much data is being created (by most estimates it is around 2.5 Exabytes per day or enough to fill 2.5 Million 1TB Hard Drives) then what is being done with it all? Is it safe? What is it used for? Does it ever go away?
This Ocean of New Data comes with many challenges but also a great many rewards and the investment in both collecting but also using and leveraging that Data is what I will look at today. There are two interesting studies that looked at in preparation for this post:
Looking through both of these documents it is clear that there is a huge potential upside of knowing more about…well everything and making smarter, more effective long term plans but to dive into the Risk vs. Rewards but to look at a deeper view of just how much the explosive world of Data has happened in a very short time frame.
- Quantifying overall Risk (ie applying mathematical formula to different situations). This can help organizations or individuals start to apply a statistical probability of success and make informed “go or no go” choices.
- The application of smaller low impact tasks that are data intensive can add up to a huge improvement. I like to think of this concept as being a baseball player who has a consistent swing and has a lifetime batting average of .300. He is likely to get into the Hall of Fame ahead of the player who batted after him who got 30 Home Runs a season for 3 seasons and then washed out. Do the little things right and often and it all adds up.
- It is not just Volume it is the “Velocity”. With so much new Data being generated, the processing of that data and the classification of it becomes vital. But “Value” is another key component. Hoarding Data just because you can is no good, having a plan to make use of it becomes vitally important.
- Extending an existing investment in Business Intelligence or Business Analytics to look at deeper relations from more data and different data can have a massive impact on the outcome of the analysis.
- Big Data is still a Buzz Word. Sure IBM built a machine that kicked but on Jeopardy but it is still out there on the Bleeding Edge of Adoption and it could be a “fad”.
- The Technology is largely unproven with new ways of looking at data (both Structured and Unstructured) being developed almost daily. This can lead to the “Squirrel Effect” taking valuable resources from tool to tool but never proving or disproving the effectiveness of the process (and therefore costing far more than is returned).
- Lack of People. This is not all that unusual, the world of Technology has been suffering a shortage for several years and Big Data is moving so fast that for the “Few the Proud and the Skilled” they are commanding top dollar with titles like Data Scientist, Data Architects, Data Engineers and Data Stewards.
- Security – This becomes a big issue. Back in November, Big Box Retailer Target had been hacked and Credit Card Data was being scraped and forwarded off to the hackers. The collection of Data is not just about gathering it up and then letting it be free range Data…no it needs to be penned, hemmed and protected. The value of the Data and the Cost of the Loss of it…is just too great to do nothing but harvest it.
Well, stay tuned for the next look at Risk Vs Reward when I look into the world of Mobility.