Chris J Powell

Big Data vs. Small Data

There is a tremendous marketing engine out there that is building up the need to have a Big Data Strategy and spend money on better Tools and Resources to enable better decision making through the use of Big Data.  While that in many cases is exactly what the eventual goal should be…I find that almost every day…companies that I work with are still being continuously challenged with the Small Data and its quality! recently posted an interesting article titled Don’t Worry About Big Data; Sweat the Small Stuff and in this article it illustrates the need to learn how to walk before you run.  The fact that most companies do not have an effective Data Management Plan in place for their existing Data but because of the hype are willing to go out and purchase and connect with more data will only lead to worse decisions being made not better.

Data Analytics is a field that while there is a growing demand for good people in it…does not take a Rocket Scientist to start to understand it (although some Rocket Scientists might disagree like Chris Mattman) but even Chris would see the benefits of learning the true benefits of quality, before adding in quantity.  The real challenge with Data of any kind is learning to look for what it is that you want.  Having a plan before you dive into the cumbersome and time consuming world of building the data sets that you want to compare.

I had to take a step back when I built my Hadoop connection that scours through the White Papers, Books and other media that I “collect” to find the relevancy.  It was not easy to do but this was mainly because I was dealing with a unstructured Data that did not easily allow myself to build data sets (over 1 TB of PDFs, ePub and Word Docs).  Had I taken a step back there would have been several other ways that I could have sorted the information and I am currently working on a very simple catalogue system that will also let me set up a better way of searching for the information that I want as I focus on Researching more in-depth topics.

I firmly believe that Data is what will set this generation apart from every other one that preceded it.  Every person should have at least a basic understanding of how to piece together two different pieces of data and build a foundation for analysis but whether you are in IT, Sales or working the floor at a Widget Factory…that is not necessarily what you care about.

Data Driven Applications can be built using the foundations of Small Data and from the ground up have the option to add in the more complex parts of Big Data as the capability builds itself into the needs of the organization.

Big Data is not going anywhere.  In a couple of years it may be called something different (just like Cloud Computing has changed from Outsourcing to Managed Services to its current incarnation) but focus on getting your current house in order.  Build a solid understanding of Data Quality and bring Order to the Chaos that is your current Master Data Management Plan.  These two things will not only enable a better and more fundamental change for how you and the organization you are working for or the reason you are looking at data in the first place.

Think back to a child.  They do not come out of the womb ready to run a 4 minute mile.  They spend time as bumps on a log, then they learn to roll over, then crawl eventually pulling themselves up onto objects and finally figuring out how to walk.  From there, as confidence and coordination kick in…then they are able to run, kick balls, hit baseballs and do all the things that we as grown adults take for granted.

Take a step back from the problem at hand.  The challenge you face is not unique, you are not alone.  But do not jump into the money pit that Big Data may present you with…get your house in order and better prepare for the onslaught that will eventually come your way.


Chris J Powell

Leave a comment