Mulling over several topics for this morning’s post I stumbled across an incredibly interesting article over at ZDNet by Ken Hess, titled Dispelling the “big data” myth. Much of what he writes about in this article is his general dislike about Tech Buzz words like Big Data (I can only imagine what he thinks about Cloud Computing). There is no doubt that what we think of as Big Data today is truly “Big” but it brought up an interesting question as to if our stuff is Big…relatively speaking…wasn’t it the same for past iterations of Data as well?
Taking a trip down memory lane like Ken Hess did in his article should put things into perspective that the reality is…while Big Data is today is absolutely massive…it was pretty big and onerous then too.
When you do a Google Search for Big Data History, there seems to be very little reference to the concept that Data was Big before the term really took off as part of our Common Tech Language in 2008-2009 when data sets started hitting the petabyte scale range. I did find an interesting reference to “A Very Short History of Big Data” and I thought what better way to share the perspective that what was big yesterday was just as onerous of a task as manipulating a multi-petabyte database is today.
The requirement to store, manipulate and share data has been a problem throughout man’s history. When the loss of the Library at Alexandria (at the time the hub for Civilizations shared knowledge) contributed to the Dark Ages from the 5th Century to the 15th Century. Man’s thirst to know all that he can and then contribute to that understanding is what drives us forward in so many ways. While Western Civilization took a 1000 yer set back with the loss of the original Big Data, imagine what would happen to us today if there was a catastrophic collapse of say…Google and all the knowledge that it shares with us every day!
By the mid 1940’s the written word had become a massive undertaking to both maintain and share. It was estimated in 1944 that American University Libraries were doubling in size every 16 years. It is amazing how the use of technology has enabled us to build a Data Library that does not require the use of 6000 miles of book shelves. The 200,000,000 volumes of man’s written word could be fit into a Digital Library that spans about 111 Terabytes (this calculation taken from an article I found on the average ePub size).
Data has always been a part of what we do. As the Digital Age approached and NASA was putting men on the moon, the calculations and data used to make the decisions that changes the course of history were monumental…and this was done on computers that were the size of a compact car and had far less computing power than we now carry in our front pocket in the form of a Smart Phone.
When we think about the number of calculations and decision support that went into something as big as the mission to the moon…that in its time was Big FREAKING Data…but it was done on computers with a maximum level of storage of just 6 Mega Bytes. Imagine 6 Mega Bytes is all you had and to bring in another data point, or do comparisons…oh my!
Fast forward to 1995 and the real birth of the Internet Age. Computers are far faster than ever before, the glorious Windows 95 is making in roads and everyone is clamoring for a Personal Computer. But still, I remember that my first Hard Drive on the Packard Bell still only had a 2.3 GB hard drive in it (of course I upgraded as soon as I could). This internet age that gave us new sources of information streamed to our 13″ monitors was a godsend for someone like me who is a sponge for information. New knowledge streamed to me at 28.8 baud…oh what a joy. I quickly filled floppy disk after floppy disk at 1.44 MB a shot with all that I could find…it was heaven.
Again we fast forward to the present. Big Data is all around us. It is the newest thing, but when I talk to my clients…I get the sense that the concept that this term was never intended to be a call to action for them, it was for all the non-technical people who sign the checks so that IT can get the proper storage to let business continue to expand and flourish.
Big Data is Relative. Whether the data will consume 6,000 miles of library Shelves, 100+ Terabyte Drives, 200 Floppy Disks or scores of CDs, DVDs and Flash Drives. It is here. It has always been here and it is not going away. Embrace the Data…it is your friend.
Chris J Powell