Technology, Big Data

Preparing Your Infrastructure and Workforce for Big Data

Blog-post by,

As CIOs plan for 2012, they may be hearing the buzz of Big Data more than Cloud (see this good write-up from Charlie Bess). While cloud computing has a significant impact on infrastructure – enterprises can either transform internal IT to a services model or outsource – big data’s impact on existing environments has not been well defined yet. Wikibon and SiliconAngle provided coverage of the recent Hadoop World conference in NYC (full collection of videos and articles here). Real customers of the technology provided proof points of how data scientists and the data analytics tools provide new insights into information that is transformational to business. While Internet bellwethers like Facebook, LinkedIn and Twitter are prominent early adopters, if Hortonworks CEO Eric Baldeschwieler’s prediction that Apache Hadoop will process half of the world’s data in five years is even close to the mark, this is a trend that can not be ignored.

The Big Data is partially an evolution of traditional business intelligence technologies meeting the intersection of the mobile and cloud waves. While the volume of “big” data get plenty of attention, we know that as an industry that today’s large amount of information will be considered small in a couple of years; the important thing to consider is that the speed (faster towards real-time), location (distributed), and type of data (trending towards unstructured) of the data requires new tools and methods to extract information. While much of this change is in software, there are optimizations needed in the underlying infrastructure. See my discussion of the impact to networking architectures. Similarly, as Colin Mahoney of HP’s Vertica group stated (see video here), HDFS (the default storage layer for a Hadoop environment) is a threat to traditional storage architectures. While the ripple effect of Hadoop and other Big Data tools are important, the biggest gap that companies looking to leverage these tools have is finding qualified data scientists. Training was a major focus at Hadoop World and since there is a shortage of trained people in this nascent field, CIOs should be sure to allocate ample budget to help educate the workforce so that they can grow into the new technologies.

Additional references on Big Data:

(2) (2)

Would you like to comment on this content? Log in or Register.
Paul Calento 255 Points | Fri, 11/25/2011 - 21:14

One of the keys to preparing for Big Data is an understanding of its potential impact. John Dodge recently posted a related blog citing an Atlantic Magazine article comparing Big Data to Anton van Leeuwenhoek's invention of the microscope in the 1670s. Lack of best practices is, to some extent, an inhibitor. The key isn't Big Data "potential", but how to implement in the context of other initiatives in the Instant-On era. Education, as noted is critical, but lack of established learnings/best practices inhibits Big Data from reaching its promise.

--Paul Calento

(note: I work on projects sponsored by and HP)

Pearl Zhu 90 Points | Fri, 11/25/2011 - 18:34

Hi, Stuart, very resurceful blog to articulate Big Data, with charisteristic of 4Vs: volume, volocity, voriety, viability, if processed intelligently, it becomes the biggest opportunity for organization to gain customer insight and marketing wisdom, otherwise, i's truely like to find the needle at haystack, a big distraction, look forward to see more big data related technology and vendors start maturing. thanks.