Cloud Technology, Big Data & Hadoop

Big Data seems to be the word of the year.  Everywhere you look there is Big Data staring you in the face: Blogs dedicated to discussing Big Data; LinkedIn groups with over 10,000 members focusing on Big Data topics.  Just under 15,000 times each month, the exact phrase Big Data, is entered into the search engines of people across the world.

Wikipedia begins to describe Big Data as datasets that grow so large that they become awkward to work with using on-hand database management tools.  Because there is so much content being created at speeds we have never before seen, there is this preconceived notion that in order to be as efficient and productive and intuitive as we all want to be, Big Data needs to be addressed to some degree or another.

Three notions that seem to be facilitating this Big Data movement include:

1. Faster analysis of larger operational data will help you make better decisions.


2. More in-depth analysis of customer data will guide you to better customer segmentation.


3. Insight into larger data sets will help you come up with innovative product design.

Usually when Big Data is mentioned, the reference or inclusion of cloud technology is not far behind.  That discussion includes public and private clouds, scalability, flexibility, open PaaS and all of the other cloud characteristics.  Therein lies the solution that many have already found, and that is Hadoop. "The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model." Not every cloud needs Hadoop, and Hadoop doesn't need the cloud.  But when leveraged collectively you have found a pretty good solution for your Big Data concerns.

Some interesting users of Hadoop include:

• Amazon, eBay, Rackspace, Twitter, among many others (http://wiki.apache.org/hadoop/PoweredBy).


• LinkedIn uses it on 5,200 cores to manage 7.2 PB or 7,200,000 GB of data.


• Facebook uses it on 11,200 cores to manage 12 PB.


• Yahoo uses it on 36,000 cores to manage 18 PB.

At OpenLogic, we store all of the world's open source code in Hadoop as part of our scanning solution.



Hadoop is now a generally accepted viable open source solution for your Big Data dilemma. It is by far the leading solution in its field. It is the Linux of large-scale data processing and is understood to be the long term solution for organizations of all sizes.  Like all open source, Hadoop is a great win-win.  It saves the wasted energy of developing many different and not nearly as capable closed solutions. Also, because Hadoop has become so capable, it can provide us with the reliable data management foundation on which so many of the large-scale Internet innovations of the last few years have been built.

Harnessing cloud technology and the power that Haddoop provides us with, we can realize the agility that we are all looking for, as we slowly migrate different parts of our enterprise, and even our personal lives, into some dynamic of the public and private clouds. Cloud technology offers the end user many different ways to calculate ROI and justify the move to the cloud.

Big Data can be managed with the right resources and technology while still providing you and your customers with the opportunity to be agile.  Ramp up ramp down, be creative, be innovative,  and use data driven decision making to do it as efficiently as possible.


 

Subscribe to The Enterprise Open Source Blog via email

Follow @openlogic
Follow @cloudswing
Follow @AaronMandelbaum

View Aaron Mandelbaum's profile

This work is licensed under a Creative Commons Attribution 3.0 Unported License

.Follow @openlogic
Follow @OSCloudServices

This work is licensed under a Creative Commons Attribution 3.0 Unported License
.