Big data is all about managing huge amount of data and deriving meaningful information. The size of data which can be analyzed using big data are as big as in several Petabytes of data. It is defined well with three Vs – volume, velocity and variety.
Volume: Today organizations are generating GBs of data everyday. These data are primarily generated in the transaction-based structured data storage, unstructured data published in social media, logs generated by various devices and software. Now a days, storage is not so costly and hence, no issues around storing but almost impossible to analyze the data using traditional approach. Big data is the technology which can successfully analyze petabytes (1000s of GBs and TBs) of data. There is huge value of analyzing these data for right decision making.
Velocity: Today data is streamed at tremendous speed from various sensors, devices and is required to be analyzed quickly. This is a major challenge for organizations and big data is the answer to solve this problem.
Variety: Today data is generated in several formats. Structured data from traditional transaction systems built on RDBMS, semi-structured data from social network feeds and unstructured data – documents, emails, videos, audios, logs generated by sensors, RFIDs, several devices. It is major challenge to analyze these variety of data using traditional RDBMS and analytical/reporting tools and hence, we have big data which is the right response is gaining popularity.
Apart from these three main characteristics of big data applicability, below are few more:
Variability. Data flows can be highly inconsistent in terms of peaks. Sometimes social media feeds can be huge if some campaign is going on, lot of reviews and ratings can be for eCommerce websites during huge offers/discounts. This is another challenge to deal with scale dynamically and elastically.
Complexity. Complexity is yet another factor with today’s data since it comes from several different sources and meaningful information requires to be generated by connecting and correlating all such data.
Above must have given you a good view of challenges big data can deal with. Before I start talking about the opportunity for big data, let me say that neither google nor any social networking sites – Facebook, Twitter, LinkedIn etc. would have existed without big data technology. Yes, all of them use big data as underlying technology to store unstructured data and fetch very quickly for users.
Below is an indicative list of opportunities which big data has today:-