The term Big Data is being increasingly used almost everywhere – online and offline. And it isn’t limited to computers. It falls under the broad term information technology, which is now part of almost every other technology, field of study, and business. Big Data is not a big deal. The hype around it can be confusing. This article explains what Big Data is. It also contains an example of how Netflix used its data, or Big Data, to better serve its clients’ needs. It also explains the Big Data 3 Vs concepts and models.
What is Big Data Analytics
The data lying in your company’s servers was just data until yesterday – sorted and filed. Suddenly, the slang term Big Data became popular, and now your company’s data is Big Data. The term covers every piece of data your organization has stored until now. It includes data stored in the cloud and even the URLs that you bookmarked. Your company might not have digitized all the data. You may not have structured all the data already. But then, all the digital, paper, structured, and unstructured data in your company is now Big Data.
In short, all the data in your servers—whether or not categorized—are collectively called BIG DATA. You can use this data to get different results through different types of analysis. Not all analyses need to use all the data. The analysis uses different parts of the BIG DATA to produce the necessary results and predictions.
Big Data is the data you analyze to produce results you can use for predictions and other uses. When you use the term Big Data, your company or organization is working with top-level Information technology to derive different types of results from the same data you stored intentionally or unintentionally over the years.
Read: Data Science vs Computer Science explained.
How big is Big Data
Essentially, all the data combined is Big Data, but many researchers agree that Big Data – as such – cannot be manipulated using normal spreadsheets and regular database management tools. They need special analysis tools like Hadoop (we’ll study this in a separate post) so they can analyze all the data at once (which may include multiple iterations).
Contrary to the above, though I am not an expert on the subject, I would say that data from any organization—big or small, organized or unorganized—is Big Data for that organization and that the organization may choose its own tools to analyze the data.
Normally, people create different data sets based on one or more common fields to make data analysis easy. In the case of Big Data, you don’t need to create subsets. We now have tools that can analyze data regardless of its size. These tools probably categorize the data as they analyze it.
I find it important to mention two sentences from the book “Big Data” by Jimmy Guterman:
“Big Data: when the size and performance requirements for data management become significant design and decision factors for implementing a data management and analysis system.”
-And-
“For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration.”
So you see that volume and analysis are important to Big Data.
Read: What is Data Mining?
Big Data Concepts
This is another point where most people don’t agree. Some experts say that the Big Data Concepts are three V’s:
- Volume
- Velocity
- Variety
Some others add few more V’s to the concept:
- Visualization
- Veracity (Reliability)
- Variability and
- Value.
Big Data Example – How Netflix used it to fix its problems
Several years back, Netflix had an outage that left many customers in the dark. While some could still access the streaming services, most could not. Some customers managed to get their rented DVDs, whereas others failed. A blog post in The Wall Street Journal says Netflix had just started on-demand streaming.
The outage made management think about future problems, so it turned to Big Data. Using that data, it analyzed high-traffic areas, weak points, network throughput, etc., and worked to reduce downtime if a future problem arose as it went global. Here is the link to the Wall Street Journal Blog if you want to see examples of Big Data.
Big Data Consumption and Usage – Simple explanation of major uses
Although Big Data is useful to almost every industry, including small-scale and even cottage industries, some sectors already depend on It. They have long implemented or, rather, incorporated Big Data – collection and analysis – into their systems to generate different types of reports for different end uses. This article focuses on how these industries use Big Data.
What are the uses of Big Data
Businesses have long depended on the data they have to analyze trends, behavior (of goods and/or users), impacts and overall profits, etc. With the kind of data they now possess – thanks to the Internet – the computing goes beyond simple spreadsheets to provide them with many accurate results. Furthermore, Big Data enables them to perform more kinds of analysis, keeping the business healthy and profitable and always on a path to growth.
Big Data Consumption
Industries Already Using Big Data: They Started Early
A] Financial Institutions: Mainly dealing with your money, these industries rely on Big Data to check previous trends and make predictions. Early data was limited, so predictions came with a bigger margin of risk. That risk is now reduced due to access to more data. Share markets, banks and other financial institutions may also check your spending patterns to develop equations that help you retain maximum profits. The following chart will assist you in understanding how financial institutions use Big Data. It will also give you an idea of how Big Data can be used.
B] Retail Marketing: The first thing that strikes mind talking about retail is the consumption of goods – area-wise or age-wise. Yes, you can use Big Data to tell how and who are using your goods and what types of goods. Beyond that, you can improve products and even introduce new ones based on what’s succeeding. The other side of using Big Data in Retail Marketing is identifying prospects (don’t forget the online window shoppers), prospect-to-client conversion rates and techniques, client retention, and similar areas.
C] Government and Public Sector: How can we forget the government regarding data? Govt. and public sector units collect data more than any other sector. You can say they are drowning in data even as they digitize and store the data onto their servers or clouds worldwide. According to a white paper by IDC
“As government leaders across the spectrum strive to become a data-driven organization to successfully accomplish their missions, they are laying the groundwork to correlate dependencies across events and track dependencies across people, processes, and information.”
Overall, this sector gains productivity because it can track the speed and accuracy of different projects. It can then analyze the data to find better ways to improve performance. There are also other benefits, such as tracking people to provide better healthcare, employment, etc.
D] Communications Sector: This is another area where Big Data plays an important role, from acquiring customers to enhancing or, at the least, maintaining the class of service being provided to them, recovery, and bad debts, too!
Since they want their services always up and running, they can use Big Data for the above and in their own infrastructure to project potential growth over the years. They can know the bandwidth requirements, they would know about fake customers and customers no longer using their services (helping bring them back), mitigate risk in case of a sudden increase in demand and much more – virtually any part of the business you can think of.
E] Media and Entertainment Businesses: The main focus here is customer retention—sometimes more important than customer acquisition. Big Data helps determine what kind of media different users enjoy, and based on that, media houses develop better content of that type.
They focus on age groups and divide production according to the analysis results. At the same time, they have to find out what kind of advertising the different age groups engage with – instead of simply watching. Earlier, it wasn’t possible to get that much data, but with Internet Marketing Agencies and years of data compilation, they can make real-time decisions and take appropriate actions for both customers and staff. It is just the beginning. There are no limits to what you can learn. With the right kind of data in hand, you can always get accurate results.
Big Data 3 Vs – Concepts & Models
This section talks about the concepts of Big Data, using the 3 V’s mentioned by Doug Laney, a pioneer in the field of data warehousing which is considered to have initiated the field of Infonomics (Information Economics).
Data, in its huge form, was accumulated through different means, filed properly in different databases, and then dumped after some time. When the concept emerged that the more data, the easier it is to find different and relevant information using the right tools, companies started storing data longer. This is like adding new storage devices or using the cloud to store data in whatever form it was procured: documents, spreadsheets, databases, HTML, etc. It is then arranged into proper formats using tools capable of processing huge chunks of Data.
NOTE: The scope of Big Data is not limited to the data you collect and store on your premises and in the cloud. It can include data from different sources, including but not limited to items in the public domain.
The 3D Model of Big Data is based on the following V’s:
- Volume: refers to the management of data storage
- Velocity: refers to the speed of data processing
- Variety: refers to grouping data of different, seemingly unrelated data sets
The following paragraphs explain Big Data modeling by talking about each dimension (each V) in detail.
A] Volume of Big Data
When talking about Big Data, one might think of volume as a huge collection of raw information. That’s true, but volume also includes data storage costs. Important data can be stored on-premises and on the cloud, the latter being the flexible option. But do you need to store and everything?
According to a white paper released by Meta Group, as data volume increases, some data starts to look unnecessary. It also states that businesses should retain only the data they intend to use. Other data may be discarded or if the businesses are reluctant to let go of “supposedly non-important data”, they can be dumped on unused computer devices and even on tapes so that businesses do not have to pay for storing such data.
I used “supposedly unimportant data” because I, too, believe that data of any type can be required by any business in the future – sooner or later – and thus, it needs to be kept for a good amount of time before you know that the data is indeed unimportant. Personally, I dump older data to hard disks from yesteryear and sometimes on DVDs. The main computers and cloud storage hold the data I consider important and know I’ll use. Among this data too, there is a use-once kind of data that may end up on an old HDD after a few years. The above example is just for your understanding. It won’t fit the description of Big Data as the amount is pretty small compared to what the enterprises perceive as Big Data.
B] Velocity in Big Data
The speed of data processing is an important factor when talking about big data concepts. There are many websites, especially e-commerce. Google has already admitted that page load speed is essential for better rankings. Beyond rankings, speed also makes shopping more comfortable for users. The same applies to processing data for other information.
When talking about velocity, it’s important to know it goes beyond higher bandwidth. It combines readily usable data with different analysis tools. Readily usable data means some homework to create data structures that are easy to process. The next dimension- Variety- sheds further light on this.
C] Variety of Big Data
When you have loads and loads of data, it becomes important to organize it so analysis tools can process it easily. Tools also help organize data. When storing, the data can be unstructured and of any form. It is up to you to figure out its relationship with other data with you. Once you determine the relationship, you can choose appropriate tools and convert the data into the desired form for structured, sorted storage.
Summary
In other words, Big Data’s 3D Model is based on three dimensions: usable data you possess, proper data tagging, and faster processing. If you take care of these three, your data can be processed and analyzed to meet your needs.
If you wish to add anything, please comment.
Read next: What is Web Scraping?
