Big data is a term that describes the large amount of data – both structured and unstructured – that a business can access on a day-to-day basis. Like the term “the cloud,” the term big data can seem complex. In reality, big data as a concept is really quite simple. It’s about understanding the methods/tools companies/individuals use to make sense of large amounts of data.
In this article, we’ll break down the term “big data” into more meaningful, fundamental concepts. We’ll define the term, explore the benefits of big data, and discuss a range of big data challenges.
What is Big Data?
What is big data?” is a question that must be answered before we can begin to explore uses, benefits, and challenges of big data. One simple big data definition is: data that is so large, fast, or complex that it’s nearly impossible to capture, process, and make sense of using traditional methods.
For those in the data analysis industry, there is an expanded definition of big data that is called “the 3 Vs of Big Data.”
3 Vs of Big Data
The term “big data” was increasingly adopted by information and data management professionals in the early 2000s. In these earlier years of big data, industry analyst Doug Laney articulated what is now considered to be a somewhat mainstream definition of big data as the three V’s: volume, velocity, and variety.
The volume of data that is swirling around us can be almost incomprehensible. Some of it is useless, but if harnessed right, much of it can be incredibly powerful. An organization can collect data from point-of-sale transactions, Internet of Things devices, social media engagement, industrial equipment, etc. Historically, storing large volumes of data was problematic. But not anymore. Data management platforms like Hadoop make storing huge amounts of data possible.
Velocity of data refers to how fast data moves. To make effective use of data, it often must be captured and analyzed in real time. Think about how sensors on a piece of equipment or location information from cell phones are constantly being produced. A company needs to analyze the sensor data within minutes to stop production if there is an issue. It’s often no use to analyze the data after the fact.
Finally, there is a wide variety of data available to be captured and analyzed. Data comes in very different forms and capturing and integrating it is no easy task. Traditional databases used structured numerical data. But today, text data, visual data, and audio data is also available.
Big Data Examples
There are an infinite number of big data examples available that can help us better grasp the concept.
Let’s use an industrial manufacturing example to understand big data on a deeper level. Imagine General Electric is operating a piece of heavy machinery in a plant. One part of the machine costs $1M to replace. If that part fails, the whole plant shuts down for at least a day, causing substantial revenue losses. The part involves rotation and can rotate between 2 billion and 4 billion times before it needs to be replaced.
The part has a sensor on it, and with every rotation, data about the health of the part is captured and can be analyzed. Now, it’s not as easy as a flashing red light that says the part is about to break. However, if you look at the sensor data from multiple parts and analyze the information using complex algorithms, concerning patterns can be identified.
Imagine that, after 2 billion rotations, GE replaces each part. In some cases, the part is being replaced only halfway through its life resulting in the company spending twice as much money as they need to. On the other extreme, imagine that GE lets the machine run until it breaks. Now the whole plant shuts down.
Effective use of big data in this context would mean analyzing sensor data in real time each week and replacing the part a few weeks before it might break. This saves both time and money for GE.
This is just one of many big data examples.
Benefits of Big Data
The benefits of big data are immense. There are four types of benefits for companies that use big data well:
- Direct cost reduction
- Time reduction
- Optimizing product or service offerings
- Smarter decision-making
An organization harnessing big data effectively might be able to establish the underlying cause of an outage in its website or app in minutes or even seconds. Armed with a deep understanding of customer buying habits, it might create a customized coupon at the point of sale. Done right, this will result in additional incremental revenue. The company that perfectly analyzes website usage can create pop-up offers that seem to read the mind of the user. Immediate and almost real-time fraud detection is another benefit of big data.
Big Data Challenges
Even with all its benefits, there are also challenges for big data. Seven common big data challenges organizations must overcome:
-
Insufficient Understanding and Acceptance
Big data is an emerging technology and set of tools. Many don’t yet understand its power. Clear communication of its benefits can help with this.
-
Confusing Variety of Big Data Technologies
If, as a data professional, you use lingo or try to introduce too many tools and technologies, you’ll lose support. Organizations should prove the value of big data with a small number of core big data technologies before introducing new ones.
-
Cost
While the benefits of big data are immense, the tools and licenses required to access these benefits can be expensive. Sometimes, the cost of a tool or subscription might not justify its benefits.
-
Data Complexity and Quality
Huge potential in big data may exist for your organization, but that may require capturing information from multiple sources, integrating it, and analyzing it. Sometimes data is duplicated or contradictory. This can make utilizing the data you have tricky.
-
Security
It’s easy to imagine how security might be an issue in a multitude of ways when you are analyzing real-time usage data of consumer gadgets. It’s key to make sure your big data adoption project considers security early on in the discussion of project scope and priorities.
-
Analytical Challenges
Integrating data from several sources in real-time is not easy. And keep in mind, your internal data may only be half of the story. You may have a great handle on your historical customer buying behavior and what items pair with what. But what happens when an emerging trend appears on Instagram, causing consumer preferences to change? Proper data analytics requires models that consider and integrate multiple factors, both internal and external.
-
Upscaling
The whole point of big data is to deal intelligently with huge amounts of information. But companies need to design a data architecture that also allows them to scale up their insights associated with that data. People need to be able to access the insights, communicate, and act on them in a reasonable amount of time.
[globalcta id="CTA-CorpTraining-v1"]
Conclusion
Though not without its challenges, we believe the benefits of big data to be greater than the cost of overcoming any issues faced. Big data helps organizations generate creative new growth opportunities and find ways to save money and time. The volume, velocity, and variety of data available to companies only continues to increase. Organizations that invest in big data, both in terms of people who know how to handle big data and tools to analyze it, are more likely to succeed in the marketplace.
Additional Reading:
- AI Replacing Jobs: Should Consultants Be Worried?
- Data Analytics: Do Businesses Really Need It?
- Becoming Data Driven
- Pyramid Principle Applied