Harnessing Data to Create Profits

Practically every organization has vast amounts of “dark data” in the form of weblogs, machine logs, and logs from sensors on everything from oil rigs, to turbine engines, to hospital patients.  The question is: Do you have a plan to create business value out of your dark data?

This is the age of data-driven digital transformation.  Your organization is looking for ways to engage more closely with customers, improve decision making, improve the operational efficiency of manufacturing, or deliver better patient healthcare outcomes.  You probably have a great deal of the data you need to accomplish this in-house.

We define “dark data” as large, often unstructured, data sets collected and stored from both internal and external sources that is not currently being used to create insights that deliver business value.  Dark data can be harnessed to detect critical patterns in customer buying, fraud in financial transactions, issues in supply chains, health care industry adoption patterns, product life-cycle issues, and more.

The challenge comes in being able to access and use data to deliver useful insights that drive business value.  Dark data by its nature is large, unstructured, and very hard to understand.

It’s not just data volume; data complexity is an even larger problem.  Much of this dark data has not structure or has an unknown structure, making it even harder to derive useful business insights.

Harnessing these extremely large data sets will drive new waves of productivity and innovation.  According to research by MGI and McKinsey's Business Technology Office, companies will use data to create value in numerous ways:

• Data drives profitability by making information transparent and usable to an ever-greater degree.
• As organizations create and store increasing amounts of transactional data in digital form, they can collect accurate and detailed performance data on everything exposing variability and ramping up performance.
• Data can be harnessed to detect critical patterns in customer buying, fraud in financial transactions, issues in supply chains, health care industry adoption patterns, product life-cycle issues, and so on.

The Dark Side of Data

With all the promise of data, there is a downside. These data types are characterized not just by the large volumes, but also by to their velocity, variety and variability. “Data drifting” commonly used to depict the fluctuation in the format, the pace, and the content of data in these new data types posing a serious challenge to organizations that are trying to collect and makes sense of the new data. In a survey done in 2016, 25 percent of respondents said they discard data to get analytic insights because they couldn’t scale to process the data being collected. The problem everybody faces is that when data “drifts’ that can cause data integrations to break and a sudden shut-down of a business process.

Not only are the existing data management architectures inadequate to handle these files, many organization are experiencing a shortage of analytical and managerial talent necessary to drive value from all this data. McKinsey says that The United States alone faces a shortage of people with deep analytical skills.

Simply put, making sense of your data, much less transforming it into tools that can allow you to make better decisions based on data, can become an impossible task. In numbers, less than 0.5% of the world’s data is actually being analyzed. Enterprise are data rich, but information poor.

The Promise of Machine Learning:

Machine learning promises to bring to the market products such as cars that are self-driving, and is already providing a complex understanding data collected about human genome that is being used to provide highly personalized healthcare.

Companies have been using artificial intelligence tools since the late ‘90s to analyze information. Large credit card processors have been using artificial intelligence to detect and prevent fraud for decades.  Business Intelligence companies utilize algorithms to provide business owners with reporting, dashboards and other visual representation of their data.

However, these tasks can be achieved only on data that had already be identified, cleansed, prepared and cleaned – in other words, data that had already been discovered and brought into the light.

Can Machine Learning be used to shed light on dark data? Machines have the processing power to rake through vast amounts of data utilizing mathematical and other tools to turn data into meaningful information, so why not? 

Although it sounds simple, this area is still in its infancy. Companies are struggling to adapt their data management architectures to handle the volume and velocity.

Building a Plan for Dark Data:

Providing trusted relevant, and timely data to fuel your transformative business initiatives is the goal.  So, having an intelligent data discovery tool that can help find data and understand the proliferation of data across your organization is the essential first step for solving the problem of dark data.

On-boarding new data sets into different systems, and putting them to work in your operations is the next crucial step for driving value from your data. Many organizations, are seeking ways to accelerate the on-boarding process but are struggling to adapt to the constant changes and variations of these types of data.

Delivering Business Value from Dark Data

IT organizations are struggling with the challenge of getting value out of dark data solving this challenge will require new thinking. The old ways of managing dark data simply will not scale.  To meet the needs of data-driven digital transformation is going to take something more.  You need the power of Artificial Intelligence and machine learning tools, to automate the understanding, management, and the extraction of valuable information from dark data.

It’s time to think differently about your approach to dark data.  Do you have a plan?