It’s not much of an exaggeration to say that, for many businesses, data is their most valuable asset. This accumulated information, hailing from various different sources, can be used by companies to answer questions that take the guesswork out of smart decision-making. Properly deployed in the right way, structured data is no longer mere information, but rather knowledge. That’s a big difference.
However, as the amount of data has exploded, it’s become increasingly important to have the proper way of storing, aggregating it, and analyzing it. This is where the idea of a data warehouse enters the picture.
In short, a data warehouse is a centralized repository for data that’s connected to (usually) multiple data streams -- including relational databases, transactional systems, and others.
A typical data warehouse works like this: To start with, a process called Extract Transform Load (ETL) processes and integrates data from multiple channels in the right format to be used by the data warehouse. After this, the data moves on to Data Marts, organized according to divisions such as the specific department that it belongs to. Finally, tools referred to as Business Intelligence (BI) assist people who need to access this data to do so. This involves periodic reports, multi-dimensional data analysis, and live dashboards.
Origins of a data warehouse
The term “big data” has only been in widespread usage for about the last decade. But the forward-looking inventors of the data warehouse were prescient enough to see the need for it much further back than that. In 1988, then-IBM researchers Barry Devlin and Paul Murphy called for the creation of a nonvolatile, integrated, subject-oriented, time-variant data collection tool they could employ to help management make data-driven decisions.
In some cases, this required only simple reporting based on data already collected. In others, it meant staging “what-if” simulations to look at the ramifications of certain courses of action that were available. What they required, they suggested, was a tool for bringing together structured, timestamped data from myriad diverse sources to serve as an ongoing record of both historical and current information.
We now live in a world in which the value -- and quantity -- of data has increased exponentially. Data, as the saying goes, is the “new oil,” and Devlin and Murphy’s call for the creation of the first data warehouse is now needed more than ever.
It’s not just the amount of data that has increased in the years since 1988, either. There are new sources and types of data that could scarcely have been imagined back then. Modern data warehouses have to be able to take in often semi-structured data and make it fit for purpose, along with combining it with other datasets along the way. By doing this, the modern data warehouse can live up to the expectations that its creators had for it more than 30 years ago.
Different ways of storing data
A data warehouse isn’t the only way of storing data. The most common, and basic, storage approach involves a database. This is a straightforward tool for recording data. However, while a database may be useful for carrying out very limited analysis of data, it is not able to offer the same rapid analytics and storage of historical data taken from multiple sources of historical data.
Another widely employed method of storing data is a data lake. The main difference between a data warehouse and a data lake is that data lakes store huge pools of raw data with no particular structure and no immediate purpose. It’s easier to establish and load data into a data lake in the same way that it’s easier to tidy by throwing everything into a closet without sorting it out first.
The downside of a data lake is that it’s harder to extract insights from unless you are a data scientist with the right tools for the job. A data lake has its uses, but it is far less accessible to non-data science experts -- meaning those employees who simply want a tool that “just works.”
Protecting your data assets
If you had a physical warehouse full of stock or equipment that represented significant assets for a business, you would make sure that it was well-protected. That would probably mean investing in the right security cameras and night watchmen, among other measures. Similarly, a data warehouse needs to be protected -- even if the way that you do this is a little bit different.
Whether your data warehouse is sited on-premises, in a hybrid environment, or on the cloud, there are different ways organizations can protect them. An essential step is carrying out proper encryption and data masking. This means that, even if data to somehow be stolen, it would be unreadable (and therefore valueless) to the thieves.
Just as important is making sure that thieves never get inside to begin with. To do this, organizations should deploy tools such as database activity monitoring and a database firewall. These will monitor data warehouses on a constant, 24/7 basis, and provide immediate alerts should anything out of the ordinary happen.
Companies use data warehouses so that they can make smarter decisions. Protecting those same data warehouses is about the smartest decision they can possibly make.
