Is a data warehouse still relevant today?

Is a data warehouse still relevant today, or is it better to have multiple data warehouses?

As of 2023, a data warehouse may seem unnecessary. After all, there are various tools available that allow users to retrieve, combine, and export data themselves. A data warehouse is expensive, outdated, hard to understand and manage. And also time-consuming. That is what often goes through managers' minds. But are these assumptions actually correct? And what options do healthcare organizations have to turn raw data into usable information? In this blog, we explore several aspects of this.

Hunger for Data

Let’s start with the users — the people in the organization who need data to gain insights for decision-making or to validate whether previous decisions have had the intended effect. The skill levels of these employees can vary widely. Yet these users all have one thing in common: they want more data, more insight, and more reports than they currently have. And that demand finds an outlet. If the centrally supported information provision is inadequate, users are resourceful enough to use alternative tools. This poses a risk. It creates an alternative data stream outside the view of those responsible within the organization. The hunger for data is not the problem — the self-found solutions can lead to problems. Lots of manual work, no documentation, multiple versions of the truth, no data governance, no validation, and minimal reuse. And with Excel sheets being shared via email, the risk of a data leak is also present.

Self-Service Tools

In recent years, low-threshold reporting tools have evolved into products that can also be used by non-BI professionals. As a result, employees can be empowered to import, process, and convert data into reports themselves. Data can be retrieved directly from source systems, with just access permissions required. Instead of a central department being responsible for creating reports, many more employees can now create reports without having to wait for a spot on the development agenda.

Data Warehouse: A Look Back

But why did we need a data warehouse in the first place? For several reasons:

1. Source systems often have different ways of retrieving data.
2. Due to performance reasons, source systems can’t be used during the day to run heavy data queries.
3. Data from different systems cannot be easily combined.
4. Data cleansing, enrichment, and calculations are required.

A traditional data warehouse periodically reads from source systems, cleans the data according to agreed rules, adds context to the raw data by performing validated calculations, and prepares this data in data marts. These data marts then serve as the basis for various information products.

If we look at a tool like Power BI, we see that Power BI is quite capable of retrieving data from source systems, performing calculations, cleansing and enriching data, and even creating visual reports and dashboards all within the same tool. At first glance, this seems to eliminate the need for a data warehouse. Looking deeper, however, we see that a data warehouse still holds additional value:

1. Keeping track of data that no longer exists in the source systems or has been changed, with the original value no longer present in the source.
2. A data warehouse increasingly contains unstructured data.
3. Calculations can be so complex that they are not easily replicated in a self-service tool.
4. The definitions of calculations are fixed in the data warehouse, making it a single source of truth.

This shows that while the technical aspects of a data warehouse can be taken over by a self-service tool, not all functional features can be replicated sufficiently to phase out the data warehouse.

Data Warehouse or Reporting Environment with a Software Package

Increasingly, software packages are delivered with a separate reporting environment or data warehouse that allows for reporting on the underlying data. The advantage here is that the supplier maintains the reports and definitions. But this advantage can also be a disadvantage. A report may be unusable for the organization unless it is customized in one or more ways. Even then, a new environment is created where reports run, often making it complicated for users to know where to find the report they need.

A Solution

Of course, the ideal situation can differ per organization. Still, there are several elements that can be set up in almost any organization for improved information management.

1. Map all users and group them. Identify the information needs of these user groups. Some users are well served with quality standard reports and do not need to work with data themselves.
2. Ensure a solid Enterprise Data Warehouse with functionally rich data marts. These data marts include cleansing and enrichment and contain uniformly applied calculations. The data marts are documented and centrally maintained by the BI department. Also assess per data mart which is the most valuable source to retrieve the required data from. If a software vendor already provides a data warehouse with the package, it may be useful to use that as a source. Weigh the pros and cons.
3. Develop high-quality standard reports in collaboration with internal user groups. It’s advisable to appoint one or more product owners. They manage the development agenda together with the development team. The product owner(s) represent the information needs of the entire organization and are also responsible for prioritization.
4. Implement self-service and include data governance for BI products. Do this per user group and train these groups in using the centrally supported Self-Service BI tool. This provides freedom for users, but also responsibility.
5. Ensure users of data marts and reports are aware of all information products. This requires not just documentation but also a product catalog and/or a clear information portal. The more users need to search for information, the greater the chance of parallel data streams emerging.

Setting up an enterprise data warehouse, data governance, and self-service is a complex process. The consultants at ilionx have extensive experience with this in healthcare institutions of various sizes. Contact us for a free consultation to see what suits your situation best.

Erik Schmitz - ilionx