How many layers are we up to now? One-Tier vs. Three-Tier vs. Medallion vs. Functional Data Warehouse Architectures

Formal, intentional Data Architectures are rare in the wild and for data warehousing they're even rarer. A few textbooks describe principles for managing data in a data warehouse, and there's of course the half-baked "It's all just Kimball!" conventional wisdom. But there's not, as far as I know, a reliable cheat sheet that can help you figure out what your data architecture should look like. And no, the Medallion architecture doesn't count as "reliable."

All this is of course extremely relevant to how you implement AI, however that particular combination of machine-learning algorithms and hype is configured in the year you actually read this post.

This is looong, 8000+ words, so after the jump a table of contents:

How long will this data modeling project take?

 

I was working with a Data Engineering executive yesterday on estimating the size of a data modeling effort. Calling her an "engineering manager" doesn't get to her depth, so "executive" will have to do. She's a serious software engineer who knows her stuff, and we've both worked on data warehouse projects for years at the same company, and we've worked with the same people off and on for at least six years. But she's still a little mystified at how to estimate a data modeling effort, and even though I've written thousands upon thousands of complex sentences, that Gemini can thankfully distill down to 25 words... well she just hasn't had time to read them all.

The ABV

The Attention-Based View of how organizations function was developed in the late 90s by William Ocasio, a Management prof at the University of Illinois. The idea is simple but powerful: An organization acts on what it pays attention to, and so understanding how attention is distributed and used throughout the group will provide insight into how the organization works. 

A survey paper a couple of years ago laid out some of the investigation that's happened since Ocasio published the original piece. It's a profitable framework from a research standpoint. The concept of "attention" is pretty intuitively immutable, but there's obviously a lot of detail to understand. 

If you're an MBA who's studied organizational behavior this might be old news. When I read Ocasio's paper sometime in the 00s it completely changed my way of looking at how data flows and should flow in an org, how BI functions and how the delivery mechanism you choose for that function matters. 

"Important points should be discussed by humans and not put into a dashboard" is clear enough guidance, as far as it goes. But how does the anomaly in the "data" get categorized properly such that it can be reliably surfaced, hypothesized, analyzed and explained? Getting a datapoint into the attention span of senior executives is hard. Finding the datapoints you want to get there, also hard. But unless you coordinate the effort, balancing the "stovepipe" with the "fabric" approaches around the blank spots on the map, you're obviously feeding the wrong data to the wrong people.

The Marketing Analytics Apocalypse

 

The Hertzprung-Russell diagram of stellar evolution

When I started this blog in 2018 I was unemployed, an immigrant in the US during Trump's first term, a data architect who didn't have any cloud experience and who didn't think it was probably possible to actually do data architecture successfully as an FTE. My family was doing their own thing, and all I could do was wait out the world and keep myself busy.

4. The data management ecosystem

 4. The data management ecosystem

There are only a few components to the modern data management ecosystem, which is displayed in the diagram below. We’ve plotted these systems on two axes, one showing the latency of the data managed by the system and the other showing the complexity, expressed in terms of the number of sources a particular kind of system organizes. These systems each organize data in a way that’s particularly suited to a class of tasks or workflows. Most of these systems don’t create data, which generally happens in the bottom left hand corner in the Services/Applications space. Instead, they rearrange and organize data. Application and service databases serve as the supplier of raw materials for the information supply chain, and they’re usually designated in this work as the “source systems.” Some systems manufacture data as a byproduct of their basic operation, such as Master Data Management systems that function as curation tools for a small percentage of the data they manage. But for the most part the job of the data management system, when it isn’t a source system, is synthesis and organization.

3.6 Patterns of Organization

3.6 Patterns of organization

There are a couple of simple patterns you should consider when organizing your data management efforts for maximum impact. In this section we’ll discuss some of those patterns of organization. We’ll look at a couple of different organizational models, and point to some fundamental properties of those models that help make teams effective. 

4.6 Master Data Management

4.6 Master Data Management

It’s a basic fact of the modern data ecosystem that critical data about critical entities will be duplicated and most importantly different simply because that data is created in different source systems. This happens in the simple case where a customer interacts with multiple applications, and each application creates its own CUSTOMER record. In the more complex case, two applications are downstream of a third, and copies of source records are sent to the downstream systems where they inevitably get updated or supplemented. Master data management or MDM is the process of creating entities and resynching records with the real-life thing. MDM is in one sense a brute-force solution to the governance problems caused by incompatible data models and inconsistent form validation. In another sense, MDM is the pragmatic connection between a data management ecosystem and the real world. And in a third sense, it’s one of those critical layers we see in good data management practice.

In this section we’ll discuss the process of mastering, which is basic to all pipeline development, and how that process eventually gets turned into the components of an MDM system. We’ll also walk through some of the use-cases, including the specific kinds of entities commonly managed in an MDM system. Finally, we’ll explain how MDMs should be integrated into the rest of their ecosystem.