How long will this data modeling project take?

 

I was working with a Data Engineering executive yesterday on estimating the size of a data modeling effort. Calling her an "engineering manager" doesn't get to her depth, so "executive" will have to do. She's a serious software engineer who knows her stuff, and we've both worked on data warehouse projects for years at the same company, and we've worked with the same people off and on for at least six years. But she's still a little mystified at how to estimate a data modeling effort, and even though I've written thousands upon thousands of complex sentences, that Gemini can thankfully distill down to 25 words... well she just hasn't had time to read them all.

4.2 The Data Lake

 4.2 The Data Lake

In the last five years or so Data Lake projects have been sold as a game-changing new type of fauna in the data management ecosystem, a system that is easy- and fast-to-build while providing a lot of short- and long-term value. The Data Lake is not new, however, and solutions of this type have a long and checkered history in data management. They’re also the best way for a beginner Data Management practitioner to start building systems. They’re an educational opportunity to make a series of useful and forgivable mistakes that create a ton of long-term value for their owners. Data Lakes are the critical early stage in the evolution of the data management ecosystem. They provide a laboratory for developing the business logic that glues business processes together, and as such they’re key to all subsequent components of the data management ecosystem. 

In this section we’ll lay out the context in which Data Lakes are built, and discuss some of the strategies used to build them. We’ll also walk through what we expect from a Data Lake, particularly the evolutionary function they play and the knowledge they create for us. Finally, we’ll talk about ways people end up wrecking their Data Lakes.

2.3 It takes time to get the data model right

More from the chapter entitled "Data Management is hard," explaining why people often ignore good data management practices in favor of the easy and expedient.

2.3 It takes time to get the data model right

There are people who find data modeling enjoyable and relaxing. While to all outward appearances the task may involve a lot of yelling and angry erasing of whiteboards, the author (for example) thinks that the process of doing data modeling is almost always exploratory, creative, bounded by the mysterious tyrannies of implementation and comprehension, and exhilarating. We’ve heard stories about data modelers in the old days who took years to reach their end state, and while that doesn’t sound like fun it would certainly be great to do what amounted to professional metaphysics all day. 

But there are many people who don’t enjoy abstraction, or who may believe they enjoy it but aren’t suited for it by training or temperament. It may also be that work on a data model is artificially truncated by philistine business people or technical managers who think the perfect is the enemy of the good and want to get on with good business. In all those sadly-prevalent cases it’s often the case that what you might call “folk” data modeling becomes the norm. The notion of the “folk” data model is worth exploring for a minute, because it’s key to understanding this difficulty. One of the reasons data management is hard is because it takes time - and thought, and effort - to get the data model right.

4.3 How and why to build an Operational Data Store

Nobody knows how to build an ODS, or why you might build one. I've had many many many arguments with "data warehouse developers" over the years who assume that the Kimball-style Analytics warehouse is the only kind of warehouse facility there is. This mistake results in bad systems design, because the ODS solves a specific set of use-cases that a Kimball-style system just simply can't.

In this chapter, which comes after a discussion of Data Lakes and source systems, I explain how and why to build an ODS.

4.3 How and why to build an Operational Data Store

In this section we’ll first discuss why the Operational Data Store or ODS, also known as an Inmon-style warehouse, is often negatively compared with the Kimball-style warehouse. Then we’ll talk about the use-cases satisfied by the ODS, and the steps for constructing one.

The Operational Data Store is the next obvious logical evolutionary step in data management systems development, after an organization has explored what a Data Lake can do. The ODS is also the most misunderstood system in the evolutionary process, from a development standpoint, and experienced developers are rare. They’ve fallen out of favor in recent years, in part because it requires what appears to be a more skilled data modeler than, say, Star Schema-based data warehouses or “Kimball warehouses,” or the Data Lake. An ODS also appears to fall short in cost-benefit comparisons with classic Kimball warehouses. 

Ad Hominems and Data Modeling: When to use the EAV

Data modeling patterns are like logical fallacies.  Almost everyone knows that ad hominem arguments are bad arguments: Attacking a person's argument because of seemingly arbitrary attributes that person holds is bad judgment.  If I say "Politician X makes the case that A, but since X has a really bad combover its clear A cannot be true" we've committed the ad hominem fallacy.  X's bad combover has no relevance to A, and because it has no relevance using X's combover to attack argument A is a logical fallacy.