One reason data management is hard is the sheer diversity of sources data management practitioners are expected to rationalize. Consider a typical Customer-360 program. The goal of a Customer-360 program is to collect all of the data in a given enterprise about a customer, organize it into a matrix of some sort, and make it available for eventual consumers throughout the organization. Those sources often include:
In the last five years or so Data Lake projects have been sold as a game-changing new type of fauna in the data management ecosystem, a system that is easy- and fast-to-build while providing a lot of short- and long-term value. The Data Lake is not new, however, and solutions of this type have a long and checkered history in data management. They’re also the best way for a beginner Data Management practitioner to start building systems. They’re an educational opportunity to make a series of useful and forgivable mistakes that create a ton of long-term value for their owners. Data Lakes are the critical early stage in the evolution of the data management ecosystem. They provide a laboratory for developing the business logic that glues business processes together, and as such they’re key to all subsequent components of the data management ecosystem.
In this section we’ll lay out the context in which Data Lakes are built, and discuss some of the strategies used to build them. We’ll also walk through what we expect from a Data Lake, particularly the evolutionary function they play and the knowledge they create for us. Finally, we’ll talk about ways people end up wrecking their Data Lakes.
More from the chapter entitled "Data Management is hard," explaining why people often ignore good data management practices in favor of the easy and expedient.
There are people who find data modeling enjoyable and relaxing. While to all outward appearances the task may involve a lot of yelling and angry erasing of whiteboards, the author (for example) thinks that the process of doing data modeling is almost always exploratory, creative, bounded by the mysterious tyrannies of implementation and comprehension, and exhilarating. We’ve heard stories about data modelers in the old days who took years to reach their end state, and while that doesn’t sound like fun it would certainly be great to do what amounted to professional metaphysics all day.
But there are many people who don’t enjoy abstraction, or who may believe they enjoy it but aren’t suited for it by training or temperament. It may also be that work on a data model is artificially truncated by philistine business people or technical managers who think the perfect is the enemy of the good and want to get on with good business. In all those sadly-prevalent cases it’s often the case that what you might call “folk” data modeling becomes the norm. The notion of the “folk” data model is worth exploring for a minute, because it’s key to understanding this difficulty. One of the reasons data management is hard is because it takes time - and thought, and effort - to get the data model right.
A good friend of mine recently consulted on performance measurement for a government ministry that combined the two seemingly unrelated portfolios of Tourism and Economic Development. Their businesses couldn’t have a more divergent customer base. The Tourism portfolio focused on direct to consumer marketing: Families flying in from all over, attracted by the local sights, the beaches, festivals and hotel deals. The Economic Development portfolio was B2B, and worked to attract companies large and small to the area as well as fostering and sponsoring local startups. The metrics used to measure performance across these disparate portfolios were necessarily different, the customers were very different, and the mandates were definitely different.
Nobody knows how to build an ODS, or why you might build one. I've had many many many arguments with "data warehouse developers" over the years who assume that the Kimball-style Analytics warehouse is the only kind of warehouse facility there is. This mistake results in bad systems design, because the ODS solves a specific set of use-cases that a Kimball-style system just simply can't.
In this chapter, which comes after a discussion of Data Lakes and source systems, I explain how and why to build an ODS.
In this section we’ll first discuss why the Operational Data Store or ODS, also known as an Inmon-style warehouse, is often negatively compared with the Kimball-style warehouse. Then we’ll talk about the use-cases satisfied by the ODS, and the steps for constructing one.
The Operational Data Store is the next obvious logical evolutionary step in data management systems development, after an organization has explored what a Data Lake can do. The ODS is also the most misunderstood system in the evolutionary process, from a development standpoint, and experienced developers are rare. They’ve fallen out of favor in recent years, in part because it requires what appears to be a more skilled data modeler than, say, Star Schema-based data warehouses or “Kimball warehouses,” or the Data Lake. An ODS also appears to fall short in cost-benefit comparisons with classic Kimball warehouses.
It is a strange historical accident that most VPs in the IT industry gain their experience in software development and not data management. One can imagine an alternate universe where this is not so, and organizations go decades producing horrible software until they decide to hire an experienced software architect but in the meantime their databases are all in fifth normal form and none of their data warehouses are crude copies of all the tables in the accounting module. We don’t live in this alternate universe.
That might be a sad comment on the triumph of PR over studious hard work. But it’s more likely due to the simple fact that software teams are often at least three times the size of the database teams they work with, if there is even a database team at all. Moreover, database teams are often said to “support” software teams, which puts DBAs into a role not much higher on the corporate totem pole than customer support or desktop support people. When you’re looking around for someone to run a large team, you look for people who’ve run large teams, and not those who’ve spent their career running small “support” teams. That consideration ends up, by design, selecting for the people who’ve run large software groups for VP positions.
A few years back, when I found myself with a lot of time on my hands, I started writing a very practical guide to data management. It was wordy and opinionated, as you might expect from me. But it filled a gap in data management, specifically the gap between cookbooks that published schemas, like Silverston's book, and data modeling practice books, which started with the Enterprise Architecture and worked their way down to the mere details. The goal of the book was practical, at the detail level. A practitioner whether beginner or advanced should come away from the book having learned something. To do that the book needed to include bits of SQL, where relevant, but also a discussion of the theory, the practice and how the two balanced in the real world.
I find myself with time on my hands once again, although this time I'm much more comfortable and in a more stable frame of mind. The Universe seems to be telling me something, and it sounds like maybe you should be writing this down. I've avoided documentation in favor of action for most of my professional life. But once again I'm presented with an opportunity, and I find myself arguing that writing it down violates the habit of action. But it's time for new habits.
So this is, practically, a different kind of action. The tension between theory and practice is fundamental to data management, which is a new science still feeling its way through the growing pains of formalization. I think I've got something to say about it. So here's an outline of the book. I'll post chapters that have been written, and chapters as they come to fruition.
1b. A note on terminology. What do I mean by…?
2.1. There are a lot of sources
2.2. Applications change over time2.3. It takes time to get the database right2.4. Most IT VPs are software people, not database people
3.1. Codd’s essential insight
3.2. Normal forms3.3. Layers3.4. Patterns of persistence3.5. Patterns of movement3.6. Patterns of development3.7. Patterns of organization
3.7.1. Analysis vs. Operations3.7.2. Customer-driven vs. Architecture-driven
3.8. Bizness Logic3.9. Theories of Knowledge and Truth3.10. Governance
4.1. Source systems
4.2. Data lakes4.3. Operational Data Stores4.4. Data warehouses4.5. Data Vaults4.6. Master Data Management4.7. Pipelines4.8. Logs
5.1. The value of a fixed schema
5.2. The value of schema-on-demand5.3. Brands of DB5.4. NoSQL choices5.5. Pipeline tool choices
6.1. Can it ever be Agile?
6.2. Practical concerns6.3. Addressing long-term concerns6.4. The value of feedback6.5. A simple model
7.1. “Agile”7.2. Throwing stuff into the lake7.3. Naming conventions7.4. Governance7.5. Short-term wins7.6. Long-term architecture
In the twenty-plus years I've worked in data management I've spent a lot of time having some version of the following conversati...
