2.4 Most IT VPs are software people, not database people

This subsection comes as the ending essay of a section entitled Data Management is Hard, explaining all the reasons why data people are the way they are. This section is more sociological and also more critical and controversial. It takes on a management problem. 

2.4 Most IT VPs are software people, not database people

It is a strange historical accident that most VPs in the IT industry gain their experience in software development and not data management. One can imagine an alternate universe where this is not so, and organizations go decades producing horrible software until they decide to hire an experienced software architect but in the meantime their databases are all in fifth normal form and none of their data warehouses are crude copies of all the tables in the accounting module. We don’t live in this alternate universe. 

That might be a sad comment on the triumph of PR over studious hard work. But it’s more likely due to the simple fact that software teams are often at least three times the size of the database teams they work with, if there is even a database team at all. Moreover, database teams are often said to “support” software teams, which puts DBAs into a role not much higher on the corporate totem pole than customer support or desktop support people. When you’re looking around for someone to run a large team, you look for people who’ve run large teams, and not those who’ve spent their career running small “support” teams. That consideration ends up, by design, selecting for the people who’ve run large software groups for VP positions.

Outline for _Practical Data Management_

A few years back, when I found myself with a lot of time on my hands, I started writing a very practical guide to data management. It was wordy and opinionated, as you might expect from me. But it filled a gap in data management, specifically the gap between cookbooks that published schemas, like Silverston's book, and data modeling practice books, which started with the Enterprise Architecture and worked their way down to the mere details. The goal of the book was practical, at the detail level. A practitioner whether beginner or advanced should come away from the book having learned something. To do that the book needed to include bits of SQL, where relevant, but also a discussion of the theory, the practice and how the two balanced in the real world.

I find myself with time on my hands once again, although this time I'm much more comfortable and in a more stable frame of mind. The Universe seems to be telling me something, and it sounds like maybe you should be writing this down. I've avoided documentation in favor of action for most of my professional life. But once again I'm presented with an opportunity, and I find myself arguing that writing it down violates the habit of action. But it's time for new habits.

So this is, practically, a different kind of action. The tension between theory and practice is fundamental to data management, which is a new science still feeling its way through the growing pains of formalization. I think I've got something to say about it. So here's an outline of the book. I'll post chapters that have been written, and chapters as they come to fruition.

Practical Data Management

1. Introduction

1b. A note on terminology.  What do I mean by…?

2. Data Management is Hard

2.1. There are a lot of sources

2.2. Applications change over time

2.3. It takes time to get the database right

2.4. Most IT VPs are software people, not database people

3. Patterns, heuristics, logic and philosophical disagreements

3.1. Codd’s essential insight

3.2. Normal forms

3.3. Layers

3.4. Patterns of persistence

3.5. Patterns of movement

3.6. Patterns of development

3.7. Patterns of organization

3.7.1. Analysis vs. Operations

3.7.2. Customer-driven vs. Architecture-driven

3.8. Bizness Logic

3.9. Theories of Knowledge and Truth

3.10. Governance

4. The pieces, defined

4.1. Source systems

4.2. Data lakes

4.3. Operational Data Stores

4.4. Data warehouses

4.5. Data Vaults

4.6. Master Data Management

4.7. Pipelines

4.8. Logs

5. Platform choices

5.1. The value of a fixed schema

5.2. The value of schema-on-demand

5.3. Brands of DB

5.4. NoSQL choices

5.5. Pipeline tool choices

6. The build process

6.1. Can it ever be Agile?

6.2. Practical concerns

6.3. Addressing long-term concerns

6.4. The value of feedback

6.5. A simple model

7. Fallacies

7.1. “Agile”

7.2. Throwing stuff into the lake

7.3. Naming conventions

7.4. Governance

7.5. Short-term wins

7.6. Long-term architecture

 

The Semantic Debt Model

The model I present below concisely is an argument of sorts, and some of the premises hint at mathematization. Mostly the premises are logical, and chained together to form a theory. A theory can be thought of as a two-part thing, (1) a set of axioms, and (2) a model or list or set of objects of which the axioms are true. Derivations of theorems from the axioms are also then true in the model. 

All technologies are prosthetics

I've long thought that the best definition of "technology" is "a prosthetic." In this post I'm going to explain what I mean by that. I won't engage the boundaries between science, engineering and technology in this post, and I won't go into much detail about the model itself. My goal is to inflate the concept and compare it with what we know about technology.

I used to think I got this definition from McLuhan. But he mentions the specific equation once as far as I know and dances around prosthetics everywhere else. McLuhan is much less interested in what I'm going to call The Prosthetic Model than I hope every time I go back to him. This philosophical definition of technology, as just and only a prosthetic, matches my personal experience with technology. This is as a person who's profession was abstract technology-creation and -use, and an avid technology user, and as someone who's thought philosophically about technology and technology use his whole life. 

A Series

 I wrote a short post at Raybeam's Focus site.

I've got a couple of follow-on posts written laying out the argument more succinctly than I've been able to so far.  Eventually this will get turned into a longer piece, and better illustrated than most of my usual.

The centerpiece of the first post is this graphic:

It's important to keep in mind that, when we talk about "data architecture" these days we're actually talking about six different paradigms (more or less) of data modeling. 

Each of the systems in the ecosystem diagram above have their own data modeling rules. You can't just apply a Kimball-style star schema to everything. For one thing, there's more than one way to define an attribute in most dbs. You can make the rule that every attribute should be a field in a table, but someome is still going to use a _TYPE field to draw a distinction that doesn't make it into a field.

That conflict is a basic input to the modeling styles for each system in the ecosystem.  (That's one conflict.  There's others.)  Each one plays a role in that ecosystem depending more or less on which other systems it gets and receives data from.  Sometimes data migration is as simple as forcing all the attributes into one long record. Immediately we solve the "we can't find our data" problem. Eventually it happens again when the table gets to be so wide no one can see one end from the other.  And that's when the dimensions multiply like literal snowflakes in little boxes of JSON.

In any event, at least six styles of data architecture in this ecosystem. I grouped all the collection points - the applications - into one because the current prevalent model of data modeling in software architecture is to stick to a "developer driven" NoSQL model of agile development. Ideally the db contains nothing but name-value-key pairs defined by whoever happened to be writing code that day.

Anything more than that? Friction. So there's not much data architecture in modern software engineering practice. 

   

Opportunity Cost

 Opportunity cost is the basis of semantic debt.

When we think about why a data management team can't deliver, we often have trouble quantifying why.  Is it because no one did the integration?  Is the pipeline management system faulty?  There's no product management function to drive the semantic layer?

OK so ALL of that is true.  All of it.  But how do you compare the cost of hiring two new data modelers to ... what?  It seems like you're developing some kind of NULL hypothesis.

This is NOT a problem, btw, if you have experience with what two new data modelers on full-time employee can do.  How do we know that?

Consider a Replication Engine.  With five licenses and a team suite I can do what I'd need five data engineers to do manually.  The tooling is a vast improvement on having five data engineers work around the clock for a year to move an orgs dbs.  Because it automates most of that work, which was never super complicated anyway.  

REs are something people in small ETL shops have been doing forever anyway, the same way they used to build their own data catalogs.

So the opportunity cost of the semantic debt in all of those dbs - that is now quantifiable, at least in part.  Before a replication engine if you'd told me that it'd cost $1 million just to move the dbs, and a year's effort, before I would even see the least bit of benefit, I wouldn't have blinked an eye.  That is why semantic layer construction is so hard - by the time the pipelines are built everyone is exhausted.

But that $1million is now fungible software talent and I don't have to wait a year.  I can do it in a week.  Is it worth it to me to get a year's worth of benefits in a week?  Yes.  Especially if my metric is something easy like "billions of records/day."

The semantic debt of your current legacy databases is the last estimate you got from your usual database management provider (internal or external) when you asked how much it would cost to make the data warehouse actually useful.     

One of the laws of data management

Quick thought: The longer you use old tools to glue old systems together, the less likely you'll be able to integrate or manage new problems.  (There's a bit of a leap there, but not much.)

One of those variables needs to change over time for your operations to stay healthy.  You stick to one toolset for a decade and you won't be able to hire anyone with less than a decade's experience in that toolset to manage it. 

Which you may choose to do, for lots of reasons.  But you're stuck in a decades-old toolset then.  Do you have any idea what you can't do because of your current toolset?