How to use the Semantic Debt Problem Matrix to solve problems

So how do you use the giant matrix I put up yesterday?

An analytical matrix is a model of the world.  This idea shouldn't be new to anyone who reads stuff like this, but let me explain a little further.

(I'm going to switch the names around here so anyone who's followed this closely might get a little confused for a moment.  I don't think anyone is actually following any of this that closely anyway, but the names of the axes have been bugging me for a while and I don't think they fit.  So I'm going to change them to something more appropriate.)

What I've proposed is that you can classify all of your data management problems using a 2X3 matrix.  The Y axis of the matrix is the Debt Category, which breaks down as follows:

 
My claim is: If you look around your data management shop you'll find you've generally got people working on two types of problems. 

1. They're trying to solve Equivalence problems: They're writing reports or building pipelines or ETL processes that combine different schemas (Type 1) or resolve different natural keys (Type 2).  
2. They're trying to solve Coverage problems: They're trying to figure out the schema for a set of files or a storage system (or trying to enforce governance rules on the same), or trying to figure out how to store something in an existing schema.

In a more traditional data management team this work is done by BI analysts and ETL developers.  In a data science group individual data scientists will wrestle with all four problems.  In a large enterprise group there may be a team of business analysts working on one problem at a time, and then handing off what they've found to specialists.

But - and this is very important! - each team will only ever have one of these four types of problem.  They're either working on how to glue two sets of data together, or they're trying to figure out what the data means.  Sometimes you need to do the one before the other, but this is all there is.

By the way, there's no real division of labor here.  Its not as if BI people are hired to work on Equivalence debts and ETL people work on Coverage debts, because they don't.  In the course of building a report a BI analyst will have to figure out what a schema means, often by analyzing apparent natural keys, and then figure out how to merge the schema with something they've already got.  The ETL people do exactly the same thing.  The primary difference between BI and ETL is that one person's output is a visualization (which might just be a table) and the other is a table or file.  If you're a data scientist then you're doing both jobs - but hey, they're calling you a scientist so that's worth something, right?

So there's only ever four kinds of problem that data management people wrestle with.

The second claim is that there's only three ways the data management team discovers they've got these problems.  They can be surprised by them (duh!), in which case they've got a Sudden failure.  They can know about them because they're a consequence of the design or scope of the systems they work with, in which case they're Immediate failures.  Or these failures can build up gradually, where no one specific failure is significant enough to be a Sudden failure but over time enough debt builds up to cause a problem.  These are Gradual failures.  These Failure Modes are our X axis.

So this is it, is what I'm claiming.  You can design your fancy stateless ETL solution around this matrix, hire people who specialize in one kind of debt or another, or go out and make a lot of consulting cash analyzing an organization's data management plans using this matrix.

But its not enough for Erin and people like her, which is to say everyone who doesn't live a life of enforced leisure such as myself.  So the next step is to cross-reference the matrix with the problems it solves, as follows:

Now that we've got the dimensions for the cube and we've filled in the cells with corresponding problems, we can flip the matrix around and figure out what to do with our problems.  To use the Index, you look up your problem, determine which Debt Category and Failure Mode its an example of, and then apply the mitigation.

Solving Coverage problems takes a specific kind of skill, one that's very different from playing around with fields to figure out an appropriate natural key.  If you've got a lot of Type 2 debt you're better off looking for someone with a lot of business knowledge to help glue schemas or keys together than you are asking a newly-hatched BI analyst to solve the problem.  If most of your problems fall into the Equivalence category you should build or expand your Data Lake (or build something more sophisticated - ask me how!) so you can start to work on the semantic components you're going to turn into joins.  On the other hand, if you're anywhere in the vicinity of the Blackberry situation you need to rethink your conceptual model.

You can find a more readable version of the Problem Index here. 

More detail on Types of Debt

Before we get to an assessment matrix for semantic debt I need to get into more detail about the types of debt.  I've posted a few times about a couple of the types, but I haven't done anything like a deep dive on what the four types of debt are and why I think there's four, as opposed to three or five or twenty-two.

Here's how our discussion will go:

1. We'll talk through a working example.
2. We'll talk about how data migration generally works.
3. A recap.
4. Type 1 debt.
5. Type 2 debt.
6. Equivalence failures in general.
7. Type 3 debt.
8. Type 4 debt.
9. Coverage failures in general.
10. Why are there only four types of debt?
11. The end.

If you've got a lot of experience with data management you may already think these four types are obvious.  You may even think there's more types, in which case please leave a comment.  After you leave your comment you can skip this post.

Everyone else who's interested should sit down, because this is a little denser than usual.

Categories of Debt

In my last post I reviewed how semantic debt happens and put up a picture of a data model staying static while a business model changed.  That's the core insight to the concept of "semantic debt": An organization changes, but the data models that describe the organization don't keep pace.  At some point the organization spends a lot of time compensating for the drift between data and business models.  That's semantic debt. 

It costs money to deal with semantic debt.  That cost takes the form of an opportunity cost.  You might incur that cost because the org's reporting system is brittle or needs a lot of machinery to manage semantic debt - perhaps because there's a lot of Excel spreadsheets that need to be merged into the system to cover for the fact that the data model doesn't hold the data that's in the spreadsheets.  Or the cost may be incurred by a virtuous cycle of data governance and change management, where the debt is managed positively and proactively by people who can balance the need for change in the data models with the software release cycle.  In either case the debt needs to be managed, but whether its a hidden or an obvious cost is a decision by the organization.

If we know that semantic debt is inevitable, how do we evaluate where we're at? 

Each of those models and the data in them is a "semantic asset."  While its certainly possible to look at an organization's semantic assets and by fiat claim anything old is deep in semantic debt, its a strange fact that sometimes the business just doesn't change.  Sometimes really old data models are still really good from a semantic standpoint; the organization still uses them productively.  And I'm not talking about a type 4 or "epistemic bubble"-type issue, either, where the organization has limited its activities consciously or unconsciously so it doesn't run afoul of its data model.  Some organizations do just fine on old semantic assets.

(I think we'd all like to think, as data modelers, that we've designed a few of these: data models that are still valid for the business ten or twenty years on.  In practice of course this may be just wishful thinking.)

Broadly speaking there are two types of failure, or two ways a semantic asset might fail: Coverage and Equivalence.

Coverage failure occurs when our semantic assets don't describe as much of the organization as we think they do.  We run into this scenario all the time with organizations that have expanded - e.g. added new product lines - but continue to use an old data model.

Coverage failures include Type 3 and 4 debt.  Type 3 debt involves scenarios with under-determined schemas or content.  Type 4 debt is "epistemic bubbles," or scenarios where the organization restricts its behavior to conform to the model and is thus blind to opportunities it might otherwise pursue.

Equivalence failure occurs when we have two semantic assets for the same organizational process, but they can't be mapped into each other.  Type 1 and 2 debt are Equivalence failures: Type 1 is "incompatible schemas or content," and Type 2 is "incompatible natural keys."
 

We often see Equivalence failure in orgs that have two applications for the same function, like email marketing.  In fact this problem is so common in email marketing I would wager that most MDM projects are designed to solve problems that have arisen because the org discovers its got rampant equivalence failures between its two email marketing systems.

Perhaps the paradigm case for me, however, of Equivalence failure is the phenomenon of irreconcilable categorization schemes, which is an instance of Type 1 debt.  Remember this image from an earlier post, where I talked about ERP migrations and the difficulty in reconciling order-management status schemes?

Its a fair bet that any time you see a categorization scheme some significant semantic debt has been built up.  Let's think through this for a minute or two, because its illustrative.  Note that we're not talking about incompatible data models here, but content schemes.  The data model for the two incompatible content schemes might be identical pre- and post-change, but the way those schemes divide up the world - the way they "model" the world, to be more precise - can't be mapped from one to the other.

One reason categorization schemes rack up debt so quickly is because they're almost always inherently unstable.  Very few categorization schemes are immutable, or as valid for me as they will be for my children or my grandparents; that is, as the Philosophers would say, very few categorization schemes are natural kinds.  Consider the category "race," which is a common personal attribute in American software applications.  Very few applications in the rest of the world even record "race" as a personal attribute.  Those that do recognize that its a constructed category and treat it as customizable, and as such can't be expected to provide consistent reporting.  But as little as a decade ago "race" was considered an immutable category in the US, and the composition of the "race" scheme considered a fixed and mutually-exclusive set of components.  That is clearly not the case now.  "Gender" is another category that until very recently was considered a fixed and binary category, with exactly two subcategories.  Now "gender" has a number of values, and depending on the purpose of the application those subcategories may or may not be mutually exclusive.  Its certainly possible to engage in a long and voluble argument about how "political correctness" has changed these categories.  But the simple facts are that (a) in thousands of years of philosophical inquiry no one in any of the traditions - European, Chinese, Indian, Aztec, Plains Indian, to take just a few - has been able to propose more than a handful of logically-neat immutable categorization schemes, and "race" and "gender" aren't even in that list for the modern European tradition, never mind anywhere else.  And (b) insulting your customers or your customer's customers is never the path to happiness and prosperity.  So its best to bite the bullet and, despite your personal preference, learn to live with the mutability of categorization schemes.  Acknowledge that your scheme will incur semantic debt, and prepare for it.

When a categorization scheme changes it is more often than not the case that any reporting that uses that scheme must declare an inflection point, a year 0, that prevents temporal comparisons.  There are mitigation strategies that can be used to lessen the impact of these inflection points.  One of the simplest is to restrict the scope of change in a categorization scheme to subcategories; you partition the world up to reflect five subcategories instead of four, but you don't change the category those subcategories belong to.  In that case while you can't make comparisons on events pre- and post-change that use the subcategories, all of those events still fall into the same category.  You might try a semantic-equivalence change, too, although usually if your categories need to be changed its because the old semantics doesn't work anymore.  A third option I've used is "managed change," where you phase in changes to subcategories and categories over time so that while one set of events can't be reported on because of an inflection point, the others can, and while you migrate some local reporting fails temporal comparisons but other reporting stays stable.  Or you can just "bite the bullet," "rip the band-aid off," "dive into the deep end," etc., and make the change as global as possible.

Second, categorization schemes are often implemented in ways that make them difficult to change.  There's a longstanding application-development heuristic that says one shouldn't put categorization schemes into code, as e.g. enums.  You'd be surprised how often that heuristic is ignored, particularly now that everything goes into a NoSQL database and nobody believes in relational lookups.  Many developers think the heuristic is intended to avoid situations where QA has to do a regression test on a change to the enums, and assume the regression would be required because of something simple like e.g. a change to the width of the strings in the enum.  But the heuristic is driven by the fact that putting enums into code increases the volume of semantic debt, and incurs at least two types of debt.  Changing the schema in code requires at the very least a software release, and as that entire process may be opaque to the business users who rely on the categorization scheme it is very often the case that the scheme and the software come to mean something very different for the users than for the developers and the architects.  To avoid the release process users may take to categorizing events out of the process flow dictated by the application, perhaps in Excel or a similar system, simply to keep the business running.  Or development teams may de-prioritize enum changes in favor of sexier or more exciting or complex changes, leaving the business team that needs the change to otherwise "make do" or otherwise restrict their processes until such time as they can get the change made. 

In an earlier post I gave the example of an organization with a CRM application where the returnable product list is an enum.  (Again, some software developers may scoff at that example as a straw man: Surely no one would actually do something that stupid!  Uh, yeah.  A little more abstraction may reveal some interesting parallels with the scoffer's own situation.)  Products that aren't in the list simply can't be returned, and so any disconnect between the process of updating the RMA-able product enum and the process of releasing products to customers will result in a bad customer experience and a lot of unhappy CS people.  I think the problem here goes deeper, though, than just the point of data entry.  In an architectural sense treating the "product" (in this case) as simply a value in a variable in a formula for "how to return a product," and not as an object in its own right, means the process of return is also fixed: It becomes very difficult to modify the return process itself, architecturally, if all returns follow a fixed formula.  What if, for example, we want to manage two return processes?  We can't manage a single list of RMA-able products in a single enum, we need to manage two.  But since that's pretty difficult to manage, we probably just restrict what we can do to what the enum-based software can handle.  So putting enums into code is bad practice architecturally because it puts us into an epistemic bubble, restricting the options available to the organization, and that's Type 4 debt. 

So the four types of debt fall neatly into these two Failure buckets.  I've identified four types of debt:

1. Debt that results from two (or more) incompatible but otherwise defined schemas or content.
• For example, two email marketing systems that are organized differently.  Perhaps one is relational and the other is hierarchical.
2. Debt that results from two incompatible natural keys.
• For example, suppose one of our email marketing systems identified individuals using their ERP-assigned customer key, and the other used email address.
3. Debt that results from under-determined schemas or content.
• For example, file systems with poorly-documented or under-governed files generated by devices of various versions.
4. Debt that results from epistemic bubbles.
• As in scenarios where an organization restricts its business opportunities because its data model can't accommodate them.

Dropped into a matrix, our immutable and almost-certainly-a-natural-kind categorization scheme looks like this:

(You can of course use "Type I" or Type III debt," if you think the Roman numerals look classier in your governance deck.)

In my next post we'll talk through how to assess the volume of debt incurred in each category and type of debt.

Type 4 Debt: Epistemic Bubbles

In my second piece on Semantic Debt at TDAN, I walked through four types of debt:

1. Incompatible schemas or content.
2. Incompatible natural keys.
3. Under-determined schemas or content.
4. Epistemic circularity.

The first three should be pretty familiar to anyone who's tried to migrate data from one system to another.  Type 1 debt is common when you need to migrate data from one ERP system to another, for example.  I've run into it fairly often with order-management systems, where one version of the ERP system uses a set of order status categories and another version uses a different set of order status categories.  The categories are incompatible perhaps because there's simply stages in the new category set that aren't in the old set, or because the order-management system has even rearranged their order.   

This used to be a very common problem in the early days of eCommerce, in the 00s.  Companies would come to realize that their long-standing and comfortable order-management schemes didn't work with ecommerce.  They'd update their ERP systems to accommodate the new style of ordering, which usually involved moving to a newer version of the ERP system.  The mechanical change required to manage the higher order volumes of ecommerce often necessitate a change in the way order status is tracked, though, and so suddenly there's these organizations discovered they had a big conversion headache as well.

We see similar shifts today, as we move from the notion of a discrete purchasable product, to a subscription to a more amorphous set of services and privileges.  Some parts of the subscription concept can be retrofitted into the old "order form" paradigm but most can't.  (Imagine, if you will, a server at a restaurant trying to write down on a single ticket what's involved in a monthly purchase of Adobe's Creative Cloud.  You can adapt a lot of behavior by rethinking the concept of SKU, but the difference between boxed product installed on a laptop that anyone can use to on-demand product with variable levels of permissions, with time-limits on access or even JIT checks on access makes that restaurant ticket a nightmare.)  As more products and services move to a SAAS model - from "buy it all" to "rent pieces we need" - organizations will need to try to migrate old schemes with discrete stages for discrete products into a much more flexible order status schema.

I'll go into more detail about the other two types of debt in future blog posts.  But the fourth is the sexy one.  Its also the most complex to describe, although once you understand it you see it everywhere.   

I initially called Type 4 debt "Epistemic Circularity," which is a term I picked up from Epistemology.  Its not appropriate for most uses because the actual concept of "epistemic circularity" is way to hard to understand.  Many philosophers and/or professional epistemologists would venture that because its hard to understand its probably not coherent.

The term I want to use instead is "Epistemic bubble."  An "epistemic bubble" is a form of confirmation bias, most commonly seen in politics.  In fact if you spend any time online you'll see some variation of the "epistemic bubble" term thrown around.  (Its common to call someone's political epistemic bubble "epistemic closure," but the latter is simply an inaccurate use of the term.)  The basic idea behind an epistemic bubble is that an what a person can believe to be true is bounded by what they allow can be true.  In politics this is often seen in limitations in what you can accept from your political polar-opposites.  In US politics, for example, Republicans won't listen to Democrats because what Democrats allow could be true is not allowed in the Republican epistemic bubble.  In more specific terms, Democrats might believe that fraud and waste in social welfare programs is a very small percentage of the total amount of money disbursed, whereas Republicans believe its a significant amount.  Democrats would as a consequence budget very little money for accountability and compliance, while Republicans would tend to want to spend a lot of time and effort on accountability.  In the Republicans epistemic bubble social welfare programs are simply rife with fraud and abuse by their participants; in the Democratic epistemic bubble fraud and abuse are a very small percentage of social welfare programs, and so time and effort are better spent on expanding the reach of those programs.  In both cases there's a set of things that could be true, and so we can make sensible statements about them.  Where the bubbles don't overlap, however, you can say things about what's true in one bubble that simply don't exist in the other.

Organizations get into epistemic bubbles too.  There's an easy formula, from an analytics standpoint, that you can use to determine how much of an epistemic bubble your organization is in:

We can only report on what we can do, and we can't do anything but what we report on.

This may appear unnecessarily cryptic, almost a Zen koan.   

Consider: If you want to expand your business, how would you go about it?  Suppose you wanted to launch a line of super-fancy SAAS products.  ("But we're in the packaged goods business, Dave..." I can hear you say.  Volvo is talking about a subscription plan for cars.  If a car company can think of cars in terms of subscriptions, you can think of your packaged goods as fungible products.)  What changes would need to be made to your current systems to accommodate a subscription product? 

For example, would you need a new order-management system?  What about your billing cycles?  How does Customer Service differentiate levels of service for customers with the old product versus the new ones?  Interlaced through all the systems collecting data about your organization are various semantic assets passing data from one system to another, and from reporting systems into the hands of users, whether that's Marketing, Finance or Customer Service.  All of those semantic assets and systems need to be changed because the launch of the new SAAS product puts them all into semantic debt.  You simply can't convey the information you need, because you don't have the models that can accommodate it.  In very simple terms, you don't have the right fields in the originating systems, and you don't have fields in the downstream semantic assets, and you need to run a regression all the way through the ensure that adding anything in at any point in the stream doesn't screw it all up.

Now some degree of "epistemic bubble" is a good thing.  We're usually in business to make money, so spending all of our time worrying about all possibilities is a good way to go bankrupt.  We thus have to accept a certain set of limits on what our data models and the semantic assets that represent them can encompass. 

But where our semantic debt becomes onerous is when we can't actually make the changes we need to make.  In machine learning terms, our model has overfit the data: It performs perfectly on the test data we've given it, and terribly on the real-world data we need it to manage.  In machine learning the simplest approach for dealing with an overfit model is to back off the training data, either by mixing the set up a little, adding in some incongruous data, or restricting the training set so its not so perfect.  This approach only works as a kind of kludge, though.  The bigger problem with overfitting is that its a sign our model doesn't really work. 

In enterprise data management we can't do quite the same things, but the strategy is often the same.  We find ourselves restricting our strategy to fit our data.  We have a data model that works, and we look at which business models can be true of that data model, and we choose those so we don't have to make any changes to our data model.

In a future post I'll go into more detail about how we can design data models to be more flexible.  This part isn't rocket science, and its a lesson most experienced data modelers have learned.  But its generally only applied piece-meal, to individual systems, and not the big picture.