All technologies are prosthetics

I've long thought that the best definition of "technology" is "a prosthetic." In this post I'm going to explain what I mean by that. I won't engage the boundaries between science, engineering and technology in this post, and I won't go into much detail about the model itself. My goal is to inflate the concept and compare it with what we know about technology.

I used to think I got this definition from McLuhan. But he mentions the specific equation once as far as I know and dances around prosthetics everywhere else. McLuhan is much less interested in what I'm going to call The Prosthetic Model than I hope every time I go back to him. This philosophical definition of technology, as just and only a prosthetic, matches my personal experience with technology. This is as a person who's profession was abstract technology-creation and -use, and an avid technology user, and as someone who's thought philosophically about technology and technology use his whole life. 

A Series

 I wrote a short post at Raybeam's Focus site.

I've got a couple of follow-on posts written laying out the argument more succinctly than I've been able to so far.  Eventually this will get turned into a longer piece, and better illustrated than most of my usual.

The centerpiece of the first post is this graphic:

It's important to keep in mind that, when we talk about "data architecture" these days we're actually talking about six different paradigms (more or less) of data modeling. 

Each of the systems in the ecosystem diagram above have their own data modeling rules. You can't just apply a Kimball-style star schema to everything. For one thing, there's more than one way to define an attribute in most dbs. You can make the rule that every attribute should be a field in a table, but someome is still going to use a _TYPE field to draw a distinction that doesn't make it into a field.

That conflict is a basic input to the modeling styles for each system in the ecosystem.  (That's one conflict.  There's others.)  Each one plays a role in that ecosystem depending more or less on which other systems it gets and receives data from.  Sometimes data migration is as simple as forcing all the attributes into one long record. Immediately we solve the "we can't find our data" problem. Eventually it happens again when the table gets to be so wide no one can see one end from the other.  And that's when the dimensions multiply like literal snowflakes in little boxes of JSON.

In any event, at least six styles of data architecture in this ecosystem. I grouped all the collection points - the applications - into one because the current prevalent model of data modeling in software architecture is to stick to a "developer driven" NoSQL model of agile development. Ideally the db contains nothing but name-value-key pairs defined by whoever happened to be writing code that day.

Anything more than that? Friction. So there's not much data architecture in modern software engineering practice. 

   

Opportunity Cost

 Opportunity cost is the basis of semantic debt.

When we think about why a data management team can't deliver, we often have trouble quantifying why.  Is it because no one did the integration?  Is the pipeline management system faulty?  There's no product management function to drive the semantic layer?

OK so ALL of that is true.  All of it.  But how do you compare the cost of hiring two new data modelers to ... what?  It seems like you're developing some kind of NULL hypothesis.

This is NOT a problem, btw, if you have experience with what two new data modelers on full-time employee can do.  How do we know that?

Consider a Replication Engine.  With five licenses and a team suite I can do what I'd need five data engineers to do manually.  The tooling is a vast improvement on having five data engineers work around the clock for a year to move an orgs dbs.  Because it automates most of that work, which was never super complicated anyway.  

REs are something people in small ETL shops have been doing forever anyway, the same way they used to build their own data catalogs.

So the opportunity cost of the semantic debt in all of those dbs - that is now quantifiable, at least in part.  Before a replication engine if you'd told me that it'd cost $1 million just to move the dbs, and a year's effort, before I would even see the least bit of benefit, I wouldn't have blinked an eye.  That is why semantic layer construction is so hard - by the time the pipelines are built everyone is exhausted.

But that $1million is now fungible software talent and I don't have to wait a year.  I can do it in a week.  Is it worth it to me to get a year's worth of benefits in a week?  Yes.  Especially if my metric is something easy like "billions of records/day."

The semantic debt of your current legacy databases is the last estimate you got from your usual database management provider (internal or external) when you asked how much it would cost to make the data warehouse actually useful.     

One of the laws of data management

Quick thought: The longer you use old tools to glue old systems together, the less likely you'll be able to integrate or manage new problems.  (There's a bit of a leap there, but not much.)

One of those variables needs to change over time for your operations to stay healthy.  You stick to one toolset for a decade and you won't be able to hire anyone with less than a decade's experience in that toolset to manage it. 

Which you may choose to do, for lots of reasons.  But you're stuck in a decades-old toolset then.  Do you have any idea what you can't do because of your current toolset?

Thoughts on counting mistakes in data management

The second part of our thinking-out-loud/not-ready-for-prime-time implementation discussion is, "how do we know that an asset has failed?"  You might also say "how does doing any of this improve my day-to-day?" 

This seems like a hard question.  I think its the big issue in the implementation program, which is only the latest subsection of what I've been yammering on about now for six months.  You might agree there's this thing called semantic debt, and you might think there's ways to assess it and so on.  But how do we know that something is actually in debt without manually inspecting every ETL package?  I've been somewhat apologetic in past posts for not getting to the answer to this question.  It seems like that should be the first thing we care about, right?

I don't think its that important we figure out how to count mistakes before we talk about how to classify them, actually.  The world is filled with malfunction right now, whether its AI-based business models, the US Senate or the water cycle in California.  And our semantic assets are failing all the time; its just the frequency that differs from one system to the next.  There are some fantastically stable ETL jobs out in the world, and some pipeline transformation code that has never generated an error.  There are probably ten year old APIs patiently waiting for requests right this minute, and microservices architectures that dutifully do what they're supposed to, and Excel workbooks some Finance department has used every month for twenty years.  And then there's stuff that just fails all the time, sometimes of its own accord, for completely mysterious reasons.

Data model patterns are *not* just for relational dbs

A short digression.  Many people who are all "NoSQL f0r3v3r!!11!!" may look at this discussion of "semantic facts" as patently archaic.  Why would you structure anything of value these days, especially when it slows you down and isn't agile?

In particular, I've been talking for the last few posts about building "facts."  Facts as in fact table, or star schema, which is to say Kimball-style data warehouse.  And we all know what that means: A relational database, just like your dad used.  Relational databases, as we all know, are the Ford F-150s of database design.  We'd all much rather drive a new modern Tesla 3 series or some kind of Musk Truck than a Ford F-150.  Why would we use a relational database when we could use the latest version of a document storage system designed by this guy I met at the Hotel Utah last weekend.  For one thing, you can just append fields wherever you want, and for another, its blazing fast on the 2Gb of data I need to store!

Well sure, if that's what you want.

But there's an enormous misunderstanding here.  I'm not talking about relational models with this work on semantic debt.  I'm talking about patterns of persistence.  How you choose to manage your data is critical to both its short-term utility and its long-term value.  Codd's relational algebra is partly about a way to manage writes and reads in a way that doesn't mess up the queue.  In those terms, much of what concerned Codd was purely a function of the mechanical engineering at the time: Once people were able to figure out how to do writes and reads to disc in ways that didn't create bottlenecks relational distributions of data became less of a queuing concern.

(The page I link to there is a piece I've written for a separate project.)

But Codd's insight - what I call Codd's essential insight at the link - is much more about semantics than it is about bottlenecks.  Codd did his PhD work on cellular automata, which we all know now as the building blocks of nanotechnology.  So he was a reductionist by training.  And his first job out of grad school was in database operations, managing the hierarchical data models that were everywhere at the time (and which are everywhere now).  Now anyone who's actually had to manage a complex data set in a NoSQL system will have to admit that they're a pain in the ass: Early modeling assumptions can quickly turn into thousands of lines of code completely devoted to keeping things in synch across "entities."  There's something of an arms race in modern NoSQL development, between making the model even more flexible and speeding up the ability of the model's users to keep everything in synch.  But Codd had to do all of that for IBM, way back when.  So he needed some heuristics that would help him model data in such a way that he wouldn't have to write thousands of lines of defensive code.

The answer was the relational model, but its really just separation of concerns for data management.  Its not relational, its architectural.

Consider two data models you might use for analytics.  (If you don't think you've got a "model" in your data because you never sat down and modeled it, then you've got a bad model anyway.)  The first is your e.g. standard modern 1000+ field file that merges customer attributes with marketing attributes with some transaction data.  What happens if you need to change customer attributes?  Do you go line by agonizing line through each record and make the change?  Well sure, you say; with the modern tools you can do all sorts of things quickly that you couldn't in Codd's day.  Tesla vs. F-150, after all.  Except that you need to put a lot of effort into making sure those cases are well thought out, the code properly defensive, etc.

Consider the second model, which compresses all the customer data into a file with unique records, which can be linked to the main 1000+ field file by guids.  Operations on that compressed file take so much less time simply because the file is smaller.  If your main file is something like a list of all the org's interactions with a customer over the last five years, because you're doing attribution or LTV or recommendations, then you might have to update a single customer's attributes 100 times.  And if there's millions of customers that could all take a very long time.

So you normalize the customer data and put it into its own file, and instead of updating 100+ events X 10 million customers you update only 10 million customers.

If you're smarter still, from an operations standpoint, you normalize that customer data even further so you can update components without touching anything else.  This makes your results both more reproducible and faster and easier to produce - you don't have to to do a regression every time you update an attribute in your 100 million row+ file.

Keep normalizing - there's a word your dad probably used - that gigantic file until its really just down to the essential formula you're trying to analyze.  This may be the straightforward ecommerce question Who bought What from Whom, When did they buy it, and How much did they pay for it?  Or it may be more complex.  But if you remember back to your Introductory Logic course, each proper name in the formula is a variable, and should get its own separate file.

Link each unique record in those dimension files with the central file with all your facts in it and you've got a star schema.

Using the star schema pattern to manage your data even if you are using a NoSQL system or a file-based system like Hadoop will make your life a lot easier.  Codd's insights about the relational nature of data aren't a recommendation about which brand of database to use, they're about how to model your data so you don't have to work so hard to keep it useful.

This is exactly the operating principle behind columnar databases, if you think about it: Systematic normalization can achieve vast rates of compression. 

But the columnar db implementation is really just one way to implement that model.  You can implement it relationally, which is how everyone thinks its usually done.  You can do it in files, too, with the right conceptual infrastructure.  Or image elements. 

So when I talk about creating "semantic facts" I'm not suggesting you have to do any of this in the latest flavor of SQL Server.  You can implement semantic fact tracking in any medium you want. 

Building the Semantic Fact for your basic Fortune 1000 company

We've got the sketch of a data model here.  The model will get some context in this post, and we'll fill out a little more of what's in the dimensions and how to identify facts.  Once we've got some idea of how to generate facts we can start to add some costs and infer some behavior.  And then we can tell our bosses why something will work and why it won't.

So let's work through some implementation questions.  (I will try to mix as few metaphors as possible.)  There's two big questions - or maybe two-and-a-half.

1. How do we get a list of semantic assets?
2. How do we know an asset has failed?