Wednesday, April 20, 2022

Opportunity Cost

 Opportunity cost is the basis of semantic debt.

When we think about why a data management team can't deliver, we often have trouble quantifying why.  Is it because no one did the integration?  Is the pipeline management system faulty?  There's no product management function to drive the semantic layer?

OK so ALL of that is true.  All of it.  But how do you compare the cost of hiring two new data modelers to ... what?  It seems like you're developing some kind of NULL hypothesis.

This is NOT a problem, btw, if you have experience with what two new data modelers on full-time employee can do.  How do we know that?

Consider a Replication Engine.  With five licenses and a team suite I can do what I'd need five data engineers to do manually.  The tooling is a vast improvement on having five data engineers work around the clock for a year to move an orgs dbs.  Because it automates most of that work, which was never super complicated anyway.  

REs are something people in small ETL shops have been doing forever anyway, the same way they used to build their own data catalogs.

So the opportunity cost of the semantic debt in all of those dbs - that is now quantifiable, at least in part.  Before a replication engine if you'd told me that it'd cost $1 million just to move the dbs, and a year's effort, before I would even see the least bit of benefit, I wouldn't have blinked an eye.  That is why semantic layer construction is so hard - by the time the pipelines are built everyone is exhausted.

But that $1million is now fungible software talent and I don't have to wait a year.  I can do it in a week.  Is it worth it to me to get a year's worth of benefits in a week?  Yes.  Especially if my metric is something easy like "billions of records/day."

The semantic debt of your current legacy databases is the last estimate you got from your usual database management provider (internal or external) when you asked how much it would cost to make the data warehouse actually useful.     

No comments:

Post a Comment

The point of this blog

Welcome to the beginning of the film

In the twenty-plus years I've worked in data management I've spent a lot of time having some version of the following conversati...

Top 3 Posts