Thoughts on Semantic Debt

Here's a link to the first piece I wrote on semantic debt.

I've read quite a few newsletter/online magazines for data management professionals over the years.  But I consistently come back to TDAN, or The Data Administration Newsletter, which is managed by Robert Seiner.  There's two main reasons for this: First, the people who write for TDAN are fully conscious of the need for planning a data management program.  Indeed many of the articles are clearly frustrated arguments for planning made by actual professionals having to deal with VPs of software development and similar people, who tend to discount the need for intentional data management until its too late.



If you've ever worked for an organization that decides somewhere far along the line - i.e. too late - that it finally needs to do data management, you'll know the patterns: "we really needed to do the software first," followed by "we think this is a great opportunity for someone who loves data," and almost inevitably "well this isn't all that hard, especially compared to building software, so I don't see why you're having so much trouble."  Everyone who does data management for a living wishes they'd been brought in earlier, if only to try to bend the curve just a little toward some organization; its very rare that software developers do their stuff with any thought to the long-term needs of the organization, and so they can easily build systems that create piles of stuff with obvious value and no way to access that value.  The incentives are just not there, whether its because there's agile coaches telling them they ain't gonna need it, or business owners who really don't see the need for even basic software development infrastructure (i.e. non-functional requirements), or a management layer that's set difficult deadlines.  And even if the development teams are aware of the state of the data they've created, its often the case that their management team is desperate to rationalize or justify the team's decisions to prevent negative consequences from accruing to the team.

None of this requires or implies malicious intent.  As in many things, a series of small expedient decisions create walls of justification that can be used to defend the current state of the organization.  But no VP of Software wants to destroy the last three or four years of work, especially given the pain they've gone through to hire the team.

So TDAN is filled with these pleas for early intervention by people who actually do data management for a living.  Which makes it a good place to publish yet another plea for early intervention.

Whether anyone outside of the data management community reads those pieces is another question entirely.  I intended the first piece to be a sketch, which is to say a pretty agile set of thoughts on a problem that move the problem toward a solution.  So it reads a little light and airy.