The Data Thesaurus – More than a Dictionary

So if we stop trying to create a universal dictionary, what are we going to do instead?

A better approach would be one that takes advantage of the tactical point solutions that most enterprises seem to succeed with. The pejorative idea of a “data silo” should now form the basis of our solution.

We start by building systems and architectures based on the recognition that different contexts exist. I posit that such concepts as “schema on read” and the associated technologies invented to support the “Big Variety”* inside of our modern data platforms is a good place to start.

My proposal: create a “data thesaurus” capturing in meta data all aspects of the structure, meaning, flow, usage and context of data as it appears and moves about. This thesaurus would contain the obvious syntactic reference material of traditional data dictionaries, but then would include data describing the semantic relationships between different collections of data. By explicitly permitting multiple, varying DOMAINS to co-exist for different purposes and uses, the same datasets could be mapped to multiple uses without forcing a premature homogenization to a unified model.

Finally, it would explicitly include content indicating the pragmatics (usage patterns) by capturing in declarative form the transformations to apply as the data moves from one domain context to another. The scientific community has been working in this data management solution space for many years, although perhaps not recognizing its potential for the wider world.

Provenance systems point to a promising potential, given that they track lineage through and across transformation and manipulation. Every dataset, from the initial registration of data when first received, through every transformation and manipulation, must be tracked. While the system would not have to retain every variation, it should be able to reproduce any variation from any point in time.

To begin, build data dictionaries for individual systems or even small groups of files (as is often the full extent attempted and completed in most organizations). But instead of trying to extend these point solutions into a universal solution, links across contexts within the organization would be filled in only as practicality required, as the by-product of data integration projects or system consolidation efforts. In this way, the thesaurus will grow in content and utility, but not require unification in its entirety, at any stage.

* Big Variety is one of the three challenges leading to the adoption of Big Data, as described by M. Stonebraker. It describes the situation where the variety of data structure that must be integrated within an enterprise has ballooned to where the traditional practice of data management can no longer keep up.

The Problem in Our Stars – Yesterday’s Data Dictionary

First published on LinkedIn here.

What’s wrong with our current data dictionary?

Let me count the ways:

1. Presumption of the existence of a single, unified model
   Integrating data from a thousand sources means not having a unified model – why can’t I have a dictionary that recognizes this? No support for capturing information around variations and special cases means also that measuring the overlap and similarity of variations of models can not be done.
2. Only one name for a concept
   When different contexts may have their own colloquialisms, it becomes hard for practitioners to find “their data”. It can even lead to the introduction of additional entries for synonyms and aliases as if they were separate things, which just makes the problem of unification worse.
3. Obtuse or idiosyncratic naming conventions 
   Abbreviations and acronyms developed by and for developers, programmatic short-hands, bending term definitions just to get a small enough reference – all are symptoms that our dictionaries are not human centered.
4. Abstracted or generalized beyond usefulness
   When the terms in the dictionary become too abstract, the utility of the dictionary becomes questionable. An abstract class model is useful when developing object oriented code, but not always useful when trying to understand the breadth and variety of  the actual things in the domain. Using the “least common denominator” and dropping the specificity and nuance present in the original contexts is a LOSS of meaning, not an enhancement.Consider the HL-7 model: “Entity Plays Role Participates in Act” – nicely abstract, but not terribly informative. Except that I named the standard, you can’t tell that this is even a healthcare model! Claiming that this model is a dictionary of the healthcare domain is like saying the grammar rule “Subject Verb Object” is a dictionary of the English language.

   When specific semantics (the interpretation of the model) requires data-driven interpretation,  as in the so-called “MOOD” code on the “Act” object which indicates what type of Act is occurring, then content becomes the driver of interpretation. But do these details make it into our software based dictionaries? Can your data model and dictionary capture the pragmatics of these configurations and still reason over them?

My constant and continuing premise is that DATA is a form of HUMAN LANGUAGE. As such, it exists with all the messiness that comes with being language – synonyms, homonyms, vaguely overlapping ideas, similes and metaphors, pragmatics…

Our collective, software-writing culture (i.e., our shared culture as software developers) largely tries to pretend that all of this variation is just engineering sloppiness.  Until we recognize the nature of data representation as language, our dictionaries and models will continue to suffer from these deficiencies.

“Master-less” Data Management – What’s in a Name?

Originally posted on LinkedIn here.

I work for a company providing SaaS applications in the healthcare and higher education space. Not only do we have multiple product lines, each with variations (loosely overlapping, perhaps, but various when looked at generally) in their data structures, but we have thousands of members providing self-structured content that also does not conform to a universal, standard submission structure.

I have been leading a project out of our Higher Ed space for the last couple of years to try to establish a unifying infrastructure that directly attacks this diversity through establishing a meta data repository describing the content and its various relationships to our various product lines. Not to get too deep into details, our solution uses the meta data to auto-generate on-the-fly transformation programs for delivering member data to each of their subscriptions.

Our meta data model assumes diversity of domains, not just in sources but also in targets, and provides a basic representation of the equivalences across the different domains. It provides as a first-order concept this notion of domain, permitting us to track aliases across domains, and to capture formula that establish construction logic to force consistency (this is how we add logic to our auto-generated code).

When we started, we performed extensive searches for existing software frameworks that could be applied to our use case. While we found many potential products, we never found one that actually addressed our conception of heterogeneous, multitudinous domains. What we found were many many products and frameworks that took the stance that there really would be only one “canonical” domain at the top of all else.

Our concern with these tools was that it would force our product lines to have to prematurely establish a single, overarching model of the business space. We felt this would be too constraining to our ability to develop new features and products in an aggressive and agile manner.

Hence, we undertook development of our own multi-domain, meta data repository, and the associated, basic transformation generation capability already mentioned. This has been slowly building in capabilities, and we are now supporting multiple product lines.

What is most interesting to me about the approach, however, is how it is beginning to show the potential power of a bottom-up approach to defining that overarching, “canonical” model. We can already see how, using such techniques as automated transitive closure to “clone” meta data across domains, a larger model of the Higher Ed space is emerging.

I’ve been thinking about terminology a lot, and I used to love to coin a phrase. But I’m now of a mind that permitting multiple, slightly overlapping names for slightly overlapping concepts can be a good thing. So, getting to the actual point of this discussion, I thought I’d ask for comments around some of the variations of names I’ve been working through to describe this technique/phenomenon. Let me know what you all think of these. What do they connote to you?

Emergent Data Management, or Emergent MDM: since the master model emerges from or is made visible through the amalgamation of hundreds of variations

Master-less Data Management – kind of a play on words, really, reflecting that the whole is, at core, not curated purposefully, but still implying that there is control and regularity to the endeavor

I thought about trying to incorporate the jaunty, self-contradictory nuance around the name “NoSql”, but I thought “No Data Management” was too easily mistaken as implying no actual management.

(I once posted about this before on another blog, and had come up with several other names,)

I’d also be interested in seeing hints/suggestions for other emerging technologies, services, frameworks or packages that might support the approach I’ve just described.

Principles of Master-less Data Management

This post should in no way be taken as a dismissal of Master Data Management or the goals MDM espouses. I have a healthy respect for the value of a little bit of MDM! Rather, following  the provocative lead of the NoSQL movement, I’ve tried to create a name for a set of principles and practices that share many traits with MDM, but also that diverge in important ways from the older practice.

I will admit at the outset that  I no longer believe in the mythical “single version of the truth” as an attainable goal at any level beyond specific, practical, business-driven needs. Put simply, I think it is a mistake to take an endless quest for a universal (or even just an enterprise-level) data model, in whatever form that might take (e.g., enterprise messaging standard, enterprise data warehouse, operational data store, semantic web, or MDM schema).

I will readily admit the value of and even actively participate in data integration and standardization efforts of even very complex and large data integration solutions. But I believe there is a limit, both practical and philosophical, beyond which the effort to unify and standardize should not go. This limit is often hidden, hard to see before reaching it, but when these efforts go past their own limit of utility, it is important that they stop.

I have described elsewhere how the quest can become absurd.  My doubts along these lines have been particularly reinforced by the arrival of the Big Data trend, and in particular by what Michael Stonebraker has dubbed the “Big Variety” problem. When there are thousands of “models”, no human activity can possibly keep up. As he has suggested, I believe the unification problem now requires a mixture of crowd-sourcing, “expert” curation, AND automated discovery (machine-learning technologies).

The integration problem has just expanded beyond the capacity of we human craftsmen to manage alone, We can no longer establish dominion over the entirety of our data resources. So, for anyone now facing the question: “How do I approach this problem now?”, I submit these Principles of Master-less Data Management.

The Principles of Master-less Data Management

1. Data is Language, Act as if You Believe It

   Remember that information is CARRIED by data. The forms and names given to the forms carrying that information are, in many important and fundamental ways, arbitrary. The technology doesn’t care, but the humans do.

2. Respect the Native Speakers

   No matter how cleverly you choose your names, remember the language is not your own. Do not act as if you think you are the “enlightened benefactor” of a hundred backwards-thinking communities, lest your efforts be perceived as the “semantic colonialism” that it can easily become.

3. Accept the Limits of Unification

   Limit your unification efforts to that which is needed to be communicated. Not everything in the “native vernacular” can be translated, neither does everything need to be unified. Stick to the immediate communication problem.

4. Be Multi-Lingual

   The best solution will accept the differences in terminology and convention. With data, transformation between points is a given. Interestingly, the most common transformation is the “alias”. Whenever possible capture and incorporate aliases into the solution.

5. Permit a Unified Model to Rise Emergent

   Crowd-sourcing and machine learning based models, while imperfect, will have the property of eventual convergence. Build your practices and systems to recognize and capture the emergent result, while still allowing it to evolve and adapt over time.

6. Humbly Accept That Your Work Adds Complexity

   Don’t pretend that your consolidated models (and systems) are different in kind to the original, unconsolidated models (and systems). Especially in situations where the sources remain operational, the introduction of the unification system represents a plus-one to what was there before. Remember that tomorrow’s merger will turn your best managed model into just another starting point.