DataOps Eight Years Later

What's Changed..and What Hasn't

Back in June of 2014, when I was an InformationWeek editor, I wrote a guest blog for IBM where I coined the term “DataOps.” At least that’s what Wikipedia says. And dammit I believe them.

I wrote that blog because I saw first-hand how organizations were attempting to monetize algorithmic processing of what we used to call “Big Data” without sufficiently considering the operational efficacy of the entire end-to-end value chain in which that algorithmic processing was situated.

My central point was that data science (a horrible name for algorithmic engineering) doesn’t exist in a vacuum—but instead depends on many other data operations: efficient intake of data at scale, control of data quality, the price/performance ratio of the infrastructure where the algorithmic processing is performed, optimized presentation of algorithmic results to non-technical decision-makers, etc.

Eight years later, a Google search for “DataOps” returns about 2.63 million results. So it seems that I may have been on to something.

And I believe I still am. That’s why I figured I’d take a minute to review the state of DataOps in 2022.

Here, then, are three present DataOps realities I believe are worth considering:

DataOps Reality #1: Organizations still ignore DataOps at their own peril

Remarkably, the core point of my original thesis still stands. Much like the unfortunates in The Treasure of Sierra Madre, too many executives at too many organizations mistakenly believe all they have to do is mine the gold in them thar data hills—and that money will then simply fall out of the sky. So they focus exclusively on recruiting the best data science talent they can find, wrongly assuming that said talent will lay the golden eggs that get them to their next level of funding (or IPO or cash-out or whatever trivial worldly goal it is that they seek).

Then they scratch their heads wondering why things don’t work out according to their mislaid plans.

There are many reasons businesspeople are susceptible to this mistaken belief. One is that they discount scale, because the cloud offers them unlimited elastic processing capacity. Another is that they fantasize about a revenue opportunity that will dwarf their AWS costs. Yet another is that their target isn’t sustainable profitability; it’s a multiple of revenue.

Regardless of their reasons, however, they continue to make the same mistake over and over again. They do data science (which is, of course, not science by any stretch of the imagination) without doing DataOps—which can be a bit like putting on a concert of Beethoven’s Ninth performed by middle schoolers in a mall parking lot and then acting surprised when it doesn’t sound like the New York Philharmonic.

DataOps Reality #2: DataOps success is increasingly vertical

Data science practitioners—like many technologists and engineers excessively infatuated with their own narrow area of expertise—tend to believe that their skills are horizontally applicable. That is, they gloss over what they see as the surface idiosyncrasies of vertical markets.

This is a mistake. The differences between business challenges faced by a retailer and a facilities security contractor are non-trivial. And those differences inform every step in the DataOps value chain, including:

• the variety, quality, and regulatory constraints on the data being sourced

• the nature of the algorithmic operations that will best address the business challenge being faced

• the dynamics of change that impact how aggressively organizations need to continuously re-tune and re-configure their DataOps value chains

• the economics that cost-justify both initial and ongoing budget allocations

• optimal visual presentation of data outcomes to the various participants in the business process

For these reasons and others, we are generally seeing organizations that develop a strong vertical focus for their DataOps disciplines greatly out-perform those that treat every business challenge as “just another data science problem.”

A horizontal approach to DataOps, in fact, is hardly DataOps at all—because it ignores a fundamental lesson that DataOps is supposed to take from DevOps: that iterative engagement with the “customer” (whether internal or external) is what makes or breaks the NPV of an organization’s investments in algorithmic operations.

DataOps Reality #3: DataOps practitioners need a moral center

DataOps is transforming society, politics, and culture. It is empowering private corporations and public agencies to surveil people’s actions. It is enabling those with that power to reward and punish behaviors as they see fit to serve their own interests. It is at the same time disempowering those without the capital and/or technological resources to engage in DataOps strategy and tactics themselves.

This places tremendous ethical responsibilities on DataOps practitioners. We have already seen plenty of examples of how DataOps can violate citizens’ privacy, facilitate monopolistic control of access to credit and other resources, codify racial bias, and otherwise work against the summum bonum.

It is incredibly easy for DataOps practitioners to abrogate these ethical responsibilities. After all, they’re hired to perform a technical job—not adjudicate moral questions. They also typically work for organizations that explicitly place responsibility for ethical behavior in the hands of in-house counsel, compliance officers, DEI managers, and the like.

So the overwhelming majority of DataOps practitioners do what they’re told. And they leave the moral decision-making to someone else.

This, of course, is the Nuremburg defense: “I was just following orders.” Or, to put it in more contemporary terms, moral issues are deemed “externalities”—i.e., they are “external” to the business DataOps practitioners are being paid to address.

Well, I guess that position is fine if you’re an essentially amoral person. But if you believe that you are in fact a human being who possesses moral agency, you better get on the stick. No one is in a better position to avoid and/or ameliorate the moral harms that are inherent in profit- or ideology-driven DataOps than the person doing the Ops.

So you better develop a moral center—and a moral spine. Otherwise, there’s a non-zero probability that you’ll wind up being a data nazi. Because, as the British philosopher John Stuart Mill explained it: “Bad men need nothing more to compass their ends, than that good men should look on and do nothing.”