Big Data

One Finger Moon-Pointed

Everyone seems enamored of the accidental scaffolding we find ourselves forced to erect around our growing tower of data exhaust by our increasing cupidity for data data DATA and the treasure we hope we’ll one day find therein.

We have to develop machinery to deal with data at the latest greatest scale, but the machines are just nuts and bolts. We needed a garage to park our data-Hummer in. We had a crew build one by hand using the poor Bronze Age tools available. The final result is eccentric and baroque; it requires constant maintenance just to avoid collapsing; but it solves the problem.

The part no-one outside well-insulated research centers seems to be getting all moon-eyed over is the part that matters long-term: you have what was till recently an unthinkable amount of data. How do you extract meaning from it?

This gets us into statistics, machine learning, curve fitting, segmentation, clustering, sentiment analysis, etc. That’s the part everyone should be getting excited about.

I want to read breathless blog posts about how you have advanced what we currently know how to do statistically. Because I can’t write those posts; I can poke machines and set them on their feet and point them one way or another and dust them off when they fall down, but I know not nearly enough about statistics and numerical analysis.

So much of today’s tech news is just oohing and ahhing and cooing at the latest gilded set of Napier’s bones. We’re an industry in love with its slide rule, polishing and primping and extolling its many virtues. We forget these Rube Goldberg contraption abstractions are tools for the real pursuit of knowledge.

We’ll use Bayes’ law for the next several centuries. We’ll use today’s wonder-systems for the next several decades, and we’ll curse them as legacy rubbish long before that time is through.

See also: Bryan O’Sullivan’s “Big Fucking Deal”

One Serf Disjointed

There’s another angle on this: Our continuous tech-wow is an obscene celebration of the enclosing of the commons. You’ve worked out how to pen up a bunch of users and soak them for all they’re worth. You’ve built the airiest friendliest most inescapable prison-hotel. To do so required innovations legal political and technical the like of which we could only have dreamt of a century ago. But this is not innovation in the service of anything noble; it’s the sad extraction of value from a benighted underclass of peon users.

See also: Hammerbacher’s comments in “This Tech Bubble Is Different”; Schneier with “More on Feudal Security”.