Data Exhaust

I was thinking about data exhaust on the ride in, and I was still thinking about it at my afternoon tea break. So now I will share. I guess the idea of spending my time collecting, redirecting and analyzing the torrential flow of data that modern life creates -- well it sounds like a dream job! It's a cyberpunk thing, the "digital seer." Or maybe they'd have said "shaman."  Hrm.  A little to froofroo mystical, maybe.

Basically it's what a sysadmin does already for part of their day. It's the especially fun part -- look at all the metrics and figure out what the *actual problem* is. Folks tell us "this server is slow" and we run a couple commands and tell you that you're probably missing some indexes on your database, even though we didn't look at the inside of your database. It's like magic, but not at all.

Wouldn't it be interesting to try to formalize or mechanize that? Watch and see what handful of tools you use to take the pulse, then make that an automatic script? Isn't that basically what the data visualization nuts are doing -- translating metrics into something that doesn't require "work" to explore? Are there tell-tale metrics (or ratios of metrics, or red flags) that let you see the proteome of a running system, so to speak? If you show others how it's done, is it still magic?

Maybe that's exactly what I like about my imagined profession.

What if our diagnostic craft stopped looking like magic and started looking like art?