Posted on

Toward an Appreciation Of Generalism

It took over twenty nine years to become the generalist I am today.

One of the best parts of independent consulting is that learning new things is baked into everyday reality. Every day, every challenging issue, brings with it an opportunity to learn new things.

This crazy circuitous path has traversed system support (mainframe ops, *nix sysadmin) application support (SAP Basis,Siebel Admin), Distributed Enterprise System Monitoring (Netview, Tivoli/ITM,Prometheus), and Database support/administration (DB2/MySQL/MariaDB) to name a couple. Ops was always front and center. Now, my new favorite activity is building out scalable monitoring and reporting infrastructures using open-source tools like Zabbix and Prometheus and working with AWS, containers, and config management.

‘Full Stack’ Paradigm (in the 90’s)

Before skill-matrixing in large enterprises was a calibrated thing, in a proprietary stack like Siebel or SAP, hidden away somewhere there may be a couple people that drifted in-between the components. They functioned like a utility players, specializing in DB migration, then automating critical manual work or fixing a weird performance issue in production. Sometimes these were senior employees,  often they were independents. With SAP, the Basis Admin was the full stack person: ABAP code, DBA, Unix admin tasks, Security perms (with Sec folks), SAPGUI, Batch; like gumbo or sauce it was all in there. The same applied to Siebel. What’s interesting is how each tower shared personnel and built domain specific improvements even though the technical areas were different. Because the business required flexibility (in when tasks performed and deadlines) much of the higher level work I and others performed in these roles was remote. It continued like this for years.

In this large enterprise the cumulative effect of the institutional knowledge coupled with the deep dive understanding of applications was, in a way,  a super-power or at least a competitive advantage. Yet, this was not done through tools or even process. It was the culture and vibe that ultimately created this value. The interconnect between teams was woven through the people and the culture. In some ways this org exhibited all three Westrum typologies and occasionally the generative one was dominant. In Ops it was much more bureaucratic and occasionally pathological, usually around RCAs and outages. No doubt still a staid corporate culture but underneath there were elements of trust to do the right thing in the senior management. The vibe of ERP vs CRM projects was quite pronounced though it seemed that the more mature nature of the SAP practice helped steer the CRM projects. In a way, one was a farm team for the other. I still see this pattern.

Now, I feel I have the same exact goal as over twenty years ago: help people solve real problems with software and help their businesses be more successful and self sufficient. That last self-sufficiency part has been a tenet of our business since the start because it just felt right. Oddly, when I  focused on liberating clients from needing us, they ended up trusting and engaging us even more. It also maintains a level of integrity sometimes conspicuously absent in ‘vendor’ delivered software solutions.

My first truly independent consulting gig was installing GNU software for an upstate NY library council (SENYLRC). Ahhh, the joy of compiling Pine from sources and checking out Tripwire or COPS.  I will share that it felt personally rewarding helping libraries leverage software to make work lives more productive and happier. So many years later, I have returned to this space and recognize there is a rediscovered truth for me.

If you would like to explore possibilities for your teams and businesses, contact me here or on twitter @devnullid

Posted on

New Beginnings

‘Out at Sea’ (and back)

I have to credit my friends in DevOps for totally changing my life, specifically John Willis . I’m deeply grateful. He encouraged, mentored and hipped me to DevOps and all the super-fly people in it (to numerous to mention here). When I did the music for his early Cloud Cafe’, I needed to look up exactly what a ‘cloud’ was. I was just pumped to have met John and was curious. All of the below is inspired and I can attribute to the ideas I heard or learned from the women and men in DevOps. When participating in DevOpsDays it is truly a pilgrimage for something missing. After so many years in Dev and Ops, there was finally inertia towards making life in tech a better human experience, a better existence. The work cultures I see at companies now make business cultures I remember in the 90s and 00s look ridiculously lame by comparison. Now, there are well known fine examples of how to be high performers in IT without breaking people. This is significant and that behavior and culture has a name. Dope!

In production support/Ops, ridiculously stressful projects/incidents manifest ‘hidden’ casualties at home (read families) just by proximity to a toxic work culture. Willis’ sea analogy was absolutely true for me. Inside a buried legacy silo in no one can hear you scream. The takeaway: even if ‘successful’, a great rate, a cool team, and building dope things with a great purpose, the eventual cost of burnout can be too damn high. Previously, every time things got a bit crispy (e.g. lead an off-shift build team), I would take time off and try to rejuvenate. As an independent contractor (ICCA member) in LegacyLand (Unix/ SAP/ Siebel/ AIX/Infrastructure/DB2) this was somewhat easy. Independent Consulting was supposed to be a hedge against toxic culture, psychopaths and traumatic on-call. One could always walk away, right?

Not quite.

In my experience, clients would hang on to a performant indie like grim death and roll the contracts or more often arrange better ones. A big takeaway was no matter how much you were banking or planning, sometimes the burn goes too damn deep. If time-off is deferred too long, there may be no coming all the way back anytime soon. In a way though, I thought at the time, isn’t that the deal I signed? I accepted and did not lament this reality of having a number on my forehead, except after a while that number wasn’t alone. It wasn’t just my hourly, there was this other number; a smaller number, one with a grimmer context; it seemed to be getting brighter. Slowly, then gradually that shit started blinking like crazy. It was my legacy Ops expiry date. Lovely.

Back in 2004, I took a year off. At the end? I still was not back; at least not all the way back. Eventually, I finally figured out that large batches for rest and relaxation to bounce back is no different than any other high stakes game/monolith. It is a fools errand. Instead I resolved to keep many music projects as the small batch activity that gets my head and heart right. Having a pathway to frequent full expression in something outside coding/IT is a lifeline. Maybe starting an Arts Camp to help others balance burnout is in my new future. I was deeply moved from discussions about burnout. So this background makes the humanism, empathy and diverse personal and technical excellence of this cultural shift to DevOps completely compelling to me. Its a cool unending journey and the only way to work now.

Tools Culture and the D Word

There was one request for the team: no one should use the D word. Ever. This is because our context was, in fact, the antithesis of DevOps: it was DevOpposite. The term, this thing of ours, or what we are up to, seemed more appropriate because it was behavior based. After a couple months, I realized that it loosely translated to La Cosa Nostra. Fine, I guess. that fits.

As an agnostic, I get that tools are like furniture in the room, sure they are important but the culture and vibe are paramount. Vibe is the room. Screw that up and all the tools and catchphrases in the world will not save you. In this latest project, there was no money for software (or anything really) so it was easier to work on culture and how work was done: automation, config management, small batches, de-coupling, killing off wack workarounds and tech debt, making work repeatable and safer, making all the things production-like, better telemetry, easing team deployment pain, not shitting on other teams. Aside from this, improving culture is a cheaper easier thing to work on but it takes more energy, attention, and patience. I discovered it takes time to flourish, is somewhat fragile in the nascent period and there needed to be a weeding of the garden. In short, great progress can be undermined by shittyness coming from anywhere. There needed to be a why behind everything we were doing. I was learning it wasn’t so important what we worked in but how and why work was done. It wasn’t enough that we were up to building this thing, the team and managers needed to be brought along.

Short story, this project eliminated 1M in spend by shifting to open source. I believed in what we were doing and more importantly why we were doing it but getting others completely onboard was proving to be a challenge. I knew I was building myself out of a gig, moving the client from expensive proprietary solution to a better less expensive one but that is my role. At every step I needed to automate (within reason) or make easily repeatable all my tasks. In retrospect, it was foolish to think this view was shared as deeply. Lesson: neglecting attention to critical human aspects was no different than wiffing a huge technical detail.

There was great progress in the culture but the tools implementation lagged. I say this as a woeful participant in what had to be the central bank of technical debt. It was soul crushing and created a learned-helplessness in the midst of core chronic conflict (I was in what amounted to dev). So tools and how work is done is also important. The familiar paradigm of proprietary software, it’s creaky and antiquated notions, its processes and limitations, shouldn’t seep into this new awesome thing we were building. Open source implementation, beside scads of technical details, mandated a new framing of how to do everything. The takeaway: tools and these new behaviors have to be part of the conversation and can be the way out. At assignment’s end I found out that one of my fave engineers had installed GitLab and it was prod but nobody was using it. I put repos out there for the projects I was finishing up before rolling off (the contract ended early due to austerity), shared these docs for all the code/scripts and ansible playbooks and hoped for the best.

Books May Not Save You But Lunches Might

For all of the Open Source projects I was involved in I created a reading list that could inform the project. For deploying Zabbix and creating a more resilient infrastructure I chose two books: Antifragile and The Art of Monitoring. In addition I used Dr. Forsgren’s Accelerate, Site Reliability Engineering, and The DevOps Handbook and Effective Devops as the touchstones of our direction and wrote a great deal around the choices being made. I figured if I read and tried to replicate concepts and behaviors of high performing IT teams we could adapt these patterns to our needs and go towards a better way. After all, if you are taking your guidance from the very people that studied high performing IT organizations for years and created rigorous systems to measure them and had quantitative data/findings, at least you have a shot a successfully emulating them, right?

Make the Work Visible

I was inspired by Dominica DeGrandis’ work. I had to hold myself accountable to getting work done and having that visible to all. Management and the team would have visibility to WIP, stuck things (elevate the constraint), planned things, done work (really done) and the like so I created some Trello boards (thanks @patrickdebois!). Folks dug it but it wasn’t used as much as I hoped. How could it? When your ass is on fire, these things get dropped out. I mostly ended up doing for real-time visibility would reduce the interruptions of chaos around fires and unplanned work. In retrospect, it was just a way to avoid time-theft and be left alone but stay accountable.

I was totally inspired by the Phoenix Project (gave a copy to my direct report) and thought there would be a way to embrace change and move toward the Third Way. The text brilliantly exemplifies in novel form everything I have experienced. We could collaborate with my old tribe in Ops (DBAs) and get their feedback and input so what being built didn’t get thrown over the wall if they ended up able to support the project. An important project activity was constant feedback from teams, especially one senior engineer, on what worked on their projects and, more importantly, what didn’t so it could be avoided. This feedback was so important. We built the first MariaDB server together, but first we went out for tacos together and had some laughs. Conventions for things like filesystems, monitoring, security would be baked in from the start. Collaboration was from the start our informal agreement. Noticeably, when accustomed to waterfall, the early collaboration and inclusiveness of this approach toward the Third Way has an almost transcendent affect on vibe. It becomes more creative and fun for everyone. I can’t express how similar this is to being in a musical ensemble context and having a blast in the studio or on stage. In both, all is peachy until some bullshit happens. It most always involves egos and recovery is Sisyphean. Keep vibe happening and that shit out of the picture and all stays groovy. Sage advice from @ajacob regarding the No Asshole Rule at Mountainview 2011 Velocity will always be present for me. Still learning that lesson everyday.

What’s Next

I have so many ideas and have been working on a couple of coding and music projects, spending most waking moments studying/learning, and questioning. I am clear about only one thing, life-long learning; learning every moment and day, with a great art-life balance is the highest objective.

Peace Love and Low Funky Tones