‘Out at Sea’ (and back)
I have to credit my friends in DevOps for totally changing my life, specifically John Willis. I’m deeply grateful. He encouraged, mentored and hipped me to DevOps and all the super-fly people in it (to numerous to mention here). When I did the music wayback for his Cloud Cafe’ podcast, I needed to look up exactly what a ‘cloud’ was. I was just pumped to have met John and was curious. All of the below is inspired and I can attribute to the ideas I heard or learned from the women and men in DevOps. When participating in DevOpsDays it is truly a pilgrimage for something missing. After so many years in Dev and Ops, there was finally inertia towards making life in tech a better human experience, a better existence. The work cultures I see at companies now make business cultures I remember in the 90s and 00s look ridiculously lame by comparison. Now, there are well known fine examples of how to be high performers in IT without breaking people. This is significant and that behavior and culture has a name. Dope!
In production support/Ops, ridiculously stressful projects/incidents manifest ‘hidden’ casualties at home (read families) just by proximity to a toxic work culture. Willis’ sea analogy was absolutely true for me. Inside a buried legacy silo in no one can hear you scream. The takeaway: even if ‘successful’, a great rate, a cool team, and building dope things with a great purpose, the eventual cost of burnout can be too damn high. Previously, every time things got a bit crispy (e.g. lead an off-shift build team), I would take time off and try to rejuvenate. As an independent contractor (ICCA member) in LegacyLand (Unix/ SAP/ Siebel/ AIX/Infrastructure/DB2) this was somewhat easy. Independent Consulting was supposed to be a hedge against toxic culture, psychopaths and traumatic on-call. One could always walk away, right?
Not quite.
In my experience, clients would hang on to a performant indies like grim death and roll the contracts or more often arrange better ones. A big takeaway was no matter how much you were banking or planning, sometimes the burn goes too damn deep. If time-off is deferred too long, there may be no coming all the way back anytime soon. In a way though, I thought at the time, isn’t that the deal I signed? I accepted and did not lament this reality of having a number on my forehead, except after a while that number wasn’t alone. It wasn’t just my hourly, there was this other number; a smaller number, one with a grimmer context; it seemed to be getting brighter. Slowly, then gradually that shit started blinking like crazy. It was my legacy Ops expiry date. Lovely.
Back in 2004, I took a year off. At the end? I still was not back; at least not all the way back. Eventually, I finally figured out that large batches for rest and relaxation to bounce back is no different than any other high stakes game/monolith. It is a fools errand. Instead I resolved to keep many music projects as the small batch activity that gets my head and heart right. Having a pathway to frequent full expression in something outside coding/IT is a lifeline. I was deeply moved from discussions about burnout. Maybe starting an Arts Camp to help others balance burnout is in my new future. So this background makes the humanism, empathy and diverse personal and technical excellence of this cultural shift to DevOps completely compelling to me. It’s a cool unending journey and the only way to work now.
Tools, Culture, and the D Word
There was one request for the team: no one should use the D word. Ever. This is because our context was, in fact, the antithesis of DevOps: it was DevOpposite. The term, this thing of ours, or what we are up to, seemed more appropriate because it was behavior based. After a couple months, I realized that it loosely translated to La Cosa Nostra. Fine, I guess. that fits.
As an agnostic, I get that tools are like furniture in the room, sure they are important but the culture and vibe are paramount. Vibe is the room. Screw that up and all the tools and catchphrases in the world will not save you. In this latest project, there was no money for software (or anything really) so it was easier to work on culture and how work was done: automation, config management, small batches, de-coupling, killing off wack workarounds and tech debt, making work repeatable and safer, making all the things production-like, better telemetry, easing team deployment pain, not shitting on other teams. Aside from this, improving culture is a cheaper easier thing to work on but it takes more energy, attention, and patience. I discovered it takes time to flourish, is somewhat fragile in the nascent period and there needed to be a weeding of the garden. In short, great progress can be undermined by shittyness coming from anywhere. There needed to be a why behind everything we were doing. I was learning it wasn’t so important what we worked in but how and why work was done. It wasn’t enough that we were up to building this thing, the team and managers needed to be brought along.
Short story, this project eliminated 1M in spend by shifting to open source. I believed in what we were doing and more importantly why we were doing it but getting others completely onboard was proving to be a challenge. I knew I was building myself out of a gig, moving the client from expensive proprietary solution to a better, less expensive one but that is my role, right? At every step I needed to automate (within reason) or make easily repeatable all my tasks. In retrospect, it was foolish to think this view around automation was shared as deeply. Lesson: neglecting attention to critical human aspects was no different than whiffing a huge technical detail.
There was great progress in the culture but the tools implementation lagged. I say this as a woeful participant in what had to be the central bank of technical debt. It was soul crushing and created a learned-helplessness in the midst of core chronic conflict (I was in what amounted to dev). So tools and how work is done is also important. The familiar paradigm of proprietary software, it’s creaky and antiquated notions, its processes and limitations, shouldn’t seep into this new awesome thing we were building. Open source implementation, beside scads of technical details, mandated a new framing of how to do everything. The takeaway: tools and these new behaviors have to be part of the conversation and can be the way out. At assignment’s end I found out that one of my fave engineers had installed GitLab and it was prod but nobody was using it. I put repos out there for the projects I was finishing up before rolling off (the contract ended early due to austerity), shared these docs for all the code/scripts and ansible playbooks and hoped for the best.
Books May Not Save You But Lunches Might
For all of the Open Source projects I was involved in I created a reading list that could inform the project. For deploying Zabbix and creating a more resilient infrastructure I chose two books: Antifragile and The Art of Monitoring. In addition I used Dr. Forsgren’s Accelerate, Site Reliability Engineering, and The DevOps Handbook and Effective Devops as the touchstones of our direction and wrote a great deal around the choices being made. I figured if I read and tried to replicate concepts and behaviors of high performing IT teams we could adapt these patterns to our needs and go towards a better way. After all, if you are taking your guidance from the very people that studied high performing IT organizations for years and created rigorous systems to measure them and had quantitative data/findings, at least you have a shot a successfully emulating them, right?
Make the Work Visible
I was inspired by Dominica DeGrandis’ work. I had to hold myself accountable to getting work done and having that visible to all. Management and the team would have visibility to WIP, stuck things (elevate the constraint), planned things, done work (really done) and the like so I created some Trello boards (thanks @patrickdebois!). Folks dug it but it wasn’t used as much as I hoped. How could it? When your ass is on fire, these things get dropped out. I mostly ended up doing for real-time visibility would reduce the interruptions of chaos around fires and unplanned work. In retrospect, it was just a way to avoid time-theft and be left alone but stay accountable.
I was totally inspired by the Phoenix Project (gave a copy to my direct report) and thought there would be a way to embrace change and move toward the Third Way. The text brilliantly exemplifies in novel form everything I have experienced. We could collaborate with my old tribe in Ops (DBAs) and get their feedback and input so what being built didn’t get thrown over the wall if they ended up able to support the project. An important project activity was constant feedback from teams, especially one senior engineer, on what worked on their projects and, more importantly, what didn’t so it could be avoided. This feedback was so important. We built the first MariaDB server together, but first we went out for tacos together and had some laughs. Conventions for things like filesystems, monitoring, security would be baked in from the start. Collaboration was from the start our informal agreement. Noticeably, when accustomed to waterfall, the early collaboration and inclusiveness of this approach toward the Third Way has an almost transcendent affect on vibe. It becomes more creative and fun for everyone. I can’t express how similar this is to being in a musical ensemble context and having a blast in the studio or on stage. In both, all is peachy until some bullshit happens. It most always involves egos and recovery is Sisyphean. Keep vibe happening and that shit out of the picture and all stays groovy. Sage advice from @ajacob regarding the No Asshole Rule at Mountainview 2011 Velocity will always be present for me. Still learning that lesson everyday.
What’s Next
I have so many ideas and have been working on a couple of coding and music projects, spending most waking moments studying/learning, and questioning. I am clear about only one thing, life-long learning; learning every moment and day, with a great art-life balance is the highest objective.
Peace, Love, and Low Funky Tones
Hit me up on Twitter @devnullid or @cpitech