The Never-Ending P1

Don’t worry, this diagram is explained in thousands of pages of documentation

As a computer systems administrator at a large organization I was introduced to a library of procedures and best practices for managing technology services called ITIL that originated in the mainframe era. ITIL is huge, complex, unwieldy, and bureaucratic. It is also a brilliant and incredibly useful resource for understanding how to manage and optimize large systems. And I am very much a fan of optimizing large systems. It’s sort of my jam.

I mention this because at a contention and stressful meeting a few days ago I was accused of being a “process person,” the implication being that I was out of touch with the reality of the situation. The situation being how to make reasonable business decisions during a global pandemic of a scale unprecedented in the last century.

Our business is keeping our community healthy and safe through effective cleaning, and that has never been more important than right now. There has been a huge surge in demand for a scarce pool of cleaning and personal protection supplies. Every day we are overcoming countless challenges to source and stock products, to manufacture disinfectants and other chemicals, to take orders, manage expectations, and to make deliveries, all while ensuring that our staff are kept safe from unnecessary risk.

In the ITIL model, incidents are classified on a scale based on urgency and impact. Our business (and our society as a whole) is facing a priority 1 incident, and, as a senior Systems Engineer, I am very familiar with P1s.

In ITIL, service requests are prioritized based on urgency and impact

How do you typically handle a P1? First you wake up, because it is probably the middle of the night. You push away the grogginess and embrace the adrenaline rush. You assess the situation, triage as best you can, then figure out who else needs to be woken up and how to brief them. Next you work to analyze the problem, determine the corrective action, and implement the fix. Then you monitor the result.

It is rare in the sysadmin world for a P1 to last more than a few hours, or at most a few days. But dealing with the business impacts of coronavirus feels like a P1 that never ends. I’ve been on that groggy/adrenaline combination for a couple weeks now. I can’t sleep at night, and I can’t stop thinking about all the things we need to do.

When I moved to Maintex three years ago and started learning about the many aspects of running a business, it quickly became clear to me that process could solve a lot of problems, but also that my biggest problem was understanding all the things that cannot be easily formalized into a process. The physical world is rife with variables a technologist would not expect.

The physical world is rife with variables a technologist would not expect.

Fifty thousand bottles from a trusted supplier are fine, then suddenly a few hundred start leaking inexplicably. A step is missed in a sanitizing procedure and an entire 5,000 gallon batch of product is contaminated. A forklift driver accidentally smashes a fire sprinkler, flooding the loading dock. A shipping company returns an entire trailer full of product because a labeling machine with a dirty optical sensor placed a regulatory label two inches askew. An inspector with an imperfect understanding of building codes delays a project by 3 months before inexplicably approving it.

ITIL is a helpful framework for me because most other business processes are designed for incremental improvement, not crisis management. But a process is only as good as the people who implement it.

In the physical world, and in the world of people, the unexpected is routine, and no amount of checklists or procedures can account for every possible variation. So I have spent three years implementing systems and analyzing data, sure, but also learning what it means to manage an organization made up of people.

Attempting to maintain business continuity during a pandemic is like a P1 incident that never ends. And that means it is a problem I cannot just solve and then go back to sleep. Every day we need to make decisions and trade-offs that are uncomfortable and might not be the right ones, simply because a decision has to be made so we can keep moving forward.

A lot of my job is to make space so that the smart, hard-working people that work for me can do their jobs.

But most importantly for me as a manager, this P1 requires stepping back. Taking a breath. Checking in with staff to make sure they are okay. Listening to their concerns and figuring out how to help them to clear obstacles. A lot of my job is to make space so that the smart, hard-working people that work for me can do their jobs.

And that is decidedly outside my comfort zone.

But I’m finding it just as exhilarating and even more challenging than doing things on my own. Most of all, dealing with this crisis, and working with this team feels more consequential and important to me than any IT incident I’ve ever participated in. The coronavirus is an incredibly daunting challenge for everyone, and the effects are rippling across society and business in ways that are scary and uncertain. The best we can hope to do in our little corner of the economy is continue to conscientiously perform the valuable service of delivering critical cleaning and safety supplies where they are most needed.

I’m proud of Maintex, and of our team. I’m proud of the work we are doing to help keep society resilient in the face of an unprecedented crisis. And I’m more sure than ever that leaving my former career to start over was the right decision.

Tortoise-centered design

Hacienda de la Tortuga sits on a relatively flat 1/4 acre that came mostly devoid of foliage. Kip’s pen was constructed of chain link fence, as was our land boundry on one side. Mountain peaks poked tantalizingly through our dense ash trees. While the space is large, it felt penned in by the neighboring properties.

Taking inspiration from the British garden show Big Dreams, Small Spaces and wanderings around our neighborhood, we decided to pursue a reasonably priced transformation. Our goal was to gain a lush natural environment that would feel larger and more isolated. We wanted native flowering plants and grasses that would attract birds, bees, and butterflies. We were looking for something low maintenance and water-wise with meandering paths and hidden surprises.

Local landscape designer Susanna Dadd, who specializes in climate-appropriate “habitat gardens,” brought just the right perspective to this journey. She immediately identified our biggest X factor: Kip the tortoise. Many plants were out of bounds because they are dangerous to tortoises, and others wouldn’t work because he would quickly devour them before they had a chance to grow.

Kip explores his new maze-like environment

Sue’s design incorporates low-walled “islands” and “meadows” that serve to give Kip plenty of edible food and enrichment while allowing us to gain the dense, colorful vegetation we desired. To keep costs down, she found steel sheeting to act as the barriers, which will rust to a lovely brown patina over time. We also built a new redwood fence in front of the ugly chain link, and got the ash trees thinned and cleaned up.

This being hot California, there was no avoiding irrigation. Most of the garden has a sophisticated drip system, while some of the more open areas where Kip roams (and might chew the hose) have traditional sprinklers.

After a month-or-so of work by two very diligent gardeners, the yard is mostly done. We are incredibly excited to see how it grows and develops as the new plants fill in and others are added as the season progresses.

I also built Kip a new tortoise house with a nice heat lamp to keep out the chill. It is made of recycled material from the old owners’ garden shed.

Kip’s new hacienda

We were also able to reuse all of the old vegetable boxes in a new horseshoe arrangement. Next week we will get some veggies going, and before long we will have an edible garden for humans as well as tortoises!

Adventure

Snow day

In the new year I’ve resolved to post updates more regularly, starting with a few interesting things from the past month or so. This adventure took place on November 30th, 2019.

I have been back in California almost three years now and I’m still not used to the level of geographic and climate variation on display. I can stand in my yard in a t-shirt and look up at the snow-covered peaks of the San Gabriel Mountains. The end of November saw heavy storms and snowfall, so Katy and I drove up the Angeles Crest Highway to see what we could find.

It did not take long to gain a few thousand feet of elevation and enter a whole different world.

Snow in the Angeles National Forest

We didn’t have a specific destination in mind, so we stopped at a couple pull-offs to marvel at the white stuff. There were plenty of other people doing the same, many quite ill equipped for the weather.

The obligatory selfie

We stumbled upon a neat hiking trail and followed it a short way, but sans map and wearing only jeans, we quickly decided to backtrack.

We had to dig deep in our closets to grab puffy jackets and warm caps, but we weren’t committed enough to get fully decked out for adventure. But perhaps next time!

We turned around and just spent some time driving and marveling at the lovely scenery. Some views were breathtaking. With all the scrub brush and smaller trees it is a very different experience than, say, Vermont or New Hampshire.

We stopped one last time to romp through one particularly nice patch of fresh deep snow. Were snowballs thrown? Of course they were.

An hour later, we were back in the relative warmth of Pasadena.

Women and people of color, of course, were originally outside the protection of [America’s] founding documents. But on Wednesday, the most diverse Congress in history declared that even the most powerful white man in the world should be bound by them. When Republicans act as if that’s a sacrilege, they show us what they worship.

Michelle Goldberg, “The Tyranny of the 63 Million”

Adventures in chimney replacement

We purchased Hacienda de la Tortuga knowing that the circa-1928 chimney was no longer fit for purpose. The chimney is built with a single brick thickness and some of the bricks and mortar had deteriorated due to age and use. Because the structure is not reinforced, an earthquake could send it tumbling into the house or onto the neighbors driveway.

A chimney liner is a flexible or rigid metal tube that is placed into the existing chimney

A traditional fix for this sort of problem is the installation of a metal chimney flue “liner”, a relatively straightforward and inexpensive operation. But due to the narrowness of the chimney and the lack of reinforcement, a liner alone would not solve our problem.

The two masons we consulted both said we needed to tear down the chimney and either seal the fireplace (making it non-functional) or rebuild the chimney at great expense. Neither option was particularly appealing, but we were also unwilling to give up on the charm and comfort of a fireplace.

How a direct vented fireplace works when vented through a rear wall

Having lived on the East coast, I was convinced that another approach would be easy and cost-effective: tearing down the existing chimney and installing a direct venting fireplace instead. While standard wood burning fireplaces are dirty, inefficient, and actually pull heat out of the house, direct vent units are sealed, clean, use natural gas for fuel, and provide substantial heating capacity. Plus, they do not require a chimney!

And this is when I started to think I was losing my mind. While I was convinced that direct vent was the way to go, the local “experts” I consulted were dismissive of the option. They told me direct vent fireplaces are ugly, cheaply made, and inflexible, since you cannot use them to burn wood or to put in other decorative elements. They said that venting through a masonry wall was not possible and would not meet code requirements. I could only find one nearby store that even stocked direct vent units; even there the salesman tried to discourage me from buying one. And if I had bought it, would I have been able to find someone to install it?

I could only find one nearby store that even stocked direct vent units; even there the salesman tried to discourage me from buying one.

Direct vented fireplaces are sealed units that can look quite nice when properly installed

Eventually I identified a local company that was willing and able to reinforce and line the existing chimney for about half the price of a full replacement. That work was completed right before Thanksgiving and, pending final city approval, we now have a functional fireplace that can burn wood.

As I write this on a 40ºF winter morning, I still regret that the direct vent gas option was not possible. It would be nice to have a crackling fire accompanied by heat. At least I know I’m not crazy — plenty of YouTube videos, including one by This Old House, espouse the benefits of the approach I was pursing.

California is a land of paradoxes, and this strange fireplace journey is just another one to add to the list. In an era of climate emergency, why wouldn’t the state be encouraging and even incentivizing the replacement of dirty wood-burning fireplaces with cleaner natural gas alternatives? I wonder how often we will actually burn wood, and how much the heartburn about our contribution to atmospheric carbon pollution will offset our enjoyment of the crackling flames.

★☆☆☆☆
Review

A Day With the New AirPods Pro

The original AirPods would not stay on my ears, which is an important prerequisite to using headphones. The new AirPods Pro utilize a proper in-ear design with changeable tips. There is a clever built-in test that plays audio and uses multiple microphones to advise on fit.

Having completed the pairing and calibration, I went for a run on a cool morning. It was a bit confusing to determine how the three modes work — normal, noise-cancelling, and “transparency,” which is supposed to provide the benefit of noise cancellation while also making voices and traffic noises audible. Transparency was particularly awful while wearing a hoodie, as every rub of hood against AirPod stem was loudly transmitted. Sometimes I heard approaching cars quite loudly, but then when a car was pulling out of a driveway right ahead of me, there was no sound at all.

A mile in, I wiped the sweat from my brow with the back of my hand, a common and subconscious action. In the process, I somehow jarred loose the right AirPod, which went flying to the ground. Luckily I did not step on it, but it did get pretty dirty. And when I put it back in my ear, it took a lot of fiddling on the phone before the music would start playing again.

Near the end of my run, I took off my sweatshirt. In the process, both AirPods came loose and fell off. That did it — another very expensive and impractical gadget is going back to the store, and I’m going back to my BackBeat Fits. They don’t sound as good and the case is bulky, but I’ve never worried about them falling out or getting lost.

Rock climbing and the economics of innovation

Because reality has a surprising amount of detail, a quick overview of the many innovations and advances in equipment, technique, travel, and knowledge that were needed before Alex Honnold was in a position to “free solo” El Capitan. Everything builds on everything, and it’s turtles all the way down.

This new abnormal is a pernicious and dangerous departure from the time when the United States sought a leadership role in designing and supporting global agreements that advanced a safer and healthier planet. The new abnormal describes a moment in which fact is becoming indistinguishable from fiction, undermining our very abilities to develop and apply solutions to the big problems of our time. The new abnormal risks emboldening autocrats and lulling citizens around the world into a dangerous sense of anomie and political paralysis.

It is still 2 minutes to midnight, Bulletin of the Atomic Scientists

Introducing Hacienda de la Tortuga

My move to California meant giving up an old house by the sea in favor of modern apartment living. And while apartments have much to recommend them, I missed living in (and owning, and DIYing) my own house. (This mostly manifest in the form of nightly binges of home improvement and woodworking videos on YouTube.)

After getting married in June, Katy and I decided to start looking for a more permanent residence. We stuck to the Pasadena area — which is overflowing with charming Craftsmans — and the surrounding cities. Alas, none of them were in our price range, but after much looking and some stressful negotiations we were able to acquire a cute little bungalow in the Spanish Colonial Revival style.

Our new home is a single-story stucco structure with a flat roof trimmed in clay tile. The lot is a flat 1/4 acre on a quiet street within walking distance of restaurants, a grocery store, and several Armenian bakeries. There is a single car detached garage that is just perfect for a small workshop. And best of all, the house came with a reptilian occupant — a 25 year old, 65 pound sulcata tortoise named Kip!

The previous homeowners adopted Kip after he was found abandoned at the nearby Huntington Library in 2004. He seems pretty active on warm days and more pokey on the cold ones. We’ve been feeding him lots of leafy greens and vegetables and he seems content.

We are planning a few major projects that involve professionals, like replacing outdated sewer systems and old windows, as well as lots of small projects that we can tackle on our own. Katy has managed much of the unpacking and organizing while I’ve been installing new locks, shower curtains, smoke detectors, speakers, and the like.

Surprisingly all of the YouTubing seems to have paid off a bit, or maybe it’s the benefit of all my experience in Hull over the years, plus a bit more patience and planning. So far I haven’t put any holes in walls that weren’t intended, haven’t electrocuted myself, and haven’t needed to make unexpected hardware store visits. I seem to be getting marginally better at patching walls, fishing wires, and adjusting stuck doors.

Our first night here was rough — we felt unsettled, we were surrounded by boxes, there were swarms of ants everywhere, and none of the light switches did what we expected. A week later we are starting to feel more settled and more at home, as well as more confident that this wasn’t a huge mistake.

We are working on coming up with a silly house name, and I’m advocating for Hacienda de la Tortuga. Hacienda for the home’s Spanish heritage and our plans to make it a homestead with vegetable gardens, chickens, and who knows what else. Tortuga for obvious reasons. But Katy would like to incorporate our gatos into the name as well. I guess we’ll keep thinking about it. We plan to be here for a while, so there is no rush, and in the meantime there is so much else to do!

The Myth of the Wealthy Welder

Dissecting a conservative myth that contends middle-America tradespeople can be as or more successful than college-educated city dwellers in the modern economy. In reality trades are rarely lucrative, and the underfunding of public education means that trade schools leave students deeply indebted even as they face a lack of opportunity.

Politicizing the weather

Trump’s Sharpie vs. reality

As all levels of government are subverted and corrupted at a pace few would have thought possible, even something as fact-based on weather predictions are subject to the lies and whims of a whiny and narcissistic chief executive:

“It makes me speechless that the leadership would put [Trump’s] feelings and ego ahead of putting out weather information accurately,” said Michael Halpern, a deputy director at the Union of Concerned Scientists. “If we’re politicizing the weather what is there left to politicize?”

NOAA staff warned in Sept. 1 directive against contradicting Trump“, Washington Post

The corruption of any notion of objective, agreed-upon reality is the point. The breakdown of social cohesion is the point. Once even the weather is a political opinion, the fabric of society is so ripped asunder that unity and common purpose may never again be achievable.