steveheadshot.jpg

Hi.

Welcome to my blog. I talk about all things tech & leadership.

On Risks and Masks

On Risks and Masks

A hallowed institution where really big judges work.
True story: One time I walked up to the door of the Supreme Court and knocked on it. A man with a hat told me to move along. I obeyed.

On Monday, April 18, federal US District Judge Kathryn Kimball Mizelle in Florida overturned the CDC’s mask mandate for air travel and in airports. I don’t know exactly what time this happened, but I can tell you this much: it was between 2:15 pm and 5:49 pm Eastern Daylight Time. How would I know that? Well, because I was on JetBlue flight 1221 from Boston to West Palm Beach, Florida at that time.

When I left Boston, masking was almost universal in the airport, aside from folks actively eating or drinking the admittedly weak offerings in Logan Airport’s terminal C. I myself had a disappointing bagel from Dunkin Donuts. The flight was delayed somewhat significantly; first due to an FAA hold on air traffic in Florida, and then because of a maintenance issue that was discovered literally while we were on the runway. (Interesting note: this is the second time that a plane I was on aborted takeoff because the plane was broken. Cool beans.)

Unlike some other stories that came out about people en route when the decision was made, once we were airborne the flight itself was uneventful. I spent most of the time listening to podcasts or writing the second installment of the “Thoughts on Mentoring” series I just concluded. Sometime while we were over the eastern US coastline, a judge decided that the CDC isn’t allowed to mandate masks in public transport settings or something to that effect. Honestly, I cannot be bothered to know the actual statutes used in this hyper-political argument – and it’s (I promise) not relevant to today’s topic. Unlike other stories I’ve heard, no one from JetBlue came over the PA and told us we could take our masks off and so the flight went very much like other flights I’ve been on since March of 2021 when I allowed myself to travel again.

Everything seemed normal right up until I stepped off the jetway in Florida.
First lizards running around, and now this?

When I got off the plane, I noticed that the captain - who was greeting people as they disembarked - was not wearing a mask and I thought that was odd. Generally, I’ve seen inflight crews be very diligent about modeling the required behavior, so this was a noticeable departure that I was thinking about as I walked down the jetway into Palm Beach International Airport. As I was wondering what the pilot’s deal was, I looked up to see something startling for someone residing in mask-compliant Boston: an airport comparatively bereft of masks. The signs were everywhere saying that masks were required, but it seemed like no one cared about that at all, because they were exposing their mouths and noses with relative abandon. It wasn’t until I made my way outside and got into the Lyft to my hotel that I checked the news to see that a judge in Florida had just upped my anxiety factor considerably.

I should make it clear at this point that I’ve been very, very cautious since Covid came on the scene in March of 2020. I have had germophobic tendencies since well before the pandemic hit. Not that you’re my counselor, but it probably comes from growing up in a home that could charitably be considered “unkempt.” Regardless from where it comes from, finding myself in a situation where germs were decidedly the enemy for two years – my guard has been up.  And with one judge’s seemingly sudden ruling, I found myself 1,450 miles from home and no path back that fit nicely with my particular neuroses. Going home in a few days was going to be riskier than I expected.

Incidentally, I have been doing a lot of thinking about risk recently. As the chair of my employer’s Change Advisory Board (CAB), I spend some amount of time each week vetting and considering various infrastructure and application changes that carry varying degrees of risk. To boil it down, we categorize changes as either low, medium, high, or critical risk, and I’ve often debated with other CAB members and change request submitters about whether something’s risk is properly categorized not. There is no objective measurement for risk. Something that may seem unacceptably risky for one may fall comfortably within another’s risk tolerance; it truly is in the eye of the beholder. It’s not hard to understand why investment advisors are constantly asking about clients’ ability to tolerate risk.

It's easy to see that in any organization larger than a sole proprietorship, there needs to be a common understanding of what the level of acceptable risk is. For sure, industry plays into this, with a large multinational financial services firm likely being on the more conservative end, and a small graphic design firm potentially being comfortable with living more on the edge. However, with no universal standard to point to, how do you define where the line is to ensure that an IT shop is using some sort of objective benchmark instead of leaving it to each various staff member to try to harmonize their own risk tolerance with that of their employer? First, you have to break down risk into its components.

In our shop, we have defined risk as being the product of both the likelihood of an adverse event or condition happening and how severe the impact of such an event would be. This can be represented by the simple chart below:

This isn’t the exact chart we use in my office, but it’s close enough to give you an idea of what I’m talking about.

You’ll notice that this is broken down into the impact it could have (across the bottom) and how likely an adverse impact is down the side. The chart is pretty intuitive, as you should be able to at least generally conceptualize the probability of something going wrong with your task. Once you have that, you should be able to move across the chart to the right until you find the column that best describes how impactful it would be if it does go bad. For instance, if there is a 10% chance your firewall upgrade could go wrong, but if it does it will take down your entire company’s public ecommerce presence where 90% of their revenue comes from, that would be a “somewhat unlikely” event with a critical impact meaning it would be a “High” risk.

I like this chart because you can change the percentages assigned to each likelihood, as well as how you define a particular severity, so you can customize the chart all you’d like. Once you’ve done that, you can train your team to use it, and you’ve started to get everyone defining risk the same way, which means less debates about what is or is not outside your company’s risk tolerance.

The other thing I love about this chart is that it forces people to not become too focused on one of these components. I cannot tell you how many conversations I’ve had with people about how risky something is only to hear an argument focused solely on one aspect. Yes, it may be highly unlikely that something will go wrong, but if the severity of that thing happening means that the company is taken offline - I would argue that you should take the requisite precautions and develop appropriate mitigations for a high-risk scenario. To make it simpler, the likelihood of a professional juggler dropping a running chainsaw would be quite low – they’re professionals, after all – but I think we can all agree that juggling chainsaws is a high-risk activity because dropping one would very likely be catastrophic.

“Oh, someone opened up the conference bridge? That’s great. I’m kind of going to check on my family first though. Then maybe few other things. I told you we should have moved to the cloud.”

Of course, I’ve seen arguments go the other way as well – that should something so infinitesimally unlikely to happen actually happen, there would be a catastrophic event. For instance, I was once across the table from an IT auditor (who happen to be lovely people, by the way, IT auditors) who kept asking me what our recovery processes were for increasingly catastrophic events. For instance, he started by asking what would happen if one particular server died, and then if the whole rack was damaged, and then if the whole datacenter was flooded. I had documented plans for all of these scenarios, with varying degrees of recovery time and cost. Then he asked me something that stopped me. He asked, “What would you do if the whole campus was destroyed?” I looked at him for a while, and then said, “I’d go home and make sure my family was safe.”

This was many years ago, when online learning was a fringe idea and the thought that we’d have any reason to recover if our entire campus was destroyed - and therefore unable to host and teach students - caught me as absurd. If you’re a local theme park and the whole thing is destroyed by a meteor hit, you may as well realize that being able to restore your database may not be super high on the list of priorities for your owners. In these extreme cases, I believe it’s OK to say that it’s acceptable risk and that you do not have a recovery plan. Your company may very well cease to exist; data loss isn’t super relevant if the entity is unable to proceed as a going concern.

I, like many other IT folks who have come up through different infrastructure disciplines, tend to be pretty risk averse. I’ve often thought that this has made me a natural fit for someone responsible for a larger organization’s datacenter and IT operations. I don’t take a lot of uncalculated or unnecessary risks in my personal life, so I naturally don’t take them in my professional life. Of course, there is a lot to lose when being too risk averse. Just ask anyone who knows anything about investing – if you put all your money in a hole, you end up losing purchasing power very quickly, thereby eroding your financial security.

The same is true in technology. If you never take any risks, you’ll eventually look up and be surrounded by legacy systems that are either losing – or have already lost – interoperability with modern services. Imagine if you were afraid to ever move off of your old Windows NT systems back in the day, or Windows 3.11 before that. Your entire organization would likely be decimated by cybersecurity attacks, and you wouldn’t even be able to buy a printer that works for your organization. Sure, upgrading may have some risks - but there is always risk in staying where you are, too. Don’t just bury your technology stack hoping that it stays safe and worthwhile.

To return to my conundrum in Florida, I evaluated the changes to the risk levels and acted accordingly on my flights. I used a high-quality N95 mask that seals to my face to prevent any airflow from getting around the mask and opted not to eat or drink anything on the flight at all; I decided to keep it in place for the entirety of the flight, which reduced the likelihood of contracting anything on the flight. I also had to rationalize that as a vaccinated and boosted adult under 50, the severity of a Covid infection would most likely be mild. By reducing both the likelihood of the adverse event and understanding it would most likely not be severe, I was comfortable flying.

I apologize to all of these people.
Well, except that one guy who took his socks off on the plane… I’m not sorry to him.

In the end, what I did on my flights was all a moot point because it turned out that I was the “walking biological weapon” and what people really needed was protection from me. I tested positive for Covid-19 the morning after I got home, despite testing negative the day before. And while my course has been mild and I’m well recovered, I am hopeful that my high-quality mask protected everyone around me on my flights. Because in the end, they also had a make a risk calculation, and I hope I didn’t mess theirs up.

Questions for reflection:

  • How do you personally define risk? Does it deviate from how your company defines it?

  • How likely does a catastrophic impact have to be before it’s too likely for your risk profile?

  • What are the risks of taking “no” risks? How do you account for that in your decision-making process?

Lessons From My First IT Crisis

Lessons From My First IT Crisis

Thoughts on Mentorship (Part III)

Thoughts on Mentorship (Part III)