Lessons From My First IT Crisis
I can draw a direct line in my career back to my high school internship at a manufacturing company in Hanover, NH. I started out as a front-line desktop support intern at a company that was such a great place to work that I miss working for them to this day. When it was time for college, I ended up finding a work study job as a PC support assistant in the office of the Dean of Students. In both of these roles, my actual job responsibilities were to do the basic first tier of desktop support. Replace keyboards, fix little glitches in Windows, etc. It was a (mostly) completely uneventful IT gig; the sort where when you go home, you don’t need to think about it much until the next day. Occasionally I got to mess around with the “servers”, but always in a low-impact and generally supervised manner.
Unlike Dwight, I generally accepted being the “Assistant to “ things
This access to servers got a little more common as I grew in my role in the Dean of Students’ Office, but I wouldn’t say that I was exactly a SysAdmin at this point. In fact, I was probably closer to the “Assistant to the SysAdmin” instead of the “Assistant SysAdmin.” Not too long into this job, that SysAdmin left for another job without much notice and I ended up being promoted to the “Only IT Resource We Have” role. It was a lackluster title, I’ll grant you, but the work was mostly unchanged. After all, without a true SysAdmin - every day is a read-only Friday. Things were mostly copacetic right up until they weren’t.
One afternoon, I got a call from someone saying that they couldn’t get access to their files on the server. To be clear, we only had one server that functioned as a Microsoft Mail server and an SMB file server. I think it also spooled print jobs, but could be bypassed by simply pointing to the printer’s (no doubt DHCP assigned) IP address. I went to look, and sure enough I couldn’t access the files either. In doing a little more clicking around, I couldn’t access any files on the server. Every one gave me an access denied error.
I pretty quickly realized that I was in over my head, so I told my manager Tom (who was the Dean of Student Services) that the server was in deep trouble and I didn’t know what to do. He made a few phone calls, and the husband of one of the folks in the office (who I’ll refer to as “Doug” for anonymity) who was very early into a professional IT career was able to come by to check on what was going on with that box. By now, the office was closing up and without really thinking about it, I told folks that I’d lock up when we would leave. Doug and I beat on that system for hours. He started a call with Dell thinking the drive may be bad, and I was on the phone with Microsoft having paid the $250 for a support call with them. We broke around 8pm to eat a few slices of pizza as we discussed options. Doug was helpful as a sounding board and we both bounced ideas off of each other, but nothing was really working. At one point, we went in to check the status of the RAID array, but it seemed fine - at least according to the rudimentary controller in the server.
Everything gets fuzzy after that. I remember thinking it was getting really late and I remember realizing that we weren’t making a lot of progress and that this might not break our way. I remember being really nervous that there were no backups; the prior SysAdmin hadn’t taken the task of data protection as seriously has he probably should have. Doug and I talked about how we’d message that to the office if everything was gone forever. The hours ticked by and I was getting really tired and losing steam.
I truly don’t remember much as my next memory is taking ownership of every folder and re-ACLing the entire drive to grant me permission to read all those files (this was well before crypto-locking was a thing, but it was the same sort of inability to access files). Then I remember the sun coming up. Doug and I (a totally unskilled and untrained admin) brought that file system back online through sheer force of will right about 7:00 in the morning.
Not all heroes wear masks.
I do wear a mask, but I am not a hero. Clearly.
Not even a half hour later, the Dean of Student Affairs came in. I was so excited to show him that the server was back up and running with all the files restored (though not properly ACL’d, yet.) When I unlocked the door for him, I somehow expected him to be relieved that Doug and I were still there at the keyboard focused on saving the world (or at least the data). I don’t know why I expected a ticker-tape parade, but I was thrown for a loop when he looked at me and in a not-at-all approving tone asked, “Why are you here?” I explained that we had been at it through the night, working diligently to get everything back to normal, and it had worked! We were heroes!
The Dean of Student Affairs, Larry, listened, and then proceeded to remind me that I was a student. I wasn’t supposed to work more than 22 hours in a whole week, and I had just used them all that day. He asked me what my first class was, and if I thought I would be able to attend it. I admitted that I probably couldn’t attend my 8:30 Physics II class. I was going to be a zombie and probably couldn’t stay awake through it.
Larry looked at me with a mix of disappointment and concern that I still haven’t forgotten. It was clear that he was worried about my overall wellbeing. He asked Doug if he could finish up, and when Doug readily agreed Larry sent me home - and told me not to come to work for the next week. I needed to focus on my classes and recover from the all-nighter. As I gathered my stuff and got ready to walk out the door, I was upset. I was the hero! I had saved the day, and I wanted my dang parade!
In reality, not only could have Doug cleaned up when I left - he likely would have solved the whole problem if I had just gone home when it was getting late. I could have stuck around to help bounce around ideas until 7 or 8, and then I could have headed back to my dorm room to study and do my homework. I pushed too hard and ended up causing myself harm. There was another way to solve the problem that didn’t have to sink me. It took me a full 20 years to realize just how right Larry was - I was fortunate enough to be able to tell him so in person just recently and thank him for the lesson.
In keeping with our hero theme, the only real heroics I’ve ever done is reading the documentation really, really fast.
I learned a lot that night about how to break a problem down into components and solve each systematically and how to isolate variables. I learned how to work through intense anxiety and communicate with someone else through it to coordinate our actions for better effect. But more than that, I learned something I’ve carried with me ever since: never ever forget to be clear about expectations - and care about the person above the labor they provide.
Ultimately, that night proved to be the beginning of the end of that semester for me. This was right around midterms, and I didn’t study and didn’t complete some assignments due to lost time. I ended up needing to drop a few classes in order to not fail them. That caused me to have to re-register and retake them the following semester. There were certainly other factors in my downfall that semester, but that server recovery ended up being a very damaging evening. I saved the server, but at what cost? I probably could have graduated a semester sooner if I hadn’t thrown myself at that problem with such gusto.
In hindsight, it’s probably a good thing I didn’t get that parade.
Sometimes, people are willing to burn themselves out without being asked to. These are the kinds of people that managers will look at and think that they’re the “rockstars” of the group because they’re always going to extra mile, they’re always willing to stay late or pick up a shift for the overnight datacenter operations. This has risks though, both to the individual and the organization. Left unchecked, and without some sort of governor on it these people may flame out and ultimately stall their growth and contributions to the team much like I did that semester.
This is quite the dramatic picture, but if you’re feeling burnt out, or think you’re trending that way - listen to yourself!
If the preceding paragraph describes you, please take a minute to consider the long-term consequences of perpetually laying everything on the line. Do not trade long-term health and success for a short-term dopamine boost of being the hero. And if you’re a leader, please recognize that sometimes people will make bad decisions that at the time seem like heroics. At the end of the day, it’s on you as a leader to step forward and prevent people from burning themselves out if they won’t self-regulate. Remember, it’s almost always better for a team’s success for everyone to be at full-capacity over the long haul.
Now that I’m in a position of leadership and I have lived through multiple crises, I try to never forget the lesson that Larry taught me with his actions and words. Every IT crisis I’ve been through is temporary. Sometimes it doesn’t feel that way while you’re in it, but they’re always temporary. What’s more important than the temporary IT crises are the people doing the work to resolve them. Don’t get me wrong, it’s OK to ask people to extend themselves, it’s OK to expect a “full court press” - but it’s essential that you recognize how far is too far for your team - and yourself. I work to make sure that each member of my team feels like I am watching their backs as people and that I care about them more than the crisis.
And some day when I retire, I hope that I have someone come up to me at a reception and thanks me for teaching them that sometimes something that seems really important – even essential – might not be worth the sacrifice.
Questions for Reflection:
Have you ever extended yourself too far and harmed yourself in the pursuit of something?
What advice would you give someone early in their career who may be risking their own wellbeing to serve their company?
How can you make sure that you’re caring about your staff or team when you are in a crisis?
