Experience

or

why I run my own infrastructure

or

diversity is important

I was reading an article the other day that used an interesting turn of phrase when pointing out a particularity in Powershell and just how it can bite if you don’t understand the implications. The phrase is:

How you interpret the above depends entirely on your experience. Not your level of experience, just your experience

It’s not just about the number of years of experience you have with something, which is often translated into level of experience, but actually the accumulation of all of the individual things that you’ve encountered along the way because they are what forms your knowledge base. Especially those that have bitten you.

The most formative of these experiences are the ones you encounter in situations of crisis since the emotional reaction has a tendency to mark those memories more firmly in your brain. This is one of the reasons why I think that it’s important to own¹ and run your own bit of the stack where you live for yourself. Screwing up your own stuff gives you that emotional kick to ensure you won’t forget, but it remains a relatively low-stakes game.

What’s interesting is the serendipity that can come from these experiences. I can’t count the number of times that I have gone down some technology rabbit-hole to get something working for myself only to have a client need help on that same thing in the following months. Some examples:

• I’ve been hosting my own mail server since about 1999 and in the latest server migration (sadly OS X Server is deprecating the tools for email), I was bringing myself up to date on the latest best practices in order to ensure that my mail would work from being hosted off a blacklisted home broadband connection. So I ended up learning lots of stuff about SPF and DKIM which was new to me since I’ve been out of the active email server management side of things for a while. A couple of weeks ago, a client was having issues with some automated mail being rejected as SPAM. Digging around I was easily able to quickly identify a possible solution which turned out to work for them.

• At one point I decided to migrate a home ZFS server to a new machine with a completely different disk setup. Since this was the media server with all of my films, there were multiple terabytes of data and it became quickly apparent that the traditional SSH connection was the bottleneck for sending ZFS snapshots. Some research led me to a solution using netcat plus some buffering tools that let me saturate my gigabit network. Later that year I ended up working on a data center migration project and we used ZFS servers to stage the data and that netcat trick let me saturate 10GbE links for duplicating the data for transport.

This brings me around to a point about diversity of experience, especially when trying to fix things (which in IT seems to be at least half of the time). There’s an old joke about the policeman walking his nightly beat when he runs across someone poking around on the sidewalk near a street lamp. When asked, the man responds that he’s looking for his lost car keys, so the two of them look for a bit until the policeman asks if this is where he remembers losing them. The man responds, “no, but this is where the light is“.

That is what experience is all about: little bits of light in the midst of the darkness that is all of the things that you don’t know. Every experience you have brings a little bit more light. Sometimes by making a spot brighter and clearer with the deeper understanding of a subject, and sometimes spreading bits of light across various subjects. But you can only have done and seen so many things in your life and career. If your team is made up of people that predominantly share the same experience, they are also going to share the same darkness and gaps. Which means when you’re looking for the keys, you’re all looking under the same lamp posts.

Bringing a diversity of experiences to your project/community/team/company means that you’re not all looking around in the same lit space, you will collectively have a sprinkling of lights scattered around that can help find better answers faster. The difficulty remains in the communications since from one side, you’re hearing a voice out of the darkness, outside of your experience and often that’s hard to trust. You’ll need to build some of your own experience to light up that area, hopefully going faster with the help of someone who’s already there. In cases where you have corporate cultural rigidity we have people working inside a nicely well delineated and well lit space and beyond those walls lie dragons in the darkness so we don’t go there.

The counterpoint to accumulating experience (and one hopes a little wisdom along the way) is the knowledge that many kinds of information have a best before date and need to be reevaluated in each new situation. My favorite story that I use to demonstrate this in seminars is the best practice concerning the deployment of the VMware vCenter Server. Back in the day, (vSphere 3.x), the best practice was a physical vCenter server, mostly because it was a resource hog and would be capable of eating most of the resources on a dual-core, dual-processor server. Then came the halcyon days of vSphere 4, and the arrival of 4,6 and 8 core processors and the best practice became a virtual machine in order to take advantage of HA. Then came the first iterations on the Distributed vSwitch, which we discovered required that the vCenter be available for each ESXi server to acquire it’s configuration from a cold boot. This was exposed in places where they had annual electrical maintenance and server rooms were taken completely down and then we couldn’t bring them back up, so the best practice went back to a physical server with no dependencies on the virtual infrastructure so that we could cold boot a room. Then VMware fixed that issue by adding a local caching feature to bring up an ESXi in the last known distributed vSwitch configuration if the vCenter server was unavailable so we’re back to a virtual machine (for good now I hope!).

Much of the wisdom of experience comes from being able to sort out your accumulated knowledge into the two buckets of things that are always true and those that you need to approach with a beginner’s mind each time. But if your job isn’t providing you with those experiences, try and find a way to make them for yourself. Build something for yourself and reach outside your local peer group if you can.