If you are in IT, then you have probably had an experience like this. You have been trying to get something to work properly and no matter what you do, it just won’t work. You know (or are at least very sure) you have done everything you were supposed to and it still does not work. You finally give in and have someone else look at it. What happens? They figure it out within minutes. They find the one spelling or punctuation mistake, the check-box that should or should not be checked, or even the capitalization mistake. They notice the one thing you did not. Once you are done banging your head against the wall, you wonder how you could have missed that. It had been staring you in the face the whole time. I recently had another of those moments and I thought I would share that experience and others with you. I am sure many of you who are reading this have experienced these moments.
I know I have had moments like this a few times in my career.
The first time in my IT career I can remember it happening was back in college. I was in a programming class. I think the language was COBOL, so that should give you an idea how long ago I was in college. I had almost finished the assignment, but had one problem. The program gave the correct response, but then changed it. I had my buddy who was in the class with me look at it. For a short while, he was also stumped as to what was happening. After reviewing it together over and over, he finally looks at me and calls me a name that had a similar meaning to idiot, but not as nice. In the main body of the program, I had forgotten a period after the ‘Stop Run’ command. One little dot. Because it was not there, instead of the program ending where it should, the subroutine directly after the main body was run one extra time. Over 20 years later, I still remember that incident. That’s how frustrating (and revealing) it was.
Last week, I was doing a VMWare server refresh with a client. We needed to swap out some host servers that were past their prime for some new servers with up-to-date hardware. On the first server we swapped out, we hit a point where the server seemed to be configured properly in vCenter, but would not talk to the SAN. I looked into the obvious areas first like cabling and configurations, but could not find anything. Admittedly, I found a couple of little things to adjust. However, there was nothing significant. I was looking at it going nuts knowing that it should work. My client was looking over my shoulder trying to see if he could catch something. I compared settings from new to old. I checked everything over and over. I changed settings I knew were correct, but had to try anyway. We finally gave up and called VMWare support to see if they could find it. The first support person knew right away that this was not his area of expertise and handed us off to someone else. That second person, took five minutes to find it. On the SAN, in the WWN name, the letters ESX were capitalized. In the host server’s configuration, they were lower case. That was it. We had been staring at it over and over and never saw it. It was past 2 am, so we were not likely to see it. Also, in my defense, I had copy/pasted the WWN from the SAN console to the VMWare console configuration. However, there it was. One of those moments where the answer was extremely simple, staring us in the face, and we were not going to see it. By the way, I observed what happened when we did the second host server replacement. I copy/pasted again and it did the same thing. On the paste, capital letters were converted to lowercase. The wise guys reading this are thinking: “Sure, that is your excuse.” However, we are replacing more hosts this week, so I can video it happening as proof. I am not saying that I am not losing my mind. I am, but this was not a symptom of that.
Let’s face it, you don’t have to be in IT to have one of these moments.
I remember helping my kids build LEGO sets. One of them could not figure out why they were unable to do what the directions show. I would look at it and realize that they had previously used a brick with three studs when they needed one of the exact same color and similar shape that had four studs. Ever try a video game where you just can’t figure out what to do next? You go on to the Internet and find a walk-through or watch a video showing you how to do that level only to realize that the answer was staring you in the face. You did not realize it was there, but it was. Like I said, we have all ‘been there, done that’. If you haven’t, think of this article when it does happen to you (and it will). It won’t help you feel less “stupid” at that moment, but you will know you are not alone.
If you have any stories like this, feel free to share them in the comments of this blog.
Craig R. Kalty
(CCIA, CCEE, CCA, MCITP:EA, MCITP:SA, VCP)
Sr. Network Consultant firstname.lastname@example.org
©2016 Custom Systems Corporation