Do you remember when growing up how kids that were 2-3 years older than you seemed so much more mature? They had it all together. But when you hit that age, you did not feel like you arrived at the level of your predecessors. And to add insult, the kids a few years younger than you did not look that much different in age than you.
Experience, time and wisdom all play a large role in perspective. Why am I speaking about perspective in a technology blog? Well I see leaders and owners get bit with the perspective bug constantly.
Here are two stories where leadership and tech support believed they were on the same page, but in reality there were not even close. I will also provide some advice on how to ask the tough questions to bridge the technology gap and protect your business [the way you already assume it is protected].
We met up with a company a little while back who had all of their IT services locked down tight according to leadership. So much that they believed it was time to optimize. They believed that their ship was so tight, why not just bring in an IT service provider and maintain what they had. No real work to do, just keep the well-oiled system humming. That would be cheaper than keeping the IT staff they had in place.
Initial interviews with leadership all echoed on how well things were operating. No problems anywhere, just looking for a cheaper way to maintain. I was a little amazed albeit skeptical. So many companies have problems, they somehow were able to stand in the right spot in the universe and all things worked. [ Sunshine and unicorns!] I was thinking, “Wow, why do they need to even speak to us?” Oh right, the only thing better than IT bliss, is IT bliss for less money.
Well, we would not be able to give them a price until we did a discovery. Depending on the scope, we generally sit down with the leadership over technology and ask some questions. Here is an example of questions that are asked:
-What are the concerns?
-What does the infrastructure look like?
-Can you please share documentation?
-What business continuity do you have?
-When was the last time it was tested?
To all questions asked, the response was that everything was all nice and pretty, and functioning smoothly.
So far story is ringing true. Good for them. Okay, now we need to look at the actual servers, review logs, and run a report or two. We asked to be led to the servers. They did not want to let us. They questioned us and claimed it was all working fine. What we heard from them was, “we just need someone to maintain it. We do not want you to change anything. Just give us a price.”
At this point, the hairs on the back of my neck were starting to rise. I felt like I was being asked to take the bait on a trap. To believe that everything would be easy, offer to take all of the responsibility for a fraction of what the normal cost is and then be trapped in a horrible relationship.
Next came our thorough assessment. It was time to actually look at the network, the servers, disk storage, backup, business continuity, monitoring, etc. To do this we perform several different tasks and build a report:
We were told they had several physical servers in their configuration. That way if something failed, there was redundancy. When we looked at the servers there was only one. I asked where the failover systems were and was told, “they are here,” as they pointed to the server rack. I informed them that, unfortunately, there is only one “server” here. There were other devices, but no other servers in the cluster they had sworn existed. Their workstations were a hodgepodge of everything. In fact, they had systems there that were still beige! Do you remember, in the 90s and early 2000’s how so many of the systems were beige? Well we found where they were hiding out.
Network security was almost nonexistent. Passwords never expired and were unsecure. Fired employees still had login access to systems, even on the remote server. The firewall was not current with updates and protection, antivirus was in a random state.
The backup system turned out to be an $89 big box retailer USB hard drive. [Consider this, if you spent 50 – 100k on a high performance car, would you put the cheapest brake pads you could find on it? I see this problem weekly. Business with multi-thousand dollar servers, with 10s of thousands in daily business, putting cheap, household grade USB hard drives as their “backup/business continuity device”. Someone needs to be accountable for letting this happen. This is as close as we can come to spit and bailing wire. If you need a server/data/application to conduct business you need to truly protect it. STOP IT!]
We were told that should something truly crash, failover was just a matter of minutes. Turns out, when we looked we could not find the system to failover to. Time to recover would be 2-3 days from the diagnosis time, work with the hardware vendor, requested replacement parts, [if they get the parts right] install them the next day, start rebuilding the systems, reconfigure the OS, patch the OS, install the backup software, to restore the database from the backup, select the backup and start the data restore. Then wait for the restore, tweak – You get the point. It was not just a simple switch.
Now what? They were looking for a simple price to maintain but in truth – this was not even close. Do you tell leadership what the truth is? You look like someone just trying to scare them. Not how you want to start out a relationship. Do you walk away? The expectations are so far off, there is no way you can win here.
We met with an internal IT director who had been with his company for 20 or so years. He was their only internal IT support. They had about 30 in-house users and 35 outside users dependent on 7-8 servers. The whole operation revolved around this one application (ERP) and most of the servers were related to supporting this key resource.
The software had been in place for years and they had found a programmer across the country to make modifications for them. Over time, this remote consultant quietly became the IT leader for the company. He knew more than the internal IT support guy. He alone held most of the knowledge that enabled the company to work. The IT director was basically just an extension of the consultant.
The consultant over time had grown his responsibility from the application to designing and partially implementing the whole network infrastructure. [Servers, Storage, Security, remote access, etc.] He flew out a couple times for a project here or there, but 99% of the time he was over 2000 miles away.
The director wanted another IT firm to be a safety valve. He realized that he was in a precarious position. But that said, he did not have the ability to bring in the level of help required to protect the business without jeopardizing the perception that he had it all covered. Once again, leadership thought they were protected.
Now in both of these situations, leadership just believed that they were “all set” with technology. They trusted that their IT support guy was doing it all. In their minds they were safe, secure, and their normal operations were protected.
Truth was it was all perception. Often in the gap between reality and perception, the liability lays hidden, until it strikes and impacts the business. And to make things worse, often the IT support or leadership will know that things are not where they really need to be, but because of time, fear and energy, they leave it alone. Nobody wants to deal with it, so they figure it is best left under the rug.
So now what?
While it may be more common for an employee or IT consultant to leave the time bombs and liabilities, owners and leaders have a deeper level of commitment to protect the company. Leaders need to engage and ask the harder questions. We have a short video on our website detailing some of the questions to ask here.
As a leader, if you do not understand about insurance or sales management, can you choose to not learn about it? How would the board feel about not crossing those T’s dotting those I’s? Technology is unarguably a key part of your business.
Take the time to get a little better perspective and protect what you have been working so hard for.
Until next time,
I wish you the best.