There is a very prevalent issue in the IT service industry for businesses and we would like to share it with you. Like death and taxes, technology problems are truly inevitable. Every one of us can point to at least one experience when we have had an IT system fail. At this point, we all have lost data ranging from financial information, client records, or even family photographs. The data we have on our systems is becoming more and more critical to us. It is a major part of our lives.
Experience has taught me that, in general, business leaders do not knowingly choose to risk their business and it is actually because they do not really know what the IT vulnerabilities are. They place trust in IT support people that may have the ability to fix PCs, but really do not understand or value what it takes to ensure business continuity.
Denial. Most of us seem to think it will not happen to us or will not happen again. Most of us learn the hard way but, sadly, we often do not take precautionary steps before a failure. Most of us do the minimum to get though a failure and then do not do much to prepare for the next, because we are rushed to recover and reboot the IT systems.
Blind trust. We often find leaders delegate the responsibility to someone else. We tell the IT support people to protect us and we just assume that they understand the business uptime requirements. This approach leaves us either spending way more than our requirements dictate or as is most often the case, having a “solution” in place that leaves us exposed to loose days of business productivity.
Q: So, Mr. CEO of ABC Company, it looks like your IT servers enable your business to function and they are pretty critical to the operation. Do you have a solid backup and business continuity system in place?
A: I’ve got an IT guy and he knew we needed to be backed up. When I hired him, he said we are all set.
Q: When was the last time you spoke with him directly about backup and business continuity?
A: It’s been a while.
Q: Have you tested the backup
A: A while ago.
Q: Do you know if you tested restoring a file or two or the whole server?
A: Well, I am not sure really, but my IT guy said we are covered and this is getting technical…
Most of us in business have experienced a gap in communication and understanding between business leadership and IT support. After a decade of witnessing this, allow me share some insights specifically in the area of business continuity to protect a business’s operations.
When a new server or system is brought in to play a role in the production of a company’s services or operations, the business leader realizes that if the system fails, it needs to return quickly to operational state as the business starts to loose productivity and opportunity at a quick rate. So the business leader tells the IT support people that the business needs to be backed up. Now when the technical support hears the business leader say back up, most of the time they think a backup of the data; a copy of files.
This articulates the exact communication gap that most businesses experience. I would bet that there has been a system failure and the time you thought it would take to return to normal operation was longer than you expected. What you need to realize is while a copy of your data is important, but a copy of your data alone does not return you to full operation.
If an IT system fails, often you need to diagnose the problem, order parties if necessary, wait for them, rebuild the system, reconfigure it, install the backup device, install and configure the backup software, then recover the backup history into the backup software so you can chose what to restore, select the recovery option and start restoring the system, and pending the amount to be restored, wait and hope the system restores. As you can see, and unfortunately many have experienced, this is not a quick process.
A business continuity based approach takes all of these layers into account and provides systems and procedures to bring the entire system back in minutes vs. days.
What difference can it make? See the 2 examples below.
Consider this comparison of two like companies with different IT standards in place.
Company 1 has 2 servers & 22 users. Their backup consisted of a 2 TB USB hard drive attached to the back of the server. They were hit with a nasty virus that corrupted files. They had Anti-Virus software installed, but this strain of virus was invoked by an employee that opened an email that they should not have. Sometimes your own users can be the biggest threats to your security albeit innocent and unknowing.
The company tried to deal with the problem for a couple hours then called in for IT help. We diagnosed that the corruption was so widespread that we needed to restore from a previous backup. We went to the hard drive, but unfortunately it was that server that was corrupted. We quickly removed the backup drive. We did not want to delete anything on the corrupted server until we could confirm that there was valid data on the backup drive.
We built a new system so we could attach the backup drive to it. We did so and found that while there were files backed up, there were only a couple of complete data sets to choose from. The most recent was 3 days old. With business leadership approval, we started the process of recovering and restoring a 3 day old version of their server.
We created new drive partitions, searched for software, started windows install, finished basic install, configured windows, installed backup software, loaded started journaling the old backups, found the agreed upon target, started restore, the first attempt failed 2/3rds of the way through, started again, 3 hours later it took. Then there was testing and verification & we rebooted.
All in it was two days of down time and when the systems did get backup, it was with 3 day old data – overall 5 days lost. The technical time cost was about $10,000. But the cost of the company losing 5 days’ worth of work and data was tremendously more.
In the end, leadership expressed that they wished they would have known that the USB hard drive was not going to bring their business back the way they thought it would. They just thought they were “all set”.
NOTE: This example was with a company who utilized their own systems and processes to “back up” their files despite our best efforts to educate them on best practices beforehand.
Company 2 has 2 servers & 19 users. Their backup is a NENS business continuity and backup service.
They were hit with same nasty virus, it was determined after 1 hour that corruption was so bad that a simple restore was not going to work. We were able to take the corrupted server offline. Then, we turned on the business continuity system and within an hour we were back to full production. Fortunately, we were able to return to an image that was 1 hour prior to the corruption. The corrupt server was then worked on off line and was restored to a clean state. Then at a convenient time to the business, production was transferred back to the original, but clean server overall down time to the business was about 4-5 hours.
Take an honest look at what systems play which roles in your organization and what value they are to you. You need to understand what the impact to your company is if a component is down. Look at the cost, lost wages, lost productivity, extra support costs, interruption in client service, etc. You really need to take account of everything. These are real costs and exposures.
Realize that there is a difference between backing up to restore a file or two and then bringing a whole system (and loaded applications and configurations) back up and into production. They are different technical problems and need different approaches.
Ask your technical resources to give you different options and have them explain the pros and cons of each as well as the related cost structures. Your technical resource should be able to give you at least five different options and approaches. If they cannot; you really need to reach out to find more answers.
Then, and only then, will you know what your business dependency and exposure is as well as your options and related costs. You can make an educated decision on what fits your business needs. It is not in your favor to leave this decision up to whatever some IT consultant or IT services provider throws out.
For more from Dan, sign up for our monthly Insights newsletter at the bottom of our home page here.