I just need to get this off my chest – so bear with me.
First off, I’ve been doing sysadmin work for scores of years now, and the idea of backups, business continuity, and “bad guys” isn’t new. However, this week it was brought to a new and interesting head for one small business.
Rewind the clock two years and we were in the conversation with this business about where they host their “golden nuggets” of their business, what servers did what, where were the users, how did the backups fare, state of malware, web filtering protection, etc. You know, all the “normal” stuff any qualified IT provider would ask a prospective customer. “We’re fine” was the answer – they had an in-house IT guru watching all that stuff. However, they did make a (wise) decision to host their ERP solution with us.
Last week, our monitoring went suspiciously quiet, it looked like the company went on vacation, or they had fallen asleep at the keyboard. I reached out to the company, and was informed that they had been the victim of the latest ransomware attack, and all their documents were encrypted and unusable. Thankfully, since they were hosting their ERP system with us, that was safe from the attack. All their ERP data was secure but everything else they controlled was locked. Backups proved unreliable or inaccessible, so the ransom was paid. The company got lucky and the recovery key worked and they got their documents back. What they didn’t get back was Active Directory. Ouch! Nobody could login, even though their documents were back on a server, nobody could access them.
A week later, a new domain, and new profiles on everyone’s desktop, new shares, new permissions, and they were back up and running. After everything, the company is back to doing business, but it could have been a much worse situation. A critical note: the ERP system was never at risk and no ERP data was lost since that was safely stored elsewhere.
Moral of the story:
- Test your backups. Not just documents, but the whole server. How long does it take to get it back? It should not be more than a few hours.
- Just because you can restore files doesn’t mean you can go out, buy a new server and restore your existing workload onto a new server.
- If you can’t live without it, and you don’t have the in-house expertise to manage it – outsource it! Let the pros handle the critical IT while you do what you do best: making essential product and making your business grow.
- In general, EstesGroup’s EstesCloud Hosted ERP (ECHO) systems are more secure than those hosted in-house, unless you are as paranoid and diligent as we are!