Time To Check Your DR/BC Plans?

The UK had some unexpected power outages yesterday and it no doubt caught a few companies off guard. One hospital had to cancel operations as the backup power generator failed. We take power for granted, you switch on a light and it works - but this shows that we shouldn’t. While it was a ‘freak event’ that caused it, this is what usually causes ‘disasters’.

Disaster Recovery / Business Continuity (DR/BC) plans are only any good if they are regularly tested. You don’t need to carry out complete tests every week, but there should be a schedule for partial and complete testing. Power outages are probably the most likely cause of a disaster (although last year it was flooding), so regular testing of emergency generators need to happen. If generators are fired up then they do use fuel, so make sure that it is replaced (unlike one company who didn’t do this and so they stopped after 15 minutes). Regular testing of critical application fail-over also needs to happen. Application dependencies are increasingly complicated, so failing over to a disaster recovery sight becomes more complex. Service Oriented Architectures (SOA) add further to the complexity providing functionality that you may not be in control of - DR plans need to take these into account as well. What if your 3rd party mapping service goes down - what will replace it?

In the US they have rolling brownouts (scheduled blackouts) in South Africa there are regular blackouts (2-3 a week), this keeps the IT department and the DR/BC teams on their feet. We are lucky to have a better power supply, but it isn’t perfect - so check your contingency plans today.

May 29, 2008 | Filed Under Availability | Leave a Comment 

Narrowing The Search…

Yet more unencrypted data has been lost… well, no surprise there to be honest. At least they know where the data is - somewhere between London and the Isle of Wight, except it could be anywhere because it was en route with a courier.

There were two process failures here. The first was the fact that it was unencrypted data - which was making two trips, one to the third party and then one back to the owners. The other was that it took more than a week to know it was missing.

So, what to do… revisit old policies! If it involves confidential customer information and it’s going offsite then it should be encrypted. [Full Stop!] Backup products today can encrypt the information - so there is really no excuse. There should also be an effective tracking mechanism for data that is traveling with or being stored whether it is with a 3rd party or even by internal personnel. That way, even if the data is encrypted and lost the disaster recovery plan won’t be a disaster itself because the data isn’t where it was expected.

The good news, well piece of process, which we should all take heed of in this case was that the data was being verified as readable / usable. Frequently backup data is not checked and you get to the point of needing it and it is inaccessible, or not complete. I remember a case a few years ago when the data was required and there wasn’t any on the tape - except the header. The reason… the data had changed mount point on the system and the backup policy hadn’t been altered. So it regularly backed up ‘nothing’… and was always successful! So, checking the data integrity on a regular basis is a great habit to get into.

May 24, 2008 | Filed Under Availability, Information Policy, data-loss | Leave a Comment 

Traditional IT Departments To Disappear

Traditional IT departments are to disappear according to a comment from a Software as a Service (SaaS) vendor. This reminds me of a similar comment 10 years ago that tape backup was dead. We all know that tape is still here and will be for the foreseeable future (ok, so you backup to disk as well - but tape still has its place.) The same will be true of the IT department, their role may change a little but their basic function will remain.

Applications will be delivered over the web - but that doesn’t mean users won’t have problems with them and need support. There will be a rise in outsourced functionality, you just need to look at salesforce.com to see the revolution that is underway in delivering business applications as a service. There will also be a rise in the number of services available to the small, and the not so small business. For example, outsourcing email for companies with 1,000 employees is now relatively simple and that number will grow over the next few years to 5,000+. However, there will always be core value-adding applications and services that will be developed in-house and need support from the IT department. The changes that occur will enable IT workers to focus on these business focused value-add applications rather than the more mundane ones - which will be a good thing, for the business and for those of us who work in IT and like to innovate.

March 28, 2008 | Filed Under Availability, SaaS | Leave a Comment 

Amazon S3’s Outage

Amazon’s S3 service suffered an outage for a few hours, so what? Well this raises a few queries in general about Software-As-A-Service (SaaS), basically if you are depending on someone to provide a service then it needs to be there all the time, as reliably as the dial-tone on the ‘phone. If not, then you lose business - and the Service Level Agreement (SLA) might not help.

“SLA’s just make money for lawyers they won’t guarantee you remain in business”

So, when looking at SaaS also look at the risks and consequences as well. If all your data is stored by a third party and that third party is down is there anything you can do, or will you just be twiddling your thumbs? Look very closely at the SLA and translate the uptime into actual time, 99% uptime means the service could be down 87 hours a year, or 10 working days! Look closely at whether this includes planned downtime (when they do upgrades) or other outages.

SaaS is a great way for smaller companies to get ‘Big IT’ benefits - but check the small print.

March 3, 2008 | Filed Under Availability | Leave a Comment