A number of client’s have asked us how we managed to get operational again so quickly when compared to other CBD based businesses which were not so lucky, so I thought the best way would be to outline what we put in place after the first earthquake that we were able to put into operation after the latest one, plus how it works and what didn’t work.
The Office
The first thing I did when we moved into out offices three years ago was to anchor to the wall everything that could fall and either trap someone, block a door or block an exit route. This included storage cupboards, book shelves, and anything else large. If you have not done this at your work then do it now.
The second thing we have discovered is that most filing cabinets are generally sited to open outwards but during an earthquake they all open their draws and create a huge danger as you try and evacuate the building by blocking your exit route. If possible try and site them so they open in parallel to the walls so if they open they don’t block your exit.
All our computers fell over as did the monitors so if there is a way of anchoring them to your desk then it may prevent them from damage.
Do take shelter under your desk or door and try and stay there until the shaking stops (which it will). Quite a number of people were killed as they exited their buildings and were hit by falling masonry, which was a real tragedy. So don’t be in a rush to get outside unless of course your building is falling down around you.
Planning
After the first earthquake we put together a plan for varying degrees of an emergency. From just not having internet access to having no access to the building. I never went as far as taking into account no access to the city which will be taken into account in our next plan.
The lack of access to your building is something you must take into account, and even the complete destruction of your building. I think it came as quite a shock for many down here to find out that if a building is condemned and flagged for demolition then you will not have any ability to recover anything from your building (servers, personal belongings, equipment, NOTHING). The demolition company will just crush your building and cart everything away to the local dump. So you need to build into your planning the above scenario as that is what has occurred here for a number of businesses.
Backups
You may find it unbelievable but organisations such as The Christchurch City Council and Ngai Tahu maintained all their backups ONSITE and were then faced with situation where (a) they had no access to their buildings, (b) there was no power, and (c) if the building was condemned then all their records would be destroyed. There was also many Legal Firms who only kept hard copy backups in their safe, onsite, of important items such as wills, contracts etc. Most Christchurch legal firms were in the CBD so have no access to their buildings and some may be even be demolished with no recourse to any copies.
All our key data is backed up every night via a company called Nexus Data Security (www.nexusdata.co.nz). It gets fully encrypted before been sent and is stored in an encrypted format. Luckily for us, after both earthquakes, we did not have to resort to retrieving our backups but it was a great relief to know that it was all available. The Nexus staff even contacted as many clients as they could get hold of to see how they could help, which considering the level of destruction was pretty impressive.
If all your critical data is currently stored only in your building then you suffer the real possibility of loosing it all should a disaster strike. At a minimum this is what we suggest:
- Maintain off-site back ups of critical data. The scenario of someone taking a tape or external hard drive home each night is not sufficient as after the last earthquake this is the last thing on anyones mind, plus there was no power in order to create a fresh backup. We highly recommend the guys at Nexus.
- Having your data backed up to another building in the same city may not be good enough. Many companies did this here and had their data safe in a secure, undamaged building, but the building next door was unsafe so the whole area is cordoned off so it is as good as useless. We suggest you ensure your data is stored either in a different city or country.
The other thing that became obvious after the first earthquake was there is a lot of important information that you need readily available, these would be passwords, account numbers, employee lists, directors details, bank details, ird number etc. After the earthquake it was very difficult to find this as (a) there was no access to the building, and (b) no power to print anything out (this was also the case with schools that relied on electronic rolls, they had no way of doing a roll call as they could not print a student list).
How we overcame this was by setting up a Private Blog (www.wordpress.com) and recording as much of this information as possible. The site is known only to our staff, is password protected, but all critical information is there. We don’t record logins and passwords but key staff will know both these. This kind of structure worked brilliantly in the second earthquake as we tried to re-direct our fax line (Telecom takes five working days to do this and you need your account number). We would strongly recommend that every company considers a similar option as you cannot rely on retrieving this from your office and is something that many have overlooked.
E-Mails
These days we rely heavily on e-mail and internet access and luckily for Christchurch neither of these structures were damaged (but of no use if you have no power). After the first earthquake we took over the management of our domain name and worked out what all the Server Settings mean (A Record, C Record, and MX Records).
We used an internal Exchange Server for our e-mail but had established in advance an e-mail accounts with Google Apps. You then need to update your MX Record to point to the Google Apps account and let it update, this can take up to 8 hours. Set the Google Apps account as a low priority setting on your MX Records and forget about it. When you need to activate this account you just reset the Google Account to top priority and within an hour all your e-mails turn up in your Google e-mail account.
Google Apps lets you setup 50 e-mail user names on their free account and you can either access them directly via the internet (if you have power) but we have installed Thunderbird onto our PC’s and download all the e-mails using the iMAP protocol and it all works brilliantly.
Your IT staff should understand all of the above but I think the key is you must maintain management of your domain (usually your Web Developers will manage this for you) so you can quickly change records as your external managers may be in a worse state than you are, and you need to have the Google Apps account all pre populated and active as otherwise it can take up to 48 hours to resolve everything.
On another note, if you do end up picking up your server and moving it to a residential address and use Xtra as your ADSL provider you either need to unblock Port 25 (you request this via Xtra) or know how to change the SMTP settings on your Exchange Server. Exchange by default uses Port 25 to send out all e-mails but if it is blocked then your e-mails go no where.
Telecommunications
Last year we changed our phone system to a Voice over IP system. This means that we make all our calls over the internet which is great provided you have internet.
The big advantage for us is that we just had to pick up our Phone server and plug it into any internet connection and all our phones were operational with no need to try and deal with Telecom to get phones diverted etc. When we tried to get our fax diverted we were told it would take five business days which was as good as useless (and they still haven’t done it after two weeks – our new fax number is 03 2818484) so we had to put in place a virtual fax number so all faxes now come to us via e-mail and took 5 minutes to setup.
We will get our old fax number ported to the new number (remember you will need your Telecom account number and exact company name to do this). It is worth keeping a fax machine onsite for outgoing faxes but not required for incoming. Also, ensure you always maintain an Analogue line into the business (usually your fax line) and an old style phone (one that does not need power) as they will come in very useful should your power fail as you could call your customers (provided you have a hard copy client list) if need be.
So, what happened on the day
When the earthquake hit I was not in the office but had left a client’s premises (which now has to be demolished) and was sitting in a cafe just having some lunch. When it hit I thought it was just another aftershock (we have had over 5,000 of them since September) so did not get overly excited. Some people had dived under the tables (obviously from out of town) but I remained where I was as nothing was falling on me and the building was totally safe.
Across the road, a building damaged in the first quake fell down and cars were rocking about on the road. After I checked that everyone was OK I jumped on the scooter and headed to the office. All around me the road was erupting with mini volcanoes of silt and many roads were already blocked with debris or very damaged buildings likely to collapse. I count my lucky stars I was on the scooter that day as within 10 minutes of the quake the roads were gridlocked as people tried to either get home or to offices. With the scooter I was able to quickly get to the office to find the staff were all unhurt, the building was still standing, but a scene of complete destruction along Colombo Street.
After ensuring everyone was ok (at this point as well the cell phone network was totally overloaded so no calls or texts were getting through), I returned to the office to check on damage and to turn off the power switch. I then had to transport Harvey across town through gridlocked streets to where his wife worked. I then quickly headed home to check on 0ur dog and elderly next door neighbour. Chucked a stray dog in with ours and headed back into town. Again, the scooter was the only means of transport that was working as everyone who had a car had got into it and gone nowhere.
Back at the office we locked the doors, told everyone to head home as quickly as possible as we did on the scooter. Our street was now flooded with burst water mains and silt still pouring out of the ground. A quick check on our neighbour and then it was out with shovels to try and clear the drains to get rid of the water but it was not going anywhere.
That night we were without power and water but luckily we had an old cellphone that also had a TV receiver built into it so we were able to watch the news and see what had happened. We used a battery from the motorbike to charge our phones (but remember many cars were stuck in Christchurch and are still there so think about how you will charge your devices with no power).
To all those clients who we missed the Tuesday payrolls we are very sorry as this was the first time we had ever missed a payroll. Thankfully most clients managed to contact their banks and put through a payment as they had no way of contacting us and we had no way of contacting them, except via our Blog as Kerry still had power out in Halswell.
The next day I ventured back into town to see what we could do. The roads were an absolute mess but totally deserted. There was a cordon around the city but with the scooter I was able to get to the office and unplugged the server and a number of computers and stacked them by the door. I then approached the police on the cordon, explained what we did and they agreed to escort us to the door so we could grab what we could. Luckily Kerry could get there quickly in her car as I could not carry much on the scooter. We were extremely lucky that (a) our building was not destroyed, (b) all our gear was ok, and (c) I found some co-operative Police who understood how important it was to pay people.
Within an hour of grabbing the gear we were up and running at Kerry’s place and processing pays as:
- We had our server (many companies were not so lucky)
- We had our phone server so had full communications
- I quickly put in place our emergency e-mail procedure (as outlined above)
- We couldn’t print anything as we had no printer but we had e-mail
- The staff at ASB Payment Services were fantastic in helping us out as we could not get our dial up working.
We spent that week at Kerry’s but then moved to our ‘planned’ backup office at Paula’s. Here we had a separate room to operate from and a box of cables that we had previously put together. Once the cordon for our part of town was lifted we were able to grab more gear (but our office has been cordoned off again because of the stability of an adjoining building).
The outcome of all our planning and good luck was we were only down for no more than 24 hours.
Looking forward
Although our building looks sounds none of us really want to remain so close to destruction that is all around us so we are looking for somewhere new but all office space is either gone or at a premium at the moment.
What we have learnt from the episode is:
- Your staff must come first
- It’s too late to start planning once a disaster strikes
- Don’t keep your backups onsite and/or even in the same city
- Keep a print out (offsite) of key contacts
- All this amazing technology we use today is useless if there is no power
- Our blog site was invaluable in keeping our clients up to date so we will be pushing again that all clients subscribe to it
- Plan now where your backup office is going to be and ensure everything you need to function there is already in place (cables, internet etc). If on Xtra get Port 25 unblocked now.
- Setup a private online blog site where you keep key account and password details
- When it happens you won’t be thinking about grabbing your car keys so maybe hide a key on your car somewhere now or it could be a long walk home
- When it happens public transport stops so either keep a bicycle at work or expect to walk
- Everyone should ride a motorbike or scooter
- Keep an old phone handy and an analogue line
- As soon as you can, store as much water as possible. Cut your downpipes to capture any rainwater and ensure you have a decent container to store it in
- If your house is 100% electricity think how you will manage to boil unsafe water when there is no power
- You get to know all your neighbours
- Don’t be tempted to down a bottle of wine as this will only dehydrate you at a time when water is a scarce commodity
- Have a radio, or a cellphone with a tv receiver as this way you can keep in touch and know what is happening when there is no power
- You can’t rely on one type of technology, for example, one might think we should move all our operations to ‘the cloud’ but this then makes you 100% dependent on internet access. We will be structuring an onsite and cloud environment so that through one form or the other we won’t need to relay on either to remain operational
- Think what would happen if your phone is destroyed or the battery is flat. Do you have a hard copy list of your contacts? I use a service that automatically saves all my contacts, photo’s messages etc to a secure website for a small annual fee. You can easily restore them to any new phone and access them via the internet. See www.wavesecure.com
Our thoughts go out to everyone who lost someone in the quake but also to all those businesses who have lost everything either through their building collapsing or not maintaining offsite backups.
Some stories:
- There was a photo in The Press of a business owner walking home after the earthquake. Under his arm was his server, he was one of the lucky ones.
- A local restaurant owner in a badly damaged building is frustrated he cannot access his chef’s knives and his payroll records.
- Many people have been without their car for three weeks.
- One man lost his wife, his home and his business.
- People living outside the Canterbury area bought water, why not just fill up containers from your tap!
Just want to specifically thank Mudhouse Wines and Horncastle Homes for the wonderful gifts they sent us.
I hope this provides some help in what you can do to keep your business operational in the event of a major earthquake. Feel free to ask for more details regarding any item mentioned.
Justin
Very impressive! Just the sort of forethought I have come to expect from your company.
Fantastic insights. If only more people have prepared for disasters as well as you and your company did, Justin. I especially agree on the comment on two-wheelers 🙂