Permanent data loss: it could happen to anyone
Permanent data loss occurs when a piece of information is lost forever either due to a system malfunction or a user error and has not been backed up.
Although web hosting companies strive to use very high quality components, there is always a risk of server drive failure. Due to the mechanical nature of server hard drives, and the fact that they are required to run 24/7, they are the most likely components to fail.
Disney’s Toy Story 2 was almost a victim to permanent data loss. As one of the technical directors of the film was reviewing an update to Woody’s hat the files started disappearing. One by one the characters vanished from the system and panic followed!
In this instance the error was most likely human: the command below was run, which tells the system to start removing files under the current directory.
In this age of information it is obvious that data loss either through human error, hardware failure, natural disaster or malicious intent will have wide ranging consequences for any company. Web hosting is no exception as errors can always occur regardless of what preventative steps have been taken. Thankfully, any reputable web hosting provider will have the capability to back up data to mitigate against human error from causing any permanent damage.
The following scenario demonstrates how a small mistake can lead to severe data loss – in this case website content – and why back-ups are the often the last and final line of defence even if you have a rock star sysadmin!
What would you do if you got a call from a customer stating there was an issue with broken site content – specifically style sheets and images but they had not logged onto the server for weeks?
If you’re a sysadmin you’d probably log onto the system and start having a look around. In this scenario let’s imagine you do this and find that the wp-content folder is missing. Your initial assessment may well point towards a compromise to the server. However, what if no evidence could be found to confirm this? Further investigation into who had logged onto the system would be needed.
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT customer pts/0 x.x.x.x Wed27 44:55m 1.22s 0.00s sshd: customer [priv]
This shows us that despite their claims to the contrary someone from within the customer site had logged into the server. Now it is imperative that you find out what the user had been doing as it is still unclear if this is a human error or a compromise.
The good news is the session was still active so the .bash_history file could not have been written yet. First we have to pull the bash process ID to find out where the heap’s located in the memory space.
root@VDED-XX-001:~# ps auxf | grep pts/0 | grep bash
customer 21233 0.0 0.3 27544 7972 pts/0 Ss Wed27 0:00 | \_ -bash
root@VDED-XX-001:~# cat /proc/21233/maps | grep heap
02622000-02ccb000 rw-p 00000000 00:00 0 [heap]
You can then use the information above to hook the process with GDB and dump the memory.
root@VDED-XX-001:~# gdb -p 21233
Attaching to process 21233
(gdb) dump memory dump.bin 0x02622000 0x02ccb000
A debugging session is active.
Inferior 1 [process 21233] will be detached.
Quit anyway? (y or n) y
Detaching from program: /bin/bash, process 21233
We now have a file named dump.bin in the $CWD that should include the current command history.
root@VDED-XX-001:~# strings dump.bin
rm -Rf /var/www/html/wp-content .bak
The eagle eyed among you will notice there is a space included in the above code after content which should not be there.
This example shows how a simple human error, such as a space in a command code, can create huge data problems. This command, now deformed, has been altered from a simple command to delete a duplicate file into a complete wipe of website content. It also shows that with the right experience you can extrapolate very accurate information about the origins of any issues.
The tech takeaway
It is important to point out that the above scenario is an unlikely event and that in most cases a good web hosting company will offer a solid back up plan as an additional element of their package.
Here are some points and facts to take away….
1. It’s important to have a backup plan and make sure the restores are regularly tested.
“93% of companies that lost their data centre for 10 days or more due to a disaster filed for bankruptcy within one year of the disaster. 50% of businesses that found themselves without data management for this same time period filed for bankruptcy immediately”. (National Archives & Records Administration in Washington)
2. Human errors are inevitable however more care can always be taken especially with recursive removes and ensuring that no spaces have been left in the commands!
96% of business workstations are not being backed up and are at serious risk. (Contingency Planning and Strategic Research Corporation)
3. When choosing a web hosting provider it would be wise to make use of any additional back up services offered such as a managed backup package just in case.
Companies that aren’t able to resume operations within ten days (of a disaster hit) are not likely to survive. (Strategic Research Institute)