If you’ve ever wondered what tools a technical data recovery engineer might use on a daily basis, then here are my top five  tools, although they may not be quite what you’re expecting.

If you think that data recovery is just about running a bunch of software tools on hard disks or RAIDs or data from backup tapes then for a professional technical company, this is far from the truth. As a data recovery expert my job at Altirium involves interrogating raw data and writing software to solve often complex problems. This is one thing that I think sets us apart from some of the other companies in the data recovery industry. Yes we use “off the shelf” recovery software where it’s appropriate but often they are found lacking and don’t give the best results or properly report their findings. Therefore to recover data where there are no tools available, we develop them in-house.

Some of the achievements we’ve delivered in the past 12 months, using my top five tools include:

• Reverse engineering the MS SQL Server data structures and written software to recover data from dropped tables, where off the shelf tools failed.
• Extending our Tivoli Storage Manager recovery software capabilities, adding extractors for more data sources and software compressed data.
• Identifying and implementing the processing of many undocumented structures in the Microsoft Tape Format, used in software such as  Symantec BackupExec.
• Reverse engineering Atempo Time Navigator backup format including processing of software compressed data.

All of this has contributed to solving genuine data recovery issues and has saved the companies that have come to us, thousands of pounds in lost revenue, many hours of support time and countless terabytes of potentially “unrecoverable” data.

Here are the top five tools that I use pretty much every day during the course of my job.

1. Winhex: Undoubtedly one of the best hex editors for Windows. I originally discovered this tool at about version 9, I think, when it was just a hex editor on steroids. It was able to work within very large data files > 4GB (big at the time), where other software couldn’t. I could create structure templates to view real numbers rather than just hex and it had some great searching capabilities. Winhex have grown up a lot and is now on version 16. Unfortunately it has been adopted by the forensic community which I think is more lucrative and therefore some of the feature are now targeted towards them rather than us data junkies and so some of the changes have be detrimental. However it is still by far one of my most used tools.

2. Textpad: A simple, easy to use, plain text editor that got some cool but easy to use features. Simple keystroke macro recording, block selection cut and pasting, long lines (good for chopping up long line of hex). Syntax higlighting, Has some great sorting and searching functionality and is fast when dealing with very large text files such as file listings.

3. Excel: What’s a spreadsheet application doing acting as a data recovery tool I hear you cry. Well, it’s great for a working out offsets, relationships between values and I’ve also used it for annotating hex dumps from time to time.

4. VMWare Workstation: I frequently have to run trial software, whether it’s backup software, database servers, email servers, other recovery tools etc. VMWare Workstation allows me to quickly configure a virtual environment in which I can operate. Although the software tends not to run as quick as if it was running on native hardware and there are limitation, the virtual machines are quick to boot, take up less storage can be isolated in their own virtual network, snapshotted, replicated and destroyed without having to leave the comfort of my desk.

5. Visual Studio C++: When working in a data recovery environment its absolutely essential to be able to develop your own tools. If you don’t then you’re relying on someone else’s  knowledge and I’ve many times in the past found this to be incomplete or in some cases misguided and wrong. I work predominantly in a Windows environment and Visual Studio C++, in my opinion has an excellent IDE to allow me to construct small widgets and test application to larger more robust software. In C++ I can address the data how I want to with few restrictions.

The problem was that the database had been migrated to a new server. Since the migration, the database had not been backed up although the transaction files were still complete from the time of the migration. In error, all the tables in the database had been dropped. As soon as the mistake was noticed the server was taken off-line and copies of the database MDF and LDF files were made so the data was still in the files, just could not be accessed.

Now, I am not a MSSQL expert I am a data recovery engineer so my knowledge of MSSQL is from a “Technical User” standpoint. Whilst examining the files with our trusted hex editor, Winhex, we thought we try downloading some “off the shelf” recovery tools that looked as if they would do what was required.

The software we downloaded and tested were:

• ApexSQL, Recover
• officerecovery.com, Recovery for SQL Server
• Kernel, SQL Database Recovery
• Stellar Phoenix SQL Recovery

Of the above however, only ApexSQL Recover claimed that it could recover from dropped tables so we weren’t very optimistic.

All the software we tested went through the motions of appearing to scan the entire data file but when it came to results, they were as suspected and no data from the dropped tables was reported to be recoverable.

Fortunately the software testing didn’t distract us too much from our usual approach of “looking at the data”.

Now it used to be that internal data structures of proprietary files, such as MDF files, had to be reverse engineered from scratch. Fortunately though for the MDF file structure, much information can be found on the internet, even if not completely, but it’s a good starting point.

From visually interrogating the file we could see that the required data was still present within the MDF file. The 1st step in actually recovering the data was to break out the pages for each of the tables, both internal and user tables. From this we could then find the data that related to what our client required recovering. The schemas for the tables were not available in the active data because the tables had been dropped and not just truncated. So once we’d located the correct data files from the data generated from our 1st process we then set about coding software to turn the data in those files into a CSV type format which could later be used to repopulate a database.

The outcome was that we were able to recover the data that the client wanted, approximately 3 million records, and they were able to use this to help rebuild their system, proving that sometimes there is nothing better for recovery than the skills of an experienced data recovery team. It also begs the question, how many times have people run recovery software that has failed to recover data and believed that this meant there was nothing that could be done, when with professional help a recovery of data could have been achieved.

References

http://improve.dk/archive/2011/05/19/reverse-engineering-sql-server-page-headers.aspx

http://www.sqlskills.com/blogs/paul/post/Inside-the-Storage-Engine-Anatomy-of-a-page.aspx

As Hamlet so eloquently put it “There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.”, just because something is outside of our normal experience does not mean that it is not possible.

One such case landed at our offices recently, 4 disks in a RAID 5 with apparently corrupted data files. Once the drives had been imaged and the data secured we set about examining the RAID and it all looked pretty good. The RAID scheme was a standard last to first data striping with parity shifting first to last every 256KB, and the file system appeared to be perfect.

Then we looked at the MS-Exchange EDB files. The PRIV.EDB started fine then after a while the EDB page checksums were all wrong. The PUB.EDB file did not even start with the correct header pages, though the data did have the appearance of Exchange Information Store data. Examining the data in free space for Exchange EDB headers we did find one for an EDB from the required date.

So, the file system was perfect but the data was not quite right, maybe not corrupted as such, but shifted.

At this point we had not looked further down the disk, but now we started to examine the data in more depth. A second data partition existed on the volume, but on the current RAID configuration the file system was not correct. We could find our way from the boot sector to the MFT, but then after a few sectors the entry numbering went awry. The RAID configuration was different for the second partition.

It is not usual for a RAID to be in this state, so it appeared to us that perhaps the RAID had somehow been re-configured, and that this explained the data issues on the first partition. Sure enough checking the allocation information for PRIV.EDB it was apparent that the data was correct if it was from a low numbered set of file system clusters, but incorrect if it was from high numbered ones, so perhaps the re-configuration had stopped after the bulk of the file system information for partition 1, but not before the end of the data.

Not finding the notion of playing spot the configuration change somewhere in the midst of a large volume of RAID data we set about a new approach.

We had 2 RAID orgnisations, one that worked for partition 1 up to a point and one that probably worked from somewhere in the middle of partition 1 though partition 2. The customer had a set of EDB files that they rather needed back and straddled the point where the RAID configuration changed.

We created 2 versions of the RAID data using each configuration and then extracted the required files.  EDB files have page checksums. And it is possible to work out the page number, so processing an EDB file you can tell if the data is almost certainly correct.  So the next stage was to process each EDB, check each page, and for any that failed check the equivalent page for the alternate file.

The result was stunning, EDB files that not only could be processed using the checking options of eseutil (the Exchange utility program) with minimal errors, and that could then be accessed via Exchange and all mail items examined.

Looking at this as a standard RAID data recovery the answer would have been “your data has gone forever”. Utilities such as R-Studio and Winhex could be used to access the data from the file system, but not to recover the files as the customer needed them, so anyone just playing point and shoot with a recovery utility would be on to a loser.

The answer for this recovery was to examine the data, to really check what was there and to not let preconceptions prejudice the outcome. By finding the facts, and letting them determine the course of the recovery and not letting the demons of “that cannot be possible” cause a distraction, the data was saved.

So, you are an investigator who has been handed a case containing 25 LTO4 cartridges from a TSM archive, now what?

Unlike relatively straightforward backup applications such as BackupExec or ARCserve you cannot simple buy a TSM installation DVD and install it on your Windows Workstation and start examining the tapes. Though even with other applications the access to data from Exchange and other agent backups is not easy without re-creating the originating infrastructure. Even if you had the experience and patience to set up a Tivoli system there is no method for importing foreign tapes.

Like the proverbial London bus, you don’t see one for ages and then two come along at once. In less then a week, one set of TSM tapes arrived as part of a forensic examination, and another for data recovery as the tapes had expired and the customer needed their Notes data restored.

Within the TSM heterogeneous infrastructure, data from many different sources can exist. In these cases the data was a mixture of Windows files, UNIX files and Notes mail database files. Using TSM was not an option, but we did find a TSM salvage utility that was available on one of the Tivioli forums. Unfortunately the writer had not seen data of the flavours we were attempting to recover so the program could not handle it. We decided the the only viable option was to create our own software for the purpose.

This was by no means a straightforward task, TSM data is encapsulated in multiple layers of data buffers each of which requires that its key structures are identified and then a process written to peel it away. Eventually, however, after much huffing and late night working we arrived at the data and were able to extract files, one customer had their email back and the other had evidence that might otherwise have not been available to them.

Tapes represent a valuable source of historical data for external investigations, and for when historical data is required under the regulatory regimes that rule over business areas such as banking. Yet there is a problem with tapes having expired, or being provided by a third party for examination the situation can arise where all of the data are present and correct on good condition well-recorded tapes, but there is not way to recover it.

Having spent the past 26 years of my life working out data from platforms as diverse as Honeywell 2000, VMS, ICL1900 and UNIX systems tackling TSM tape data recovery should have been a walk in the park but it still took us quite a while, and a lot of stress, but ultimately the results were good for all concerned.

When a file system has become severely corrupt, or has been wiped out by re-partitioning or re-formatting, the ability to examine every block of data within a partition, hard drive or RAID array to identify and extract specific fragments of a specific file could be your only hope of recovery. This may sound extreme but it does happen, especially with business critical systems such as email and database servers.

Back in the days of Exchange 5.5, Exchange 2000 and Exchange 2003 (pre service pack 1) the internal structure of the 4K or 8K data pages that made up the information store files had enough encoded data within them to allow the pages to not only be recognised but also to identify where in the file the page was located. The values that allowed us to do this included the page CRC (cyclic redundancy check) checksum value and the pgnoThis page number. These were two 32bit values that sat side by side at the start of the the page structure.

Whilst the CRC value allowed a page to be checked to see if the data was valid and the pgnoThis could identify if the page was in the correct position within the file, it did not allow any errors within the page to be rectified. Research by Microsoft apparently identified that approximately 40 percent of CRC errors identified were caused by single bit “bit flip” errors so, with the release of Exchange 2003 service pack 1 in May 2004, they replaced the CRC and pgnoThis values in the page header structure with an updated CRC value and an ECC value to allow these single bit errors to be corrected when the page is loaded.

The algorithm to produce the first CRC value is seeded with the page number. This means that Exchange can still determine if the page it read is the page it expected to read however the CRC value can now no longer easily be used to identify a page of Exchange data if you don’t know its location as is the case when scavenging data during a data recovery process. Therefore, new techniques have had to be developed. Microsoft’s changes to the ESE (Extensible Storage Engine) architecture to allow a single bit within a block of Exchange data mean it is no longer easy to recovery a block or page of Exchange data when the allocation information for that file no longer exists.

Data recovery work of this nature require an in depth knowledge of the internal structures of files and a lot of dedication to produce the best possible result and this is how we approach every data recovery we do.

References:
Extensible Storage Engine Architecture: http://technet.microsoft.com/en-us/library/aa998171%28EXCHG.65%29.aspx

A NAS storage device  in need of recovery was delivered to us last week. The customer had been using the device and on Friday evening all data was present and correct, but when the customer went to access the device on the Monday morning it was operational but there was no data present. So where had it gone and could a data recovery be achieved?

Once the data from the disks had been secured by imaging the disks and backing up the data our diagnosis began. The RAID unit was basically a Linux system in a box with four hard drives, and used the XFS file system to store data. Using the file system structures from within the XFS file system we were able to identify that the RAID used a stripe size of 64KB and with a standard data order, so far so good. The problem was that when we then examined that data from the disks using this RAID configuration, and attempted to follow the XFS INODE information to access the files something did not seem quite right. We dropped the data from the RAID onto a single disk and connected it to a Linux system, and were able to mount it without any errors, but only three directories were visible so it appeared that there had been either file deletion of a file system re-creation, but our utilities did not indicate a problem, so what was wrong?

On closer examination of the hard drive images, we were able to find the answer and it makes for quite scary reading. Yes, there had been a file system re-creation, but before this the RAID had been re-configured. A previous RAID configuration on the device had used a stripe length of 16KB and whatever had happened over the weekend had resulted in the RAID being reconfigured, rebuilt and a new file system written to it. Armed with this information the new RAID configuration setting were used with our data recovery software. When we examined the resultant data produced with these new settings, we were surprised to find that the previously existing filesystem was almost entirely intact and approximately 99% of the customers data could be recovered.

How was this possible? First, the reconfiguration of the RAID did not cause the parity blocks to be recalculated for the entire RAID volume, parity was only written in the stripes where new data was written. Second, the change in the RAID stripe size caused the allocation group size of the XFS filesystem to change, shifting the location of internal filesystem data and therefore preserving many of the previous XFS structures that we were then able to use to reconstruct the previous filesystem.

The moral of this story? We see many instances of NAS RAID systems that have apparent failures or where some level of rebuilding has taken place. This is not an attempt to disparage what is a highly important type of storage in terms of ease of use and cost, but it does demonstrate a failure to comprehend the risk of reliance upon a single storage device for critical data almost as if the fact that a RAID is in use gives complete protection against failure. For the cost of a NAS data recovery a tape device, or additional NAS units to backup up to, could have been put in place resulting in no need for data recovery work and no interruption to business and to sleep patterns.

It might seem an odd request, to copy tapes that went out with the ark, but if you have a dedicated system based around a VAX workstation (for example some CAM systems used VAX) you might still be using it or have a request for design information from a couple of decades ago. VAX systems were well built so will often be in good working order, but if the method of booting is from TK50 tape then the problems can begin.

One of the biggest problems we see with TK50 tapes in our tape data recovery lab is “stiction”, the tape sticks to the read/write head. This is a result of media decay, the material bonded within the media to keep it sliding over the heads smoothly breaks down and suddenly the tape that was running nice and quickly comes to an abrupt halt. Worse still, there is some momentum involved and the dragging of the tape over the heads whilst waiting for it to stop moving can leave a lot of the recorded surface as a gooey mess on the heads. The result, a nice bit of see-through tape and data that will never been seen again.

Is all lost when the media decays in this manner? Not always, in our data recovery lab we have developed methods for keeping the tape moving and stopping it sticking. Usually this allows a recovery of the majority of the data from tape, though we often only get hold of a tape after several failed attempts have already been made and there are already some holes in the media.

The thing to consider if you do have data stored on TK50, TK70 or half-inch open reel tape is that it could be suffering from decay, this could be getting worse, and if there is a likelihood that data will be needed at some time in the future it is worth weighing up the cost of a data migration project now against the cost of a data recovery exercise some time hence.

There are many data recovery software tools on the market these days that, in some circumstances, can provide a quick and cheap solution if the problem being experienced is relatively straight forward. Because of this market in data recovery tools anyone who has some IT skill and is capable of pressing the ‘go’ button in the data recovery software can claim to provide a service. What happens however if the problem is too complex for the recovery tool or there is a hardware fault?

We often see cases that have already been to other recovery companies and have been declare a ‘no-fix’, stating the data cannot be recovered, where we have been able to successfully recover the data the customer required.

We are able to do this because of our level of knowledge and the commitment we give to each and every job. We do this because we have a passion for the work and take pride in the service and results we can offer.

I have been reverse engineering backup formats, file structures and file systems for many years and have been writing software to recover data from some unimaginable scenarios and I am not the only member of the team with these credentials. We can therefore provide a high level of service to all of the jobs we see.

The level of service we can provide is not beholden to any third-party utilities, if we encounter a data recovery situation, legacy backup format or file system corruption that our existing tool set does not cater for we will develop the solution required, no-fix is not an answer.

Having faithfully adhered to a regime of nightly backup of a SUN UNIX system, the failure of the hard drive appeared to be an inconvenience that would soon be overcome for this customer. The hard drive having swiftly been replaced, the backup tape was brought back from storage and the restore process initiated. Five minutes in, however, and the DLT drive made a nasty noise, the cleaning lights came on and ufsrestore became ufs-cannot-restore.

We received a call from the customer wanting to know if the data from the backup tape could be recovered. The disk was with another data recovery company who had found that there were a significant number of bad sectors that meant the required data could not be recovered intact. This is not a smug self-congratulatory tale of “then we got everything back from the tape” as at the point where the restore failed there was nothing to be done, that was where the backup had been terminated as the result of a tape problem and no-one had noticed.

The process followed by the customer was to use ufsdump to create a backup file on the hard drive, then to use dd to transfer this to tape. This meant that they had an immediately accessible backup on-line in case of file deletion or corruption, and the fall back of a tape backup to cover against a total failure (or not in this instance).

So, the data could not be recovered from the hard drive nor from the tape, so was all lost? Well, no it was not. An examination of the disk revealed that the ufsdump archive could be partially recovered, but that the critical section at the start of the backup where all of the directory information (which includes slightly important data such as the file names) was lost forever. The tape, however, contained a valid backup up to the point of failure, and an examination of this data showed that it extended into the area of data where files were stored.

In the motor trade a “cut-n-shut” car is viewed as a dangerous beast that should never be allowed on the road, for this customer it was the answer to their UNIX data recovery problem as the data from two sources could be pieced together to create a completed backup and a full restore was initiated.

A happy story, a success, but how often are opportunities such as this missed? The best data recovery service companies, not the biggest but those who understand data, will always pick up on such opportunities to get the right result. But, if the data recovery work is being done by someone who just runs some software and if it says “failed” they move on to the next job then there will be unnecessary data losses.

Some people seem to think so, but have been dangerously mis-informed. The error correction used in a RAID5 array is there to provide a level of protection for what will often be business critical data, but can still only survive the loss of a single hard drive from the array. Even RAID 6 with double error correction can only cope with a failure of two drives.

You might wonder how likely it is that multiple disks will fail in a RAID, the answer is that it is quite likely.

Environmental factors, for example heat, can have a severe impact upon the operation of a hard disk drive. With a RAID array the disks are in the same enclosure and so an environmental factor will influence all hard drives.

Electronics and connectivity must also be taken into account. In a RAID 5 array all disks are connected to a single controller, and may be on a single data bus. Instability on the bus can result in hard drives being determined to have failed even when they have not. We have received RAID for data recovery with multiple hard drive failures, but under closer examination it has transpired that one drive has failed and resulted in three drives disappearing from the bus and being logged as failed.

Failure with a RAID 5 array can result from minor problems. A single bad sector on a disk will often result in that disk being flagged as failed. This makes sense as the failed sector cannot be read nor written so the hard drive should no longer be used and must be replaced. But, this does mean that a failure of two disk sectors on separate drives, that is 1024 bytes becoming unreadable, can result in a RAID no longer being accessible.

So, whilst RAID 5 brings many benefits in terms of capacity, performance and data protection, the security given by the RAID’s fault tolerance should not be used as an excuse to relax other procedures for data protection such as tape backup.