As Hamlet so eloquently put it “There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.”, just because something is outside of our normal experience does not mean that it is not possible.

One such case landed at our offices recently, 4 disks in a RAID 5 with apparently corrupted data files. Once the drives had been imaged and the data secured we set about examining the RAID and it all looked pretty good. The RAID scheme was a standard last to first data striping with parity shifting first to last every 256KB, and the file system appeared to be perfect.

Then we looked at the MS-Exchange EDB files. The PRIV.EDB started fine then after a while the EDB page checksums were all wrong. The PUB.EDB file did not even start with the correct header pages, though the data did have the appearance of Exchange Information Store data. Examining the data in free space for Exchange EDB headers we did find one for an EDB from the required date.

So, the file system was perfect but the data was not quite right, maybe not corrupted as such, but shifted.

At this point we had not looked further down the disk, but now we started to examine the data in more depth. A second data partition existed on the volume, but on the current RAID configuration the file system was not correct. We could find our way from the boot sector to the MFT, but then after a few sectors the entry numbering went awry. The RAID configuration was different for the second partition.

It is not usual for a RAID to be in this state, so it appeared to us that perhaps the RAID had somehow been re-configured, and that this explained the data issues on the first partition. Sure enough checking the allocation information for PRIV.EDB it was apparent that the data was correct if it was from a low numbered set of file system clusters, but incorrect if it was from high numbered ones, so perhaps the re-configuration had stopped after the bulk of the file system information for partition 1, but not before the end of the data.

Not finding the notion of playing spot the configuration change somewhere in the midst of a large volume of RAID data we set about a new approach.

We had 2 RAID orgnisations, one that worked for partition 1 up to a point and one that probably worked from somewhere in the middle of partition 1 though partition 2. The customer had a set of EDB files that they rather needed back and straddled the point where the RAID configuration changed.

We created 2 versions of the RAID data using each configuration and then extracted the required files.  EDB files have page checksums. And it is possible to work out the page number, so processing an EDB file you can tell if the data is almost certainly correct.  So the next stage was to process each EDB, check each page, and for any that failed check the equivalent page for the alternate file.

The result was stunning, EDB files that not only could be processed using the checking options of eseutil (the Exchange utility program) with minimal errors, and that could then be accessed via Exchange and all mail items examined.

Looking at this as a standard RAID data recovery the answer would have been “your data has gone forever”. Utilities such as R-Studio and Winhex could be used to access the data from the file system, but not to recover the files as the customer needed them, so anyone just playing point and shoot with a recovery utility would be on to a loser.

The answer for this recovery was to examine the data, to really check what was there and to not let preconceptions prejudice the outcome. By finding the facts, and letting them determine the course of the recovery and not letting the demons of “that cannot be possible” cause a distraction, the data was saved.

So, you are an investigator who has been handed a case containing 25 LTO4 cartridges from a TSM archive, now what?

Unlike relatively straightforward backup applications such as BackupExec or ARCserve you cannot simple buy a TSM installation DVD and install it on your Windows Workstation and start examining the tapes. Though even with other applications the access to data from Exchange and other agent backups is not easy without re-creating the originating infrastructure. Even if you had the experience and patience to set up a Tivoli system there is no method for importing foreign tapes.

Like the proverbial London bus, you don’t see one for ages and then two come along at once. In less then a week, one set of TSM tapes arrived as part of a forensic examination, and another for data recovery as the tapes had expired and the customer needed their Notes data restored.

Within the TSM heterogeneous infrastructure, data from many different sources can exist. In these cases the data was a mixture of Windows files, UNIX files and Notes mail database files. Using TSM was not an option, but we did find a TSM salvage utility that was available on one of the Tivioli forums. Unfortunately the writer had not seen data of the flavours we were attempting to recover so the program could not handle it. We decided the the only viable option was to create our own software for the purpose.

This was by no means a straightforward task, TSM data is encapsulated in multiple layers of data buffers each of which requires that its key structures are identified and then a process written to peel it away. Eventually, however, after much huffing and late night working we arrived at the data and were able to extract files, one customer had their email back and the other had evidence that might otherwise have not been available to them.

Tapes represent a valuable source of historical data for external investigations, and for when historical data is required under the regulatory regimes that rule over business areas such as banking. Yet there is a problem with tapes having expired, or being provided by a third party for examination the situation can arise where all of the data are present and correct on good condition well-recorded tapes, but there is not way to recover it.

Having spent the past 26 years of my life working out data from platforms as diverse as Honeywell 2000, VMS, ICL1900 and UNIX systems tackling TSM tape data recovery should have been a walk in the park but it still took us quite a while, and a lot of stress, but ultimately the results were good for all concerned.

When a file system has become severely corrupt, or has been wiped out by re-partitioning or re-formatting, the ability to examine every block of data within a partition, hard drive or RAID array to identify and extract specific fragments of a specific file could be your only hope of recovery. This may sound extreme but it does happen, especially with business critical systems such as email and database servers.

Back in the days of Exchange 5.5, Exchange 2000 and Exchange 2003 (pre service pack 1) the internal structure of the 4K or 8K data pages that made up the information store files had enough encoded data within them to allow the pages to not only be recognised but also to identify where in the file the page was located. The values that allowed us to do this included the page CRC (cyclic redundancy check) checksum value and the pgnoThis page number. These were two 32bit values that sat side by side at the start of the the page structure.

Whilst the CRC value allowed a page to be checked to see if the data was valid and the pgnoThis could identify if the page was in the correct position within the file, it did not allow any errors within the page to be rectified. Research by Microsoft apparently identified that approximately 40 percent of CRC errors identified were caused by single bit “bit flip” errors so, with the release of Exchange 2003 service pack 1 in May 2004, they replaced the CRC and pgnoThis values in the page header structure with an updated CRC value and an ECC value to allow these single bit errors to be corrected when the page is loaded.

The algorithm to produce the first CRC value is seeded with the page number. This means that Exchange can still determine if the page it read is the page it expected to read however the CRC value can now no longer easily be used to identify a page of Exchange data if you don’t know its location as is the case when scavenging data during a data recovery process. Therefore, new techniques have had to be developed. Microsoft’s changes to the ESE (Extensible Storage Engine) architecture to allow a single bit within a block of Exchange data mean it is no longer easy to recovery a block or page of Exchange data when the allocation information for that file no longer exists.

Data recovery work of this nature require an in depth knowledge of the internal structures of files and a lot of dedication to produce the best possible result and this is how we approach every data recovery we do.

References:
Extensible Storage Engine Architecture: http://technet.microsoft.com/en-us/library/aa998171%28EXCHG.65%29.aspx

A NAS storage device  in need of recovery was delivered to us last week. The customer had been using the device and on Friday evening all data was present and correct, but when the customer went to access the device on the Monday morning it was operational but there was no data present. So where had it gone and could a data recovery be achieved?

Once the data from the disks had been secured by imaging the disks and backing up the data our diagnosis began. The RAID unit was basically a Linux system in a box with four hard drives, and used the XFS file system to store data. Using the file system structures from within the XFS file system we were able to identify that the RAID used a stripe size of 64KB and with a standard data order, so far so good. The problem was that when we then examined that data from the disks using this RAID configuration, and attempted to follow the XFS INODE information to access the files something did not seem quite right. We dropped the data from the RAID onto a single disk and connected it to a Linux system, and were able to mount it without any errors, but only three directories were visible so it appeared that there had been either file deletion of a file system re-creation, but our utilities did not indicate a problem, so what was wrong?

On closer examination of the hard drive images, we were able to find the answer and it makes for quite scary reading. Yes, there had been a file system re-creation, but before this the RAID had been re-configured. A previous RAID configuration on the device had used a stripe length of 16KB and whatever had happened over the weekend had resulted in the RAID being reconfigured, rebuilt and a new file system written to it. Armed with this information the new RAID configuration setting were used with our data recovery software. When we examined the resultant data produced with these new settings, we were surprised to find that the previously existing filesystem was almost entirely intact and approximately 99% of the customers data could be recovered.

How was this possible? First, the reconfiguration of the RAID did not cause the parity blocks to be recalculated for the entire RAID volume, parity was only written in the stripes where new data was written. Second, the change in the RAID stripe size caused the allocation group size of the XFS filesystem to change, shifting the location of internal filesystem data and therefore preserving many of the previous XFS structures that we were then able to use to reconstruct the previous filesystem.

The moral of this story? We see many instances of NAS RAID systems that have apparent failures or where some level of rebuilding has taken place. This is not an attempt to disparage what is a highly important type of storage in terms of ease of use and cost, but it does demonstrate a failure to comprehend the risk of reliance upon a single storage device for critical data almost as if the fact that a RAID is in use gives complete protection against failure. For the cost of a NAS data recovery a tape device, or additional NAS units to backup up to, could have been put in place resulting in no need for data recovery work and no interruption to business and to sleep patterns.

It might seem an odd request, to copy tapes that went out with the ark, but if you have a dedicated system based around a VAX workstation (for example some CAM systems used VAX) you might still be using it or have a request for design information from a couple of decades ago. VAX systems were well built so will often be in good working order, but if the method of booting is from TK50 tape then the problems can begin.

One of the biggest problems we see with TK50 tapes in our tape data recovery lab is “stiction”, the tape sticks to the read/write head. This is a result of media decay, the material bonded within the media to keep it sliding over the heads smoothly breaks down and suddenly the tape that was running nice and quickly comes to an abrupt halt. Worse still, there is some momentum involved and the dragging of the tape over the heads whilst waiting for it to stop moving can leave a lot of the recorded surface as a gooey mess on the heads. The result, a nice bit of see-through tape and data that will never been seen again.

Is all lost when the media decays in this manner? Not always, in our data recovery lab we have developed methods for keeping the tape moving and stopping it sticking. Usually this allows a recovery of the majority of the data from tape, though we often only get hold of a tape after several failed attempts have already been made and there are already some holes in the media.

The thing to consider if you do have data stored on TK50, TK70 or half-inch open reel tape is that it could be suffering from decay, this could be getting worse, and if there is a likelihood that data will be needed at some time in the future it is worth weighing up the cost of a data migration project now against the cost of a data recovery exercise some time hence.

There are many data recovery software tools on the market these days that, in some circumstances, can provide a quick and cheap solution if the problem being experienced is relatively straight forward. Because of this market in data recovery tools anyone who has some IT skill and is capable of pressing the ‘go’ button in the data recovery software can claim to provide a service. What happens however if the problem is too complex for the recovery tool or there is a hardware fault?

We often see cases that have already been to other recovery companies and have been declare a ‘no-fix’, stating the data cannot be recovered, where we have been able to successfully recover the data the customer required.

We are able to do this because of our level of knowledge and the commitment we give to each and every job. We do this because we have a passion for the work and take pride in the service and results we can offer.

I have been reverse engineering backup formats, file structures and file systems for many years and have been writing software to recover data from some unimaginable scenarios and I am not the only member of the team with these credentials. We can therefore provide a high level of service to all of the jobs we see.

The level of service we can provide is not beholden to any third-party utilities, if we encounter a data recovery situation, legacy backup format or file system corruption that our existing tool set does not cater for we will develop the solution required, no-fix is not an answer.

Having faithfully adhered to a regime of nightly backup of a SUN UNIX system, the failure of the hard drive appeared to be an inconvenience that would soon be overcome for this customer. The hard drive having swiftly been replaced, the backup tape was brought back from storage and the restore process initiated. Five minutes in, however, and the DLT drive made a nasty noise, the cleaning lights came on and ufsrestore became ufs-cannot-restore.

We received a call from the customer wanting to know if the data from the backup tape could be recovered. The disk was with another data recovery company who had found that there were a significant number of bad sectors that meant the required data could not be recovered intact. This is not a smug self-congratulatory tale of “then we got everything back from the tape” as at the point where the restore failed there was nothing to be done, that was where the backup had been terminated as the result of a tape problem and no-one had noticed.

The process followed by the customer was to use ufsdump to create a backup file on the hard drive, then to use dd to transfer this to tape. This meant that they had an immediately accessible backup on-line in case of file deletion or corruption, and the fall back of a tape backup to cover against a total failure (or not in this instance).

So, the data could not be recovered from the hard drive nor from the tape, so was all lost? Well, no it was not. An examination of the disk revealed that the ufsdump archive could be partially recovered, but that the critical section at the start of the backup where all of the directory information (which includes slightly important data such as the file names) was lost forever. The tape, however, contained a valid backup up to the point of failure, and an examination of this data showed that it extended into the area of data where files were stored.

In the motor trade a “cut-n-shut” car is viewed as a dangerous beast that should never be allowed on the road, for this customer it was the answer to their UNIX data recovery problem as the data from two sources could be pieced together to create a completed backup and a full restore was initiated.

A happy story, a success, but how often are opportunities such as this missed? The best data recovery service companies, not the biggest but those who understand data, will always pick up on such opportunities to get the right result. But, if the data recovery work is being done by someone who just runs some software and if it says “failed” they move on to the next job then there will be unnecessary data losses.

Some people seem to think so, but have been dangerously mis-informed. The error correction used in a RAID5 array is there to provide a level of protection for what will often be business critical data, but can still only survive the loss of a single hard drive from the array. Even RAID 6 with double error correction can only cope with a failure of two drives.

You might wonder how likely it is that multiple disks will fail in a RAID, the answer is that it is quite likely.

Environmental factors, for example heat, can have a severe impact upon the operation of a hard disk drive. With a RAID array the disks are in the same enclosure and so an environmental factor will influence all hard drives.

Electronics and connectivity must also be taken into account. In a RAID 5 array all disks are connected to a single controller, and may be on a single data bus. Instability on the bus can result in hard drives being determined to have failed even when they have not. We have received RAID for data recovery with multiple hard drive failures, but under closer examination it has transpired that one drive has failed and resulted in three drives disappearing from the bus and being logged as failed.

Failure with a RAID 5 array can result from minor problems. A single bad sector on a disk will often result in that disk being flagged as failed. This makes sense as the failed sector cannot be read nor written so the hard drive should no longer be used and must be replaced. But, this does mean that a failure of two disk sectors on separate drives, that is 1024 bytes becoming unreadable, can result in a RAID no longer being accessible.

So, whilst RAID 5 brings many benefits in terms of capacity, performance and data protection, the security given by the RAID’s fault tolerance should not be used as an excuse to relax other procedures for data protection such as tape backup.

It is worth debunking the myth quickly. With a hard disk there is a single layer of recordable material on each side of each platter. When an area of this disk is written to, whatever was at the area previously has gone forever. No need to write over it seven times, no need to sandblast the disk.

If a hard disk drive is re-formatted and then new data written to it, some or all of the original data will be written over and will never be found, there is no hard drive data recovery technique that will get it back.

So, why does data keep turning up where it shouldn’t? Medical records on hard disks in computers that have been sold on etc.? Just as when military laptops are left in cars or on trains, this is not a technical issue, just shear carelessness. With the exception of a few sectors that are mapped out by a hard drive’s defect management system there is nothing to stop all data being eradicated from any hard disk, and however much Brasso is used you can’t polish through to any older data.

Windows NTFS, for example, marks a file as deleted but until the MFT entry is reused all of the file’s allocation information is still present. Other systems clear some or all of a file’s allocation information as soon as there is deletion, which is why the recovery of deleted files from heavily fragmented FAT file systems is such a nightmare.

What about UNIX? Well there are a wide range of file systems available under UNIX and LINUX, common amongst these is UFS. We have received a number of UFS data recovery jobs lately where the problem has been file deletion and the owners of the hard drives have been told by some data recovery companies and other UNIX specialists that file cannot be un-deleted.

This assertion has some basis in truth, there is no generalised way of pointing a software application at a UFS file system and recovering the deleted files. The inode for a file is cleared when the deletion is done and so the primary allocation information is no more. What is ignored is that there is more allocation information then that stored in the file inode and that this can be used to make a substantial recovery of some files. What is required is for the data recovery professional to have an understanding of data rather than just being a user of data recovery software.

So the 12 months up until today has seen an 80% success rate in the recovery of deleted files from UFS file systems when the customer had been told by several other “experts” that the chances were 0%. This was not achieved through clever software or massively in depth R&D but by having the right attitude and knowledge.

So if you have a problem with a UNIX file system and are being told flatly that it is not possible, get an explanation as to why, and get some other opinions, in case you are talking to the wrong UNIX data recovery company.