What is almost certain is that these regulatory paper tigers are about to be forced to become real, and with it will come new zeal for enforcing regulatory compliance leading to an increase in eDiscovery and eDisclosure requests.

If, for example, the FSA knock at the door with a demand for information what do you do? There will be a deadline, and there are plenty of cases of well known companies being fined very heavily for non-disclosure within the time specified, so life can get fraught and very expensive.

The reality is that, unless there are sophisticated data management systems in place, by the time a demand for disclosure is made it is often too late. Where large volumes of data are stored in tape archives has to be re-instated somewhere, inspected and the required data found and returned. This can be quite an onerous task even with a properly maintained archive catalog.

Our experience is that vague requests for information regarding the transfer of data from a couple of hundred tapes often undergo a metamorphosis to become an urgent requirement to track down particular files from some or all of the tapes, “and it must be done by next Tuesday”. It usually transpires that the deadline is for regulatory purposes.

There is a lot that can be done in advance of any disclosure demands, including the following:

• Assess the potential task. If you hold large volumes of data on tape do you have all of the hardware and infrastructure in place to recover data when required?
• Is the archive adequately catalogued so to allow the rapid identification of any required data.
• Do you have the operational capacity within your IT department to deal with a demand for data whilst still maintaining daily operations.

If the answer to any of these is “no” then the cost saving in avoiding re-cataloguing and possibly data migration to new media might soon be revealed to be a false economy.

Well the problem is that the data from any single backup is distributed throughout the data from several other backups, and to identify the data from the required backup you have to read all of the data, including that from the other backups. For example, one backup set might occupy only 10% of the space on an LTO4 tape, but the restoration might require that 100% of the tape be read as each block must be read to see if it belongs to the required backup set.

The problem then with an archive wide tape data migration project is that if there are 1000 tapes containing backups from 100 different sources, the migration process might require that each tape be read 100 times, so the entire reading process will be the equivalent of transfering data from 100,000 tapes.

A bit more than one of those jobs to leave for a wet afternoon.

I’m not denigrating multiplexing as a procedure. It improves backup performance and logistics, and it is backup that occupies the most time. Unless we are very unlucky a lot less time is spent restoring data from tapes than is spent writing to them. It is just that when the time comes to move a large amount of data the option of using the original backup software and systems can be just too daunting and resource hungry.

There are ways around the problem. We write our tape data migration software to cater for this type of issue, so that a faster high level scan of the tapes can then mean that the tapes have only to be read once. This means that the data migration process can be completed in an acceptable time without the need for a massive investment in infrastructure and staff.

So what happened? Back in the 1980’s hard drives were expensive, not much trusted and low capacity. Optical disk offered a far higher capacity, and being a removable media technology, the ability to expand storage by simply using more disks. Tape was then seen as a technology in transition, again not adequate on the capacity front, and there was a perception of reliability issues.

Optical disk storage ticked all of the boxes, appeared to be rugged, WORM variants satisfied those for whom data retention was an issue, and they were shiny. Not being as flippant as it might seem, they had the appearance of robustness, and were close enough in terms of usability to resemble hard drive, and so felt comfortingly familiar and trustworthy.

These days we only really see Magneto Optical when people want to migrate away from it or need an optical disk data recovery. Phase Change optical died with the loss of market share in the early 1990’s, some of the ablative formats soldiered on at the high end in the banking and pharmaceutical sectors, and the mainstream of the format evolved into UDO (Ultra Density Optical). But, with only one company pushing the format, against the massive technological advances in hard drive storage and LTO tape, the inevitable happened and the format died.

There is still a lot of data stored on optical disks around the world, we see a steady stream through our optical data migration and data conversion operation, they come from VMS systems, medical scanners and a variety of UNIX platforms, and all finally abandoning the technology for pastures new.

Other optical technologies do still thrive in the form of CD, DVD and Blue Ray, all of which are excellent for data transfer, and for gaming and software distribution, but no longer for the long term archive of data.

The more complex the system the more pitfalls when attempting to migrate the data. Where data is being streamed from multiple sources a technique known as multiplexing is often preferred as this gives the best use of the bandwidth available for data backup. The problem comes with restoring as each data set is potentially spread across multiple tapes, and the restore process using the originating backup software might well require that each set be restored individually and so you can end up having to read each tape multiple times. This effectively means that if you have an archive of 1000 tapes, you have to read 5,000 or 10,000 tapes to restore everything.

Factor in the need to have systems set up for the restoration, and that this could include NAS and Filer devices, Exchange Servers and SQL servers, and the temptation might well be to lock the archive and lose the key. This is fine until you need some data back or the regulators come-a-knocking.

Faced with customers who are despairing at the size of the data migration project to turn their 500 or 1000 DLT tapes the contain NetBackup, TSM or NetWorker archives into a smaller number of LTO3 or LTO4 tapes we decided the only approach was the development one. Trying to restore tapes under these circumstances using the originating software is just too restrictive, requires too much time and other resources from an IT department.

Having developed our own data migration utilities and so being able to scan the tapes and then restore them sequentially means that a 1000 tape data migration job might require only 1000 tape reads and that these can be done in parallel over multiple systems. Use 10 systems and rather than 1000 tapes requiring 10,000 read passes, we end up needing only 100. This equates to a job taking less than a month rather than several years. An extreme example but a fair illustration of the point.