New long term archiving technologies

A few of our clients have expressed concerns about their aging Microform archives and many are having problems finding reliable service and parts for these products to maintain it’s viability in their organizations. So we took some time to look into this carefully and here is what we found.

Archiving to External Hard Drives:

Archiving data to an External Hard Drive can be a fast and economical solution and is an excellent way to back up your data in the short term.

The advantages to storing data on External hard drives are the ease with which you can retrieve this data, plus in can be easily accessed from multiple systems of different operating systems, such as Windows, Mac and Linux systems. However, hard disk backups may not always be the best long-term solution if you are archiving data for long periods of time. The data on hard disks stored for more than one year without being started up can begin to become unreliable. Simply turning on each archive disk and reading all of the data on the disk can prevent this problem, but this plan requires diligence and often is not possible for some archivist.

We had a hard time believing that data degradation could begin so quickly on a hard drive so we decided to perform a few tests of our own. We had several IDE hard drives that were all over 10 years old and one RLL drive that we believe has not been turned on since the late 80’s and the oldest IDE has a manufactured date of 1994. Sadly about 25% of the drives no longer worked, a few ran for a few minutes and we were able to access some of the data but within one hour of operation we were no longer able to access any of the data stored on the drives. The rest of the drives we were able to retrieve all of the data on the drives and continued to use them for a few days with no issues even the RLL drive.

We reached out to a few Hard Drive manufactures to see what the storage life expectancy was on these economical products and this is what we learned,

Hard Drives all have the following weaknesses/concerns in regards to long term storage.

1. Magnetic thermal decay of recorded bits and control signals

2. Media corrosion

3. Media lubricant evaporation

4. Fluid dynamic bearing oil evaporation

5. Electronics corrosion and degradation

Hard Drives are physical mechanisms, and there’s always the chance of mechanical failure. It was only because Google and Carnegie-Mellon studied vast numbers of hard drives for long periods of time that they were able to reveal clearly the differences between specifications and performance. A particular disk product may only be on the market for about 18 months where the service life of a spinning drive might be five years. Reliability based on experience in service of a particular product will typically be available after the product is obsolete in most cases. Useful reliability data about a particular product will necessarily be a prediction based on testing a sample of early production units.

ReliaSoft’s ALTA provides a tool called distribution wizard which helps determine the best lifetime distribution for a data set. The wizard estimates the parameters for each of several distributions, compares the log-likelihood values and then recommends the one that is the best statistical fit for the data.

Consider this; most hard drives on the market today only carry a one, three or a five year warranty. Hard Drives sold in the early 90’s often carried a warranty of 20 years! We believe competitive pressures in the market are influencing hard drive manufactures to look for ways to cut cost and lower product quality to match the life of the systems these products are designed for. We replace most computers every three to five years. It doesn’t make much sense to manufacturer drives to last 20 years when the computers they are going in will likely be out dated in three to five years. For most archivist External Hard Drives simply are not a good fit, here are a few more choices that may better address the needs of professional Archivist.

Archiving to tape:

Many Archivists archive their data to Linear Tape-Open (LTO) tapes because of the known life expectancy. LTO drives store data on tape cartridges that, when properly stored, are considered to have a lifetime of 15 to 30 years. We have found claims from tape manufactures that the life expectancy is 30 years. It is a common practice to create two tapes for an additional level of safety and the result is an extremely reliable long-term backup solution. We recommend that Archivist review the tape technology every decade to assess the need to have archived data on tapes migrated to the latest tape technology to ensure against degradation and to be certain the current technology has the ability to read the type of tapes used at the time the data was archived.

There have been several generations of LTO tape technology available from several manufacturers. The current standard is LTO-5, with a maximum uncompressed capacity of 1.5TB per tape. However you decide to back up your media, for the greatest level of safety, it’s ideal to have two sets of backups, including one that’s stored offsite in a secure location in case of fire, theft, or natural disaster.

Many industries have taken measures to safeguard their data, and most of these industries have settled on Write-Once/Read-Many (WORM) technology as today’s most secure long-term storage solution. When this media is stored and handled properly WORM technology is non-editable data storage that cannot be over-written, altered or corrupted.

Archiving to Optical Storage:

The life expectancy of optical storage disc’s depends upon a number of factors including such things as the intrinsic properties of the materials used in the disc’s construction, its manufactured quality, how well it is recorded and its physical handing and storage. As a result, the life span of a recorded disc is extremely difficult to estimate reliably. However, to calculate optical disc’s life spans within some practical timeframe blank media manufacturers do conduct accelerated age testing by subjecting samples of their discs to environments much beyond those experienced under normal storage conditions. Generally speaking, only the effects of varying temperature and humidity are considered. The results some optical disc manufacturers have claimed life-spans ranging from 50 to 200 years for CD-R discs. Most consultants we reached out to for this article only recommended using optical disc for storage for length’s less than 10 years. There is quite a bit to consider before you decide on an optical storage solution for your archiving needs. It is recommended Discs are best stored upright, like a book in “jewel” cases that are designed specifically for CDs/DVDs. Ideally, store the cases in plastic or steel containers manufactured specifically for that type of use in cool, dry storage that is free of large temperature fluctuations. Generally, useful life will be increased by storing discs at a low temperature and low relative humidity, since chemical degradation is reduced in these conditions. Store at 62-70 degrees F. and 35-50% relative humidity. Fluctuations in the storage area should not exceed +/- 2 degrees F. in temperature; relative humidity should not fluctuate more than +/- 5%. Another thought should be supplied power to the storage facility where the disc’s will be kept, if your in Atlanta and it’s July you wouldn’t want the power staying off in the facility for more than a day or so.

Archiving to Flash Media:

Flash Drives, often referred to as “thumb drives” are Solid State Device’s ( SSD ) and have no moving parts to fail mechanically. These types of drives are typically not sensitive to orientation, vibration, or shock. Each sector of a flash-based SSD can be written to only a limited number of times (typically 1-5 million) before it fails. Some software controllers manage this limitation in such a way that it is believed that drives when used regularly can last for many decades. However once the drive is written to and stored away for long periods of time we found little to no projects that involved the testing of longevity of the data. This is a fairly new product and just does not have the track record at this time for our approval as a long term storage device. We recommend using solid state devices no more than 5 years for archiving data with out regular use of the device.

In Summary:

Other preservation strategies like replication and migration are necessary for the long-term preservation of your data when using any type of digital media for long term storage. Digital sustainability implies a more active and continuous process mostly due to the ever changing and evolving technology products and market. We have found the majority of professional archivist are working hard to replace Microfilm by migrating data archives to digital media such as LTO WORM tapes with policies in place to review data migration to the newest media every 10 years based on availability of current hardware to read and safely recover archived data. We also recommend keeping your Microfilm even after you have migrated copies to digital media as the Microform industry is not completely out of the picture yet, despite low demand and limited access to replacement parts and services.