smartData Protection & Archiving

The bytes are the easy part

Moving an archive is a metadata and access-path problem wearing a data-transfer costume.

Copying twenty years of files onto newer media is arithmetic. Keeping retention clocks, ownership, checksums and application stub paths intact across the move is the work that actually decides the outcome.

Aban Smart designs, supplies, integrates and supports archive migrations onto QStar, INCOM/StorEasy and DISC platforms across the GCC.

Almost every migration plan we are shown opens with a throughput calculation. Petabytes divided by an assumed transfer rate, multiplied by a contingency factor, presented as a schedule. That number is rarely the thing that slips. Projects overrun because a decade-old library returns read errors on a meaningful share of its cartridges, because nobody could establish which retention dates were already running, or because a PACS was pointing at stub paths that the new platform quietly renamed. This page is about that second category of problem.

Why the archive has to move at all

The trigger is usually external rather than strategic. Tape drives fall out of vendor support and spares become auction purchases. Archive software reaches end of life and the last engineer who understood its catalogue has retired. Media passes the manufacturer's rated shelf life, which is a statistical statement about degradation, not an expiry date, but auditors treat it as one anyway. Sometimes it is a vendor exiting the market, a data-centre consolidation in Dubai or Riyadh, or simply a floor of ageing hardware drawing power to hold data nobody reads.

None of those triggers give you an open schedule. Each one carries a hard date, and that date sets how much verification you can afford before cutover. Establishing it early changes the design.

Staged archive migration with parallel runMigration proceeds from assessing and indexing the legacy archive, validating that the source media is still readable, copying in parallel, verifying checksums and metadata, cutting over the access path, and only then decommissioning the legacy system. Both platforms run in parallel until cutover, and metadata and access paths must be preserved alongside the bytes.BOTH SYSTEMS LIVE IN PARALLELLEGACY ARCHIVEstays authoritative until cutover1Assess &index legacy2Validate sourcereadability3Parallel copy4Verify checksums& metadata5Cutoveraccess path6DecommissionlegacyNEW PLATFORMproven before it is trustedCUTOVERMUST SURVIVE THE MOVE — NOT JUST THE BYTESByteschecksum verifiedMetadata & indexnot just the filesAccess pathslinks, mounts, IDsNothing is deleted until the new copy is verified and the access path works.Step 2 often reveals unreadable media — the point of validating the source before committing to a schedule.
Conceptual archive migration flow. Legacy and new platforms run in parallel until a verified cutover, with metadata and access paths migrated alongside the data. Final topology depends on verified product compatibility.

Old and new run in parallel until verification completes.

What has to survive besides the file content

A file is not the unit of value in an archive. The unit of value is the file plus everything that lets someone find it, trust it, and prove how long it must be kept.

  • Directory structure and path depth. Some targets impose path-length or character limits that the source never had.
  • Timestamps. Created, modified, last accessed and archived are four distinct facts. Naive copies collapse them into one and reset the rest to the migration date, which destroys any age-based retention rule.
  • Ownership and ACLs. Cross-domain moves and NAS-to-object transitions are where permission models stop mapping cleanly.
  • Custom metadata and tags. Case numbers, patient identifiers, study codes. Often held outside the file, in the old archive's catalogue.
  • Retention state and the remaining clock. A record with three years left must arrive with three years left, not a fresh full term and not an expired one.
  • Checksums and integrity records. The old system's hashes are your only evidence about what the file was before you touched it.
  • Stub and link paths. Covered below, because this one strands more projects than the rest combined.

Migration risks and the control that addresses each

Risk What it looks like in practice Control
Unreadable source media Cartridges or discs fail mid-run, discovered at 60% complete Full read-verification pass on the source before the cutover plan is written
Broken stub paths Applications return "file not found" for archived items after cutover Path inventory, remapping table, and application-level retrieval testing before writes are switched
Reset retention clocks Records become deletable early, or lock for a fresh full term Export retention state as data, import explicitly, reconcile expiry dates by sample
Silent metadata loss Timestamps and ACLs arrive flattened, noticed months later Metadata-level verification on every batch, not only content verification
Schedule pressure at the end Verification is cut short to hit a support-expiry date Parallel running, so the deadline is met by the new system going live, not by finishing every byte
Premature decommissioning Source destroyed before a gap is found Defined read-capability tail period and a written sign-off gate
Undetected duplication The same dataset migrated twice through overlapping batches Migrate only files absent from the target, then reconcile object counts per batch

Preserving metadata and access paths

Tiering and archiving software leaves a stub, a link or a placeholder in the original location so that applications keep working after the real data has moved to tape, optical or object storage. QStar Network Migrator, for example, replaces the original with a stub when it migrates a file, and the file remains reachable from its original file system path. Those stubs are the archive's user interface. Applications, scripts, DICOM nodes and mapped drives all hold references to them.

When the archive moves, every one of those references has to resolve to the new location, or be rewritten. Two failure shapes are common. Either the stubs are recreated correctly but point at an old server name that no longer answers, or the target enforces a naming rule the source did not and paths shift silently for a subset of files. Neither surfaces during a bulk copy. Both surface the first time a clinician or a case worker requests a fifteen-year-old record.

The practical control is a path inventory taken before anything moves, a documented remapping table, and retrieval testing driven through the actual application rather than through a file browser. Where the old catalogue holds metadata outside the files, that catalogue has to be exported as data in its own right. If it cannot be exported, that limitation belongs in the plan before the contract is signed, not after.

Proving the source before you plan the move

Legacy media is not uniformly readable, and you do not know which parts are not until you read them. A cartridge written in 2011 may have lived in a rack at 30°C for years. Optical platters may have surface damage on a small percentage of the set.

A read-verification pass across the source, before the cutover plan is finalised, converts an unknown into a number. That number changes the design. If a few percent of a tape set turns out to be unreadable, as an illustration, the conversation shifts to specialist recovery, to secondary copies, or to a documented and accepted loss. Discovering it in week nine of a migration turns the same conversation into an incident.

Migration tooling helps here. QStar's Technology Migration Tool is built to read archives written by older or non-QStar software, including LTFS, UDF and Plasmon-era file systems, and supports many-to-one consolidation from low-capacity source media onto current generations. It also runs in time-boxed increments that suspend and resume, which suits a source you are handling carefully.

Running old and new in parallel

A big-bang cutover on an archive assumes verification is fast. It is not. Checksum comparison across hundreds of millions of objects takes weeks, and until it completes you have no evidence that the new archive holds what the old one held. A cutover weekend gives you a switch with no proof behind it and no obvious way back.

Parallel running separates the two decisions. The new archive takes all writes from an agreed date. Reads are served from whichever system currently holds the object, with the migration engine transparently retrieving from the source when a request arrives for something not yet moved. The source goes read-only. Bulk movement continues in the background at whatever pace the media allows. This is the pattern the older PoINT Storage Manager migration collateral described in 2016, and it remains the sane shape for a file-system move today, whatever product implements it.

Two things make it reversible. The source stays intact and readable until sign-off, and the write cutover date is recorded so the delta written to the new system is a bounded, identifiable set. If something fundamental is wrong, you fall back to a known state plus a known list of changes. That is a rollback plan. A completed one-way copy is not.

Verification and what counts as done

Verification runs at two levels, and they cost very differently. Metadata comparison checks that names, sizes, timestamps and attributes match between source and target. It is fast and catches most structural faults. Full content verification recomputes and compares hashes for the file bodies. It is slow, and it is the only thing that proves the bytes survived.

Most estates use both: metadata verification on everything, full content verification on all high-value or regulated datasets plus a defined random sample of the rest. State the sample rate and the tolerance in the plan. "Ninety-nine point nine percent verified" is meaningless unless someone has decided in advance what the remaining fraction may contain.

Reconciliation is separate from verification. Object counts, total capacity and per-directory totals should agree between source and target, and where they do not, each difference needs a named explanation. Done means the reconciliation report is signed by whoever owns the records, not by the storage team.

Decommissioning without losing the tail

The source is not safe to destroy at cutover. It becomes safe when verification is signed off and a defined tail period has passed without a retrieval gap being reported. Six to twelve months is a common range, but it should be set from your own retrieval patterns rather than borrowed.

During that tail you need retained read capability, which means keeping at least one working drive, the software able to mount the old format, and someone who knows how to use it. Budget for it. It is routinely forgotten and then bought back at short notice.

When disposal comes, media sanitisation is a records decision as much as a technical one. Rewritable tape can be cryptographically erased, or degaussed, though degaussing an LTO cartridge wipes its servo tracks and retires the cartridge for good. Written optical WORM has no erase operation, so physical destruction with a certificate is the only path, and the certificate is the evidence your auditor will ask for.

For the target architecture itself, see long-term data archiving, the archive management software that runs the policies, and tape and LTO systems if LTFS is the destination. The full range sits under solutions. If you are holding an archive with a support cliff in front of it, request an assessment and start with the read-verification pass.

Related resources

Manufacturer documentation relevant to this page. Availability, specifications, and configurations are subject to verification.

PDFTechnical guide
Legacy document

PoINT Storage Manager: File System Migration

File-system migration module overview. Dated collateral; migration context only.

Vendor:
PoINT (INCOM)
Date:
2016
Format:
PDF
Size:
2.0 MB

Provided as historical reference. Not current product documentation; confirm current availability and configuration.

Frequently asked questions

Longer than the transfer arithmetic suggests. Source media condition, the number of small files, and the retrieval latency of the old library usually dominate over raw bandwidth. A realistic figure comes from a measured pilot batch against your actual source, not from a datasheet rate. We will not quote a duration before that pilot has run.

Turn your requirement into a defensible architecture

Share the workload, capacity, retention, access, and resilience requirements. Aban Smart will identify the next discovery inputs and the appropriate engagement path.