smartData Protection & Archiving

The tier you read from

Spinning media earns its cost when recall latency, not capacity price, is the binding constraint.

Disk and object archive systems sit between primary storage and cold media. The design question is rarely how many terabytes. It is whether applications reach the data through a filesystem path or an S3 bucket.

Configurations are quoted against your access pattern and namespace size, not against a headline capacity figure.

Every archive has a tier where data still has to come back quickly. A radiologist opening a prior study, a rendering job pulling source frames, an investigator pulling three months of camera footage: none of these tolerate a robot fetching a cartridge. That tier is built from disk, usually presented as an object store, and it is the most expensive part of an archive per terabyte. Getting its size and its access model right is what keeps the rest of the design affordable.

Disk, object and S3 archive storage topologyData is ingested over SMB, NFS or S3 into a disk cache tier, then written to an erasure coded object store exposing an S3 endpoint, with an optional cold tier on tape or optical WORM. Durability is a choice between erasure coding and replication. Object storage suits archive because of its flat namespace, metadata at scale and absence of directory-tree limits.INGESTIngestApplications andscanners overSMB / NFS / S3TIER 1Disk cache tierFast landing zonefor recent andrecalled dataTIER 2Object storeErasure codedS3 endpointFlat namespaceOPTIONALCold tierTape or opticalWORM, deepretention copyrecallDURABILITY CHOICEErasure codingk data + m parity fragmentsSurvives m concurrent failuresAbout 1.3x to 1.5x raw overheadBest for large archive objectsReplicationN complete copies per objectSurvives N minus 1 failuresAbout 2x to 3x raw overheadSimpler, faster small-object readsWHEN OBJECT BEATS FILE FOR ARCHIVEFlat namespaceNo directory tree towalk, split or rebalanceMetadata at scaleRich per-object metadata,indexed and searchableNo tree limitsBillions of objects inone flat address space
Conceptual archive storage topology across disk cache, erasure coded object storage with an S3 endpoint and an optional cold tier. Final topology depends on verified product compatibility.

Erasure coding trades overhead for rebuild behaviour.

Four jobs a disk tier is asked to do

The same hardware gets bought for four quite different reasons, and the correct configuration differs in each case.

  • Ingest landing. Data arrives faster than the media tier can absorb it. Disk buffers the burst and releases it at the drive's comfortable rate.
  • Cache in front of cold media. Recently written and recently read objects stay resident so that the common request never triggers a mount. Cache hit ratio, not raw capacity, is the number that matters.
  • Backup repository. A landing zone for backup software before copies are pushed to a second location or to removable media. This is the tier most often targeted during an intrusion, which is why retention enforcement belongs here rather than only downstream.
  • Active archive. Data that is formally archived, removed from the backup scope, but still read often enough that offline storage would be an operational nuisance.

Confusing the cache case with the active archive case is the most common sizing error we see. A cache can be small and fast. An active archive has to hold the whole working set or it behaves like neither one thing nor the other.

Object versus file access for archive

A filesystem is a tree. Every lookup walks directory entries, and metadata operations queue behind each other. That model is fine at millions of files. At hundreds of millions it becomes the bottleneck long before the spindles do, and backup, indexing and antivirus scans of the namespace stop completing in a working day.

Object storage discards the tree. Objects sit in a flat namespace inside buckets, addressed by key, with arbitrary user metadata attached to each object. Listing and lookup costs stop tracking directory depth, and the metadata you need for retrieval travels with the data instead of living in a separate database that can drift out of step.

The practical argument for object access is integration. Applications that speak S3 can write straight to the archive without a mapped drive, a mount point or an agent. QStar's product line reflects both routes: Kaleidos is an S3-compliant distributed object platform that scales across nodes and can extend the object store onto tape and commercial cloud, while the S3 Application Gateway takes storage already presented over SMB and NFS and exposes it as buckets, so an S3-native application can use infrastructure that was never built for it.

File access is still the right answer in three situations. When the writing application only knows SMB or NFS and will not be changed. When users need to browse the archive in a file manager as part of the workflow. And when an existing HSM or archive manager already presents a namespace that your operational procedures are written around. QStar's Object Storage Manager sits deliberately in this space, offering CIFS and NFS shares over an object engine with a REST API alongside, so the access method is a presentation choice rather than an architectural commitment.

File access Object access
Namespace behaviour Degrades as directory counts grow Flat, scales with metadata index
Application integration Mount point or share S3 API call
Metadata Limited to filesystem attributes Arbitrary key-value per object
Typical fit Existing workflows, user browsing New applications, large namespaces
Retention control Filesystem or appliance-level Per-object lock semantics
S3 gateway in front of archive tiersS3 clients talk to an S3-compatible endpoint that applies buckets, object lock and lifecycle rules, then translates requests to back-end tiers. Writes are acknowledged from disk cache, while reads of data already on tape or optical are recalled asynchronously and take longer.S3 CLIENTSS3 ENDPOINTARCHIVE TIERSBackup softwareS3 SDK / CLIArchive & MAMBuckets & keysObject Lock (WORM)Lifecycle & tieringGateway translationRetention set at thebucket, not the appDisk cachemillisecondsTape libraryseconds to minutesOptical WORMseconds to minutesPUTrecallWrite lands fastPUT acknowledged from disk cacheCold read takes timeAsync recall — mount, seek, stream
Conceptual S3 gateway fronting archive tiers, with object lock at the bucket layer and asynchronous recall from cold media. Final topology depends on verified product compatibility.

Writes land fast. Cold reads do not.

Erasure coding and durability maths

Two mechanisms protect data on a disk tier, and they trade different things.

Replication keeps whole copies. Three copies of an object means three times the raw capacity for one usable terabyte, and the recovery path is a straight read from a surviving copy. It is simple, it is fast to repair, and it is expensive.

Erasure coding splits an object into k data fragments and adds m parity fragments, distributing all k+m across failure domains. Any k fragments reconstruct the object. As an illustration only, a 10+4 scheme carries 40% overhead and survives four simultaneous fragment losses, where triple replication carries 200% overhead to survive two. That is the whole appeal: better failure tolerance per terabyte purchased.

The cost is on the recovery path. Rebuilding a replicated copy is a copy operation. Rebuilding an erasure-coded fragment means reading k fragments from across the cluster and recomputing, which consumes network and CPU while the system is already degraded.

This is why the advertised durability figure deserves scepticism. Vendors quote durability as a probability derived from an assumed rebuild window. Double the time to rebuild, because the drives are large, the cluster is busy or the replacement disk arrived a week later, and the exposure window doubles with it. Ask instead how long a full rebuild of your largest drive size takes under production load, and whether the system throttles rebuild to protect client I/O. QStar describes Kaleidos, for example, as combining distributed copies with erasure coding and repairing without taking the service down. That is the behaviour to put under test during commissioning, rather than the durability number to accept on paper.

Object Lock as a retention mechanism on this tier

S3-compatible archive storage can expose Object Lock, which applies a retain-until date or a legal hold to an object version. Where it is available it is the cleanest retention control at this tier, because the rule attaches to the object rather than to a share permission that an administrator can change.

Two cautions. Object Lock is not universally implemented across S3-compatible products, and governance mode permits privileged override by design, so the mode matters as much as the feature. And a retention lock on disk still lives on rewritable media under one administrative domain. It reduces the risk of deletion by a compromised account. It does not give you the media-level write-once property described on our WORM storage appliances and optical archive systems pages.

HIT-branded archive storage in the INCOM line

INCOM Storage supplies archive storage systems under the StorEasy and HIT brands. These are one supplier and one product line, not two vendors. The HIT name covers the HMS archive storage series and a separate HDL line, and we do not publish capacities, media types or model designations for either, because the documents in circulation do not agree closely enough to be quoted.

Where a design calls for HDL specifically, we qualify it further. The published documentation for HDL is noticeably thinner than for HMS, so we confirm the current specification, drive options and support terms with INCOM before it appears in a proposal rather than quoting from a datasheet. That check happens before pricing, not after.

Deciding how much of this tier you need

The disk tier should be sized against behaviour, not against total archive size. Three inputs settle it in most engagements: peak ingest rate over a sustained window, the proportion of reads that land on data written in the last 90 days, and the largest single restore anyone will realistically request. From those, the working set falls out, and everything below it belongs on cheaper media managed by the software described under archive management software.

If most reads are recent, the disk tier can be a fraction of the total and the economics work. If reads are uniformly distributed across a decade of data, disk is doing the whole job and you should be honest about that in the budget. See long-term data archiving for how the tiers combine, backup and recovery storage for the repository case, and research, AI and HPC for the read-heavy end of the spectrum. Other technology options are listed on the technologies overview.

Rack-mount archive storage array
Manufacturer image. Capacities confirmed per engagement.

Related resources

Manufacturer documentation relevant to this page. Availability, specifications, and configurations are subject to verification.

Frequently asked questions

No. Object access wins on namespace scale and application integration, and it is the right default for new workloads and very large file counts. But if your archiving application only speaks SMB or NFS, an object store behind a file gateway adds a layer without adding much. Choose object when something will actually call the S3 API, or when directory-tree performance is already a measured problem.

Turn your requirement into a defensible architecture

Share the workload, capacity, retention, access, and resilience requirements. Aban Smart will identify the next discovery inputs and the appropriate engagement path.