A retention store holding ninety billion call records is an indexing problem wearing a storage costume.

A CDR is small, structured and highly repetitive. Fields are mostly fixed-width identifiers, timestamps and integers. Compression works well on them precisely because of that repetition, and columnar or delimited batch formats compress far better than the same records stored individually.

xDR is the broader family. Data records from packet gateways, IMS session records, SMS and messaging records, roaming records exchanged with other operators, and increasingly signalling and network-function event records. They differ in schema and in volume by orders of magnitude, but they share the shape: tiny, numerous, and needed later in ranges rather than one at a time.

Three consequences follow from that shape.

• The read pattern is a range scan, not a random get. Nobody asks for one record by its own identifier. They ask for everything matching a subscriber identifier or a number across a date window.
• The index is a first-class capacity item. It must be budgeted, sized and protected as deliberately as the payload, because it can approach the same order of magnitude.
• Ingest never stops. There is no quiet window. Any design that assumes a maintenance pause for compaction or rebalancing will meet the network's opinion of that assumption.

The following figures are invented for illustration. They are not a measurement from any operator and should be replaced with your own mediation output before anything is sized.

Assume a network emitting 250 million records per day at an average of 250 bytes per record, retained for five years.

• Daily volume: 250,000,000 × 250 bytes = 62,500,000,000 bytes, roughly 62.5 GB per day
• Annual volume: 62.5 GB × 365 = roughly 22.8 TB per year
• Five-year volume: 22.8 TB × 5 = roughly 114 TB

114 TB is unremarkable. Now count the records instead.

• Annual record count: 250,000,000 × 365 = 91.25 billion records per year
• Five-year record count: 91.25 billion × 5 = 456.25 billion records

Allow a deliberately conservative 64 bytes of index entry per record to support subscriber and date-range lookup. 456,250,000,000 × 64 bytes is approximately 29 TB of index sitting alongside 114 TB of payload. The index is a quarter of the payload volume again, and unlike the payload it needs to be fast. Cheap deep storage does not help it.

Continue the illustration. Suppose each record were written as its own object. Object stores and filesystems carry a fixed cost per object for metadata, allocation and naming. Take 1 KB as an illustrative figure for that overhead.

456,250,000,000 objects × 1 KB is roughly 456 TB of overhead carrying 114 TB of payload. The bookkeeping is four times the data. Long before capacity became an issue, the namespace, the metadata database and the listing operations would have failed.

Batching is the fix, and the batch boundary is the important decision. Group one hour of one stream into a single archive object and, on the same figures, each object holds about 10.4 million records at roughly 2.6 GB, producing 8,760 objects per stream per year instead of 91.25 billion. That is a namespace an archive system can manage, migrate and verify.

The cost of batching is retrieval granularity. You now fetch a 2.6 GB object to answer a question about one subscriber, unless the index records byte offsets within the batch. Whether the batch interval should be an hour, fifteen minutes or a day is set by how quickly retrieval has to complete, not by storage preference.

Records arrive from network elements into mediation, which normalises, deduplicates and correlates them before billing consumes them. The retention copy should be taken from a defined point in that chain, and which point matters. Pre-mediation records are raw and complete but inconsistent in schema. Post-mediation records are clean but have already been filtered, and what was discarded is gone.

Write the retention copy immutably once the batch closes. If the copy can be amended after the fact, its evidential value drops sharply, and the reason for keeping it at all was evidential. Archive management software handles the placement, the media abstraction and the verification underneath, so mediation writes to a filesystem or S3 endpoint and does not need to know what the data eventually lands on.

Keep the schema with the data. A five-year-old batch is unreadable if the field definitions live only in a mediation configuration that has since been replaced twice. Store the schema version alongside each batch and treat schema evolution as an archive concern, not only an application one.

Operators receive properly authorised requests for records, and the storage design either supports answering them within the time allowed or it does not. This section describes engineering requirements, not legal ones. What any authority may request, and within what deadline, is a question for your own legal advisers and your regulatory affairs function.

Four engineering properties usually decide whether the retrieval works.

1. Bounded latency. The answer must arrive within the window a request specifies. If the archive tier is offline media with a mount and seek time, that time is part of the answer, and it needs measuring rather than estimating.
2. Selective retrieval. Restoring an entire month to find one subscriber is not a workable procedure. The index must resolve a query to specific batches, ideally to offsets inside them.
3. Integrity you can attest to. Whoever produces the extract should be able to show that the batch it came from is bit-identical to what was written. That means checksums recorded at ingest and verified at retrieval.
4. An audit trail of the retrieval itself. Who ran the query, under what authorisation reference, what was extracted and when. This record is often as sensitive as the data.

Access to the retrieval function should be separated from ordinary storage administration. The same separation-of-duty logic applies in government and banking and financial services environments, where a privileged administrator should not be able to search records unobserved.

At the end of the retention window, records should go. Over-retention creates discoverable data with no business reason to exist, and in a store of 456 billion records it also creates cost that compounds quietly.

Expiry against batched data is straightforward if the batch boundary aligns with the retention clock, and painful if it does not. An hourly batch expires as a unit. A batch containing records from mixed dates cannot expire cleanly without being rewritten, which defeats immutability. Align the batch to the clock at design time.

Two operational points are worth stating plainly. Immutable storage that enforces a retention period is designed to prevent early deletion, and that includes deletion you later decide you wanted. Second, the index must expire in step with the payload, or your search results will start returning references to objects that no longer exist.

We size, design and integrate the retention store and its index tier. We do not decide your retention periods, and no storage product makes an operator compliant with anything. Periods and their interpretation are a legal question for your advisers. See long-term data archiving for the underlying lifecycle model, browse the other industries we work in, or request an assessment with your record counts to hand.