HuskHoard architect argues metadata can cut compliance storage costs

Financial firms facing six- to seven-year record-retention requirements can cut compliance archive costs by rethinking storage architecture, according to Joichiro Mitaka, a system architect at HuskHoard. In a Finextra community post, Mitaka estimated that a 500 TB compliance archive kept in premium cloud storage can cost about $130,000 to $140,000 a year, while a metadata-first cold-storage model could reduce infrastructure costs by 70% to 90%.

Mitaka said the pressure comes from a mismatch between regulatory retention periods and cloud pricing. Regulators require firms to hold transaction and operational records for long periods so events can be reconstructed years later, while cloud providers charge more for data kept in tiers designed for frequent access.

The result, he argued, is a choice many firms find unsatisfactory: keep rarely read compliance records in higher-cost “hot” storage, or move them into cheaper deep archive tiers and accept slower retrieval when regulators, lawyers or internal investigators ask for specific records.

Index and archive as separate layers

Mitaka’s proposed model separates the searchable record of an archive from the underlying payload. The first layer is a compact catalog containing metadata about each record and its exact location, including byte-level references. Because this index is small relative to the full archive, he said it can remain on faster storage and respond to queries in milliseconds.

The second layer holds the bulk data, such as transaction logs, API traces and raw records, in lower-cost storage, including deep cloud archive tiers or tape. Mitaka said the price difference between the highest- and lowest-cost cloud storage tiers is roughly 20 times on a per-gigabyte basis.

Under that structure, searches run against the catalog rather than the cold archive. The slower storage tier is contacted only after the index has identified the specific object or byte range to retrieve.

Cold files with application-level access

Mitaka also described a file-stubbing approach, where data moved to cold storage leaves behind a sparse placeholder on the original system. Applications see a file that appears to be in place, while the full payload sits elsewhere.

When an application opens the stub, a kernel-level component can intercept the access, pause the process, restore the data and resume the read operation, according to Mitaka. He said this can reduce manual restore workflows, where users submit a request, wait for processing and then work with the retrieved data.

He also argued that catalog data should be available in open formats such as Apache Parquet. That would allow compliance teams to query archive metadata using tools including DuckDB, Athena or Spark, rather than relying only on proprietary archive software.

In Mitaka’s view, the file-centric model of compliance storage relies too heavily on folder structures and filenames. A catalog-led system instead extracts attributes such as merchant IDs, account numbers, timestamps and risk scores at the point of ingestion, making them searchable across years of records without restoring the underlying files.

Finextra identified the post as external content provided without editing and representing the author’s views. Mitaka linked the approach to open-source data-tiering archives such as HuskHoard, where he works.

This story draws on original reporting from Finextra Research.