A Belgian distributor searches its analytics platform for a former customer's contact record. The current table returns nothing. The deletion ticket is closed. Later, an analyst opens last month's saved snapshot and the record is back on the screen.
The system may be doing exactly what it was designed to do: preserve historical state for reproducibility. The problem is that the deletion workflow only checked one view of the data. Open table formats make time travel convenient, which means teams must decide what historical access means when data should no longer be available.
For European businesses investing in a lakehouse, this is a procurement question as much as a database question. Ask for a demonstrable data lifecycle before accepting a dashboard that displays a green deletion status.
Define what the word deleted means in your product
There are several distinct outcomes: hidden from the normal user interface, absent from the current table, unavailable through historical queries, removed from active storage and handled appropriately in backups. These outcomes can occur at different times under an approved policy.
Give each outcome its own status and evidence. An operations user should not need to understand table metadata, but the system must not compress a multi-stage process into an inaccurate promise. Use language such as current view removed, retained under policy or cleanup verified where it reflects the actual state.
The responsible privacy and legal teams should determine the applicable obligations and exceptions. An engineering implementation follows those decisions; choosing Iceberg or any other storage technology does not decide them automatically.
Understand the history you are deliberately preserving
Iceberg writes create snapshots that can support historical queries or rollback. Its maintenance documentation explains that data files remain while referenced by snapshots used for these purposes. Snapshot expiration can remove no-longer-needed files, while also removing the ability to query those expired snapshots. Apache Iceberg maintenance
That trade-off belongs in the business conversation. Finance may need to reproduce an approved report, while another dataset should have a short operational life. A single global retention value is unlikely to express both requirements well.
Iceberg also supports branches and tags with retention properties. Those references can preserve particular table histories for testing or historical use. Include them in your inventory rather than assuming the main branch describes the complete retention picture. Iceberg branching and tagging
Trace one record across the entire journey
Begin with a synthetic record created specifically for the test. Track it from its source into ingestion, staging, curated tables, reports, extracts and any application cache. Record which systems hold a copy, which merely hold a reference and which generate derived information.
The exercise frequently reveals an overlooked export folder or a spreadsheet used in monthly reporting. Those are not minor details if the product promises that a request covers all relevant destinations. Assign an owner to each copy and identify how deletion or retention decisions propagate.
Derived data needs an explicit decision too. A customer identifier embedded in a materialised report behaves differently from an aggregate that no longer identifies the original person. Do not let an engineer make that classification implicitly while implementing a cleanup job.
Separate approval, execution and verification
A robust workflow first validates the request and its scope. It then records the authorised lifecycle action, executes the relevant changes and checks the agreed destinations. Each stage needs a stable request identity so that a retry resumes the same operation rather than creating a second, conflicting instruction.
If one destination is unavailable, keep the overall request visibly incomplete. A successful current-table query should not conceal a failed export cleanup. Route the exception to someone who can decide whether to retry, investigate or document a legitimate retention condition.
Verification should be independent enough to catch an implementation mistake. Checking only the success response from the deletion job repeats the job's own assumptions. Use targeted queries and storage evidence appropriate to the policy, with access controls on the resulting reports.
Treat maintenance as production work
Snapshot expiration, compaction and orphan cleanup affect the platform's operational behaviour. They need scheduling, monitoring and ownership. Iceberg explicitly warns that aggressive orphan cleanup can remove files still involved in a write and corrupt the table. A retention setting must accommodate the platform's real write duration and failure recovery. Iceberg maintenance guidance
Test a long-running ingestion job while maintenance runs. Test a failed writer that leaves temporary material behind. Test an intentionally retained branch. The aim is to prove both sides of the contract: unneeded data is eventually removed, and active work remains intact.
Storage versioning, retention locks and backup policies can introduce further constraints outside the table format. Make those dependencies visible in the design. A table-level operation cannot override a storage-level policy simply because the product interface calls it delete.
Make evidence useful without creating another data copy
The deletion ledger should record scope, policy version, execution outcomes and verification references. It should not store the complete deleted record merely to prove that the record once existed. Minimise personal information in logs and restrict access to diagnostic details.
For a customer support view, expose a comprehensible status and escalation owner. For technical reviewers, provide the execution trail and unresolved destinations. For the policy owner, show requests approaching their agreed completion target and the reason for each exception.
Our open table format comparison helps with storage choices. The article on data governance engineers can use addresses ownership around those choices.
Commission a deletion rehearsal before expanding the platform
Choose one dataset and one synthetic customer journey. The pilot should create current and historical copies, perform the approved action, test retained references and downstream exports, interrupt one cleanup step and produce a reviewable completion report.
Ask for the maintenance configuration, exception procedure and an explanation of what the evidence does not establish. This makes the handover usable when the original developer is unavailable and the next request arrives under time pressure.
If your lakehouse has grown faster than its lifecycle controls, review our custom software development service and scope a data-retention engineering pilot. Bring the data sources, reporting tools and retention decisions you already have; the first deliverable is a map of what the platform actually keeps.