File Lifecycle Management: From Creation to Deletion

File lifecycle management defines what happens to every file at each stage: created (tagged, classified, encrypted), actively used (in hot storage, versioned, shared under access control), inactive (moved to IA or cool tier at 30-90 days), archived (Glacier Deep Archive or Azure Archive), and destroyed (cryptographic erasure plus audit record). A well-designed lifecycle cuts storage cost 60-80% while satisfying GDPR Article 5(1)(e) storage limitation and HIPAA §164.316 retention — all enforced by policy rather than manual effort. The trick is writing the policy as code and testing on a small sample before applying to petabytes.

Stage 1: Creation With Intent

A file's lifecycle starts at creation. Metadata set at this moment determines everything later: retention class, confidentiality level, owner, project. An upload endpoint that doesn't capture these forces manual tagging that never happens.

A minimum metadata set at ingest:

• owner: AD/SSO user ID or service account
• created-by-app: the system that wrote the file
• content-class: document, log, media, backup, temp
• retention-class: sox-7y, hipaa-6y, temp-30d, indefinite
• confidentiality: public, internal, restricted, secret

Embed these via S3 Object Tags or Azure Blob Index Tags at PutObject time. A Lambda that validates required tags and rejects uploads missing them keeps the catalog clean from day one.

Stage 2: Active Use in Hot Tier

Files in active use live in the fastest, most expensive tier: S3 Standard ($0.023/GB), Azure Hot ($0.0184/GB), GCS Standard ($0.020/GB). Latency matters (sub-100 ms), throughput matters, and access patterns are unpredictable.

During active use:

• Versioning enabled to catch overwrites and accidental deletions
• Access logs flowing to SIEM for audit
• Encryption with KMS customer-managed keys
• Presigned URLs for external sharing (max 7 days)

Typical residence in hot tier: 30-90 days. Beyond that, access frequency usually drops enough that cooler tiers pay off.

Stage 3: Transition to Warm Storage

After 30 days without access, most files transition to Infrequent Access tiers: S3 Standard-IA ($0.0125/GB), Azure Cool ($0.0152/GB), GCS Nearline ($0.010/GB). The retrieval latency stays sub-100 ms but retrieval cost appears — S3 IA charges $0.01/GB retrieved.

Lifecycle rule for this transition:

{
  "ID": "active-to-ia",
  "Status": "Enabled",
  "Filter": {"Prefix": "documents/"},
  "Transitions": [{"Days": 30, "StorageClass": "STANDARD_IA"}]
}

Watch the minimum object size — S3 IA charges for 128 KB per object even if it's 4 KB. Objects under 128 KB cost more in IA than Standard. Filter lifecycle rules to exclude small objects with ObjectSizeGreaterThan: 131072.

Stage 4: Archival for Long Tail

Files that haven't been touched in 90+ days rarely get touched again but may be subject to retention requirements. Move them to cold archive tiers: S3 Glacier Flexible Retrieval ($0.0036/GB), S3 Glacier Deep Archive ($0.00099/GB), Azure Archive ($0.00099/GB), GCS Archive ($0.0012/GB).

Retrieval delays become significant:

• Glacier Instant Retrieval: milliseconds
• Glacier Flexible Retrieval: 3-5 hours standard, 1-5 minutes expedited
• Glacier Deep Archive: 12 hours standard, 48 hours bulk
• Azure Archive: up to 15 hours rehydration

Match the tier to the realistic restore SLA. Don't put quarterly audit files in Deep Archive if auditors give 24-hour notice.

Stage 5: Retention Hold and Compliance

Regulations often require files to be retained beyond business usefulness. SOX: 7 years. HIPAA: 6 years. SEC Rule 17a-4: 3-6 years. GDPR: as long as necessary for the stated purpose, with the right to erasure exceptions.

Implement with tag-driven retention and WORM storage:

• retention-class tag drives lifecycle timing
• S3 Object Lock with Compliance Mode prevents deletion for the retention period — even root can't delete
• Legal hold tag (legal-hold: true) freezes lifecycle transitions indefinitely
• Audit log captures all access and tag changes

For broker-dealers under FINRA 4511 and SEC 17a-4(f), WORM storage is mandatory. Implementing it wrong (Governance Mode instead of Compliance Mode, for example) has cost firms seven-figure fines.

Stage 6: Secure Destruction

Eventual deletion closes the lifecycle. "Secure" means the data is unrecoverable and the action is documented.

Approaches:

• Soft delete + delayed purge: mark deleted, purge after 30 days (allows recovery)
• Cryptographic erasure: delete the encryption key so ciphertext becomes noise
• Physical overwrite: DoD 5220.22-M three-pass write (only relevant for on-prem)

For cloud object storage, cryptographic erasure is the practical method. Each bucket or object class uses a KMS key; deleting the key renders objects unrecoverable within minutes. AWS, Azure, and GCP all support this pattern. Log the deletion with CloudTrail equivalents.

Audit log requirement: capture object key, VersionId, timestamp, actor, reason code, and the retention rule that matched. Keep the audit log longer than the data it describes — if you delete files after 7 years, keep the deletion log for 10.

Handling Cross-Lifecycle Sharing

Files in different lifecycle stages get shared differently. An active project file: Drive link with editor permissions. An archived contract needed by outside counsel: restore to warm tier, generate a presigned URL, delete the warm copy after the engagement.

For one-off sharing of archived files, a transfer tool avoids the complexity of granting external access to the archive system. HexaTransfer sends up to 10 GB with AES-256-GCM end-to-end encryption and a one-time link — retrieve the archived file, send it, done. The archive stays untouched; the counsel gets what they need without an IAM role provisioning.

The Policy-as-Code Pattern

Encode every lifecycle rule in Terraform, CloudFormation, Pulumi, or Bicep. Never hand-click in the console. Benefits:

• Peer review via pull requests
• Version history of policy changes
• Dry-run with terraform plan
• Rollback by reverting a commit

Example Terraform snippet:

resource "aws_s3_bucket_lifecycle_configuration" "docs" {
  bucket = aws_s3_bucket.docs.id
  rule {
    id     = "tiering"
    status = "Enabled"
    transition { days = 30  storage_class = "STANDARD_IA" }
    transition { days = 90  storage_class = "GLACIER_IR" }
    transition { days = 365 storage_class = "DEEP_ARCHIVE" }
    expiration { days = 2555 }
  }
}

Commit, review, apply. Test on a dev bucket first. Roll out to production by expanding filter prefixes gradually.

Measuring Lifecycle Effectiveness

Track monthly:

• Total storage by tier (hot/warm/archive)
• Cost by tier
• Object age distribution
• Retrieval rate from cold tiers (high rate means tiers are wrong)
• Deletion rate and reasons

S3 Storage Lens, Azure Cost Management, and GCP's storage insights all produce these dashboards. A healthy mature lifecycle: 10% hot, 20% warm, 70% archive, with cold retrieval under 1% per year. Anything else means the policy needs tuning.

Lifecycle Design Exercise

When designing a lifecycle for a new workload, answer these five questions:

1. What regulatory retention applies?
2. What's the realistic access pattern after 30, 90, 365 days?
3. What's the acceptable restore time if an old file is needed?
4. What's the deletion trigger — time, event, request?
5. What's the audit evidence requirement?

The answers dictate the tier schedule, the retention locks, the legal hold capability, and the audit pipeline. Get these five answers before writing any Terraform. Most lifecycle-gone-wrong stories start with a team that skipped the question phase and went straight to copying someone else's policy.

Try it at hexatransfer.com — free, no account, 10 GB max.