Automated File Archival: Set It and Forget It

Automated file archival moves inactive files from hot storage to cold tiers on a schedule, without anyone clicking anything. S3 Lifecycle rules, Azure Blob Lifecycle Management, and GCS Object Lifecycle let you write policies like "after 90 days untouched, move to Glacier Deep Archive; after 7 years, delete unless tagged legal-hold." Set them once, test them on a 1% sample, and the system reclaims 50-80% of storage cost while satisfying retention rules — SOX 7-year, HIPAA 6-year, GDPR's data minimization — without manual sweeps.

Choosing the Trigger: Age, Access, or Tag

Three archival triggers dominate. Age-based rules (object.age > 90d) are simplest and work for time-decaying data like logs, transactions, or email exports. Access-based rules (S3 Intelligent-Tiering, which tracks last-access timestamps) are smarter for datasets with unpredictable access patterns — a financial audit file might be dormant for 11 months and critical in month 12.

Tag-based rules (tag retention-class = sox-7y) give humans and applications explicit control. A compliance team can mark files retention-class: hipaa-6y at ingest, and lifecycle rules archive accordingly without changing the rule when categories evolve. Combine them: age for the default, tags for exceptions.

Writing the Policy in YAML or JSON

S3 lifecycle configurations use XML or JSON; Azure uses JSON; GCS uses JSON. A representative S3 rule for a document repository:

{
  "Rules": [
    {
      "ID": "archive-documents",
      "Status": "Enabled",
      "Filter": {"Prefix": "documents/"},
      "Transitions": [
        {"Days": 30, "StorageClass": "STANDARD_IA"},
        {"Days": 90, "StorageClass": "GLACIER_IR"},
        {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
      ],
      "Expiration": {"Days": 2555},
      "NoncurrentVersionExpiration": {"NoncurrentDays": 180}
    }
  ]
}

Version control the policy in Git. Every change reviewed in a pull request, applied with Terraform or CloudFormation, audited in the CI log. Hand-edited console policies drift and break.

Testing Before Production

A lifecycle policy that deletes the wrong bucket is career-limiting. Test on a small prefix first. Create documents/test/ with sample files and a dedicated test policy that applies only to that prefix. Use short durations — Days: 1 — to exercise transitions quickly.

Check the results with aws s3api list-objects-v2 --prefix documents/test/ and verify storage classes changed as expected. Then widen the filter. For high-stakes buckets (legal, finance), run the policy in dry-run mode if the provider supports it, or simulate with a script that reports what would happen without applying changes.

Minimum Storage Durations Are Non-Negotiable

Lifecycle rules respect minimum storage duration charges. Moving an object into Glacier Flexible Retrieval and deleting it 7 days later still charges for 90 days. Deep Archive charges for 180 days minimum. A poorly designed rule that transitions objects from Standard to Deep Archive to deletion within 30 days pays triple — once for the transition, once for the minimum duration, once for the egress if you restore.

The safe pattern: only transition to a colder tier if you expect the object to live there at least the minimum duration. If files are typically deleted after 60 days, don't push them through Deep Archive (180-day minimum). Stop at Standard-IA.

Handling Legal Holds and Exceptions

Every archival system needs an escape hatch for litigation. A tag like legal-hold: true should exempt objects from lifecycle deletion. Implement with a rule filter that excludes tagged objects:

Filter:
  And:
    Prefix: "documents/"
    Tag: {Key: "legal-hold", Value: "false"}

Combined with S3 Object Lock in Governance or Compliance mode, legal-hold tagged objects become immutable until released. Compliance mode prevents even root deletion, which is what FINRA 4511, SEC 17a-4, and HIPAA §164.316 WORM requirements expect.

Automating the Tagging Pipeline

Archival is only as good as the tags driving it. Automate tagging at ingest: a Lambda triggered by S3 ObjectCreated examines the file and applies content-type, department, retention-class tags. For existing unsprayed data, run a batch tagging job with S3 Batch Operations.

Sources for tag values:

File extension and magic bytes (a .pdf with "HIPAA" in metadata gets healthcare tag)
Upload source (an API key tied to the finance app auto-tags finance)
Content scanning (Amazon Macie detects PII and tags accordingly)
User input at upload (a form with a Department dropdown)

Tags cost $0.01 per 10,000 tag requests in S3; worth the investment for precise lifecycle targeting.

Archival Audit Trail

Log every lifecycle action. CloudTrail captures LifecycleTransition and LifecycleExpiration events. Pipe them to Athena, Splunk, or Datadog for queryable history. When an auditor asks "when was this 2019 contract moved to Glacier," the query should return in seconds.

Include in the audit log:

Object key and VersionId
Transition from/to storage class
Trigger (age, tag, explicit)
Timestamp
Policy ID that matched

Retain the audit log for at least as long as the data retention requires — if you archive for 7 years, keep the lifecycle audit for 7 years plus buffer.

Cross-Region and Cross-Cloud Archival

For disaster recovery, archives should live in a second region or cloud. S3 Cross-Region Replication can target a different storage class in the destination — replicate to us-west-2 directly into Glacier Deep Archive. Azure supports similar with RA-GRS and object replication. GCS has multi-region and turbo replication.

For cross-cloud archival (AWS primary, Backblaze B2 secondary), rclone with cron runs scheduled sync jobs. The secondary copy provides independence from vendor-specific outages, ransomware that targets IAM of one provider, or billing disputes.

Retrieval-Ready Archives

An archive you can't restore is just a slow delete. Document the restore path for each tier:

Glacier Instant Retrieval: direct GET, no restore needed
Glacier Flexible Retrieval: RestoreObject with Tier=Standard (3-5 hours) or Expedited (1-5 minutes)
Glacier Deep Archive: RestoreObject with Standard (12 hours) or Bulk (48 hours)
Azure Archive: Set Blob Tier to Hot/Cool, 15 hours rehydration

Practice quarterly. Pick a random archived object, restore it, verify against the manifest hash, and time the whole process. Untested restores become discovered-the-hard-way during incidents.

Handling One-Off Archival Transfers

Automated lifecycle handles steady-state, but sometimes you need to move a specific 50 GB batch immediately — a departing employee's files, a client engagement closeout. For these, a manual transfer tool with integrity checks beats editing lifecycle policies. HexaTransfer moves files up to 10 GB with end-to-end AES-256-GCM encryption and a one-time link, useful for handing archives to outside counsel or a successor without provisioning them access to your whole archive bucket.

Set up the lifecycle policy, automate the tagging, pipe the audit logs, and the system hums. Review the cost dashboard quarterly; adjust thresholds when the access pattern shifts. That's all archival needs — once configured, it takes less attention than coffee orders.

Try it at hexatransfer.com — free, no account, 10 GB max.