Skip to content

Object Storage — Notes#

Functional#

  • PUT / GET / DELETE / LIST by key.
  • Multipart upload + resumable.
  • Versioning, lifecycle transitions, object lock.
  • Server-side / client-side encryption.
  • Cross-region replication.
  • Event notifications.

Non-functional#

  • 11×9 durability.
  • 99.99% availability per region.
  • Strong read-after-write consistency.
  • Effectively unlimited capacity.

Capacity#

  • S3 reportedly hosts trillions of objects, exabytes total.
  • Per-bucket: no hard limit; sharding by key prefix handles hot ranges.

API#

PUT /{bucket}/{key}            (Object body)
GET /{bucket}/{key}            (Range supported)
DELETE /{bucket}/{key}
POST /{bucket}/{key}?uploads    (multipart init)

Data model#

  • bucket(name, region, policy, lifecycle)
  • object(bucket, key, version_id, etag, size, mtime, storage_class, encryption)
  • Storage: chunked + EC across nodes.

Trade-offs#

  • Flat keyspace scales but listing by prefix can become a hot spot — randomize prefixes for high-throughput buckets.
  • Erasure coding vs replication: EC saves storage (~1.4× overhead) but rebuilds cost CPU/network on failure.
  • Strong consistency needed adapting metadata layer; older S3 used to be eventually consistent.

Refs#

  • AWS S3 white papers, "Building S3" re:Invent talks.
  • GCS architecture papers; Ceph RADOS paper.
  • Reed-Solomon coding; FAST conference papers on object storage.