chaowen.← Notes

Cloud platforms / object-storage protocol deep dive

2026-10-07

S3 multipart uploads: a distributed commit protocol hiding in object storage

Uploading a large object over one HTTP request makes retries expensive and ties success to one long-lived connection. Multipart upload breaks transfer into independently retryable pieces—but it also creates durable intermediate state. The final CompleteMultipartUpload call is a publication boundary, not just a file-join operation.

1. The concrete problem

Imagine a 200 GiB backup traveling over an unstable link. A single PUT fails at 199 GiB; the client may need to start again. Multipart upload divides the payload into numbered parts, uploads parts independently (often concurrently), retries only missing pieces, then asks the service to assemble the object. This improves throughput and recovery, but the service must track an upload session, retain uncommitted parts, validate the caller's manifest, publish one object atomically, and clean up abandoned state.

Model: multipart upload is a small staged-write protocol. Initiate allocates a session; UploadPart writes tentative components; Complete commits a chosen ordered manifest into the visible object key; Abort discards the session.

2. State machine and data flow

The API has three phases. CreateMultipartUpload returns an opaque UploadId bound to a bucket and key. Each UploadPart supplies that ID and a part number (1–10,000). A successful response returns an ETag for that part. Finally, CompleteMultipartUpload supplies the ordered list of part numbers and ETags. The service checks that the manifest identifies stored parts, assembles the object, and makes the completed object available under the key.

Client Object store | CreateMultipartUpload(key) | |--------------------------------->| allocate uploadId |<---------------------------------| | UploadPart(uploadId, n, bytes) | repeat / retry parts |--------------------------------->| persist part, return ETag |<---------------------------------| | Complete(uploadId, [(n, ETag)]) | |--------------------------------->| validate manifest; commit object |<---------------------------------| final object ETag | | Abort(uploadId) ---------------->| discard unfinished upload

This resembles a transaction, but it is not a general multi-object transaction and does not provide rollback after completion. Parts are staged data; the manifest is the client's commit record. The service is not asked to infer intended order from arrival order: parts may arrive out of order, so the completion list defines the assembly sequence.

3. Part identity, retries, and integrity

Part numbers provide logical position; ETags provide a version token for the uploaded part. Re-uploading the same part number replaces that part's prior content in the upload. Therefore a lost response creates ambiguity: the client may not know whether its request succeeded. Retrying the same part number is usually safer than inventing another number, but the client must retain the returned ETag corresponding to the final version of each part.

ETag must not be treated as a universal content hash. For multipart objects it is generally not the MD5 digest of the entire object, and encryption or provider implementation details further weaken assumptions. Use explicit checksum fields supported by the API and verify the service's documented checksum semantics. A robust uploader persists a manifest containing upload ID, key, part number, byte range, checksum, ETag, and completion state.

Concurrency and manifest correctness

Workers can upload separate part numbers concurrently. Avoid two workers writing the same number: last successful replacement wins, and one worker's ETag can become stale. Before completion, list the parts and compare them to the intended manifest. Completion requests can also be retried after timeouts; first inspect whether the object exists and whether the upload session remains active rather than blindly assuming either commit or rollback.

4. Publication and consistency boundary

Until completion, the intended object is not the assembled final object at its key. Consumers should not be pointed at a partially uploaded key. The successful completion response is the application-level signal to publish downstream references, update a database row, or emit an event. If the caller crashes after storage commits but before writing its database record, the object exists but application metadata may lag. Solve that cross-system gap with idempotent completion handling, an outbox/reconciliation process, or content-addressed staging keys—not by assuming a database and object store share one transaction.

Use unique or versioned keys when overwriting a live object would be dangerous. For conditional publication, use the storage service's supported preconditions and versioning semantics. “Multipart” by itself does not mean compare-and-swap on the destination key, and it does not serialize independent upload sessions targeting the same key.

5. Failure modes, cost, and security

Encryption settings are normally associated with the initiated upload; ensure the initiator chooses required server-side encryption and key policy. Treat upload IDs as sensitive capability-like identifiers even when authorization is separately enforced. A leaked ID alone should not replace access control, but it can expose operational state or aid misuse when combined with weakly scoped credentials.

6. Reproducible lab with AWS CLI

Goal: upload two parts, inspect staged state, complete, and verify final bytes. Prerequisites: AWS CLI configured for a disposable S3-compatible bucket with multipart permissions; a test key with no valuable existing object. Commands shown use AWS S3 API and a local scratch directory. Never run against a production key.

BUCKET='your-disposable-bucket'
KEY="lab/multipart-$(date +%s).bin"
head -c 6291456 /dev/urandom > part1.bin
head -c 1048576 /dev/urandom > part2.bin
aws s3api create-multipart-upload --bucket "$BUCKET" --key "$KEY"

Save the returned UploadId as UPLOAD_ID. The first part is 6 MiB, above S3's usual 5 MiB minimum for non-final parts; part 2 can be smaller because it is final.

UPLOAD_ID='paste-the-returned-id'
aws s3api upload-part --bucket "$BUCKET" --key "$KEY" \
  --upload-id "$UPLOAD_ID" --part-number 1 --body part1.bin
aws s3api upload-part --bucket "$BUCKET" --key "$KEY" \
  --upload-id "$UPLOAD_ID" --part-number 2 --body part2.bin
aws s3api list-parts --bucket "$BUCKET" --key "$KEY" --upload-id "$UPLOAD_ID"

Record each response's ETag exactly, including quotes if your shell preserves them. Build parts.json using the API's required structure; do not guess ETags:

{
  "Parts": [
    {"ETag": "ETAG_FROM_PART_1", "PartNumber": 1},
    {"ETag": "ETAG_FROM_PART_2", "PartNumber": 2}
  ]
}
aws s3api complete-multipart-upload --bucket "$BUCKET" --key "$KEY" \
  --upload-id "$UPLOAD_ID" --multipart-upload file://parts.json
aws s3api head-object --bucket "$BUCKET" --key "$KEY"
aws s3 cp "s3://$BUCKET/$KEY" result.bin
cat part1.bin part2.bin | shasum -a 256
shasum -a 256 result.bin

Expected: list-parts reports both numbered parts and their ETags; completion returns an object ETag; HEAD reports the final object's size; the concatenated local parts and downloaded result have identical SHA-256 digests. The ETag may differ from that digest.

Revealing variations

  1. Upload part 1 again with different bytes but the same part number. Observe that the part listing reflects its new ETag; completion with the old ETag should fail or identify a mismatch. Rebuild the manifest from current part state.
  2. Initiate another upload and abort it: aws s3api abort-multipart-upload --bucket "$BUCKET" --key "$KEY" --upload-id "$UPLOAD_ID". Listing or completion should no longer treat it as a live session. In a real environment, lifecycle policy provides cleanup when clients never return.

CLI quoting and checksum behavior can vary across AWS CLI versions and compatible stores. If the lab fails, preserve the exact service error, check IAM permissions (s3:CreateMultipartUpload, s3:UploadPart, s3:ListMultipartUploadParts, s3:CompleteMultipartUpload, s3:AbortMultipartUpload), and consult that provider's API docs. Do not interpret a failed CLI demonstration as proof that the protocol lacks these semantics.

7. Production debugging checklist

  1. Identify bucket, key, upload ID, and request IDs; avoid logging credentials or presigned query strings.
  2. List parts and compare number, size, checksum, and ETag against the persisted manifest.
  3. Determine whether the destination object was committed (HEAD/version history) before retrying completion.
  4. Measure incomplete-upload count, age, bytes, completion errors, and abort rate; set lifecycle expiration and alert on old sessions.
  5. Separate network retry policy from application publication: exponential backoff with jitter for transient failures, but reconciliation for ambiguous commit outcomes.
  6. Test interruption at each boundary: before part request, after server accepts but before response, during completion, and after commit but before metadata update.

Multipart upload is useful precisely because work is resumable. Reliability comes from treating its intermediate state as durable distributed state—not as invisible temporary files.

Primary sources