Improving storage efficiency in Magic Pocket, our immutable blob store

This post explains how Dropbox improved storage efficiency in Magic Pocket, their immutable exabyte-scale blob store, after a fragmentation incident caused by a new service.

• •Magic Pocket is an immutable blob store where deleted data stays on disk until reclaimed by compaction, since volumes cannot be modified once closed
• •A new Live Coder service for on-the-fly erasure coding caused severe volume under-filling (less than 5% capacity in worst cases), sharply increasing storage overhead
• •The existing L1 compaction strategy tops off already-dense volumes with donor data, but was too slow against a large tail of severely under-filled volumes
• •L2 compaction uses dynamic programming to group under-filled volumes and pack their combined live data into a single near-full destination volume