3 ms·

Yeah, S3 is frankly a disaster. People use it like a filesystem and if you have billions of objects, it very quickly becomes a nightmare to manage. Size of data

Yeah, S3 is frankly a disaster. People use it like a filesystem and if you have billions of objects, it very quickly becomes a nightmare to manage. Size of data is a complete non-issue, it's all about the number of objects.

grogenaut 6y ago
I will point out deleting that much data from harddrives isn't a fast operation either if you were storing it on drive. Same for tape. Sure you can destructively destroy the drives. But you could also just use KMS on the S3 bucket object and then toss the key and close the buckets. Gonezo.
- dylan604 6y ago
  except you'd still be on the hook for the monthly cost for the useless bucket full of data
- the8472 6y ago
  > I will point out deleting that much data from harddrives isn't a fast operation either Dropping a btrfs subvolume or snapshot is pretty fast. ditto for zfs.
  - danielheath 6y ago
    That doesn't help if you want the data to be gone instead of hidden.
    - Dylan16807 6y ago
      If it's encrypted then destroying the key gives you a blank-with-noise drive. There's no equivalent for S3.
- jiggawatts 6y ago
  It's strange that people just shrug their shoulders and accept this state of things in 2020. Any system these days should allow near-instantaneous admin operations for just about any reasonable request. Subtree move, delete, or rename should all be instant. Similarly, any simple ACL changes should be instant. Similarly, data movement should scale linearly with the volume of data, not the number of objects.
nstart 6y ago
I'm definitely one of those who treat it like a filesystem :') . I'd love to hear a different point of view. Would you feel ok to share thoughts on how I could be thinking of s3 to fully appreciate the right way of using it?
- manigandham 6y ago
  That entirely depends on what you're using it for. Mimicking a typical block-level filesystem with object storage will lead to problems beyond a few thousand objects or shallow "folder" levels. Object storage is best for larger files with bulk updates. Either write new files entirely (best for S3) or append/replace where supported (like Azure/GCP). Any further structure and metadata should be maintained at a higher level with pointers to the raw files in the object store. There are products like Stablebit Cloud Drive [1], ObjectiveFS [2], and other storage companies that handle all this for you. You can also look UtahFS [3] to see open-source code on building something yourself. 1) https://stablebit.com/CloudDrive https://stablebit.com/CloudDrive 2) https://objectivefs.com/ https://objectivefs.com/ 3) https://blog.cloudflare.com/utahfs/ https://blog.cloudflare.com/utahfs/

grogenaut 6y ago

I will point out deleting that much data from harddrives isn't a fast operation either if you were storing it on drive. Same for tape. Sure you can destructively destroy the drives. But you could also just use KMS on the S3 bucket object and then toss the key and close the buckets. Gonezo.

dylan604 6y ago
except you'd still be on the hook for the monthly cost for the useless bucket full of data
the8472 6y ago
> I will point out deleting that much data from harddrives isn't a fast operation either Dropping a btrfs subvolume or snapshot is pretty fast. ditto for zfs.
- danielheath 6y ago
  That doesn't help if you want the data to be gone instead of hidden.
  - Dylan16807 6y ago
    If it's encrypted then destroying the key gives you a blank-with-noise drive. There's no equivalent for S3.
jiggawatts 6y ago
It's strange that people just shrug their shoulders and accept this state of things in 2020. Any system these days should allow near-instantaneous admin operations for just about any reasonable request. Subtree move, delete, or rename should all be instant. Similarly, any simple ACL changes should be instant. Similarly, data movement should scale linearly with the volume of data, not the number of objects.

nstart 6y ago

I'm definitely one of those who treat it like a filesystem :') . I'd love to hear a different point of view. Would you feel ok to share thoughts on how I could be thinking of s3 to fully appreciate the right way of using it?

manigandham 6y ago
That entirely depends on what you're using it for. Mimicking a typical block-level filesystem with object storage will lead to problems beyond a few thousand objects or shallow "folder" levels. Object storage is best for larger files with bulk updates. Either write new files entirely (best for S3) or append/replace where supported (like Azure/GCP). Any further structure and metadata should be maintained at a higher level with pointers to the raw files in the object store. There are products like Stablebit Cloud Drive [1], ObjectiveFS [2], and other storage companies that handle all this for you. You can also look UtahFS [3] to see open-source code on building something yourself. 1) https://stablebit.com/CloudDrive https://stablebit.com/CloudDrive 2) https://objectivefs.com/ https://objectivefs.com/ 3) https://blog.cloudflare.com/utahfs/ https://blog.cloudflare.com/utahfs/