13 ms·

Not be that guy, but we never send an object to S3 without compressing it on our end first.

flatiron 4y ago
We never send anything unencrypted to S3. I have no clue why someone would not want to encrypt their data before putting it in the cloud…
- cmeacham98 4y ago
  For people also using other AWS services (in particular compute ones like EC2) this doesn't do anything meaningful.
  - flatiron 4y ago
    We use other aws services. We decrypt the files on those services.
- thecleaner 4y ago
  Because there is already server side encryption. Which I guess means data on disk is always encrypted not sure about in-memory.
  - flatiron 4y ago
    We also use server side encryption. But honestly encrypting the files before putting on s3 has given us 0 grief and gives us a feeling even if we somehow are “hacked” the files are useless to the hackers.
    - yunohn 4y ago
      Where is your decryption key stored?
      - flatiron 4y ago
        We have a one and done rest api that will only send the key once after being staged as part of our pipeline. So that’s the delivery mechanism. Key is on an internal server that isn’t on the public internet except the one and done. Super simple.
      - moltar 4y ago
        Post-it as a Service
- barkingcat 4y ago
  Would you encrypt files that are supposed to be publicly readable and accessible? I get hashing it and providing hashes, but encrypting public files seems excessive.
- 101011 4y ago
  There's a very common data/ETL pattern wherein raw (unencrypted) data is stored into S3 at the very beginning of any pipeline. Adding encryption adds a layer for failure, which can grind your pipeline to a halt. I've seen a pattern of: drop raw data into an S3 bucket that has a very restrictive policy with a long retention policy. Then, process that data asynchronously (encrypt, transform, filter, etc) and drop it into a different bucket/area that is accessed by other consumers. Then, if any part of your ETL fails (encryption included), you can fix your bug and reprocess from your raw data without writers seeing any impact.

We never send anything unencrypted to S3. I have no clue why someone would not want to encrypt their data before putting it in the cloud…

cmeacham98 4y ago
For people also using other AWS services (in particular compute ones like EC2) this doesn't do anything meaningful.
- flatiron 4y ago
  We use other aws services. We decrypt the files on those services.
thecleaner 4y ago
Because there is already server side encryption. Which I guess means data on disk is always encrypted not sure about in-memory.
- flatiron 4y ago
  We also use server side encryption. But honestly encrypting the files before putting on s3 has given us 0 grief and gives us a feeling even if we somehow are “hacked” the files are useless to the hackers.
  - yunohn 4y ago
    Where is your decryption key stored?
    - flatiron 4y ago
      We have a one and done rest api that will only send the key once after being staged as part of our pipeline. So that’s the delivery mechanism. Key is on an internal server that isn’t on the public internet except the one and done. Super simple.
    - moltar 4y ago
      Post-it as a Service
barkingcat 4y ago
Would you encrypt files that are supposed to be publicly readable and accessible? I get hashing it and providing hashes, but encrypting public files seems excessive.
101011 4y ago
There's a very common data/ETL pattern wherein raw (unencrypted) data is stored into S3 at the very beginning of any pipeline. Adding encryption adds a layer for failure, which can grind your pipeline to a halt. I've seen a pattern of: drop raw data into an S3 bucket that has a very restrictive policy with a long retention policy. Then, process that data asynchronously (encrypt, transform, filter, etc) and drop it into a different bucket/area that is accessed by other consumers. Then, if any part of your ETL fails (encryption included), you can fix your bug and reprocess from your raw data without writers seeing any impact.