4 ms·

Uploading a single file to S3 seems like it should be a simple job, that is well-documented: https://docs.aws.amazon.com/AmazonS3/latest/user-guide/upload-obje

Uploading a single file to S3 seems like it should be a simple job, that is well-documented: https://docs.aws.amazon.com/AmazonS3/latest/user-guide/upload-objects.html https://docs.aws.amazon.com/AmazonS3/latest/user-guide/uploa... There are parts of AWS that are hard to use, and non-intuitive. But S3 didn't ever seem to be one of them, though perhaps I'm forgetting how hard it was initially. You might consider a "friendly" system for static hosting if raw-S3 is too hard though, such as netlify. There are a lot of services out there which basically wrap and resell AWS services. (I run my own to do git-based DNS hosting, which is a thin layer upon the top of route53 for example.)

sho 7y ago
Ha, I can't upload anything into S3. I'm unable to, from memory, create or modify any of my own buckets. My account is in some weird half-suspended state from a past billing incident, long resolved, where things mostly work - but modifying S3 is not one of them. Support insisted everything was fine even as I uploaded videos of the failures and eventually I just gave up. I leave a few legacy things running there (billing works, of course) but these days just put personal stuff on Digital Ocean, which seems to meet basically all my needs without the complexity and cheaper to boot.
m0xte 7y ago
Uploading to S3 is easy until it isn't. It works pretty nicely for one-offs but when you have to blast a few hundred gigs into it or large files from a bit of software you wrote it's a royal pain in the butt. The phrase "multi-part upload" makes me cry inside.
- scarface74 7y ago
  Why is that hard? The CLI automatically does multi part uploads. It’s also simple with the various AWS SDKs.
  - m0xte 7y ago
    It looks like it works but it's not reliable so your reliability concerns then get externalised into your application which multiplies complexity terribly.
    - nostrebored 7y ago
      What do you mean by it's not reliable?
      - scarface74 7y ago
        I’ve never had the experience myself, but I would assume he means you would have to build in some type of retry logic in your script. Just from a cursory glance, I couldn’t find any samples of how to do a multipart upload with retries in Python with Boto3. This is an example of how to do multipart uploads though. https://medium.com/@niyazi_erd/aws-s3-multipart-upload-with-python-and-boto3-9d2a0ef9b085 https://medium.com/@niyazi_erd/aws-s3-multipart-upload-with-... Fun Trivia note: when you do a multipart upload, the S3 hash of the object is not the same as it is when you do a single part upload. I had a file with the same contents but a different hash when I used Python than when it was transferred with the CLI or CloudBerry. The quick and dirty way to fix the hash is to copy the file to itself with Boto3.
drchewbacca 7y ago
Yeah I managed to get the file into the bucket. However I couldn't get the permissions to let it be publicly visible despite working through several tutorials. I ended up on Netlify, it's 1000x more my speed.
- cloakandswagger 7y ago
  I've never used S3 to host a website before so I decided to try this and see if it's really that complicated. First, I searched for "host static website on S3" and found an AWS docs page with a walkthrough. Then, I created an S3 bucket through the console. I uploaded a "hello world" index.html file to it. Then I went to Properties, checked "Use this bucket to host a website" and gave it the name of my index file. When I clicked Save, I had a hosted HTML file that I could navigate to. I'm struggling to see how it could be simpler than that. What exact problems did you run into?
  - drchewbacca 7y ago
    So this was a while ago and I fully admit it might have been me being stupid. However when I was doing it I don't think there was a "Use this bucket to host a website" button. Instead you had to give the bucket a special type of permissions which meant anyone could see the contents but no one could write to it. I kept following the guide as closely as I could but I couldn't get it to take.
    - cloakandswagger 7y ago
      Maybe it's been updated since then. This is the guide I followed: https://docs.aws.amazon.com/AmazonS3/latest/dev/HostingWebsiteOnS3Setup.html https://docs.aws.amazon.com/AmazonS3/latest/dev/HostingWebsi...
      - drchewbacca 7y ago
        Ok yeah I think it was changing the policy that access I couldn't get to work. Again it may be that is super easy and I fucked up something obvious however I tried a lot of times a lot of different ways until I gave up in rage.

sho 7y ago

Ha, I can't upload anything into S3. I'm unable to, from memory, create or modify any of my own buckets. My account is in some weird half-suspended state from a past billing incident, long resolved, where things mostly work - but modifying S3 is not one of them. Support insisted everything was fine even as I uploaded videos of the failures and eventually I just gave up. I leave a few legacy things running there (billing works, of course) but these days just put personal stuff on Digital Ocean, which seems to meet basically all my needs without the complexity and cheaper to boot.

m0xte 7y ago

Uploading to S3 is easy until it isn't. It works pretty nicely for one-offs but when you have to blast a few hundred gigs into it or large files from a bit of software you wrote it's a royal pain in the butt. The phrase "multi-part upload" makes me cry inside.

scarface74 7y ago
Why is that hard? The CLI automatically does multi part uploads. It’s also simple with the various AWS SDKs.
- m0xte 7y ago
  It looks like it works but it's not reliable so your reliability concerns then get externalised into your application which multiplies complexity terribly.
  - nostrebored 7y ago
    What do you mean by it's not reliable?
    - scarface74 7y ago
      I’ve never had the experience myself, but I would assume he means you would have to build in some type of retry logic in your script. Just from a cursory glance, I couldn’t find any samples of how to do a multipart upload with retries in Python with Boto3. This is an example of how to do multipart uploads though. https://medium.com/@niyazi_erd/aws-s3-multipart-upload-with-python-and-boto3-9d2a0ef9b085 https://medium.com/@niyazi_erd/aws-s3-multipart-upload-with-... Fun Trivia note: when you do a multipart upload, the S3 hash of the object is not the same as it is when you do a single part upload. I had a file with the same contents but a different hash when I used Python than when it was transferred with the CLI or CloudBerry. The quick and dirty way to fix the hash is to copy the file to itself with Boto3.

drchewbacca 7y ago

Yeah I managed to get the file into the bucket. However I couldn't get the permissions to let it be publicly visible despite working through several tutorials. I ended up on Netlify, it's 1000x more my speed.

cloakandswagger 7y ago
I've never used S3 to host a website before so I decided to try this and see if it's really that complicated. First, I searched for "host static website on S3" and found an AWS docs page with a walkthrough. Then, I created an S3 bucket through the console. I uploaded a "hello world" index.html file to it. Then I went to Properties, checked "Use this bucket to host a website" and gave it the name of my index file. When I clicked Save, I had a hosted HTML file that I could navigate to. I'm struggling to see how it could be simpler than that. What exact problems did you run into?
- drchewbacca 7y ago
  So this was a while ago and I fully admit it might have been me being stupid. However when I was doing it I don't think there was a "Use this bucket to host a website" button. Instead you had to give the bucket a special type of permissions which meant anyone could see the contents but no one could write to it. I kept following the guide as closely as I could but I couldn't get it to take.
  - cloakandswagger 7y ago
    Maybe it's been updated since then. This is the guide I followed: https://docs.aws.amazon.com/AmazonS3/latest/dev/HostingWebsiteOnS3Setup.html https://docs.aws.amazon.com/AmazonS3/latest/dev/HostingWebsi...
    - drchewbacca 7y ago
      Ok yeah I think it was changing the policy that access I couldn't get to work. Again it may be that is super easy and I fucked up something obvious however I tried a lot of times a lot of different ways until I gave up in rage.