3 ms·

TIL that Azure blob storage can be told where to download blobs from, that's neat. > Pre-GTR, the backup procedure would have taken at least 3 hours even with

TIL that Azure blob storage can be told where to download blobs from, that's neat. > Pre-GTR, the backup procedure would have taken at least 3 hours even with a VPS Setup facilitating the transfer from Google Takeout as even large instances on the cloud with large disks, much memory, and many CPUs would eventually choke with too many files being downloaded in parallel. ("VPS" above links to https://sjwheel.net/cloud/computing/2019/08/01/aws_backup.html https://sjwheel.net/cloud/computing/2019/08/01/aws_backup.ht... , which describes spinning up an EC2 machine with enough local storage to download the whole takeout, and using that to download from google and then upload to s3) I think one could do this "transloading" comfortably inside the AWS Lambda limits. The golang AWS sdk's s3.s3manager.UploadInput takes an io.Reader and puts its contents into an S3 bucket+key. You could make a lambda function that downloads a Takeout part and chunks it into s3. Lambda scales to zero and will be patiently waiting for you two months from now when you need it again.

crazysim 4y ago
Azure uses this functionality for their "Transfer me data from GCS/S3" functionality in their CLI. Unfortunately, product-wise, they seemed to have missed the memo about making an S3 compatible interface like GCS. Oh well. At least we got GTR!
crazysim 4y ago
That AWS centric idea with Lambda seems pretty reasonable cost-wise too. One issue with Cloudflare Workers is that it is limited in only being able to PUT 100MB at a time to any host, Cloudflare or not, in an undocumented limit presumably for anti-abuse. I don't believe a similar limitation is present in AWS Lambda. GTR was organically grown from seeing Azure's Blob Storage download API. I got disappointed at the bugs seen and then just patched over it with Cloudflare Workers, but didn't change up to just AWS/S3. I could see myself also going this route too in a clean slate. With the 100MB limit gone, and having to pay for CPU anyway, maybe we could also get proper GPG public key encryption in it too.
- kentonv 4y ago
  > One issue with Cloudflare Workers is that it is limited in only being able to PUT 100MB at a time to any host, Cloudflare or not, in an undocumented limit presumably for anti-abuse. It's documented here: https://developers.cloudflare.com/workers/platform/limits/#request-limits https://developers.cloudflare.com/workers/platform/limits/#r... This limit actually isn't specific to Workers. It's a general Cloudflare thing, because Cloudflare buffers uploads to disk in order to defend against slow loris attacks. That is, this is actually meant to protect your origin. We've occasionally discussed letting people turn this off if they don't need it but it hasn't come up very often... (I'm the lead engineer for Workers.)
  - crazysim 4y ago
    Ah! I guess I missed that. It confused the CF Discord a bit but I guess it is documented.
  - crazysim 4y ago
    Actually, the surprise was that it applied to non-Cloudflare hosts such as Azure Storage's endpoint. Request limits against Cloudflare itself? Famous. Against another non-Cloudflare host from a Cloudflare Worker's environment? Much less famous or well known.
    - kentonv 4y ago
      Yeah, that's more or less a bug. The reason for it is that outgoing requests from Workers pass through a lot of the same proxy logic as incoming requests. This made Workers a lot easier to build initially (six months from inception to beta!) but there are a lot of quirks as a result. Outgoing requests from Workers destined for external servers (i.e. not other sites on Cloudflare) really have no reason to be buffered and thus no reason to impose these limits. We should fix that. I also suspect most Workers-based apps don't really need incoming buffering either.

crazysim 4y ago

Azure uses this functionality for their "Transfer me data from GCS/S3" functionality in their CLI. Unfortunately, product-wise, they seemed to have missed the memo about making an S3 compatible interface like GCS. Oh well. At least we got GTR!

crazysim 4y ago

That AWS centric idea with Lambda seems pretty reasonable cost-wise too. One issue with Cloudflare Workers is that it is limited in only being able to PUT 100MB at a time to any host, Cloudflare or not, in an undocumented limit presumably for anti-abuse. I don't believe a similar limitation is present in AWS Lambda. GTR was organically grown from seeing Azure's Blob Storage download API. I got disappointed at the bugs seen and then just patched over it with Cloudflare Workers, but didn't change up to just AWS/S3. I could see myself also going this route too in a clean slate. With the 100MB limit gone, and having to pay for CPU anyway, maybe we could also get proper GPG public key encryption in it too.

kentonv 4y ago
> One issue with Cloudflare Workers is that it is limited in only being able to PUT 100MB at a time to any host, Cloudflare or not, in an undocumented limit presumably for anti-abuse. It's documented here: https://developers.cloudflare.com/workers/platform/limits/#request-limits https://developers.cloudflare.com/workers/platform/limits/#r... This limit actually isn't specific to Workers. It's a general Cloudflare thing, because Cloudflare buffers uploads to disk in order to defend against slow loris attacks. That is, this is actually meant to protect your origin. We've occasionally discussed letting people turn this off if they don't need it but it hasn't come up very often... (I'm the lead engineer for Workers.)
- crazysim 4y ago
  Ah! I guess I missed that. It confused the CF Discord a bit but I guess it is documented.
- crazysim 4y ago
  Actually, the surprise was that it applied to non-Cloudflare hosts such as Azure Storage's endpoint. Request limits against Cloudflare itself? Famous. Against another non-Cloudflare host from a Cloudflare Worker's environment? Much less famous or well known.
  - kentonv 4y ago
    Yeah, that's more or less a bug. The reason for it is that outgoing requests from Workers pass through a lot of the same proxy logic as incoming requests. This made Workers a lot easier to build initially (six months from inception to beta!) but there are a lot of quirks as a result. Outgoing requests from Workers destined for external servers (i.e. not other sites on Cloudflare) really have no reason to be buffered and thus no reason to impose these limits. We should fix that. I also suspect most Workers-based apps don't really need incoming buffering either.