3 ms·

I didn't read the documentation yet. Is it possible to make it work with Wasabi instead of Azure?

mdaniel 4y ago
I don't know Wasabi well enough to comment, but GTR uses Azure because of its unique(?) ability to do cloud-initiated fetches, versus "download to my computer, upload to the S3 endpoint" that every other cloud storage API uses. All of the words you haven't read yet were about trickery to exfiltrate the signed Takeout URLs so that Azure's blob storage could fetch them (along with some Cloudflare Worker trickery to work around some kind of bugs in Azure) So, I guess the TL;DR (heh, quite literally in this case) is that if Wasabi allows you to download something like an Ubuntu ISO to Wasabi storage by giving it the ISO URL from your phone, it'll likely work for you. Otherwise, you're back in the VPS setup TFA was seeking to avoid
- rodolphoarruda 4y ago
  Thanks for the "extended" TL;DR =) Wasabi allows cloud-to-cloud migrations[1]. Disclaimer: I'm an official partner in Brazil. [1] https://wasabi.com/cloud-migration-tools/ https://wasabi.com/cloud-migration-tools/
  - mdaniel 4y ago
    That was an objectively terrible blog post. But, because I was curious, I actually went digging into the docs for it and it seems that "WCSM" only supports fetching from S3 Compatible origins, which those pre-signed Takeout URLs are not: https://docs.wasabi.com/docs/wcsm-for-data-migration https://docs.wasabi.com/docs/wcsm-for-data-migration Hypothetically one could trick the Cloudflare workers into exposing those Takeout URLs under fake GetObject request-response handshake, but ... what a lot of work just for Wasabi
    - crazysim 4y ago
      I don't see any documentation about an API here at all. It's just yeah we support these sources for our transfer tool and contact our team for them. From the outside it's a lot closer to the storage transfer job stuff I've seen on GCP and yet also kind of far as it's gated behind those infernal "talk to our team" for access gates.
crazysim 4y ago
At the moment it's just Azure. Azure's API is special as mentioned by other comments. That said, I've tried PUT-ing directly the Azure but quickly discovered that Azure actually has an ingress(!) quota on bytes, not requests like S3 or S3-likes I've seen before. I have not implemented a S3 target yet. I might try. I'm particularly interested in targeting Cloudflare R2 as a temporary staging area so I can download the takeout files via a resumable or bulk download manager for a local backup without heavy bandwidth costs. AWS S3 also has their own competitive Archive tier too. Maybe keep an eye on https://github.com/nelsonjchen/gargantuan-takeout-rocket/issues/7 https://github.com/nelsonjchen/gargantuan-takeout-rocket/iss...
- rodolphoarruda 4y ago
  > Azure's API is special Yes, so is Wasabi. It doesn't charge you for ingress/egress data, so there is a clear cost advantage over similar services. Disclaimer: I'm a Wasabi official partner in Brazil.
  - crazysim 4y ago
    In another branch of replies, it does not appear there is an API to do transfers. Just a transfer tool. https://news.ycombinator.com/item?id=34904775 https://news.ycombinator.com/item?id=34904775 . Nothing about any deviations from the S3 API for transloading or an API offered. As for that special-ness you've mentioned, Wasabi is actually falling behind on that "charge you for ingress/egress" point since has a serious asterisk. One of the reasons I'm far more interested in Cloudflare R2 for a possible future temporary staging platform for local downloads is because it's genuine not "charge you for ingress/egress" as Wasabi says "don't egress more than you have stored" or we'll ban you. R2 does not have that limitation there. In fact, I've used it to serve a 28GB file that have been downloaded many, many times with only 28GB stored in R2. Such a thing is against Wasabi's use and the threat makes it unsuitable for GTR. As for cost, like a final destination, there is no option to do Deep Glacier or Archive tier like on AWS or Azure. It is significantly more expensive.

I don't know Wasabi well enough to comment, but GTR uses Azure because of its unique(?) ability to do cloud-initiated fetches, versus "download to my computer, upload to the S3 endpoint" that every other cloud storage API uses. All of the words you haven't read yet were about trickery to exfiltrate the signed Takeout URLs so that Azure's blob storage could fetch them (along with some Cloudflare Worker trickery to work around some kind of bugs in Azure) So, I guess the TL;DR (heh, quite literally in this case) is that if Wasabi allows you to download something like an Ubuntu ISO to Wasabi storage by giving it the ISO URL from your phone, it'll likely work for you. Otherwise, you're back in the VPS setup TFA was seeking to avoid

rodolphoarruda 4y ago
Thanks for the "extended" TL;DR =) Wasabi allows cloud-to-cloud migrations[1]. Disclaimer: I'm an official partner in Brazil. [1] https://wasabi.com/cloud-migration-tools/ https://wasabi.com/cloud-migration-tools/
- mdaniel 4y ago
  That was an objectively terrible blog post. But, because I was curious, I actually went digging into the docs for it and it seems that "WCSM" only supports fetching from S3 Compatible origins, which those pre-signed Takeout URLs are not: https://docs.wasabi.com/docs/wcsm-for-data-migration https://docs.wasabi.com/docs/wcsm-for-data-migration Hypothetically one could trick the Cloudflare workers into exposing those Takeout URLs under fake GetObject request-response handshake, but ... what a lot of work just for Wasabi
  - crazysim 4y ago
    I don't see any documentation about an API here at all. It's just yeah we support these sources for our transfer tool and contact our team for them. From the outside it's a lot closer to the storage transfer job stuff I've seen on GCP and yet also kind of far as it's gated behind those infernal "talk to our team" for access gates.

crazysim 4y ago

At the moment it's just Azure. Azure's API is special as mentioned by other comments. That said, I've tried PUT-ing directly the Azure but quickly discovered that Azure actually has an ingress(!) quota on bytes, not requests like S3 or S3-likes I've seen before. I have not implemented a S3 target yet. I might try. I'm particularly interested in targeting Cloudflare R2 as a temporary staging area so I can download the takeout files via a resumable or bulk download manager for a local backup without heavy bandwidth costs. AWS S3 also has their own competitive Archive tier too. Maybe keep an eye on https://github.com/nelsonjchen/gargantuan-takeout-rocket/issues/7 https://github.com/nelsonjchen/gargantuan-takeout-rocket/iss...

rodolphoarruda 4y ago
> Azure's API is special Yes, so is Wasabi. It doesn't charge you for ingress/egress data, so there is a clear cost advantage over similar services. Disclaimer: I'm a Wasabi official partner in Brazil.
- crazysim 4y ago
  In another branch of replies, it does not appear there is an API to do transfers. Just a transfer tool. https://news.ycombinator.com/item?id=34904775 https://news.ycombinator.com/item?id=34904775 . Nothing about any deviations from the S3 API for transloading or an API offered. As for that special-ness you've mentioned, Wasabi is actually falling behind on that "charge you for ingress/egress" point since has a serious asterisk. One of the reasons I'm far more interested in Cloudflare R2 for a possible future temporary staging platform for local downloads is because it's genuine not "charge you for ingress/egress" as Wasabi says "don't egress more than you have stored" or we'll ban you. R2 does not have that limitation there. In fact, I've used it to serve a 28GB file that have been downloaded many, many times with only 28GB stored in R2. Such a thing is against Wasabi's use and the threat makes it unsuitable for GTR. As for cost, like a final destination, there is no option to do Deep Glacier or Archive tier like on AWS or Azure. It is significantly more expensive.