6 ms·

Took a quick look at the API. For context, I was involved in the early days of Google Cloud Storage. It is surprising that they didn't make it compatible with

by jbeda 11y ago

Took a quick look at the API. For context, I was involved in the early days of Google Cloud Storage. It is surprising that they didn't make it compatible with the S3 API -- at least for common object/bucket create/delete. This will require more code to be written and it will be harder to adapt client libraries. The API documentation is here: https://www.backblaze.com/b2/docs/ https://www.backblaze.com/b2/docs/ Other notes: * The lack of scalable front-end load balancing is shown by the fact that they require users to first make an API call to get an upload URL followed by doing the actual upload. * They require a SHA1 hash when uploading objects. This is probably overkill over a cheaper CRC. In addition, it means that users have to make 2 passes to upload -- first to compute the hash and then another to upload. This can slow uploads of large objects dramatically. A better method is to allow users to omit the hash and return it in the upload response. Then compare that response with a hash computed while uploading. In the rare case that the object was corrupted in transit, delete/retry. GCS docs here: https://cloud.google.com/storage/docs/gsutil/commands/cp#checksum-validation https://cloud.google.com/storage/docs/gsutil/commands/cp#che...

brianwski 11y ago
> It is surprising that they didn't make it compatible with the S3 API .... The lack of scalable front-end load balancing is shown by the fact that they require users to first make an API call to get an upload URL And... you answered your own question. :-) We reduce our operating costs by not having as many load balancers in the datacenter and pushing off the responsibility to the API. It all comes from our traditional backup product where we wrote all the software on both sides so we could save money this way. With that said, we are actively considering offering an S3 compatible API for a slightly higher cost (basically what it would cost us to deploy the larger load balancing tech).
- sagivo 11y ago
  S3 compatible API is the only reason preventing us from migrating to your cheaper option right now.
  - AYBABTME 11y ago
    Wouldn't be very hard to make your own proxy.
    - Gigablah 11y ago
      Famous last words :)
- mwcampbell 11y ago
  I, for one, prefer the directness of not having to go through a front-end proxy. It probably eliminates some failure modes. I think that, instead of Backblaze providing an S3-compatible API, someone should do an open-source S3-compatible front-end for B2, that any interested user can run on a cheap VPS.
  - khc 11y ago
    If someone writes a jclouds provider for B2, you can then use https://github.com/andrewgaul/s3proxy https://github.com/andrewgaul/s3proxy to interface with it
  - toomuchtodo 11y ago
    https://libcloud.apache.org/ https://libcloud.apache.org/
  - narsil 11y ago
    I work at https://kloudless.com https://kloudless.com. While not S3-compatible, we offer a similar proxy that provides a single API to multiple storage services such as Dropbox, Box, Google Drive, SharePoint, etc. We've also released support for S3 and Azure and are looking into B2.
    - deleted 11y ago
      [deleted]
heavenlyhash 11y ago
> They require a SHA1 hash when uploading objects. This is probably overkill over a cheaper CRC. Having been on the receiving end of entirely too many corrupted files in my life, I strongly approve of their use of a hash that's been standardized and fast for decades and remains cryptographically strong. "But fast" if you fail to store it isn't very helpful. TCP has a CRC too. We're wallpapering over it with better ones and everyone serious has been for years: it's time to accept that cheap CRCs aren't a good place to get stuck. Improving the API to avoid the 2-pass problem is spot-on though. Another possible solution is to require either a subsequent API call, or format the first message as a multipart, and use that route to have the caller submit the hash that's used to confirm and commit the file to storage after the body upload. This would solve the 2pass problem while still ensuring the client is actually doing the integrity check -- and since Backblaze is more than likely to take the heat on any corruption issues, it's probably a good policy for them to make sure lazy client implementations aren't going to cause problems that their storage then gets the publicity smear for.
- jbeda 11y ago
  Using a hash or CRC here is totally necessary. Often times CRCs in TCP fail due to corruption outside the network stack. Having an end to end check will catch, say, memory bit flips and such after data comes off the wire. But there is no call for a cryptographic hash here. This isn't being used as any sort of ID or to verify integrity outside of corruption.
  - devit 11y ago
    No, it's pretty much totally unnecessary. The API works on top of TLS, which already includes cryptographic authentication of all data (usually via SHA-1/2 HMAC or AES-GCM). The hash would be computed at the client right after reading from disk and right before TLS enryption, and since they seem to terminate TLS at the storage server it would be computed right after TLS decryption and right before storage, so it doesn't seem to provide any gain. I think they should just remove it, or at least make it optional.
    - 11y ago

by jbeda 11y ago

brianwski 11y ago

> It is surprising that they didn't make it compatible with the S3 API .... The lack of scalable front-end load balancing is shown by the fact that they require users to first make an API call to get an upload URL And... you answered your own question. :-) We reduce our operating costs by not having as many load balancers in the datacenter and pushing off the responsibility to the API. It all comes from our traditional backup product where we wrote all the software on both sides so we could save money this way. With that said, we are actively considering offering an S3 compatible API for a slightly higher cost (basically what it would cost us to deploy the larger load balancing tech).

sagivo 11y ago
S3 compatible API is the only reason preventing us from migrating to your cheaper option right now.
- AYBABTME 11y ago
  Wouldn't be very hard to make your own proxy.
  - Gigablah 11y ago
    Famous last words :)
mwcampbell 11y ago
I, for one, prefer the directness of not having to go through a front-end proxy. It probably eliminates some failure modes. I think that, instead of Backblaze providing an S3-compatible API, someone should do an open-source S3-compatible front-end for B2, that any interested user can run on a cheap VPS.
- khc 11y ago
  If someone writes a jclouds provider for B2, you can then use https://github.com/andrewgaul/s3proxy https://github.com/andrewgaul/s3proxy to interface with it
- toomuchtodo 11y ago
  https://libcloud.apache.org/ https://libcloud.apache.org/
- narsil 11y ago
  I work at https://kloudless.com https://kloudless.com. While not S3-compatible, we offer a similar proxy that provides a single API to multiple storage services such as Dropbox, Box, Google Drive, SharePoint, etc. We've also released support for S3 and Azure and are looking into B2.
  - deleted 11y ago
    [deleted]

heavenlyhash 11y ago

> They require a SHA1 hash when uploading objects. This is probably overkill over a cheaper CRC. Having been on the receiving end of entirely too many corrupted files in my life, I strongly approve of their use of a hash that's been standardized and fast for decades and remains cryptographically strong. "But fast" if you fail to store it isn't very helpful. TCP has a CRC too. We're wallpapering over it with better ones and everyone serious has been for years: it's time to accept that cheap CRCs aren't a good place to get stuck. Improving the API to avoid the 2-pass problem is spot-on though. Another possible solution is to require either a subsequent API call, or format the first message as a multipart, and use that route to have the caller submit the hash that's used to confirm and commit the file to storage after the body upload. This would solve the 2pass problem while still ensuring the client is actually doing the integrity check -- and since Backblaze is more than likely to take the heat on any corruption issues, it's probably a good policy for them to make sure lazy client implementations aren't going to cause problems that their storage then gets the publicity smear for.

jbeda 11y ago
Using a hash or CRC here is totally necessary. Often times CRCs in TCP fail due to corruption outside the network stack. Having an end to end check will catch, say, memory bit flips and such after data comes off the wire. But there is no call for a cryptographic hash here. This isn't being used as any sort of ID or to verify integrity outside of corruption.
- devit 11y ago
  No, it's pretty much totally unnecessary. The API works on top of TLS, which already includes cryptographic authentication of all data (usually via SHA-1/2 HMAC or AES-GCM). The hash would be computed at the client right after reading from disk and right before TLS enryption, and since they seem to terminate TLS at the storage server it would be computed right after TLS decryption and right before storage, so it doesn't seem to provide any gain. I think they should just remove it, or at least make it optional.
  - 11y ago