4 ms·

I would love to do this. Cross-connects aren't free, but they are certainly less expensive than going out across the Internet. And enabling compute/storage use

I would love to do this. Cross-connects aren't free, but they are certainly less expensive than going out across the Internet. And enabling compute/storage use cases by (possibly) partnering with other providers would be excellent. I'll reach out to a few of those compute providers and explore. Can you tell me more about how you use/would use cloud storage? (with or without compute)

Veratyr 9y ago
Currently I don't really use cloud storage, it's cheaper for me to colocate a 2U stuffed with disks and buy transit. Primarily though I store media. I have (for a home user) large amounts of high resolution (lossless audio, RAW images, high bitrate (even ProRes) video) media that I like to keep for archival and transcoding but don't need for playback. I want to store the original media somewhere safe and cheap but have the ability to transcode it to a format suitable for playback/streaming on demand. Actually it just occurred to me that it might be worth you talking to the folks at Plex (https://plex.tv https://plex.tv) about this too, as they're working on their Cloud product and need more places for their users to store media.
- budmang 9y ago
  Got it. Transcoding & making image thumbnails are two common use cases we hear. Thanks.
- jjeaff 9y ago
  If I was a storage company, I would consider Plex a hot potato and would stay far, far away. There is a reason Plex isn't building out their own storage and the reason isn't cost. It's because they know that 99% of all the media their customers handle is pirated content. At some point, Plex will become large enough to get on the radar of the MPAA and at that point, I assume they will go after the cloud storage providers.
  - Veratyr 9y ago
    I feel like it'd be the opposite. DMCA protects the storage provider as long as they're unaware that they're storing infringing content and Plex limits sharing enough that any infringement is unlikely to reach the MPAA/RIAA.
    - jjeaff 9y ago
      I think inking a deal with Plex would be pretty obvious. I'm not sure where the line is for knowing that you are storing infringing content, but MEGA claims they didn't know what users were uploading.
      - Veratyr 9y ago
        Plex itself is a US company and were it "obvious" that its existing Cloud product was infringing on copyrights, it would have already been the target of a lawsuit.
  - cookiecaper 9y ago
    Yeah, I'm not sure about that. Plex has nothing to do with how users are getting the content, it just offers playback and cataloging after they've already retrieved it. I'd agree if they allowed something like library sharing between users, but I don't see them as a likely target just for allowing playback of consumer media. I use a (local) Plex installation for home videos, downloaded clips, and various other non-infringing things. It's the most convenient way to share the library out to a Chromecast/phone/Roku.
    - jjeaff 9y ago
      Ya, I don't think Plex would be necessarily at fault, although they could be susceptible to DMCA takedown notices. But a storage provider could get into deep trouble if they know tons of pirated content was being uploaded. Kim Dotcom claims MEGA had no idea if the content being uploaded was infringing.
      - cookiecaper 9y ago
        Megaupload (which I assume you mean because afaik MEGA has not been in legal trouble) was a distribution platform; the purpose was to upload something and give the link to others. If it was a personal file locker that couldn't link out or be shared, as Plex intends to be, it probably would not have had issues.
derekp7 9y ago
I can give you my example -- I have an open source backup tool called Snebu which uses an sqlite DB to store the backup catalog (file names, file hash, metadata etc), whereas the individual files are stored in a vault area (currently just a disk volume, with files named after the sha1 hash). Backup data is sent to it by piping in tar format to the main process, which updates the catalog DB and deposits files into the storage vault. I'd like to add cloud capabilities, but for that I'd have to run the backend process somewhere that can easily update the DB, and easily read/write the data in the vault, along with syncing the full DB to the vault location after a backup. My other option is to run everything locally, and compute a delta file between versions of the DB and send that delta over to the vault (i.e., do D2D2C backups). Edit: Another possible requirement -- I'd like to pack multiple smaller files into an archive file, and deposit that into a bucket. But as files age out of the archive, I'd like to be able to move active ones out of it and into a new one. Right now, I'd have to re-create the still-valid backup data out of on-hand information, upload the new archive, then send a command to delete the old one.
loxias 9y ago
Speaking for myself, I'm excited about using B2 as a replacement for more expensive local disk in not-heavy-iops-role machines. It's not the most exotic use, but if you look at every vm provider (linode, digital ocean, even amazon) the prices of local disk are huge in comparison to the prices of cloud storage like B2 & S3. By combining them I can have a cheap box (Linode for instance) with a local write cache, with B2 providing a terabyte of raw storage, and get a slow but effective VM with a 1TB "local disk" for under $20 month. Which is great.
- Veratyr 9y ago
  Unless you're going to be doing pretty much entirely sequential reads of entire files, I don't think this is likely to work well for you. B2 (and other cloud storage) isn't just priced by bandwidth but by "transactions" (requests).
  - loxias 9y ago
    It's true, there is a price for some API requests, but it's actually quite low if you're using the cloud storage to populate a local cache! Backblaze categorizes download as a "class B" transaction with 2500 free per day and $0.004/10k. I fully admit it's not a suitable solution for many workloads, but I've been playing with a VM using B2 as slow additional disk (as an additional volume, not suggesting putting / or /usr..) and am yet to exceed the free tier.
    - budmang 9y ago
      So are you actually running a VM image somewhere and using B2 as a virtual disk drive from it?

Veratyr 9y ago

Currently I don't really use cloud storage, it's cheaper for me to colocate a 2U stuffed with disks and buy transit. Primarily though I store media. I have (for a home user) large amounts of high resolution (lossless audio, RAW images, high bitrate (even ProRes) video) media that I like to keep for archival and transcoding but don't need for playback. I want to store the original media somewhere safe and cheap but have the ability to transcode it to a format suitable for playback/streaming on demand. Actually it just occurred to me that it might be worth you talking to the folks at Plex (https://plex.tv https://plex.tv) about this too, as they're working on their Cloud product and need more places for their users to store media.

budmang 9y ago
Got it. Transcoding & making image thumbnails are two common use cases we hear. Thanks.
jjeaff 9y ago
If I was a storage company, I would consider Plex a hot potato and would stay far, far away. There is a reason Plex isn't building out their own storage and the reason isn't cost. It's because they know that 99% of all the media their customers handle is pirated content. At some point, Plex will become large enough to get on the radar of the MPAA and at that point, I assume they will go after the cloud storage providers.
- Veratyr 9y ago
  I feel like it'd be the opposite. DMCA protects the storage provider as long as they're unaware that they're storing infringing content and Plex limits sharing enough that any infringement is unlikely to reach the MPAA/RIAA.
  - jjeaff 9y ago
    I think inking a deal with Plex would be pretty obvious. I'm not sure where the line is for knowing that you are storing infringing content, but MEGA claims they didn't know what users were uploading.
    - Veratyr 9y ago
      Plex itself is a US company and were it "obvious" that its existing Cloud product was infringing on copyrights, it would have already been the target of a lawsuit.
- cookiecaper 9y ago
  Yeah, I'm not sure about that. Plex has nothing to do with how users are getting the content, it just offers playback and cataloging after they've already retrieved it. I'd agree if they allowed something like library sharing between users, but I don't see them as a likely target just for allowing playback of consumer media. I use a (local) Plex installation for home videos, downloaded clips, and various other non-infringing things. It's the most convenient way to share the library out to a Chromecast/phone/Roku.
  - jjeaff 9y ago
    Ya, I don't think Plex would be necessarily at fault, although they could be susceptible to DMCA takedown notices. But a storage provider could get into deep trouble if they know tons of pirated content was being uploaded. Kim Dotcom claims MEGA had no idea if the content being uploaded was infringing.
    - cookiecaper 9y ago
      Megaupload (which I assume you mean because afaik MEGA has not been in legal trouble) was a distribution platform; the purpose was to upload something and give the link to others. If it was a personal file locker that couldn't link out or be shared, as Plex intends to be, it probably would not have had issues.

derekp7 9y ago

I can give you my example -- I have an open source backup tool called Snebu which uses an sqlite DB to store the backup catalog (file names, file hash, metadata etc), whereas the individual files are stored in a vault area (currently just a disk volume, with files named after the sha1 hash). Backup data is sent to it by piping in tar format to the main process, which updates the catalog DB and deposits files into the storage vault. I'd like to add cloud capabilities, but for that I'd have to run the backend process somewhere that can easily update the DB, and easily read/write the data in the vault, along with syncing the full DB to the vault location after a backup. My other option is to run everything locally, and compute a delta file between versions of the DB and send that delta over to the vault (i.e., do D2D2C backups). Edit: Another possible requirement -- I'd like to pack multiple smaller files into an archive file, and deposit that into a bucket. But as files age out of the archive, I'd like to be able to move active ones out of it and into a new one. Right now, I'd have to re-create the still-valid backup data out of on-hand information, upload the new archive, then send a command to delete the old one.

loxias 9y ago

Speaking for myself, I'm excited about using B2 as a replacement for more expensive local disk in not-heavy-iops-role machines. It's not the most exotic use, but if you look at every vm provider (linode, digital ocean, even amazon) the prices of local disk are huge in comparison to the prices of cloud storage like B2 & S3. By combining them I can have a cheap box (Linode for instance) with a local write cache, with B2 providing a terabyte of raw storage, and get a slow but effective VM with a 1TB "local disk" for under $20 month. Which is great.

Veratyr 9y ago
Unless you're going to be doing pretty much entirely sequential reads of entire files, I don't think this is likely to work well for you. B2 (and other cloud storage) isn't just priced by bandwidth but by "transactions" (requests).
- loxias 9y ago
  It's true, there is a price for some API requests, but it's actually quite low if you're using the cloud storage to populate a local cache! Backblaze categorizes download as a "class B" transaction with 2500 free per day and $0.004/10k. I fully admit it's not a suitable solution for many workloads, but I've been playing with a VM using B2 as slow additional disk (as an additional volume, not suggesting putting / or /usr..) and am yet to exceed the free tier.
  - budmang 9y ago
    So are you actually running a VM image somewhere and using B2 as a virtual disk drive from it?