20 ms·
Polycloud: S3-compatible object storage
- elcomet 6y agoThe important point in the FAQ: > The encrypted data pieces are scattered across our community of users, using small portions of our member’s unused hard drive space So it's not really comparable to S3 or B2. It's more like storj or sia.
- lcabral 6y agoI think they meant int the sense of a API compatible > Polycloud is 100% AWS S3 bit-compatible. If you’re used to using an S3 API, you can access Polycloud using the same API.
- zokier 6y agoDo you also provide same consistency guarantees as S3?
- onionjake 6y agoYes. We are strongly consistent.
- ahachete 6y agoS3 only became strongly consistent recently. It is a major feat to do so. I can imagine doing it without controlling the storage layer, network topology and specially network latency is only harder. Could you please elaborate a bit more on how do you achieve it?
- onionjake 6y agoS3 has been around for a long time and they had a lot of objects to transition when they upgraded, so I imagine that is why it took awhile. There are other object storage systems that have strong consistency guarantees that came out after S3. It greatly simplifies things that an object written to S3 it is immutable. On a high level, all writes to your storage use some UUID. All reads use a consistent metadata storage (pick a modern database). After your write is complete and you are sure it is persisted, do the metadata update and return success. Everyone gets a consistent view of the operation.
- KaiserPro 6y agoI'm assuming the catch with this is that you are letting your data be stored on some randomer's set of disks. which means availability is directly tied to the interest of nerds who are loaning out their hdd space for a pittance. https://crowdstorage.com/faq/#where-is-my-data-stored https://crowdstorage.com/faq/#where-is-my-data-stored
- aequitas 6y agoI also cannot find any link to join the community and become a storage provider. Maybe it operates under a different name?
- onionjake 6y agoAnswered in another thread https://news.ycombinator.com/item?id=26581857 https://news.ycombinator.com/item?id=26581857
- ClumsyPilot 6y agoThe avaliability of coffee is directly tied to the interest of random farmers who are growing it for a pittance. How is it different than any other market? To bitcoin nodes? To Torrents, Tor network? Wikipedia depends on donations of random nerds. Like this is such a self-depreciating perspective, that nothing can be done or relied upon unless its done by a megacorp.
- KaiserPro 6y agoThanks for that, I hadn't truly grasped the concept of supply and demand until you so clearly summed up macro economics. To just fill you in, Coffee growers grow coffee because the cost of growing coffee is less than the local wholesale price (most of the time.) It is possible to subsist on the profits of growing coffee(depending on where you are). The price per TB being charged to the consumer is $48 a year. which is significantly less than the initial setup cost to become a data host. (pi + sd card + hdd) That's before we get to the opex of paying for an ISP, (and any bandwidth overages) plus power and general maintenance. Thats assuming that the company is selling at the price it pays the "hosts". I'm assuming they have some ambition for profitability. which leads me back to the original statement: you're reliant on a bunch of randomers Who didn't really think about the economics subsidising this company.
- reacharavindh 6y agoWhat is the catch here? How can it be cheaper than b2 while essentially offering the same service to the end-user?(geographically distributing data does not directly result in any reduction of costs). It’s okay to say that “we have lower profit margin” than B2, what is it technically that enables this lower cost? , and what are the trade offs made to get there?..
- toolslive 6y agothere are several things you can do. - First, like you indicated is: you can take less margin (and believe me, there's still considerable margin). - You can play with erasure coding policy. (n+10)/n makes that you can tolerate 10 failures. a higher n makes your storage overhead less. (which means more margin) - you can use even fancier storage schemes (fe online codes will make your storage overhead something like 3%) - you can use cheaper hardware. - ...
- wctwct 6y agoManaging a distributed network has a much lower cost structure - no capital investment for hard drives, no electrical cost for running the hard drives, no cooling costs, etc.
- dodyg 6y agoIf you want people to upload data to your system, it helps to give them more information about your organization.
- szszrk 6y agoIt may not be clear at first sight, but they provide info about the company [0] and that data is actually stored on drives provided by community [1]. It could have be laid out more clearly, though. [0] https://crowdstorage.com/privacy/ https://crowdstorage.com/privacy/ [1] https://crowdstorage.com/faq/#where-is-my-data-stored https://crowdstorage.com/faq/#where-is-my-data-stored
- dubcanada 6y agoWhat happens if say 3 shards of a specific file you need are not online that day? Or never come back online again? I understand the files are copied to multiple locations but we are not talking about a 24/7 datacenter but random computers. People can decide to uninstall the program or never power back on or anything. Unless it’s copied to basically every computer I don’t see how eventually someone would have a corrupted file just because they are unable to piece it back together with available computers.
- PurpleFoxy 6y agoThe solution for this is to require hosts to enter some kind of deposit in to the system. If they go offline too much, the deposit is forfeit.
- ohashi 6y agoBasically what Storj does. I am kind of wondering if they are backed by it based on descriptions
- rglullis 6y agoI confess that I haven't gone deep into Storj, but I am running a node for some weeks now and I didn't have to pay anything (besides the operational costs of keeping the server online, of course) to get into it.
- ClumsyPilot 6y agoYou are earning money every month, but they keep a colateral and payment to you is delayed. If your nodes combusts one day, that colateral dissapears broadly speaking. Also there are like 32 shards for each data, so 32 conpuetrs don't go offline in one day.
- wctwct 6y agoWe now manage a network of over 250,000 NAS devices that are part of our network. So we know how to handle devices going offline without losing data. This approach has been used at scale since 2014.
- v02zfl8pg9 6y ago> Yes, Polycloud is GDPR compliant. [...] processed and stored on servers located across the United States [...] certified with the EU-U.S. Privacy Shield Framework. I don't think this means Polycloud is GDRP compliant. It's my understanding the Privacy Shield Framework was struck down by the EU courts. Might be a bit careful here if you are a EU business.
- quyse 6y agoIt seems the catch is the same as with Wasabi. It says everywhere on the site that egress is free and there's no hidden fees. Except for this FAQ entry [1]: > Pricing for Polycloud is only $0.004 per GB per month. There are no other fees as long as you do not egress more than 100% of your data during the month. If you do egress more than 100% of the amount of data you have stored then you will only be charged $0.01 per GB for any overages. Looks like a hidden fee to me. Note that even the Pricing page [2] does not mention it, and the calculator does not allow to enter egress volume for calculation. [1] https://crowdstorage.com/faq/#what-is-the-pricing-for-polycloud https://crowdstorage.com/faq/#what-is-the-pricing-for-polycl... [2] https://crowdstorage.com/products/polycloud/pricing/ https://crowdstorage.com/products/polycloud/pricing/
- onionjake 6y agoThis is great feedback. We are redesigning the website right now and will make sure it is more prominent.
- gccs 6y agoChange it to: exgress is free as long as you don't use it
- nine_k 6y agoEgress is free as long as you use it for individual backup, or for other data you want to keep just in case and mostly discard, like old logs.
- igetspam 6y agoIf I do the math but I store a lot of data that gets served by a CDN. It rarely changes and is requested enough that it will stay in the cache. I'm already paying for the CDN, so I can't save there but if can save on storage, it could be useful. I'm not disagreeing with the point that that pricing is clear or not "hidden" but more that there's a use case here that can make sense.
- ing33k 6y agoalso check out https://tardigrade.io/ https://tardigrade.io/
- wctwct 6y agoVery similar to tardigrade, just much lower cost, $0.004/GB for Polycloud vs. $0.010/GB for Tardigrade.
- mike-cardwell 6y ago$4/month for 1TB of storage and 1TB of bandwidth? Just get a 1TB Storage VPS (+8TB BW) from https://www.time4vps.com/storage-vps/ https://www.time4vps.com/storage-vps/ for 3 EURO a month instead and stick Minio on it ;) (Just a happy customer)
- onionjake 6y agoWe are pay-as-you-go and charge per GB, so you can store 1-10GB and pay nothing, 13 GB and pay $0.01/month, etc. It scales up and handles PB of data as well.
- Dylan16807 6y ago> this discount is valid only for the first invoice and it is not reoccurring So it'll double in price after that. And that's on top of having to round up to the next terabyte or two. But if you want more bandwidth then it looks like it's worth considering. Side note: It's interesting that above 2TB the deal gets steadily worse.
- deallocator 6y ago"We designed this service to reliably hold a huge amount of data. This setup will serve you best if it’s used to store compressed data archives or backups." "Do you offer backups with this service? Unfortunately, no. Users have to regularly back up the data themselves. " That sounds a little contradicting
- 6y ago
- headmelted 6y agoWe’ve seen these types of systems before and I always wonder how well thought out this is - either on the part of the hosts, the company or the client. A couple of questions I have right off the bat are: How do we know how secure this solution is? Even if it were incredibly well secured - what are the laws around this setup? If someone stores illegal materials on this system who is responsible? (The person who stored it? The company? The unsuspecting host?) What happens if the hosts lose interest or it fails commercially? Does the data get lost without warning when hosts start uninstalling the software? The problems with these systems is almost never the technology - it’s finding a way to negotiate the millions of different implications storing information on other people’s behalf brings.
- onionjake 6y agoWe've been storing data like this for several years and have kept hundreds of PB and 10's of billions of objects without data loss despite having many nodes in areas affected by large regional outages. One recent example is when Texas was affected by blackouts. We are working on getting as much content up on the website as we can and telling more about us!
- headmelted 6y agoI’d genuinely be interested in seeing that, it would certainly give me more confidence. Have you looked into the legal aspects of your setup as regards liabilities? It probably doesn’t matter as much from the customer’s perspective as it’s their data so they should know what it contains, but would be worth knowing where this leaves you as a company and/or the network of nodes if some other customer pushed data onto the system that was objectionable.
- ddorian43 6y agoHow much free space do you guys have ? If a big customers comes and stores 50PB, what do you do since you don't have servers ?
- TruthWillHurt 6y agoYou do realize storing your data away from where processing takes place is going to incur network charges and latency, right?
- wctwct 6y agoIt isn't the best solution for applications that require a lot of compute. Better for pure archive or backup applications.
- teknopurge 6y agoIn this case it seems twice as expensive as AWS glacier. I like the idea of this but do not yet see the competitive advantage. Also your FAQ mentions your community of users. Is there a forum/discord where the community can discuss/learn?
- onionjake 6y agoHow do you arrive at twice as expensive as AWS glacier? AWS S3 Glacier is the same $0.004/GB/month and requires the same minimum of 90 days, but retrieval takes "from 1 minute to 12 hours". They also have large retrieval costs (on top of the regular egress to the internet) and API call charges. AWS S3 Deep glacier is only $0.00099/GB/month, but requires it to be stored 180 days and also have operation/retrieval fees. "For long-term data archiving that is accessed once or twice in a year and can be restored within 12 hours" https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/ We feel like the immediate availability of our offering is compelling and a competitive advantage over glacier. > Is there a forum/discord where the community can discuss/learn? Not yet, we will have one created soon.
- teknopurge 6y agofrom https://aws.amazon.com/glacier/pricing/ https://aws.amazon.com/glacier/pricing/ Select Bulk $0.001 per GB (Storage) Select Bulk $0.0025 per GB (Transfer)
- Mortiffer 6y agoWould love to have lower cost S3. Last time I tried one I got burned when they changed their prices and then made it really hard to migrate with super slow outbound bandwidth. Anyway my advice is giving people something to de risk the cost of migration. IDK maybe an abstraction layer that automatically keeps a backup copy inside s3 glacier . Or something else ...
- viraptor 6y agoThe first thing I check on services which advertise as S3-compatible is permissions. As usual - no permissions, no ACLs. So practically it looks like anyone with an access key can wipe out everything you store with them.
- onionjake 6y agoWe are working on it! We plan to have support for ACL, object locking, versioned buckets, etc.
- jjt-yn_t 6y agoSome/many of these solutions have an unstated low speed UPLOAD bottleneck. Anyone with with definitive answer(s) to that?
- drip-egg-jack 6y agohttps://tebi.io/ https://tebi.io/ has a solution - you always upload to the closest server in your region that is much faster then upload to a central location.
- JCM9 6y agoThe catch is always in the fine print. The storage is cheap but the savings go away if you actually do anything with the data. Storage is a foundational service in the cloud. There are huge advantages to having that storage sit adjacent to everything else. Such storage as a service somewhere else doesn’t make a lot of sense in many/most use cases. Now you’re having to pay to move data across comparatively slow networks links to where it’s actually needed. It’s a catchy headline but when putting this to the test in real world scenarios those numbers don’t pan out.
- bad_username 6y ago> storage is cheap but the savings go away if you actually do anything with the data. This is the "data back up" use case, where you store heaps of data, and hope to never need to access it. For this use case the conditions seem excellent.
- JCM9 6y agoYes, but then you wouldn’t use S3 for that either so the price comparisons are misleading.
- wctwct 6y agoBest for archive and backup. You can also see how Vivint uses them to store video clips and stream them to customers when needed: https://crowdstorage.com/solutions/ https://crowdstorage.com/solutions/
- the_duke 6y ago> but then you wouldn’t use S3 for that either Why do you have that impression? S3 and similar are used a lot for backups, especially Glacier.
- onionjake 6y agoAnother thread I posted a little bit about how we are compared to glacier: https://news.ycombinator.com/item?id=26582429 https://news.ycombinator.com/item?id=26582429 Glacier can get pricey when you store/retrieve your data because of ops, retrieval fees, and egress (if going to the internet). We feel like immediate availability is a compelling advantage of our product.
- mtlynch 6y ago>This means that with our 20/40 encoding scheme, a malicious attacker would need to physically access 20 different nodes within our network of almost 300,000 devices. Which community is providing 300k devices? Is Polycloud building on top of IPFS or Sia?
- onionjake 6y agoDiscussed in another thread: https://news.ycombinator.com/item?id=26581399 https://news.ycombinator.com/item?id=26581399 There is a 'Contact Us' button at the bottom of the page you can always reach out.
- onionjake 6y agoHi, CTO of CrowdStorage here. Happy to answer any questions. I'm doing my best to respond to comments. Our website is right now being redesigned and your comments here are helpful to help us know where we need to improve!
- iagovar 6y agoYour site says it's distributed, but honestly it has no details of how it works. Is it built on top fo SIA, STORJ or similar? How does it work? How does one "become a node", etc.
- tw04 6y agoThe fact they seem to refuse to answer WHERE the storage is coming from is honestly pretty concerning. It's been asked multiple times in this thread and they don't seem interested in addressing it. For all we know the storage is coming compromised hosts that don't even know they're participating.
- iagovar 6y agoYeah, I wouldn't trust my data to a service I have no idea how it works.
- onionjake 6y agoNot trying to obscure anything, just busy this morning :) I answered the parent.
- neverartful 6y agoI just found this poking around on their website: https://crowdstorage.com/faq/#where-is-my-data-stored https://crowdstorage.com/faq/#where-is-my-data-stored Where is my data stored? Your data is intelligently encrypted on your My Cloud Home and sliced into tiny pieces that only you can put back together. The encrypted data pieces are scattered across our community of users, using small portions of our member’s unused hard drive space. Spreading your data out in this way makes it more safe and secure than it would be if it was all stored in one location.
- robjan 6y agoWhere is the other side, i.e how are people with hard drives recruited? It didn't seem to be on this site but it's important to know otherwise how do we know it's not, for instance, a botnet?
- bennyp101 6y agoSays it is GDPR compliant - but I don't see it explained how? You say you would use "Standard Data Protection Clauses" if the data goes outside of the US ... but I don't want my data in the US in the first place, so is this really a US only service? How would those clauses be implemented? Does every node have a contractual agreement with you? How would you know if someone took their computer with them on holiday to another country? Also, how is the data encrypted? "State of the art encryption" is just marketing fluff :)
- onionjake 6y ago> so is this really a US only service We plan on launching regions outside of the U.S. that only store data outside of the U.S. soon. > Also, how is the data encrypted? "State of the art encryption" is just marketing fluff :) AES256 and/or xsalsa20poly1305. You can also use SSE-C to provide our servers with a key to encrypt the data (which our servers then promptly forget). https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/Server...
- bennyp101 6y agoThanks. How do you handle the data leaving the country? With someone just taking their machine on holiday etc
- shay_ker 6y agoIs there a technical reason that cloud stores charge egress fees, or is it just purely "we want to disincentivize people from using competitors"?
- jnwatson 6y agoI think it is to encourage write-once, read-rarely use cases like logs and backups, as opposed to serving web assets.
- onionjake 6y agoPrecisely. Although we also believe you shouldn't have to worry about going bankrupt if you ever need to restore all your data or want to migrate somewhere else, which is where the 100% egress of what you have stored comes from.
- EricE 6y agoWell data transfer isn't free - so they have to have some model to help cover that overhead. The roach motel model would be considered making lemonaide by marketing types - not sure as a customer I'd readily agree :/
- killingtime74 6y agoProfit. Its the Biggest part of the margin
- clarkevans 6y agoAre there S3 compatible services where they charge for storage/ingress/egress straight-up, without clever billing?
- e12e 6y agoDigital Ocean spaces has pretty straightforward pricing? Hetzner storagebboxes/shares are great value - but no s3 api.
- treesknees 6y agoIf you scroll a bit on the Backblaze B2 pricing page, they have an easy calculator that will tell you exactly what the costs will be. $0.005/GB/Month for storage $0.01/GB for egress The only "hidden" cost I'd say they have is they have a limit on some API calls. For example their b2_download_file_by_name is limited to 2500 calls per day and then $0.004 per 10k calls after that. https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/b2-transactions-price.html https://www.backblaze.com/b2/b2-transactions-price.html
- acejam 6y agoFilebase is S3 compatible and has pretty straight forward pricing [1] [1] https://filebase.com https://filebase.com
- drip-egg-jack 6y agohttps://tebi.io/ https://tebi.io/ charge for the storage, you can define how many copies and where you want to store your data and a flat egress fee
- Proven 6y agoSounds okay but it'd have to be much cheaper than S3 for non-individual buyers (orgs, enterprises) to use it... 10 years ago I used a backup app that worked on similar principle, the name escapes me now, it kept my home NAS very busy (high churn due to people joining and leaving, and lots of network traffic) - I had to turn it off.
- chaz6 6y agoOne killer feature could be some form of ransomware protection. If a sudden entropy change is detected, snapshot the data, then provide the customer with the option to revert changes from that time (up to, say, a week).