4 ms·

I have been asking for non-SSD on DO for a long time now. My heart jumped when I saw the HN title, only to be dashed on the rocks. What are us data nerds suppo

I have been asking for non-SSD on DO for a long time now. My heart jumped when I saw the HN title, only to be dashed on the rocks. What are us data nerds supposed to do? We want to take 10 terabytes, run a batch process on it, keep the 20TB, then continue with about 5GB of working data until the next month's terabyte comes in, then we want to batch through the 21TB. Right now the price slider doesn't even go up to 21TB, and clicking on the "need more storage button" doesn't go anywhere, but I'm assuming it would be $2100 / month which is more than 3x as expensive than vanilla S3.

mattbee 10y ago
At Bytemark we've done network as standard since 2012, so we can do 100GB of higher-capacity storage for £2/month, and all discs are expandible / migrateable online: bigv.io/prices
boulos 10y ago
Not being snarky, but at that scale why would you still be using Digital Ocean? Are you processing this dataset with just a single droplet? Are you going to keep each months 10 TiB forever? Something like Nearline is going to be a much better fit even if you process the data and have to pay the $.01/GB retrieval fee. Disclaimer: I work on compute engine (so of course I want your money), but I'm honestly curious.
mikeash 10y ago
"All SSD all the time" is kind of DO's thing. There are lots of other VPS providers with round storage if that's what you're after.
- raiyu 10y ago
  All SSD all the time, we may make that our new tagline. But yeah, we basically look to SSD for everything as that continues to take over market share from spinning disk and when you have to support a product 5-10 years, it's necessary to look ahead, so as SSD prices continue to drop we will eventually switch to SSD for everything and spinning disk will become the equivalent of tape drives in the old days.
Veratyr 10y ago
You're looking at the wrong market. At Hetzner, you can rent a dedicated server with 2x3TB drives for about $25/month ($0.005/GB, to scale to multiple servers, use Ceph) and a few larger machines for under $100. At OVH you can buy object storage for around 1c/GB and rent a few dedicated servers for less than $100/month, or use their cloud. If you go to the _really_ low end, time4vps will give you a 1TB VPS for 2EUR/month, if you pay for 2 years (and they give you 4x the storage as bandwidth). I don't work for any of these companies but I have services with all of them.
- 3pt14159 10y ago
  Yeah, but I like the DO per hour billing, design, and API. Other than their insecure-by-default SSH key thing, https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/9307569-return-the-droplet-s-ssh-public-key-as-part-of-api https://digitalocean.uservoice.com/forums/136585-digitalocea... that is.
  - Veratyr 10y ago
    You can get per hour billing for compute at OVH and I believe they're Openstack based so their API isn't too bad.
  - happyslobro 10y ago
    How many hours does it take to load 10TB and then reduce it to 5GB? ;)
    - 3pt14159 10y ago
      Actually because I use the DO API and lots of small instances that work together, not that long. Right now I use S3, which blows because I hate the S3 cli and API, so I would love a better solution.
      - tedmiston 10y ago
        Not sure if it's much better for you use case, but Backblaze introduced an S3 competitor called B2 a few months ago. Price calculation is straightforward: - $0.005/GB/month for storage - $0.05/GB download - $0.004 per 10k downloads There is a base free tier as well. https://www.backblaze.com/b2/ https://www.backblaze.com/b2/
raiyu 10y ago
You can actually create multiple volumes and then stripe them together so you should be able to get above 16TB with a bit of work. The reason we focus on SSD exclusively for most of our services is that there are obvious performance benefits and as we look into the future the prevalence of spinning disk will continue to decrease over time. That being said, if we are only providing SSD there are going to be customer use cases where it's not going to make sense, but that's also part of looking into the future. So you should be able to create a larger striped volume to run through your batch processing and then depending on how you plan to access that data afterwards you can ship that off to an object store if it's going to be accessed infrequently.
KaiserPro 10y ago
For the price of 16 Tbs plus, real steel becomes very appealing. You've got to remember that block storage is >> faster than s3/http. Its also has significantly lower latency. Also if you want fast processing you need to start looking at shared filesystems/NFS
jclulow 10y ago
If you have a lot of data that you want to ingest and store, without keeping it in an active instance, you might want to take a look at Joyent Manta[1]. We allow you to run compute jobs directly on the storage servers themselves -- basically anything you can do in a UNIX environment with a script you can run against an object in the store, from "grep" up to "python" and beyond. If you were to select "single copy" storage, the public service pricing[2] might be more interesting for you; if you grow too big or want to move onto your own hardware, the stack is open source[3] and/or we can sell you support! Disclaimer: I work at Joyent. [1]: https://www.joyent.com/manta https://www.joyent.com/manta [2]: https://www.joyent.com/pricing/manta https://www.joyent.com/pricing/manta [3]: https://github.com/joyent/manta https://github.com/joyent/manta

mattbee 10y ago

At Bytemark we've done network as standard since 2012, so we can do 100GB of higher-capacity storage for £2/month, and all discs are expandible / migrateable online: bigv.io/prices

boulos 10y ago

Not being snarky, but at that scale why would you still be using Digital Ocean? Are you processing this dataset with just a single droplet? Are you going to keep each months 10 TiB forever? Something like Nearline is going to be a much better fit even if you process the data and have to pay the $.01/GB retrieval fee. Disclaimer: I work on compute engine (so of course I want your money), but I'm honestly curious.

mikeash 10y ago

"All SSD all the time" is kind of DO's thing. There are lots of other VPS providers with round storage if that's what you're after.

raiyu 10y ago
All SSD all the time, we may make that our new tagline. But yeah, we basically look to SSD for everything as that continues to take over market share from spinning disk and when you have to support a product 5-10 years, it's necessary to look ahead, so as SSD prices continue to drop we will eventually switch to SSD for everything and spinning disk will become the equivalent of tape drives in the old days.

Veratyr 10y ago

You're looking at the wrong market. At Hetzner, you can rent a dedicated server with 2x3TB drives for about $25/month ($0.005/GB, to scale to multiple servers, use Ceph) and a few larger machines for under $100. At OVH you can buy object storage for around 1c/GB and rent a few dedicated servers for less than $100/month, or use their cloud. If you go to the _really_ low end, time4vps will give you a 1TB VPS for 2EUR/month, if you pay for 2 years (and they give you 4x the storage as bandwidth). I don't work for any of these companies but I have services with all of them.

3pt14159 10y ago
Yeah, but I like the DO per hour billing, design, and API. Other than their insecure-by-default SSH key thing, https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/9307569-return-the-droplet-s-ssh-public-key-as-part-of-api https://digitalocean.uservoice.com/forums/136585-digitalocea... that is.
- Veratyr 10y ago
  You can get per hour billing for compute at OVH and I believe they're Openstack based so their API isn't too bad.
- happyslobro 10y ago
  How many hours does it take to load 10TB and then reduce it to 5GB? ;)
  - 3pt14159 10y ago
    Actually because I use the DO API and lots of small instances that work together, not that long. Right now I use S3, which blows because I hate the S3 cli and API, so I would love a better solution.
    - tedmiston 10y ago
      Not sure if it's much better for you use case, but Backblaze introduced an S3 competitor called B2 a few months ago. Price calculation is straightforward: - $0.005/GB/month for storage - $0.05/GB download - $0.004 per 10k downloads There is a base free tier as well. https://www.backblaze.com/b2/ https://www.backblaze.com/b2/

raiyu 10y ago

You can actually create multiple volumes and then stripe them together so you should be able to get above 16TB with a bit of work. The reason we focus on SSD exclusively for most of our services is that there are obvious performance benefits and as we look into the future the prevalence of spinning disk will continue to decrease over time. That being said, if we are only providing SSD there are going to be customer use cases where it's not going to make sense, but that's also part of looking into the future. So you should be able to create a larger striped volume to run through your batch processing and then depending on how you plan to access that data afterwards you can ship that off to an object store if it's going to be accessed infrequently.

KaiserPro 10y ago

For the price of 16 Tbs plus, real steel becomes very appealing. You've got to remember that block storage is >> faster than s3/http. Its also has significantly lower latency. Also if you want fast processing you need to start looking at shared filesystems/NFS

jclulow 10y ago

If you have a lot of data that you want to ingest and store, without keeping it in an active instance, you might want to take a look at Joyent Manta[1]. We allow you to run compute jobs directly on the storage servers themselves -- basically anything you can do in a UNIX environment with a script you can run against an object in the store, from "grep" up to "python" and beyond. If you were to select "single copy" storage, the public service pricing[2] might be more interesting for you; if you grow too big or want to move onto your own hardware, the stack is open source[3] and/or we can sell you support! Disclaimer: I work at Joyent. [1]: https://www.joyent.com/manta https://www.joyent.com/manta [2]: https://www.joyent.com/pricing/manta https://www.joyent.com/pricing/manta [3]: https://github.com/joyent/manta https://github.com/joyent/manta