4 ms·

It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very qui

It is a cool project. S3 can be cost efficient, but only if you don't touch data :) Their price calculation doesn't mention cost of S3 requests, which very quickly adds up and is often neglected. It costs $1 for 2.5M GET requests to S3. They have 180 shards, in a general case query seems to fetch all of them. Presumably they don't download full shard per request, but download an index + some relevant ranges. Lets say that is 10 requests per shard. So that would be 1800 S3 GET requests = ~1400 search queries cost them $1. Assuming their service is reasonably popular and serve 1 req/second on average, that would be $1,440 per 30 days in addition to advertised $1,000 spent on EC2 and S3 storage. Seems comparable to AWS ElasticSearch service costs: - 3 nodes m5.2xlarge.elasticsearch = $1,200 - 20TB EBS storage = $1,638

fulmicoton 5y ago
I tend to agree :). If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. For that kind of use case, I'd probably start using minio. > Seems comparable to AWS ElasticSearch service costs: > - 3 nodes m5.2xlarge.elasticsearch = $1,200 > - 20TB EBS storage = $1,638 Don't forget S3 includes replication. Also EBS throughput (even with SSD) is not good at all. Also our memory footprint is tiny. This is necessary to make it run on two servers. Finally, cpu-wise, our search engine is almost 2x faster than lucene. If you don't believe us, try to replicate our demo on an elastic search :D. Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.
- rossmohax 5y ago
  > If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. How many req/s do you have in mind for your system to be a viable option? > Also EBS throughput (even with SSD) is not good at all. It is not worse than S3 still, right? > Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes. I have no deep ES experience. Are you saying, that to host 6TB of indexed data (before replication) you'd need 120 nodes ES cluster? If so, then reducing it to just 2 nodes is the real sales pitch, not S3 usage :)
- klohto 5y ago
  What about d3en instances? Clustered, and together with minio you might reach similar performance. Only issue is the inter-region traffic, it would need to be inside the same AZ EDIT: Realizing that d3 has just slow HDD
- pcnix 5y ago
  Have you checked out the new EBS gp3 disks? Throughout vs cost is much better on those than gp2, and also cheaper than Provisioned IOPS
tpetry 5y ago
I had the same feeling when reading the post. Their remark that they "estimated the cost" to be that low is in my experience a bad signal. Estimating costs on the cloud is really hard, there are so many (hidden) costs you may miss making it a lot more expensive.
heipei 5y ago
For what it's worth, if you want to run ElasticSearch on AWS I would always go with local-NVMe instances from the i3 family, this is also what AWS and Elasticsearch themselves recommend. 4x i3en.2xlarge (64GB / 5TB NVMe) at $449 / month (1yr reserved) is $1796, or $2636 without reservation, but much better performance due to the NVMe drives.
bufferoverflow 5y ago
AWS is almost never cost efficient. Maybe if you stay in their free tier.
- arcturus17 5y ago
  > AWS is almost never cost efficient. A ridiculous blanket statement, despite the "almost never" cop-out... It is cost-efficient in a wide array of scenarios. Many companies pay for it because they have calculated the different investment scenarios and AWS comes on top of alternatives such as owning the hardware or using competing cloud vendors. I own a consultancy that builds complex web apps and while I appreciate how occasionally a dev has tried to save costs for me by cramming every piece of the stack (web server, cache, db, queue, etc.), in a single Docker image to host in a droplet, I'd much rather pay for separate services, as I consider it cheaper in the long run.
  - tjoff 5y ago
    Many companies pay for it because they have spent an inordinate amount of time learning the ecosystem and know of nothing else.
  - bufferoverflow 5y ago
    Name an AWS tier that there's no cheaper alternative for. I'm only aware of Glacier, I haven't seen anything cheaper than it. AWS is convenient and reliable, but it's not cheap.
    - weird-eye-issue 5y ago
      Cheap is a whole lot different than cost efficient.
    - arcturus17 5y ago
      Convenience and reliability are deeply related with cost efficiency - that’s the point the example I made was meant to illustrate.
  - vagrantJin 5y ago
returningfory2 5y ago
For Digital Ocean object storage, data transfer to/from a Digital Ocean VM is free. You only pay for bytes-at-rest. But it seems S3 doesn't have a similar offering. Data transfer is free between S3 and EC2 instances, but you still pay the per-request charge. I wonder can you factor this into the pricing algorithm.
- rossmohax 5y ago
  Obvious optimization would be to cache chunks locally on every worker nodes.
- remram 5y ago
  Digital Ocean has a limit of 750 requests per IP per second, so you wouldn't be able to complete a search query at all (1800 > 750). https://docs.digitalocean.com/products/spaces/#limits https://docs.digitalocean.com/products/spaces/#limits
fizx 5y ago
It's easy to put a block cache in front of the index, and I'm sure they'll get to it sooner or later. The benefit of using S3 in that case is that unlike e.g. Elastic, your block cache servers don't need replication, and you can tear them down when you're done. You can put them in a true autoscaling group as well.

fulmicoton 5y ago

I tend to agree :). If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. For that kind of use case, I'd probably start using minio. > Seems comparable to AWS ElasticSearch service costs: > - 3 nodes m5.2xlarge.elasticsearch = $1,200 > - 20TB EBS storage = $1,638 Don't forget S3 includes replication. Also EBS throughput (even with SSD) is not good at all. Also our memory footprint is tiny. This is necessary to make it run on two servers. Finally, cpu-wise, our search engine is almost 2x faster than lucene. If you don't believe us, try to replicate our demo on an elastic search :D. Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes.

rossmohax 5y ago
> If we get 1 req/s, even for a dataset of that size, this is not as cost efficient. How many req/s do you have in mind for your system to be a viable option? > Also EBS throughput (even with SSD) is not good at all. It is not worse than S3 still, right? > Chatnoir.eu is the only other common crawl cluster we know of. It consists of 120 nodes. I have no deep ES experience. Are you saying, that to host 6TB of indexed data (before replication) you'd need 120 nodes ES cluster? If so, then reducing it to just 2 nodes is the real sales pitch, not S3 usage :)
klohto 5y ago
What about d3en instances? Clustered, and together with minio you might reach similar performance. Only issue is the inter-region traffic, it would need to be inside the same AZ EDIT: Realizing that d3 has just slow HDD
pcnix 5y ago
Have you checked out the new EBS gp3 disks? Throughout vs cost is much better on those than gp2, and also cheaper than Provisioned IOPS

tpetry 5y ago

I had the same feeling when reading the post. Their remark that they "estimated the cost" to be that low is in my experience a bad signal. Estimating costs on the cloud is really hard, there are so many (hidden) costs you may miss making it a lot more expensive.

heipei 5y ago

For what it's worth, if you want to run ElasticSearch on AWS I would always go with local-NVMe instances from the i3 family, this is also what AWS and Elasticsearch themselves recommend. 4x i3en.2xlarge (64GB / 5TB NVMe) at $449 / month (1yr reserved) is $1796, or $2636 without reservation, but much better performance due to the NVMe drives.

bufferoverflow 5y ago

AWS is almost never cost efficient. Maybe if you stay in their free tier.

arcturus17 5y ago
> AWS is almost never cost efficient. A ridiculous blanket statement, despite the "almost never" cop-out... It is cost-efficient in a wide array of scenarios. Many companies pay for it because they have calculated the different investment scenarios and AWS comes on top of alternatives such as owning the hardware or using competing cloud vendors. I own a consultancy that builds complex web apps and while I appreciate how occasionally a dev has tried to save costs for me by cramming every piece of the stack (web server, cache, db, queue, etc.), in a single Docker image to host in a droplet, I'd much rather pay for separate services, as I consider it cheaper in the long run.
- tjoff 5y ago
  Many companies pay for it because they have spent an inordinate amount of time learning the ecosystem and know of nothing else.
- bufferoverflow 5y ago
  Name an AWS tier that there's no cheaper alternative for. I'm only aware of Glacier, I haven't seen anything cheaper than it. AWS is convenient and reliable, but it's not cheap.
  - weird-eye-issue 5y ago
    Cheap is a whole lot different than cost efficient.
  - arcturus17 5y ago
    Convenience and reliability are deeply related with cost efficiency - that’s the point the example I made was meant to illustrate.
- vagrantJin 5y ago

returningfory2 5y ago

For Digital Ocean object storage, data transfer to/from a Digital Ocean VM is free. You only pay for bytes-at-rest. But it seems S3 doesn't have a similar offering. Data transfer is free between S3 and EC2 instances, but you still pay the per-request charge. I wonder can you factor this into the pricing algorithm.

rossmohax 5y ago
Obvious optimization would be to cache chunks locally on every worker nodes.
remram 5y ago
Digital Ocean has a limit of 750 requests per IP per second, so you wouldn't be able to complete a search query at all (1800 > 750). https://docs.digitalocean.com/products/spaces/#limits https://docs.digitalocean.com/products/spaces/#limits

fizx 5y ago

It's easy to put a block cache in front of the index, and I'm sure they'll get to it sooner or later. The benefit of using S3 in that case is that unlike e.g. Elastic, your block cache servers don't need replication, and you can tear them down when you're done. You can put them in a true autoscaling group as well.