3 ms·

It cost $250,000 to do this run but it feels worth it.

handfuloflight 24d ago
I did not know men could build such things.
whalesalad 24d ago
I estimated the cluster to achieve this was ~$3-4k per-hour. I am thinking there is a typo on the r8g.16xlarge and they are actually r8gd.16xlarge (notice the d) which comes with directly attached nvme disks.
- svuiv 24d ago
  We used r8g.16xlarge instances with EBS disks, no nvmes
  - whalesalad 24d ago
    would love to hear more about the ebs volumes, iops/size/raid configuration
    - svuiv 24d ago
      each shard had a 4TiB volume with 65k IOPS and 1,500 mbps of throughput
- rcrowley 24d ago
  It felt rude to take so many r8gd instances away from our customers who really love those (and i8g and i8ge).
jeffbee 24d ago
That's roughly 250x more than it would cost to perform this stunt using on-demand Cloud Bigtable, if my math checks out (~1150 nodes @ 85¢/hour for 1h).
- dataviz1000 24d ago
  > 512 shards, each with one Postgres primary each on an r8g.16xlarge > 480 Neki routers, each on its own 8xlarge instance > We sustained 118,538,803 QPS for 16 minutes across 512 shards and 1.22 PiB of data. Our largest recording was 118,747,267. Component Detail Monthly Hourly 16-min burst --------------------------------------------------------------------------------------------- Shard compute 512x r8g.16xlarge $1.41M $1,930 $515 Router compute 480x r8g.8xlarge* $661K $905 $241 Storage (gp3 floor) 1.22 PiB @ $0.08/GB-mo $102K $140 $37 Storage (io2 floor) 1.22 PiB @ $0.125/GB-mo $160K $219 $58 IOPS (io2, light) 5K IOPS/shard, tiered rate $166K $228 $61 IOPS (io2, medium) 20K IOPS/shard, tiered rate $666K $912 $243 IOPS (io2, worst-case) 231,517 IOPS/shard (0% cache) $4.56M $6,251 $1,667 --------------------------------------------------------------------------------------------- Total (gp3 floor) $2.17M $2,975 $793 Total (io2 floor) $2.23M $3,054 $814 Total (io2 + light IOPS) $2.40M $3,282 $875 Total (io2 + medium IOPS) $2.90M $3,966 $1,058 Total (io2 + worst-case IOPS) $6.79M $9,305 $2,481
  - jeffbee 23d ago
    Sounds more right. Where did $250k come from?
    - fizx 23d ago
      I think Sam is an exec at the company. Presumably it took more than one attempt, or there's some humanpower in the budget.
      - deleted 23d ago
        [deleted]

I did not know men could build such things.

I estimated the cluster to achieve this was ~$3-4k per-hour. I am thinking there is a typo on the r8g.16xlarge and they are actually r8gd.16xlarge (notice the d) which comes with directly attached nvme disks.

svuiv 24d ago
We used r8g.16xlarge instances with EBS disks, no nvmes
- whalesalad 24d ago
  would love to hear more about the ebs volumes, iops/size/raid configuration
  - svuiv 24d ago
    each shard had a 4TiB volume with 65k IOPS and 1,500 mbps of throughput
rcrowley 24d ago
It felt rude to take so many r8gd instances away from our customers who really love those (and i8g and i8ge).

jeffbee 24d ago

That's roughly 250x more than it would cost to perform this stunt using on-demand Cloud Bigtable, if my math checks out (~1150 nodes @ 85¢/hour for 1h).

dataviz1000 24d ago
> 512 shards, each with one Postgres primary each on an r8g.16xlarge > 480 Neki routers, each on its own 8xlarge instance > We sustained 118,538,803 QPS for 16 minutes across 512 shards and 1.22 PiB of data. Our largest recording was 118,747,267. Component Detail Monthly Hourly 16-min burst --------------------------------------------------------------------------------------------- Shard compute 512x r8g.16xlarge $1.41M $1,930 $515 Router compute 480x r8g.8xlarge* $661K $905 $241 Storage (gp3 floor) 1.22 PiB @ $0.08/GB-mo $102K $140 $37 Storage (io2 floor) 1.22 PiB @ $0.125/GB-mo $160K $219 $58 IOPS (io2, light) 5K IOPS/shard, tiered rate $166K $228 $61 IOPS (io2, medium) 20K IOPS/shard, tiered rate $666K $912 $243 IOPS (io2, worst-case) 231,517 IOPS/shard (0% cache) $4.56M $6,251 $1,667 --------------------------------------------------------------------------------------------- Total (gp3 floor) $2.17M $2,975 $793 Total (io2 floor) $2.23M $3,054 $814 Total (io2 + light IOPS) $2.40M $3,282 $875 Total (io2 + medium IOPS) $2.90M $3,966 $1,058 Total (io2 + worst-case IOPS) $6.79M $9,305 $2,481
- jeffbee 23d ago
  Sounds more right. Where did $250k come from?
  - fizx 23d ago
    I think Sam is an exec at the company. Presumably it took more than one attempt, or there's some humanpower in the budget.
    - deleted 23d ago
      [deleted]