4 ms·

Let's do some math and take their metrics and divide by their number of servers: - 6.3 million feed updates per day per server - 111 API requests per minute p

Let's do some math and take their metrics and divide by their number of servers: - 6.3 million feed updates per day per server - 111 API requests per minute per server If you guess 4 cores per server it even sounds worse. (Ps, will HN ever support better formatting?)

latch 9y ago
They measured their response time using average. Collecting metrics to look good. Since we can only speak in broad terms, those requests/updates aren't going to be evenly distributed throughout the day and they might not be consistent from day to day so some over-provisioning is reasonable. Since they're on EC2 a) they shouldn't be that over-provisioned though and b) are getting comparatively (to bare metal) awful performance (say 2x-8x depending on exactly what they're doing) and paying for the privilege. I'd like to better understand why "Fanout-on-write" wasn't their go-to solution. All they said was that it was "expensive".
- tschellenbach 9y ago
  We started out with fanout on write but it was tricky. Fanout on write can get incredibly expensive. Customers would implement things like featured accounts, or topics that you can follow. Basically feeds that are followed by a significant percentage of your userbase and update all the time. Companies like Twitter and Instagram control their product and can avoid some of these pitfalls. A full fanout on write approach means you need to enforce certain product limitations, have a large budget and accept that feed propagation is not always as fast.
- xstartup 9y ago
  How does fan out write help?
  - latch 9y ago
    Lets you shard and makes your reads dead simple (1 table, 1 predicate, 1 sort)
tbarbugli 9y ago
There are more factors to take into account than plain API / feed update traffic when looking at server count: - High Availability requires 2x or 3x more servers - We run our own monitoring, log collection and tracing infrastructure - ML/Analytics features - Distribution of traffic - Disaster Recovery Edit: fix list formatting

latch 9y ago

They measured their response time using average. Collecting metrics to look good. Since we can only speak in broad terms, those requests/updates aren't going to be evenly distributed throughout the day and they might not be consistent from day to day so some over-provisioning is reasonable. Since they're on EC2 a) they shouldn't be that over-provisioned though and b) are getting comparatively (to bare metal) awful performance (say 2x-8x depending on exactly what they're doing) and paying for the privilege. I'd like to better understand why "Fanout-on-write" wasn't their go-to solution. All they said was that it was "expensive".

tschellenbach 9y ago
We started out with fanout on write but it was tricky. Fanout on write can get incredibly expensive. Customers would implement things like featured accounts, or topics that you can follow. Basically feeds that are followed by a significant percentage of your userbase and update all the time. Companies like Twitter and Instagram control their product and can avoid some of these pitfalls. A full fanout on write approach means you need to enforce certain product limitations, have a large budget and accept that feed propagation is not always as fast.
xstartup 9y ago
How does fan out write help?
- latch 9y ago
  Lets you shard and makes your reads dead simple (1 table, 1 predicate, 1 sort)

tbarbugli 9y ago

There are more factors to take into account than plain API / feed update traffic when looking at server count: - High Availability requires 2x or 3x more servers - We run our own monitoring, log collection and tracing infrastructure - ML/Analytics features - Distribution of traffic - Disaster Recovery Edit: fix list formatting