3 ms·

Thanks for sharing the name of that problem! I've encountered it before while optimizing batched LLM inference. The whole batch would last until all queries in

Thanks for sharing the name of that problem! I've encountered it before while optimizing batched LLM inference. The whole batch would last until all queries in a batch were done, and by changing the batch size, you'd trade off per-query-speed (better in a larger batch) with overall performance (worse with a larger batch). Nowadays I think this is solved in an entirely different way, though.

deleted 1y ago
[deleted]
srean 1y ago
It gets more entertaining. It's common to wrap API calls with retry on failure, or spawn an identical request if taking longer than x,or recursively spawn an identical request if taking longer than x,or retry on failure but no more than k times. All of these and similar patterns/decorators can be analysed using the same idea.
- incognito124 1y ago
  Oh wow, pretty cool stuff! If you have more to share, you can always dump it in my mail inbox
  - silloncito 1y ago
    You should be careful with your estimation. The events should be independent to apply those properties but it is very common that one cause can influence many factors, so they are not independent and all the beauty math does not work as with independence. In the worst day, all fail, because one resource can block others and the system get strangled. There is the black swan book when one rare event make the financial market realize what is risk.
    - srean 1y ago
      A load-balancer transparently sitting in front of the api end-point (not an uncommon scenario) usually decouples things well enough to be practically independent. That said, silloncito's warning does need to be paid heed. While independence is essential for the proof to go through, the relationships need not break catastrophically with break of independence, usually it is graceful degradation with degree of independence. There are however specific, often degenerate, theoretical edge cases where the degradation is rapid.

deleted 1y ago

[deleted]

srean 1y ago

It gets more entertaining. It's common to wrap API calls with retry on failure, or spawn an identical request if taking longer than x,or recursively spawn an identical request if taking longer than x,or retry on failure but no more than k times. All of these and similar patterns/decorators can be analysed using the same idea.

incognito124 1y ago
Oh wow, pretty cool stuff! If you have more to share, you can always dump it in my mail inbox
- silloncito 1y ago
  You should be careful with your estimation. The events should be independent to apply those properties but it is very common that one cause can influence many factors, so they are not independent and all the beauty math does not work as with independence. In the worst day, all fail, because one resource can block others and the system get strangled. There is the black swan book when one rare event make the financial market realize what is risk.
  - srean 1y ago
    A load-balancer transparently sitting in front of the api end-point (not an uncommon scenario) usually decouples things well enough to be practically independent. That said, silloncito's warning does need to be paid heed. While independence is essential for the proof to go through, the relationships need not break catastrophically with break of independence, usually it is graceful degradation with degree of independence. There are however specific, often degenerate, theoretical edge cases where the degradation is rapid.