4 ms·

I know this will be down voted but I've learned to be a bit skeptical of Axboe's monumental claims. There have been some unfair benchmarks posted in the past su

by junon 5y ago

I know this will be down voted but I've learned to be a bit skeptical of Axboe's monumental claims. There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Take these with a grain of salt - his results are notoriously very difficult to reproduce, even on identical hardware. Nevertheless, io_uring is certainly the better design and I'm happy to see progress still being made.

Datagenerator 5y ago
Nitpicking aside he has twenty years of Linux kernel development experience and does great things for everyones benefit. Congratulations Jens Axboe!
- junon 5y ago
  This wasn't nitpicking. It was claiming a nontrivial speedup over epoll for identical test cases. What was the point of your comment?
  - yuffaduffa 5y ago
    What was the point of yours? “Take benchmarks with a grain of salt” is like saying refrigerating food is important or, more appropriately, that it’s important to verify surprising claims. (You made it a personal observation for some reason, but your whole point is still about as insightful as both of those.) The person you’re going after here probably felt compelled to counter the needless personal nature of your remarks. It’s difficult to experimentally verify relativity but we don’t criticize Einstein as a result.
    - junon 5y ago
      Comparing very verifiable, applied science to very theoretical science is a strawman. Science doesn't care about feelings or tenure, so while his experience is relevant, it does not excuse the provably inflated performance figures that have been boasted historically.
marcodiego 5y ago
Not disagreeing with you, but even if benchmarks are unfair, these reports are still a good illustration of progress. Unless there are some drawbacks to these optimizations, there should be no negative side-effects. AFAIK as I know, these changes shouldn't negatively impact tasks that are not making any use of these features. So, even if improvements are not directly proportional to what is being reported, they're still real.
- junon 5y ago
  > Unless there are some drawbacks to these optimizations, there should be no negative side-effects. Historically the benchmarks have been crafted in a way to make them seem faster than their real-world and well-formed opponents (e.g. the io_uring vs epoll benchmark). One issue that was pointed out was that the io_uring benchmark eschewed proper error handling in order to avoid some branches, whereas the epoll benchmark properly error checked. This reduced the benchmark times considerably, though no correction was ever published after the fact. This is why in the Github issue I mentioned in another comment there are some people understandably a bit annoyed. I don't choose to jump to conclusions about Axboe's intent, and my original comment wasn't meant to do so.
- spicybright 5y ago
  Why wouldn't you report more rigorous benchmarks for typical use cases though? Why can't others reproduce it the same?
pengaru 5y ago
> There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Could you provide any links to these discussions?
- junon 5y ago
  Sure. https://github.com/axboe/liburing/issues/189 https://github.com/axboe/liburing/issues/189 Original claims were in the 90% and above performance increase over epoll. Then issues were found, and the figure was adjusted to 60% over epoll. Then more issues were found, and now real-world performance tests are showing minimal speedups if any. Unfortunately the sibling commentors don't see "computer science" as a science but instead as a "feel good hobby", it seems. My point wasn't to hurt feelings, it was to provide a word of caution with these sorts of groundbreaking claims with respect specifically to the io_uring efforts, as they have been disingenuous quite a few times historically. I don't doubt Jens does fantastic work. I don't doubt that he's seen these speedups in very specific cases. But people are celebrating this as a win where they'd be skeptical of e.g. "breakthrough" treatments of cancer (footnote: in mice). It's the same thing.
  - deleted 5y ago
    [deleted]
  - servytor 5y ago
    But if you read the thread, you realize he is testing on a 2011 laptop, and the older architecture may be causing the issues. He reported a 3x speedup running the benchmark on a Raspberry Pi 4 in the same thread[0] Someone claimed that you don't get to see the benefit of io_uring in a hypervisor[1], but they did not provide benchmark results. [0]: https://github.com/axboe/liburing/issues/189#issuecomment-942316843 https://github.com/axboe/liburing/issues/189#issuecomment-94... [1]: https://github.com/axboe/liburing/issues/189#issuecomment-739903710 https://github.com/axboe/liburing/issues/189#issuecomment-73...
  - pengaru 5y ago
    If that's the best example you've got I'm writing your entire comment off as a steaming pile of FUD.

by junon 5y ago

Datagenerator 5y ago

Nitpicking aside he has twenty years of Linux kernel development experience and does great things for everyones benefit. Congratulations Jens Axboe!

junon 5y ago
This wasn't nitpicking. It was claiming a nontrivial speedup over epoll for identical test cases. What was the point of your comment?
- yuffaduffa 5y ago
  What was the point of yours? “Take benchmarks with a grain of salt” is like saying refrigerating food is important or, more appropriately, that it’s important to verify surprising claims. (You made it a personal observation for some reason, but your whole point is still about as insightful as both of those.) The person you’re going after here probably felt compelled to counter the needless personal nature of your remarks. It’s difficult to experimentally verify relativity but we don’t criticize Einstein as a result.
  - junon 5y ago
    Comparing very verifiable, applied science to very theoretical science is a strawman. Science doesn't care about feelings or tenure, so while his experience is relevant, it does not excuse the provably inflated performance figures that have been boasted historically.

marcodiego 5y ago

Not disagreeing with you, but even if benchmarks are unfair, these reports are still a good illustration of progress. Unless there are some drawbacks to these optimizations, there should be no negative side-effects. AFAIK as I know, these changes shouldn't negatively impact tasks that are not making any use of these features. So, even if improvements are not directly proportional to what is being reported, they're still real.

junon 5y ago
> Unless there are some drawbacks to these optimizations, there should be no negative side-effects. Historically the benchmarks have been crafted in a way to make them seem faster than their real-world and well-formed opponents (e.g. the io_uring vs epoll benchmark). One issue that was pointed out was that the io_uring benchmark eschewed proper error handling in order to avoid some branches, whereas the epoll benchmark properly error checked. This reduced the benchmark times considerably, though no correction was ever published after the fact. This is why in the Github issue I mentioned in another comment there are some people understandably a bit annoyed. I don't choose to jump to conclusions about Axboe's intent, and my original comment wasn't meant to do so.
spicybright 5y ago
Why wouldn't you report more rigorous benchmarks for typical use cases though? Why can't others reproduce it the same?

pengaru 5y ago

> There have been some unfair benchmarks posted in the past surrounding io_uring that were called out by people on the liburing repositories. Could you provide any links to these discussions?

junon 5y ago
Sure. https://github.com/axboe/liburing/issues/189 https://github.com/axboe/liburing/issues/189 Original claims were in the 90% and above performance increase over epoll. Then issues were found, and the figure was adjusted to 60% over epoll. Then more issues were found, and now real-world performance tests are showing minimal speedups if any. Unfortunately the sibling commentors don't see "computer science" as a science but instead as a "feel good hobby", it seems. My point wasn't to hurt feelings, it was to provide a word of caution with these sorts of groundbreaking claims with respect specifically to the io_uring efforts, as they have been disingenuous quite a few times historically. I don't doubt Jens does fantastic work. I don't doubt that he's seen these speedups in very specific cases. But people are celebrating this as a win where they'd be skeptical of e.g. "breakthrough" treatments of cancer (footnote: in mice). It's the same thing.
- deleted 5y ago
  [deleted]
- servytor 5y ago
  But if you read the thread, you realize he is testing on a 2011 laptop, and the older architecture may be causing the issues. He reported a 3x speedup running the benchmark on a Raspberry Pi 4 in the same thread[0] Someone claimed that you don't get to see the benefit of io_uring in a hypervisor[1], but they did not provide benchmark results. [0]: https://github.com/axboe/liburing/issues/189#issuecomment-942316843 https://github.com/axboe/liburing/issues/189#issuecomment-94... [1]: https://github.com/axboe/liburing/issues/189#issuecomment-739903710 https://github.com/axboe/liburing/issues/189#issuecomment-73...
- pengaru 5y ago
  If that's the best example you've got I'm writing your entire comment off as a steaming pile of FUD.