10 ms·

>Single-threaded sequential 1KB writes for a total of 4GB, without even oflag=sync? Bonnie++? Sorry, but these are not "good benchmarks to run" at all. They're

by dice 10y ago

>Single-threaded sequential 1KB writes for a total of 4GB, without even oflag=sync? Bonnie++? Sorry, but these are not "good benchmarks to run" at all. They're garbage. People who know storage would never suggest these. As a non-storage person: what should I be using instead of dd and bonnie++?

subway 10y ago
fio is a favorite of mine. https://github.com/axboe/fio https://github.com/axboe/fio
SEJeff 10y ago
fio, maintained by Jens Axboe, the guy who maintains the Linux kernel block layer.
- deleted 10y ago
  [deleted]
Terribledactyl 10y ago
There are two main issues with generalized benchmarking. 1) You (probably) aren't testing the things most done by your application/platform/etc. And worse yet, the hardware/software maybe simply going for big numbers in the common benchmarks. 2) The test may or may not be garbage depending on your use cases.
bcantrill 10y ago
bonnie++ is, as Jeff says, well-known as a canonical example of a benchmark gone horribly wrong; see Brendan Gregg's excellent post on active benchmarking and bonnie++[1] for the gory details. Beyond bonnie++, storage benchmarks are fraught with peril; years ago, I dismembered SPEC SFS as being similarly unsafe at any speed when benchmarking storage systems, albeit for much more subtle reasons than the glaring mechanical flaws in bonnie++.[2] And as for dd, I actually think it's okay as long as you explain clearly what it is (and isn't); Jeff's complaint is that they seem to be treating this single dd invocation as "write performance", when it fact the truth is subtler -- and things like blocksize and synchronicity matter a great deal. More generally, anyone interested in storage benchmarks would be wise to read essentially everything that Brendan Gregg has written on the subject, starting with his five-part (!!) series on file system latency.[3][4][5][6][7] [1] http://www.brendangregg.com/ActiveBenchmarking/bonnie++.html http://www.brendangregg.com/ActiveBenchmarking/bonnie++.html [2] http://dtrace.org/blogs/bmc/2009/02/02/eulogy-for-a-benchmark/ http://dtrace.org/blogs/bmc/2009/02/02/eulogy-for-a-benchmar... [3] http://dtrace.org/blogs/brendan/2011/05/11/file-system-latency-part-1/ http://dtrace.org/blogs/brendan/2011/05/11/file-system-laten... [4] http://dtrace.org/blogs/brendan/2011/05/13/file-system-latency-part-2/ http://dtrace.org/blogs/brendan/2011/05/13/file-system-laten... [5] http://dtrace.org/blogs/brendan/2011/05/18/file-system-latency-part-3/ http://dtrace.org/blogs/brendan/2011/05/18/file-system-laten... [6] http://dtrace.org/blogs/brendan/2011/05/24/file-system-latency-part-4/ http://dtrace.org/blogs/brendan/2011/05/24/file-system-laten... [7] http://dtrace.org/blogs/brendan/2011/06/03/file-system-latency-part-5/ http://dtrace.org/blogs/brendan/2011/06/03/file-system-laten...
- notacoward 10y ago
  +1 for anything Brendan writes or says. Seriously, to those of us who do this stuff, he's idol material.
  - deleted 10y ago
    [deleted]
  - SEJeff 10y ago
    Ditto for Mr Cantrill, one of the fathers of DTrace fwiw
notacoward 10y ago
I usually recommend iozone and fio. Iozone isn't the most powerful tool, but it's simple to use and with the right options can at least provide some useful information. Fio is much more powerful, in terms of the workloads it can generate and the information it can provide, but it's a real PITA to work with. As it turns out, I've given a few presentations on this stuff. Here's one of the more recent ones: https://docs.google.com/presentation/d/1CQgn_fXivdXZ-fcqJ39v09uQUcIeQQMILma9Hd-cJj8/edit#slide=id.p4 https://docs.google.com/presentation/d/1CQgn_fXivdXZ-fcqJ39v...
DeepYogurt 10y ago
CompileBench is a decent one. It's made by the author of btrfs. https://oss.oracle.com/~mason/compilebench/ https://oss.oracle.com/~mason/compilebench/
wildlogic 10y ago
Intel developed tool - http://www.iometer.org/ http://www.iometer.org/
bogomipz 10y ago
Fio. https://github.com/axboe/fio https://github.com/axboe/fio
sijoe 10y ago
TL;DR version: Anything that is not your application won't use the resources in the same way, and will have very different properties (scaling, performance, contention, etc.) Longer: bonnie++ is not a load generator. Really. Start a typical run with a command line test case you find online and you'll see no actual IO. Lots of cache hits. We used to use it, 10 years ago, to play around with load generation, but found that it didn't generate enough IO, or in a way that actually matched what people do. IOzone is marginal ... I wouldn't use it for a serious test, and when people suggest dd or IOzone, I ask them how well the code actually matches their use case. Chances are that it is also largely irrelevant to this. Worse still is that the IOzone throughput measurements are basically bogus, using a naive sum of bandwidths, rather than showing the interesting data (the actual histogram or distribution of performance, including the start/end times, and the rates per thread/process as a function of time). fio is good, in that you can implement many types of tests that have a reasonable chance at being meaningful. I caught Sandforce controllers compressing non-random data on benchmarks they were using for the SSDs that used them with fio. Actual SF performance was lower than spinning rust once you fed it real random data. We wrote something called io-bm (https://gitlab.scalableinformatics.com/joe/io-bm https://gitlab.scalableinformatics.com/joe/io-bm) for pounding on parallel file systems (specifically to stress them and see how they scaled and dealt with contention for network, metadata, etc.). I am not sure if the repo has our histogramming and time series bits in it so we can see individual thread performance as well as overall performance, might be in a private repo. Basically it boils down to the TL;DR above. If its not your code, then you are likely testing with an application that is somewhere between partially to wholly irrelevant to your use case.

by dice 10y ago

subway 10y ago

fio is a favorite of mine. https://github.com/axboe/fio https://github.com/axboe/fio

SEJeff 10y ago

fio, maintained by Jens Axboe, the guy who maintains the Linux kernel block layer.

deleted 10y ago
[deleted]

Terribledactyl 10y ago

There are two main issues with generalized benchmarking. 1) You (probably) aren't testing the things most done by your application/platform/etc. And worse yet, the hardware/software maybe simply going for big numbers in the common benchmarks. 2) The test may or may not be garbage depending on your use cases.

bcantrill 10y ago

bonnie++ is, as Jeff says, well-known as a canonical example of a benchmark gone horribly wrong; see Brendan Gregg's excellent post on active benchmarking and bonnie++[1] for the gory details. Beyond bonnie++, storage benchmarks are fraught with peril; years ago, I dismembered SPEC SFS as being similarly unsafe at any speed when benchmarking storage systems, albeit for much more subtle reasons than the glaring mechanical flaws in bonnie++.[2] And as for dd, I actually think it's okay as long as you explain clearly what it is (and isn't); Jeff's complaint is that they seem to be treating this single dd invocation as "write performance", when it fact the truth is subtler -- and things like blocksize and synchronicity matter a great deal. More generally, anyone interested in storage benchmarks would be wise to read essentially everything that Brendan Gregg has written on the subject, starting with his five-part (!!) series on file system latency.[3][4][5][6][7] [1] http://www.brendangregg.com/ActiveBenchmarking/bonnie++.html http://www.brendangregg.com/ActiveBenchmarking/bonnie++.html [2] http://dtrace.org/blogs/bmc/2009/02/02/eulogy-for-a-benchmark/ http://dtrace.org/blogs/bmc/2009/02/02/eulogy-for-a-benchmar... [3] http://dtrace.org/blogs/brendan/2011/05/11/file-system-latency-part-1/ http://dtrace.org/blogs/brendan/2011/05/11/file-system-laten... [4] http://dtrace.org/blogs/brendan/2011/05/13/file-system-latency-part-2/ http://dtrace.org/blogs/brendan/2011/05/13/file-system-laten... [5] http://dtrace.org/blogs/brendan/2011/05/18/file-system-latency-part-3/ http://dtrace.org/blogs/brendan/2011/05/18/file-system-laten... [6] http://dtrace.org/blogs/brendan/2011/05/24/file-system-latency-part-4/ http://dtrace.org/blogs/brendan/2011/05/24/file-system-laten... [7] http://dtrace.org/blogs/brendan/2011/06/03/file-system-latency-part-5/ http://dtrace.org/blogs/brendan/2011/06/03/file-system-laten...

notacoward 10y ago
+1 for anything Brendan writes or says. Seriously, to those of us who do this stuff, he's idol material.
- deleted 10y ago
  [deleted]
- SEJeff 10y ago
  Ditto for Mr Cantrill, one of the fathers of DTrace fwiw

notacoward 10y ago

I usually recommend iozone and fio. Iozone isn't the most powerful tool, but it's simple to use and with the right options can at least provide some useful information. Fio is much more powerful, in terms of the workloads it can generate and the information it can provide, but it's a real PITA to work with. As it turns out, I've given a few presentations on this stuff. Here's one of the more recent ones: https://docs.google.com/presentation/d/1CQgn_fXivdXZ-fcqJ39v09uQUcIeQQMILma9Hd-cJj8/edit#slide=id.p4 https://docs.google.com/presentation/d/1CQgn_fXivdXZ-fcqJ39v...

DeepYogurt 10y ago

CompileBench is a decent one. It's made by the author of btrfs. https://oss.oracle.com/~mason/compilebench/ https://oss.oracle.com/~mason/compilebench/

wildlogic 10y ago

Intel developed tool - http://www.iometer.org/ http://www.iometer.org/

bogomipz 10y ago

Fio. https://github.com/axboe/fio https://github.com/axboe/fio

sijoe 10y ago

TL;DR version: Anything that is not your application won't use the resources in the same way, and will have very different properties (scaling, performance, contention, etc.) Longer: bonnie++ is not a load generator. Really. Start a typical run with a command line test case you find online and you'll see no actual IO. Lots of cache hits. We used to use it, 10 years ago, to play around with load generation, but found that it didn't generate enough IO, or in a way that actually matched what people do. IOzone is marginal ... I wouldn't use it for a serious test, and when people suggest dd or IOzone, I ask them how well the code actually matches their use case. Chances are that it is also largely irrelevant to this. Worse still is that the IOzone throughput measurements are basically bogus, using a naive sum of bandwidths, rather than showing the interesting data (the actual histogram or distribution of performance, including the start/end times, and the rates per thread/process as a function of time). fio is good, in that you can implement many types of tests that have a reasonable chance at being meaningful. I caught Sandforce controllers compressing non-random data on benchmarks they were using for the SSDs that used them with fio. Actual SF performance was lower than spinning rust once you fed it real random data. We wrote something called io-bm (https://gitlab.scalableinformatics.com/joe/io-bm https://gitlab.scalableinformatics.com/joe/io-bm) for pounding on parallel file systems (specifically to stress them and see how they scaled and dealt with contention for network, metadata, etc.). I am not sure if the repo has our histogramming and time series bits in it so we can see individual thread performance as well as overall performance, might be in a private repo. Basically it boils down to the TL;DR above. If its not your code, then you are likely testing with an application that is somewhere between partially to wholly irrelevant to your use case.