3 ms·

Did you link the wrong script? The script you show runs everything to statistical significance using Chairmarks.@b. Also, I don't understand what the issue wou

Did you link the wrong script? The script you show runs everything to statistical significance using Chairmarks.@b. Also, I don't understand what the issue would be with mixing Python and Julia code in the benchmark script. The Julia side JIT compiles the invocations which we've seen removes pretty much all non-Python overhead and actually makes the resulting SciPy calls faster than doing so from Python itself in many cases, see for example https://github.com/SciML/SciPyDiffEq.jl?tab=readme-ov-file#measuring-overhead https://github.com/SciML/SciPyDiffEq.jl?tab=readme-ov-file#m... where invocations from Julia very handily outperform SciPy+Numba from the Python REPL. Of course, that is a higher order function so it's benefiting from the Julia JIT in other ways as well, but the point is in previous benchmarks we've seen the overhead floor as so low (~100ns IIRC) that it didn't effect benchmarks negatively for Python and actually improved many Python timings in practice. Though it would be good to show in this case what exactly the difference is in order to isolate any potential issue, I would be surprised if it's more than a 100ns overhead for an invocation like this and with 58ms being the benchmark size, that cost is well below the noise floor). Though trying different datasets is of course valid. There's no reason to reject a benchmark just because it doesn't fit into L3 cache, there's many use cases for that. But it does not mean that all use cases are likely to see such a result.

gauge_field 2y ago
[dead]
menaerus 2y ago
If it doesn't fit into the L3 cache then what the benchmark is measuring is not the CPU efficiency of your SIMD approach but also inherently includes different (and complex) effects of L1+L2+L3 cache hierarchy, memory prefetchers and main memory latency. Latency of accessing the RAM is alone ~two to ~three orders of magnitude larger then the latency of executing the single instruction. Per-se this isn't wrong but you may get the skewed picture of your algorithm efficiency because you're measuring multiple things at once, and perhaps the ones you may not be interested in or even knew that they even exist or are included in the numbers of your experiment, e.g. how good the memory prefetcher understands the access patterns of your experiment. For the same reason, I think it's better to avoid mixing many different things into the same benchmark run. I stand corrected wrt statistical significance since I am not familiar with Julia. I missed to understand that part.

gauge_field 2y ago

[dead]

menaerus 2y ago

If it doesn't fit into the L3 cache then what the benchmark is measuring is not the CPU efficiency of your SIMD approach but also inherently includes different (and complex) effects of L1+L2+L3 cache hierarchy, memory prefetchers and main memory latency. Latency of accessing the RAM is alone ~two to ~three orders of magnitude larger then the latency of executing the single instruction. Per-se this isn't wrong but you may get the skewed picture of your algorithm efficiency because you're measuring multiple things at once, and perhaps the ones you may not be interested in or even knew that they even exist or are included in the numbers of your experiment, e.g. how good the memory prefetcher understands the access patterns of your experiment. For the same reason, I think it's better to avoid mixing many different things into the same benchmark run. I stand corrected wrt statistical significance since I am not familiar with Julia. I missed to understand that part.