4 ms·
In many applications I can get 10x faster I/O using stdio.h just by radically increasing the buffer size. It turns out that less you talk to the kernel, faster
by nabla9 6y ago
In many applications I can get 10x faster I/O using stdio.h just by radically increasing the buffer size. It turns out that less you talk to the kernel, faster you are.
/* typically 16 to 32 times
stat.st_blksize or BUFSIZ
is good */
int new_size = 16 * 4096;
setvbuf(fp, NULL, _IOFBF, new_size);
- michaelcampbell 6y agoI can't remember how many times I've suggested that people use the optional pre-sizing parameter to Java's StringBuffer class constructor in code reviews. Many times the size of the final string is known absolutely, or at least close enough to pre-allocate. IIRC, the default initial size is 16 bytes which is absurdly low.
- underdeserver 6y agoIt's a shame there's no way to globally set the default StringBuffer size. I bet if you optimize it just by dumb binary search, on your average project you could save on both runtime and memory usage.
- grandinj 6y agoThe default size is in fact optimally sized for "normal" Java applications, which frequently do very little with string buffers, or use them for small tokens or numbers. Source: mailing list discussion with one of the Sun devs.
- jrpelkonen 6y agoI completely agree on the buffer size statement, but at the same time, find it interesting that you would use StringBuffer enough to make that comment. I personally haven’t found any compelling use cases for it since Java 1.5 introduced StringBuilder. Would you like to share some ways you find StringBuffer useful?
- cogman10 6y agoYou shouldn't use StringBuffer. The reason it is slow is because it is "thread safe". 99% of the time, you don't really want that (especially in cases where you know how big the string will be at the end of everything). StringBuilder should be preferred for pretty much everything.
- hyperman1 6y agoWhile I agree with the conclusion to strongly prefer StringBuilder, in most cases the difference is zero: The JVM will prove no other thread can access the object, so it remooves the locks
- cogman10 6y agoI'm not sure if I agree about most cases. It is fairly trivial to come up with code that goes down a sad path (you just need to defeat the Escape analysis). That happens all the time. The JVM's Escape analysis isn't perfect.
- fpoling 6y agoIs it 64K that made the difference on your system? From benchmarking on Linux few years ago the gains after 16k buffer were in the noise.
- nabla9 6y agoSomething like decade ago it did at least in one system.
- pdimitar 6y agoNowadays anywhere between 512KB and 1MB on my computers does make huge difference as opposed to going with default buffer sizes which are usually 4KB - 16KB.
- hyperman1 6y agoSwitching to kernel has become much more expensive thanks to recent security bugs, though
- specialist 6y agoThat indeed does work great. Potentially at the expense of trashing the heap. I once wrote my own JSON parser in Java. Mostly because I couldn't abide by how hard serialization was with the popular libs. I just wanted JSON to HashMap and back. Then I got nutty trying to hyperoptimize it. Stuff like resizing methods so that JIT inlining could work. At the time Boon was the benchmark leader. Its trick was preallocating a huge buffer per parse (and as well as taking shortcuts with the spec, accepting invalid JSON). Which I deemed wasteful. I had an HTTP server and I wanted each client connection to be lighter vs heavier. And for some reason I was committed to "stream processing" (incremental pull parser). I could never match Boon's performance. Close. But never quite catching it. Which offended me in some way. More simple should always win, right? Some time later I forced myself to crawl out of the rabbit hole. I backed out all my goofy optimizations. Opting for the most simple implementation I could conceive. Given my use case, my parser was more than good enough. It's fun to tinker now and again. If only to relearn the "don't preoptimize" lesson.
- anon946 6y agoIf much of your I/O is FP, however, that will still be a significant bottleneck. FP <-> ASCII is inherently slow.
- hinkley 6y agoSomeone else was complaining about this library using 50k of extra space for some operations. I wonder what the benchmark would look like if you used that for buffer.