2 ms·
Well with coreutils like wc etc. you get low level control over such things, but with postgres there would be less flexibility. For illustration one could use G
by pixelbeat 13y ago
Well with coreutils like wc etc. you get low level control over such things, but with postgres there would be less flexibility. For illustration one could use GNU dd to take advantage of the page cache only for readahead purposes like:
dd if=clickstream.csv.1 iflag=nocache bs=1M | wc -l
A more common technique is to bypass the page cache altogether and is often use to avoid the many unfortunate characteristics of the current Linux VM. This is done usually with directIO:
dd if=clickstream.csv.1 iflag=direct bs=1M | wc -l
Now postgres might be able to use directIO as an option?
Another related problem with too much caching when writing
to slow device can be seen in this thread:
http://thread.gmane.org/gmane.linux.kernel.mm/108708 http://thread.gmane.org/gmane.linux.kernel.mm/108708
That thread actually describes two problems.
1. That Linux waits too long before writing
2. When it does write large amounts to a slow device it locks out everything else