5 ms·
Oh wow, I had no idea that write buffers needed to be on the order of a megabyte on modern systems.
by im3w1l 10y ago
Oh wow, I had no idea that write buffers needed to be on the order of a megabyte on modern systems.
- mpweiher 10y agoNeither did I! And in fact it took almost 2 days to question/overcome my assumptions. "It's not what you don't know that kills you. It is what you know for sure that ain't true" -- Mark Twain
- danieldk 10y agoInteresting... You never dd data to a USB3 disk or SSD? The default buffer size for dd on OS X is also 512 bytes. I've been using `bs=1m` on OS X for ages.
- sz4kerto 10y agoWhy are you using dd for USB disks?
- danieldk 10y agoWriting bootable installation images?
- chungy 10y agoYou could use cp.
- feld 10y agoExplain?
- jsjohnst 10y agocp is designed to copy data across file systems. dd is designed to copy raw block data across devices. Sure, you can abuse both for either purpose, but seems rather misguided to call it out explicitly. What did you intend to gain by doing it?
- chungy 10y agocp is designed to copy data from one file to another. Block devices are represented as files, and cp won't be speed limited by block size. dd is also designed to copy data from one file to another, in blocks, with optional conversion of the data (eg, from EBCDIC to ASCII), and with optional offsets/limits to copy regions of a file. dd is not designed to copy raw block data, it is merely a different file copy tool.
- sz4kerto 10y agoYou don't know what cp is for. cp is exactly for copying data from a file to another file -- and under Unix, devices are represented as files. Using cp for writing disk ISOs to an USB drive is proper use, not abuse. dd is for transform-and-copy, so you're abusing dd when writing disk ISOs. The fact that most of the Internet is wrong doesn't make you any less wrong.
- sz4kerto 10y agoYou should use cp, that's just better for this task. "dd is not a disk writing tool" http://www.vidarholen.net/contents/blog/?p=479 http://www.vidarholen.net/contents/blog/?p=479
- fulafel 10y agoIt's just OS X being slow, Linux gets close to peak with a 16 kB buffer: http://blog.tdg5.com/tuning-dd-block-size/ http://blog.tdg5.com/tuning-dd-block-size/
- mpweiher 10y agoMaybe I am reading it wrong, but I am not getting the 16KB optimal buffer size from that article. The summary at the end is as follows: 'In the above example it can be seen that an input block size of 128K is optimal for my particular setup.' However, the SSD seems to be significantly slower, so the relative overhead of the syscall is going to be respectively less: 'Reading from /dev/zero and writing out to a SSD with a more optimal block size of 256K yields a throughput of 280 MB/s.' I was getting 250MB/s with the 512 byte buffer size, although the test in the article is doing two syscalls (one read(), one write()) whereas mkfile is just doing one. At 256KB buffer size I was already getting around 1.8GB/second writing to SSD.
- feld 10y agoOr is it another case of Linux taking shortcuts and sacrificing safety for speed?
- alain94040 10y agoReading from /dev/zero and writing out to RAM with a more optimal block size of 256K yields a throughput of 1,536 MB/s Actually, in this article, Linux writing to RAM is slower than MacOS writing to SSD. Hard to conclude that "it's just OS X being slow". I suspect the hardware used is wildly different from the one in the original article.
- ivoras 10y agoJust saw the performance of this "consumer" device and remembered this thread: http://www.anandtech.com/show/11104/the-patriot-hellfire-m2-480gb-review-phison-nvme-tested http://www.anandtech.com/show/11104/the-patriot-hellfire-m2-... . This SSD can write over a GB/s and read nearly 3 GB/s. We've come a long way since 32 KiB buffers were a sane choice.