9 ms·
Mkfile(8) is severely syscall limited on OS X
- deleted 10y ago[deleted]
- im3w1l 10y agoOh wow, I had no idea that write buffers needed to be on the order of a megabyte on modern systems.
- mpweiher 10y agoNeither did I! And in fact it took almost 2 days to question/overcome my assumptions. "It's not what you don't know that kills you. It is what you know for sure that ain't true" -- Mark Twain
- danieldk 10y agoInteresting... You never dd data to a USB3 disk or SSD? The default buffer size for dd on OS X is also 512 bytes. I've been using `bs=1m` on OS X for ages.
- sz4kerto 10y agoWhy are you using dd for USB disks?
- danieldk 10y agoWriting bootable installation images?
- chungy 10y agoYou could use cp.
- feld 10y agoExplain?
- jsjohnst 10y agocp is designed to copy data across file systems. dd is designed to copy raw block data across devices. Sure, you can abuse both for either purpose, but seems rather misguided to call it out explicitly. What did you intend to gain by doing it?
- chungy 10y agocp is designed to copy data from one file to another. Block devices are represented as files, and cp won't be speed limited by block size. dd is also designed to copy data from one file to another, in blocks, with optional conversion of the data (eg, from EBCDIC to ASCII), and with optional offsets/limits to copy regions of a file. dd is not designed to copy raw block data, it is merely a different file copy tool.
- sz4kerto 10y agoYou don't know what cp is for. cp is exactly for copying data from a file to another file -- and under Unix, devices are represented as files. Using cp for writing disk ISOs to an USB drive is proper use, not abuse. dd is for transform-and-copy, so you're abusing dd when writing disk ISOs. The fact that most of the Internet is wrong doesn't make you any less wrong.
- sz4kerto 10y agoYou should use cp, that's just better for this task. "dd is not a disk writing tool" http://www.vidarholen.net/contents/blog/?p=479 http://www.vidarholen.net/contents/blog/?p=479
- fulafel 10y agoIt's just OS X being slow, Linux gets close to peak with a 16 kB buffer: http://blog.tdg5.com/tuning-dd-block-size/ http://blog.tdg5.com/tuning-dd-block-size/
- mpweiher 10y agoMaybe I am reading it wrong, but I am not getting the 16KB optimal buffer size from that article. The summary at the end is as follows: 'In the above example it can be seen that an input block size of 128K is optimal for my particular setup.' However, the SSD seems to be significantly slower, so the relative overhead of the syscall is going to be respectively less: 'Reading from /dev/zero and writing out to a SSD with a more optimal block size of 256K yields a throughput of 280 MB/s.' I was getting 250MB/s with the 512 byte buffer size, although the test in the article is doing two syscalls (one read(), one write()) whereas mkfile is just doing one. At 256KB buffer size I was already getting around 1.8GB/second writing to SSD.
- feld 10y agoOr is it another case of Linux taking shortcuts and sacrificing safety for speed?
- alain94040 10y agoReading from /dev/zero and writing out to RAM with a more optimal block size of 256K yields a throughput of 1,536 MB/s Actually, in this article, Linux writing to RAM is slower than MacOS writing to SSD. Hard to conclude that "it's just OS X being slow". I suspect the hardware used is wildly different from the one in the original article.
- ivoras 10y agoJust saw the performance of this "consumer" device and remembered this thread: http://www.anandtech.com/show/11104/the-patriot-hellfire-m2-480gb-review-phison-nvme-tested http://www.anandtech.com/show/11104/the-patriot-hellfire-m2-... . This SSD can write over a GB/s and read nearly 3 GB/s. We've come a long way since 32 KiB buffers were a sane choice.
- amelius 10y agoSo is this any different on Linux? Why is OSX in the title? Are syscalls more expensive on OSX?
- mpweiher 10y agoGood point, the reason is simple: I didn't check on Linux (and have updated the post to reflect this). I am guessing it would be similar, but don't have any data.
- throwawayish 10y agoIt is.
- akandiah 10y ago> So is this any different on Linux? Why is OSX in the title? mkfile is not available on Linux. The equivalent utility is xfs_mkfile or fallocate.
- throwawayish 10y agoNo, mkfile creates a file with written extents (no sparse or uninitialized areas), while fallocate will (with ext4/xfs/...) only map some extents which are marked uninitialized. So while mkfile 10G would have to write ~10 GB of data, fallocate 10G is nearly instanteneous, because it most likely just allocated a single extent. xfs_mkfile could do what mkfile does, the description isn't conclusive enough.
- gcr 10y agoYou can use good old `dd` for this: dd if=/dev/zero of=test bs=1k seek=2m count=1 Or python: python -c 'f=open("test", "w"); f.seek(2e9); f.write("\x00")' Both these calls create a ~2GB file with holes in it. They return almost instantly on OSX. EDIT: oh wait, this isn't what you want. you want a file _without_ holes, my mistake.
- pedrocr 10y agoThe graph could use an X axis label. I'm assuming it's "Buffer size in kB". It would also be nice to include the datapoint you started with (250MB/s from a 512byte buffer).
- masklinn 10y agoThat's actually written below the graph, rather than as a graph label > X-Axis is buffer size in KB. The original 512 byte size isn't on there because it would be 0.5KB or the entire axis would need to be bytes, which would also be awkward at the larger sizes. Also note that the X-Axis is logarithmic.
- Retr0spectrum 10y agoFurthermore, the X and Y axis should really have been the other way around.
- mpweiher 10y agoThough I thought it was fairly obvious from the text this is still a good point. Added as comment underneath.
- Matt3o12_ 10y agoThis still begs the question I've had since day one when learning about buffers: what the heck is the recommended buffer size. I've seen a lot of old code that use extremely small buffer size as well as some recent code that use extremely high buffer sizes (up to 10MB). What is a good sweet spot that runs well on older hardware (less then 10 years old) and new hardware? And should network buffers be bigger or smaller then disk buffers?
- RyanZAG 10y agoJudging by the post here, 512kb buffers seems like a good bet?
- mpweiher 10y agoFor a an SSD with 2GB/s throughput and the syscall overhead of OS X: yes. Go higher if you want some headroom for the next generation of faster SSDs.
- throwawayish 10y agoNetwork buffers are a completely different animal, also harder to optimize... Disk IO buffers, it's easy nowadays, just use something like a MB, which is just fine for almost any application, and doesn't stack up to much memory use (unless you're writing many files concurrently, which can bring it's own problems as well)
- yjftsjthsd-h 10y agoThat actually does sound about right, but do you know how to derive that? I'm curious how to figure it out.
- gens 10y agoJust do a test with dd or some small piece of code. Why rely on heresay.
- dom0 10y agoAnd that's why sane people will tell you to not use something like st_blksize as the IO size, because that's far to small. Many tools use 32 KiB, but that can be limiting even on Linux and needlessly uses more CPU (especially if FUSE is involved -- reads are not coalesced at the VFS layer! Read and write merging is done at the block device layer by the IO scheduler there!). Something like 512k-1M is a sane default these days.
- gens 10y agoMaybe i'm misunderstanding you, but.. Before every one a "echo 3 > /proc/sys/vm/drop_caches" is executed. Processor is in "performance" mode. dd if=./big_file.file of=/dev/null bs=64K 535020540 bytes (535 MB, 510 MiB) copied, 8.70128 s, 61.5 MB/s dd if=./big_file.file of=/dev/null bs=4K 535020540 bytes (535 MB, 510 MiB) copied, 8.36283 s, 64.0 MB/s dd if=./big_file.file of=/dev/null bs=1K 535020540 bytes (535 MB, 510 MiB) copied, 8.82508 s, 60.6 MB/s Only thing gone up is the cpu usage (same is negligible at ~60MB/s). dd if=/dev/zero of=/dev/null bs=1K count=1024K 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.06113 s, 1.0 GB/s dd if=/dev/zero of=/dev/null bs=4K count=256K 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.343595 s, 3.1 GB/s dd if=/dev/zero of=/dev/null bs=32K count=32K 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.160191 s, 6.7 GB/s
- brobinson 10y agoDon't forget "oflag=direct"
- dom0 10y agoUhm, does dd contain all the logic to work with that or is that literally just O_DIRECT |ed into the flags?
- dom0 10y agoAt just 60 MB/s it doesn't matter much (even 512 byte blocks would just be a little more than 100k syscalls / second). At a couple hundred MB/s it begins to matter, and with faster NVMe SSDs it's a legit issue. This becomes a bigger issue if it's not a kernel-based, local FS, but something that does more work, I mentioned FUSE above. Since no read merging happens, small reads like 4k or 32k start to hurt earlier, especially if the FS is written in an interpreter. Some of these also mishandled small reads, incurring an additional penalty (I know because I fixed that in an implementation).
- JelteF 10y agoInteresting post, but this xkcd is really relevant here: https://xkcd.com/833/ https://xkcd.com/833/
- mkj 10y agoIt's a command, not a syscall!
- drewg123 10y ago" I did not check on other operating systems, but my guess is that the results would be similar." Actually, no. Irrespective of mkfile being mostly MacOSX specific, most sycalls on MacOSX are just plain slow as compared to Linux (or FreeBSD). I think this is partially an artifact of performance not being a major metric for most system calls on MacOSX. The systems calls where performance is a critical metric (like timekeeping) use a kernel / user shared page interface to avoid the syscall entirely in the vast majority of the cases. I recall doing benchmarking roughly 10 years ago to find the best interface to communicate with a mostly in an OS-bypass HPC driver. MacOSX ioctls were some multipler more expensive than Linux, but the native IOKit Mach IPC was even more expensive than that. Sigh. There was a similar story for sockets in non-OS bypass mode, where simply writing on a socket was far more expensive than Linux. Somebody needs to resurrect lmbench & do a comparison of the various x86 kernels available these days. Maybe that would shame Apple into focusing on performance. Drew
- developer2 10y ago>> most syscalls on MacOSX are just plain slow as compared to Linux (or FreeBSD). I think this is partially an artifact of performance not being a major metric for most system calls on MacOSX This. When flat out milliseconds per syscall is deemed to trump actual functionality and usability of the operating system as a whole, your priorities are simply wrong. Nerding out over obsessive metrics is literally that - nerding out over useless micro-details rather than real-world needs. When your loyalties fall solely upon a race-to-the-bottom speed on POSIX standards that are 40 years old... sigh. We come across this kind of thing whenever someone releases a new pre-alpha, first-time-endeavour, operating system. Arguments like "It's not 100% POSIX!". Excuse me, but if you think that POSIX is the only valid foundation for an operating system, you're pushing your previous experience as the only facet that matters, rather than any actual measure of quality or merit. The ability to easily port existing applications is not the only metric that is going to dictate the operating system of the future. POSIX is so ancient at this point that it should have been iterated upon a multitude of times by now. It's so out of date it's almost a joke - we're basically still building our operating systems with horses, without even considering that a mechanical or electrical engine is even a possibility. For 99.9% of use cases, this kind of metric is so far beyond worth even mentioning that it sounds like a farce. Nobody cares about the performance of mkfile. Nobody. If you're one of the very few people wasting your energy on such a thing... holy shit... get over it and modernize your way of thinking. We're not living in 1980; and no, it's not that the "reasonable old ways no longer gain any respect or recognition" - it's that you are trying to live in the past. Move on.
- raverbashing 10y agoWell, there's this option (which looks like it would be the way to go for its purpose): -n Create an empty filename. The size is noted, but disk blocks aren't allocated until data is written to them Also the description on the man file says: mkfile creates one or more files that are suitable for use as NFS-mounted swap areas. The sticky bit is set, and the file is padded with zeroes by default
- grigjd3 10y agoThat's missing the point. The author wanted to test the write speed and found that mkfile was not a good tool to test with.
- rollthehard6 10y agoSome years ago I picked up the habit from a predecessor of testing such things with dd instead, that way you can experiment with the effect of different block sizes, so like - dd of=/dev/zero of=/ddtest.out myfile bs=64k count=65536
- mcguire 10y agoDoes anyone know offhand the architecture of the Mach/BSD hybrid kernel in this case? It sounds suspiciously like a problem with the IBM Microkernel back in about 1992, and with the OSF's mkLinux, which is apparently a predecessor of Mac OS. Specifically, do the syscalls writing the file go through multiple protection domains? https://maniagnosis.crsr.net/2011/07/this-is-comment-i-made-on-gluster-blog.html https://maniagnosis.crsr.net/2011/07/this-is-comment-i-made-...
- Someone 10y ago"the OSF's mkLinux, which is apparently a predecessor of Mac OS." Apple worked on MkLinux, but it isn't technically a predecessor to Mac OS X. The two do not share a single line of code; if they did, Apple would have to license Mac OS under the GPL. XNU, the Mac OS kernel (https://en.wikipedia.org/wiki/XNU https://en.wikipedia.org/wiki/XNU) isn't a real microkernel; the functionality of a microkernel is there, but quite a bit of code was added that, in a true microkernel, would live in userspace.
- gwu78 10y agoWhile HN is discussing XNU sycalls, anyone know why the most fundamental of syscalls, execve, according to NeXT/Apple's post-UNIX wisdom needs to have an extra "char *apple[]" argument vector? Not to imply that it is "hidden" but I am curious if any HN users know about this and understand its purpose?
- seba_dos1 10y agohttps://unixjunkie.blogspot.com/2006/02/char-apple-argument-vector.html https://unixjunkie.blogspot.com/2006/02/char-apple-argument-...
- msbarnett 10y agoWhen in doubt, consult the source: https://github.com/opensource-apple/xnu/blob/53c5e2e62fc4182595609153d4b99648da577c39/bsd/kern/kern_exec.c#L4131 https://github.com/opensource-apple/xnu/blob/53c5e2e62fc4182... The upshot is that it looks like it constructs some non-env variable environment data on the program's stack after the posix environment. The absolute path of the executable, followed by preemption free zone addresses, entropy, a configuration setting for malloc allocation strategy, and the address of the main thread's stack afaict.
- pixelbeat__ 10y agoThe GNU coreutils default to 128KiB buffers as per the test script at https://github.com/coreutils/coreutils/blob/master/src/ioblksize.h#L23 https://github.com/coreutils/coreutils/blob/master/src/ioblk...
- snorrah 10y agoIs this something that would be affected by file system defaults? I see it writes in 512 byte chunks by default, when MacOS moves to APFS, is this something that might be altered as a result and thus get a 'free' speedup?