4 ms·

Tup also relies on FUSE. Do you know if Ekam does as well?

by kolev 12y ago

kentonv 12y ago
Ekam uses LD_PRELOAD to intercept system calls. I considered FUSE but it would have involved considerably more overhead and synchronization issues. - The LD_PRELOAD trick simply rewrites filenames on open() and similar calls, allowing subsequent I/O to happen directly. FUSE unfortunately requires going through the userspace FUSE server for every read or write operation. - FUSE would require either mounting a new FS on-the-fly for each build step (probably inefficient) or using a pool of FUSEs where each one is reused for multiple commands (requires carefully making sure all state is purged in between). - Something would have to be done to ensure that no other process on the system (say, a backup or indexing daemon) accidentally wanders into the Ekam FUSE mounts, performing I/O that Ekam interprets as coming from the build step. Mount namespaces could solve this, but assuming you don't want to run Ekam as root, this would require user namespaces, and currently there's no way to mount FUSE devices inside a user namespace. LD_PRELOAD is not without its own share of problems, though. On Linux, you cannot override intra-library calls, so it's not enough to just override the syscall wrappers, you must also override all libc calls that take paths (e.g. fopen()). Also, some people don't use the libc syscall wrappers, preferring syscall(2) or even raw assembly code (especially true of languages that don't like calling into C), which can't be intercepted by LD_PRELOAD. Oh yes, and statically-linked programs are a problem. (Luckily I haven't run into these problems in anything I actually wanted to run as a build tool.) If I were rewriting Ekam today, I'd probably use seccomp to raise SIGSYS on actual system calls, and use LD_PRELOAD only to install a SIGSYS handler. This should be cleaner and work for a wider range of programs, though it would be totally Linux-specific. (Currently Ekam only runs on Linux, but its strategy has been shown to work at least on FreeBSD and OSX in the past.)
- kolev 12y ago
  Thanks for taking the time to clarify. FUSE was actually a big drawback of tup and especially on the OS X platform, so, I'm definitely gonna look into Ekam!

Tup also relies on FUSE. Do you know if Ekam does as well?

by kolev 12y ago

Tup also relies on FUSE. Do you know if Ekam does as well?

Ekam uses LD_PRELOAD to intercept system calls. I considered FUSE but it would have involved considerably more overhead and synchronization issues. - The LD_PRELOAD trick simply rewrites filenames on open() and similar calls, allowing subsequent I/O to happen directly. FUSE unfortunately requires going through the userspace FUSE server for every read or write operation. - FUSE would require either mounting a new FS on-the-fly for each build step (probably inefficient) or using a pool of FUSEs where each one is reused for multiple commands (requires carefully making sure all state is purged in between). - Something would have to be done to ensure that no other process on the system (say, a backup or indexing daemon) accidentally wanders into the Ekam FUSE mounts, performing I/O that Ekam interprets as coming from the build step. Mount namespaces could solve this, but assuming you don't want to run Ekam as root, this would require user namespaces, and currently there's no way to mount FUSE devices inside a user namespace. LD_PRELOAD is not without its own share of problems, though. On Linux, you cannot override intra-library calls, so it's not enough to just override the syscall wrappers, you must also override all libc calls that take paths (e.g. fopen()). Also, some people don't use the libc syscall wrappers, preferring syscall(2) or even raw assembly code (especially true of languages that don't like calling into C), which can't be intercepted by LD_PRELOAD. Oh yes, and statically-linked programs are a problem. (Luckily I haven't run into these problems in anything I actually wanted to run as a build tool.) If I were rewriting Ekam today, I'd probably use seccomp to raise SIGSYS on actual system calls, and use LD_PRELOAD only to install a SIGSYS handler. This should be cleaner and work for a wider range of programs, though it would be totally Linux-specific. (Currently Ekam only runs on Linux, but its strategy has been shown to work at least on FreeBSD and OSX in the past.)

kolev 12y ago
Thanks for taking the time to clarify. FUSE was actually a big drawback of tup and especially on the OS X platform, so, I'm definitely gonna look into Ekam!