6 ms·
On Go, Portability, and System Interfaces
- smegel 11y ago> Basically, the Go folks want to minimize external dependencies and the web of failure that can lead to. What is he ranting on about? The reason Go doesn't link against system libraries is that Go in binary-incompatible with C, as in it's calling convention is different. There is a way to call C through a shim but it's extremely slow which is why they implemented their own syscall interface for Go. Static linking is just a side-effect of this. Edit: the Go team want to support dynamic linking and it is on the roadmap but not fully implemented as of 1.5 https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit# https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGh...
- readams 11y agoStatic linking is independent of this concern. Go could implement dynamic linking but doesn't for ideological reasons. I'd really hate to be a distribution maintainer in a world where go components are common. Security bug in the TLS library? Time to download 14GB of updates!
- nulltype 11y agoJust download the deltas instead?
- xyproto 11y agoNot ideological reasons. The latest version of Go supports dynamic linking for some platforms.
- pjmlp 11y agoOn Android there is no way around it, as native code can only be accessed via dynamic linking. This is what some Go designers think of dynamic linking: http://harmful.cat-v.org/software/dynamic-linking/ http://harmful.cat-v.org/software/dynamic-linking/
- shadowmint 11y agoYeeesss... but ultimately, Go is also a pragmatic language. Dynamic linking is coming to Go, like it or not; because it's a feature that is needed in certain circumstances, and that has been acknowledged, even by the Go team.
- GauntletWizard 11y agoAnd those reasons are clear and clear-cut. Dynamic linking is a net negative in most use cases, and certainly in all the use-cases that the Go developers are developing for, where reproducible builds are mandatory, code size is a tiny fraction of main memory, and code quality is very strictly maintained. When you've built a good development and deploy strategy, rebuilding and redistributing your binary is an everyday activity. There's no additional cost.
- bahamat 11y agoThat same web page lists Linux as harmful. http://harmful.cat-v.org/software/operating-systems/linux/ http://harmful.cat-v.org/software/operating-systems/linux/
- marssaxman 11y agoI'm happy to be an end user in a world where go components are becoming more common, because statically-linked executables are far less likely to break when I upgrade unrelated programs.
- tedunangst 11y agoI doubt go to C calling conventions are going to be the slowest part of calling tcsetattr(). Using ioctl() is being difficult for the sake of being difficult.
- pcwalton 11y ago> I doubt go to C calling conventions are going to be the slowest part of calling tcsetattr(). Well, by "C calling conventions" it really means "performing a stack switch to get out of M:N, allocating a big stack if necessary or taking a lock to check one out". It's quite a lot of overhead and I wouldn't be surprised if it dwarfs the cost of the SYSENTER/SYSEXIT pair in many cases.
- tedunangst 11y agoAll of which is likely dwarfed by all the code that runs between SYSENTER and SYSEXIT. And how many freaking times do you need to call tcsetattr()? Even worst case, if calling overhead is a problem, the Go team can likely improve it at some point, but they are less likely to magically map special cases of syscall.SYS_IOCTL to tcsetattr for the sake of portability.
- 4ad 11y agoNot sure I understand your point, tcsetattr calls a system call on its own... So the alternatives are between doing a trivial system call on the Go stack, or doing a complex stack switch operation that does the (too complicated) System V calling convention ABI translation, and then calls a function that does a system call...
- 4ad 11y agoIt's not necessary to allocate a large stack, there is always one available because threads have their own stack, and you can switch to it at all times because goroutine threading is cooperative. In other words, if you are scheduled to run on a thread, you and only you have access to that thread stack. Stack switching doesn't take more than a few machine instructions. The problem with tcsetattr is that it's a library function, so you (generally[1]) need the target library in order to create a program that calls it. This is bad because in Go we value cross-compiling a lot, so we can't depend on having other people's shared libraries available in the build environment. Syscalls, however, we know how to do them and we know what they are, so we don't need any special target library in order to generate programs for the target. [1] On platforms like Solaris and Windows, where we are forced to use target shared libraries, we cheat a little and encode the target symbol names that we need, and we rely on standardized ELF/PE symbol resolution mechanism creating a scheme that's essentially equivalent to having import libraries available. But this is very complicated and takes a lot of code in the linker, so it's not at all unexpected that we avoid it whenever possible.
- 4ad 11y ago> The reason Go doesn't link against system libraries is that Go in binary-incompatible with C, as in it's calling convention is different. I'm sorry, but the system call "calling convention" is different too. This argument doesn't hold water. Calling convention translation is a trivial issue. Go doesn't usually use system libraries in order to minimise dependencies and to allow easy cross-compiling. > There is a way to call C through a shim but it's extremely slow Not really, it's actually pretty fast. Of course it will be always slower than just a regular function call, but it's sufficiently low-overhead that in most cases you can ignore the overhead. Of course there are still cases where you can't ignore the overhead. On systems like Windows and Solaris, the cgo mechanism is actually used to do back-end "system calls" all the time.
- smegel 11y ago> I'm sorry, but the system call "calling convention" is different too. Sufficiently different to not be a problem? As I understand it system calls don't usually rely on the stack whereas C calls do, and it is the stack mismatch between C and Go that is the issue. > but it's sufficiently low-overhead that in most cases you can ignore the overhead Maybe for user called function...but if every single system call had to swap out the stack surely that would have a pretty heavy impact...
- geofft 11y agoThe Go calling convention passes everything, in both directions, on the stack. Most platform-native ("C") calling conventions put as many things in registers as possible, and reuse one of the registers for the return value when possible. However, this is just a convention from one function to the next, and they can share the same stack; the caller of a C standard library function merely needs to know what to put on the stack and what to put in registers (and which registers to save). And in fact, the C calling convention is more efficient than Go's from the caller's point of view (fewer things have to go on the stack, some registers are promised not to be clobbered), so calling a C library function with the native calling convention is, on all reasonable platforms, a fine idea performance-wise. There is the problem of the stack itself. The C model envisions a large, contiguous stack, possibly with some kernel magic to grow the stack as you approach the end (by handling page faults). Go previously used "segmented stacks", essentially a linked list of heap-allocated buffers used as stack, where functions knew how to allocate and deallocate segments when necessary. C doesn't know how to do that, so a C call previously would have to allocate plenty of memory just in case the function wanted to use a bunch of stack space. Go no longer uses segmented stacks, so that's essentially not a concern. Go still does reallocate stacks, but does so by copying the entire stack, and the stack is basically never freed. So there might be a one-time performance penalty on the first C call (unless the compiler is smart enough to tell when a Goroutine might possibly call into C, and pre-allocate that space), or the first C call after most of the existing stack was used in Go code, but not after that. tl;dr: The C calling convention is strictly faster than the Go one (but more complicated), and in Go 1.3+ the existing stack is fine to use, so there shouldn't be performance concerns. (Unless, of course, you think the C library itself is slow. But that's a very different problem.)
- geofft 11y agoThis is confounding two separate issues: using the system call interface instead of the platform C library, and static vs. dynamic linking. It's an easy thing to confound, because (as the author states) on many platforms, including not only Solaris but also OS X, the only interface to the libc is dynamic linking. But you can certainly dynamically link to libc.so without dynamically linking to anything else, and without using dynamic linking within your own language community. (Rust takes roughly this approach.) There's one more subtler problem with the "defined to exist on POSIX" thing: strictly speaking, what's defined to exist is a C-language interface. Some interfaces may be defined to use macros, and some platforms may make things work when compiled through the C compiler but not through the dynamic-linker interface. One example that I've run into recently, when binding things in Rust, is the cmsg API, which is primarily defined in terms of macros to walk a heterogeneous array, and has to be reimplemented in Rust (with platform-specific code): https://github.com/carllerche/nix-rust/pull/179 https://github.com/carllerche/nix-rust/pull/179. Another example is Android, which before 5.0 (Lollipop) exposed a handful of signal-handling functions as inline functions in a C-language header file, so they were not actually dynamically linkable: https://github.com/rust-lang/rust/commit/a8dbb92b https://github.com/rust-lang/rust/commit/a8dbb92b. In both of these cases I would have loved to just #include the C header, but cross-language, that wasn't an option. It does seem that cgo has some ability to just #include the C header, as demonstrated here. I'd be curious to know if it's powerful enough to handle these two cases. (Also, the author is totally right in saying that this doesn't belong inline in random application code. If you find yourself writing code like this, it should end up in an abstraction library so that porters have a single place to look, and any application calling `tcsetattr` gains the portability benefits.)
- pcwalton 11y ago> Some interfaces may be defined to use macros, and some platforms may make things work when compiled through the C compiler but not through the dynamic-linker interface. One example that I've run into recently, when binding things in Rust, is the cmsg API, which is primarily defined in terms of macros to walk a heterogeneous array, and has to be reimplemented in Rust (with platform-specific code): The worst offender I've seen here is Xlib, which has to recreate the internal layout of large structs like Display that have haphazardly grown fields over the decades in order to deal with macros that reach into them: https://github.com/servo/rust-xlib/blob/master/src/xlib.rs#L1101 https://github.com/servo/rust-xlib/blob/master/src/xlib.rs#L...
- pcwalton 11y agoUsing the syscall interface is a lot faster in Go than calling to libc, because the system libc is going to expect large stacks and so you incur the overhead of a stack switch when you switch out of M:N threading into the C world. I assume that's why Go calls syscalls to begin with.
- teacup50 11y agoHowever, calling directly into the syscall interface is illegal on most platforms other than Linux. I'm not sure Go can fix this without getting rid of their M:N threading.
- 4ad 11y agoOn Windows and Solaris, Go doesn't do its own syscalls.
- teacup50 11y agoHow about OS X? (and, how does that work when calling into userspace APIs that assume fixed 1:1 thread/stack correlation)
- 4ad 11y agoOn OS X Go does its own system calls, ignoring the OS X guidlines. Go uses M:N scheduling, but you still have N threads with (relatively) large stacks. When calling into C code, Go switches stacks, so it runs on the (relatively) large stack where C code can run. Of course there's some overhead associated with this, but system call overhead is large on its own compared to this anyway. During the system call the thread (obviously) can't run any other Go code, but that's ok, the runtime makes sure there are always threads available to run user Go code (that's one reason why the runtime stands between user code and system calls).
- deleted 11y ago[deleted]
- 11y ago
- laxk 11y agoThe method proposed by the author does not work for packed structs. The btrfs library has a lot of them. main.go package main /* struct packed { unsigned char a; unsigned long long b; unsigned char c; } __attribute__((packed)); */ import "C" type Packed C.struct_packed go tool cgo -godefs main.go package main type Packed struct { A uint8 Pad_cgo_0 [8]byte C uint8 } We lost packed.b field. Go doesn't support packed structs. More info here: https://groups.google.com/forum/#!topic/golang-nuts/UX5srUMtSSw https://groups.google.com/forum/#!topic/golang-nuts/UX5srUMt...