4 ms·

Have there been any thoughts around making the compiler something akin to a stateful server with an embedded datastore for the caching rather than using the fil

Have there been any thoughts around making the compiler something akin to a stateful server with an embedded datastore for the caching rather than using the filesystem?

steveklabnik 9y ago
Not directly, but there's been some work on changing the compiler's interfaces to ones that would work with that model, in my understanding. It's not out of the realm of possibility.
Nelkins 9y ago
I think this is something that C# and VB.NET do.
AceJohnny2 9y ago
What would be the benefit of this approach over using the filesystem?
- azdle 9y ago
  I'd imagine it would be a benefit where, let's say, every project has `byteorder` in its dependency tree somewhere. If you had a local, stateful, build server that processes every build on your machine, you could re-use the object file from the last time you compiled that library regardless of if it was for this project or not. That's probably not a big gain for just `byteorder`, but if you multiply that by many libs or take into account some of the much bigger ones it could make quite a difference in build time. Even "first" build time for new projects you're building. IIRC, some group in mozilla has already has something working that works like ccache that works (or rewrote ccache to work) with compiling rust. (Which I think is what steveklabnik is alluding to in his neighboring comment.) Although I guess that might only work within a single code base, not sure about that.
  - steveklabnik 9y ago
    You're thinking of https://github.com/mozilla/sccache https://github.com/mozilla/sccache which is not quite the same thing.
  - foota 9y ago
    Man, I bet that would be blazing fast!
- yazaddaruvala 9y ago
  Any time you put data onto the file system there is overhead (eg. Memcache, Redis, most databases). Simple things like parsing the bytes into the binary representation go away. You can have more granularity of the cache because you don't need to worry about creating too many files. Not only from a performance perspective but also from a code perspective. More nuanced things like cache locality can be improved with an embedded data store rather than creating files and hoping the OS understands they will always be used together. More advanced things like retaining inverted indexes becomes possible and search performance is dramatically improved.
  - zbentley 9y ago
    The shortest path to something like that (assuming it's a good idea) would probably not be writing an in-memory incremental compile state server, but storing intermediate artifacts as very cheap-to-serialize files (flatpak or just mmap and a cast), and then storing the files on /dev/shm or some other memory filesystem. That would give you most of the benefits you discussed (unless you clear out the cache/restart the machine) without requiring a huge rewrite of how state is stored. Most of the assumptions I'm making here are unverified, though, so take it with a large grain of salt.
deleted 9y ago
[deleted]

Have there been any thoughts around making the compiler something akin to a stateful server with an embedded datastore for the caching rather than using the filesystem?

steveklabnik 9y ago

Not directly, but there's been some work on changing the compiler's interfaces to ones that would work with that model, in my understanding. It's not out of the realm of possibility.

Nelkins 9y ago

I think this is something that C# and VB.NET do.

AceJohnny2 9y ago

What would be the benefit of this approach over using the filesystem?

azdle 9y ago
I'd imagine it would be a benefit where, let's say, every project has `byteorder` in its dependency tree somewhere. If you had a local, stateful, build server that processes every build on your machine, you could re-use the object file from the last time you compiled that library regardless of if it was for this project or not. That's probably not a big gain for just `byteorder`, but if you multiply that by many libs or take into account some of the much bigger ones it could make quite a difference in build time. Even "first" build time for new projects you're building. IIRC, some group in mozilla has already has something working that works like ccache that works (or rewrote ccache to work) with compiling rust. (Which I think is what steveklabnik is alluding to in his neighboring comment.) Although I guess that might only work within a single code base, not sure about that.
- steveklabnik 9y ago
  You're thinking of https://github.com/mozilla/sccache https://github.com/mozilla/sccache which is not quite the same thing.
- foota 9y ago
  Man, I bet that would be blazing fast!
yazaddaruvala 9y ago
Any time you put data onto the file system there is overhead (eg. Memcache, Redis, most databases). Simple things like parsing the bytes into the binary representation go away. You can have more granularity of the cache because you don't need to worry about creating too many files. Not only from a performance perspective but also from a code perspective. More nuanced things like cache locality can be improved with an embedded data store rather than creating files and hoping the OS understands they will always be used together. More advanced things like retaining inverted indexes becomes possible and search performance is dramatically improved.
- zbentley 9y ago
  The shortest path to something like that (assuming it's a good idea) would probably not be writing an in-memory incremental compile state server, but storing intermediate artifacts as very cheap-to-serialize files (flatpak or just mmap and a cast), and then storing the files on /dev/shm or some other memory filesystem. That would give you most of the benefits you discussed (unless you clear out the cache/restart the machine) without requiring a huge rewrite of how state is stored. Most of the assumptions I'm making here are unverified, though, so take it with a large grain of salt.

deleted 9y ago

[deleted]