4 ms·
> warp is currently able to preprocess many source files, one after the other, in a single command. [...] warp is set up/torn down only once for the entire set
by jedbrown 13y ago
> warp is currently able to preprocess many source files, one after the other, in a single command. [...] warp is set up/torn down only once for the entire set of files, rather than once for each source file.
I'd like to learn more about this. I spend a fair amount of time building on HPC systems. Frustratingly, compiling on a $100M computer is typically 50x slower than compiling on a laptop due to the atrocious metadata performance of the shared file system. Configuration is even worse because there is typically little or no parallelism. Moving my source tree to a fast local disk barely helps so long as system and library headers continue to reside on the slow filesystem. A compiler system that transparently caches file accesses across an entire project build would save computational scientists an enormous amount of time building on these systems.
- srean 13y agoThis is why I asked Walter about use of asynchronous idioms in Warp. It ought to help even if there is no parallelism involved. It is perhaps a TODO. Walter would be able to elaborate.
- mrich 13y agoSo your build speed is limited by metadata file accesses to network storage? Your network/storage must be really bad then. How about mirroring all the needed headers locally before building? You could establish that as a makefile rule.
- _delirium 13y agoOn large HPC clusters a common setup is to have a distributed cluster file system, rather than disk set up as "local" and "network" volumes. All the cluster machines' disks are integrated into a distributed volume running a filesystem like https://en.wikipedia.org/wiki/IBM_General_Parallel_File_System https://en.wikipedia.org/wiki/IBM_General_Parallel_File_Syst... or https://en.wikipedia.org/wiki/GFS2 https://en.wikipedia.org/wiki/GFS2