4 ms·
If there is an in-memory filesystem library for this, it could be useful for unit testing things that want to write files.
by epylar 5y ago
If there is an in-memory filesystem library for this, it could be useful for unit testing things that want to write files.
- catlifeonmars 5y agoWhat’s wrong with /tmp?
- mixmastamyk 5y agoStringIo is probably faster than temp files.
- calpaterson 5y agoMy own experience is that nameless temporary files (eg tempfile.TemporaryFile()) are _considerably_ faster than (c)StringIO and worth using whenever the "file" is bigger than a few bytes.
- klmr 5y agoOdd, why would this be? Read and write operations on temporary files need to go through the kernel and thus lead to frequent context switches. The same shouldn’t be true for a string buffer except on reallocation.
- WGH_ 5y agoMaybe page cache (the thing tmpfs relies on) doesn't do full copy when reallocation, but rather, roughly speaking, appends extra pages to some sort of linked list? It would explain why it's faster.
- mixmastamyk 5y agoIf that’s the case, there’s a bug somewhere.
- calpaterson 5y agoI'm sure I must have made a mistake as I wasn't able to replicate this with some quick local benchmarking.
- mdurant 5y agoI don't know if it was a question to fsspec; we have MemoryFileSystem, addressed with "memory://" URLs.