4 ms·
We had a similar task a year ago in a Python codebase and we ended up building a function in Rust for it. Rust's splitting iterators over `&str` return referenc
by iamed2 8y ago
We had a similar task a year ago in a Python codebase and we ended up building a function in Rust for it. Rust's splitting iterators over `&str` return references to the same data as their own `&str`s, which is a huge performance win. We would filter the split strings and conditionally store them in a `HashMap<&str, &str>`, which we later drained, both automatically freeing memory behind the scenes as able. We also shaved off a lot of time in some cases by avoiding parsing and comparing strings in cases where we had fixed-length integers (e.g., UNIX timestamps in a known range).
- srean 8y agoYes this is one aspect that surprised me about Python. I expect a scripting language to be really good with strings and I/O. Python does a pretty good job at both. Strings are immutable, which IMO is a good choice and then I get this really disappointing surprise that slices don't share content. I understand their motivation though. Memoryviews can substitute for immutable string slices at times, but since these are not strings, its annoying. One can get a string out of a memoryview but I believe Python copies at that point.
- rndgermandude 8y agoIf you'd be sharing slices naively you might run into cases like a = "large" * 10000000 self.b = a[1:10] this tiny slice would then "magically" keep the entire huge large string alive. It would be easy to defacto leak memory with that. Even if you are less naive and introduce some kind of heuristic to determine when to share and when to copy, you might still be leaking "lots" of memory if you have a lot of small slices (into small-ish strings), which might hurt on low speced/embedded systems with tiny memory available.
- jerf 8y agoI would point out that by "it would be easy to leak memory with that", in reality almost every run time that has ever implemented this "optimization" to automatically apply has had to either pull it back out, or make it explicit whether or not you're taking a copy of the underlying string. I've seen it at least three times. The probability of a given program having this problem is lowish (not that low, but reasonably low), but the probability of one of a language community's flagship projects having this problem shoots to 100% almost instantly for any non-trivial language or community.
- srean 8y agoI like it how Guile does it. There it is very explicit whether you are getting a copy or a view.
- Razengan 8y ago> Rust's splitting iterators over `&str` return references to the same data as their own `&str`s, which is a huge performance win. Seems similar to Swift's Substrings (and other collection "slices"): https://developer.apple.com/documentation/swift/substring https://developer.apple.com/documentation/swift/substring
- iamed2 8y agoSimilar, except for this key difference: > A substring holds a reference to the entire storage of the string it comes from, not just to the portion it presents, even when there is no other reference to the original string. Rust's `&str` doesn't do that, since the borrow checker will ensure at compile time that the `&str` reference doesn't live longer than the data it points to. At runtime there is no structural difference between an "original" `&str` and a `&str` pointing to a substring of the original.
- pjc50 8y agoZero-copy substring classes are hugely useful for parsing. I first encountered this approach in the Zeus Web Server back in the early 2000s, in non-STL C++.
- phyzome 8y agoHeck, even in Java, String.split returns shared references (with different offsets) to the same backing byte array. Mind you, that can bite you pretty hard if you're taking in giant documents, splitting down to a few tiny tokens, and then holding those tiny tokens in a long-lived in-memory cache (ugh, voice of experience here) but most of the time that's the Right Thing.