4 ms·
> Making a header with the city string mapped to an integer would dramatically shrink the file and speed up parsing. Indeed, Parquet will do this for you if yo
by amluto 2y ago
> Making a header with the city string mapped to an integer would dramatically shrink the file and speed up parsing.
Indeed, Parquet will do this for you if you let it.
I also wonder how good Java is at optimizing closure creation in a loop, as in:
k -> new ResultObserver()
The vast majority of those closures are created and never called. C++ might optimize this well if calling a template but has basically no chance if the parameter is an std::function. Java probably has more ability to optimize it, but I don’t know whether it actually does so.
- the-alchemist 2y agoI believe the term for this is "partial escape analysis" (PEA). GraalVM is the SOTA in the JVM world for this. Here's an oldish blog on this: https://chrisseaton.com/truffleruby/seeing-escape-analysis/ https://chrisseaton.com/truffleruby/seeing-escape-analysis/ I'm sure PEA is better now, but it's not sure if it's moved beyond scalars (int, float, etc.)
- vips7L 2y agoWouldn't the lambda here by always escape? It gets created in the function and passed into another one. Looking at the byte code it looks like it always allocates in when its in the interpreter: invokedynamic #16, 0 // InvokeDynamic #0:apply:()Ljava/util/function/Function;
- amluto 2y agoI would expect the relevant optimization to be inlining and/or specialization. An inlined function could allow the JIT detect that the hot path creates and destroys an object without ever using it. Alternatively, if the JIT could specialize the callee for the specific function that is passed in, then the overhead would go away. (One think I like about less heavily JIT-reliant languages is that it can be more obvious when something will incur a runtime cost.)
- vips7L 2y agoReading more about how invokedynamic works, it appears that if your lambda doesn't capture it shouldn't allocate it more than one time.
- WASDx 2y agoThe closure itself is only being created once, it's essentially a singleton. Only if it would capture variables it would have to be recreated every iteration.