3 ms·

I'm mostly write embedded code. But when I have to deal with files and associated API's I'm always like why does anyone tolerate this stuff. string[] lines =

I'm mostly write embedded code. But when I have to deal with files and associated API's I'm always like why does anyone tolerate this stuff. string[] lines = ReadFile("path", "filename"); Why is the above somehow hard?

riku_iki 3y ago
actually your code is not well designed too: - it will produce OOM if file is very large - iterating array is usually verbose in many of languages (at least in Java) - not sure why you would want to have path and filename as separate params In Java, it would be more ergonomical if they add something like following: Iterable<String> Files.readAllLines(String fileName);
- kaba0 3y ago
  How is OOM not a concern in case of C++, python? Java’s List is much more ergonomic that arrays. Also, I really don’t understand your last example - that’s literally available in java returning a List of lines. If you add a .stream().collect(joining(“\n”)), you get the whole string in one go.
  - riku_iki 3y ago
    > How is OOM not a concern in case of C++, python? I think in python it will lazely iterate through the file, and not load everything at once > Java’s List is much more ergonomic that arrays. right, but commenter above proposed array.. > that’s literally available in java returning a List of lines. current API has two issues: - OOM if file is large - it takes Path instance which you need to construct and not "String file" parameter, thus hit on ergonomics
    - kaba0 3y ago
      Path.of(yourString)
      - riku_iki 3y ago
        I know, but why it is necessary?
- Gibbon1 3y ago
  - it will produce OOM if file is very large A well designed API won't give you an OOM it'll throw a 'this file is too big' - not sure why you would want to have path and filename as separate params Throws path not found. Throws file not found. Why does the programmer need to deal with twee-fiddly path magic for say each file in a directory of config files? Iterable<String> Files.readAllLines(String fileName); Oh sure that's totally more readable. I think in python it will lazely iterate through the file, I wrote a program to suck in a list of log files and generate a bunch of metrics and display them. Users liked to select a dozen 100MB log files. Lazy iteration was way way too slow. Fastest was suck each one into memory and call split.
  - riku_iki 3y ago
    > A well designed API won't give you an OOM it'll throw a 'this file is too big' this is possible solution, still it doesn't allow to iterate lines one by one and handle large files > Throws path not found. Throws file not found. I think it is some very niche use cases when you need to distinguish between these cases. From another hand there are many places in the code which operate absolute file paths already, and you force them to split path, which is cumbersome. At least two alternative APIs probably are justified to have. > Fastest was suck each one into memory and call split. I really don't think pre-splitting is any kind of performance bottleneck, it is likely 1% of performance compared to all IO stack and subsequent string/objects memory management.

riku_iki 3y ago

actually your code is not well designed too: - it will produce OOM if file is very large - iterating array is usually verbose in many of languages (at least in Java) - not sure why you would want to have path and filename as separate params In Java, it would be more ergonomical if they add something like following: Iterable<String> Files.readAllLines(String fileName);

kaba0 3y ago
How is OOM not a concern in case of C++, python? Java’s List is much more ergonomic that arrays. Also, I really don’t understand your last example - that’s literally available in java returning a List of lines. If you add a .stream().collect(joining(“\n”)), you get the whole string in one go.
- riku_iki 3y ago
  > How is OOM not a concern in case of C++, python? I think in python it will lazely iterate through the file, and not load everything at once > Java’s List is much more ergonomic that arrays. right, but commenter above proposed array.. > that’s literally available in java returning a List of lines. current API has two issues: - OOM if file is large - it takes Path instance which you need to construct and not "String file" parameter, thus hit on ergonomics
  - kaba0 3y ago
    Path.of(yourString)
    - riku_iki 3y ago
      I know, but why it is necessary?
Gibbon1 3y ago
- it will produce OOM if file is very large A well designed API won't give you an OOM it'll throw a 'this file is too big' - not sure why you would want to have path and filename as separate params Throws path not found. Throws file not found. Why does the programmer need to deal with twee-fiddly path magic for say each file in a directory of config files? Iterable<String> Files.readAllLines(String fileName); Oh sure that's totally more readable. I think in python it will lazely iterate through the file, I wrote a program to suck in a list of log files and generate a bunch of metrics and display them. Users liked to select a dozen 100MB log files. Lazy iteration was way way too slow. Fastest was suck each one into memory and call split.
- riku_iki 3y ago
  > A well designed API won't give you an OOM it'll throw a 'this file is too big' this is possible solution, still it doesn't allow to iterate lines one by one and handle large files > Throws path not found. Throws file not found. I think it is some very niche use cases when you need to distinguish between these cases. From another hand there are many places in the code which operate absolute file paths already, and you force them to split path, which is cumbersome. At least two alternative APIs probably are justified to have. > Fastest was suck each one into memory and call split. I really don't think pre-splitting is any kind of performance bottleneck, it is likely 1% of performance compared to all IO stack and subsequent string/objects memory management.