3 ms·

I struggled with this as well, and ended up making the lexer itself have a lifetime which is that of the full String read from a file, so that every token (and

by brabel 5y ago

I struggled with this as well, and ended up making the lexer itself have a lifetime which is that of the full String read from a file, so that every token (and AST node) has the lifetime of the parsed file. I keep their indexes as well so that I can show error messages like rustc itself does (with code snippets and all) which would be hard to do if I didn't have the full source code text available. Hope that helps.

amelius 5y ago
This sounds a bit like cheating. What if you want to parse a really big file and delete the tokens as soon as you don't need them? If Rust claims to solve memory management, then we should not fool ourselves by allocating everything in a huge chunk which is then freed when the program ends.
- steveklabnik 5y ago
  If this is cheating, many real compilers “cheat.” Heck, some of them famously don’t even free it when the program ends, leaving that up to the operating system!
  - amelius 5y ago
    The solution doesn't scale. The moment you need to deal with very large inputs, you will have to rewrite your entire application from scratch.
- scns 5y ago
  I prolonged my own path many times in my life, by making things harder than they need to be. (edit) Aspirations acting as a anti-shortcut somehow.
- brabel 5y ago
  That's why this was not my first choice... but it was just far too hard to implement it otherwise... and then again, once you really think about it, you must keep ALL tokens/AST nodes in memory to be able to analyse and type-check/lint any decent language... so keeping a big Sring in memory while all nodes just point to slices of it is not nearly as wasteful as it might appear at a first glance.
mamcx 5y ago
This sound nice and simple. I moving to arenas but is a extra complexity. Can you show how is done? (ie:code!)
- brabel 5y ago
  https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=6d98ea7c6922c8c42aaf2d681ac24e5a https://play.rust-lang.org/?version=stable&mode=debug&editio... This loses the whitespace and consequently positions... but should be enough to get you started... I use a UTF-8 crate to split words which keeps all whitespace, so I can keep positions correctly, in case you want to do that: https://unicode-rs.github.io/unicode-segmentation/unicode_segmentation/index.html https://unicode-rs.github.io/unicode-segmentation/unicode_se...

by brabel 5y ago

amelius 5y ago

This sounds a bit like cheating. What if you want to parse a really big file and delete the tokens as soon as you don't need them? If Rust claims to solve memory management, then we should not fool ourselves by allocating everything in a huge chunk which is then freed when the program ends.

steveklabnik 5y ago
If this is cheating, many real compilers “cheat.” Heck, some of them famously don’t even free it when the program ends, leaving that up to the operating system!
- amelius 5y ago
  The solution doesn't scale. The moment you need to deal with very large inputs, you will have to rewrite your entire application from scratch.
scns 5y ago
I prolonged my own path many times in my life, by making things harder than they need to be. (edit) Aspirations acting as a anti-shortcut somehow.
brabel 5y ago
That's why this was not my first choice... but it was just far too hard to implement it otherwise... and then again, once you really think about it, you must keep ALL tokens/AST nodes in memory to be able to analyse and type-check/lint any decent language... so keeping a big Sring in memory while all nodes just point to slices of it is not nearly as wasteful as it might appear at a first glance.

mamcx 5y ago

This sound nice and simple. I moving to arenas but is a extra complexity. Can you show how is done? (ie:code!)

brabel 5y ago
https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=6d98ea7c6922c8c42aaf2d681ac24e5a https://play.rust-lang.org/?version=stable&mode=debug&editio... This loses the whitespace and consequently positions... but should be enough to get you started... I use a UTF-8 crate to split words which keeps all whitespace, so I can keep positions correctly, in case you want to do that: https://unicode-rs.github.io/unicode-segmentation/unicode_segmentation/index.html https://unicode-rs.github.io/unicode-segmentation/unicode_se...