3 ms·

So I want to preface my next point by stating actually really like how syntax reads, hitting a sweet spot of leaning just enough into APL's symbol-focused terse

So I want to preface my next point by stating actually really like how syntax reads, hitting a sweet spot of leaning just enough into APL's symbol-focused terseness without losing the familiar structure of literally every other popular programming language ever that makes it easy to follow the code. I imagine it could be a great language to pick up for working out an algorithm with pen & paper (the best limus test for how well a language functions as pseudocode imo). Having said that, I have my doubts about the reduced token claims because of a lack of "begin" and "end" or "{" and "}" delimiters. Surely, if your language is identation based, LLMs will treat significant whitespace as a token too, since they need to keep track of it? Which would result in an extra token per newline compared to using explicit begin/end token? I suspect that to really optimize for tokens one would have to go the concatenative route, which significant whitespace very much isn't.

CompanyGardener 3mo ago
Apologies for the slow reply, my lack of karma is throttling how often I can comment. I appreciate you sharing your thoughts, the whole point of a public preview is to find out how things are received, what’s working, what isn’t. We more or less agree on the vibe-coded website (especially the copywriting). It’ll improve over time, but this is a side-project, time is limited, and I wanted to release something on 7/4. Tungsten’s grammar and design are mine. Tungsten’s implementation, is almost entirely LLM (lowering, escape analysis, SSA), with a good bit of prodding, benchmarking, troubleshooting; in order to have the performance land remotely close to where I knew it could. While my primary motivation was to build the language that I wish existed, if I can make something that others enjoy or find useful, even better. To address your other point: Whitespace is almost certainly getting tokenized whether it’s significant to the language or not. And I would expect most folks are indenting their blocks anyway, even when they are delimited. I think you’ll find the token reduction comes from more than just dropping delimiters. I won’t try to litigate the point here, I’ll save that for a longer-form write up that is lacking on the website. Thanks again for the feedback. Related (in the repo): WValue layout https://github.com/tungsten-lang/tungsten/blob/d5f505acca24547e92acb4fc27a68dd1bd55c0e3/doc/WVALUE.md https://github.com/tungsten-lang/tungsten/blob/d5f505acca245... Ruby Lexer (regex-based) https://github.com/tungsten-lang/tungsten/blob/d5f505acca24547e92acb4fc27a68dd1bd55c0e3/implementations/ruby/lib/tungsten/lexer.rb#L1281 https://github.com/tungsten-lang/tungsten/blob/d5f505acca245... Ruby Lexer (spec) https://github.com/tungsten-lang/tungsten/blob/d5f505acca24547e92acb4fc27a68dd1bd55c0e3/implementations/ruby/spec/lexer_spec.rb#L426 https://github.com/tungsten-lang/tungsten/blob/d5f505acca245...
- vanderZwan 3mo ago
  Thanks for the in-depth reply, appreciate the elaboration on where you did and didn't use LLMs. Looking forward to reading the blog posts and other future updates!

CompanyGardener 3mo ago

Apologies for the slow reply, my lack of karma is throttling how often I can comment. I appreciate you sharing your thoughts, the whole point of a public preview is to find out how things are received, what’s working, what isn’t. We more or less agree on the vibe-coded website (especially the copywriting). It’ll improve over time, but this is a side-project, time is limited, and I wanted to release something on 7/4. Tungsten’s grammar and design are mine. Tungsten’s implementation, is almost entirely LLM (lowering, escape analysis, SSA), with a good bit of prodding, benchmarking, troubleshooting; in order to have the performance land remotely close to where I knew it could. While my primary motivation was to build the language that I wish existed, if I can make something that others enjoy or find useful, even better. To address your other point: Whitespace is almost certainly getting tokenized whether it’s significant to the language or not. And I would expect most folks are indenting their blocks anyway, even when they are delimited. I think you’ll find the token reduction comes from more than just dropping delimiters. I won’t try to litigate the point here, I’ll save that for a longer-form write up that is lacking on the website. Thanks again for the feedback. Related (in the repo): WValue layout https://github.com/tungsten-lang/tungsten/blob/d5f505acca24547e92acb4fc27a68dd1bd55c0e3/doc/WVALUE.md https://github.com/tungsten-lang/tungsten/blob/d5f505acca245... Ruby Lexer (regex-based) https://github.com/tungsten-lang/tungsten/blob/d5f505acca24547e92acb4fc27a68dd1bd55c0e3/implementations/ruby/lib/tungsten/lexer.rb#L1281 https://github.com/tungsten-lang/tungsten/blob/d5f505acca245... Ruby Lexer (spec) https://github.com/tungsten-lang/tungsten/blob/d5f505acca24547e92acb4fc27a68dd1bd55c0e3/implementations/ruby/spec/lexer_spec.rb#L426 https://github.com/tungsten-lang/tungsten/blob/d5f505acca245...

vanderZwan 3mo ago
Thanks for the in-depth reply, appreciate the elaboration on where you did and didn't use LLMs. Looking forward to reading the blog posts and other future updates!