3 ms·

Even lexing ruby for highlighting is really hard. I had to use so many lookahead hacks to get the lexer in rouge working: https://github.com/jayferd/rouge/blob

by jayferd 14y ago

Even lexing ruby for highlighting is really hard. I had to use so many lookahead hacks to get the lexer in rouge working: https://github.com/jayferd/rouge/blob/master/lib/rouge/lexers/ruby.rb https://github.com/jayferd/rouge/blob/master/lib/rouge/lexer... I learned a ton about ruby when implementing that. Did you know that this totally works: my_method <<-ONE, <<-TWO, <<-THREE text of one ONE text of two TWO text of three THREE And all the % delimited strings are super hard. (also note that pygments chokes on my perfectly valid ruby file, at `%r(\\\\)`)

Udo 14y ago
For non-rubyists, what does this example do? Are those labels?
- venus 14y ago
  That's ruby's heredoc syntax: http://en.wikipedia.org/wiki/Here_document http://en.wikipedia.org/wiki/Here_document I had no idea it could be nested like that and I've been using ruby for 8 years. Crazy!
  - mitchty 14y ago
    Holy cow, 10 years for me and I've never even thought of nesting heredocs. Some days I wish Matz would formalize the grammar so it is sane.
- ahlatimer 14y ago
  It's a way of writing strings without having to escape things or having line breaks mess things up (a heredoc, if you're familiar with that terminology). For instance, <<-END this is some text END will give you a string containing " this is some text\n". END can be anything. It's particularly useful in metaprogramming, since you can maintain formatting, and you don't have to escape " or ' if they happen to show up in your code. What's interesting in the OP's example is that you end up passing three strings " text of one\n", " text of two\n", " text of three\n". I had no idea that would happen, though I have seen things like define_method <<-RUBY, __FILE__, __LINE__ + 1 some_code RUBY so it makes sense. Ruby treats anything on the same line after the comma as separate arguments, so even though __FILE__ comes after <<-RUBY and before RUBY, it really appears after the final RUBY to the interpreter.
- dllthomas 14y ago
  As another non-rubyist with too much perl and bash under my belt, I'm guessing it's heredoc - the <<-FOO syntax means "grab the lines following until you see FOO alone on a line, shove it all in a string, and pass it as this argument (in bash it passes a filename that represents half of a pipe which has been sent the content).
the_mitsuhiko 14y ago
> (also note that pygments chokes on my perfectly valid ruby file, at `%r(\\\\)`) Pygments was actually better at lexing ruby at one point but the regular expressions were too complex and some other bug fixes broke other stuff. I think at the moment it's good enough. But yeah, I was in the same boat. I learned Ruby for Pygments. //EDIT: vim does considerably worse on that file btw.
- jayferd 14y ago
  Yep. It's surprising how many things can be missing from a lexer and still be "good enough" for most people. I don't know anybody who's actually used that heredoc-stacking thing. :D I guess you're the author of all those frustrated "wtf ruby" comments I found in the pygments lexer, then :). I seriously doubt I could have done it at all without pygments as a reference. My vim does fine on that file: http://imgur.com/VSYr4aa http://imgur.com/VSYr4aa . Maybe I'm using a more recent version...? It misses some of the `end` keywords, but that's just a quirk that it has ;) Another example that bit me - this took the longest, I think: foo = 10 foo %(2) # call method foo with string "2" foo % 2 # the value of foo modulo 2 % 2 # the string "2"
SeoxyS 14y ago
These are the kinds of things that make me appreciate s-expressions so much more. It's crazy how programming languages like Ruby or C can have ridiculously complex syntax, and LISP can express all of that an then some with just three or four simple grammar rules.

by jayferd 14y ago

Udo 14y ago

For non-rubyists, what does this example do? Are those labels?

venus 14y ago
That's ruby's heredoc syntax: http://en.wikipedia.org/wiki/Here_document http://en.wikipedia.org/wiki/Here_document I had no idea it could be nested like that and I've been using ruby for 8 years. Crazy!
- mitchty 14y ago
  Holy cow, 10 years for me and I've never even thought of nesting heredocs. Some days I wish Matz would formalize the grammar so it is sane.
ahlatimer 14y ago
It's a way of writing strings without having to escape things or having line breaks mess things up (a heredoc, if you're familiar with that terminology). For instance, <<-END this is some text END will give you a string containing " this is some text\n". END can be anything. It's particularly useful in metaprogramming, since you can maintain formatting, and you don't have to escape " or ' if they happen to show up in your code. What's interesting in the OP's example is that you end up passing three strings " text of one\n", " text of two\n", " text of three\n". I had no idea that would happen, though I have seen things like define_method <<-RUBY, __FILE__, __LINE__ + 1 some_code RUBY so it makes sense. Ruby treats anything on the same line after the comma as separate arguments, so even though __FILE__ comes after <<-RUBY and before RUBY, it really appears after the final RUBY to the interpreter.
dllthomas 14y ago
As another non-rubyist with too much perl and bash under my belt, I'm guessing it's heredoc - the <<-FOO syntax means "grab the lines following until you see FOO alone on a line, shove it all in a string, and pass it as this argument (in bash it passes a filename that represents half of a pipe which has been sent the content).

the_mitsuhiko 14y ago

> (also note that pygments chokes on my perfectly valid ruby file, at `%r(\\\\)`) Pygments was actually better at lexing ruby at one point but the regular expressions were too complex and some other bug fixes broke other stuff. I think at the moment it's good enough. But yeah, I was in the same boat. I learned Ruby for Pygments. //EDIT: vim does considerably worse on that file btw.

jayferd 14y ago
Yep. It's surprising how many things can be missing from a lexer and still be "good enough" for most people. I don't know anybody who's actually used that heredoc-stacking thing. :D I guess you're the author of all those frustrated "wtf ruby" comments I found in the pygments lexer, then :). I seriously doubt I could have done it at all without pygments as a reference. My vim does fine on that file: http://imgur.com/VSYr4aa http://imgur.com/VSYr4aa . Maybe I'm using a more recent version...? It misses some of the `end` keywords, but that's just a quirk that it has ;) Another example that bit me - this took the longest, I think: foo = 10 foo %(2) # call method foo with string "2" foo % 2 # the value of foo modulo 2 % 2 # the string "2"

SeoxyS 14y ago

These are the kinds of things that make me appreciate s-expressions so much more. It's crazy how programming languages like Ruby or C can have ridiculously complex syntax, and LISP can express all of that an then some with just three or four simple grammar rules.