5 ms·

Great blog post! I'd just like to add for the curious, that usually you'd use goyacc for parsing SQL. And most serious SQL projects in Go have started with th

Great blog post! I'd just like to add for the curious, that usually you'd use goyacc for parsing SQL. And most serious SQL projects in Go have started with the SQL parser from vitess and adapted it to their use case (which is just funny trivia, but for anything big, I recommend it, did the same for OctoSQL [0]). [0]: https://github.com/cube2222/octosql https://github.com/cube2222/octosql

pizza234 7y ago
This is not the first educational project I've seen that insists on manually writing the parser. Call me contrarian but I find the article's approach poor (at least, as it is now). If one looks at the overall content of the two articles of the series, 70%/80% of the content (or possibly more) is parsing, which is arguably the least interesting, or at least, the most boilerplate, part of a database system.
tbrock 7y ago
Why did all of the Golang SQL parsers come from Vitess? I would love to know more about this history. Was it because they were the first and people just started using it or is it the best for some reason?
- eatonphil 7y ago
  SQL is a humongous spec. Any serious project would rather piggy-back off an existing parser. Most of the interesting parts for most people is implementing backends against in-memory, disk, S3, HDFS, etc.
  - eatonphil 7y ago
    Also, a contributor to dolthub shared on Reddit last time this post came up that they originally wrote their own SQL frontend but gave up because it was so much to maintain and get correct. They ended up going with go-mysql-server which uses vitess. https://www.reddit.com/r/golang/comments/fgwwlx/database_basics_writing_a_sql_database_from/fk84dq9/?context=3 https://www.reddit.com/r/golang/comments/fgwwlx/database_bas...
- cube2222 7y ago
  No idea about the history. From my perspective (as the author of OctoSQL): I need a SQL parser. Vitess has a fully fledged MySQL compatible one. I'd probably want to copy the open source vitess one and add the necessary features myself, instead of writing it from scratch. It's just a big endeavor, a few thousand lines of goyacc code and even more Go boilerplate. I'm not saying everybody is using the same library, because the code has been heavily modified after being copied. A look at the cockroachdb parses suffices to see how much changed it is. If you look at OctoSQL, there too have been many changes to add stuff like table valued functions.
eatonphil 7y ago
I like this project! I think I've asked you about it supporting parquet before... I'll add it to my list of similar projects. Regarding goyacc and "usually", I myself was curious about this years ago and asked on Stack Exchange: Do modern languages still use parser generators? [0]. Most of the time, major languages _do_ use hand-written parsers. Of course it may literally be the case that most other SQL projects in Go do use goyacc. My point is that in general, hand-written parsers are more common in real software than you might think. [0] https://softwareengineering.stackexchange.com/questions/250256/do-modern-languages-still-use-parser-generators/250264 https://softwareengineering.stackexchange.com/questions/2502...
- cube2222 7y ago
  Indeed, I remember! :) The parquet support is mostly ready as far as I know here https://github.com/cube2222/octosql/pull/153/ https://github.com/cube2222/octosql/pull/153/ but I haven't tested it myself yet. However, we'll surely have it ready for the "streaming" release which is planned for May/June. Regarding the parsing part, that's interesting. Though the cockroachdb parser is a _great_ lecture about how to add nice error messages with goyacc. (You add a special error token which captures anything after a syntax error, so the parser doesn't break down. At least something around this, haven't yet dived deep into it.) Though I know Go itself uses a handwritten parser.

pizza234 7y ago

This is not the first educational project I've seen that insists on manually writing the parser. Call me contrarian but I find the article's approach poor (at least, as it is now). If one looks at the overall content of the two articles of the series, 70%/80% of the content (or possibly more) is parsing, which is arguably the least interesting, or at least, the most boilerplate, part of a database system.

tbrock 7y ago

Why did all of the Golang SQL parsers come from Vitess? I would love to know more about this history. Was it because they were the first and people just started using it or is it the best for some reason?

eatonphil 7y ago
SQL is a humongous spec. Any serious project would rather piggy-back off an existing parser. Most of the interesting parts for most people is implementing backends against in-memory, disk, S3, HDFS, etc.
- eatonphil 7y ago
  Also, a contributor to dolthub shared on Reddit last time this post came up that they originally wrote their own SQL frontend but gave up because it was so much to maintain and get correct. They ended up going with go-mysql-server which uses vitess. https://www.reddit.com/r/golang/comments/fgwwlx/database_basics_writing_a_sql_database_from/fk84dq9/?context=3 https://www.reddit.com/r/golang/comments/fgwwlx/database_bas...
cube2222 7y ago
No idea about the history. From my perspective (as the author of OctoSQL): I need a SQL parser. Vitess has a fully fledged MySQL compatible one. I'd probably want to copy the open source vitess one and add the necessary features myself, instead of writing it from scratch. It's just a big endeavor, a few thousand lines of goyacc code and even more Go boilerplate. I'm not saying everybody is using the same library, because the code has been heavily modified after being copied. A look at the cockroachdb parses suffices to see how much changed it is. If you look at OctoSQL, there too have been many changes to add stuff like table valued functions.

eatonphil 7y ago

I like this project! I think I've asked you about it supporting parquet before... I'll add it to my list of similar projects. Regarding goyacc and "usually", I myself was curious about this years ago and asked on Stack Exchange: Do modern languages still use parser generators? [0]. Most of the time, major languages _do_ use hand-written parsers. Of course it may literally be the case that most other SQL projects in Go do use goyacc. My point is that in general, hand-written parsers are more common in real software than you might think. [0] https://softwareengineering.stackexchange.com/questions/250256/do-modern-languages-still-use-parser-generators/250264 https://softwareengineering.stackexchange.com/questions/2502...

cube2222 7y ago
Indeed, I remember! :) The parquet support is mostly ready as far as I know here https://github.com/cube2222/octosql/pull/153/ https://github.com/cube2222/octosql/pull/153/ but I haven't tested it myself yet. However, we'll surely have it ready for the "streaming" release which is planned for May/June. Regarding the parsing part, that's interesting. Though the cockroachdb parser is a _great_ lecture about how to add nice error messages with goyacc. (You add a special error token which captures anything after a syntax error, so the parser doesn't break down. At least something around this, haven't yet dived deep into it.) Though I know Go itself uses a handwritten parser.