4 ms·

I don't have a lot of experience writing Lisp-y code, so perhaps I'm speaking from ignorance, but I think there is a reason that syntax never gained huge tracti

by erezsh 5y ago

I don't have a lot of experience writing Lisp-y code, so perhaps I'm speaking from ignorance, but I think there is a reason that syntax never gained huge traction. Imho a syntax that's concise and expressive is important for effective coding. Having operators for the most common operations is just a small complication that yields a big reward. Having said that, the amount of keywords and operators that you see in Preql right now, isn't likely to grow by much. I have the basic and common operators covered, and for the most part, the rest can be done with regular functions. I agree about introspection, which is why in Preql you can ask for the type of any value, list the columns of a table, get the resulting SQL of an expression as a string, and so on. And certainly more can and should be done towards that.

dvdkon 5y ago
Thanks for the reply. I'm happy that others are also tackling the problem of a better query language. My approach isn't actually a full-on Lisp with parentheses and all, rather it's based on a single compound data structure (like Lisp's cons cell, but more like Lua tables). I call it an "arg-tuple", and it's basically a function's arguments in Python, but as a data type (which allows nesting). Add in a simple function call syntax, infix operartors and a "pipeline" operator (like F#'s |>) and you get something like this: from stops | let region :be case(zoneid, :when ("P", "0", "B") :then "Prague", :when ("1", "2") :then "Almost-Prague", :else "Regional") | where lat != 0 && lon != 0 && region != "Prague" | select name, region "from", "let", "where" and "select" are filters, functions that take a query description object and return a new one. "name, region" is just an arg-tuple with two unnamed elements and "case" is a function taking one polymorphic value and a variable number of nested arg-tuples with ":when: and ":then" named elements. Filters can be arranged in any order, unlike SQL, so this would work: from stops | group_by lat, lon | group_by lat | select min(group.lon)
- erezsh 5y ago
  Python has a data structure like this called NamedTuple. I also noticed it's popping up everywhere, in this case sql rows (kind of). Conceptually I like the idea of using it as the mechanism for function arguments, but I didn't want the syntax to be too confusing for beginners. Although yours seems a little bit different, because different fields can have the same name. Just for the heck of it, here's how your second example would look in preql: (stops { lat, lon => } { lat => lon: min(lon) } { lon } // not necessary. just drops the lat column )
  - dvdkon 5y ago
    It's not quite that arg-tuples can contain multiple values under the same name, but rather ":a 1, :a 2" is actually an arg-tuple containing two nested arg-tuples as unnamed values. I agree that basing all of syntax on a single element (especially one that's so "light" on syntax), it's hard making the resulting syntax intuitive. I think it's worth it, but I can see other approaches' benefits.
    - erezsh 5y ago
      It's definitely an interesting approach. I imagine it makes the parsing much easier.
- culebron21 5y ago
  I tried developing a syntax like this: a table is set as context, then you do some transformations in pipeline fashion. That seems promising, but trying to rewrite some of my reallife examples, I found out that pipelines are much shorter than I expected. The reason was that you take a couple of tables, then make 3-4 of them (by filtering and grouping), then mix them together. And this does not fit well into pipeline structure, even with heavy nesting. In the end, the most gain on real examples (Python & Pandas) was where I made a function-like operation, which can be as well written in Python. I can share some sketches if you wish (maybe it can give some ideas). But generally, I think the code could be rather structured around calculation graph, than a linear sequence of operations.
  - dvdkon 5y ago
    I've also tried to rewrite some of my challenging queries [0] in my hypothetical syntax and while I think your observation about pipeline length is correct, the result still came out much better than SQL. Frankly, even in F#, most of my pipelines are around 5 functions too. In my view, pipelines are just a convenient mental model. I'd love to see your sketches, here are my (very WIP) concepts: [1] I don't quite see how a computation graph would work as the core of a textual language and I'm sceptical about using it with existing DBMSs (I'd like to actually use my creation one day :) ), but I'm open to ideas. [0]: for example this monstrosity: https://gitlab.com/dvdkon/jrutil/-/blob/dbb971c18526e68dcc97415676298b4404e47970/rtview/src/Trip.fs#L73 https://gitlab.com/dvdkon/jrutil/-/blob/dbb971c18526e68dcc97... [1]: https://gitlab.com/-/snippets/2147895 https://gitlab.com/-/snippets/2147895

by erezsh 5y ago

dvdkon 5y ago

Thanks for the reply. I'm happy that others are also tackling the problem of a better query language. My approach isn't actually a full-on Lisp with parentheses and all, rather it's based on a single compound data structure (like Lisp's cons cell, but more like Lua tables). I call it an "arg-tuple", and it's basically a function's arguments in Python, but as a data type (which allows nesting). Add in a simple function call syntax, infix operartors and a "pipeline" operator (like F#'s |>) and you get something like this: from stops | let region :be case(zoneid, :when ("P", "0", "B") :then "Prague", :when ("1", "2") :then "Almost-Prague", :else "Regional") | where lat != 0 && lon != 0 && region != "Prague" | select name, region "from", "let", "where" and "select" are filters, functions that take a query description object and return a new one. "name, region" is just an arg-tuple with two unnamed elements and "case" is a function taking one polymorphic value and a variable number of nested arg-tuples with ":when: and ":then" named elements. Filters can be arranged in any order, unlike SQL, so this would work: from stops | group_by lat, lon | group_by lat | select min(group.lon)

erezsh 5y ago
Python has a data structure like this called NamedTuple. I also noticed it's popping up everywhere, in this case sql rows (kind of). Conceptually I like the idea of using it as the mechanism for function arguments, but I didn't want the syntax to be too confusing for beginners. Although yours seems a little bit different, because different fields can have the same name. Just for the heck of it, here's how your second example would look in preql: (stops { lat, lon => } { lat => lon: min(lon) } { lon } // not necessary. just drops the lat column )
- dvdkon 5y ago
  It's not quite that arg-tuples can contain multiple values under the same name, but rather ":a 1, :a 2" is actually an arg-tuple containing two nested arg-tuples as unnamed values. I agree that basing all of syntax on a single element (especially one that's so "light" on syntax), it's hard making the resulting syntax intuitive. I think it's worth it, but I can see other approaches' benefits.
  - erezsh 5y ago
    It's definitely an interesting approach. I imagine it makes the parsing much easier.
culebron21 5y ago
I tried developing a syntax like this: a table is set as context, then you do some transformations in pipeline fashion. That seems promising, but trying to rewrite some of my reallife examples, I found out that pipelines are much shorter than I expected. The reason was that you take a couple of tables, then make 3-4 of them (by filtering and grouping), then mix them together. And this does not fit well into pipeline structure, even with heavy nesting. In the end, the most gain on real examples (Python & Pandas) was where I made a function-like operation, which can be as well written in Python. I can share some sketches if you wish (maybe it can give some ideas). But generally, I think the code could be rather structured around calculation graph, than a linear sequence of operations.
- dvdkon 5y ago
  I've also tried to rewrite some of my challenging queries [0] in my hypothetical syntax and while I think your observation about pipeline length is correct, the result still came out much better than SQL. Frankly, even in F#, most of my pipelines are around 5 functions too. In my view, pipelines are just a convenient mental model. I'd love to see your sketches, here are my (very WIP) concepts: [1] I don't quite see how a computation graph would work as the core of a textual language and I'm sceptical about using it with existing DBMSs (I'd like to actually use my creation one day :) ), but I'm open to ideas. [0]: for example this monstrosity: https://gitlab.com/dvdkon/jrutil/-/blob/dbb971c18526e68dcc97415676298b4404e47970/rtview/src/Trip.fs#L73 https://gitlab.com/dvdkon/jrutil/-/blob/dbb971c18526e68dcc97... [1]: https://gitlab.com/-/snippets/2147895 https://gitlab.com/-/snippets/2147895