4 ms·
Interesting approach! I've been contemplating a similar query DSL for compressed time-series data on embedded devices. It's easier to make a specialized languag
by elcritch 8y ago
Interesting approach! I've been contemplating a similar query DSL for compressed time-series data on embedded devices. It's easier to make a specialized language than commonly thought. Though generally the trickiest part is creating appropriate tooling for things like source maps, debugging, etc.
It surprises me that the author used Elixir for implementation but didn't just write a query DSL utilizing the Elixir macro facilities [0]. A macro based DSL likely would be much simpler than writing a new grammar and builds on existing language tooling. Ecto for example does a great job (IMHO) of making a sane DSL for SQL [1]. Given data processing primitives provided by Elixir libraries like Streams and GenStage [2] it seems it'd be pretty straightforward to implement.
Take for example the "segmented query path expression". It is handled fine with a minor modification as an Elixir quoted statement:
`quote do: @query_path//foo/goo[instance='a2']/blah/x[a='S 512']/y`
That yields a well defined AST tree base on the Elixir syntax:
```
{:@, [context: Elixir, import: Kernel],
[
{:path, [context: Elixir],
[
{:/, [context: Elixir, import: Kernel],
[ ...
```
The closures work as well by switching the dollar signs to ampersands: `quote do: select(@path/foo/&x/&goo/z/y/w, { value(&1) < 3 })`
Makes me want to write an Ecto adapter for dealing with Elixir Streams. =)
0: https://hackernoon.com/understanding-elixir-macros-3464e141434c https://hackernoon.com/understanding-elixir-macros-3464e1414...
1: https://github.com/elixir-ecto/ecto https://github.com/elixir-ecto/ecto
2: https://hexdocs.pm/gen_stage/GenStage.html https://hexdocs.pm/gen_stage/GenStage.html
- heavenlyblue 8y agoWhat’s with the macros? With Python you can build fully-featured DSLs without needing macros by simply overriding standard operators.
- 4thaccount 8y agoCan you give an example? I've never heard people talk about Python DSLs (just Lisp, Smalltalk, Rebol, and Forth usually).
- bollu 8y agoPolymage is a good example: http://mcl.csa.iisc.ac.in/polymage.html#examples http://mcl.csa.iisc.ac.in/polymage.html#examples Python embedded DSL for image processing / general matrix munging
- heavenlyblue 8y agoTheano: http://deeplearning.net/software/theano/library/index.html http://deeplearning.net/software/theano/library/index.html Define arbitrary execution graphs that compile to GPU or CPU.
- elcritch 8y agoTrue, you can but it results in a more limited DSL. This is mainly due to variable names needing to exist, be a valid object and chained used method invocations, which results in more awkward expressions (IMHO). Importantly macros enable compile time checking of a DSL rather than only runtime checking. That alone is very powerful. Take a SQL query in Python's SQLAlchemy: s = select([db.func.sum(users.c.id)]).\ .select_from( Users.outerjoin(Posts, Users.id == Posts.user_id)) where(and_( users.c.id == a1.c.user_id, users.c.id == a2.c.user_id, a1.c.email_address == 'jack@msn.com', a2.c.email_address == 'jack@yahoo.com' )).\ group_by(users.c.age) That's not bad, but compare it to a similar query in Elixir's Ecto: from account in App.Account, left_join: t0 in App.Transaction, on: t0.account_id == account.id and t0.user_id == ^current_user.id and t0.deleted == false and t0.type == "inflow", where: account.id == ^id and t1.user_id == t0.user_id, group_by: account.id, select: {account, {sum(t0.amount), sum(t1.amount)}}) The Elixir DSL allows a bit more flexibility when re-using language built-in's such as `and`/`or`, or undefined ones likes `sum` without needing the hacked names like `and_` or `db.func.sum` as in Python (or Java or others OOs). Generally the formatting is easier with macro based DSL's. P.S. The examples might not be exactly correct as I cobbled a couple of different ones together.
- heavenlyblue 8y agoI agree - the truth is that in Python it's easy to define DSLs only if they consist of expressions. It's not that easy to define an arbitrary language that allows you to write code that is close to something else completely (like Ecto vs SQL).
- virtualwhys 8y agoVery cool, does it compose? Some statically typed languages allow you to build up composable query snippets along the lines of (from a Scala DSL): val accountQ = for { (a, t) <- Account leftJoin Transaction (_.id is _.accountId) if a.id is ... && t.userId ... groupBy a.id } yield (a, t.id, t.amount.sum, ...) val invoiceQ = for { (a, transId, total) <- accountQ i <- Invoice join (_.transId is transId) } yield (a, i, total) Basically you can build up arbitrarily complex queries, all compile time checked -- it's really quite wonderful. Haskell has something similar with Esqueleto and C#/F# has LINQ to SQL. I love query DSLs, the more we move away from string-y SQL the better :)
- humanrebar 8y ago> It's easier to make a specialized language than commonly thought. Though generally the trickiest part is creating appropriate tooling for things like source maps, debugging, etc. Right. Maintenance, documentation, training, tooling, packaging, debugging, etc., of a DSL far outstrips its development cost. When you use regular expressions to solve a problem, you have two problems. And regular expressions have (basically) well understood parsers, logic generators, and so on. When you use a DSL to solve a problem, you have N problems. Maybe that's much, much better than the alternative (query languages are particularly interesting), but people definitely shouldn't stop at "this was easy to make... what's the big deal?"