3 ms·

grep -rE '```\w+' |uniq seems like the same query across text files to me. it's not "trivial" but at least grep has a man page and people have used it reliabl

by fjorde 5y ago

grep -rE '```\w+' |uniq seems like the same query across text files to me. it's not "trivial" but at least grep has a man page and people have used it reliably for decades now. Spinning up whatever it takes to get groq going is not the same thing as having widely available tools that just work. Somehow structuring content in json is a new idea? There are a lot of html to json parsers out there already. jq and grep can do everything his post suggests and require no less or more investment from a dev perspective.

atombender 5y ago
Correct me if I'm wrong — my use of Jq is limited to basic shell stuff — but I believe Jq does not have joins at all. You can do joining by writing your own functions, but it's not built into the syntax. GROQ was designed to query and join multiple sources of data, and to make it easy to plan for efficient execution (e.g. on top of a relational database or search engine). For example: *[_type == "author"] { _id, name, "books": *[_type == "book" && author == ^._id] { title, year, publisher-> // Join referenced publisher genres[]-> // and all the genres ] } This gives you all the authors with each author's books. You can do a lot of the same transformations that Jq does, but you can also do it across multiple collections of data; it's not a pipeline with an implicit input. (Disclosure: I work at Sanity.)
- fjorde 5y ago
  I don't know if jq can do joins but getting a list of codeblock languages is a pretty simple task for almost anyone in a text file. For complex queries I agree that you typically need a more expressive query language. I'm really not judging this syntax based on the 2 examples I've looked at, but I agree that I can mostly read all the words and stuff, but without the comments etc this is very dense and I don't think it's "intuitive". Compared to sql it's interesting, and I hope developers like it. I just am almost automatically revolted by the marketing hype train language of the last 15 years. None of the hype comes close to being accurate and it's just all so misleading. AWS services are among the most notorious in this regard, having sold an entire class of devs on the notion that aws would "abstract away complexity" only to have devs learning about db indices not at the cost of a slow query execution time for some customers, but $50,000 row-scan bills. AWS hype beasts have been "abstracting away complexity" for years just to tell the poor sap devs that they need intimate knowledge of query execution plans, for some poorly conceived and managed service aws put together after ripping off some os community. And the connection to aws is not immaterial. Every query is potentially business critical and also susceptible to tanking the entire business. I don't think we should necessarily try to treat queries as "trivial" or whatever.
  - atombender 5y ago
    Actually, I'm not sure anything can be truly "intuitive". I think you're really talking about offering familiar concepts. In this case, the syntax might make more sense if you think of GROQ as a superset of JSON, because this is valid GROQ: { "a": 42, "b": [true, null] } But we also have functions, subqueries, joins, and so on: { "allArticles": *[_type == "article"] } Here, the asterisk simply means the array of "all objects", and [] is used to apply a filter predicate. To me, I don't find Jq at all intuitive. The fact that filtering a collection is expressed this way, for example: .[] | select(.type == "article") is not "intuitive", because it doesn't look like anything else. The pipe operator certainly behaves very differently from Unix shells; it's the "[]" part that causes iteration, not the pipe. I absolutely see your point about abstracting away complexity. Developers have a lot of knowledge in their heads about how SQL translates to execution complexity, and GROQ is more opaque. We do have an "explain" mode that shows the query plan, but we are still thinking about ways to make the execution model more transparent and understandable.
    - fjorde 5y ago
      > To me, I don't find Jq at all intuitive. Same here. Intuitive is probably the wrong word. I think I'm still looking for the right word to use to describe the idea. I think there was some progress from a fb team that translated english language queries into sql. I hope there is more progress in the field so that the same people who write the content can search it and use it like a dev would. I think that given a stack of 5K blog posts, any dev can pump out a dozen charts and word graphs or whatever given some time, and it would all look pretty good but probably not have as much value as an editor's summary. But if the editor could also make use of the dev tooling to do more in depth research or produce some new insights, or get to the "important" information more quickly, than that would represent a step in the right direction.
- deleted 5y ago
  [deleted]
- fjorde 5y ago
  I did probably misstate the overall capabilities of jq and grep compared with groq. I wasn't intending to criticize the product and I'm excited about all the offerings in this space that find a niche or provide wider value. Joins across markdown is a great idea and I'm glad this exists.

by fjorde 5y ago

atombender 5y ago

Correct me if I'm wrong — my use of Jq is limited to basic shell stuff — but I believe Jq does not have joins at all. You can do joining by writing your own functions, but it's not built into the syntax. GROQ was designed to query and join multiple sources of data, and to make it easy to plan for efficient execution (e.g. on top of a relational database or search engine). For example: *[_type == "author"] { _id, name, "books": *[_type == "book" && author == ^._id] { title, year, publisher-> // Join referenced publisher genres[]-> // and all the genres ] } This gives you all the authors with each author's books. You can do a lot of the same transformations that Jq does, but you can also do it across multiple collections of data; it's not a pipeline with an implicit input. (Disclosure: I work at Sanity.)

fjorde 5y ago
I don't know if jq can do joins but getting a list of codeblock languages is a pretty simple task for almost anyone in a text file. For complex queries I agree that you typically need a more expressive query language. I'm really not judging this syntax based on the 2 examples I've looked at, but I agree that I can mostly read all the words and stuff, but without the comments etc this is very dense and I don't think it's "intuitive". Compared to sql it's interesting, and I hope developers like it. I just am almost automatically revolted by the marketing hype train language of the last 15 years. None of the hype comes close to being accurate and it's just all so misleading. AWS services are among the most notorious in this regard, having sold an entire class of devs on the notion that aws would "abstract away complexity" only to have devs learning about db indices not at the cost of a slow query execution time for some customers, but $50,000 row-scan bills. AWS hype beasts have been "abstracting away complexity" for years just to tell the poor sap devs that they need intimate knowledge of query execution plans, for some poorly conceived and managed service aws put together after ripping off some os community. And the connection to aws is not immaterial. Every query is potentially business critical and also susceptible to tanking the entire business. I don't think we should necessarily try to treat queries as "trivial" or whatever.
- atombender 5y ago
  Actually, I'm not sure anything can be truly "intuitive". I think you're really talking about offering familiar concepts. In this case, the syntax might make more sense if you think of GROQ as a superset of JSON, because this is valid GROQ: { "a": 42, "b": [true, null] } But we also have functions, subqueries, joins, and so on: { "allArticles": *[_type == "article"] } Here, the asterisk simply means the array of "all objects", and [] is used to apply a filter predicate. To me, I don't find Jq at all intuitive. The fact that filtering a collection is expressed this way, for example: .[] | select(.type == "article") is not "intuitive", because it doesn't look like anything else. The pipe operator certainly behaves very differently from Unix shells; it's the "[]" part that causes iteration, not the pipe. I absolutely see your point about abstracting away complexity. Developers have a lot of knowledge in their heads about how SQL translates to execution complexity, and GROQ is more opaque. We do have an "explain" mode that shows the query plan, but we are still thinking about ways to make the execution model more transparent and understandable.
  - fjorde 5y ago
    > To me, I don't find Jq at all intuitive. Same here. Intuitive is probably the wrong word. I think I'm still looking for the right word to use to describe the idea. I think there was some progress from a fb team that translated english language queries into sql. I hope there is more progress in the field so that the same people who write the content can search it and use it like a dev would. I think that given a stack of 5K blog posts, any dev can pump out a dozen charts and word graphs or whatever given some time, and it would all look pretty good but probably not have as much value as an editor's summary. But if the editor could also make use of the dev tooling to do more in depth research or produce some new insights, or get to the "important" information more quickly, than that would represent a step in the right direction.
deleted 5y ago
[deleted]
fjorde 5y ago
I did probably misstate the overall capabilities of jq and grep compared with groq. I wasn't intending to criticize the product and I'm excited about all the offerings in this space that find a niche or provide wider value. Joins across markdown is a great idea and I'm glad this exists.