4 ms·

That's pretty cool. Feature docs including an example: https://murex.rocks/docs/optional/select.html https://murex.rocks/docs/optional/select.html So apparentl

That's pretty cool. Feature docs including an example: https://murex.rocks/docs/optional/select.html https://murex.rocks/docs/optional/select.html So apparently it runs the query against an in-memory sqlite instance. I'm curious what the rough time overhead for that is (compared to, say, the overhead of calling out to grep). Does it feel noticeably slow to run a query on ps output? And the other thing I'm wondering, can this drop you into a sqlite command line? I.e., » ps aux -> select -i sql> SELECT ...

jhgb 5y ago
Also a comparison with http://www.strozzi.it/cgi-bin/CSA/tw7/I/en_US/NoSQL/ http://www.strozzi.it/cgi-bin/CSA/tw7/I/en_US/NoSQL/ or similar might be in order if we're talking about shell pipelines and relational data.
hnlmorg 5y ago
> I'm curious what the rough time overhead for that is (compared to, say, the overhead of calling out to grep). Does it feel noticeably slow to run a query on ps output? It took 0.2 seconds to run the command on my relatively modest 4 yr old MBP. Running a similar pipeline (below) took 0.4 seconds. But there are optimisations that can be made to the command line: time {ps aux -> awk {print $1} -> sort -> uniq -c -> sort} This is just loading 510 rows of data though. While sqlite3 is very well optimised for loading in millions of records, the interface that sits between sqlite3 and the shell isn't (yet). Currently it reads in the entire dataset then imports it into the database. The reason it does this is so that it can work with any tabulated data format that is supported by the shell eg jsonlines that might look like the following: ["Name", "Session", "Score", "Completed"] ["Gilbert", "2013", 24, true] ["Alexa", "2013", 29, true] ["May", "2012B", 14, false] ["Deloise", "2012A", 19, true] However since UNIX pipes run concurrently it would mean if you were to load in millions of lines, the `sort`/`uniq` version would be better optimised I would imagine. Grep certainly would be. The long term plan for this feature I've written is to make it streamable. In fact I do already have some draft code written to do that. But I figured I'd get an early draft of this out there to gauge peoples thoughts. > And the other thing I'm wondering, can this drop you into a sqlite command line? Currently no. And it's not something I'm looking to add support for because there's already a lot of decent SQL tools out there. But I was thinking of adding an option to write the database to disk as a standard .db file and you can then use your favourite interactive sqlite3 tool to query the data.
- spockz 5y ago
  > However since UNIX pipes run concurrently it would mean if you were to load in millions of lines, the `sort`/`uniq` version would be better optimised I would imagine. Iirc, sort still needs to load the input in memory to be able to sort correctly, I presume. Uniq indeed works on the stream directly.

jhgb 5y ago

Also a comparison with http://www.strozzi.it/cgi-bin/CSA/tw7/I/en_US/NoSQL/ http://www.strozzi.it/cgi-bin/CSA/tw7/I/en_US/NoSQL/ or similar might be in order if we're talking about shell pipelines and relational data.

hnlmorg 5y ago

> I'm curious what the rough time overhead for that is (compared to, say, the overhead of calling out to grep). Does it feel noticeably slow to run a query on ps output? It took 0.2 seconds to run the command on my relatively modest 4 yr old MBP. Running a similar pipeline (below) took 0.4 seconds. But there are optimisations that can be made to the command line: time {ps aux -> awk {print $1} -> sort -> uniq -c -> sort} This is just loading 510 rows of data though. While sqlite3 is very well optimised for loading in millions of records, the interface that sits between sqlite3 and the shell isn't (yet). Currently it reads in the entire dataset then imports it into the database. The reason it does this is so that it can work with any tabulated data format that is supported by the shell eg jsonlines that might look like the following: ["Name", "Session", "Score", "Completed"] ["Gilbert", "2013", 24, true] ["Alexa", "2013", 29, true] ["May", "2012B", 14, false] ["Deloise", "2012A", 19, true] However since UNIX pipes run concurrently it would mean if you were to load in millions of lines, the `sort`/`uniq` version would be better optimised I would imagine. Grep certainly would be. The long term plan for this feature I've written is to make it streamable. In fact I do already have some draft code written to do that. But I figured I'd get an early draft of this out there to gauge peoples thoughts. > And the other thing I'm wondering, can this drop you into a sqlite command line? Currently no. And it's not something I'm looking to add support for because there's already a lot of decent SQL tools out there. But I was thinking of adding an option to write the database to disk as a standard .db file and you can then use your favourite interactive sqlite3 tool to query the data.

spockz 5y ago
> However since UNIX pipes run concurrently it would mean if you were to load in millions of lines, the `sort`/`uniq` version would be better optimised I would imagine. Iirc, sort still needs to load the input in memory to be able to sort correctly, I presume. Uniq indeed works on the stream directly.