dmoura

9 ms·

31.
by dmoura 5y ago
The idea was to focus on querying tools. ujson and orjson (as well as the json module from python's standard library) offer json decoding and decoding but not a querying language: you need to implement the query logic in Python, result
32.
by dmoura 5y ago
:-D out of curiosity, jq, one of the tools in the benchmark, is written in C
33.
The fastest tool for querying large JSON files is written in Python (benchmark)(colab.research.google.com)
175 points by dmoura 5y ago|84 comments
34.
Show HN: Create Matplotlib visualizations from the command-line(github.com)
11 points by dmoura 5y ago|0 comments
35.
Gistemp Climate Spiral 1880-2021 (visualization)(svs.gsfc.nasa.gov)
2 points by dmoura 5y ago|0 comments
36.
FOSDEM: Free event for software developers to meet, share ideas and collaborate(fosdem.org)
18 points by dmoura 5y ago|2 comments
37.
Fig – Your Terminal, Reimagined(fig.io)
2 points by dmoura 5y ago|0 comments
38.
by dmoura 5y ago
yep, unfortunately due to security concerns is not supported everywhere (DB as a service): `ERROR: Extension "plpythonu" is not supported by Amazon RDS`
39.
by dmoura 5y ago
Demo available here: https://vimeo.com/danielcmoura/spyqldemo
40.
by dmoura 5y ago
:-) It’s an open-source tool, so it should be community-driven. If you can share an example I can try to understand if spyql can make your life easier.
41.
by dmoura 5y ago
Thank you :-) You can use https://github.com/mikefarah/yq to convert yaml to json on the fly and pipe it into spyql. There’s an example in the Readme.
42.
by dmoura 5y ago
Thank you! One of my main goals was making data processing in the command-line more accessible and intuitive. If you use a shell you can leverage an extensive array of tools. please take a look at the recipes in the Readme. The shell is man
43.
by dmoura 5y ago
Nice! It would perfectly fit my use-case if it supports the following flow: 1. Person A makes a research and collects and ranks a set of products 2. When done, person A sends to person B 3. Person B looks at the result and provides feedback
44.
by dmoura 5y ago
Yes, column storage is great, specially when you only need a small subset of columns as there much less reading operations. I have experience working with Parquet files, so I can see the benefits of CTFs, even if we don't take into acc
45.
by dmoura 5y ago
Agree! There’s a lot of code you don’t see behind a simple SELECT statement. I would also say that when writing SQL you start thinking on the end result, describing what you want to get and not so much how you want it done. This is also par
46.
by dmoura 5y ago
Thank you!
47.
by dmoura 5y ago
The tool has a flow: stdin -> spyql -> stdout You can choose the data format of the input and output. One of the output formats writes INSERT statements to stdout, enabling to import data to any SQL database. Currently, spyql does not
48.
by dmoura 5y ago
Interesting! I will take a look
49.
by dmoura 5y ago
Yes it leverages eval. I did not know about RBQL... I will check it out! Also curious here, there a lot of non-trivial decisions :-)
50.
by dmoura 5y ago
I am just very lazy! :-)
51.
by dmoura 5y ago
I know OctoSQL, I found the project super interesting, especially as you can merge different data sources. SPyQL is very single-stream oriented, much in line with command line tools (e.g. awk, grep, sed). I should be implementing soon a lef
52.
by dmoura 5y ago
Thank you! I know DSQ, great work! Unlike q and others it manages JSONs, which is great. Yes, there are a number of tools that use an in-memory DB. I work with JSONs and CSVs with several GBs, and loading them to sqlite is not an option, it
53.
by dmoura 5y ago
Thank you! Made my day :-)
54.
Show HN: SPyQL – SQL with Python in the middle(github.com)
105 points by dmoura 5y ago|45 comments
55.
by dmoura 5y ago
I am writing a language and CLI that mixes SQL with Python (spyql: https://github.com/dcmoura/spyql ), and it is interesting to see that we are tackling some of the same problems: - code/formula reutilization wher
56.
SPyQL – SQL with Python in the Middle(github.com)
69 points by dmoura 5y ago|1 comments
57.
Freeing the data scientist mind from the curse of vectoRization(towardsdatascience.com)
1 points by dmoura 7y ago|0 comments