captaintobs

9 ms·

61.
by captaintobs 4y ago
It's easy to add errors for dialects by calling #unsupported when a function is used. In terms of versioning of engines, I haven't implemented that yet, but presumably it could be done by adding a dialect subclass and having versi
62.
by captaintobs 4y ago
This is what the sqlglot optimizer is used for. The optimizer converts EXISTS and IN into a canonicalized SQL (some variant of left join) which can then be compared to another query. So if you run the optimizer first and then the diff tool,
63.
How Airbnb Safeguards Changes in Production(medium.com)
3 points by captaintobs 4y ago|0 comments
64.
by captaintobs 4y ago
yep, we do the same but using duckdb, it's got a lot more analytical functions and closer to hive than sqlite
65.
by captaintobs 4y ago
I've added support now for any, all, and some. Some is identical to Any so I always default to Any. You can see what it took to add this in this commit. https://github.com/tobymao/sqlglot/commit/ab49a3a29
66.
by captaintobs 4y ago
SQLGlot supports nested queries and EXISTS. SOME, ANY, and ALL aren't fully supported yet but it's only a couple lines to add support since I already support EXISTS. https://github.com/tobymao/sqlglot/blo
67.
by captaintobs 4y ago
You should try it out and see. Here are the test cases https://github.com/tobymao/sqlglot/blob/main/tests/fixtures/... I can even parse and optimize TPC-H https://github.com/tob
68.
by captaintobs 4y ago
The parser is very permissive. You don’t need the exact dialect upfront, it handles unknown udfs internally as anonymous funcs.
69.
by captaintobs 4y ago
Id love to support more engines. PRs are very welcome!
70.
by captaintobs 4y ago
I don’t do join order optimizations because that relies on cardinality estimation, I leave that up to the physical plan. But there’s plenty more to optimize like predicate and projection pushdown.
71.
by captaintobs 4y ago
I actually built the system you work on :). It was the main inspiration for this project because PyPika is not a great experience for data scientists.
72.
by captaintobs 4y ago
Awesome, let me know if you have any questions or suggestions if you check it out!
73.
by captaintobs 4y ago
At the large tech companies I've been working at, there are many different big data engines that speak different dialects of sql (presto / spark). People write SQL queries in one language and want to run it in another, but it does
74.
by captaintobs 4y ago
i rarely ever use exists and prefer to do left joins or left semi joins (in spark) i'm not exactly sure what you're asking though, in terms of sql generation, it's not difficult for me because i just take in sql and output sq
75.
by captaintobs 4y ago
performance is not that important, usually the time to run a sql query takes 100ms > 1 hour, and so it doesn't matter that parsing it takes 0.5 ms.
76.
by captaintobs 4y ago
Author here, feel free to ask me any questions! Something that I'm working on is a pure python SQL engine https://github.com/tobymao/sqlglot/blob/main/sqlglot/executo... . It does the whole sheb
77.
by captaintobs 5y ago
I wrote an extremely performant and simple async worker framework called SAQ because I couldn't find any that fit my use case. https://github.com/tobymao/saq
78.
Show HN: SAQ – Simple Async Queues in Python based on Redis (includes a web UI)(github.com)
2 points by captaintobs 5y ago|0 comments
79.
by captaintobs 6y ago
Thanks! It’s not too difficult, to write one parser, the tricky part is supporting many variants. I learned a lot from this site https://craftinginterpreters.com/
80.
Show HN: SQLGlot – Customizable Python SQL Transpiler(github.com)
5 points by captaintobs 6y ago|2 comments
81.
Snabberb – A simple component view framework for Ruby Opal based on Snabbdom(github.com)
1 points by captaintobs 7y ago|0 comments
82.
Scaling Experimentation Methodology at Netflix with Compression(medium.com)
4 points by captaintobs 7y ago|0 comments
83.
by captaintobs 8y ago
Netflix | Software Engineer - Experimentation Platform | Los Gatos, CA | VISA | ONSITE Our team is looking for talented and motivated engineers to work on our experimentation platform. You'll build the tools and infrastructure for data
84.
by captaintobs 9y ago
Scribd | Senior Recommendations/Search Engineer | San Francisco | Toronto | VISA | ONSITE | Remote Scribd is a reading subscription that gives you access to the best books, audiobooks, magazines, and more. Our mission is to re-imagine
85.
by captaintobs 9y ago
Scribd | Senior Recommendations/Search Engineer | San Francisco | VISA | ONSITE Scribd is a reading subscription that gives you access to the best books, audiobooks, magazines, and more. Our mission is to re-imagine the way the world r
86.
by captaintobs 9y ago
Scribd | Senior Recommendations/Search Engineer | San Francisco | VISA | ONSITE Scribd is a reading subscription that gives you access to the best books, audiobooks, magazines, and more. Our mission is to re-imagine the way the world r
87.
by captaintobs 9y ago
Scribd | Senior Recommendations/Search Engineer | San Francisco | VISA | ONSITE Scribd is a reading subscription that gives you access to the best books, audiobooks, magazines, and more. Our mission is to re-imagine the way the world r
88.
by captaintobs 9y ago
Scribd | Senior Recommendations/Search Engineer | San Francisco | VISA | ONSITE Scribd is a reading subscription that gives you access to the best books, audiobooks, magazines, and more. Our mission is to re-imagine the way the world r
89.
by captaintobs 9y ago
Scribd | Senior Recommendations/Search Engineer | San Francisco | VISA | ONSITE Scribd is a reading subscription that gives you access to the best books, audiobooks, magazines, and more. Our mission is to re-imagine the way the world r
90.
by captaintobs 9y ago
Scribd | Senior Recommendations/Search Engineer | San Francisco | VISA | ONSITE Scribd is a reading subscription that gives you access to the best books, audiobooks, magazines, and more. Our mission is to re-imagine the way the world r

More ›