wenc

36 ms·

511.
by wenc 2y ago
It’s also the aggregation operation. If there are many unique groups it can take a lot of memory. Newer DuckDbs are able to handle out of core operations better. But in general just because data fits in memory doesn’t mean the operation wil
512.
by wenc 2y ago
What is the error message you’re getting? Or is it simply that the query will not finish? (Does it continue to run?) Parquet files are compressed, and many analytic operations require more memory than the on disk size. When you don’t have e
513.
by wenc 2y ago
Are you using it for simple SQL retrieval or complex analytic queries? They’re both similar for the former use case, but DuckDB — being an analytic engine — supports the latter use case much better.
514.
by wenc 2y ago
The problem is America is that tipping has become a norm, and it's hard to put the genie back in the bottle. You have 3 parties with non-aligned goals. Staff will optimize for earnings, so they actually prefer tips. Many oppose a minim
515.
by wenc 2y ago
> We’re decades into the pattern. I continue to be stunned that anyone is surprised by Chinese product anymore. The problem with this kind of thinking is that it is "narrative based" thinking. This type of thinking typically as
516.
by wenc 2y ago
> OS makers (here mainly Microsoft and Apple) to increase margins via including tracking and ads into the system Not sure I see it on Apple. macOS is free. There are no ads in it. I don't see ads on Windows either, unless you mean b
517.
by wenc 2y ago
I really appreciate it! Thanks.
518.
by wenc 2y ago
Agreed. On a scale of 10 in terms of current technology, CSV is a 1 while Parquet is 7. ORC is maybe 7.2. But parquet is far more ubiquitous than ORC (I’ve never seen ORC in prod but I also have limited sample sizes) I’m sure there are more
519.
by wenc 2y ago
Do you have any details on this? Duckdb over vanilla S3 has latency issues because S3 is optimized for bulk transfers, not random reads. The new AWS S3 Express Zone supports low-latency but there's a cost. Caching Parquet reads from va
520.
by wenc 2y ago
Parquet the most popular columnar format. (owing to support in Spark and various other big data tools, as well as local tools like pandas, polars and duckdb) It's technically not the very best format (ORC has some advantages), but it&#
521.
by wenc 2y ago
>But in reality looks like parsing CSV string works faster than bloated and overengineered parquet format with libs. Anecdotally having worked with large CSVs and large on-disk Parquet datasets, my experience is the opposite of yours. My
522.
by wenc 2y ago
itertools (iterators) and collections (data structures) are both underrated modules in stdlib.
523.
by wenc 2y ago
Yes, it does that, assuming you read in the entire CSV, which works for CSVs that fit in memory. With Parquet you almost never read in the entire dataset and it's fast on all the projections, joins, etc. while living on disk.
524.
by wenc 2y ago
I would love to see the benchmarks. That is not my experience, except in the rare case of a linear read (in which CSV is much easier to parse). CSV underperforms in almost every other domain, like joins, aggregations, filters. Parquet lets
525.
by wenc 2y ago
I don't think devs need to be nerds -- after all, it's just a job. FAANG devs I meet are rarely nerds -- they studied CS in college and got into tech because it fit their aptitudes and paid well. They work in sprints, clear out ba
526.
by wenc 2y ago
I don’t mind critical but considered comments. But I find so many many comments are critical and ill-informed. They’re contrarian but unnuanced. And people with high karma (>15k) overwhelmingly seem to be the least insightful and most pe
527.
by wenc 2y ago
On a related note, I took an online course at the UChicago Graham School a couple of years ago on How To Read Classic Texts. [1] We used Mortimer Adler's "How to Read a Book" as our guide. It was only a 3-4 week class ($275,
528.
by wenc 2y ago
Anyone know the backstory behind Apple including logo stickers in their products? I own lots of Apple products and thus I have a bunch of stickers, but I can't think of any surface I'd stick them on. They'd just seem misplace
529.
by wenc 2y ago
Hence the shortcomings of limited sampling (and data). I know many people with both devices in my circles (including me). Typically the laptop is for work and the iPad is for travel or for kids.
530.
by wenc 2y ago
I remember Google being marginally better than Altavista but not much more. The cool kids in those days used Metacrawler, which meta searched all the search engines.
531.
by wenc 2y ago
1 USD = 156 JPY For years it has been 1 USD = 110 JPY. Japan is also overrun by tourists right now.
532.
by wenc 2y ago
I wear my Apple Watch at all times except when I shower. I charge it when I shower and it lasts till the next time I shower. I'm not German so I do shower every day, which makes this routine work.
533.
by wenc 2y ago
I think there’s a difference between saying “ok I don’t know how to do this, what’s an easier way?” Rather than going on a rant that “bunch of parquet files sucks!” When it’s clearly untrue and all that was needed was knowledge of the tools
534.
by wenc 2y ago
Athena is great. It’s essentially querying your data using lambdas on your S3 data. DuckDB outperforms it slightly on a single node but Athena is a good choice if you need elastic scaling.
535.
by wenc 2y ago
I appreciate the caveats. Notice I mentioned analytics use cases which is a subset of all the software engineering use cases. CSVs have their place as a simple interchange format. They’ve been overextended to use cases they were never mean
536.
by wenc 2y ago
This is true. There are two tools you need to know for this: duckdb and visidata. With these tools, Parquet is almost as easy as CSVs (but a few orders of magnitude more powerful and faster) Parquet is also usable in polars and pandas, and
537.
by wenc 2y ago
If you’re dealing with non trivial data sizes and if you have an analytics use case and you’re using CSV, you’re already doing it wrong. You bring up partitioning — that can help performance but with Parquet, you can get performance they is
538.
by wenc 2y ago
The problem with Parquet on S3 is not parquet but S3. S3 is great for bulk reads but very high latency for random reads. It’s not really designed for data querying use cases. There are two ways around this: parquet on S3 Express Zone (a new
539.
by wenc 2y ago
I would say on the contrary, please stop dumping CSV data just because that’s all you know all to deal with. CSV is full of gotchas which is why CSV readers have so many switches. They are huge (and hugely inefficient at scale), have no typ
540.
by wenc 2y ago
Kialo looks interesting but it looks like it ends up being a binary tree of pros and cons? (Ie debate outcome) One shortcoming of debates is assuming that there are only two positions. I get that it’s a simplification to help us manage our

More ›