wenc

12 ms·

151.
by wenc 1y ago
Can you give me an example? Spell check only checks if a word is in dictionary. It doesn’t check grammar or context.
152.
by wenc 1y ago
I'm in my late 40s and have Instagram on desktop. I watch reels from time to time. The clips I watch are definitely >6 seconds (more like 30s to 1.5 minutes -- there are very few clips under 10 seconds on my feed). And agree with em
153.
by wenc 1y ago
Teflon rice cookers don't really release PFAS at appreciable levels. Teflon is remarkably stable (and one of the reasons it was DuPont's miracle discovery). PTFE's degradation point is 260C, but rice cookers only cook up to
154.
by wenc 1y ago
Can be approximated yes. Approximated well? No, but you can get away with it sometimes with saturation functions like softmax. But badly. It’s like trying to solve an integer program as a linear program. You end up with a relaxation that is
155.
by wenc 1y ago
Deep learning specifically yes. It needs a manifold assumption. But not data in general which was what I was responding to.
156.
by wenc 1y ago
Many types of data don’t. Disconnected spaces like integer spaces don’t sit on a manifold (they are lattices). Spiky noisy fragmented data don’t sit on a (smooth) manifold. In fact not all ML models treat data as manifolds. Nearest neighbor
157.
by wenc 1y ago
I often see Finance and IT departments merged, usually with the CIO reporting to the CFO or some variant of that. Both Finance and IT are cost centers that manage assets, risks, audit and compliance, and the mindset behind both are similar.
158.
by wenc 1y ago
> I do not believe that people are more productive after about 4-5 hours a day of work. Off topic: I think that’s generally true in an 8 hr work day, but I find in a 10 hr work day I can actually be productive for up to 7 hrs. It’s a non
159.
by wenc 1y ago
That’s not bad. I had to get from JFK to midtown during peak hours. It was Airtrain ($8.50) + LIRR to Woodside ($11) + Subway 7 train to midtown ($2.90) = $22.40. (I didn’t know LIRR had city ticket, it would have been $16.40. But it took 1
160.
by wenc 1y ago
This sounds like something Via is doing https://ridewithvia.com/ I signed up for Via in Chicago but it didn’t quite work out for me. I guess Uber’s network is bigger so high probability of coincidence routes.
161.
by wenc 1y ago
Yes and no. They produce “nice” people who are polite, but distant. Dreary places encourage introversion so people are much less warm. People are more inward looking and private. They want to mostly keep to themselves. Seattle is known for
162.
by wenc 1y ago
Can confirm map. Seattle is truly dreary.
163.
by wenc 1y ago
Btw, you can plug your comment into ChatGPT and it'll give you a reasonable answer. The short answer is: distortions.
164.
by wenc 1y ago
Did Athena on CSV work for you? I've used Athena and it struggles with CSV at scale too. Btw I'm not suggesting to use Spark. I'm saying that even Spark didn't work on large TSV datasets (it only takes a JOIN or GROUP B
165.
by wenc 1y ago
It doesn’t usually choke on 500 gb of data. I query 600 gb (equivalent to a few TBs of CSVs?) of parquets daily. It’s not the size of the data. It’s the type of data. If date partitioning doesn’t work, just find another chunking key. The ke
166.
by wenc 1y ago
Typically small data is batched. While theoretically you could, I wouldn't create 1 file per row (there would be too many files and your filesystem would struggle). But maybe you can batch 1 day's worth of data (or whatever partit
167.
by wenc 1y ago
I take photos so I can go back to them as a different person and see details I didn't see the first time. I first visited Chicago in 2006 and took a bunch of pictures. It was all a blur to me. Later I became a Chicago resident in 2017,
168.
by wenc 1y ago
CSV is a pretty bad format any engine will choke on it. It basically requires a full table scan to get at any data. You need to convert it into Parquet or some columnar format that lets engines do predicate pushdowns and fast scans. Each pa
169.
by wenc 1y ago
Parquet files are already built for append only. Just add a new file. This is a new paradigm for folks who aren’t in big data — the conventional approach usually involves doing a row INSERT. In big data, appending simply means adding a new
170.
by wenc 1y ago
If the data is in Parquet they are already indexed in a sense. No further indexing necessary. If they are stored in DuckDB’s native format (which I don’t use), it supports some state of the art indices. https://duckdb.org/do
171.
by wenc 1y ago
Thank you for your work on OLX. I miss Turbo Vision. It was way better than many of the TUIs of the day, not only on DOS but also UNIX.
172.
by wenc 1y ago
Reprojection is accurate locally but inaccurate at scale. Geodesics are the most accurate (Vincenty etc) but are computationally heavy. Haversine is a nice middle ground.
173.
by wenc 1y ago
DuckDB can be thought of as an analytics engine rather than storage engine. It’s OLAP rather than OLTP.
174.
by wenc 1y ago
This is the more trivial use case (static data, heavy reads) and DuckDB is absolutely optimized for this use case. DuckDB also provides a vectorized, parallelized engine. When I run a query all of my 32 cores light up on htop.
175.
by wenc 1y ago
I don't know if it's ever been a core skill. You can ask any dev. Many tell me SQL is not something they use every day, so even if they learned it, they usually forget it in no time. SQL is burned into my muscle memory because I w
176.
by wenc 1y ago
I don't quite agree. SQL queries are one area where correctness matters a lot, because downstream applications rely on them to be correct. If you get a query wrong, especially in an ETL process, you can generate a lots of garbage data
177.
by wenc 1y ago
> importing them into postgres shouldn't be that long and then you can do the same or more than with DuckDB. Usually new data is generated regularly and would require creating a separate ETL process to ingest into Postgres. With Duc
178.
by wenc 1y ago
That requires data to already be in Postgres, otherwise you have to ETL data into it first. DuckDB on the other hand works with data as-is (Parquet, TSV, sqlite, postgres... whether on disk, S3, etc.) with requiring an ETL step (though if t
179.
by wenc 1y ago
Probably no difference for your use-case (ST_Distance). If you already have data in Postgres, you should continue using Postgis. In my use case, I use DuckDB because of speed at scale. I have 600GBs of lat-longs in Parquet files on disk. If
180.
by wenc 1y ago
How is the adoption among DevOps folks? I'm still getting feedback that many devs are not too comfortable with reading and writing SQL. They learned simple SELECT statements in school, but get confused by JOINs and GROUP BYs.

More ›