kwillets

6 ms·

31.
by kwillets 1y ago
I've been reading up on lakehouse stuff, and everything seems to agree with the idea that data warehouse functionality deteriorates as they try to accommodate raw data access. The databricks people suggest that a better "cache for
32.
by kwillets 1y ago
I don't know; my neighbors never talked about it. But one guy came back from retirement after 9/11 and moved into a trailer in the Lockheed parking lot.
33.
by kwillets 1y ago
I see Iceberg as a nice complement to a real data warehouse, as a way to manage raw data files and ETL's that load them. Getting file updates as transactions a few times a day seems a lot cleaner. But I constantly underestimate the ten
34.
by kwillets 1y ago
Extended-width multiplication works, but the cost of extra random bits is often a lot higher than the range arithmetic. Somewhere in my github there's an indefinite-width multiply that only adds bits while there's a risk of carry
35.
by kwillets 1y ago
Some folks I work with have a terabyte of short string records that they regularly scan with regexes to develop classification criteria; it's a prime application for a substring index that can accelerate their queries, but the scale is
36.
by kwillets 1y ago
>The judge expressed his "sorrow at this devastating turn of events, and my sympathy and deepest condolences", adding that he "admired" Mr Lynch despite ruling against him. The UK is looking increasingly like a third
37.
by kwillets 1y ago
Snowflake seems to have peaked; 2023 was hellish dealing with roomfuls of inexperienced devs and even architects convinced it was the fastest cheapest thing ever.
38.
by kwillets 1y ago
Kakao ended the English names a year or so ago due to inevitable confusion, but now people are supposed to add the "nim" (dear) suffix to each other's Korean names instead, which sounds creepy. The funny part of the Kakao CEO
39.
by kwillets 1y ago
Snowflake is becoming the Juicero of data.
40.
by kwillets 1y ago
There's a why page in their docs, but basically it's cheaper than a multitasking OS for doing the same thing. The valueprop of Erlang is lightweight threads and message-passing. My knowledge of microcontrollers is dated, but the f
41.
by kwillets 1y ago
The UK has cemented its place in tech as the land without Sarbanes-Oxley.
42.
by kwillets 1y ago
Stonebraker is one of the few whose criticism is listened to. He recently updated his "What goes around comes around" paper; it's worth a read: https://db.cs.cmu.edu/papers/2024/whatgoesaround-sigmod
43.
by kwillets 1y ago
Cloud and SaaS were good for a while because they took away the old sales-CTO pipeline that often saw a whole org suffering from one person's signature. But they also took away the benefits of a more formal evaluation process, and nowa
44.
by kwillets 1y ago
It's been a commodity for decades now. Metrics like price-performance have a long history, but the SnowBricks products fail at them quite dramatically. The difference is hard-sell vs. soft or no-sell.
45.
by kwillets 1y ago
It's only serverless in the way it commits transactions to cloud storage, making the server instance ephemeral; otherwise it has a server process with compute and in-memory buffer pool almost identical to pg, with the same overheads.
46.
by kwillets 1y ago
That's an interesting reference. I came up with what may be a simpler case of that -- a structure that estimates the last time a key was seen. It's an upper bound, as collisions make the time more recent, but never the other way.
47.
by kwillets 1y ago
Nothing in C-store seems to have sunk in. In clickhouse's case I can forgive them since it was an open source, bootstrap type of project, and their cash infusion seems to be going into basic re-engineering, but in general slowly re-imp
48.
by kwillets 1y ago
Late Materialization, 19 years later. https://dspace.mit.edu/bitstream/handle/1721.1/34929/MIT-CSA...
49.
by kwillets 2y ago
I don't know much about the financial side of the company, but it seems like a client-led effort by telco's etc. against the dreck that tech VC's keep pushing on them. That can't translate into decent salaries unfortunat
50.
by kwillets 2y ago
Zero is certainly on my scale. Bonus points if you build a server and keep it under your desk.
51.
by kwillets 2y ago
I spent the past year puzzling over the DB market as well, but I don't feel like I'm much closer to understanding it. It appears that a lot of attention is now directed at the folks doing 100 MB queries, and the high end has moved
52.
by kwillets 2y ago
The animation, pseudocode, and join order discussion all imply that the cartesian product is being generated.
53.
by kwillets 2y ago
The description of nested loop join is confusing; it's mainly a single pass through the outer table with one B-tree probe per row to each inner table. The linked paper is clearer: "However, the inner loops in the join are typicall
54.
by kwillets 2y ago
SSD's are also a bit of an achilles heel for AWS -- they have their own Nitro firmware for wear levelling and key rotations, due to the hazards of multitenant. It's possible for one EC2 tenant to use up all the write cycles and th
55.
by kwillets 2y ago
That's my guess as well. Bitstring rank/select is a well-known problem, and the BMI and non-BMI (Hacker's Delight) versions are available as a reference.
56.
by kwillets 2y ago
I've never run across it, but I would describe the job of an analytics engineer as doing this over and over for analysts -- it's probably semantically clearer to do the join first and then aggregate, so I end up pushing it down fo
57.
by kwillets 2y ago
To follow up, it's about 3x faster: SELECT pickup.zone AS pickup_zone, dropoff.zone AS dropoff_zone, cnt AS num_trips FROM (select pickup_location_id, dropoff_location_id, count(*) as cnt from taxi_data_2
58.
by kwillets 2y ago
One possible hand optimization is to push the aggregation below the joins, which makes the latter a few hundred rows.
59.
by kwillets 2y ago
It's a sad example of my C++ knowledge that I didn't know about the checked add intrinsics, but I do know that unsigned (x + y) < x compiles into checking the carry flag on most compilers.
60.
by kwillets 2y ago
Conspiracy theories are common in repressive regimes.

More ›