iknownothow

10 ms·

61.
by iknownothow 3y ago
I would consider the following optimizations first before attempting to rewrite an HTTP API since you already did the hard part: 1. For multiples processes use `gunicorn` [1]. Runs your app across multiple processes without you having to to
62.
by iknownothow 3y ago
It makes me think of "cute aggression"
63.
by iknownothow 3y ago
The problem is, which "database"? If you know the data is always going to be in the range of ~1TB, use Postgres or some other ACID database. But I don't know of any petabyte scale ACID compliant database. Also I'm not su
64.
by iknownothow 3y ago
I'm very interested in such products. It appears to be similar to Materialize DB. I use dbt and Postgres for managing a data warehouse. It appears Risingwave doesn't have support for dbt yet, there is an open issue [1]. Materializ
65.
by iknownothow 3y ago
Question for OP - How big is the company you work for? I work for a company that has around 50 technical employees and I'd say use of LLMs is putting pressure on the company against hiring more employess rather than actually laying peo
66.
by iknownothow 3y ago
I don't know much about him apart from him being an amazing story teller. I've listened to this story told by him about 25 times. I hope y'all enjoy it too. It's called the LA speed check story. https://www.yo
67.
by iknownothow 3y ago
If this is you, then take a moment and empathize with yourself. If you realize that your situation can't be helped then embrace opening packets and not feeling guilty about being unable to cook. Once you let go of the guilt, your inner
68.
by iknownothow 3y ago
I had investigated if this was possible to do but I couldn't find a way to do it myself. It's possible that pd.Timestamp might be intricately tied interally to vectorization optimizations using numpy or others. Therefore separatin
69.
by iknownothow 3y ago
I know some of the HN crowd are gonna wag their finger at me but the moment I have to deal with timestamps in Python, even for the most simplest of cases, I parachute in pandas. `pd.Timestamp` (and its friends like `pd.Timedelta`) in my opi
70.
by iknownothow 3y ago
Does anyone know how many Falcon 9s were required to make it to 200 flights? I assume they didn't actually make 200 rockets.
71.
by iknownothow 3y ago
We use dbt at work. The entire analytics lineage (not the data) is laid bare for all to see. There is no analytics done outside of dbt. If PII were to enter into the lineage accidentally, it's easy to spot and remove. You also get to k
72.
by iknownothow 3y ago
My opinion, comparing Americans with Germans I see and I find two obvious differences: * On average Americans are likely to be obese and very likely to be overweight. * Fit & muscular Americans are much bulkier than fit Germans.
73.
by iknownothow 3y ago
No thanks. Here's why I dislike it. There's a lot of libraries out there in Python that appear to be DSLs or have custom syntaxes that heavily use decorators etc e.g. Flask or FastAPI. In exchange for learning these custom syntaxe
74.
by iknownothow 3y ago
If you're a beginner absolutely give it a try! I implemented all of the book almost 10 years ago in Python and pygame. I was starting to code professionally back then and it is responsible for much of the confidence I needed. It defini
75.
by iknownothow 3y ago
I'm a data engineer and I only know Python. It appears Golang hits a sweetspot on many metrics such as performance, parallelism, ease of use etc and since 2016 there's been a lot of new data products and tools written in Golang. S
76.
by iknownothow 4y ago
That's interesting! I don't have much experience with Clickhouse, especially not in the last two years. I'll have to try this out myself. That's a pretty incredible if it can handle batching internally.
77.
by iknownothow 4y ago
The total dataset size doesn't seem much at ~1TB. So you can use Clickhouse for fast analytics like others have suggested. Consider hosting on Hetzner since they have cheap Nvme disks. The biggest problem you're going to face is i
78.
by iknownothow 4y ago
If it's a database or similar product, then a docker image. If it's a utility that I can test against my databases, then a binary :)
79.
by iknownothow 4y ago
Many many demo gifs or videos for as many use cases you can imagine right in your home page. Preferably these demos should progress from the simplest use cases to more complex ones. Show me the code snippet of the hairiest aggregation. Show
80.
by iknownothow 4y ago
You can watch the full interview here [1]. Jon Stewart knew he was just grilling a person of power with no real significance. The real underlying problem was laid out in an example in Jon Stewart's closing comments from 01:05:00 to 01:
81.
by iknownothow 4y ago
I'm a visual learner and you have wired my intuitions about this topic permanently. Amazing work!
82.
by iknownothow 4y ago
1. How many insertions do you expect per second or per minute? 2. What's the size of each insert? 3. At the end of one year, what's the total size of your dataset? 4. How long can your largest and most complex analytical query tak
83.
by iknownothow 4y ago
We use Postgres as a Datawarehouse, once AlloyDB Omni is stable there is virtually no reason for us not to use it. Obviously, we use Postgres not for its performance but for its ergonomics, first class dbt support and unbeatable extension e
84.
by iknownothow 4y ago
Cleaning my cats' litter boxes. I turn the vacuum cleaner on and put the hose into the box so it sucks all the bad air out. It also sucks all of the dust out when I'm scooping and when I put in new litter. I also set the air purif
85.
by iknownothow 4y ago
Almost all European companies will be inclined to use Azure in EU regions to get data privacy guarantees than use OpenAI's API directly.
86.
by iknownothow 4y ago
How much memory would a spark worker need to process a single JSON file that is 25GB? To clarify, this is not JSONL or NDJSON file. Just a single JSON object.
87.
by iknownothow 4y ago
I'm not a prompt engineer but prompt engineering is a real skill* and you can specialize for different models. For example, a good and well informed prompt engineer will have intuitions that GPT like LLMs are bad at arithmetic tasks ev
88.
by iknownothow 4y ago
It looks incredible and I truly hope your project takes off for my sake since I have to work with Kafka from time to time! I had a quick look through the code snippets and docs so maybe I missed it, what happens if the message received on t
89.
by iknownothow 4y ago
What you're describing is the common symptom among many programmers which is their unfamiliarity with disk limitations. It's not the database's fault for not being able to write 10M rows to the disk quickly. Disks, especially
90.
by iknownothow 4y ago
Sorry but in my opinion you can't beat free styling with Excalidraw and a bunch of nerds leaning back in their own basements and home offices thinking deeply and being critical of ideas. The overhead of having such meetings is next to

More ›