efxhoy

6 ms·

31.
by efxhoy 2y ago
One of my teachers did a paper on maritime disasters and concluded that women were more likely to die in shipwrecks, despite “women and children first”. The Titanic and Birkenhead were the exceptions. https://www.ifn.se/medi
32.
by efxhoy 2y ago
Strong migrations helps writing migrations that are safe for ZDD deploys. We use it in our rails app, catches quite a few potential footguns. https://github.com/ankane/strong_migrations
33.
by efxhoy 2y ago
I recently wrote a little data transfer service in python that runs in ECS. When developing it locally it was easy to handle SIGINT: try write a batch, except KeyboardInterrupt, if caught mark the transfer as incomplete and finally commit t
34.
by efxhoy 2y ago
> Running on-call well is a culture problem. You need management to prioritize observability (you can't fix what you can't show as being broken), then you need management to build a no-broken-windows culture (feature developmen
35.
by efxhoy 2y ago
Impossible, they even put “libre” in the name!
36.
by efxhoy 2y ago
Samba and tailscale.
37.
by efxhoy 2y ago
I think search needs to split in two. One isn’t even search, it’s generic questions: “how do I make pickled onions” type queries where an LLM can actually make sense of the millions of recipes and summarize them for you. I guess this is the
38.
by efxhoy 2y ago
Typically data warehouses are OLAP databases that have much better performance than OLTP databases for large queries. There might also be several applications in a company, each with their own database, and a need to produce reports based o
39.
by efxhoy 2y ago
We run an image scaler on aws lambda based on libvips. We cache the responses from it with cloudflare. We compared to letting cloudflare handle the scaling and the lambda was several times cheaper.
40.
by efxhoy 2y ago
No unfortunately not. We tried out snowflake and bigquery and landed on bigquery mainly because of how well it integrated with existing company workflows (a looot of google sheets floating around the org).
41.
by efxhoy 2y ago
We just ported our ~500gb data warehouse from an 8 core postgres rds instance to bigquery. Rebuilds went from 5 hours to 11 minutes and costs are about the same or lower. and that’s with caching of some very big tables in postgres and rebu
42.
by efxhoy 3y ago
I find this too at work all the time. People are very quick to branch off some “independent” feature into its own service or lib and repo. Great for them when building, tiny repos do have that greenfield clean slate feeling, but awful when
43.
by efxhoy 3y ago
I think it could be done with a monolithic app with a single db and truly stateless external services in a single repo. I only have experience from a single shop though and that’s not how the stuff we build ends up.
44.
by efxhoy 3y ago
The main problem is state, it’s too big and scattered. If it was just one db that knew everything that would be easy enough to snapshot, but there’s multiple dbs and s3 buckets and messages on queues and its all spread out. We could, with a
45.
by efxhoy 3y ago
I work at a 60 dev shop so not a startup per se. Our biggest problem is feature environments, or actual integration tests where multiple services have to change. Because infra is in its own repo in terraform and the apps have their own repo
46.
by efxhoy 3y ago
We’re adopting kubernetes to run “ai” with kubeflow. Modern problems require more problems taps forehead
47.
by efxhoy 3y ago
v1 or v2?
48.
by efxhoy 3y ago
Completely agree. It’s also a fantastic way to get to know how the org is doing, many people will be happy to share what parts of the product they’re happy with and where the dumpster fires are. I also think it’s useful for the people you’r
49.
by efxhoy 3y ago
I’m no computer scientist but I think the fundamental problem is that the CAP theorem makes it really tricky to do in a “cost free” way. You fundamentally need to sacrifice one, at least a tiny bit.
50.
by efxhoy 3y ago
Forgot to add this, which IMO is the canonical "Data warehouse on postgres" talk: https://www.youtube.com/watch?v=whwNi21jAm4 Lots of really good tips and info in there.
51.
by efxhoy 3y ago
I love postgres too. We went from a brittle event sourcing architecture to postgres_fdw, works really well. We do something similar with a few differences: We swap entire schemas on every run (one schema being one upstream database or coll
52.
by efxhoy 3y ago
Lovely! We just migrated from postgres 14 to 16 for 3 postgres clusters (servers) on RDS containing about 2TB of data across 8 databases. We were down from 00:00 to 04:00. Steps we took: * enabled our fallback "maintenance mode&quo
53.
by efxhoy 3y ago
Yes, the dataframe is very tricky to get right function signature wise. I used to write a lot of pandas heavy software and converged on writing most functions to take a dataframe or series and when possible just return a series, then put th
54.
by efxhoy 3y ago
I wrote scientific simulation software in academia for a few years. None of us writing the software had any formal software engineering training above what we’d pieced together ourselves from statistics courses. We wrote our simulations to
55.
by efxhoy 3y ago
No RAID. All the pictures and music I have duplicated on other devices. I’m ok with losing my video media.
56.
by efxhoy 3y ago
I run samba via docker compose and bind mounts. I use Infuse on my apple tv and ios devices.
57.
by efxhoy 3y ago
At home I run: Raspberry pi 4 8GB with a 22TB external hdd and 1tb external ssd. For uhhhh media. Runs *arr stack and samba. Raspberry pi os (debian) with everything defined in a docker compose file. Also runs tailscale. Having my entire me
58.
by efxhoy 3y ago
I keep queries in .sql files in git repo. I run longer queries by writing them in a file and including/running it with \i. There's also \e to open $PSQL_EDITOR.
59.
by efxhoy 3y ago
psql and the meta commands like \d to view schema of a table, combined with your regular text editor and \i is IMO the greatest postgres UI. I spend a lot of time there though so I can see the appeal of a gui for someone who only occasional
60.
by efxhoy 3y ago
I'm trying to peek at the data in the dev tools. It's doing a long running request to https://bfcm-globe-data-service-central.shopifycloud.com/pub... which is constantly downloading more data, but when I go to pe

More ›