mrkurt

12 ms·

121.
by mrkurt 4y ago
I think people who are earnest about the sentiment aren't screening specific languages. They're trying to find people who think hard about business problems and follow with implementation. We do this. Our hiring process is impossi
122.
by mrkurt 4y ago
You really only have to worry if HN goes away.
123.
by mrkurt 4y ago
i think the argument is: TikTok is a tool China will use to push propaganda. The problem is that a nation state is using a viral, addictive app for its own purposes. So the thing to stop everyone from is "letting a maybe-hostile foreig
124.
by mrkurt 4y ago
This would be more interesting if you did have time to go into details. I'd love to hear about the company getting eviscerated for messing up sales tax.
125.
by mrkurt 4y ago
Accurate.
126.
by mrkurt 4y ago
We got ourselves into a bind because of Nomad. We outlined a bunch of it here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchest... The short version is that "using Nomad and Consul for the type of glob
127.
by mrkurt 4y ago
Oh yes, I should edit. We launched this product a little over 2.5 years ago. We built a bunch of different things no one really wanted before that.
128.
by mrkurt 4y ago
It was really 7 out of 60, but yes. We ran into two problems: 1) the infra ops jobs is pretty intensely difficult here and 2) actually building an infra ops _org_ instead of just hiring individual engineers and overloading them seemed impor
129.
by mrkurt 4y ago
I appreciate that. I think we'll be in a really good spot in ~60 days.
130.
by mrkurt 4y ago
<3
131.
by mrkurt 4y ago
Not right now, no. We've been running this product for about 2.5 years.
132.
by mrkurt 4y ago
Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".
133.
by mrkurt 4y ago
We chose Nomad and adopted Consul as a result. Nomad and Consul work well together. I don't believe etcd would have been any better for us, though. Centralized service discovery that runs through raft consensus doesn't make a lot
134.
by mrkurt 4y ago
There are really only a few frameworks where our experience approaches Heroku. And even for those, it's only the newest versions. Phoenix, Rails, Laravel, and Remix are all pretty seamless to launch. Most others require pretty decent D
135.
by mrkurt 4y ago
You wouldn't necessarily know this from the outside, but we have _exceptional_ internal support when things go sideways. This is relatively new, up until about two months ago most incidents were run by 1.5 people. We had 7 people worki
136.
by mrkurt 4y ago
We still pipe service discovery through Consul, we just propagate it with a different, gossip based mechanism. Services are stored in local sqlite DBs on every host that runs our Proxy. They are designed to keep running, even when we can&#x
137.
by mrkurt 4y ago
Thank you! I'm both sorry it didn't work out (because $$$$) and also glad we didn't create any agony for you. Someday, we hope to create mild irritation for you, though, if we can.
138.
by mrkurt 4y ago
The design of Consul is wrong for what we need to do. Consul has been pretty good when it's running, but it's a huge pain in the ass to recover when it falls over. And when it does fall over, it's usually with no notice.
139.
by mrkurt 4y ago
We've scaled infra ops from 3 to 7 people in the past few weeks. Our very first VP was a VP Infra Ops, because that's the thing we have to get best at to succeed as a business. Note that we grew the whole company from 25 to 60 ove
140.
by mrkurt 4y ago
This has been a rough week, and I'm sorry we broke peoples' apps. We had a big Nomad outage on Monday, and then a suspiciously similar Consul outage today. Both tipped over faster than we could detect and mitigate, and we ended up
141.
by mrkurt 4y ago
I know blocking deploys sucks, I'm sorry. We disabled them to prevent otherwise healthy apps from going down. When Consul fails, we can't boot new app processes. The ones that are already running continue running. A restart is rou
142.
by mrkurt 4y ago
If you get any kind of loan from SVB, you're required to keep your cash with SVB. I think it was entirely reasonable for Series B and earlier startups to keep all their money in SVB. It was wrong, in hindsight, but reasonable. Bank fai
143.
by mrkurt 4y ago
We use Mercury, they're a frontend though. They're not a bank. Our checking/savings are with Choice Financial, and swept into at least four of these banks: https://co-mercury-prod.s3.amazonaws.com/legal/C
144.
by mrkurt 4y ago
I think that's what I said? There's a cost, we don't know what it is. It's not as high as it sounds when you say "we should only let people get $250k liquid and everyone else should pound sand". The cost to let
145.
by mrkurt 4y ago
$10mm of deposits at SVB is worth way more than $250k. We don't know exactly how much though. That money didn't just vanish, it got tied up in ways that make it hard for depositors to withdraw it. So no, it's not a depositor
146.
by mrkurt 4y ago
Why does FDIC insurance exist at all?
147.
by mrkurt 4y ago
That's an incredibly normal amount for a business. That's one payroll cycle for ~50 people making $75k per year each.
148.
by mrkurt 4y ago
Yes that's reasonable. That's probably not what's happening here. In some ways, the _promise_ of the FDIC getting depositors almost all their money back is enough. The biggest problem people have right now is time. Over time,
149.
by mrkurt 4y ago
Does the EU require 100% _liquid_ collateral? Because European banks are also at risk from bank runs. Any bank that makes loans is. SVB (probably) more than 100% deposit collateral. It just lost value very quickly.
150.
by mrkurt 4y ago
SVB isn't getting a bailout. That company is gone. Their shareholders lost everything they invested. Depositors don't get bailouts. That's not what bailouts are.

More ›