2 ms·
I built bonsai.io/websolr.com while working at Twitter in 2010. I left in 2012, keeping a promise to a co-founder that I'd leave if we hit $1M ARR. There were
by fizx 8y ago
I built bonsai.io/websolr.com while working at Twitter in 2010. I left in 2012, keeping a promise to a co-founder that I'd leave if we hit $1M ARR.
There were definitely a couple months of pager/ops overload where I hit 100hr workweeks and multiple hundreds of pagers per week. Hypervisor bugs meant that some tasks were hard to automate, and that automation would stochastically fail, so I stayed up til 4am watching the logs and correcting issues, then went back into the office at 10am.
Basically, it was a reasonable hedging strategy against either company's failure, but given both succeeded, I lost a good chunk of what I would have made if I was "all in" on either. Plus, it was about as miserable as it sounds, though I learned a ton.
- shdh 8y agoAny insightful takeaways from "watching the logs" and "correcting issues" in terms of what you would do differently?
- fizx 8y agoIf you aren't automatically rolling instances frequently, you will inevitably have some machines that were theoretically cattle but in practice are pets. (This is much easier today with k8s/mesos) More generally, when you claim in system design that you have invariants, if you aren't using automation to enforce your invariants, they will inevitably be bypassed to your detriment by eager and/or lazy engineers. You will find this out at the worst possible time.