3 ms·

Thanks for sharing this, not easy to admit. In retrospect what would you have done differently though?

bsaul 6y ago
"why i screwed up" is the kind of talk you don't see often, and yet it's probably as instructive as the other ones.
nkassis 6y ago
That's a good question, a lot of the issues we hit now are mostly due to early assumption informed by our business requirements at the time that didn't hold true long term (argh blaming the business trope sorry). Our company being a a startup still learning market fit for our product went from being SASS first to doing mostly on prem and private cloud installations for example. That changes a lot of how I would have approached the design to reduce operational complexity had that been obvious early on. Similarly, we did a lot of learning from the tools we used that is probably very much just something you earn with battle scars. For example the behavior and issues we hit with our choice of core stack components (celery, redis, rabbitmq, kubernetes,...) are often things you learn by using them and gaining experience. The next thing is there were a lot of conscious tradeoffs that I would still argue we would take today. We prioritized speed of building features to make the product viable and attractive to customers at the expense of quality of those features and reliability. Doing the hard real world testing instead of relying on simple unit/integration tests before shipping. That's something we now are having to address but the question is was that something we should have done differently? No sure I would, we may not be around now if we hadn't built the feature we needed to sell the product and prove market viability. On the topic of this video. I think the issue it not using microservices or building a monolith but controlling complexity. The biggest thing I dislike about our current infrastructure is the cognitive load and complexity that a new engineer joining has to deal with to be productive. That can happen in any architecture if you aren't careful. Sure having to run multiple services, tracing and testing acrross them (basically the usual gripes about microservice architectures) dealing with inter-service calls, performance tradeoff etc.. is not as good as I'd like it to be but it could be improved. What sucks most is there a lot of raw edges we left exposed that trip new people and that I think was a mistake we should be prioritizing fixing those as they come up and think of them while buidling up our core framework and components more thoroughly.

"why i screwed up" is the kind of talk you don't see often, and yet it's probably as instructive as the other ones.

That's a good question, a lot of the issues we hit now are mostly due to early assumption informed by our business requirements at the time that didn't hold true long term (argh blaming the business trope sorry). Our company being a a startup still learning market fit for our product went from being SASS first to doing mostly on prem and private cloud installations for example. That changes a lot of how I would have approached the design to reduce operational complexity had that been obvious early on. Similarly, we did a lot of learning from the tools we used that is probably very much just something you earn with battle scars. For example the behavior and issues we hit with our choice of core stack components (celery, redis, rabbitmq, kubernetes,...) are often things you learn by using them and gaining experience. The next thing is there were a lot of conscious tradeoffs that I would still argue we would take today. We prioritized speed of building features to make the product viable and attractive to customers at the expense of quality of those features and reliability. Doing the hard real world testing instead of relying on simple unit/integration tests before shipping. That's something we now are having to address but the question is was that something we should have done differently? No sure I would, we may not be around now if we hadn't built the feature we needed to sell the product and prove market viability. On the topic of this video. I think the issue it not using microservices or building a monolith but controlling complexity. The biggest thing I dislike about our current infrastructure is the cognitive load and complexity that a new engineer joining has to deal with to be productive. That can happen in any architecture if you aren't careful. Sure having to run multiple services, tracing and testing acrross them (basically the usual gripes about microservice architectures) dealing with inter-service calls, performance tradeoff etc.. is not as good as I'd like it to be but it could be improved. What sucks most is there a lot of raw edges we left exposed that trip new people and that I think was a mistake we should be prioritizing fixing those as they come up and think of them while buidling up our core framework and components more thoroughly.