6 ms·

This was linked to me right after coming out of a meeting with my product team where I was the engineer in this video. Worse, I had a large hand in defining how

This was linked to me right after coming out of a meeting with my product team where I was the engineer in this video. Worse, I had a large hand in defining how our software architecture is designed so I can't even blame other people. It's largely my fault.

makstaks 7y ago
Thanks for sharing this, not easy to admit. In retrospect what would you have done differently though?
- bsaul 7y ago
  "why i screwed up" is the kind of talk you don't see often, and yet it's probably as instructive as the other ones.
- nkassis 7y ago
  That's a good question, a lot of the issues we hit now are mostly due to early assumption informed by our business requirements at the time that didn't hold true long term (argh blaming the business trope sorry). Our company being a a startup still learning market fit for our product went from being SASS first to doing mostly on prem and private cloud installations for example. That changes a lot of how I would have approached the design to reduce operational complexity had that been obvious early on. Similarly, we did a lot of learning from the tools we used that is probably very much just something you earn with battle scars. For example the behavior and issues we hit with our choice of core stack components (celery, redis, rabbitmq, kubernetes,...) are often things you learn by using them and gaining experience. The next thing is there were a lot of conscious tradeoffs that I would still argue we would take today. We prioritized speed of building features to make the product viable and attractive to customers at the expense of quality of those features and reliability. Doing the hard real world testing instead of relying on simple unit/integration tests before shipping. That's something we now are having to address but the question is was that something we should have done differently? No sure I would, we may not be around now if we hadn't built the feature we needed to sell the product and prove market viability. On the topic of this video. I think the issue it not using microservices or building a monolith but controlling complexity. The biggest thing I dislike about our current infrastructure is the cognitive load and complexity that a new engineer joining has to deal with to be productive. That can happen in any architecture if you aren't careful. Sure having to run multiple services, tracing and testing acrross them (basically the usual gripes about microservice architectures) dealing with inter-service calls, performance tradeoff etc.. is not as good as I'd like it to be but it could be improved. What sucks most is there a lot of raw edges we left exposed that trip new people and that I think was a mistake we should be prioritizing fixing those as they come up and think of them while buidling up our core framework and components more thoroughly.
raghava 7y ago
Is there a method/framework to measure/manage architectural complexity? Please share any pointers you might have!
- nkassis 7y ago
  Observe how well and quickly new engineers come up to speed in your system. What problems do they hit? Think about them while designing. For example what hinders them from solving issues as they come up with customers. Their experience in my view is a representation of the architectural complexity and cognitive load you are inflicting with your design.

makstaks 7y ago

Thanks for sharing this, not easy to admit. In retrospect what would you have done differently though?

bsaul 7y ago
"why i screwed up" is the kind of talk you don't see often, and yet it's probably as instructive as the other ones.
nkassis 7y ago
That's a good question, a lot of the issues we hit now are mostly due to early assumption informed by our business requirements at the time that didn't hold true long term (argh blaming the business trope sorry). Our company being a a startup still learning market fit for our product went from being SASS first to doing mostly on prem and private cloud installations for example. That changes a lot of how I would have approached the design to reduce operational complexity had that been obvious early on. Similarly, we did a lot of learning from the tools we used that is probably very much just something you earn with battle scars. For example the behavior and issues we hit with our choice of core stack components (celery, redis, rabbitmq, kubernetes,...) are often things you learn by using them and gaining experience. The next thing is there were a lot of conscious tradeoffs that I would still argue we would take today. We prioritized speed of building features to make the product viable and attractive to customers at the expense of quality of those features and reliability. Doing the hard real world testing instead of relying on simple unit/integration tests before shipping. That's something we now are having to address but the question is was that something we should have done differently? No sure I would, we may not be around now if we hadn't built the feature we needed to sell the product and prove market viability. On the topic of this video. I think the issue it not using microservices or building a monolith but controlling complexity. The biggest thing I dislike about our current infrastructure is the cognitive load and complexity that a new engineer joining has to deal with to be productive. That can happen in any architecture if you aren't careful. Sure having to run multiple services, tracing and testing acrross them (basically the usual gripes about microservice architectures) dealing with inter-service calls, performance tradeoff etc.. is not as good as I'd like it to be but it could be improved. What sucks most is there a lot of raw edges we left exposed that trip new people and that I think was a mistake we should be prioritizing fixing those as they come up and think of them while buidling up our core framework and components more thoroughly.

raghava 7y ago

Is there a method/framework to measure/manage architectural complexity? Please share any pointers you might have!

nkassis 7y ago
Observe how well and quickly new engineers come up to speed in your system. What problems do they hit? Think about them while designing. For example what hinders them from solving issues as they come up with customers. Their experience in my view is a representation of the architectural complexity and cognitive load you are inflicting with your design.