4 ms·
Thank you for the detailed feedback! Great insight and makes a lot of sense. > Particularly in open source, you see remarkably little investment in performanc
by szvsw 2y ago
Thank you for the detailed feedback! Great insight and makes a lot of sense.
> Particularly in open source, you see remarkably little investment in performance engineering or scalability, so it is always straightforward to write your own implementation that is many times faster on the same hardware. Even basic things like popular data format conversion libraries are surprisingly consistently suboptimal. In most cases, the kinds of changes we'd want to make cannot be upstreamed to that library; most open source libraries heavily optimize for simplicity and the widest set of users to the exclusion of maximum performance.
Yeah, I have even encountered this myself once or twice - i.e. re-implementing some optimization/search algos, sampling processes, or numerical simulation methods to be more tightly coupled to our requirements/context for great performance gain - despite not being in a context where heavy performance engineering has become necessary yet. This obviously overlaps with your point about using only a tiny subset of library functionality.
> Another issue is the impracticality of threading fine-grained metrics and observability through a hodgepodge of libraries that were never designed to be used that way or to work together. This is a big deal if you want to heavily automate operations.
Yes, I personally have not ever run in to this because of the relatively small scales I am operating at but makes perfect sense. Same goes for the security concerns. Obviously these are more serious concerns for larger and longer timescale projects (as discussed in the original blog post).
> engineer-month here, a few engineer-months there
I've only worked in contexts where there are no more than 5 simultaneous active devs on a project, so the long-term investment for longer term gains is probably in higher tension with the short term needs. obvious red flag for increasing technical debt/deferred maintenance/downstream complexity, but still can be difficult to justify diverting 20% of team's total productivity for a long term investment when there are short term demands. But always good to be reminded that those long term investments shouldn't be rejected outright and should be more seriously considered.
> I also like that this forces a discussion of actual requirements for that functionality. Far too much software makes their requirements whatever a library offers. I arrived at this point gradually over many, many years. The practical maintainability and opportunity for optimization have been key. In far too many cases, it turns out that a library dependency could be replaced with a couple hundred lines of code that actually worked better and which matched the design and style of the rest of the code base. There is a learning curve but once you've written a common class of library once, it is straightforward to do it again.
This is a really excellent and concise way of summarizing it. I'm definitely going to try to push this with my team more often and now have a great resource for how to articulate it. Thanks again for taking the time to explain your perspective.