I think that everything I'm willing to list publicly is in that post. Understand that almost all of it is written in pure C. [Edited to add: I do not use Java i

I think that everything I'm willing to list publicly is in that post. Understand that almost all of it is written in pure C. [Edited to add: I do not use Java in any part of ontolo.] An example is that I do use Redis for some queueing operations. But there's a bottleneck in part of the process (not with Redis, but the nature of being unable to POP multiple items off of a list at once using the blocking pop requests, requiring multiple requests over a socket, or two separate data stores; one to notify, another to hold the data). So I'm in the process of writing my own queue for the crawler system. I'm guessing that your question comes from a curiosity or desire to build something similar, or something similarly performant. I don't know if this is a useful perspective or not, but it's mine: that every time I don't know what's happening to a byte of data and how it's processed, there's the possibility of some unknown amount of inefficiency there, and that inefficiency could be small or large. But the not knowing is the problem. So I've tried to eliminate as many of those as possible. I don't always get it right, and I'm sure there are many areas for improvement. But if I find a performance bottleneck, I go in and edit my own code, rather than having to learn someone else's codebase that is written to handle edge- and use-cases I'll never need. I then either have to modify their code - and have to maintain future updates as well, etc - or I write my own, and keep refining it. I'm certainly in no position to give any sort of programming advice to anyone, but that's just the philosophy I've had from the beginning while I worked on this. And I think it's an important one that's helped me get the performance I've achieved with it. It's a conversation I have often with other friends who are programmers and who make more liberal use of tech stacks. But those are often problems of a very different nature than the ones I'm interested in solving. And if you do hope to build something similar, or something different but similarly performant, be sure to test everything. You don't need amazing tests, just consistent tests with consistent units of measure. Learn where the compiler is optimizing things out and how to make sure that doesn't happen, etc (I will often insert an integer that gets incremented by a random number generated by a fast xorshift prng at the critical place of measure, in order to increase the odds the compiler won't optimize it out), to ensure a level of quality in your tests. And when you're working at the byte level, keep refining the operations that run the most frequently and test all possibly ways you could imagine doing it.