4 ms·
It's a system design question. High throughout on the scale of Google means you'd laser any log ingestion service to death. He was probably looking for a distr
by dumpster_fire 4y ago
It's a system design question. High throughout on the scale of Google means you'd laser any log ingestion service to death.
He was probably looking for a distributed way to label logs and build an ingestion service that shards regionally with high availability, but still able to search and filter within an acceptable time for debugging purposes.
- Vayl 4y agoHow can I learn about these topics? Junior self-taught dev, I read these things and kind of see what they're about, but I wouldn't know where to even start with this...
- shubhamkrm 4y agoDesigning Data Intensive Applications (DDIA)[1] is a great introductory book for such things. It covers a broad number of system design topics, and you can then go into detail for stuff that interests you. Other than that, several companies (such as Uber) operating on a large scale maintain engineering blogs. [1] https://dataintensive.net/ https://dataintensive.net/
- sowbug 4y agoYou can practice the mindset by writing a toy solution for the problem, then listing the reasons why you consider your solution just a toy. This develops a fundamental skill of engineering, which isn't understanding whether a system will fail (it will always eventually fail, even if the first reason is that the sun explodes), but rather when and how it will fail. You can make a whole career out of pulling on that thread. (I worked at Google with Bob in the mid-2000s. I might have interacted with him once or twice on the internal Java and Guice mailing lists, but never met him in person. He was supremely helpful and knowledgeable, and I admired him from a distance.)