My own experience says that you should not attempt to make a serious change without being able to evaluate whether or not your attempt has worked. That could b

My own experience says that you should not attempt to make a serious change without being able to evaluate whether or not your attempt has worked. That could be as simple as fixing a bug - you need a test that would have caught the bug and it (and all your other tests) needs to pass once your fix is in. That could be as complex as something like this, improving your recommendation system. To evaluate something like this you'd likely need a few key metrics, latency and throughput, sure, but also recommendation quality. Maybe click through rate? Growth in recommended subs? Time that users spend on reddit with and without recommendation? Things like that. If you can't evaluate your change then you will be stumped by a very simple question. Did you succeed? Are you done? If you fix a bug by thinking "I know what the problem is", then change a few lines of code and say "Done" how do you know you actually fixed it? How do you know you didn't make things much worse? Same thing here. So, you changed the recommendation system - did you make it better or worse? I was quite interested to see how they were evaluating the system and whether or not it was an improvement. Their section on evaluation says "Scaling this component of the system is something we think about a lot and are actively working on." That seems really, really, backwards to me. You are thinking about evaluating it after you've changed the system? Huh?