4 ms·

I have worked with engineers who treated their work as glueing together various services to get the results required. They were absolutely shit engineers, whose

I have worked with engineers who treated their work as glueing together various services to get the results required. They were absolutely shit engineers, whose approach to building services for massive scale were simply 1. Glue together whatever services are available and seem to make sense. 2. Throw increasing amounts of traffic at it and check what fails 3. Try to patch around failure point 4. blame the service provider for "not scaling" 5. Ask for more and more machines 6. Exclaim how great their design is on the basis of the cool class hierarchies and interfaces they wrote. Teach "junior devs" their "OOP design skills and patterns" 7. Go to step 2 They think of themselves as great engineers who scale things. The reality is they are absolute shit engineers who have no clue what they are building, frequently have crazy O(n^2) loops in their code, dont recognize a graph problem if its staring them in their face. Examples 1. They needed a way to rerun map reduce jobs to regenerate data for previous 30 days. It was a straight forward, graph topological sort problem. Instead they built a crazy solution using multiple instances of Airbnb airflow. One server ran out of memory for these airflow instances, so they allocated a whole damn cluster of machines to run 30 instances of airflow! The job of the cluster? Issuing map reduce jobs to the hadoop cluster. They needed a cluster of machines just to issue commands to hadoop. They gave an elaborate tech talk about "scaling to 30X capacity" to process 30 days of data. Meanwhile, my team was processing 3 years of data. We considered hamake for our data rebuild process. Eventually, I wrote a 70 line scala topological sort routine that would do the job for us, because it gave us more flexibility than hamake and linked to our code libraries. 2. Some guy wrote a data pipeline to summarize a days results that took 24 hours to run, thanks to 40 different joins. I rewrote it to use a single group by. The code ran in 10s of minutes. He didn't know how hadoop worked or external sorting, beyond writing arbitrary sql. So much for looking up algorithms when required. 3. Guys presents beautiful OOP code that has major race conditions in a distributed environment. I point out the race condition. Keeps proposing hacks to "solve the problem". Does not understand transaction processing or locking algorithms or what options are available to him. Hadn't heard if read write locks. What are the pros or cons. Doesn't understand the difference between local locks and distributed fault tolerant locks. I explain it to him. He eventually comes up with a solution using zookeeper that will need zookeeper to process 1000s of updates per second. I give up. "DSA, what DSA? They are for kids. I will look up an algorithm when I need it".

YZF 5y ago
You're missing the point. What you're describing has nothing to do with memorizing data structures and algorithms. Having those guys memorize those won't make them better engineers, right? Sounds you understand locking and distributed locks. Cool. Your next project is a video encoder. Write an arithmetic coder in a 20 minute interview? Motion detection? Your next project is a chess engine. Write minimax with alpha-beta cutoff in an interview? Your next project is public key cryptography ... A distributed k/v database... Can you write a bloom filter for me in an interview? Gradient descent in some machine learning application? If you're such a great software engineer, which likely you are, you should be able to do all the above (at least somewhat), by doing your research, and writing code. I'm not saying hire bad engineers. I'm saying the game is silly. Sure, a good engineer (and people that aren't so good) can play the game. Also not arguing with the person that said if your goal is to get paid better and you want that job then play the game. Doesn't make it less silly.
- random314 5y ago
  You are missing the point > Having those guys memorize those won't make them better engineers, right? They can't realistically memorize all the algorithms, they have to develop an understanding of graphs and algorithmic complexity to be able to get through. They should be able to map arbitrary problems to a corresponding graph problem. These are absolutely fundamental. At least I should be able to communicate to them, "use a topological sort" and they should be able to understand what I am talking about. Sure, some one could set aside their job for 1 year and memorize everything and they might just get through the interview loop. But it is impossible to have a perfect filter, however DSA and complexity analysis is a bare minimum. > Your next project is a video encoder. Write an arithmetic coder in a 20 minute interview? Motion detection? Your next project is a chess engine. Write minimax with alpha-beta cutoff in an interview? Your next project is public key cryptography ... A distributed k/v database... Can you write a bloom filter for me in an interview? Gradient descent in some machine learning application? Funnily enough, none of these questions are asked in a FAANG interview at all. So, I don't know why you are constructing a strawman and demolishing it. The questions actually asked are from basic CS201. These are very specialized questions. FWIW, I have already built a bloom filter, minhash and consistent hash at work and I am a machine learning engineer. So, I can implement an in memory consistent hash store, bloomfilter, minhash, count min sketch and gradient descent in an interview setting comfortably. For motion detection a 2 dimensional derivative function should work. A simple delta between 2 images is a basic implementation . Huffman coding is not too complex to write. The basic entropy formula is pretty simple. In fact, most ML classifiers minimize cross entropy loss, so I am very familiar with entropy. Minimax - I have partly forgotten. But if the interviewer prompts me with some high level details I should be able to do it. I have actually been asked to derive gradient descent in a Google staff eng equivalent interview and also demonstrate that gradient descent converges to global minima for the single layer perceptron[1]. Mind you, this was a specialist ML IC6+ position. Not a new college hire loop. [1] You need to prove that the Hessian is positive semi definite. I did the basic set up but couldn't complete the derivation. However, I got an offer.

YZF 5y ago

You're missing the point. What you're describing has nothing to do with memorizing data structures and algorithms. Having those guys memorize those won't make them better engineers, right? Sounds you understand locking and distributed locks. Cool. Your next project is a video encoder. Write an arithmetic coder in a 20 minute interview? Motion detection? Your next project is a chess engine. Write minimax with alpha-beta cutoff in an interview? Your next project is public key cryptography ... A distributed k/v database... Can you write a bloom filter for me in an interview? Gradient descent in some machine learning application? If you're such a great software engineer, which likely you are, you should be able to do all the above (at least somewhat), by doing your research, and writing code. I'm not saying hire bad engineers. I'm saying the game is silly. Sure, a good engineer (and people that aren't so good) can play the game. Also not arguing with the person that said if your goal is to get paid better and you want that job then play the game. Doesn't make it less silly.

random314 5y ago
You are missing the point > Having those guys memorize those won't make them better engineers, right? They can't realistically memorize all the algorithms, they have to develop an understanding of graphs and algorithmic complexity to be able to get through. They should be able to map arbitrary problems to a corresponding graph problem. These are absolutely fundamental. At least I should be able to communicate to them, "use a topological sort" and they should be able to understand what I am talking about. Sure, some one could set aside their job for 1 year and memorize everything and they might just get through the interview loop. But it is impossible to have a perfect filter, however DSA and complexity analysis is a bare minimum. > Your next project is a video encoder. Write an arithmetic coder in a 20 minute interview? Motion detection? Your next project is a chess engine. Write minimax with alpha-beta cutoff in an interview? Your next project is public key cryptography ... A distributed k/v database... Can you write a bloom filter for me in an interview? Gradient descent in some machine learning application? Funnily enough, none of these questions are asked in a FAANG interview at all. So, I don't know why you are constructing a strawman and demolishing it. The questions actually asked are from basic CS201. These are very specialized questions. FWIW, I have already built a bloom filter, minhash and consistent hash at work and I am a machine learning engineer. So, I can implement an in memory consistent hash store, bloomfilter, minhash, count min sketch and gradient descent in an interview setting comfortably. For motion detection a 2 dimensional derivative function should work. A simple delta between 2 images is a basic implementation . Huffman coding is not too complex to write. The basic entropy formula is pretty simple. In fact, most ML classifiers minimize cross entropy loss, so I am very familiar with entropy. Minimax - I have partly forgotten. But if the interviewer prompts me with some high level details I should be able to do it. I have actually been asked to derive gradient descent in a Google staff eng equivalent interview and also demonstrate that gradient descent converges to global minima for the single layer perceptron[1]. Mind you, this was a specialist ML IC6+ position. Not a new college hire loop. [1] You need to prove that the Hessian is positive semi definite. I did the basic set up but couldn't complete the derivation. However, I got an offer.