4 ms·
It does seem a bit of a strange challenge - a bit reminiscent of high school math problems where understanding the question was as much part of it as actually s
by HarHarVeryFunny 9mo ago
It does seem a bit of a strange challenge - a bit reminiscent of high school math problems where understanding the question was as much part of it as actually solving the problem when you understood it.
Since the focus of the challenge appears(?) intended to be optimization, not reverse engineering, it's a bit odd that they don't give a clear statement of what the kernel is meant to be computing. Perhaps the challenge is intended to be a combination of the two, but then the correct reverse engineering part of it becomes a gate for the optimization part, else you'll be solving the wrong problem.
Given the focus on results achieved by Opus 4.5, maybe that's the main point - to show how well Opus can reverse engineer something like this. If they gave the actual clear problem statement, then maybe you could brute force an optimal solution using tree search.
- HarHarVeryFunny 9mo agoI just threw this prompt at Gemini, and it seems (I haven't analyzed the problem to see if it is correct), to be able to extract a clear understanding of the problem, and a specification for the kernel. "Can you "reverse engineer" what the kernel in this optimization exercise is actually doing - write a specification for it? https://github.com/anthropics/original_performance_takehome https://github.com/anthropics/original_performance_takehome" Gemini says it's doing inference on a random forest - taking a batch of inputs, running each one through each decision tree, and for each input outputting the sum of these decision tree outputs - the accumulated evidence.
- HarHarVeryFunny 9mo agoSo looking at the actual code (reference_kernel() in problem.py), this "random forest inference" is completely wrong! It's doing some sort of binary tree traversal, but the hashing and wrap around looks weird - maybe just a made up task rather than any useful algorithm?
- saagarjha 9mo agoYes, it’s made up.
- fc417fc802 9mo agoThis isn't "reverse engineering" it's merely "being able to read fairly simple code you didn't write". A much simpler version of the kernel is provided at the end of problem.py as reference_kernel2. If you can't make sense of such a small codebase or don't immediately recognize the algorithm that's being used (I'm guilty of the latter) then you presumably aren't someone that they want to hire.
- HarHarVeryFunny 9mo agoFair enough, and there are clues in the comments too, but why not just provide the specification of the kernel (inputs and outputs) as part of the problem?
- fc417fc802 9mo agoThey do. They provide reference_kernel which shows the algorithm itself, build_mem_image which shows the data format you will be working with, and finally reference_kernel2 which implements said algorithm on said data format. They then provide you with a very naive implementation that runs on their (very simple) VLIW architecture that you are to optimize. If at the end of that someone is still lost I think it is safe to say it was their goal that person should fail.
- HarHarVeryFunny 9mo agoWell, yes, they have a reference implementation as documentation, just as they have the simulator as documentation for the ISA ... The problem is about pipelining memory loads and ALU operations, so why not just give clear documentatation and state the task rather than "here's a kernel - optimize it"? \_(ツ)_/
- fc417fc802 9mo agoPresumably that is only one of two purposes, with the other being to test your ability to efficiently read, understand, and edit low level code that you didn't write. I imagine you'd regularly run into raw PTX if you worked for them in the relevant capacity. And perhaps a third purpose is to use the simulator to test your ability to reason about hardware that you are only just getting familiar with.