4 ms·

Too easy. But can your LLM understand this Go code and explain to you how it works? func extract(x, m uint64) uint64 { x &= m var ( ho

Too easy. But can your LLM understand this Go code and explain to you how it works? func extract(x, m uint64) uint64 { x &= m var ( holes = ^m shift = uint(1) ) for ; shift < 64; shift *= 2 { odd := holes odd ^= odd << 1 odd ^= odd << 2 odd ^= odd << 4 odd ^= odd << 8 odd ^= odd << 16 odd ^= odd << 32 holes &^= odd move := x & odd x ^= move x |= move >> shift } return x } What does this compute, and how is that computation achieved? More broadly, I'm curious to know how much of the BUGFIX-66 debugging gauntlet your LLM can solve. You have to Google it, because I'm not allowed to link to it on Hacker News. Banned.

botanical76 2y ago
What is your point? Does anyone really believe LLMs are capable of understanding such dense, undocumented, and advanced code today? Or is the state of AI today - specifically at this nontrivial task - supposed to predict that it will never be able to?
- AnimalMuppet 2y ago
  First: Undocumented? So what? Either it can understand the code, or it can't. (And it better not depend on the comments, same as a programmer shouldn't, because they can be lies.) You think it's unreasonable to expect it to understand 21 lines? Of go, one of the least-dense languages out there? I think it actually demonstrates the point: (some) LLMs really cannot read and understand code if it doesn't fit in with their training data. They don't know the language, they know the patterns they have seen. If it doesn't fit, they can't reason about it from a knowledge of the language syntax, because they don't actually have such knowledge.
  - botanical76 2y ago
    This will probably come down to your perspective and experience. The function is written in Go, but this is not relevant, because it does not use any Go-specific constructs. This is solely about the algorithm that is implemented. I don't understand the function provided, so I would need some comments to have a chance to parse it. That said, I feel I am an experienced programmer, so I would not fault ChatGPT for being confused with this function. Perhaps you find understanding this class of algorithm - doing bit manipulation in a loop - relatively easy, or even trivial. It would make sense if you expect an LLM to breeze through this task if that were the case. Today, a computer is able to work with natural language, and even perform useful tasks with it. It can write and understand surprisingly complex code, and sometimes it works. Many different types of people of various professions have been able to incorporate it into their workflows with impressive results. I think it lacks perspective to scoff at LLMs if they can't understand this function right now. That said, as other people have pointed out, it probably gets half points on this function already.
deleted 2y ago
[deleted]
emptiestplace 2y ago
4o did a great job at this, both in terms of actually understanding it and explaining in a way that is very easy to follow. I suspect Claude would as well. I pasted it here but the formatting got fucked. Have you ever used an LLM?
gaganyaan 2y ago
It also did a step by step breakdown, but here's what GPT-4o summarized the function as: What does the code achieve? This function essentially extracts and compacts the bits from x that correspond to the 1 bits in the mask m. It does so by successively shifting the extracted bits towards the least significant positions, while keeping the relative order of the bits intact. The algorithm is likely inspired by techniques used in bit packing or bit extraction, where the goal is to gather specific bits from a number based on a mask and pack them together efficiently.
- atomic128 2y ago
  Right, so it's trained on this text, from the BUGFIX-66 page where that code appears: Bug 47: Bit Extraction Misguided By Mask This is Guy L. Steele Jr.'s parallel suffix algorithm for bit extraction (U.S. Patent 6715066). Bits of x that correspond to 1-bits in m are shifted low and made consecutive, preserving relative order. The method masks x with m to isolate such bits and then repeatedly shifts 1- bits of x low by powers of two. A 1-bit in x with n less significant 0-bits in m is shifted low by a series of powers of two that sum to n. And it's trained on the patent, too. But there's something very clever happening in the repeated prefix sums of the holes, and that's the magic that makes it work. Does it understand how repetition of this works to achieve the extraction? odd := holes odd ^= odd << 1 odd ^= odd << 2 odd ^= odd << 4 odd ^= odd << 8 odd ^= odd << 16 odd ^= odd << 32 holes &^= odd What's happening to holes when this piece of code is repeated over and over? Does it really understand the algorithm (i.e., the magic of the code sequence above) or is it just recognizing the "bit extraction code" from its training data, and executing it like a machine? That's my question.

botanical76 2y ago

What is your point? Does anyone really believe LLMs are capable of understanding such dense, undocumented, and advanced code today? Or is the state of AI today - specifically at this nontrivial task - supposed to predict that it will never be able to?

AnimalMuppet 2y ago
First: Undocumented? So what? Either it can understand the code, or it can't. (And it better not depend on the comments, same as a programmer shouldn't, because they can be lies.) You think it's unreasonable to expect it to understand 21 lines? Of go, one of the least-dense languages out there? I think it actually demonstrates the point: (some) LLMs really cannot read and understand code if it doesn't fit in with their training data. They don't know the language, they know the patterns they have seen. If it doesn't fit, they can't reason about it from a knowledge of the language syntax, because they don't actually have such knowledge.
- botanical76 2y ago
  This will probably come down to your perspective and experience. The function is written in Go, but this is not relevant, because it does not use any Go-specific constructs. This is solely about the algorithm that is implemented. I don't understand the function provided, so I would need some comments to have a chance to parse it. That said, I feel I am an experienced programmer, so I would not fault ChatGPT for being confused with this function. Perhaps you find understanding this class of algorithm - doing bit manipulation in a loop - relatively easy, or even trivial. It would make sense if you expect an LLM to breeze through this task if that were the case. Today, a computer is able to work with natural language, and even perform useful tasks with it. It can write and understand surprisingly complex code, and sometimes it works. Many different types of people of various professions have been able to incorporate it into their workflows with impressive results. I think it lacks perspective to scoff at LLMs if they can't understand this function right now. That said, as other people have pointed out, it probably gets half points on this function already.

deleted 2y ago

[deleted]

emptiestplace 2y ago

4o did a great job at this, both in terms of actually understanding it and explaining in a way that is very easy to follow. I suspect Claude would as well. I pasted it here but the formatting got fucked. Have you ever used an LLM?

gaganyaan 2y ago

It also did a step by step breakdown, but here's what GPT-4o summarized the function as: What does the code achieve? This function essentially extracts and compacts the bits from x that correspond to the 1 bits in the mask m. It does so by successively shifting the extracted bits towards the least significant positions, while keeping the relative order of the bits intact. The algorithm is likely inspired by techniques used in bit packing or bit extraction, where the goal is to gather specific bits from a number based on a mask and pack them together efficiently.

atomic128 2y ago
Right, so it's trained on this text, from the BUGFIX-66 page where that code appears: Bug 47: Bit Extraction Misguided By Mask This is Guy L. Steele Jr.'s parallel suffix algorithm for bit extraction (U.S. Patent 6715066). Bits of x that correspond to 1-bits in m are shifted low and made consecutive, preserving relative order. The method masks x with m to isolate such bits and then repeatedly shifts 1- bits of x low by powers of two. A 1-bit in x with n less significant 0-bits in m is shifted low by a series of powers of two that sum to n. And it's trained on the patent, too. But there's something very clever happening in the repeated prefix sums of the holes, and that's the magic that makes it work. Does it understand how repetition of this works to achieve the extraction? odd := holes odd ^= odd << 1 odd ^= odd << 2 odd ^= odd << 4 odd ^= odd << 8 odd ^= odd << 16 odd ^= odd << 32 holes &^= odd What's happening to holes when this piece of code is repeated over and over? Does it really understand the algorithm (i.e., the magic of the code sequence above) or is it just recognizing the "bit extraction code" from its training data, and executing it like a machine? That's my question.