2 ms·
I think the trend is away from semantic tokens, and away anything that is remotely like 'feature engineering'. Maybe even they will just use character level or
by ftxbro 3y ago
I think the trend is away from semantic tokens, and away anything that is remotely like 'feature engineering'. Maybe even they will just use character level or byte level encoding in the future. They have used it before, and probably something like it will be used again if the Rich Sutton 'bitter lesson' continues to be true.
- jiggawatts 3y agoHave you seen how many steps an AI would have to go through to do something as simple as division or square root of “just” a single number? This is an operation a calculator from the 1970s can do instantly but an AI running on a computer worth half a million and sucking down ten kilowatts of juice struggles with! PS: I just asked GPT 4 to use long form division to calculate 73732/64. It took over ten seconds and the result was wrong.
- ftxbro 3y agoThey are making machines that think basically like people. People suck at mental math for whatever reason and it looks like the machines are sucking at it too, probably for the same reason. As someone else said here, these thinking machines can just use a calculator themselves or write a small python code or use wolfram alpha plugin.
- jiggawatts 3y agoI’m saying instead of emulating the AI pressing buttons on a calculator, just a plug a calculator into its brain. Make a cyborg without surgery.
- ftxbro 3y agoOK I see what you mean. I don't think using your suggestion of semantic tokens is a good way to do it. They will do it by wrapping the raw text completion with more steps that are hidden from you, which will make it do the math right, and then it won't make such obvious math mistakes anymore.
- jiggawatts 3y agoThe problem with that is that the AI can’t reason over the results of computations it can’t see.