4 ms·
If your premises and assumptions are sufficiently corrupted, you can come to any conclusion and believe you are being rational. Like those dreams where you walk
by satisfice 1y ago
If your premises and assumptions are sufficiently corrupted, you can come to any conclusion and believe you are being rational. Like those dreams where you walk around without pants on and you are more worried about not having pants than you are about how it could have come to be that your pants kept going missing. Your brain is not present enough to find the root of the problem.
An LLM is not unbiased, and you would know that if you tested LLMs.
Apart from biases, an LLM is not a reliable oracle, you would know that if you tested LLMs.
The reliabilities and unreliabilities of LLMs vary in discontinuous and unpredictable ways from task to task, model to model, and within the same model over time. You would know this if you tested LLMs. I have. Why haven’t you?
Ideas like this are promoted by people who don’t like testing, and don’t respect it. That explains why a concept like this is treated as equivalent to a tested fact. There is a name for it: wishful thinking.
- walterbell 1y ago> wishful thinking Given the economic component of LLM wishes, we can look at prior instances of wishing-at-scale, https://en.wikipedia.org/wiki/Tulip_mania https://en.wikipedia.org/wiki/Tulip_mania
- troupo 1y agoThere's a more recent one: https://blog.mollywhite.net/blockchain/ https://blog.mollywhite.net/blockchain/
- roenxi 1y agoBlockchains are past the gauntlet where they can be described as a mania, it is clear they are a permanent addition to the world of finance; probably as a multi-billion or -trillion dollar market cap asset class. If crypto was going to fail the interest rate rises would have done it by now.
- troupo 1y agoTulips. You're describing tulips.
- MarcoDewey 1y agoI believe that I have unintentionally misled you. When I say "unbiased oracle" I am talking specifically about the test oracle being unbiased by how the software was implemented. ie. Black Box testing. I don't think I made the point very clear in the blog (I will rectify that), but I am saying that because LLMs are so easily biased by their prompting that they sometimes perform better when doing black box testing tasks than they do when performing white box testing.
- satisfice 1y agoI appreciate that you replied. It warms my heart, frankly. It gives me hope. I don't want to have a big argument about this right at this moment. But-- truly-- thank you for replying!