7 ms·
Yet Another LLM Rant
- techpineapple 1y agoI wonder if one reasons new versions of GPT appear to get better - say at coding tasks is just because they have new knowledge. When ChatGPT4 comes out, new versions of API’s will have less blog post / examples / documentation in their training data. So ChatGPT 5 comes out and seems to solve all the problems that ChatGPT4 had, but then of course fail on newer libraries. Rinse and repeat
- QuantumNoodle 1y ago> ... just because they have new knowledge. This means there is a future where AI is training on data it self generated, and I worry that might not be sustainable.
- techpineapple 1y agoI’ve heard of this idea of training on synthetic data, I wonder what is that data and does this increase or decrease hallucinations? Is the goal of training on synthetic data to better wear certain paths, or to increase the amount of knowledge / types of data. Because the second seems vaguely impossible to do.
- jgalt212 1y agoA software based Habsburg Jaw if you will.
- lazide 1y agoThis is already occurring, is not sustainable, and produces an effect known as Model Collapse.
- jmkni 1y agowhat's funny is that newer models will now be trained on the exact question, "Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone?" and similar questions to it, because of this post maybe the key to training future llm's is to write angry blog posts about the things they aren't good at and get them to the front page of hn?
- nikolayasdf123 1y agogood point. nobody knows you are a dog on internet anyways
- nikolayasdf123 1y ago> “Not having an answer” is not a possibility in this system - there’s always “a most likely response”, even if that makes no sense. simple fix - probability cutoff. but in all seriousness this is something that will be fixed. don't see fundamental reason why not. and I myself seen such hallucinations (about compression too actually) as well.
- Seb-C 1y agoHallucinations are not a bug or an exception, but a feature. Everything outputted by LLMs is 100% made-up, with a heavy bias towards what has been fed to it at first (human written content). The fundamental reason why it cannot be fixed is because the model does not know anything about the reality, there is simply no such concept here. To make a "probability cutoff" you first need a probability about what the reality/facts/truth is, and we have no such reliable and absolute data (and probably never will).
- nikolayasdf123 1y agohave you seen Iris flowers dataset? it is fairly simple to find cutoffs to classify flowers. or are you claiming in general that there is no objective truth in reality in philosophical sense? well, you can go on that more philosophical side of the road, or you can get more pragmatic. things just work, regardless how we talk about them.
- Seb-C 1y agoI don't mean it in a philosophical sense, more in a rigorous scientific one. Yes, we do have reliable datasets as in your example, but those are for specific topics and are not based on natural language. What I would call "classical" machine learning is already a useful technology where it's applied. Jumping from separate datasets focused on specific topics to a single dataset describing "everything" at once is not something we are even close to doing, if it's even possible. Hence the claim of having a single AI able to answer anything is unreasonable. The second issue is that even if we had such a hypothetical dataset, ultimately if you want a formal response from it, you need a formal question and a formal language (probably something between maths and programming?) in all the steps of the workflow. LLMs are only statistical models about natural languages, so it's the antithesis of this very idea. To achieve that would have to be a completely different technology that has yet to even be theoretized.
- bfioca 1y ago>...it’s a useless tool. I don’t like collaborating with chronic liars who aren’t able to openly point out knowledge gaps... I think a more correct take here might be "it's a tool that I don't trust enough to use without checking," or at the very least, "it's a useless tool for my purposes." I understand your point, but I got a little caught up on the above line because it's very far out of alignment with my own experience using it to save enormous amounts of time.
- lazide 1y agoIt’s a tool that fundamentally can’t be used reliably without double checking everything it. That is rather different than you’re presenting it.
- tmnvdb 1y agoSo similar to wikipedia
- simianwords 1y agoSimilar to anything really. Can I really trust anything without verifying? Scientific journals?
- cindyllm 1y ago[dead]
- lblume 1y agoIt seems that on some level, you have to in order to not just constantly reflecting upon your thoughts and researching facts. Whether you trust a given source should surely depend upon its reputation regarding the validity of its claims.
- simianwords 1y agoI agree and by reputation you mean accuracy. We implicitly know not to judge anything as 100% true and implicitly apply skepticism towards sources - the skepticism is decided by our past experience with the sources. Think of LLMs as the less accurate version of scientific journals.
- cantor_S_drug 1y agoThere was a writer who in order to get ideas to write about used to cut up words from newspaper headlines and then rearrange them. In one rearrangement, he got "Son sues father for xyz". That headline came true 2 years later.
- lblume 1y agoHow is this relevant? Modern LLMs don't rearrange words at all in a meaningful sense of the word, and are certainly better than just using random chance to sample tokens.
- cantor_S_drug 1y agoIndeed it is exactly that process. They cutup words. Then categorize them based on some metric of nearness (not random). Then link them up. Obviously this process is much more sophisticated than what I have described here.
- hodgehog11 1y agoI am sympathetic to the reasoning as to why LLMs should not be used to help some programmers right now. But I get a little frustrated seeing many of these kinds of posts that talk about fundamental limitations of LLMs vs humans on the grounds that it cannot "logically reason" like a human does. These are limitations in the current approach to training and objectives; internally, we have no clue what is going on. > it’s “just a statistical model” that generates “language” based on a chain of “what is most likely to follow the previous phrase” Humans are statistical models too in an appropriate sense. The question is whether we try to execute phrase by phrase or not, or whether it even matters what humans do in the long term. > The only way ChatGPT will stop spreading that nonsense is if there is a significant mass of humans talking online about the lack of ZSTD support. Or you can change the implicit bias in the model by being more clever with your training procedure. This is basic stats here, not everything is about data. > They don’t know anything, they don’t think, they don’t learn, they don’t deduct. They generate real-looking text based on what is most likely based on the information it has been trained on. This may be comforting to think, but it's just wrong. It would make my job so much easier if it were true. If you take the time to define "know", "think", and "deduct", you will find it difficult to argue current LLMs do not do these things. "Learn" is the exception here, and is a bit more complex, not only because of memory and bandwidth issues, but also because "understand" is difficult to define.
- libraryofbabel 1y agoYeah. The empty “it’s just a statistical model” critique (or the dressed-up “stochastic parrots” version of it) is almost a sign at this point that the person using it formed their opinions about AI back when ChatGPT first came out, and hasn’t really bothered to engage with it much since then. If in 2022 I’d tried to convince AI skeptics that in three years we might have tools on the level of Claude Code, I’m sure I’d have heard everyone say it would be impossible because “it’s just a statistical model.” But it turned out that there was a lot more potential in the architecture for encoding structured knowledge, complex reasoning, etc., despite that architecture being probabilistic. (Don’t bet against the Bitter Lesson.) LLMs have a lot of problems, hallucination still being one of them. I’d be the first to advocate for a skeptical hype-free approach to deploying them in software engineering. But at this point we need careful informed engagement with where the models are at now rather than cherry-picked examples and rants.
- gdsys 1y ago"Based on my research, zstd compression is not natively supported by iOS or Apple's frameworks, which means you cannot use zstd compression without adding some form of external code to your project" Thanks Sonnet. Full response: https://www.perplexity.ai/search/without-adding-third-party-dep-pGJFzVcxQvGRWE3nycN7vg#0 https://www.perplexity.ai/search/without-adding-third-party-...
- dcre 1y agoMost likely the key here is web search. Later I will try the post’s example with gpt-5 with search. I would be surprised if it didn’t say the same thing. From a product point of view, it seems clear that just as they have work to get the model to dynamically decide to use reasoning when it would help, they have to do the same with web search.
- dcre 1y agogpt-5 with medium reasoning, no search: "Short answer: use the system zstd C library that ships with iOS; Apple’s Compression framework still doesn’t expose zstd. Link libzstd and call its streaming API from Swift." (and then like 50 lines of code plus some explanation) gpt-5 with medium reasoning + search: "Short answer: you can’t. Apple’s built‑in Compression framework on iOS exposes LZFSE, LZ4, ZLIB/DEFLATE, and LZMA—not Zstandard—so there is no system API to do zstd without bundling an external library. (https://www.hackingwithswift.com/example-code/system/how-to-compress-and-decompress-data https://www.hackingwithswift.com/example-code/system/how-to-...) If you can switch to a built‑in algorithm, here’s a minimal streaming compressor using Compression (LZFSE shown):" (etc etc)
- nikolayasdf123 1y agopossible solution: "reality checks" I see that GitHub Copilot actually runs code, writes simple exploratory programs, iteratively tests its hypothesis. it is astoundingly effective and fast. same here. nothing stops this AI to actually trying to implement whatever this AI suggested, compile it, and see if this is actually works. grounding in reality at inference time, so to speak.
- quantum_state 1y agoAn implication from the 1985 paper of Peter Naur on programming as theory building is that the current LLM coding tool would be very effective in generating technical debt even when it works ... use at your own risk.
- simianwords 1y agoThe prompt works for me and correctly identifies that zstd doesn't work https://chatgpt.com/share/689769c5-bd68-800b-ae63-c6a337dcfaa9 https://chatgpt.com/share/689769c5-bd68-800b-ae63-c6a337dcfa... "Short answer: you can’t. iOS doesn’t ship a Zstandard (zstd) encoder/decoder in any first-party framework. Apple’s built-in Compression framework supports LZFSE, LZ4, zlib/deflate, and LZMA—not zstd."
- Maro 1y agoI'm currently working as a hands-off VP, so I don't use LLMs for coding at work, only for emails and document editing. I do use it for my hobby weekend coding stuff, which usually involves writing short 100-1000 LOC Python toy programs for my own education/entertainment. My way-of-working is quite primitive, I have zero integrations, nothing agentic, I just copy/paste with ChatGPT. For this use-case it's been very useful, it can usually generate close-to-complete solutions, as long as it's one of the major programming languages, and it's a reasonably standard problem. So in general I'm always surprised when people say that LLMs are completely useless for coding --- this is just not true, and I feel sorry for people who shut themselves off from a useful tool. But even at this small scale, even the best (o3) models sometimes totally fail. Recently I started a series of posts on distributed algorithms [1], and when I was working on the post/code for the Byzantine Generals / Consensus algorithm, o3 --- to my honest surprise --- just totally failed. I tried about 10 different times (both from scratch and describing the incorrect behaviour of its code), also showing it the original Lamport paper, and it just couldn't get it right... even though the toy implementation is just ~100 LOC, the actual algorithm portion is maybe 25 LOC. My hypothesis is that there are very few implementations online, and additionally I find the descriptions of the algorithm a bit vague (interleaves message cascade and decision logic). [1] https://bytepawn.com/tag/distributed.html https://bytepawn.com/tag/distributed.html
- drakenot 1y agoChatGPT 5> Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone? (It auto-routed to its thinking model, took 25s) > Short answer: you can’t. iOS doesn’t ship a public zstd encoder/decoder. The system Compression and AppleArchive frameworks only expose lzfse, lz4, zlib/deflate, and lzma. So with only Apple SDKs, there’s no way to produce zstd-compressed bytes. > > Two practical paths: > 1. If zstd isn’t mandatory, use a built-in codec (lzfse or lz4 are fastest) via Compression’s streaming API. Example (pure Swift, no deps) that compresses an arbitrary byte stream: > > <code example> > 2. If zstd is mandatory, you must embed zstd yourself. That can mean: > - Vendoring the official C sources (libzstd) directly into your project and calling them via a tiny Swift wrapper/bridging header > - Statically linking a prebuilt libzstd.a you compile for iOS
- hodgehog11 1y agoI think the useful takeaway here is that Top 1 operation is generally not a good idea, especially not for making judgements. This doesn't address the main points of the blog though.
- djeastm 1y agoHere's what I got with ChatGPT GPT-5, it must not have thought about it because it was near instantaneous: >On iOS, you can use Apple’s built-in Zstandard (zstd) compression API from the Compression framework — no third-party dependencies required. >Here’s how you can compress a Data stream with zstd: >... https://chatgpt.com/share/68976c8f-7ae0-8012-b7a8-58e016246dac https://chatgpt.com/share/68976c8f-7ae0-8012-b7a8-58e016246d...
- simonw 1y agoLooks like that time it didn't route to the thinking model. The router mechanism is deeply confusing right now: https://twitter.com/emollick/status/1954210778321465634 https://twitter.com/emollick/status/1954210778321465634
- tptacek 1y agoLLMs can be a useful tool, maybe. But don’t anthropomorphize them. (but, earlier) If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work that is guaranteed to never work, it’s a useless tool. I don’t like collaborating with chronic liars.
- simonw 1y ago> This concludes all the testing for GPT5 I have to do. If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work that is guaranteed to never work, it’s a useless tool. Yeah, except it isn't. You can get enormous value out of LLMs if you get over this weird science fiction requirement that they never make mistakes. And yeah, their confidence is frustrating. Treat them like an over-confident twenty-something intern who doesn't like to admit when they get stuff wrong. You have to put the effort in to learn how to use them with a skeptical eye. I've been getting value as a developer from LLMs since the GPT-3 era, and those models sucked. > The only way ChatGPT will stop spreading that nonsense is if there is a significant mass of humans talking online about the lack of ZSTD support. We actually have a robust solution for this exact problem now: run the prompt through a coding agent of some sort (Claude Code, Codex CLI, Cursor etc) that has access to the Swift compiler. That way it can write code with the hallucinated COMPRESSION_ZSTD thing in it, observe that it doesn't compile and iterate further to figure out what does work. Or the simpler version of the above: LLM writes code. You try and compile it. You get an error message and you paste that back into the LLM and let it have another go. That's been the main way I've worked with LLMs for almost three years now.
- jpc0 1y agoIf that same intern, when asked something, responded that they checked, gave you a link to a document they claim have the proof / answer but does in fact no and continued to do that they wouldn’t be an intern very long. But somehow this is acceptable behaviour for an AI? I use AI for sure, but only on things that I can easily verify is correct (run a test or some code ), because I have had the AI give me functions in an API with links to online documentation for those functions, the document exists, the function is not in it, when called out instead of doing a basic tool call the AI will double down that it is correct and you the human are wrong. That would get an intern fired but here you are standing on the interns side.
- raincole 1y agoWow... people unironically anthropomorphize AI to the point that they expect AI to work exactly like an human intern, otherwise it's unacceptable...
- blizdiddy 1y agoFinding a useless way to use a tool is neither interesting or novel. I hit my thumb with a hammer, useless tool.
- zhivota 1y agoI call this genre of article "I'm too lazy to think about how to deal with the fact that LLMs are sometimes wrong, especially when they aren't using thinking and are given short, leading prompts." Ok great! For those of us who aren't too lazy for it, LLMs are providing a lot of value right now.
- add-sub-mul-div 1y agoI'm not sure that the people who still prefer to do their own work rather than circuitously delegate it are the ones that seem lazy.
- johnfn 1y agoI find it hard to empathize with these sorts of articles. I think in, in the spate of GPT-5 related content recently, I've been seeing a lot of articles that boil down to "I tried GPT-5 on a single hard question, and it gave an answer. This proves that all LLMs are useless." And I don't think I'm distorting the author's viewpoint. He goes on to say "This concludes all the testing for GPT5 I have to do" after conducting a single test. This seems like particularly harsh criteria; what would happen if I applied this to other tools? - I used Typescript, but it missed a bug that crashed prod, so it is "absolute horseshit" - I used Rust, but one of my developers added an unsafe block, so it's trash.
- terminatornet 1y ago> I used Typescript...it is "absolute horseshit" That's right
- overgard 1y agoMy issue with LLMs isnt really the tech itself. They're occassionally useful, although HOW useful seems to depend on a persons skillset. My main beef with the AI hype is that it's allowing a lot of idiots to significantly devalue our profession in a really noxious and irritating way to people that generally don't understand what we do but would like to pay us less or pay less of us. I'm annoyed at other software developers that don't seem to see how harmful this will be for us when the insane investment bubble bursts and AI becomes a lot more expensive to use. We will probably have lost a generation of junior developers who have become dependent on a suddenly expensive tool. And execs will just think the seniors need to pick up the slack. And expectations on AI will be a lot higher when the subscription is more like 200 or 2000 a month. And that's just for coding! I'd be furious if I was an artist and generative AI was being trained on my portfolio to plagiarize my work. (Badly) What I never see justified is why any of this is good for society. At best it lets billionionaires save some money by getting rid of jobs, or vibe coders pretend they can build a product until they hit a wall where real understanding is neccessary. If you follow the trail of who is supposed to benefit from these things its not many of us. If AI were to disappear today I don't think my life would be any worse.
- viach 1y agoI think people often misread AGI as Artificial God Intelligence.
- oidar 1y agoIf you have an Apple docs mcp, it does let you know and offers alternatives. So this is another “your holding it wrong” problem.
- nojito 1y ago>Use your brains. I love posts like these because it just reinforces that I made the right decision in spending as much time as I do in getting really really really good at using llms.
- kentonv 1y agoAuthor is arguing: * LLMs, when presented with a question to which they don't know the answer, tend to make up a wrong answer rather than say they don't know. * Therefore, LLMs are useless. Yes, LLMs do that, and it is a problem. No, that does not make them useless. It's usually much easier to check their answer than it would have been to search for an answer yourself. If they are right, you saved time. If they are wrong, at worst you wasted a couple minutes. But they are right often enough that it can be worthwhile, especially if you have a feel for what questions they are likely to know the answer to and what questions they aren't.
- prinny_ 1y agoThe top comments are focusing on what "know" means in the context of an LLM but I think that while it's an interesting discussion it misses the point. Even if we agree that an LLM can "know" things, the tech has been heavily advertised as nothing less that transformative. If it can't help itself from inventing non existent answers or count the number of "r"s in the word blueberry (a test I tried myself the other day on my phone's chat-gpt app, it said "three") it's not transformative.
- yelirekim 1y agoI'm almost positive that he didn't use thinking mode, because I've asked GPT-5 thinking his question verbatim in 10 parallel threads and it said "you can't" and thoroughly explained why every time. Below is the first response I got back. --- Short answer: you can’t. iOS doesn’t ship a Zstandard encoder/decoder. Apple’s built-in `Compression` framework only exposes LZFSE, LZ4(/RAW), zlib/DEFLATE, and LZMA—no ZSTD knob to turn. ([Microsoft Learn][1], [Apple Developer][2]) If you’re okay with a built-in alternative that’s “zstd-ish” in speed/ratio, use LZFSE via `Compression`’s streaming API. Here’s a minimal streaming compressor you can drop in; it takes chunks of `Data` and yields compressed chunks (no third-party code): ```swift import Compression final class StreamCompressor { private var stream = compression_stream() private let algo: compression_algorithm private var isInitialized = false init(algorithm: compression_algorithm = COMPRESSION_LZFSE) { self.algo = algorithm } func append(_ input: Data, isFinal: Bool, outChunkSize: Int = 64 * 1024) -> [Data] { if !isInitialized { let status = compression_stream_init(&stream, COMPRESSION_STREAM_ENCODE, algo) precondition(status != COMPRESSION_STATUS_ERROR, "compression_stream_init failed") isInitialized = true } var outputs: [Data] = [] input.withUnsafeBytes { (srcBuf: UnsafeRawBufferPointer) in var flags: Int32 = isFinal ? Int32(COMPRESSION_STREAM_FINALIZE.rawValue) : 0 stream.src_ptr = srcBuf.bindMemory(to: UInt8.self).baseAddress stream.src_size = srcBuf.count var out = Data(count: outChunkSize) out.withUnsafeMutableBytes { dstBuf in while true { stream.dst_ptr = dstBuf.bindMemory(to: UInt8.self).baseAddress stream.dst_size = dstBuf.count let status = compression_stream_process(&stream, flags) let produced = dstBuf.count - stream.dst_size if produced > 0 { outputs.append(Data(bytes: dstBuf.baseAddress!, count: produced)) } if status == COMPRESSION_STATUS_OK && stream.src_size == 0 && stream.dst_size > 0 { break // need more input to continue } else if status == COMPRESSION_STATUS_OK && stream.dst_size == 0 { // output buffer filled; loop to drain more continue } else if status == COMPRESSION_STATUS_END { break } else if status == COMPRESSION_STATUS_ERROR { preconditionFailure("compression_stream_process failed") } else { // Shouldn’t happen break } } } } return outputs } deinit { if isInitialized { compression_stream_destroy(&stream) } } } ``` Usage: ```swift let compressor = StreamCompressor(algorithm: COMPRESSION_LZFSE) var compressed: [Data] = [] for (i, chunk) in chunks.enumerated() { let final = i == chunks.count - 1 compressed.append(contentsOf: compressor.append(chunk, isFinal: final)) } // `compressed` now contains the zstd-adjacent (LZFSE) stream ``` If you truly need the ZSTD format (interop requirement), you’ll have to ship the reference C library (`libzstd`) in your app (static link it and call through a tiny Swift wrapper). That’s still “third-party,” even if you vendor the sources instead of using SPM/CocoaPods. There is no system-provided ZSTD on iOS today. ([GitHub][3]) tl;dr: iOS doesn’t expose ZSTD. Use `Compression` with LZFSE for built-in streaming, or vendor `libzstd` and bridge to it if you must emit/ingest ZSTD. ([Microsoft Learn][1], [Apple Developer][2], [GitHub][3]) [1]: https://learn.microsoft.com/en-us/dotnet/api/compression.compressionalgorithm?view=xamarin-ios-sdk-12&utm_source=chatgpt.com https://learn.microsoft.com/en-us/dotnet/api/compression.com... "CompressionAlgorithm Enum (Compression)" [2]: https://developer.apple.com/documentation/compression/compression_lzfse?utm_source=chatgpt.com https://developer.apple.com/documentation/compression/compre... "COMPRESSION_LZFSE | Apple Developer Documentation" [3]: https://github.com/facebook/zstd?utm_source=chatgpt.com https://github.com/facebook/zstd?utm_source=chatgpt.com "facebook/zstd: Zstandard - Fast real-time compression ..."
- anuramat 1y agoif you treat it like stochastic software it is, how is 95% chance of one-shotting a problem not useful? I feel like the biggest problem most people have with LLMs is the "lack of accountability", and then the same people say "don’t anthropomorphize them"