7 ms·
An LLM company using regexes for sentiment analysis? That's like a truck company using horses to transport parts. Weird choice.
by BoppreH 7mo ago
An LLM company using regexes for sentiment analysis? That's like a truck company using horses to transport parts. Weird choice.
- harikb 7mo agoNot everything done by claude-code is decided by LLM. They need the wrapper to be deterministic (or one-time generated) code?
- lou1306 7mo agoThey're searching for multiple substrings in a single pass, regexes are the optimal solution for that.
- noosphr 7mo agoThe issue isn't that regex are a solution to find a substring. The issue is that you shouldn't be looking for substrings in the first place. This has buttbuttin energy. Welcome to the 80s I guess.
- deleted 7mo ago[deleted]
- 8cvor6j844qw_d6 7mo agoVery likely vibe coded. I've seen Claude Code went with a regex approach for a similar sentiment-related task.
- mr_00ff00 7mo agoMy understanding of vibe coding is when someone doesn’t look at the code and just uses prompts until the app “looks and acts” correct. I doubt you are making regex and not looking at it, even if it was AI generated.
- deleted 7mo ago[deleted]
- rdiddly 7mo agoClbuttic!
- lou1306 7mo ago> The issue is that you shouldn't be looking for substrings in the first place. Why? They clearly just want to log conversations that are likely to display extreme user frustration with minimal overhead. They could do a full-blown NLP-driven sentiment analysis on every prompt but I reckon it would not be as cost-effective as this.
- noosphr 7mo ago>Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems. The only time to use a regex is when searching with a human in the loop. All other uses are better handled some other way. >They could do a full-blown NLP-driven sentiment analysis on every prompt but I reckon it would not be as cost-effective as this. Every conversation is sent to an llm at least a thousand times the size of gpt2 which could one shot this nearly a decade ago.
- lou1306 7mo ago> Every conversation is sent to an llm at least a thousand times the size of gpt2 which could one shot this nearly a decade ago. Yes, but that is _what the product does_. What we are talking about is _telemetry_.
- deleted 7mo ago[deleted]
- BoppreH 7mo agoIt's fast, but it'll miss a ton of cases. This feels like it would be better served by a prompt instruction, or an additional tiny neural network. And some of the entries are too short and will create false positives. It'll match the word "offset" ("ffs"), for example. EDIT: no it won't, I missed the \b. Still sounds weird to me.
- hk__2 7mo agoIt’s fast and it matches 80% of the cases. There’s no point in overengineering it.
- NitpickLawyer 7mo ago> There’s no point in overengineering it. I swear this whole thread about regexes is just fake rage at something, and I bet it'd be reversed had they used something heavier (omg, look they're using an LLM call where a simple regex would have worked, lul)...
- vharuck 7mo agoThe pattern only matches if both ends are word boundaries. So "diffs" won't match, but "Oh, ffs!" will. It's also why they had to use the pattern "shit(ty|tiest)" instead of just "shit".
- BoppreH 7mo agoYou're right, I missed the \b's. Thanks for the correction.
- stingraycharles 7mo agoBecause they want it to be executed quickly and cheaply without blocking the workflow? Doesn’t seem very weird to me at all.
- deleted 7mo ago[deleted]
- _fizz_buzz_ 7mo agoThey probably have statistics on it and saw that certain phrases happen over and over so why waste compute on inference.
- mycall 7mo agoThe problem with regex is multi-language support and how big the regex will bloat if you to support even 10 languages.
- b112 7mo agoDid you just complain about bloat, in anything using npm?
- TeMPOraL 7mo agoWe're talking about Claude Code. If you're coding and not writing or thinking in English, the agents and people reading that code will have bigger problems than a regexp missing a swear word :).
- MetalSnake 7mo agoI talk to it in non-English. But have rules to have everything in code and documentation in english. Only speaking with me should use my native language. Why would that be a problem?
- ekropotin 7mo agoBecause 90% of training data was in English and therefore the model perform best in this language.
- codegladiator 7mo agowhat you are suggesting would be like a truck company using trucks to move things within the truck
- argee 7mo agoThat’s what they do. Ever heard of a hand truck?
- eadler 7mo agoI never knew the name of that device. Thanks
- freedomben 7mo agoDepending on the region you live in, it's also frequently called a "dolly"
- SmellTheGlove 7mo agoIsn’t a dolly a flat 4 wheeled platform thingy? A hand truck is the two wheeled thing that tilts back.
- eszed 7mo agoHa! Where I'm from a "dolly" was the two-wheeled thing. The four-wheeler thing wasn't common before big-boxes took over the hardware business, but I think my dad would have called it a "cart", maybe a "hand-cart".
- sethammons 7mo agoGrew up with two wheeled: dolly and four wheeled: piano dolly. Was an adult when I heard hand-truck. I prefer dolly. Nicer mouth feel.
- istoleabread 7mo agoDo we have a hand llm perchance?
- draxil 7mo agoGood to have more than a hammer in your toolbox!
- ojr 7mo agoI used regexes in a similar way but my implementation was vibecoded, hmmm, using your analysis Claude Code writes code by hand.
- blks 7mo agoBecause they actually want it to work 100% of the time and cost nothing.
- orphea 7mo agoThen they made it wrong. For example, "What the actual fuck?" is not getting flagged, neither is "What the *fuck*".
- vntok 7mo agoThey evidently ran a statistical analysis and determined that virtually no one uses those phrases as a quick retort to a model's unsatisfying answer... so they don't need to optimize for them.
- arcfour 7mo agoIt is exceedingly obvious that the goal here is to catch at least 75-80% of negative sentiment and not to be exhaustive and pedantic and think of every possible way someone could express themselves.
- Zamaamiro 7mo agoClassic over-engineering. Their approach is just fine 90% of the time for the use case it’s intended for.
- orphea 7mo ago75-80% [1], 90%, 99% [2]. In other words, no one has any idea. I doubt it's anywhere that high because even if you don't write anything fancy and simply capitalize the first word like you'd normally do at the beginning of a sentence, the regex won't flag it. Anyway, I don't really care, might just as well be 99.99%. This is not a hill I'm going to die on :P [1]: https://news.ycombinator.com/item?id=47587286 https://news.ycombinator.com/item?id=47587286 [2]: https://news.ycombinator.com/item?id=47586932 https://news.ycombinator.com/item?id=47586932
- 7mo ago
- sumtechguy 7mo agohmm not a terrible idea (I think). You have a semi expensive process. But you want to keep particular known context out. So a quick and dirty search just in front of the expensive process. So instead of 'figure sentiment (20seconds)'. You have 'quick check sentiment (<1sec)' then do the 'figure sentiment v2 (5seconds)'. Now if it is just pure regex then your analogy would hold up just fine. I could see me totally making a design choice like that.
- pfortuny 7mo agoThey had the problem of sentiment analysis. They use regexes. You know the drill.
- throwaw12 7mo agobecause impact of WTF might be lost in the result of the analysis if you solely rely on LLM. parsing WTF with regex also signifies the impact and reduces the noise in metrics "determinism > non-determinism" when you are analysing the sentiment, why not make some things more deterministic. Cool thing about this solution, is that you can evaluate LLM sentiment accuracy against regex based approach and analyse discrepancies
- kjshsh123 7mo agoUsing regex with LLMs isn't uncommon at all.
- deleted 7mo ago[deleted]
- floralhangnail 7mo agoWell, regex doesn't hallucinate....right?
- geon 7mo agobuttbuttination
- mmh0000 7mo agoThe Clbuttical problem[1] [1] https://en.wikipedia.org/wiki/Scunthorpe_problem https://en.wikipedia.org/wiki/Scunthorpe_problem
- raw_anon_1111 7mo agoI just went to expertSexChange.com…
- mghackerlady 7mo agoMore like a car company transporting their shipments by truck. It's more efficient
- deleted 7mo ago[deleted]
- __alexs 7mo agoUsing some ML to derive a sentiment regex seems like a good actually?
- lazysheepherd 7mo agoBecause they are engineers? The difference between an engineer and a hobbyist is an engineer has to optimize the cost. As they say: any idiot can build a bridge that stands, only an engineer can build a bridge that barely stands.
- susupro1 7mo ago[dead]
- feketegy 7mo agoIt's all regex anyways
- irthomasthomas 7mo agoThis just proves its vibe coded because LLMs love writing solutions like that. I probably have a hundred examples just like it in my history.
- irthomasthomas 7mo agoActually, this could be a case where its useful. Even it only catches half the complaints, that's still a lot of data, far more than ordinary telemetry used to collect.
- j45 7mo agoAsking a non deterministic software to act like a deterministic one (regex) can be a significantly higher use of tokens/compute for no benefit. Some things will be much better with inference, others won’t be.
- intended 7mo agoThe amount of trust and safety work that depends on google translate and the humble regex, beggars the imagination.
- lopsotronic 7mo agoThe difference in response time - especially versus a regex running locally - is really difficult to express to someone who hasn't made much use of LLM calls in their natural language projects. Someone said 10,000x slower, but that's off - in my experience - by about four orders of magnitude. And that's average, it gets much worse. Now personally I would have maybe made a call through a "traditional" ML widget (scikit, numpy, spaCy, fastText, sentence-transformer, etc) but - for me anyway - that whole entire stack is Python. Transpiling all that to TS might be a maintenance burden I don't particularly feel like taking on. And on client facing code I'm not really sure it's even possible.
- cyanydeez 7mo agoSo, think of it as a business man: You don't really care if your customers swear or whatever, but you know that it'll generate bad headlines. So you gotta do something. Just like a door lock isn't designed for a master criminal, you don't need to design your filter for some master swearer; no, you design it good enough that it gives the impression that further tries are futile. So yeah, you do what's less intesive to the cpu, but also, you do what's enough to prevent the majority of the concerns where a screenshot or log ends up showing blatant "unmoral" behavior.
- true_religion 7mo agoThis door lock doesn’t even work against people speaking French, so I think they could have tried a mite harder.
- sebastiennight 7mo agoEn toute honnêteté, je pense avoir dit "damn it" plus d'une fois à chat gépété avant de fermer la fenêtre dans un accès de rage
- ben_w 7mo agoThe up-side of the US market is (almost) everyone there speaks English. The down side is, that includes all the well-networked pearl-clutchers. Europe (including France) will have the same people, but it's harder to coordinate a network of pearl-clutching between some saying "Il faut protéger nos enfants de cette vulgarité!" and others saying "Η τηλεόραση και τα μέσα ενημέρωσης διαστρεβλώνουν τις αξίες μας!" even when they care about the exact same media. For headlines, that's enough. For what's behind the pearl-clutching, for what leads to the headlines pandering to them being worth writing, I agree with everyone else on this thread saying a simple word list is weird and probably pointless. Not just for false-negatives, but also false-positives: the Latin influence on many European languages leads to one very big politically-incorrect-in-the-USA problem for all the EU products talking about anything "black" (which includes what's printed on some brands of dark chocolate, one of which I saw in Hungary even though Hungarian isn't a Latin language but an Ugric language and only takes influences from Latin).
- sfn42 7mo agoIt's almost as if LLMs are unreliable
- deleted 7mo ago[deleted]
- apgwoz 7mo ago> That's like a truck company using horses to transport parts. Weird choice. Easy way to claim more “horse power.”
- makeitrain 7mo agoDon’t worry, they used an llm to generate the regex.
- make3 7mo agoit's like a faster than light spaceship company using horses. There's been infinite solutions to do this better even CPU only for years lol.
- nojs 7mo agoOh it’s worse than that. This one ended up getting my account banned: https://github.com/anthropics/claude-code/issues/22284 https://github.com/anthropics/claude-code/issues/22284
- cryptonector 7mo agoWow, that's horrible.
- lanbin 7mo agoThis is a tricky problem, I mean, Pinyin also uses the English alphabet.
- foodevl 7mo agoIt is not a tricky problem because it has a simple and obvious solution: do not filter or block usage just because the input includes a word like "gun".
- toraway 7mo ago... and closed for inactivity like basically every issue in the repo, of course.
- raw_anon_1111 7mo agoCloud hosted call centers using LLMs is one of my specialties. While I use an LLM for more nuanced sentiment analysis, I definitely use a list of keywords as a first level filter.
- pdntspa 7mo agoLLMs cost money, regular expressions are free. It really isn't so strange.
- nitekode 7mo agoA lot if things dont make sense until you involve scale. Regex could be good enough do give a general gist.
- ldobre 7mo agoIt's more like a truck company using people to transport some parts. I could be wrong here, but I bet this happens in Volvo's fabrics a lot.
- arnarbi 7mo agoIt's more like workers on a large oil tanker using bicycles to move around it, rather than trying to use another oil tanker.
- scotty79 7mo agoAs far as I can tell they do nothing with it. They just log it.
- artrockalter 7mo agoLLMs are good at writing complex regex, from my experience
- slashdave 7mo agoMaybe. Could just be a pre filter.