5 ms·
and you'll be saying that about GPT5 when it comes out to. This is what hype looks like.
by PH95VuimJjqBqy 3y ago
and you'll be saying that about GPT5 when it comes out to.
This is what hype looks like.
- fnordpiglet 3y agoYou can choose to believe what you choose. However I’m working with these things at a senior tech lead level at a megacorp and what I’m doing is actually really real, with real material benefits. I don’t know that another order of magnitude performance improvement would matter as much to me as better tooling, performance, context, lower costs, and higher GPU capacity. But if it does, 1000% I’ll be hyping it. I hype things that work and dismiss things that don’t, that’s what an engineer is supposed to do.
- NewJazz 3y agoOK... So care to share what that benefit looks like? This whole thread was about asking for concrete examples of generative AI's utility.
- humansareok1 3y agoGive an LLM the text of a PDF document. Ask the model to extract values in the document or in tables. Input the values into a spreadsheet. This is at minimum a task which costs companies around the world Hundreds of Millions of dollars a year.
- PH95VuimJjqBqy 3y agowhat you're describing is automation and companies have been doing it for years.
- NewJazz 3y agoYeah I'd love to see the results... If anything if there is a multimillion dollar benefit on the table one might argue that companies should publish this data in a more useful format. But no, let's bandaid over outdated practices like PDF-only data and burn GPU cycles to do it, with 92% accurate results.
- humansareok1 3y agoIf it replaces hundreds or thousands of man-hours of expensive labor then yes lets do it.
- fnordpiglet 3y agoA lot of things don’t require 100% accuracy, and raging against the worlds outdated practices doesn’t solve problems faced in the immediate present - and after spending 30 years of my career rating at those practices to absolutely no meaningful effect, a more effective bandaid that has a semantic understanding of the content is probably as good as you get. And GPU cycles are meant to be wasted.
- joquarky 3y ago"Should" is just a curse word. Be the change.
- NewJazz 3y agoLol I just don't read those PDFs. They are probably generated by shoddy automated tooling anyway. Garbage in, garbage out.
- fnordpiglet 3y agoA lot of garbage encodings encode valuable information and resilient systems follow postels law, even if you personally do not.
- humansareok1 3y agoHaving worked on this directly and used basically every other piece of "automation" software available to do this, I can tell you that the GenAI solution is far superior. It's not close.
- PH95VuimJjqBqy 3y agobecause apparently somehow ChatGPT magically knows what you're trying to extract. Oh it doesn't? odd that. ChatGPT cannot make judgement calls like what you're trying to imply it can. ChatGPT can do some really cool things, but it's not magic.
- fnordpiglet 3y agoIf you tell a multi modal LLM to extract and structure the contents of a PDF it will absolutely be able to do that successfully. Further, they display a surprising ability at abductive “reasoning” (acknowledging of course they don’t reason at all), and are thus able to make pretty reasonable assumption given the semantic context of a request. Unlike traditional extraction tools that require very specific tuning and are very fragile to structural changes in layout, LLMs tend to be very resilient to such things.
- PH95VuimJjqBqy 3y agowhat are you extracting?
- aurareturn 3y agoJust go use GPT4 and try it. You keep repeating the same rhetoric that it doesn’t work but people have made it work, including myself.
- PH95VuimJjqBqy 3y agoI pay for GPT4 enterprise, try again, only this time answer the question.
- joquarky 3y agoThis sort of bespoke automation ~~is~~ was expensive.
- fnordpiglet 3y agoSure. An example is classification. Take suppose a company that provides a platform for other companies but only if their businesses are supportable under a variety of regimes, such as industry (no cannabis say), sanctions and export controls, etc. Typically this is done by training up an army of people that poorly review websites and collateral by hand and make some determinations with a very low precision and recall, and a very high expense profile including training, benefits, HR, etc, with high attrition. Building these operations is complex, time consuming, and often fails to produce meaningfully useful results for years. With LLM at GPT4 level ability and a modest investment in scraping, context, building an agent based tool you can automate the entire thing with a pretty high precision and recall (such as a 0.9 precision with a 0.8 recall, vs 0.4 / 0.5 for humans). Using classical NLP, schema matching, and classification techniques you could triage and improve human performance by as much as 20%. But that requires a lot of effort and provides important but modest improvements. The LLM approach however is almost effortless from an engineering effort (note all of the infrastructure is the same as the NLP/schema matching/classification/etc effort). This may not seem earth shatteringly important to you, but for an enterprise this is monumental. It dramatically reduces risk, complexity, cost, and all the while improves outcomes by an enormous amount. Another example is incident management. Having LLM agents monitoring incidents on slack as reduced our MTRR dramatically. They provide summaries for new joiners, broad cast management updates, etc, and take care of all the glue out response teams typically did but with a high degree of quality, low degree of variability, and extremely low latencies compared to humans. Another example is policy management. Many enterprises are saddled with thousands of pages of external requirements from regulators, and those in turn create tens of thousands of pages of policy, and hundreds of thousands of LoC of control implementations. By indexing them into a Context+RAG LLM you can create a policy oracle, allowing you to create coverage and effectiveness testing, analyze for policy or control gaps, draft responses to auditors and regulators with total policy knowledge, answer legals questions, advise management on the implications of a business driven exception, etc. The state of the art before was a search engine dumping phrase based matches into a grid of thousands of rows of results, which was effectively useless for all of these use cases.
- PH95VuimJjqBqy 3y ago> training up an army of people that poorly review websites and collateral by hand and make some determinations with a very low precision and recall, and a very high expense profile including training, benefits, HR, etc, with high attrition. Building these operations is complex, time consuming, and often fails to produce meaningfully useful results for years. followed by > GPT4 level ability and a modest investment in scraping oh yeah, these speech patterns are those of someone who isn't unnecessarily hyped while trying to downplay other approaches. no bias here at all folks...
- joquarky 3y agoI just started using one of OpenAI's Assistants to turn unstructured job listings into structured JSON with inferred scores on dimensions that are aligned with my preferences and my attached resume.txt file. Right now, I have to copy+paste my input and outputs, but my next task will be to write a Tampermonkey script to automate it.