28 ms·

Anyone else witnessing a panic inside NLP orgs of big tech companies?

398 points by georgehill 4y ago

api 4y ago
It does seem like the (misnamed because it’s not open) OpenAI is very far ahead of most other efforts, especially at the edges in areas like instruction training and output filtering. Playing with Llama 65G gave me a sense for what the median raw effort is probably like. It seems to take a lot of work to fine tune and harness these systems and get them reliably producing useful output.
- CuriouslyC 4y ago
  I don't think it's possible to build a moat around models at all. The model architectures are public, and there are already distributed group training projects so the compute isn't a barrier. The only moat is data.
  - levidos 4y ago
    OpenAI stopped publishing the architecture of GPT-4, so I'm worried that architecture availability will not be as available in the futire
davidkuennen 4y ago
I tried translating something from English to German (my native language) yesterday with ChatGPT4 and compared it to Microsoft Translate, Google Translate and DeepL. My ranking: 1. ChatGPT4 - flawless translation. I was blown away 2. DeepL - very close, but one mistake 3. Google Translate - good translation, some mistakes 4. Microsoft Translate - bad translation, many mistakes I can understand the panic.
- davidktr 4y ago
  Fellow German here. Funny thing about DeepL: It translates "pathetisch" as "pathetic". For example: "Das war eine pathetische Rede." -> "That was a pathetic speech." I guess we have to get used to software redefining the meaning of words. It was kind of funny when that happened regarding Google Maps / neighborhood names, but with LLMs it's a different ballgame.
  - pohuing 4y ago
    Pathetic can mean emotional in English as well. Though I only discovered that by reading the dictionary. For anyone who doesn't speak German, pathetisch means with pathos, impassioned.
    - harimau777 4y ago
      This strikes me as a good example of how nuanced language can be. A native English speaker probably would only use "pathetic" to mean "emotional" if the emotions were specifically negative. They also would use pathetic to describe someone experiencing non-emotional suffering such as injury or poverty. Therefore, a native English speaker probably would not use "pathetic" to mean "emotional" in everyday writing. However, I could definitely see someone using it to mean emotional when they were being more poetic. For example, I could see someone calling an essay on the emotional toll of counseling "The Pathetic Class" in order to imply that social workers are a class that society has tasked with confronting negative emotions.
    - pyuser583 4y ago
      That’s a definition you see as technical term in Ancient Philosophy. Beyond literal translations from Greek, it doesn’t come up much.
      - xdennis 4y ago
        Yeah, if someone gives a heartfelt speech at a wedding and I call it pathetic, I don't think people will appreciate it.
    - xdennis 4y ago
      > For anyone who doesn't speak German, pathetisch means with pathos, impassioned. It's the same in Romanian, and I guess many other languages. Many of the common words of European languages are derived from Greek and Latin, and where the meaning has diverged in English, now (because of its ubiquity) these false friends are being realigned to mean what they do in English.
- zirgs 4y ago
  Does it translate hate speech too?
  - groffee 4y ago
    "In the end we shall make thoughtcrime literally impossible, because there will be no words in which to express it."
  - blurbleblurble 4y ago
    Of course it can
    - zirgs 4y ago
      ButI thought ChatGPT has guardrails that prevent it from outputting hate speech, praising certain politicians and so on.
- epups 4y ago
  I'm actually not sure what will become of tools like DeepL. Whatever edge they may have with dataset tuning and other tricks under the hood are likely superseded by a better architecture, which in turn requires a ton of capital to train. By the time they come up with a GPT4 equivalent, we will be using GPT5.
- famouswaffles 4y ago
  Tested these before GPT4 but 100%, Bi/Multi-lingual LLMs are the key to solving Machine Translation. https://github.com/ogkalu2/Human-parity-on-machine-translations https://github.com/ogkalu2/Human-parity-on-machine-translati...
- AlanYx 4y ago
  I've spent a lot of time comparing ChatGPT4 to DeepL Pro on English to French translation. ChatGPT4 is quite good, but not as consistently good as DeepL. But the interesting thing IMHO is the nature of the mistakes ChatGPT makes... often they're quite elementary mistakes (e.g., the occasional subject-adjective word order) while it gets the big picture right. Whereas DeepL is sometimes the reverse. ChatGPT also has the advantage of being able to tailor its output to a particular context, e.g., it can tailor legal translations to terms used in Canadian law rather than French law. However, for longer texts, I've noticed that ChatGPT will sometimes omit small parts of the source text from the translation, which is unfortunate. I have a colleague who says that the Mandarin translations done by ChatGPT are an order of magnitude better than DeepL though, which is interesting.
oars 4y ago
If you were an NLP researcher at a university whose past years of experience is facing existential threat due to this rapid innovation causing your area to become obsolete, what would be some good areas to pivot to or refocus on?
- echelon 4y ago
  Get out of academia and into industry. Why the hell stay in in academia? This is clearly the next technological wave, and you shouldn't sleep on it. Especially when you're so well positioned to take advantage of your experience. You can make $500,000/yr (maybe more with all the new startups and options) and be on the bleeding edge. If you want to go back to academia later, you can comfortably do so. Most don't, but that doesn't mean it isn't an option.
  - matthewdgreen 4y ago
    If you go into industry you’ll be given a chance to deploy these models and rush them into products. You’ll also make good money. If you go into academia (or research, whether it’s in academia or industry) you’ll be given the chance to try to understand what they’re doing. I can see the appeal of making money and rushing products out. But it wouldn’t even begin to compete with my curiosity. Makes me wish I was younger and could start my research career over. ETA: And though it may take longer, people who understand these models will eventually be in possession of the most valuable skill there is. Perhaps one of the last valuable human skills, if things go a certain direction.
    - CuriouslyC 4y ago
      Plot twist: as these models increase in function, complexity and size, behaviors given activations will be as inscrutable to us as our behaviors are given gene and neuron activations.
    - siva7 4y ago
      This is as likely to happen as that someone will fully understand how the brain works. I don't think you're missing much out in academia
bippingchip 4y ago
As one of the comments on reddit posts - it's not just big tech companies, but also entire university teams which feel the goalposts moving miles ahead all of a sudden. Imagine working on your PhD on chat bots since start of 2022. Your entire PhD topic might be irrelevant already...
- ChuckNorris89 4y ago
  >Imagine working on your PhD on chat bots since start of 2022. Your entire PhD topic might be irrelevant already... In fairness most PhD topics people work on these days, outside of the select few top research universities in the world, are obsolete before they begin. At least from what my friends in the field tell me.
  - Yoric 4y ago
    Anecdata of one: I finished my PhD about 20 years ago in programming language theory. I created something innovative but not revolutionary. Given how slowly industry is catching up on my domain, it will probably take another 20-30 years before something similarly powerful makes it into an industrial programming language. Counter-anecdata of one: On the other hand, one of the research teams of which I've been a member after my PhD was basically inventing Linux containers (in competition with other teams). Industry caught up pretty quickly on that. Still, academia arrived first. edit Rephrased to decrease pedantism.
    - nemaar 4y ago
      > something as powerful as what I created Could you give us more detail? It sounds intriguing.
      - Yoric 4y ago
        I developed a new static analysis (a type system, to be precise) to guarantee statically that a concurrent/distributed system could fail gracefully in case of (D)DoS or other causes of resource exhaustion. Other people in that field developed comparable tools to statically guarantee algorithmic space or time complexity of implementations (including the good use of timeouts/resource sandboxes if necessary). Or type system-level segregation between any number of layers of classified/declassified information within a system. Or type systems to guarantee that binary (byte)code produced on a machine could find all its dependencies on another machine. Or type systems to prove that an algorithm was invariant with respect to all race conditions. Or to guarantee that a non-blocking algorithm always progresses. Or to detect deadlocks statically. etc. All these things have been available in academia for a long time now. Even languages such as Rust or Scala, that offer cutting edge (for the industry) type systems, are mostly based on academic research from the 90s. For comparison, garbage-collectors were invented in the 60s and were still considered novelties in the industry in the early 2000s.
version_five 4y ago
This is imo a wake-up call about the value of having "AI teams" embedded in companies. Bad analogy- if you had an integrated circuit team in your product company building custom CPUs and Intel came out with the 8080 (or whatever was the first modern commercial chip), probably time to disband the org and use the commercial tech
martindbp 4y ago
Not big tech (or PhD level research), but half the work I did on my side project (subtitles for Chinese learning/OCR) is sort of obsolete now, most of the rest of it within a year or two. I put months into an NLP pipeline to segment Chinese sentences, classifying pinyin and translating words in-context, something ChatGPT is great at out the box. My painstaking heuristic for determining show difficulty using word frequencies and comparing distributions to children's shows is now the simple task of giving part of the transcript and asking ChatGPT how difficult it is. Next up, the OCR I did will probably be solved by ChatGPT4. It seems the writing is on the wall: most tasks on standard media (text/images/video), will be "good enough" for non-critical use. The only remaining advantage of bespoke solutions is speed and cost and that will also be a fleeting advantage. But it's also extremely exciting, we'll be able to build really great things very easily, and focus our efforts elsewhere. Today anyone can throw together a language learning tutor to rival Duolingo. As long as you're in it for solving problems you shouldn't be too threatened by whatever tool set you're currently becoming obsolete.
djous 4y ago
During my master's degree in data science, we had several companies visit our faculty to recruit students. Not a single one was a specialized NLP company, but many of them had NLP projects going on. Most of those projects were the usual "solution looking for a problem to solve". Even those projects that might have had _some_ utility, would have been way more effective to buy/license a product than to develop an in-house solution. Because really, what's the use of throwing a dozen 25-30 years old with non-specialized knowledge, when there are companies full of guys with PhDs in NLP that devote all their resources to NLP? Yeah, you can pipe together some python, but these kind of products will always be subpar and more expensive long-term than just buying a proper solution from a specialized company. To me it was pretty clear that those projects were just PR so that c-levels could sell how they were preparing their company for a digital world. Can't say I'm sorry for all the people working on those non-issues though. From the attitude of recruiters and employees, you'd think they were about to find a cure for cancer. Honestly, I can't wait for GPT and other productivity tools to wrech havock upon the tech labour market. Some people in tech really need to be taken down a notch or two.
- echelon 4y ago
  > Honestly, I can't wait for GPT and other productivity tools to wrech havock upon the tech labour market. Some people in tech really need to be taken down a notch or two. That's an odd reason to want this.
  - MomoXenosaga 4y ago
    Less bullshit jobs. Society needs doctors, nurses, plumbers and teachers not tech bros.
    - piva00 4y ago
      Society definitely needs those, but the incentives of the system most societies live under do not align to those needs. We are 100% into a society of wants, not needs, and the rewards are for those who sell stuff for these wants. Our needs went into the "cost center" of society's calculation, not an investment, and so it's been a race to the bottom for those professions. While adtech, crypto and other bullshit gets massive funding because it can turn a profit. The incentives to have a good society don't align with the incentives of financial capitalism.
credit_guy 4y ago
They may panic, but they shouldn't. They can quickly pivot. GPT programs can be used off the shelf, but they can also use custom training. Every large org has a huge internal set of documents, plus a large external set of documents relevant to its work (research articles, media articles, domain relevant rules and regulations). They can train a GPT bot to their particular codebase. And that is now. Soon (I'd give it at most one year), we'll be able to train GPT bots to videos. All this training does not happen by itself.
- nr2x 4y ago
  100%. Anybody with experience in distributed systems, networking, or SRE knows the plumbing can be as challenging as the “big idea”. Training these models is a plumbing job. And that’s actually really hard to pull off.
- MonkeyMalarky 4y ago
  Yeah this thread has been the motivation for me to sign up on the wait list and cost out what it would take to try fine-tuning their older models on our data. There's still plenty of work out there when it comes to building a solution to a problem.
- Oras 4y ago
  Have you seen Microsoft demo yesterday https://youtu.be/Bf-dbS9CcRU https://youtu.be/Bf-dbS9CcRU
hnbad 4y ago
When I was studying Computational Linguistics I kept running into the unspoken question: given that Google Translate already exists, what is even the point of all of this? We were learning all these ideas about how to model natural language and tag parts of speech using linguistic theory so we could eventually discover that utopian solution that would let us feed two language models into a machine to make it perfectly translate a sentence from one language into another. And here was Google Translate being "good enough" for 80% of all use cases using a "dumb" statistic model that didn't even have a coherent concept of what a language is. It's been close to two decades and I still wonder if that "pure" approach has any chance of ever turning into something useful. Except now it's not just language but "AI" in general: ChatGPT is not an AGI, it's a model fed with prose that can generate coherent responses for a given input. It doesn't always work out right and it "hallucinates" (i.e. bullshits) more than we'd like but it feels like this is a more economically viable shot at most use cases for AGI than doing it "right" and attempting to create an actual AGI. We didn't need to teach computers how language works in order to get them to provide adequate translations. Maybe we also don't need to teach them how the world works in order to get them to provide answers about it. But it will always be a 80% solution because it's an evolutionary dead end: it can't know things, we have only figured out how to trick it into pretending that it does.
- hnfong 4y ago
  I learnt some very basics of computational linguistics since it was related to a side project. I kept wondering why people were spending huge amounts of resources into tagging and labelling corpora of thousands of words, while to me it seems that in theory it should be possible to feed wikipedia (of a certain language) into a program and have it spit out some statistically correct rules about words and grammar. I guess the same intuition led to these new AI technologies...
  - hnbad 4y ago
    I think a huge part is that computational linguistics still chases the idea of a universal language model, which may simply not be possible. I haven't followed the science in general linguistics but something feels off when most of the information ends up being tagged onto nil particles (i.e. parts of speech present neither in utterances nor written language and not affecting intonation or otherwise being detectable except by contrasting the structure with related languages).
epups 4y ago
Everyone here is saying that people can simply transition easily into startups and other big companies. To a certain extent that's true, but what exactly are they going to do? As technology consolidates into one or two major LLM's, likely only accessible by API, I feel most orgs would be better served by relying heavily on finetuning or optimizing those for their purpose. Previous experience with NLP certainly helps with that, although this type of work would not necessarily be as exciting as trying to build the next big thing, which everyone was scrambling for before. OpenAI could build a state-of-the-art tool with a few hundred developers - to me, that means that money will converge to them and other big orgs rather than the opposite.
- Yoric 4y ago
  That's definitely a risk. With a PhD in the domain, I consider myself pretty good at (a subset of) distributed programming. But these days, when companies hire for distributed programming, they seem to want developers who know a specific set of tools and APIs. I'm more suited at reimplementing them for scratch.
twawaaay 4y ago
I think education goal for people shifted. I teach my kids to be flexible and embrace the change. Invest in abilities that transfer well to various things you could be doing during your life. Be a problem solver. In the future -- forget about cosy job you can be doing for the rest of your life. You no longer have any guarantees even if you own the business and even if you are farmer. What you absolutely don't want is spend X years at uni learning something, and then 5-10 years into your "career" finding out it was obsoleted overnight and you now don't have plan B.
- rr888 4y ago
  Liberal arts education will one day be back in fashion.
  - twawaaay 4y ago
    Oh I do believe it. There will always be a market for snobs who will want to pay extra for handmade things vs AI-generated. The issue here is that it is all driven by fads and unstable. If you want to make money you will have to be flexible.
- SketchySeaBeast 4y ago
  > What you absolutely don't want is spend X years at uni learning something, and then 5-10 years into your "career" finding out it was obsoleted overnight and you now don't have plan B. That seems to be running directly opposite of the current trend of admin assistant jobs requiring 2 years specialized admin assistant diplomas. Tech (and I would guess the world of the business MBA) is a unique space where people are learning and changing so quickly, but for a lot of those outside the bubble things seem to be calcifying and requiring more and more training at the expensive of the worker.
- yoyohello13 4y ago
  Really the only safe career in the moderate future is going to be manual labor. There is always need to send a bunch of humans into the middle of nowhere to dig ditches.
twa34532 4y ago
oh no?! so finally the tech sector is experiencing themselves what they have done to other lines of professions for the past decades, namely eradicting them (rightfully) with innovation? well same advice applies then: * embrace, move on and retrain for another profession * learn empathy from the panic and hurt
TMWNN 4y ago
Is the entire field of data science (Itself maybe a decade old in terms of being a college major?) now obsolete, in terms of being a distinct job field? Are all data science majors now going to be "just" coming up with the proper prompts to get GPT to correctly massage datasets?
- theGnuMe 4y ago
  No. It's always been about posing the right question.
belter 4y ago
A whole thread on AI experts discussing how AI is making them obsolete...back to gardening...
- izacus 4y ago
  This fad too shall pass. And the tech will end up where it always does: helping some, changing some but nowhere near as much as the gold rush profiteers would make you believe.
  - emptysongglass 4y ago
    This is not an event that calls for pithy adages. The fruits of ML are not a fad just like personal computing was not a fad. It's a watershed event that cuts across every knowledge worker's domain. If you're not currently using these LLMs it may not be obvious to you but those of us that have tried to apply them to our current fields see huge gains in productivity. Just in my own little slice of knowledge work, I've seen yield increases that have saved me multiple days of work on a single project. Everyone is going to feel this, most prominently people in the sorts of industries that frequent HN. If you haven't yet, you will or you will be forced to when you discover everyone in your field is out-producing you armed with these tools.
    - izacus 4y ago
      Uh-huh. How's them NFTs and Blockchain doing the watershed world changin these days?
      - deleted 4y ago
        [deleted]
      - eggsmediumrare 4y ago
        Your comment seems almost intentionally obtuse... How's them internet and smartphones doing the watershed world changing these days?
MonkeyMalarky 4y ago
I'm not at a big tech company, and we don't sell algorithms, but my team does use a lot of NLP stuff in internal algorithms. The only panic I have is trying to keep up and take the time to learn the new stuff. If anything, things like GPT-4 are going to make my team 10x more successful without having to hire an army of PhDs.
- jarebear6expepj 4y ago
  The PhD army will rise up against us one day... as soon as they are finish their TA appointments.
- not-chatgpt 4y ago
  What does your team do? It feels like GPT4 can handle any task out there. Only drawback is latency and cost.
  - MonkeyMalarky 4y ago
    The price isn't even that bad, even the most expensive at 6cents per 1k tokens, it won't cost me much. It's the context size that's amazing. Gone are the days of only being able to pass ~500 tokens into something like BERT.
dserban 4y ago
The PR folks at my current company are in full panic mode on Linkedin, judging from the passive-aggressive tone of their posts (sometimes very nearly begging customers not to use ChatGPT and friends). They fully understand that LLMs are stealing lunch money from established information retrieval industry players selling overpriced search algorithms. For a long time, my company was deluded about being protected by insurmountable moats. I'm watching our PR folks going through the five stages of grief very loudly and very publicly on social media (particularly noticeable on Linkedin). Here's a new trend happening these days. Upon releasing new non-fiction books to the general public, authors are simultaneously offering an LLM-based chatbot box where you can ask the book any question. There is no good reason this should not work everywhere else, in exactly the same way. Take for example a large retailer who has a large internal knowledge base. Train an LLM on that corpus, ask the knowledge base any question. And retail is a key target market of my company. Needless to say I'm looking for employment elsewhere.
- mashygpig 4y ago
  > Here's a new trend happening these days. Upon releasing new non-fiction books to the general public, authors are simultaneously offering an LLM-based chatbot box where you can ask the book any question. Can you link to an example?
  - dserban 4y ago
    I saw at least two examples of this here on HN. One of the books was about tech entrepreneurship 101, and I remember asking how to launch if you're a sole developer with no legal entity behind the product. I remember the answer being fairly coherent and useful. I don't have the URL handy, I suspect if you search HN for "entrepreneur book" you'll find it.
  - org3 4y ago
    https://portal.konjer.xyz/ https://portal.konjer.xyz/
    - throwayyy479087 4y ago
dongobread 4y ago
I worked in a research capacity in the voice assistant org of a big tech company until very recently. There was a lot of panic when ChatGPT came out, as it became clear that the vast bulk of the org's modeling work and research essentially had no future. I feel bad for some of my colleagues who were really specialized in specific NLP technology niches (e.g. building NLU ontologies) which have been made totally obsolete by these generalized LLMs. Personally - I'm moving to more of a focus on analytical modeling. There is really nothing interesting about deep learning to me anymore. The reality is that any new useful DL models will be coming out of mega-teams in a few companies, where improving output through detailed understanding of modeling is less cost effective than simply increasing data quality and scale. Its all very boring to me.
- mr_toad 4y ago
  “ Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. “ http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
  - shawntan 4y ago
    I've seen many interpretations of this article and I'm curious as to the mainstream CS reading of it. One could look at the move from linear models to non-linear models or the use of ConvNets (yes I know ViTs exist, to my knowledge the base layers are still convolution layers) as 'leveraging human knowledge'. Only after those shifts were made did the leveraging of computation help. It would seem to me that the naive reading of that quote only rings true between breakthroughs.
  - jnwatson 4y ago
    See also the GPT-4 technical report, page 37. https://images.app.goo.gl/vRP8368Z17zW2hvC9 https://images.app.goo.gl/vRP8368Z17zW2hvC9
    - 4y ago
macinjosh 4y ago
Is the compute for running an LLM cheap enough to scale at the moment? LLMs seem to be a great generalist solution but could specifically targeted NLP solutions still outperform in terms of speed/cost when you are processing high volumes of inputs?
wunderland 4y ago
Some big tech companies are witnessing a panic inside their entire org because they focus almost entirely on their competitors (except for the business divisions which are monopolies).
- KrugerDunnings 4y ago
  Some people look at ChatGPT and think its all over and other look at it and start imagining all the things they can use it for.
rdedev 4y ago
My university professor who specialises in NLP kinda feels like what's the point of research in the time of chatgpt. He says for now it's not possible to scale retrieval easily when using these llms so that's what he is looking into for now
hn_throwaway_99 4y ago
Wow - this is just wild. I've seen lots of arguments around "AI won't take everyone's job, it will just open up new areas for new jobs." Even if you take that with the benefit of the doubt (which I don't really think is warranted): 1. You don't need to take everyone's job. You just need to take a shitload of people's jobs. I think a lot of our current sociological problems, problems associated with wealth inequality, etc., are due to the fact that lots of people no longer have competitive enough skills because technology made them obsolete. 2. The state of AI progress makes it impossible for humans in many fields to keep up. Imagine if you spent your entire career working on NLP, and now find GPT-4 will run rings around whatever you've done. What do you do now? I mean, does anyone think that things like human translators, medical transcriptionists, court reporters, etc. will exist as jobs at all in 10-20 years? Maybe 1-2 years? It's fine to say "great, that can free up people for other thing", but given our current economic systems, how are these people supposed to eat? EDIT: I see a lot of responses along the lines of "Have you seen the bugs Google/Bing Translate has?" or "Imagine how frustrated you get with automated chat bots now!" Gang, the whole point is that GPT-4 blows these existing models out of the water. People who work in these fields are blown away by the huge advances in quality of output in just a short time. So I'm a bit baffled why folks are comparing the annoyances of ordering at a McDonald's automated kiosk to what state-of-the-art LLMs can do. And reminder that the first LLM was only created in 2018.
- mostlysimilar 4y ago
  This is possibly a death spiral. GPT is only possible because it's been trained on the work humans have learned to do and then put out in the world. Now GPT is as good as them and will put them all out of work. How can it improve if the people who fed it are now jobless?
  - jgust 4y ago
    Presumably this problem is solved with technology improvements or the need is recognized to hire experts capable of generating high quality training material. In either situation, there's going to be extreme discomfort.
gniv 4y ago
I remember thinking about this when AlphaFold was announced. Did it happen back then? Were there large shifts in companies/universities that were doing folding research?
- jhrmnn 4y ago
  I’ve been thinking about this. My current theory is that molecular simulation is a much more heterogeneous activity than language modeling. Language is always the same kind of data. Molecular simulations span orders of magnitude in space and time and depending on that, data and even objectives have very different form. AlphaFold is just one small piece in this puzzle and it’s very easy for a research project to incorporate AlphaFold into an existing pipeline and shift its goal.
deleted 4y ago
[deleted]
jurassic 4y ago
Maybe this is alarmist, but I don't see how LLMs don't collapse our entire economic system over the next decade or so. This is coming for all of us, not just the NLP experts in big company research groups. Being able to cheaply/instantly perform virtually any task is great until you realize there is now nobody left to buy your product or service because the entire middle class has been put out of work by LLMs. And the service industries that depend on those middle class knowledge workers will be out of work because nobody can afford to purchase their services. I don't see how this doesn't end with guillotines coming out for the owner class and/or terrorism against the companies powering this revolution. I hope I'm wrong.
- qwerty3344 4y ago
  There are entire sectors of the economy that LLMs can't touch - hospitality, manufacturing, caregivers, religious sectors, live-action entertainment, etc. Sure some of these will be replaced by robots but there will always be new jobs too.
  - jurassic 4y ago
    No, there are not. Everything in the economy is connected and you can't have a vibrant industry without customers. The customers of hospitality/entertainment/healthcare/etc businesses are largely the middle class who will be put out of work by LLMs. So the person who today makes $200/night in tips waiting tables at a nice restaurant.... who will be buying those meals?
    - woah 4y ago
      Someone who uses an LLM as a tool to perform a useful service
      - seydor 4y ago
        That would be another LLM or a robot then
      - bilbo0s 4y ago
tippytippytango 4y ago
Not even experts in the domain could see themselves being replaced and pivot in time. What hope does an ordinary person have in preparing for what’s coming? Telling people to retrain will not be an acceptable answer because no one can predict which skills will be safe from AI in 5 years.
bsder 4y ago
I guess I'm not panicked about my job in the face of AI because objective correctness is required. I dream about the day that OpenAI can write the 100 lines of code that connect the BLE stack, the ADC sensor and the power management code so that my IoT sensor doesn't crash once every 8 days. I see the AI stuff as very different from, say, the microcomputer revolution. People had LOTS of things they wanted to use computers for, but the computers were simply too expensive. As soon as microprocessors arrived, people had LOTS of things they were already waiting to apply them to. Factory automation was screaming for computers. Payroll management was screaming for computers. I don't see that with the current AI stuff. What thing was waiting for NLP/OpenAI to get good enough? Yes, things like computer games opened up whole new vistas, and maybe AI will do that, but that's a 20 year later thing. What stuff was screaming for AI right now? Maybe transcription? When I see the search bar on any of my favorite forums suddenly become useful, I'll believe that OpenAI stuff actually works. Finally, the real problem is that OpenAI needs to cough up what I want but then it needs to cough up the original references to what I want. I normally don't make other humans do that. If I'm asking someone for advice, I've already ascertained that I can trust them and I'm probably going to accept their answers. If it's random conversation and interesting or unusual, I'll mark it, but I'm not going to incorporate it until I verify. Although, given the current political environment, pehaps I should ask other humans to give me more references.
drewda 4y ago
I wonder if this will be a repeat of what happened with speech recognition. It used to be a specialized field dominated by smaller companies like Nuance. More recently Google, Microsoft, Apple, etc. decided they wanted to have speech recognition as an internal piece of their platforms. Google poached lots of Nuance's talent. And then Microsoft bought what remained of the company. Now speech recognition is a service integrated into the larger tech company's platforms, and also uses their more statistical/ML approaches, rather than being a component created by specialist companies/groups. (I'm sure I'm grossly simplifying this — just seeing a potential parallel.)
deepsquirrelnet 4y ago
I work at a small company, but it’s hard for me to imagine that generative AI will replace predictive AI/ML any time soon. Smaller models trained supervised/in-domain are simply more efficient and more accurate than unsupervised/out-of-domain. Plus we own and operate the technology much more cheaply. I don’t doubt that if your were trying to build a competing product to what OpenAI is doing that you’d feel affected, but there’s also a lot of other problems that are not being solved by generative models.
dr_dshiv 4y ago
Sunk cost! This is why so many AI scientists are so skeptical. They don’t use chatGPT because they spent years developing skills that enabled them to use LLMs and now any kid on the street can.
anshumankmr 4y ago
Here's my two cents, as I work with NLP in a tech company, mostly with Dialogflow and Rasa, in my current project, we are using Chat GPT (and previously GPT 3) to summarize articles, and I see that it can really handle FAQ questions really well. One of most common requirements was to train our bot to handle FAQ type question apart from complex conversation stories/flows,but this thing can straight up take the content from an article,summarize it neatly and send a response back. We have had some issues and complaints with the API,( mostly with GPT 3 as the fine tuning was only open for the base model and that had some trouble with some questions). Also there is a finicky response time, despite using having paid access. Response time varies from 10 seconds to even a minute (during some downtime that occured a few days ago, and a few days even before that there was a complete outage).
nathan_f77 4y ago
I would have never guessed that menial jobs would be the hardest ones to automate. I realized that humans have some incredible skills that were vital to our survival, and were honed over billions of years. But we take these skills for granted because most humans are born with these natural abilities, which are actually very impressive in the grand scheme of things. A janitor might not get paid very much relative to other jobs, but it took billions of years of evolution to hone their brain, muscles, balance, hand eye coordination, energy efficiency, information processing, etc. We're actually very impressive creatures!
orsenthil 4y ago
The book "Who Moved My Cheese?" tries to explain the human behavior for the moments like this. It is happening for many people now.
xvilka 4y ago
If someone build a software that would be able to attend a meeting instead of you - it will become as popular as Zoom. Combine LLM with voice and video recognition and synthesis, and voila - an immense chunk of wasted time in every corporation is gone.
WalterBright 4y ago
I see chatGPT as eliminating an awful lot of drudgery. For example, back in the 60's my dad was working on his book. The text was typed out double spaced, and he (and others) would make corrections. After a while, my mom would retype the whole thing. Imagine typing a whole book. Again and again and again. She'd type hour after hour. It's dehumanizing. And then came word processors. What a magical revolution! You could edit text instead of typing it all over again. I bet few people today realize what a great achievement that was. All chatgpt does is select the most likely next word out of a corpus of existing text. It is not creative. We don't need rooms full of typists anymore. Good riddance. I bet we get rid of a bunch of drudgery jobs with chatgpt.
- KyeRussell 4y ago
  It sounds like you’re giving a lot of human ‘occupations’ a whole lot of undeserved credit. There are stark differences between these two scenarios. Remixing existing ideas in an incredibly abstract way is not drudgery, it’s what a lot of people do, and get enjoyment out of.
  - WalterBright 4y ago
    AI will automate the drudgery end, which is not "incredibly" anything.
sidcool 4y ago
We thought AI will take out the mechanical and manual jobs first that don't need intellectual capabilities, but only repetition. Who knew AI will first learn to write poems, novels and code.
PaulHoule 4y ago
I look at it the other way. Much of what people use GPT-4 can be done better (with a little more work) with transformer models that are specialized with the task. Things that were a struggle 5 years ago are about to be easy.
- billythemaniam 4y ago
  Definitely true for GPT-3, but after using GPT-4, I am not so sure. I am sure there are cases where a specialized NLP model can beat GPT-4, but I think they might be in the minority now. Whether I tried summarization, data extraction, or code generation, GPT-4 just kinda worked.
mdip 4y ago
Fascinating -- I think the comments on the HN post are almost as good. I think everyone mostly agrees that AI is coming for a lot of jobs. There's disagreement about how many, how it will impact society and the like. The pace of technology is not linear, it accelerates. I've never seen something that has so rapidly crossed into the "magical" territory as "nearly every single big LLM/Generative AI thing" seems to. It redefines what was previously laughably impossible ... a decade ago. We're riding a curve upward that is making it extremely hard to see what's coming next. All of the pontificating, all of the attempts at finding solutions to imagined problems ... I can't see one that doesn't feel like a blindfolded person aiming at what they were told was a dart board with what they were told was a dart. There's really nothing to do but hang on and hope you land where any new opportunities creep up. Expect bubbles, black swans, and purple unicorns.
chatmasta 4y ago
Maybe semantics, but... The implication of this post is that ChatGPT is causing the demise of NLP jobs, which, sure, that's true - but I don't think it's fair to classify this as replacing the job of anyone who works on NLP. This is not the same as GPT replacing a human translator, which is a direct replacement of their role - it's not like ChatGPT is going to start researching NLP. It's just a fairly mundane example of a superior technology outpacing an inferior one. The technology of NLP is being obsoleted, not the job of researching NLP (which is just unnecessary, not being replaced with something else). And in fact, NLP researchers are probably some of the most well positioned people to move laterally to working on transformers and LLMs.
thinkr42 4y ago
This mostly makes me wonder about the varying definitions of NLP. Most business folks I’ve worked with in this field don’t even know the question to ask or prompt- the implementation details of models like GPT are the easy part.
skc 4y ago
Somebody in that thread is speculating that the company in question is grammarly. I'm not sure but I'm now curious as to what the execs there are thinking, especially now with the recent Microsoft 365 news. Feels like the body blows keep coming.
specproc 4y ago
As someone who's a crap NLP practitioner, everything is just fine and dandy. I've never really had the gear or the skills to put together anything that improves over what I can pull from huggingface. What I do have, and virtually none of my (not remotely technical) colleagues have, is a clue what to do with all this stuff. They reckon it's about churning out poems and boilerplate text, the minute I figured it could give me whatever json I could reasonably ask for from a source doc, I was overjoyed. I see more things I can be doing now, not a risk of being replaced.

398 points by georgehill 4y ago

api 4y ago

It does seem like the (misnamed because it’s not open) OpenAI is very far ahead of most other efforts, especially at the edges in areas like instruction training and output filtering. Playing with Llama 65G gave me a sense for what the median raw effort is probably like. It seems to take a lot of work to fine tune and harness these systems and get them reliably producing useful output.

CuriouslyC 4y ago
I don't think it's possible to build a moat around models at all. The model architectures are public, and there are already distributed group training projects so the compute isn't a barrier. The only moat is data.
- levidos 4y ago
  OpenAI stopped publishing the architecture of GPT-4, so I'm worried that architecture availability will not be as available in the futire

davidkuennen 4y ago

I tried translating something from English to German (my native language) yesterday with ChatGPT4 and compared it to Microsoft Translate, Google Translate and DeepL. My ranking: 1. ChatGPT4 - flawless translation. I was blown away 2. DeepL - very close, but one mistake 3. Google Translate - good translation, some mistakes 4. Microsoft Translate - bad translation, many mistakes I can understand the panic.

davidktr 4y ago
Fellow German here. Funny thing about DeepL: It translates "pathetisch" as "pathetic". For example: "Das war eine pathetische Rede." -> "That was a pathetic speech." I guess we have to get used to software redefining the meaning of words. It was kind of funny when that happened regarding Google Maps / neighborhood names, but with LLMs it's a different ballgame.
- pohuing 4y ago
  Pathetic can mean emotional in English as well. Though I only discovered that by reading the dictionary. For anyone who doesn't speak German, pathetisch means with pathos, impassioned.
  - harimau777 4y ago
    This strikes me as a good example of how nuanced language can be. A native English speaker probably would only use "pathetic" to mean "emotional" if the emotions were specifically negative. They also would use pathetic to describe someone experiencing non-emotional suffering such as injury or poverty. Therefore, a native English speaker probably would not use "pathetic" to mean "emotional" in everyday writing. However, I could definitely see someone using it to mean emotional when they were being more poetic. For example, I could see someone calling an essay on the emotional toll of counseling "The Pathetic Class" in order to imply that social workers are a class that society has tasked with confronting negative emotions.
  - pyuser583 4y ago
    That’s a definition you see as technical term in Ancient Philosophy. Beyond literal translations from Greek, it doesn’t come up much.
    - xdennis 4y ago
      Yeah, if someone gives a heartfelt speech at a wedding and I call it pathetic, I don't think people will appreciate it.
  - xdennis 4y ago
    > For anyone who doesn't speak German, pathetisch means with pathos, impassioned. It's the same in Romanian, and I guess many other languages. Many of the common words of European languages are derived from Greek and Latin, and where the meaning has diverged in English, now (because of its ubiquity) these false friends are being realigned to mean what they do in English.
zirgs 4y ago
Does it translate hate speech too?
- groffee 4y ago
  "In the end we shall make thoughtcrime literally impossible, because there will be no words in which to express it."
- blurbleblurble 4y ago
  Of course it can
  - zirgs 4y ago
    ButI thought ChatGPT has guardrails that prevent it from outputting hate speech, praising certain politicians and so on.
epups 4y ago
I'm actually not sure what will become of tools like DeepL. Whatever edge they may have with dataset tuning and other tricks under the hood are likely superseded by a better architecture, which in turn requires a ton of capital to train. By the time they come up with a GPT4 equivalent, we will be using GPT5.
famouswaffles 4y ago
Tested these before GPT4 but 100%, Bi/Multi-lingual LLMs are the key to solving Machine Translation. https://github.com/ogkalu2/Human-parity-on-machine-translations https://github.com/ogkalu2/Human-parity-on-machine-translati...
AlanYx 4y ago
I've spent a lot of time comparing ChatGPT4 to DeepL Pro on English to French translation. ChatGPT4 is quite good, but not as consistently good as DeepL. But the interesting thing IMHO is the nature of the mistakes ChatGPT makes... often they're quite elementary mistakes (e.g., the occasional subject-adjective word order) while it gets the big picture right. Whereas DeepL is sometimes the reverse. ChatGPT also has the advantage of being able to tailor its output to a particular context, e.g., it can tailor legal translations to terms used in Canadian law rather than French law. However, for longer texts, I've noticed that ChatGPT will sometimes omit small parts of the source text from the translation, which is unfortunate. I have a colleague who says that the Mandarin translations done by ChatGPT are an order of magnitude better than DeepL though, which is interesting.

oars 4y ago

If you were an NLP researcher at a university whose past years of experience is facing existential threat due to this rapid innovation causing your area to become obsolete, what would be some good areas to pivot to or refocus on?

echelon 4y ago
Get out of academia and into industry. Why the hell stay in in academia? This is clearly the next technological wave, and you shouldn't sleep on it. Especially when you're so well positioned to take advantage of your experience. You can make $500,000/yr (maybe more with all the new startups and options) and be on the bleeding edge. If you want to go back to academia later, you can comfortably do so. Most don't, but that doesn't mean it isn't an option.
- matthewdgreen 4y ago
  If you go into industry you’ll be given a chance to deploy these models and rush them into products. You’ll also make good money. If you go into academia (or research, whether it’s in academia or industry) you’ll be given the chance to try to understand what they’re doing. I can see the appeal of making money and rushing products out. But it wouldn’t even begin to compete with my curiosity. Makes me wish I was younger and could start my research career over. ETA: And though it may take longer, people who understand these models will eventually be in possession of the most valuable skill there is. Perhaps one of the last valuable human skills, if things go a certain direction.
  - CuriouslyC 4y ago
    Plot twist: as these models increase in function, complexity and size, behaviors given activations will be as inscrutable to us as our behaviors are given gene and neuron activations.
  - siva7 4y ago
    This is as likely to happen as that someone will fully understand how the brain works. I don't think you're missing much out in academia

bippingchip 4y ago

As one of the comments on reddit posts - it's not just big tech companies, but also entire university teams which feel the goalposts moving miles ahead all of a sudden. Imagine working on your PhD on chat bots since start of 2022. Your entire PhD topic might be irrelevant already...

ChuckNorris89 4y ago
>Imagine working on your PhD on chat bots since start of 2022. Your entire PhD topic might be irrelevant already... In fairness most PhD topics people work on these days, outside of the select few top research universities in the world, are obsolete before they begin. At least from what my friends in the field tell me.
- Yoric 4y ago
  Anecdata of one: I finished my PhD about 20 years ago in programming language theory. I created something innovative but not revolutionary. Given how slowly industry is catching up on my domain, it will probably take another 20-30 years before something similarly powerful makes it into an industrial programming language. Counter-anecdata of one: On the other hand, one of the research teams of which I've been a member after my PhD was basically inventing Linux containers (in competition with other teams). Industry caught up pretty quickly on that. Still, academia arrived first. edit Rephrased to decrease pedantism.
  - nemaar 4y ago
    > something as powerful as what I created Could you give us more detail? It sounds intriguing.
    - Yoric 4y ago
      I developed a new static analysis (a type system, to be precise) to guarantee statically that a concurrent/distributed system could fail gracefully in case of (D)DoS or other causes of resource exhaustion. Other people in that field developed comparable tools to statically guarantee algorithmic space or time complexity of implementations (including the good use of timeouts/resource sandboxes if necessary). Or type system-level segregation between any number of layers of classified/declassified information within a system. Or type systems to guarantee that binary (byte)code produced on a machine could find all its dependencies on another machine. Or type systems to prove that an algorithm was invariant with respect to all race conditions. Or to guarantee that a non-blocking algorithm always progresses. Or to detect deadlocks statically. etc. All these things have been available in academia for a long time now. Even languages such as Rust or Scala, that offer cutting edge (for the industry) type systems, are mostly based on academic research from the 90s. For comparison, garbage-collectors were invented in the 60s and were still considered novelties in the industry in the early 2000s.

version_five 4y ago

This is imo a wake-up call about the value of having "AI teams" embedded in companies. Bad analogy- if you had an integrated circuit team in your product company building custom CPUs and Intel came out with the 8080 (or whatever was the first modern commercial chip), probably time to disband the org and use the commercial tech

martindbp 4y ago

Not big tech (or PhD level research), but half the work I did on my side project (subtitles for Chinese learning/OCR) is sort of obsolete now, most of the rest of it within a year or two. I put months into an NLP pipeline to segment Chinese sentences, classifying pinyin and translating words in-context, something ChatGPT is great at out the box. My painstaking heuristic for determining show difficulty using word frequencies and comparing distributions to children's shows is now the simple task of giving part of the transcript and asking ChatGPT how difficult it is. Next up, the OCR I did will probably be solved by ChatGPT4. It seems the writing is on the wall: most tasks on standard media (text/images/video), will be "good enough" for non-critical use. The only remaining advantage of bespoke solutions is speed and cost and that will also be a fleeting advantage. But it's also extremely exciting, we'll be able to build really great things very easily, and focus our efforts elsewhere. Today anyone can throw together a language learning tutor to rival Duolingo. As long as you're in it for solving problems you shouldn't be too threatened by whatever tool set you're currently becoming obsolete.

djous 4y ago

During my master's degree in data science, we had several companies visit our faculty to recruit students. Not a single one was a specialized NLP company, but many of them had NLP projects going on. Most of those projects were the usual "solution looking for a problem to solve". Even those projects that might have had _some_ utility, would have been way more effective to buy/license a product than to develop an in-house solution. Because really, what's the use of throwing a dozen 25-30 years old with non-specialized knowledge, when there are companies full of guys with PhDs in NLP that devote all their resources to NLP? Yeah, you can pipe together some python, but these kind of products will always be subpar and more expensive long-term than just buying a proper solution from a specialized company. To me it was pretty clear that those projects were just PR so that c-levels could sell how they were preparing their company for a digital world. Can't say I'm sorry for all the people working on those non-issues though. From the attitude of recruiters and employees, you'd think they were about to find a cure for cancer. Honestly, I can't wait for GPT and other productivity tools to wrech havock upon the tech labour market. Some people in tech really need to be taken down a notch or two.

echelon 4y ago
> Honestly, I can't wait for GPT and other productivity tools to wrech havock upon the tech labour market. Some people in tech really need to be taken down a notch or two. That's an odd reason to want this.
- MomoXenosaga 4y ago
  Less bullshit jobs. Society needs doctors, nurses, plumbers and teachers not tech bros.
  - piva00 4y ago
    Society definitely needs those, but the incentives of the system most societies live under do not align to those needs. We are 100% into a society of wants, not needs, and the rewards are for those who sell stuff for these wants. Our needs went into the "cost center" of society's calculation, not an investment, and so it's been a race to the bottom for those professions. While adtech, crypto and other bullshit gets massive funding because it can turn a profit. The incentives to have a good society don't align with the incentives of financial capitalism.

credit_guy 4y ago

They may panic, but they shouldn't. They can quickly pivot. GPT programs can be used off the shelf, but they can also use custom training. Every large org has a huge internal set of documents, plus a large external set of documents relevant to its work (research articles, media articles, domain relevant rules and regulations). They can train a GPT bot to their particular codebase. And that is now. Soon (I'd give it at most one year), we'll be able to train GPT bots to videos. All this training does not happen by itself.

nr2x 4y ago
100%. Anybody with experience in distributed systems, networking, or SRE knows the plumbing can be as challenging as the “big idea”. Training these models is a plumbing job. And that’s actually really hard to pull off.
MonkeyMalarky 4y ago
Yeah this thread has been the motivation for me to sign up on the wait list and cost out what it would take to try fine-tuning their older models on our data. There's still plenty of work out there when it comes to building a solution to a problem.
Oras 4y ago
Have you seen Microsoft demo yesterday https://youtu.be/Bf-dbS9CcRU https://youtu.be/Bf-dbS9CcRU

hnbad 4y ago

When I was studying Computational Linguistics I kept running into the unspoken question: given that Google Translate already exists, what is even the point of all of this? We were learning all these ideas about how to model natural language and tag parts of speech using linguistic theory so we could eventually discover that utopian solution that would let us feed two language models into a machine to make it perfectly translate a sentence from one language into another. And here was Google Translate being "good enough" for 80% of all use cases using a "dumb" statistic model that didn't even have a coherent concept of what a language is. It's been close to two decades and I still wonder if that "pure" approach has any chance of ever turning into something useful. Except now it's not just language but "AI" in general: ChatGPT is not an AGI, it's a model fed with prose that can generate coherent responses for a given input. It doesn't always work out right and it "hallucinates" (i.e. bullshits) more than we'd like but it feels like this is a more economically viable shot at most use cases for AGI than doing it "right" and attempting to create an actual AGI. We didn't need to teach computers how language works in order to get them to provide adequate translations. Maybe we also don't need to teach them how the world works in order to get them to provide answers about it. But it will always be a 80% solution because it's an evolutionary dead end: it can't know things, we have only figured out how to trick it into pretending that it does.

hnfong 4y ago
I learnt some very basics of computational linguistics since it was related to a side project. I kept wondering why people were spending huge amounts of resources into tagging and labelling corpora of thousands of words, while to me it seems that in theory it should be possible to feed wikipedia (of a certain language) into a program and have it spit out some statistically correct rules about words and grammar. I guess the same intuition led to these new AI technologies...
- hnbad 4y ago
  I think a huge part is that computational linguistics still chases the idea of a universal language model, which may simply not be possible. I haven't followed the science in general linguistics but something feels off when most of the information ends up being tagged onto nil particles (i.e. parts of speech present neither in utterances nor written language and not affecting intonation or otherwise being detectable except by contrasting the structure with related languages).

epups 4y ago

Everyone here is saying that people can simply transition easily into startups and other big companies. To a certain extent that's true, but what exactly are they going to do? As technology consolidates into one or two major LLM's, likely only accessible by API, I feel most orgs would be better served by relying heavily on finetuning or optimizing those for their purpose. Previous experience with NLP certainly helps with that, although this type of work would not necessarily be as exciting as trying to build the next big thing, which everyone was scrambling for before. OpenAI could build a state-of-the-art tool with a few hundred developers - to me, that means that money will converge to them and other big orgs rather than the opposite.

Yoric 4y ago
That's definitely a risk. With a PhD in the domain, I consider myself pretty good at (a subset of) distributed programming. But these days, when companies hire for distributed programming, they seem to want developers who know a specific set of tools and APIs. I'm more suited at reimplementing them for scratch.

twawaaay 4y ago

I think education goal for people shifted. I teach my kids to be flexible and embrace the change. Invest in abilities that transfer well to various things you could be doing during your life. Be a problem solver. In the future -- forget about cosy job you can be doing for the rest of your life. You no longer have any guarantees even if you own the business and even if you are farmer. What you absolutely don't want is spend X years at uni learning something, and then 5-10 years into your "career" finding out it was obsoleted overnight and you now don't have plan B.

rr888 4y ago
Liberal arts education will one day be back in fashion.
- twawaaay 4y ago
  Oh I do believe it. There will always be a market for snobs who will want to pay extra for handmade things vs AI-generated. The issue here is that it is all driven by fads and unstable. If you want to make money you will have to be flexible.
SketchySeaBeast 4y ago
> What you absolutely don't want is spend X years at uni learning something, and then 5-10 years into your "career" finding out it was obsoleted overnight and you now don't have plan B. That seems to be running directly opposite of the current trend of admin assistant jobs requiring 2 years specialized admin assistant diplomas. Tech (and I would guess the world of the business MBA) is a unique space where people are learning and changing so quickly, but for a lot of those outside the bubble things seem to be calcifying and requiring more and more training at the expensive of the worker.
yoyohello13 4y ago
Really the only safe career in the moderate future is going to be manual labor. There is always need to send a bunch of humans into the middle of nowhere to dig ditches.

twa34532 4y ago

oh no?! so finally the tech sector is experiencing themselves what they have done to other lines of professions for the past decades, namely eradicting them (rightfully) with innovation? well same advice applies then: * embrace, move on and retrain for another profession * learn empathy from the panic and hurt

TMWNN 4y ago

Is the entire field of data science (Itself maybe a decade old in terms of being a college major?) now obsolete, in terms of being a distinct job field? Are all data science majors now going to be "just" coming up with the proper prompts to get GPT to correctly massage datasets?

theGnuMe 4y ago
No. It's always been about posing the right question.

belter 4y ago

A whole thread on AI experts discussing how AI is making them obsolete...back to gardening...

izacus 4y ago
This fad too shall pass. And the tech will end up where it always does: helping some, changing some but nowhere near as much as the gold rush profiteers would make you believe.
- emptysongglass 4y ago
  This is not an event that calls for pithy adages. The fruits of ML are not a fad just like personal computing was not a fad. It's a watershed event that cuts across every knowledge worker's domain. If you're not currently using these LLMs it may not be obvious to you but those of us that have tried to apply them to our current fields see huge gains in productivity. Just in my own little slice of knowledge work, I've seen yield increases that have saved me multiple days of work on a single project. Everyone is going to feel this, most prominently people in the sorts of industries that frequent HN. If you haven't yet, you will or you will be forced to when you discover everyone in your field is out-producing you armed with these tools.
  - izacus 4y ago
    Uh-huh. How's them NFTs and Blockchain doing the watershed world changin these days?
    - deleted 4y ago
      [deleted]
    - eggsmediumrare 4y ago
      Your comment seems almost intentionally obtuse... How's them internet and smartphones doing the watershed world changing these days?

MonkeyMalarky 4y ago

I'm not at a big tech company, and we don't sell algorithms, but my team does use a lot of NLP stuff in internal algorithms. The only panic I have is trying to keep up and take the time to learn the new stuff. If anything, things like GPT-4 are going to make my team 10x more successful without having to hire an army of PhDs.

jarebear6expepj 4y ago
The PhD army will rise up against us one day... as soon as they are finish their TA appointments.
not-chatgpt 4y ago
What does your team do? It feels like GPT4 can handle any task out there. Only drawback is latency and cost.
- MonkeyMalarky 4y ago
  The price isn't even that bad, even the most expensive at 6cents per 1k tokens, it won't cost me much. It's the context size that's amazing. Gone are the days of only being able to pass ~500 tokens into something like BERT.

dserban 4y ago

The PR folks at my current company are in full panic mode on Linkedin, judging from the passive-aggressive tone of their posts (sometimes very nearly begging customers not to use ChatGPT and friends). They fully understand that LLMs are stealing lunch money from established information retrieval industry players selling overpriced search algorithms. For a long time, my company was deluded about being protected by insurmountable moats. I'm watching our PR folks going through the five stages of grief very loudly and very publicly on social media (particularly noticeable on Linkedin). Here's a new trend happening these days. Upon releasing new non-fiction books to the general public, authors are simultaneously offering an LLM-based chatbot box where you can ask the book any question. There is no good reason this should not work everywhere else, in exactly the same way. Take for example a large retailer who has a large internal knowledge base. Train an LLM on that corpus, ask the knowledge base any question. And retail is a key target market of my company. Needless to say I'm looking for employment elsewhere.

mashygpig 4y ago
> Here's a new trend happening these days. Upon releasing new non-fiction books to the general public, authors are simultaneously offering an LLM-based chatbot box where you can ask the book any question. Can you link to an example?
- dserban 4y ago
  I saw at least two examples of this here on HN. One of the books was about tech entrepreneurship 101, and I remember asking how to launch if you're a sole developer with no legal entity behind the product. I remember the answer being fairly coherent and useful. I don't have the URL handy, I suspect if you search HN for "entrepreneur book" you'll find it.
- org3 4y ago
  https://portal.konjer.xyz/ https://portal.konjer.xyz/
  - throwayyy479087 4y ago

dongobread 4y ago

I worked in a research capacity in the voice assistant org of a big tech company until very recently. There was a lot of panic when ChatGPT came out, as it became clear that the vast bulk of the org's modeling work and research essentially had no future. I feel bad for some of my colleagues who were really specialized in specific NLP technology niches (e.g. building NLU ontologies) which have been made totally obsolete by these generalized LLMs. Personally - I'm moving to more of a focus on analytical modeling. There is really nothing interesting about deep learning to me anymore. The reality is that any new useful DL models will be coming out of mega-teams in a few companies, where improving output through detailed understanding of modeling is less cost effective than simply increasing data quality and scale. Its all very boring to me.

mr_toad 4y ago
“ Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. “ http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- shawntan 4y ago
  I've seen many interpretations of this article and I'm curious as to the mainstream CS reading of it. One could look at the move from linear models to non-linear models or the use of ConvNets (yes I know ViTs exist, to my knowledge the base layers are still convolution layers) as 'leveraging human knowledge'. Only after those shifts were made did the leveraging of computation help. It would seem to me that the naive reading of that quote only rings true between breakthroughs.
- jnwatson 4y ago
  See also the GPT-4 technical report, page 37. https://images.app.goo.gl/vRP8368Z17zW2hvC9 https://images.app.goo.gl/vRP8368Z17zW2hvC9
  - 4y ago

macinjosh 4y ago

Is the compute for running an LLM cheap enough to scale at the moment? LLMs seem to be a great generalist solution but could specifically targeted NLP solutions still outperform in terms of speed/cost when you are processing high volumes of inputs?

wunderland 4y ago

Some big tech companies are witnessing a panic inside their entire org because they focus almost entirely on their competitors (except for the business divisions which are monopolies).

KrugerDunnings 4y ago
Some people look at ChatGPT and think its all over and other look at it and start imagining all the things they can use it for.

rdedev 4y ago

My university professor who specialises in NLP kinda feels like what's the point of research in the time of chatgpt. He says for now it's not possible to scale retrieval easily when using these llms so that's what he is looking into for now

hn_throwaway_99 4y ago

Wow - this is just wild. I've seen lots of arguments around "AI won't take everyone's job, it will just open up new areas for new jobs." Even if you take that with the benefit of the doubt (which I don't really think is warranted): 1. You don't need to take everyone's job. You just need to take a shitload of people's jobs. I think a lot of our current sociological problems, problems associated with wealth inequality, etc., are due to the fact that lots of people no longer have competitive enough skills because technology made them obsolete. 2. The state of AI progress makes it impossible for humans in many fields to keep up. Imagine if you spent your entire career working on NLP, and now find GPT-4 will run rings around whatever you've done. What do you do now? I mean, does anyone think that things like human translators, medical transcriptionists, court reporters, etc. will exist as jobs at all in 10-20 years? Maybe 1-2 years? It's fine to say "great, that can free up people for other thing", but given our current economic systems, how are these people supposed to eat? EDIT: I see a lot of responses along the lines of "Have you seen the bugs Google/Bing Translate has?" or "Imagine how frustrated you get with automated chat bots now!" Gang, the whole point is that GPT-4 blows these existing models out of the water. People who work in these fields are blown away by the huge advances in quality of output in just a short time. So I'm a bit baffled why folks are comparing the annoyances of ordering at a McDonald's automated kiosk to what state-of-the-art LLMs can do. And reminder that the first LLM was only created in 2018.

mostlysimilar 4y ago
This is possibly a death spiral. GPT is only possible because it's been trained on the work humans have learned to do and then put out in the world. Now GPT is as good as them and will put them all out of work. How can it improve if the people who fed it are now jobless?
- jgust 4y ago
  Presumably this problem is solved with technology improvements or the need is recognized to hire experts capable of generating high quality training material. In either situation, there's going to be extreme discomfort.

gniv 4y ago

I remember thinking about this when AlphaFold was announced. Did it happen back then? Were there large shifts in companies/universities that were doing folding research?

jhrmnn 4y ago
I’ve been thinking about this. My current theory is that molecular simulation is a much more heterogeneous activity than language modeling. Language is always the same kind of data. Molecular simulations span orders of magnitude in space and time and depending on that, data and even objectives have very different form. AlphaFold is just one small piece in this puzzle and it’s very easy for a research project to incorporate AlphaFold into an existing pipeline and shift its goal.

deleted 4y ago

[deleted]

jurassic 4y ago

Maybe this is alarmist, but I don't see how LLMs don't collapse our entire economic system over the next decade or so. This is coming for all of us, not just the NLP experts in big company research groups. Being able to cheaply/instantly perform virtually any task is great until you realize there is now nobody left to buy your product or service because the entire middle class has been put out of work by LLMs. And the service industries that depend on those middle class knowledge workers will be out of work because nobody can afford to purchase their services. I don't see how this doesn't end with guillotines coming out for the owner class and/or terrorism against the companies powering this revolution. I hope I'm wrong.

qwerty3344 4y ago
There are entire sectors of the economy that LLMs can't touch - hospitality, manufacturing, caregivers, religious sectors, live-action entertainment, etc. Sure some of these will be replaced by robots but there will always be new jobs too.
- jurassic 4y ago
  No, there are not. Everything in the economy is connected and you can't have a vibrant industry without customers. The customers of hospitality/entertainment/healthcare/etc businesses are largely the middle class who will be put out of work by LLMs. So the person who today makes $200/night in tips waiting tables at a nice restaurant.... who will be buying those meals?
  - woah 4y ago
    Someone who uses an LLM as a tool to perform a useful service
    - seydor 4y ago
      That would be another LLM or a robot then
    - bilbo0s 4y ago

tippytippytango 4y ago

Not even experts in the domain could see themselves being replaced and pivot in time. What hope does an ordinary person have in preparing for what’s coming? Telling people to retrain will not be an acceptable answer because no one can predict which skills will be safe from AI in 5 years.

bsder 4y ago

I guess I'm not panicked about my job in the face of AI because objective correctness is required. I dream about the day that OpenAI can write the 100 lines of code that connect the BLE stack, the ADC sensor and the power management code so that my IoT sensor doesn't crash once every 8 days. I see the AI stuff as very different from, say, the microcomputer revolution. People had LOTS of things they wanted to use computers for, but the computers were simply too expensive. As soon as microprocessors arrived, people had LOTS of things they were already waiting to apply them to. Factory automation was screaming for computers. Payroll management was screaming for computers. I don't see that with the current AI stuff. What thing was waiting for NLP/OpenAI to get good enough? Yes, things like computer games opened up whole new vistas, and maybe AI will do that, but that's a 20 year later thing. What stuff was screaming for AI right now? Maybe transcription? When I see the search bar on any of my favorite forums suddenly become useful, I'll believe that OpenAI stuff actually works. Finally, the real problem is that OpenAI needs to cough up what I want but then it needs to cough up the original references to what I want. I normally don't make other humans do that. If I'm asking someone for advice, I've already ascertained that I can trust them and I'm probably going to accept their answers. If it's random conversation and interesting or unusual, I'll mark it, but I'm not going to incorporate it until I verify. Although, given the current political environment, pehaps I should ask other humans to give me more references.

drewda 4y ago

I wonder if this will be a repeat of what happened with speech recognition. It used to be a specialized field dominated by smaller companies like Nuance. More recently Google, Microsoft, Apple, etc. decided they wanted to have speech recognition as an internal piece of their platforms. Google poached lots of Nuance's talent. And then Microsoft bought what remained of the company. Now speech recognition is a service integrated into the larger tech company's platforms, and also uses their more statistical/ML approaches, rather than being a component created by specialist companies/groups. (I'm sure I'm grossly simplifying this — just seeing a potential parallel.)

deepsquirrelnet 4y ago

I work at a small company, but it’s hard for me to imagine that generative AI will replace predictive AI/ML any time soon. Smaller models trained supervised/in-domain are simply more efficient and more accurate than unsupervised/out-of-domain. Plus we own and operate the technology much more cheaply. I don’t doubt that if your were trying to build a competing product to what OpenAI is doing that you’d feel affected, but there’s also a lot of other problems that are not being solved by generative models.

dr_dshiv 4y ago

Sunk cost! This is why so many AI scientists are so skeptical. They don’t use chatGPT because they spent years developing skills that enabled them to use LLMs and now any kid on the street can.

anshumankmr 4y ago

Here's my two cents, as I work with NLP in a tech company, mostly with Dialogflow and Rasa, in my current project, we are using Chat GPT (and previously GPT 3) to summarize articles, and I see that it can really handle FAQ questions really well. One of most common requirements was to train our bot to handle FAQ type question apart from complex conversation stories/flows,but this thing can straight up take the content from an article,summarize it neatly and send a response back. We have had some issues and complaints with the API,( mostly with GPT 3 as the fine tuning was only open for the base model and that had some trouble with some questions). Also there is a finicky response time, despite using having paid access. Response time varies from 10 seconds to even a minute (during some downtime that occured a few days ago, and a few days even before that there was a complete outage).

nathan_f77 4y ago

I would have never guessed that menial jobs would be the hardest ones to automate. I realized that humans have some incredible skills that were vital to our survival, and were honed over billions of years. But we take these skills for granted because most humans are born with these natural abilities, which are actually very impressive in the grand scheme of things. A janitor might not get paid very much relative to other jobs, but it took billions of years of evolution to hone their brain, muscles, balance, hand eye coordination, energy efficiency, information processing, etc. We're actually very impressive creatures!

orsenthil 4y ago

The book "Who Moved My Cheese?" tries to explain the human behavior for the moments like this. It is happening for many people now.

xvilka 4y ago

If someone build a software that would be able to attend a meeting instead of you - it will become as popular as Zoom. Combine LLM with voice and video recognition and synthesis, and voila - an immense chunk of wasted time in every corporation is gone.

WalterBright 4y ago

I see chatGPT as eliminating an awful lot of drudgery. For example, back in the 60's my dad was working on his book. The text was typed out double spaced, and he (and others) would make corrections. After a while, my mom would retype the whole thing. Imagine typing a whole book. Again and again and again. She'd type hour after hour. It's dehumanizing. And then came word processors. What a magical revolution! You could edit text instead of typing it all over again. I bet few people today realize what a great achievement that was. All chatgpt does is select the most likely next word out of a corpus of existing text. It is not creative. We don't need rooms full of typists anymore. Good riddance. I bet we get rid of a bunch of drudgery jobs with chatgpt.

KyeRussell 4y ago
It sounds like you’re giving a lot of human ‘occupations’ a whole lot of undeserved credit. There are stark differences between these two scenarios. Remixing existing ideas in an incredibly abstract way is not drudgery, it’s what a lot of people do, and get enjoyment out of.
- WalterBright 4y ago
  AI will automate the drudgery end, which is not "incredibly" anything.

sidcool 4y ago

We thought AI will take out the mechanical and manual jobs first that don't need intellectual capabilities, but only repetition. Who knew AI will first learn to write poems, novels and code.

PaulHoule 4y ago

I look at it the other way. Much of what people use GPT-4 can be done better (with a little more work) with transformer models that are specialized with the task. Things that were a struggle 5 years ago are about to be easy.

billythemaniam 4y ago
Definitely true for GPT-3, but after using GPT-4, I am not so sure. I am sure there are cases where a specialized NLP model can beat GPT-4, but I think they might be in the minority now. Whether I tried summarization, data extraction, or code generation, GPT-4 just kinda worked.

mdip 4y ago

Fascinating -- I think the comments on the HN post are almost as good. I think everyone mostly agrees that AI is coming for a lot of jobs. There's disagreement about how many, how it will impact society and the like. The pace of technology is not linear, it accelerates. I've never seen something that has so rapidly crossed into the "magical" territory as "nearly every single big LLM/Generative AI thing" seems to. It redefines what was previously laughably impossible ... a decade ago. We're riding a curve upward that is making it extremely hard to see what's coming next. All of the pontificating, all of the attempts at finding solutions to imagined problems ... I can't see one that doesn't feel like a blindfolded person aiming at what they were told was a dart board with what they were told was a dart. There's really nothing to do but hang on and hope you land where any new opportunities creep up. Expect bubbles, black swans, and purple unicorns.

chatmasta 4y ago

Maybe semantics, but... The implication of this post is that ChatGPT is causing the demise of NLP jobs, which, sure, that's true - but I don't think it's fair to classify this as replacing the job of anyone who works on NLP. This is not the same as GPT replacing a human translator, which is a direct replacement of their role - it's not like ChatGPT is going to start researching NLP. It's just a fairly mundane example of a superior technology outpacing an inferior one. The technology of NLP is being obsoleted, not the job of researching NLP (which is just unnecessary, not being replaced with something else). And in fact, NLP researchers are probably some of the most well positioned people to move laterally to working on transformers and LLMs.

thinkr42 4y ago

This mostly makes me wonder about the varying definitions of NLP. Most business folks I’ve worked with in this field don’t even know the question to ask or prompt- the implementation details of models like GPT are the easy part.

skc 4y ago

Somebody in that thread is speculating that the company in question is grammarly. I'm not sure but I'm now curious as to what the execs there are thinking, especially now with the recent Microsoft 365 news. Feels like the body blows keep coming.

specproc 4y ago

As someone who's a crap NLP practitioner, everything is just fine and dandy. I've never really had the gear or the skills to put together anything that improves over what I can pull from huggingface. What I do have, and virtually none of my (not remotely technical) colleagues have, is a clue what to do with all this stuff. They reckon it's about churning out poems and boilerplate text, the minute I figured it could give me whatever json I could reasonably ask for from a source doc, I was overjoyed. I see more things I can be doing now, not a risk of being replaced.