15 ms·
I work on a research team in FAANG. What it really feels like is one company made everyone else obsolete. And we are going to work working on NLP models that un
by fdgsdfogijq 3y ago
I work on a research team in FAANG. What it really feels like is one company made everyone else obsolete. And we are going to work working on NLP models that underperform ChatGPT by a huge margin. Twiddling my thumbs and keeping quiet while no one wants to recognize the elephant in the room.
Also, there is no "working in AI", a few thousand people are doing real AI at most. The rest of us are calling an API.
- lacker 3y agoThis reminds me of back in the mid 2000's, there were a lot of smart people working on search algorithms at different companies. But eventually, you'd talk to someone smart working on Yahoo Search, and they would just be kind of beaten down by the frustration of working on a competing search engine when Google was considered to be by far the best. It got harder for them to recruit, and eventually they just gave up. So... I don't know where you're working. But don't twiddle your thumbs for too long! It's no fun to be in the last half of people to leave the sinking ship.
- deleted 3y ago[deleted]
- anon7725 3y agoCan confirm. People are scrambling to remain relevant.
- TechnicolorByte 3y agoHow does that manifest specifically?
- ProllyInfamous 3y agoGo read the Goldman Sachs report from last week, which predicts 300m jobs disappearing and 60% of jobs "affected beyond minimally."
- BulgarianIdiot 3y agoCalling an API doesn't mean no value is captured. There are vastly complex integrations of LLM as a small component in larger systems, with their own programming, memory, task models and so on. If you think GPT is just about chat, you've misunderstood LLMs.
- UncleOxidant 3y agoWhy do you think OpenAPI is so far out in front? It's not like there's a lot of secret sauce here - most of this stuff (transformers,etc.) is all out there in papers. And places like Google & Meta must have a lot more computing resources to train on that OpenAI does thus they should be able to train faster. Do you think OpenAI has discovered something they haven't been open about?
- nicpottier 3y agoUntil we see otherwise, don't we have to assume there's some secret sauce? Bard doesn't match GPT4 and it isn't for a lack of trying. (though perhaps that will change, so far that's the case)
- xnx 3y agoGoogle is going slow. They might be behind, but we haven't seen their best effort yet. Google has a 540 billion parameter model.
- nicpottier 3y agoSo they say, and I agree this seems likely. But until they release it that's all talk, and they have plenty of incentive to release it.
- nr2x 3y agoAnd plenty of reason not to: search ads.
- nr2x 3y agoGoogle also has Sundar and Ruth who’d rather focus on how to get another ad on the SERP than kill the golden goose. They’re not going slow, they just don’t have the leadership for the moment.
- saynay 3y agoBard would not trick anyone into thinking it was sentient, yet something they have supposedly did. I just think Google has far more to lose than Bing, so they are being more cautious.
- xnx 3y agoNot sure why LLM would make Facebook (ads), Apple (hardware), Amazon (hosting, retail), Netflix (tv) obsolete. It's definitely something Google needs to think about, but there's no reason to think they won't again be the leader soon.
- atonse 3y agoI actually think Apple is in a unique position here again with the hardware/software integration. Once again, their ability to do computation on device and optimize silicon to do it, is unparalleled. A huge Achilles heel of current models like GPT-4 is that they can’t be run locally. And there are tons of use cases where we don’t necessarily want to share what we’re doing with OpenAI. That’s why if Apple wasn’t so behind on the actual models (Siri is still a joke a decade later), they’d be in great shape hardware-wise.
- xnx 3y agoGoogle has some impressive on device AI software such as Google Translate (translation), Google Photos (object detection, removal, inpainting), and Recorder (multi-speaker speech to text). Most of this is possible without their Tensor chip, but is more efficient with it.
- letitgo12345 3y agoImagine Walmart launching a ChatGPT interfaced bot for shopping that customers take a liking to. Walmart starts acquiring both new customers as well as high quality data they can use for RLHF for shopping. Eventually Walmart's data moat becomes so big, that Amazon retail cannot catch up and customers start leaving Amazon. For AWS, if MS starts giving discounts for OAI model usage to regular Azure customers, that's gonna be a strong incentive to switch For Apple, A Windows integrated with GPT tech may become a tough beast to beat.
- twelve40 3y ago> Imagine Walmart launching a ChatGPT interfaced bot for shopping that customers take a liking to. I can't imagine that, because it doesn't seem to fit the use case. Especially not to the point of bankruptcy of Amazon, maybe as a small novelty? Can you list some killer features that the chat would bring that would make the existing shopping experience irrelevant? Maybe not everything is a nail to the hammer?
- tayo42 3y agoSounds like the same thing that happened with datacenters? No one has ops or hardware sysadmins, no one sets up large networks except a few in those centralized cloud companies and couple other niche uses. website ops job changed
- 2-718-281-828 3y agocome on, it's not that bad, at least you're doing linear regression
- VirusNewbie 3y agoI work at a FAANG and our unreleased models are fantastic. Now, there might be panic about how to productize it all, but tech wise i'm pretty surprised how good they are.
- Robotbeat 3y agoNot releasing the models may be the same as the models never existing in the end.
- philjohn 3y agoIt depends - a great model tuned to your business domain could be insanely valuable as a competetive advantage.
- letitgo12345 3y agoHope so. Don't want a single corporate entity (OAI/MS) dominating the entire economy. This sector desperately needs competition
- carabiner 3y agoNot just NLP, even 3D art: https://www.youtube.com/watch?v=SzGEfYh9ITQ https://www.youtube.com/watch?v=SzGEfYh9ITQ Top comment: I love seeing my job get transformed from 3D artist into prompt writer into jobless in a year or less, yay!
- Washuu 3y agoI do a lot of stylized 3D art: I still have time before AI figures that out!~
- zone411 3y agoI'm quite surprised at how little progress FAANG companies have made in recent years, as I believe much of what's happening now with ChatGPT was predictable. Here's a slide from a deck for a product I was developing in 2017: https://twitter.com/LechMazur/status/1644093407357202434/photo/1 https://twitter.com/LechMazur/status/1644093407357202434/pho.... Its main function was to serve as a writing assistant.
- Analog24 3y agoScaling up an LM from 2017 would not achieve what GPT-4 does. It's nowhere near that simple. Of course companies saw the potential of natural language interfaces, there has been billions spent on it over the years and a lot of progress was made prior to ChatGPT coming along.
- zone411 3y agoYou're making incorrect assumptions. This project wasn't about scaling any published approaches. It was original neural net research that produced excellent results with a new architecture without self-attention, using a new optimizer, new regularization and augmentation ideas, sparsity, but with some NLP feature engineering, etc. Scaling it up to GPT-2 size matched its performance for English (my project was English-only and it was bidirectional unlike GPT so not a perfect comparison), and very likely scaling it up to GPT-3 size would have matched it as well, since GPT-3 wasn't much of an improvement over GPT-2 besides scale. Unclear for GPT-4 since there is very little known about it. Of course, in the meantime, most of these ideas are no longer SOTA and there has been a ton of progress in GPU hardware and frameworks like PyTorch/TF. You can check out my melodies project from a year ago as a current example. There is nothing matching it yet: https://www.youtube.com/playlist?list=PLoCzMRqh5SkFPG0-RIAR8jYRaICWubUdx https://www.youtube.com/playlist?list=PLoCzMRqh5SkFPG0-RIAR8.... And that's just my personal project. What you're saying about companies recognizing the commercial potential is clearly wrong. It's six years later and Siri, Alexa, and Google Home are still nearly as dumb as they were back then. Microsoft is only now working on adding a writing assistant to Word, and that's thanks to OpenAI. Why do you think Google had to have "code red" if they saw the potential? Low-budget startups are also very slow - they should've had their products out when the GPT-3 API was published, not now. One thing I didn't expect is how well this same approach would work for code. I haven't even tried to do it.
- qqtt 3y agoChatGPT is cool and novel, but FAANG's requirements for ML/AI go far beyond what ChatGPT provides as a product. ChatGPT is good at answering questions based on an older data set. FAANG typically requires up to date real time inference for huge rapidly changing data sets. Working on the practical side of ML/AI at FAANG, you will probably be working with some combination of feature stores, training platforms, inference engines, and so on - all attempting to optimize inference and models for specific use cases - largely ranking - which ads to show which customers based on feature store attributes, which shows to show which customers - all these ranking problems exist orthogonal to ChatGPT, which is using relatively stale datasets to answer knowledge based questions. The scaling problems for AI/ML for productionizing these ranking models from training to inference is a huge scaling problem. ChatGPT hasn't really come close to solving it in a general way (and also solves a different class of problems).
- fdgsdfogijq 3y agoI can tell you that we have applied teams working on open problems, which can be solved out of the box with ChatGPT. Its a huge deal
- yanderekko 3y agoAgreed. For my job maintaining real-time models with high business value to be disrupted by a chatbot, an LLM would have to be able to plug into our entire data ecosystem and yield insights in realtime. The backend engineering work required to facilitate this will be immense, and if the answer to that is "an LLM will create a new backend data architecture required to support the front-end prompt systems", then... well, suffice to say I can't see that happening overnight. It will require several major iterative and unpredictable pivots to re-envisage what exactly engineers are doing at the company. For the time being, I expect LLMs to start creeping their tendrils into various workflows where the underlying engineering work is light but the rate of this will be limited by the slow adaptability of the humans that are not yet completely disposable. The "low hanging fruit" is obvious, but EVPs who are asking "why can't we just replace our whole web experience with a chatbot interface?" may end up causing weird overcorrections among their subordinates.
- blazespin 3y agoFolks need to start getting over themselves. It's pretty trivial to get GPT4 to explain how transformers work, where the bottlenecks are in both performance and learning, and start modifying pytorch. It's really not that complicated. Gatekeeping is so over.
- rlt 3y ago> Also, there is no "working in AI", a few thousand people are doing real AI at most. The rest of us are calling an API. I would call that “applied AI” and there’s no shame in figuring out novel ways to apply a new technology.