5 ms·

GPT-4 might be close to the best we'll get on the general LLM model front for a while since they trained on a huge chunk of web text. Next real advances will pr

by m_ke 4y ago

GPT-4 might be close to the best we'll get on the general LLM model front for a while since they trained on a huge chunk of web text. Next real advances will probably be in tuning them for specific applications in medicine, law, accounting, marketing, coding and etc. As someone running a one man company I can't wait for the cost of accounting, legal and copywriting to approach 0. Cost of shipping products will also go down 10-20x. As a fun experiment I asked ChatGPT to write me a terraform and k8s script to deploy a django app on GCP and it was able to do what would have taken me a few days in under a minute, including CICD. I then asked it to write code to compress a pytorch model and export it for iOS with coreml, and not only did it do 90% of that but also wrote the Swift code to load the model and do inference with it. EDIT: For example in medicine I recommend checking out this lecture that's actually live now: https://www.youtube.com/watch?v=gArDvIFCzh4 https://www.youtube.com/watch?v=gArDvIFCzh4

jakeinspace 4y ago
I’m not sure I’m looking forward to the politics that would come out of 10-20% of the previously middle class becoming instantly redundant and out of (middle-salary) work. That’s the fast path to fascism, unless we’re able to quickly implement UBI and other major societal overhauls.
- pineaux 4y ago
  This is isn't speculating, this is truthing.
  - paulryanrogers 4y ago
    Truthing as in theory quakery, not backed by evidence?
- JohnFen 4y ago
  > unless we’re able to quickly implement UBI and other major societal overhauls. Which are not going to happen.
  - jakeinspace 4y ago
    Something more approachable would be dropping payroll taxes to zero, or even making them negative for some positions, and significantly increasing corporate and capital gains.
    - mrguyorama 4y ago
      The problem isn't the specific policy, the problem is that right now the people who will be empowered and enriched the most by any theoretical "good at stuff" AI are the same people who already spend mountains of cash and effort stopping those things. How will a functional AI model do anything other than make them better at getting the outcomes they want? CEOs and the megarich have never had any problems watching people burn for their bank account.
- m_ke 4y ago
  Yeah I share these concerns as well (https://news.ycombinator.com/item?id=35305791 https://news.ycombinator.com/item?id=35305791). My hope is that some countries will see this as an opportunity to expand their safety nets and reduce the work burden on their citizens, which might convince citizens of countries that don't to demand similar policies.
- ngngngng 4y ago
  How would we keep it as low as 10-20%? 60% of employees in the US are knowledge workers. That number is similar for most wealthy nations.
layer8 4y ago
I wonder how it will be able to do that for the tech that will be current in 10 years, if mostly everyone will be using AI by then instead of asking on Stack Overflow.
- AlbertCory 4y ago
  Stack Overflow is headed for oblivion? What's the downside?
  - layer8 4y ago
    Lack of training data for the AI. Stack Overflow is just an e exemplary stand-in, you can replace it with any other resources you like.
    - AlbertCory 4y ago
      you missed the snark there. I hate SO.
boringg 4y ago
I think if history has bearing on things I don't see the cost of accounting, legal or copywriting ever approaching 0. If anything you will see those paywalled behind a company who will extract that from you. It's wishful thinking that somehow that goes to 0.
- m_ke 4y ago
  ChatGPT is already better at copywriting than 90% of startup founders and marketing people at big cos. You'll soon be able to let it generate 1000s of different versions of marketing material to A/B test or personalize based on user info. Soon you'll have multi modal transformers from dozens of companies and open source projects that will be able to parse and categorize all of your financial data and they'll have all of the incentives in the world to get it down to the cost of a quickbooks subscription.
tome 4y ago
> Cost of shipping products will also go down 10-20x. How can a large language model achieve that?
- m_ke 4y ago
  Ask chatgpt to implement some of the things you worked on the last few months. I was very skeptical too until I tried this. Here are some sample prompts that I tried and got full working code for: - "write pytorch code to train a transformer model on common crawl data and an inference service using fastapi" - "write react native code for a camera screen that can read barcodes and look them up using an API and then display info for matched results in a widget under the camera view" - "write react code for a wedding website" - "write code to deploy a django website on GCP using terraform and kubernetes" - "how do I dockerize the app, it uses pytorch and faiss, also push it to a container registry" - "implement a GPT style transformer model in pytorch", "write a training loop for it with distributed support and fp16" - "how would you implement reinforcement learning with human feedback (RLHF)", "can you implement it in pytorch" - "write code to compress a model trained in pytorch and export for inference on iOS" - "how would you distill a large vision model to a small one" - "what are the best CV architectures for mobile inference?" For all of these it gave me code that was 95% usable, all in under 15 minutes, and which would have taken me a week or two to do on my own.
  - tome 4y ago
    Oh sorry, I misunderstood "shipping products" to mean "physical shipping of physical products".
  - csmpltn 4y ago
    You know what's funny? I just asked ChatGPT to implement those exact same things and it shat all over itself producing embarrassing nonsense that won't compile, let alone do what they're expected to do. Bugs and incomplete code everywhere. You'd have a much better time just Googling those asks and re-using a working examples from SO or GitHub. Which is ironic, given how ChatGPT is supposedly trained on those exact things. I'm wondering how come we're both getting such vastly different results. Maybe your bar is just lower than mine? I don't know. I'm honestly shocked at the contrast between the PR given to ChatGPT, and the results on the ground. Try this simple ask (the results of which you'll find plastered everywhere): produce a Python function that decodes a Base64 string and prints the results, without using any "imports" or libraries. Every single output I got back was embarrassing garbage, and I gave it something like 15 shots.
yreg 4y ago
>since they trained on a huge chunk of web text What did they trained it on? Why is it unprobable to train on a better/bigger dataset any time soon?
- sarchertech 4y ago
  Because they trained it on a huge percentage of the existing web. There's isn't a (much) bigger and better data set available.
  - yreg 4y ago
    What percentage? What about books, newspapers, documents, etc.?
    - JW_00000 4y ago
      The LLaMA paper [1] (Meta's model) contains details about what they trained it on. This includes all of Wikipedia, a huge part of the internet (3.3 TB + 783 GB), a huge set of books (85 GB). My guess is basically all high-quality English articles on the web have been included. Also almost all English books must be included. Newspaper archives is about the only thing I see as missing, as well as more non-English sources. [1] https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971
- m_ke 4y ago
  OpenAI is working with Microsoft so they definitely had access to the full Bing index and data from their other platforms like Github and Linkedin. They also paid for private datasets, from what I heard they might have gotten a copy of Quora and I'm sure they got a dump of all digitized books from someone. Their best bet now is getting more supervised conversational data, which they should be getting a ton of from Bing and ChatGPT usage (they can use it as is with RLHF dataset which they had to pay people to generate by having fake conversations). I wouldn't be surprised if they partner with Microsoft and hire a large team of doctors to tune it to handle specific medical conditions like diabetes.
dumbaccount123 4y ago
Wrong https://www.digitaltrends.com/computing/gpt-5-artificial-general-intelligence/ https://www.digitaltrends.com/computing/gpt-5-artificial-gen...
- NameError 4y ago
  As far as I can tell, the only source in this article is a tweet from a developer with no connection to OpenAI

by m_ke 4y ago

jakeinspace 4y ago

I’m not sure I’m looking forward to the politics that would come out of 10-20% of the previously middle class becoming instantly redundant and out of (middle-salary) work. That’s the fast path to fascism, unless we’re able to quickly implement UBI and other major societal overhauls.

pineaux 4y ago
This is isn't speculating, this is truthing.
- paulryanrogers 4y ago
  Truthing as in theory quakery, not backed by evidence?
JohnFen 4y ago
> unless we’re able to quickly implement UBI and other major societal overhauls. Which are not going to happen.
- jakeinspace 4y ago
  Something more approachable would be dropping payroll taxes to zero, or even making them negative for some positions, and significantly increasing corporate and capital gains.
  - mrguyorama 4y ago
    The problem isn't the specific policy, the problem is that right now the people who will be empowered and enriched the most by any theoretical "good at stuff" AI are the same people who already spend mountains of cash and effort stopping those things. How will a functional AI model do anything other than make them better at getting the outcomes they want? CEOs and the megarich have never had any problems watching people burn for their bank account.
m_ke 4y ago
Yeah I share these concerns as well (https://news.ycombinator.com/item?id=35305791 https://news.ycombinator.com/item?id=35305791). My hope is that some countries will see this as an opportunity to expand their safety nets and reduce the work burden on their citizens, which might convince citizens of countries that don't to demand similar policies.
ngngngng 4y ago
How would we keep it as low as 10-20%? 60% of employees in the US are knowledge workers. That number is similar for most wealthy nations.

layer8 4y ago

I wonder how it will be able to do that for the tech that will be current in 10 years, if mostly everyone will be using AI by then instead of asking on Stack Overflow.

AlbertCory 4y ago
Stack Overflow is headed for oblivion? What's the downside?
- layer8 4y ago
  Lack of training data for the AI. Stack Overflow is just an e exemplary stand-in, you can replace it with any other resources you like.
  - AlbertCory 4y ago
    you missed the snark there. I hate SO.

boringg 4y ago

I think if history has bearing on things I don't see the cost of accounting, legal or copywriting ever approaching 0. If anything you will see those paywalled behind a company who will extract that from you. It's wishful thinking that somehow that goes to 0.

m_ke 4y ago
ChatGPT is already better at copywriting than 90% of startup founders and marketing people at big cos. You'll soon be able to let it generate 1000s of different versions of marketing material to A/B test or personalize based on user info. Soon you'll have multi modal transformers from dozens of companies and open source projects that will be able to parse and categorize all of your financial data and they'll have all of the incentives in the world to get it down to the cost of a quickbooks subscription.

tome 4y ago

> Cost of shipping products will also go down 10-20x. How can a large language model achieve that?

m_ke 4y ago
Ask chatgpt to implement some of the things you worked on the last few months. I was very skeptical too until I tried this. Here are some sample prompts that I tried and got full working code for: - "write pytorch code to train a transformer model on common crawl data and an inference service using fastapi" - "write react native code for a camera screen that can read barcodes and look them up using an API and then display info for matched results in a widget under the camera view" - "write react code for a wedding website" - "write code to deploy a django website on GCP using terraform and kubernetes" - "how do I dockerize the app, it uses pytorch and faiss, also push it to a container registry" - "implement a GPT style transformer model in pytorch", "write a training loop for it with distributed support and fp16" - "how would you implement reinforcement learning with human feedback (RLHF)", "can you implement it in pytorch" - "write code to compress a model trained in pytorch and export for inference on iOS" - "how would you distill a large vision model to a small one" - "what are the best CV architectures for mobile inference?" For all of these it gave me code that was 95% usable, all in under 15 minutes, and which would have taken me a week or two to do on my own.
- tome 4y ago
  Oh sorry, I misunderstood "shipping products" to mean "physical shipping of physical products".
- csmpltn 4y ago
  You know what's funny? I just asked ChatGPT to implement those exact same things and it shat all over itself producing embarrassing nonsense that won't compile, let alone do what they're expected to do. Bugs and incomplete code everywhere. You'd have a much better time just Googling those asks and re-using a working examples from SO or GitHub. Which is ironic, given how ChatGPT is supposedly trained on those exact things. I'm wondering how come we're both getting such vastly different results. Maybe your bar is just lower than mine? I don't know. I'm honestly shocked at the contrast between the PR given to ChatGPT, and the results on the ground. Try this simple ask (the results of which you'll find plastered everywhere): produce a Python function that decodes a Base64 string and prints the results, without using any "imports" or libraries. Every single output I got back was embarrassing garbage, and I gave it something like 15 shots.

yreg 4y ago

>since they trained on a huge chunk of web text What did they trained it on? Why is it unprobable to train on a better/bigger dataset any time soon?

sarchertech 4y ago
Because they trained it on a huge percentage of the existing web. There's isn't a (much) bigger and better data set available.
- yreg 4y ago
  What percentage? What about books, newspapers, documents, etc.?
  - JW_00000 4y ago
    The LLaMA paper [1] (Meta's model) contains details about what they trained it on. This includes all of Wikipedia, a huge part of the internet (3.3 TB + 783 GB), a huge set of books (85 GB). My guess is basically all high-quality English articles on the web have been included. Also almost all English books must be included. Newspaper archives is about the only thing I see as missing, as well as more non-English sources. [1] https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971
m_ke 4y ago
OpenAI is working with Microsoft so they definitely had access to the full Bing index and data from their other platforms like Github and Linkedin. They also paid for private datasets, from what I heard they might have gotten a copy of Quora and I'm sure they got a dump of all digitized books from someone. Their best bet now is getting more supervised conversational data, which they should be getting a ton of from Bing and ChatGPT usage (they can use it as is with RLHF dataset which they had to pay people to generate by having fake conversations). I wouldn't be surprised if they partner with Microsoft and hire a large team of doctors to tune it to handle specific medical conditions like diabetes.

dumbaccount123 4y ago

Wrong https://www.digitaltrends.com/computing/gpt-5-artificial-general-intelligence/ https://www.digitaltrends.com/computing/gpt-5-artificial-gen...

NameError 4y ago
As far as I can tell, the only source in this article is a tweet from a developer with no connection to OpenAI