4 ms·

Claude is a mystery/surprise to me. My mental model has been to train these cutting edge closed source models you need 1) Bespoke supercomputer (no public clou

Claude is a mystery/surprise to me. My mental model has been to train these cutting edge closed source models you need 1) Bespoke supercomputer (no public cloud will cut it) 2) Great dataset (which takes a long time to collect unless you have a partnership with with a search engine) 3) Couple hundred lines of pytorch code to run on the supercomputer 4) A couple of employees with experience in the dark arts of malfunctioning GPU's and exploding gradients Anthropic is a relatively new startup that probably has 3) & 4) from their history at OpenAI. But I don't see how they could have 1) & 2).

mareko 3y ago
For 2) it looks like they partnered with duckduckgo.
- Kiro 3y ago
  DDG has no search index (they are using Bing) so I wonder what that actually constitutes.
  - mkl 3y ago
    DDG does have their own index, but also use Bing and many other sources. See the CEO's numerous comments to that effect: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Ayegg%20index%20&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu..., https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Aepi0Bauqu%20index%20&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
    - Kiro 3y ago
      Only for things such as widgets, not regular search results.
espadrine 3y ago
For 1) a public cloud partnership is typically enough. OpenAI didn’t build a bespoke supercomputer, but trained on Azure (with a preferential contract thanks to their investors): https://openai.com/gpt-4 https://openai.com/gpt-4 > GPT-4 was trained on Microsoft Azure AI supercomputers Cohere trained on GCP: https://techcrunch.com/2021/11/17/google-cloud-teams-up-with-nlp-startup-cohere-on-multi-year-partnership/ https://techcrunch.com/2021/11/17/google-cloud-teams-up-with... > Aidan Gomez, co-founder and CEO at Cohere[:] “We scrape the data to train these big models, we train them on massive TPU pods” Stability AI trains on AWS: https://aws.amazon.com/blogs/machine-learning/stability-ai-builds-foundation-models-on-amazon-sagemaker/ https://aws.amazon.com/blogs/machine-learning/stability-ai-b... > With Amazon SageMaker, Stability AI will build AI models on compute clusters with thousands of GPU or AWS Trainium chips Given Anthropic’s recent partnership announcement, they likely train on GCP nowadays: https://www.anthropic.com/index/anthropic-partners-with-google-cloud https://www.anthropic.com/index/anthropic-partners-with-goog... > Anthropic will leverage Google Cloud's cutting-edge GPU and TPU clusters to train, scale, and deploy its AI systems

mareko 3y ago

For 2) it looks like they partnered with duckduckgo.

Kiro 3y ago
DDG has no search index (they are using Bing) so I wonder what that actually constitutes.
- mkl 3y ago
  DDG does have their own index, but also use Bing and many other sources. See the CEO's numerous comments to that effect: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=by%3Ayegg%20index%20&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu..., https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Aepi0Bauqu%20index%20&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
  - Kiro 3y ago
    Only for things such as widgets, not regular search results.

espadrine 3y ago

For 1) a public cloud partnership is typically enough. OpenAI didn’t build a bespoke supercomputer, but trained on Azure (with a preferential contract thanks to their investors): https://openai.com/gpt-4 https://openai.com/gpt-4 > GPT-4 was trained on Microsoft Azure AI supercomputers Cohere trained on GCP: https://techcrunch.com/2021/11/17/google-cloud-teams-up-with-nlp-startup-cohere-on-multi-year-partnership/ https://techcrunch.com/2021/11/17/google-cloud-teams-up-with... > Aidan Gomez, co-founder and CEO at Cohere[:] “We scrape the data to train these big models, we train them on massive TPU pods” Stability AI trains on AWS: https://aws.amazon.com/blogs/machine-learning/stability-ai-builds-foundation-models-on-amazon-sagemaker/ https://aws.amazon.com/blogs/machine-learning/stability-ai-b... > With Amazon SageMaker, Stability AI will build AI models on compute clusters with thousands of GPU or AWS Trainium chips Given Anthropic’s recent partnership announcement, they likely train on GCP nowadays: https://www.anthropic.com/index/anthropic-partners-with-google-cloud https://www.anthropic.com/index/anthropic-partners-with-goog... > Anthropic will leverage Google Cloud's cutting-edge GPU and TPU clusters to train, scale, and deploy its AI systems