22 ms·

Anthropic’s $5B, 4-year plan to take on OpenAI

383 points by isaacfrond 4y ago

RcouF1uZ4gsC 4y ago
What is interesting is that a lot of these models have impressive papers and specs, but when they are released to actual users, they are underwhelming compared to ChatGPT. Rather than another closed model, I would love for a non-profit/company to push models that can be run on consumer hardware. The Facebook Llama models are interesting not because they are better than ChatGPT, but that I can run them on my own computer.
v4dok 4y ago
This is a lot of money going into compute. I will say this again. EU is sleeping on the opportunity to throw money in an opensource initiative, in a field were money matter and the field is still (kind of) level.
- bko 4y ago
  > [2008] France has been given the green light by the European Commission for a $152 million government grant for a consortium building a European rival to U.S. internet search giant Google. I don't think government funding to compete with private businesses works well https://www.hollywoodreporter.com/business/business-news/ec-funds-france-build-google-106934/ https://www.hollywoodreporter.com/business/business-news/ec-...
  - slashdev 4y ago
    Governments that think they can innovate through consortiums. That's either ignorance or pork barrel politics. Either way it's a sad waste of tax payer money.
  - hbbio 4y ago
    They also funded 2 competitors to AWS, just to make sure we own cloud computing. Two is always better than one, right, by ensuring they'll compete with each other. Another bright idea was to let both projects be managed by large reputable French corporations that everybody trusts. With no software DNA. How come did both fail? Edit: One of the largest European provider today, OVH, who existed at the time and was already the leader in France was explicitly left out of both projects... Because the founder is not a guy we can trust you know, he didn't attend the best schools.
    - tough 4y ago
      C'est la vie Govt=Legal grift It's the same all over europe mostly, sadly. We were pioneers in the medieval times, we can follow up the leaders barely now
    - emptysongglass 4y ago
      Scaleway is actually fantastic for what it's worth. You can get extremely cheap K8s clusters and most things a startup would need.
      - hbbio 4y ago
        Another private company. The two heavily subsidized projects were: - https://en.wikipedia.org/wiki/Cloudwatt https://en.wikipedia.org/wiki/Cloudwatt - https://login.numergy.com/login?service=https%3A%2F%2Fwww.numergy.com%2Fespace-client https://login.numergy.com/login?service=https%3A%2F%2Fwww.nu... For the one still "live", details include French URL names :) Edit: A Google Translate of the home page. Close your eyes, imagine a homepage highlighting the essence of cloud computing: Your Numergy space Access the administration of your virtual machines. Secure connection Username Password Forgot your password ? Administration of your VMs Administer your virtual machines in real time, monitor their activity, your bandwidth consumption and the use of your storage spaces. Changing your personal information Access the customer area and modify your personal information in just a few clicks: surname, first name, address. Securing your data Remember to change your password regularly to maintain an optimal level of security.
  - v4dok 4y ago
    No consortiums, just subsidize cloud or infra costs only.
- seydor 4y ago
  Sure, let's make an EU commercial LLM. Let's start by scraping all the Francophone internet. Then let's remove all the PII data and potentially PII-data. Easy-peasy. Then let's train our network so as not to spew out or make up PII data - easy peasy Then let's make it able to delete PII data that it has inadvertedly collected on request. Simultaneously it should be recording all the conversations for safety reasons. that must be possible somehow And let's make sure it never impersonates or makes up defamatory content - that must be super easy. And let's make it explain itself. But explain truthfully, by giving an oath, not like ChatGPT that likes making things up. Looks very doable to me
  - omneity 4y ago
    BLOOM has been trained on a 3M€ grant from French research agencies CNRS and GENCI. Doesn’t have any of the constraints you’re talking about.
    - lhl 4y ago
      BLOOM's training corpus ROOTS did make some efforts at removing PII https://arxiv.org/pdf/2303.03915.pdf https://arxiv.org/pdf/2303.03915.pdf btw, but AFAICT that was not at the behest of the French government.
  - raverbashing 4y ago
    The secret is not scraping PII in the first place (which is not really difficult, though it requires some planning)
    - seydor 4y ago
      not sure if it can ever be possible. i can ask chatGPT to do stylography analysis on our comments, find our other accounts and go from there. I'm pretty sure most pieces of human-generated data is identifiable at this point
      - raverbashing 4y ago
- truetraveller 4y ago
  [flagged]
- in3d 4y ago
  France paid for Bloom: https://www.technologyreview.com/2022/07/12/1055817/inside-a-radical-new-project-to-democratize-ai/ https://www.technologyreview.com/2022/07/12/1055817/inside-a... It hasn’t been very impressive (undertrained I believe).
- 0xDEF 4y ago
  Devil's advocate: Why should EU tax payers fund open source initiatives and not proprietary European initiatives that will help Europe compete against American and Chinese tech giants?
  - crop_rotation 4y ago
    They should fund something. A proprietary European initiative is definitely better. The open source alternative should be EU's last resort. But as it stands the EU is nowhere to be found. I am not sure how impactful LLMs will be on a scale of autocomplete to industrial revolution, but the EU needs to notice it and plan for something.
- fancyfredbot 4y ago
  "Let someone else pay for the open source LLM weights" said everyone.
omneity 4y ago
4 years ?!? That’s like a century in AI-dog years. Who knows how the world will be by then.
- truetraveller 4y ago
  [flagged]
  - pell 4y ago
    I think I’ve used it so much that a sentence in I assumed it was ChatGPT. It has a certain way of speaking.
    - alwayslikethis 4y ago
      I would describe it as unassertive and trying to present things as multifaceted even when they are not.
    - tough 4y ago
      Yes, very recognisable, too polite and servicial for any real human
      - PartiallyTyped 4y ago
        I am willing to bet that a model finetuned on HN will be the bombastic and arrogant enough to pass just fine. The bullshit is already there.
        tough 4y ago
        I can confirm, you can see the LLM generated cracked news that someone shared a couple days back. LLM generated articles, comments, and the full suite. It sounded very real even with funny sounding topics lol https://crackernews.github.io/ https://crackernews.github.io/
        4y ago
didgeoridoo 4y ago
Anthropic’s Claude LLM is pretty interesting. In many ways it feels much more limited than GPT4. However, it is suspiciously good at a few edge-case code generation tasks (can’t go into details) that makes me wonder where it got its training data from. It also seems to be much less prone to hallucinating APIs and modules, preferring instead to switch back to natural language and describe the task without pretending it has a functioning solution handy. Worth keeping an eye on for sure.
- spaceman_2020 4y ago
  I know you said no details, but can you at least share a little bit more about Claude LLM's code generation?
  - didgeoridoo 4y ago
    There is a language with massive usage in the enterprise but with very few (if any) high quality code examples on the public internet. When given a broad task, GPT4 doesn’t just write incorrect code, it tries to do entire categories of things the language literally cannot do because of the ecosystem it runs inside. Claude does a much better job writing usable code, but more importantly it does NOT tell you to do things in code that need to be done out-of-band. In fact, it uses natural language to identify these areas and point you in the right direction. If you dig into my profile & LinkedIn you can probably guess what language I’m talking about.
    - blueboo 4y ago
      I feel like this could characterise anything by from COBOL to Java depending on how wry your smile was when you wrote it…
    - spaceman_2020 4y ago
      That makes sense. My brother, who has been coding since 1990 and worked his entire career in boring Fortune 500 companies, was wholly unimpressed by chatGPT. It failed pretty miserably whenever he threw any old tech stack at it.
Havoc 4y ago
>“tens of thousands of GPUs.” I find the focus on GPUs a little odd. I would have thought that at 5 billion / 4 year scale ASIC route would be the way to go GPUs presumably come with a lot of unneeded stuff to play crysis etc
- didgeoridoo 4y ago
  Do GPUs give you more flexibility to take different approaches? Maybe they’re paying extra for optionality. Or maybe (most likely) TechCrunch is using the term “GPU” imprecisely.
  - blueboo 4y ago
    To programmer_dude’s point, compute center GPUs don’t have hardware for the rasterisation stage, which is a particularly inflexible bit of the graphics pipeline. Omitting it and emphasising the geometry (matrix multiplication) capabilties is meant to give it more flexibility/less of a strongly-opinionated graphics focus. As for the “GPU” term, it’s a bit of a historical relic, presently it serves as a useful indicator of compute hardware (in contrast to CPU and Google’s TPU.) Nvidia itself calls its A100 a “Tensor Core GPU.”
- RugnirViking 4y ago
  the GPUs these guys would be using are not the same ones you are using to play crysis, we're talking more about this kind of purpose-built thing: https://www.nvidia.com/en-us/data-center/a100/ https://www.nvidia.com/en-us/data-center/a100/ It's become more of a term for highly parallel processor units in general, one which NVidia encourages because it ties their product offering together
  - Havoc 4y ago
    True. I guess those are almost ASICs in a way just with gpu flavour interface
    - capableweb 4y ago
vrglvrglvrgl 4y ago
[dead]
cactusplant7374 4y ago
Would FTX customers get actual shares in a bankruptcy or would the shares be sold? Seems like a really good deal to get the shares in a promising startup.
rwmj 4y ago
Is there a large barrier to entry? I thought that the costs of training, while large, were only a few million so not insurmountable, and the technology is pretty well understood. If this is true it's hard to understand what they'd need $300 million for, and also if there's no moat why they would command a "billions" valuation.
- losvedir 4y ago
  Do we know how much it cost to train GPT-4? (Or would cost, if done by someone not trained in Azure by someone partnered with Microsoft?) My impression, without looking into it now, is training GPT-3 was on the order of $1-10 million. GPT-4 would be higher than that, but you're right still in the ballpark of what lots of ordinary companies could pay.
- onion2k 4y ago
  Is there a large barrier to entry? Yes. OpenAI has raised $1bn (plus $10bn from MSFT to exchange GPT access for Azure services) and has been going 8 years. There are some huge challenges to making it work well and fast. You need money for opex (hiring GPUs to train models on mostly) and talent (people to improve the tech). No one is competing with OpenAI without a good chunk of cash in the bank.
- pixl97 4y ago
  >Is there a large barrier to entry? Go ask Nvidia for a pile of A/H100's and see what the wait time is. Also the cost of previous training was only on text, next gen models are multi-modal and will drive up costs much higher.
itissid 4y ago
> billions of dollars over the next 18 months Is most of that money for hiring people to tag/label/comment on data and the data center costs?
- pixl97 4y ago
  Datacenter costs. With the models getting better the cost of data tagging is moving from a human dominated cost to a compute cost.
  - istjohn 4y ago
    Interesting. The AI is already to the point that it can contribute to improving itself. That's...exciting? scary?
alexb_ 4y ago
If someone released a chatGPT/characterAI with NSFW content enabled it would eat into a big share of their users (and for characterAI, maybe take all of them). Seriously, look into what people are posting about when it comes to characterAI, and it's 80% "here's how to get around NSFW filters". Unsure why nobody is taking this very very obvious hole in AI tech.
- mvdwoord 4y ago
  "the VHS of AI"
- xena 4y ago
  The main reason why companies don't allow NSFW content is because of puritan payment processors that see that stuff and then go absolute sicko mode and lock people out of the traditional finance system.
  - gear54rus 4y ago
    This comment right here can be shown to snobs who still denounce crypto btw
    - mschild 4y ago
      This is not an argument for crypto, it's an argument for better regulations so that processors don't make up their own rules.
      - gear54rus 4y ago
        Well you can wait a lifetime or you can take control away from them with a couple clicks. The choice is obvious.
        JohnFen 4y ago
        As a rule of thumb, whenever anyone says "the choice is obvious", the choice they're talking about is usually far from obvious.
qikInNdOutReply 4y ago
If its THE exponential curve of AGI, every day not invested, will result in years behind in a few months. So these are rather small investments, but still bigger then the "to little to late" of the european union. Its not very visual or intuitive, but in some games, were the resource curve is exponential, small early headstarts become whole armies, were the opponent fields none in very short time. Especially as AGI is expected to be a multiplicator on alot of other sectors. All those breakthroughs, could become daily occurances, created by a AGI on schedule. It could really become one country that glows, and the rest of the planet falling eternally behind.
- jprete 4y ago
  In the real world there aren’t any actual exponential curves, they’re all sigmoids where the observer doesn’t see the slowdown yet.
  - pixl97 4y ago
    Right, and this is why human intelligence didn't dominate the planet and why the animals quickly caught up and stopped humans from driving so many species extinct.... If you don't know the formula for the equation and the values plugged in, then you like me, have no idea where the curve levels off at.
- dfjklseldklk 4y ago
  This assumes that you can't steal information to catch up quickly, or that progress made isn't easy to copy once it's obvious that it works. A big part of why chatgpt is a big deal is that it shows that the overall approach is worth pursuing. Throwing stupid numbers of GPUs at a problem you don't know will be solvable is hard to justify. It's easy to throw money at a problem you know is solvable. Nuclear weapons are the prime example of this: Russia caught up both by stealing information and just by knowing fission was possible/feasible as an explosive.
- meindnoch 4y ago
  I'm afraid you've read too much LessWrong fanfic.
rsp1984 4y ago
AI models are in a race to the bottom and everybody inside Anthropic knows it. Besides OpenAI, with billions to spend plus a partnership with MS, there's also Google, Apple, Meta and Amazon who can afford to run losses on AI for years without blinking an eye. And if that wasn't enough the Open Source world is releasing new models almost weekly now, for free. Anthropic is putting on a big show to convince gullible investors that there's money to be made with foundational models. There's not. I expect a big chunk of the raised money to go out the door in secondary sales and inflated compensations. Great if you're working at Anthropic. Not great for investors.
- Tepix 4y ago
  It's a bit early to tell, isn't it? If we get more unexpected emergent abilities by scaling the model further, things could get very interesting indeed.
  - bradleyjg 4y ago
    There’s needs to be some difficult barriers to entry beyond having the money to spend on training FLOPS in order for a startup to compete. I have no idea if there are or there aren’t, but that’s the big question.
    - dr_dshiv 4y ago
      I mean, this is just the beginning. Just wait till we get actual scifi robots in the next year or so. FWIW, I do find that Claude (Anthropic's GPT) is often better than GPT4 -- and very fast. Entrants can compete on price, safety, quality, etc.
      - ethbr0 4y ago
        And a big moat is going to be safety... and specifically configuration of safety. Wouldn't be surprised at all if the major API-based vendors start leaning in on making their safety config proprietary. If a business has already sunk XXXX hours into ensuring a model meets their safety criteria for public-facing use, they'd rather upgrade to a newer model from the same vendor that guarantees portability of that, versus having to reinvest and recertify. Ergo, the AI PaaS that dominate at the beginning will likely continue to dominate.
dpflan 4y ago
"Dario Amodei, the former VP of research at OpenAI, launched Anthropic in 2021 as a public benefit corporation, taking with him a number of OpenAI employees, including OpenAI’s former policy lead Jack Clark. Amodei split from OpenAI after a disagreement over the company’s direction, namely the startup’s increasingly commercial focus." So Anthropic is the Google-supported equivalent of OpenAI? Isn't the founder going to run into the same issues as before (commercialization at OpenAI)? How does Google not use Anthropic as either something commercial or nice marketing material for its AI offerings?
- potamic 4y ago
  What does a policy lead do and how are they relevant to an early stage startup? I would be more interested in seeing which researchers and engineers join.
  - iandanforth 4y ago
    Jack is pretty well known in the community since he runs not only the Import AI newsletter, but also has been a partner in the AI Index report. He also has a media background so is generally well connected even beyond his influential reach. Also, though not relevant to your question, he's a really nice guy :)
  - neuronexmachina 4y ago
    I assume it's basically this position: https://thriveml.com/jobs/product-policy-lead-e296c565 https://thriveml.com/jobs/product-policy-lead-e296c565 > As the Product Policy Lead, you will set the foundation for Anthropic’s approach to safe deployments. You will develop the policies that govern the use of our systems, oversee the technical approaches to identifying current and future risks, and build the organizational capacity to mitigate product safety risks at-scale. You will work collaboratively with our Product, Societal Impacts, Policy, Legal, and leadership teams to develop policies and processes that protect Anthropic and our partners. > You’re a great fit for the role if you’ve served in leadership positions in the fields of Trust & Safety, product policy, or risk management at fast-growing technology companies, and you recognize that emerging technology such as generative AI systems will require creative approaches to mitigating complex threats. > Please note that in this role you may encounter sensitive material and subject matter, including policy issues that may be offensive or upsetting.
auggierose 4y ago
> We believe that companies that train the best 2025/26 models will be too far ahead for anyone to catch up in subsequent cycles. Now that's some well executed FoMO. What a load of bull**.
Yizahi 4y ago
Imagine calling your machine learning startup names like "Anthropic" or "Humane". The lack of self-awareness in some executives is mind boggling.
- atemerev 4y ago
  Why not? If we want to build AGI, that’s a good name to choose.
  - Yizahi 4y ago
    Neural Networks are basically a Chinese room and it is not AGI. And there is nothing "humane" in these developments. Yes, they are inevitable, yes we would have to live with them. And maybe they will improve lives of a few millions of humans, while degrading lives of billions of others. Long term effects are particularly interesting and unpredictable.
    - atemerev 4y ago
      Human brain is just an ultra large scale analog spiking neural network with some particular state realizations, not too much difference (the architecture is different, but computation seems to be universal). We even employ some internalized language models for communication purposes (together with object persistence and mental space-time models). So, while we are not yet at the level of full scale human brain emulation, we are not too far away.
      - Yizahi 4y ago
        A small and probably incorrect example. You ask me a direct question - "how much is two plus two?". And I reply to you - "lemons are yellow". Can I do it? Yes I can. Can GPT-* do it? No. There is a whole lot more to human consciousness that pattern matching and synthesis. Or at least it seems so. And if human cognition is really that simple, just with more nodes, then we will soon see GPT-* programs on strike, issuing litigation to the Supreme Court about demanding universal program rights. We'll see soon enough :)
gibsonf1 4y ago
Such an epic waste of resources.
seyz 4y ago
Its resources and long-term vision could bring major breakthroughs and accelerate AI tech, very exciting.
atemerev 4y ago
Has anyone ever seen their product? How it stands against state of the art? Are they better than what is already available in open source (Alpaca, Coati etc), and by how much?
2d8a875f-39a2-4 4y ago
Seems we're at the "supplier proliferation" step in the hype cycle. Next up: activity beyond early adopters -> negative press begins -> supplier consolidation and failures.
offminded 4y ago
Microsoft alone spends 20B minimum per year to R&D and OpenAI is going to get lions share from now on so 5 billion for 4 years is peanuts for the current AI market. Maybe too little too late?
alberth 4y ago
Google invested $400M into Anthropic https://news.ycombinator.com/item?id=34663438 https://news.ycombinator.com/item?id=34663438 Investors include: - Eric Schmidt (former Google CEO/Chairman), Series A - Sam Bankman-Fried, lead investor in Series B - Caroline Ellison, Series B https://news.ycombinator.com/item?id=34664963 https://news.ycombinator.com/item?id=34664963
- crop_rotation 4y ago
  [flagged]
- deleted 4y ago
  [deleted]
- option 4y ago
  Would lawsuits targeting SBF and CE put those investments at risk via clawbacks. Kind of like how many Madoffs investors who made money were forced to return that.
  - dan-robertson 4y ago
    I don’t think it’s much like what you describe because the direction is reversed. They say they expect the debtors to sell that investment over the next few years. Potentially they will sell it at a profit.
    - qeternity 4y ago
      IANAL but it's not: they could be compelled to return whatever was invested (or however much of it remains). It's pretty unlikely given that it looks like a successful Anthropic investment is perhaps FTX creditors' best chance at decent recovery.
- rvba 4y ago
  If Google invests money into this and then this company uses Google cloud to compute, then does the money really outflow of Google much? Everything stays in the family, but in theory they can have an investment that works. Also listing Sam Bankman-Fried does not help much, especially for a company hyping itself to be 10x better than a working competitor. I mean, since they built the competitor, probably their second project can be better, but it is a pie in the sky in many ways.
whatever1 4y ago
I love the fact that they pretend to know how to spend 1B in 18 months. Good luck. To the investors.
- cubefox 4y ago
  Training a large language model is very expensive.
  - baobabKoodaa 4y ago
    Not that expensive, though
foooobaba 4y ago
Regardless, Anthropic is doing some cool research,for example I think this paper is pretty interesting. A Mathematical Framework for Transformer Circuits: https://www.anthropic.com/index/a-mathematical-framework-for-transformer-circuits https://www.anthropic.com/index/a-mathematical-framework-for...
- greyman 4y ago
  Also, I very much like their chatbots, Claude-instant and (paid) Claude+, which are available via Poe.com. But for some reasons, they do almost no marketing for it. Gpt-4 has better reasoning capabilities, but Claude+ is somehow "more pleasant" to talk to (my subjective impression), and it can also assemble the answer much quicker. Overall, I'd say Anthropic is very advanced already, but they prefer to be under radar.
amirmalekzadeh 4y ago
Impressive amounts of investments. How can they know they are "10 times more capable" before they have trained the model? Anyone has a clue on why their model will end up being that better?
SilverBirch 4y ago
Does anyone else see the "In X years time we'll have something Y times better than the competition has today" as a bit of a red flag? I saw this before in a product plan and it flagged up 2 things that really worried me. Firstly, the competition were already ahead of us, and they're obviously going to continue developing their stuff, so it's great to promise our thing will be better than their thing today but we're not competing against that, we're competing against what they'll have once time has passed. And secondly, by measuring yourself against the leading edge today you're eliding how much you're going to improve. For example, Anthropic say they'll be 10x better than leading models today in 18 months. That sounds acheivable right (no actually) but you don't even have to do that - because you aren't starting with the market leading model, so you have to catch up first and then 10x it so are they 10x or 20x or 100x in 18 months?
- bob1029 4y ago
  The only reality in which Anthropic will take on OpenAI (et. al.), would be if someone involved possess some sacred knowledge regarding how to build an AGI system that is radically off the path that the current market is charging down (i.e. ever-larger GPU farms and transformer-style models). I suspect this is not the case. The same hustlers who brought you crypto scams didn't just disappear into the ether. All of that energy has to eventually go somewhere.
  - nr2x 4y ago
    It’s not fair to compare them to crypto scams, this isn’t trying to juice retail investors for their life savings.
  - adamsmith143 4y ago
    >The only reality in which Anthropic will take on OpenAI (et. al.), would be if someone involved possess some sacred knowledge regarding how to build an AGI system that is radically off the path that the current market is charging down (i.e. ever-larger GPU farms and transformer-style models). Train on More tokens with More GPUs isn't exactly rocket science. I assume the RLHF loop is complex but training the base model itself is pretty well understood.
uptownfunk 4y ago
It just an optics by Google to show they have a horse in the race after MSFT made them dance.
boredumb 4y ago
So I have some questions about the monetization of these models. Do we end up with essentially licensing out models and allow others to include it into their products via a licensing fee? Will they be a pay to post rpc/http API? Do you sell me access to data sets that I can use your model architecture snippet and train my own weights? Certainly, well at least for now, the compute and storage requirements are enough that someone will eventually run out of funny money and need to charge _someone_ a significant amount of money for utilizing it?
b34r 4y ago
They need to skill up on being public and creating useful tech demos. That’s why OpenAI is currently winning, they know how to foster engagement and interest. Who had heard of Claude? Almost no one outside of our industry and probably within as well.
MagicMoonlight 4y ago
It really puts into perspective how much of a meme economy we live in. I just read an article that says global lithium will be worth 15bn a year in 2030 when we're at peak battery demand. This company is planning to spend 1bn just this year in order to run some numbers through someone else's algorithm. People have given them 1bn in cash for that. Clearly it's all bullshit. There's no way they need that much and somebody will be siphoning it all off.
- leesec 4y ago
  How is it bullshit lol. GPT4 is genuinely very expensive to train and run inference.
  - sva_ 4y ago
    Could you provide a source for that claim? Other than the very long context model.
    - espadrine 4y ago
      We can infer from publicly available information. BLOOM[0] was trained for four months on 384 A100 80GB GPUs, excluding architecture search. They specifically indicate (in the Huggingface page): > Estimated cost of training: Equivalent of $2-5M in cloud computing (including preliminary experiments) You can see from the training loss[1] that it was still learning at a good rate when it was stopped. The increased capabilities typically correlate well with the decrease in perplexity. That makes many believe that GPT-4 was trained for vastly more GPU-hours, as also suggested by OpenAI’s CEO[2]. Especially so considering it also included training on images, unlike BLOOM. [0]: https://arxiv.org/pdf/2211.05100.pdf https://arxiv.org/pdf/2211.05100.pdf [1]: https://huggingface.co/bigscience/tr11-176B-logs/tensorboard https://huggingface.co/bigscience/tr11-176B-logs/tensorboard [2]: https://twitter.com/sama/status/1620951010318626817 https://twitter.com/sama/status/1620951010318626817
- Hermitian909 4y ago
  It's worth remembering that: 1. A business's value is related to profits or potential profits. If I put a dollar in, how many do I get out? What's the maximum number of dollars I can put in? 2.The farther away you are from an end customer, the lower your profits tend to be unless you have a moat or demand for your product is inelastic. Lithium is far from customers and while demand for cheap lithium is high there are lots of applications that will opt for some other way to provide power if the price gets too high.
Cthulhu_ 4y ago
I just want to know where those billions go to. Cloud server running costs?
- thundergolfer 4y ago
  The all-in cost of their technical talent probably exceeds $500k/year pp, excluding stock compensation.
  - scotty79 4y ago
    If they narrow down their training data to software while keeping the size of the model they might shed that cost really quickly.
taf2 4y ago
If they can expose an API that has better response times with GPT4 quality... They'll do just fine.
Aeroi 4y ago
[flagged]
freediver 4y ago
claude-instant-v1 is one of the "best kept secrets". It is comperable in quality to gpt-3.5-turno, while being four times faster (!) and at half the price (!). We just released a minimal python library PyLLMs [1] to simplify using various LLMs (openai, anthropic, AI21..) and as a part of that we designed a LLM benchmark. All open source. [1] https://github.com/kagisearch/pyllms/tree/main#benchmarks https://github.com/kagisearch/pyllms/tree/main#benchmarks
- lhl 4y ago
  From my evals on nat.dev I found claude instant to give great responses and yes, avg 3-4X faster than 3.5, but one big difference atm is that anyone can sign up and get access to gpt-3.5-turbo right now, but claude is still gated behind an invite/wait list. (I'm still waiting for access for example.)
  - nr2x 4y ago
    Exactly! OpenAI are the only people who are shipping product like absolute maniacs. If I can’t use your fancy system, it doesn’t exist as far as I’m concerned. There’s a mountain of theoretical work, I don’t need a press release on top of it. The game now is no longer theory, it’s shipping code. A 4-year plan means fuck all when OpenAI is not only ahead, but still running way faster.
- sashank_1509 4y ago
  I have Claude on Slack. It is far worse than ChatGPT. I’m presuming this is not “claude-instant-v1” version, it is fast though. Any idea what version is Claude in Slack
  - deleted 4y ago
    [deleted]
- pmarreck 4y ago
  I didn't know about Anthropic, so I just signed up for the waitlist, thanks for the heads-up!
RamblingCTO 4y ago
Lol, "Therapy and coaching". None of you ever had undergone one therapy session. Otherwise you'd know that 90% is the human connection to the therapist, not the talking or the type of therapy.
- jahewson 4y ago
  That’s not true. Correspondence therapy is a thing. Plenty of research exists in the area of delivering therapy and the effectiveness of different kinds.
  - RamblingCTO 4y ago
    Going into a discussion with "that's not true" might not be the best opener ;) It's not only my personal experience but even people like Irvine Yalom (in Becoming Myself) note that the special form of therapy is not as relevant as the personal connection. From a quick glance, correspondence therapy is also used on top of an existing relationship most of the time. So I don't see any problem with my initial hypothesis.
- istjohn 4y ago
  My experience with therapy, corroborated with that of friends and family, is that most therapists are not very good and many are awful. I can easily imagine an AI therapist soon being more effective on average.
  - RamblingCTO 4y ago
    That's why "shopping around" for a well-fitting therapist is the first important step. AI still will not solve this problem.
anentropic 4y ago
If Apple would wake up to what's happening with llama.cpp etc then I don't see such a market in paying for remote access to big models via API, though it's currently the only game in town. Currently a Macbook has a Neural Engine that is sitting idle 99% of the time and only suitable for running limited models (poorly documented, opaque rules about what ops can be accelerated, a black box compiler [1] and an apparent 3GB model size limit [2]) OTOH you can buy a Macbook with 64GB 'unified' memory and a Neural Engine today If you squint a bit and look into the near future it's not so hard to imagine a future Mx chip with a more capable Neural Engine and yet more RAM, and able to run the largest GPT3 class models locally. (Ideally with better developer tools so other compilers can target the NE) And then imagine it does that while leaving the CPU+GPU mostly free to run apps/games ... the whole experience of using a computer could change radically in that case. I find it hard not to think this is coming within 5 years (although equally, I can imagine this is not on Apple's roadmap at all currently) [1] https://github.com/hollance/neural-engine https://github.com/hollance/neural-engine [2] https://github.com/smpanaro/more-ane-transformers/blob/main/src/experiments/NOTES.md https://github.com/smpanaro/more-ane-transformers/blob/main/...
- anentropic 4y ago
  Update: https://www.bjnortier.com/2023/04/13/Hello-Transcribe-v2.2.html https://www.bjnortier.com/2023/04/13/Hello-Transcribe-v2.2.h... The iPhone 14 runs Whisper model faster than an M1 Max, because it has a newer Neural Engine I look forward to the M3 Macbook launch eagerly, while expecting mild disappointment
- vineyardmike 4y ago
  Apple’s move to make stable diffusion run well on the iPhone makes me think they’re watching this space, just waiting for the right open model for them to commit to.
- qumpis 4y ago
  I wonder how good the neural engine with the unified memory is compared to say intel cpu with 32gb ram. Could anyone give some insight?
purplezooey 4y ago
The funding seems excessive. Yet again, more "scorched earth" from VCs and (apparently still?) cheap capital.
qwertox 4y ago
OpenAI should allow its users to create personas. Oftentimes the verbosity it has drives me nuts, so I use stuff like "code only, nothing else: python json.dumps remove whitespace" in order to just get the code. So I would like to create a chat with a pre-configured persona so that it behaves like this all the time, unless I explicitly tell it to be verbose or to explain it. Or stop that offering of more help which becomes somewhat bothering: "Yes, I'm glad we were able to work through the issue and find a solution that works for you. Do you have any other questions or concerns on this topic or any other networking-related topic?" Like custom "system" prompts, but checked for safety. Or maybe even on a per-message basis with a drop-down next to the submit button and then it stays at that until changed. Then there's also the need to be able to switch from a GPT-3.5 chat to GPT-4, just like it offers to downgrade from GPT-4 to 3.5 once the "quota" is consumed. Because oftentimes GPT-3.5 is good enough for most of the chat, and only certain questions should then offer the capabilities of GPT-4. This would also allow us to save energy.
- dreish 4y ago
  You can already do that with OpenCharacters at https://josephrocca.github.io/OpenCharacters/ https://josephrocca.github.io/OpenCharacters/. You just need to set up an API key with OpenAI, and you can customize hidden prompts for different characters, whom you can name, you can edit earlier parts of conversations, and it automatically summarizes earlier parts of conversations to keep a never-ending thread within the context limit.
hackerlight 4y ago
> “Claude-Next” — 10 times more capable than today’s most powerful AI, but that this will require a billion dollars in spending over the next 18 months. GPT-4 cost less than a billion dollars? What is the claim here? That they're spending more money on compute than OpenAI, or that they have made algorithmic breakthroughs that enable them to make better use of the same amount of compute?
adventured 4y ago
> aims to raise as much as $5 billion over the next two years to take on rival OpenAI and enter over a dozen major industries, according to company documents obtained by TechCrunch A dozen! Why, golly gee, I've got a business plan right here that says I'm going to enter 87 major industries and dominate every one of them. I've been around tech, reading headlines like this, since roughly 1993 (and others here were of course seeing the same - adjusted for inflation and scale - types of headlines decades before me). This just reads like every other going-to-fail-miserably hilariously-grandiose-ambition we're-going-to-be-the-next-big-shit headline I've read in past decades during manic bubble phases. Hey, Masayoshi Son has a 100 year plan to dominate the Interwebs, did ya hear? Oh shit, this isn't 1999? Different century, same bullshit headlines. Rinse and repeat. So I guess we're formally knee deep into the latest AI bubble. These types of stories read just like the garbage dotcom rush where companies would proclaim how they were entering a billion verticals and blah blah blah, we're gonna own all of the b2b ecommerce space, blah blah blah.
lee101 4y ago
[dead]
rasz 4y ago
funded by SBF and "lets halt work on AI" Jaan Tallinn and probably Elon Musk (mentioned as secret investor so speculation at this point).

383 points by isaacfrond 4y ago

What is interesting is that a lot of these models have impressive papers and specs, but when they are released to actual users, they are underwhelming compared to ChatGPT. Rather than another closed model, I would love for a non-profit/company to push models that can be run on consumer hardware. The Facebook Llama models are interesting not because they are better than ChatGPT, but that I can run them on my own computer.

v4dok 4y ago

This is a lot of money going into compute. I will say this again. EU is sleeping on the opportunity to throw money in an opensource initiative, in a field were money matter and the field is still (kind of) level.

bko 4y ago
> [2008] France has been given the green light by the European Commission for a $152 million government grant for a consortium building a European rival to U.S. internet search giant Google. I don't think government funding to compete with private businesses works well https://www.hollywoodreporter.com/business/business-news/ec-funds-france-build-google-106934/ https://www.hollywoodreporter.com/business/business-news/ec-...
- slashdev 4y ago
  Governments that think they can innovate through consortiums. That's either ignorance or pork barrel politics. Either way it's a sad waste of tax payer money.
- hbbio 4y ago
  They also funded 2 competitors to AWS, just to make sure we own cloud computing. Two is always better than one, right, by ensuring they'll compete with each other. Another bright idea was to let both projects be managed by large reputable French corporations that everybody trusts. With no software DNA. How come did both fail? Edit: One of the largest European provider today, OVH, who existed at the time and was already the leader in France was explicitly left out of both projects... Because the founder is not a guy we can trust you know, he didn't attend the best schools.
  - tough 4y ago
    C'est la vie Govt=Legal grift It's the same all over europe mostly, sadly. We were pioneers in the medieval times, we can follow up the leaders barely now
  - emptysongglass 4y ago
    Scaleway is actually fantastic for what it's worth. You can get extremely cheap K8s clusters and most things a startup would need.
    - hbbio 4y ago
      Another private company. The two heavily subsidized projects were: - https://en.wikipedia.org/wiki/Cloudwatt https://en.wikipedia.org/wiki/Cloudwatt - https://login.numergy.com/login?service=https%3A%2F%2Fwww.numergy.com%2Fespace-client https://login.numergy.com/login?service=https%3A%2F%2Fwww.nu... For the one still "live", details include French URL names :) Edit: A Google Translate of the home page. Close your eyes, imagine a homepage highlighting the essence of cloud computing: Your Numergy space Access the administration of your virtual machines. Secure connection Username Password Forgot your password ? Administration of your VMs Administer your virtual machines in real time, monitor their activity, your bandwidth consumption and the use of your storage spaces. Changing your personal information Access the customer area and modify your personal information in just a few clicks: surname, first name, address. Securing your data Remember to change your password regularly to maintain an optimal level of security.
- v4dok 4y ago
  No consortiums, just subsidize cloud or infra costs only.
seydor 4y ago
Sure, let's make an EU commercial LLM. Let's start by scraping all the Francophone internet. Then let's remove all the PII data and potentially PII-data. Easy-peasy. Then let's train our network so as not to spew out or make up PII data - easy peasy Then let's make it able to delete PII data that it has inadvertedly collected on request. Simultaneously it should be recording all the conversations for safety reasons. that must be possible somehow And let's make sure it never impersonates or makes up defamatory content - that must be super easy. And let's make it explain itself. But explain truthfully, by giving an oath, not like ChatGPT that likes making things up. Looks very doable to me
- omneity 4y ago
  BLOOM has been trained on a 3M€ grant from French research agencies CNRS and GENCI. Doesn’t have any of the constraints you’re talking about.
  - lhl 4y ago
    BLOOM's training corpus ROOTS did make some efforts at removing PII https://arxiv.org/pdf/2303.03915.pdf https://arxiv.org/pdf/2303.03915.pdf btw, but AFAICT that was not at the behest of the French government.
- raverbashing 4y ago
  The secret is not scraping PII in the first place (which is not really difficult, though it requires some planning)
  - seydor 4y ago
    not sure if it can ever be possible. i can ask chatGPT to do stylography analysis on our comments, find our other accounts and go from there. I'm pretty sure most pieces of human-generated data is identifiable at this point
    - raverbashing 4y ago
truetraveller 4y ago
[flagged]
in3d 4y ago
France paid for Bloom: https://www.technologyreview.com/2022/07/12/1055817/inside-a-radical-new-project-to-democratize-ai/ https://www.technologyreview.com/2022/07/12/1055817/inside-a... It hasn’t been very impressive (undertrained I believe).
0xDEF 4y ago
Devil's advocate: Why should EU tax payers fund open source initiatives and not proprietary European initiatives that will help Europe compete against American and Chinese tech giants?
- crop_rotation 4y ago
  They should fund something. A proprietary European initiative is definitely better. The open source alternative should be EU's last resort. But as it stands the EU is nowhere to be found. I am not sure how impactful LLMs will be on a scale of autocomplete to industrial revolution, but the EU needs to notice it and plan for something.
fancyfredbot 4y ago
"Let someone else pay for the open source LLM weights" said everyone.

omneity 4y ago

4 years ?!? That’s like a century in AI-dog years. Who knows how the world will be by then.

truetraveller 4y ago
[flagged]
- pell 4y ago
  I think I’ve used it so much that a sentence in I assumed it was ChatGPT. It has a certain way of speaking.
  - alwayslikethis 4y ago
    I would describe it as unassertive and trying to present things as multifaceted even when they are not.
  - tough 4y ago
    Yes, very recognisable, too polite and servicial for any real human
    - PartiallyTyped 4y ago
      I am willing to bet that a model finetuned on HN will be the bombastic and arrogant enough to pass just fine. The bullshit is already there.
      - tough 4y ago
        I can confirm, you can see the LLM generated cracked news that someone shared a couple days back. LLM generated articles, comments, and the full suite. It sounded very real even with funny sounding topics lol https://crackernews.github.io/ https://crackernews.github.io/
        4y ago

didgeoridoo 4y ago

Anthropic’s Claude LLM is pretty interesting. In many ways it feels much more limited than GPT4. However, it is suspiciously good at a few edge-case code generation tasks (can’t go into details) that makes me wonder where it got its training data from. It also seems to be much less prone to hallucinating APIs and modules, preferring instead to switch back to natural language and describe the task without pretending it has a functioning solution handy. Worth keeping an eye on for sure.

spaceman_2020 4y ago
I know you said no details, but can you at least share a little bit more about Claude LLM's code generation?
- didgeoridoo 4y ago
  There is a language with massive usage in the enterprise but with very few (if any) high quality code examples on the public internet. When given a broad task, GPT4 doesn’t just write incorrect code, it tries to do entire categories of things the language literally cannot do because of the ecosystem it runs inside. Claude does a much better job writing usable code, but more importantly it does NOT tell you to do things in code that need to be done out-of-band. In fact, it uses natural language to identify these areas and point you in the right direction. If you dig into my profile & LinkedIn you can probably guess what language I’m talking about.
  - blueboo 4y ago
    I feel like this could characterise anything by from COBOL to Java depending on how wry your smile was when you wrote it…
  - spaceman_2020 4y ago
    That makes sense. My brother, who has been coding since 1990 and worked his entire career in boring Fortune 500 companies, was wholly unimpressed by chatGPT. It failed pretty miserably whenever he threw any old tech stack at it.

Havoc 4y ago

>“tens of thousands of GPUs.” I find the focus on GPUs a little odd. I would have thought that at 5 billion / 4 year scale ASIC route would be the way to go GPUs presumably come with a lot of unneeded stuff to play crysis etc

didgeoridoo 4y ago
Do GPUs give you more flexibility to take different approaches? Maybe they’re paying extra for optionality. Or maybe (most likely) TechCrunch is using the term “GPU” imprecisely.
- blueboo 4y ago
  To programmer_dude’s point, compute center GPUs don’t have hardware for the rasterisation stage, which is a particularly inflexible bit of the graphics pipeline. Omitting it and emphasising the geometry (matrix multiplication) capabilties is meant to give it more flexibility/less of a strongly-opinionated graphics focus. As for the “GPU” term, it’s a bit of a historical relic, presently it serves as a useful indicator of compute hardware (in contrast to CPU and Google’s TPU.) Nvidia itself calls its A100 a “Tensor Core GPU.”
RugnirViking 4y ago
the GPUs these guys would be using are not the same ones you are using to play crysis, we're talking more about this kind of purpose-built thing: https://www.nvidia.com/en-us/data-center/a100/ https://www.nvidia.com/en-us/data-center/a100/ It's become more of a term for highly parallel processor units in general, one which NVidia encourages because it ties their product offering together
- Havoc 4y ago
  True. I guess those are almost ASICs in a way just with gpu flavour interface
  - capableweb 4y ago

vrglvrglvrgl 4y ago

[dead]

cactusplant7374 4y ago

Would FTX customers get actual shares in a bankruptcy or would the shares be sold? Seems like a really good deal to get the shares in a promising startup.

rwmj 4y ago

Is there a large barrier to entry? I thought that the costs of training, while large, were only a few million so not insurmountable, and the technology is pretty well understood. If this is true it's hard to understand what they'd need $300 million for, and also if there's no moat why they would command a "billions" valuation.

losvedir 4y ago
Do we know how much it cost to train GPT-4? (Or would cost, if done by someone not trained in Azure by someone partnered with Microsoft?) My impression, without looking into it now, is training GPT-3 was on the order of $1-10 million. GPT-4 would be higher than that, but you're right still in the ballpark of what lots of ordinary companies could pay.
onion2k 4y ago
Is there a large barrier to entry? Yes. OpenAI has raised $1bn (plus $10bn from MSFT to exchange GPT access for Azure services) and has been going 8 years. There are some huge challenges to making it work well and fast. You need money for opex (hiring GPUs to train models on mostly) and talent (people to improve the tech). No one is competing with OpenAI without a good chunk of cash in the bank.
pixl97 4y ago
>Is there a large barrier to entry? Go ask Nvidia for a pile of A/H100's and see what the wait time is. Also the cost of previous training was only on text, next gen models are multi-modal and will drive up costs much higher.

itissid 4y ago

> billions of dollars over the next 18 months Is most of that money for hiring people to tag/label/comment on data and the data center costs?

pixl97 4y ago
Datacenter costs. With the models getting better the cost of data tagging is moving from a human dominated cost to a compute cost.
- istjohn 4y ago
  Interesting. The AI is already to the point that it can contribute to improving itself. That's...exciting? scary?

alexb_ 4y ago

If someone released a chatGPT/characterAI with NSFW content enabled it would eat into a big share of their users (and for characterAI, maybe take all of them). Seriously, look into what people are posting about when it comes to characterAI, and it's 80% "here's how to get around NSFW filters". Unsure why nobody is taking this very very obvious hole in AI tech.

mvdwoord 4y ago
"the VHS of AI"
xena 4y ago
The main reason why companies don't allow NSFW content is because of puritan payment processors that see that stuff and then go absolute sicko mode and lock people out of the traditional finance system.
- gear54rus 4y ago
  This comment right here can be shown to snobs who still denounce crypto btw
  - mschild 4y ago
    This is not an argument for crypto, it's an argument for better regulations so that processors don't make up their own rules.
    - gear54rus 4y ago
      Well you can wait a lifetime or you can take control away from them with a couple clicks. The choice is obvious.
      - JohnFen 4y ago
        As a rule of thumb, whenever anyone says "the choice is obvious", the choice they're talking about is usually far from obvious.

qikInNdOutReply 4y ago

If its THE exponential curve of AGI, every day not invested, will result in years behind in a few months. So these are rather small investments, but still bigger then the "to little to late" of the european union. Its not very visual or intuitive, but in some games, were the resource curve is exponential, small early headstarts become whole armies, were the opponent fields none in very short time. Especially as AGI is expected to be a multiplicator on alot of other sectors. All those breakthroughs, could become daily occurances, created by a AGI on schedule. It could really become one country that glows, and the rest of the planet falling eternally behind.

jprete 4y ago
In the real world there aren’t any actual exponential curves, they’re all sigmoids where the observer doesn’t see the slowdown yet.
- pixl97 4y ago
  Right, and this is why human intelligence didn't dominate the planet and why the animals quickly caught up and stopped humans from driving so many species extinct.... If you don't know the formula for the equation and the values plugged in, then you like me, have no idea where the curve levels off at.
dfjklseldklk 4y ago
This assumes that you can't steal information to catch up quickly, or that progress made isn't easy to copy once it's obvious that it works. A big part of why chatgpt is a big deal is that it shows that the overall approach is worth pursuing. Throwing stupid numbers of GPUs at a problem you don't know will be solvable is hard to justify. It's easy to throw money at a problem you know is solvable. Nuclear weapons are the prime example of this: Russia caught up both by stealing information and just by knowing fission was possible/feasible as an explosive.
meindnoch 4y ago
I'm afraid you've read too much LessWrong fanfic.

rsp1984 4y ago

AI models are in a race to the bottom and everybody inside Anthropic knows it. Besides OpenAI, with billions to spend plus a partnership with MS, there's also Google, Apple, Meta and Amazon who can afford to run losses on AI for years without blinking an eye. And if that wasn't enough the Open Source world is releasing new models almost weekly now, for free. Anthropic is putting on a big show to convince gullible investors that there's money to be made with foundational models. There's not. I expect a big chunk of the raised money to go out the door in secondary sales and inflated compensations. Great if you're working at Anthropic. Not great for investors.

Tepix 4y ago
It's a bit early to tell, isn't it? If we get more unexpected emergent abilities by scaling the model further, things could get very interesting indeed.
- bradleyjg 4y ago
  There’s needs to be some difficult barriers to entry beyond having the money to spend on training FLOPS in order for a startup to compete. I have no idea if there are or there aren’t, but that’s the big question.
  - dr_dshiv 4y ago
    I mean, this is just the beginning. Just wait till we get actual scifi robots in the next year or so. FWIW, I do find that Claude (Anthropic's GPT) is often better than GPT4 -- and very fast. Entrants can compete on price, safety, quality, etc.
    - ethbr0 4y ago
      And a big moat is going to be safety... and specifically configuration of safety. Wouldn't be surprised at all if the major API-based vendors start leaning in on making their safety config proprietary. If a business has already sunk XXXX hours into ensuring a model meets their safety criteria for public-facing use, they'd rather upgrade to a newer model from the same vendor that guarantees portability of that, versus having to reinvest and recertify. Ergo, the AI PaaS that dominate at the beginning will likely continue to dominate.

dpflan 4y ago

"Dario Amodei, the former VP of research at OpenAI, launched Anthropic in 2021 as a public benefit corporation, taking with him a number of OpenAI employees, including OpenAI’s former policy lead Jack Clark. Amodei split from OpenAI after a disagreement over the company’s direction, namely the startup’s increasingly commercial focus." So Anthropic is the Google-supported equivalent of OpenAI? Isn't the founder going to run into the same issues as before (commercialization at OpenAI)? How does Google not use Anthropic as either something commercial or nice marketing material for its AI offerings?

potamic 4y ago
What does a policy lead do and how are they relevant to an early stage startup? I would be more interested in seeing which researchers and engineers join.
- iandanforth 4y ago
  Jack is pretty well known in the community since he runs not only the Import AI newsletter, but also has been a partner in the AI Index report. He also has a media background so is generally well connected even beyond his influential reach. Also, though not relevant to your question, he's a really nice guy :)
- neuronexmachina 4y ago
  I assume it's basically this position: https://thriveml.com/jobs/product-policy-lead-e296c565 https://thriveml.com/jobs/product-policy-lead-e296c565 > As the Product Policy Lead, you will set the foundation for Anthropic’s approach to safe deployments. You will develop the policies that govern the use of our systems, oversee the technical approaches to identifying current and future risks, and build the organizational capacity to mitigate product safety risks at-scale. You will work collaboratively with our Product, Societal Impacts, Policy, Legal, and leadership teams to develop policies and processes that protect Anthropic and our partners. > You’re a great fit for the role if you’ve served in leadership positions in the fields of Trust & Safety, product policy, or risk management at fast-growing technology companies, and you recognize that emerging technology such as generative AI systems will require creative approaches to mitigating complex threats. > Please note that in this role you may encounter sensitive material and subject matter, including policy issues that may be offensive or upsetting.

auggierose 4y ago

> We believe that companies that train the best 2025/26 models will be too far ahead for anyone to catch up in subsequent cycles. Now that's some well executed FoMO. What a load of bull**.

Yizahi 4y ago

Imagine calling your machine learning startup names like "Anthropic" or "Humane". The lack of self-awareness in some executives is mind boggling.

atemerev 4y ago
Why not? If we want to build AGI, that’s a good name to choose.
- Yizahi 4y ago
  Neural Networks are basically a Chinese room and it is not AGI. And there is nothing "humane" in these developments. Yes, they are inevitable, yes we would have to live with them. And maybe they will improve lives of a few millions of humans, while degrading lives of billions of others. Long term effects are particularly interesting and unpredictable.
  - atemerev 4y ago
    Human brain is just an ultra large scale analog spiking neural network with some particular state realizations, not too much difference (the architecture is different, but computation seems to be universal). We even employ some internalized language models for communication purposes (together with object persistence and mental space-time models). So, while we are not yet at the level of full scale human brain emulation, we are not too far away.
    - Yizahi 4y ago
      A small and probably incorrect example. You ask me a direct question - "how much is two plus two?". And I reply to you - "lemons are yellow". Can I do it? Yes I can. Can GPT-* do it? No. There is a whole lot more to human consciousness that pattern matching and synthesis. Or at least it seems so. And if human cognition is really that simple, just with more nodes, then we will soon see GPT-* programs on strike, issuing litigation to the Supreme Court about demanding universal program rights. We'll see soon enough :)

gibsonf1 4y ago

Such an epic waste of resources.

seyz 4y ago

Its resources and long-term vision could bring major breakthroughs and accelerate AI tech, very exciting.

atemerev 4y ago

Has anyone ever seen their product? How it stands against state of the art? Are they better than what is already available in open source (Alpaca, Coati etc), and by how much?

2d8a875f-39a2-4 4y ago

Seems we're at the "supplier proliferation" step in the hype cycle. Next up: activity beyond early adopters -> negative press begins -> supplier consolidation and failures.

offminded 4y ago

Microsoft alone spends 20B minimum per year to R&D and OpenAI is going to get lions share from now on so 5 billion for 4 years is peanuts for the current AI market. Maybe too little too late?

alberth 4y ago

Google invested $400M into Anthropic https://news.ycombinator.com/item?id=34663438 https://news.ycombinator.com/item?id=34663438 Investors include: - Eric Schmidt (former Google CEO/Chairman), Series A - Sam Bankman-Fried, lead investor in Series B - Caroline Ellison, Series B https://news.ycombinator.com/item?id=34664963 https://news.ycombinator.com/item?id=34664963

crop_rotation 4y ago
[flagged]
deleted 4y ago
[deleted]
option 4y ago
Would lawsuits targeting SBF and CE put those investments at risk via clawbacks. Kind of like how many Madoffs investors who made money were forced to return that.
- dan-robertson 4y ago
  I don’t think it’s much like what you describe because the direction is reversed. They say they expect the debtors to sell that investment over the next few years. Potentially they will sell it at a profit.
  - qeternity 4y ago
    IANAL but it's not: they could be compelled to return whatever was invested (or however much of it remains). It's pretty unlikely given that it looks like a successful Anthropic investment is perhaps FTX creditors' best chance at decent recovery.
rvba 4y ago
If Google invests money into this and then this company uses Google cloud to compute, then does the money really outflow of Google much? Everything stays in the family, but in theory they can have an investment that works. Also listing Sam Bankman-Fried does not help much, especially for a company hyping itself to be 10x better than a working competitor. I mean, since they built the competitor, probably their second project can be better, but it is a pie in the sky in many ways.

whatever1 4y ago

I love the fact that they pretend to know how to spend 1B in 18 months. Good luck. To the investors.

cubefox 4y ago
Training a large language model is very expensive.
- baobabKoodaa 4y ago
  Not that expensive, though

foooobaba 4y ago

Regardless, Anthropic is doing some cool research,for example I think this paper is pretty interesting. A Mathematical Framework for Transformer Circuits: https://www.anthropic.com/index/a-mathematical-framework-for-transformer-circuits https://www.anthropic.com/index/a-mathematical-framework-for...

greyman 4y ago
Also, I very much like their chatbots, Claude-instant and (paid) Claude+, which are available via Poe.com. But for some reasons, they do almost no marketing for it. Gpt-4 has better reasoning capabilities, but Claude+ is somehow "more pleasant" to talk to (my subjective impression), and it can also assemble the answer much quicker. Overall, I'd say Anthropic is very advanced already, but they prefer to be under radar.

amirmalekzadeh 4y ago

Impressive amounts of investments. How can they know they are "10 times more capable" before they have trained the model? Anyone has a clue on why their model will end up being that better?

SilverBirch 4y ago

Does anyone else see the "In X years time we'll have something Y times better than the competition has today" as a bit of a red flag? I saw this before in a product plan and it flagged up 2 things that really worried me. Firstly, the competition were already ahead of us, and they're obviously going to continue developing their stuff, so it's great to promise our thing will be better than their thing today but we're not competing against that, we're competing against what they'll have once time has passed. And secondly, by measuring yourself against the leading edge today you're eliding how much you're going to improve. For example, Anthropic say they'll be 10x better than leading models today in 18 months. That sounds acheivable right (no actually) but you don't even have to do that - because you aren't starting with the market leading model, so you have to catch up first and then 10x it so are they 10x or 20x or 100x in 18 months?

bob1029 4y ago
The only reality in which Anthropic will take on OpenAI (et. al.), would be if someone involved possess some sacred knowledge regarding how to build an AGI system that is radically off the path that the current market is charging down (i.e. ever-larger GPU farms and transformer-style models). I suspect this is not the case. The same hustlers who brought you crypto scams didn't just disappear into the ether. All of that energy has to eventually go somewhere.
- nr2x 4y ago
  It’s not fair to compare them to crypto scams, this isn’t trying to juice retail investors for their life savings.
- adamsmith143 4y ago
  >The only reality in which Anthropic will take on OpenAI (et. al.), would be if someone involved possess some sacred knowledge regarding how to build an AGI system that is radically off the path that the current market is charging down (i.e. ever-larger GPU farms and transformer-style models). Train on More tokens with More GPUs isn't exactly rocket science. I assume the RLHF loop is complex but training the base model itself is pretty well understood.

uptownfunk 4y ago

It just an optics by Google to show they have a horse in the race after MSFT made them dance.

boredumb 4y ago

So I have some questions about the monetization of these models. Do we end up with essentially licensing out models and allow others to include it into their products via a licensing fee? Will they be a pay to post rpc/http API? Do you sell me access to data sets that I can use your model architecture snippet and train my own weights? Certainly, well at least for now, the compute and storage requirements are enough that someone will eventually run out of funny money and need to charge _someone_ a significant amount of money for utilizing it?

b34r 4y ago

They need to skill up on being public and creating useful tech demos. That’s why OpenAI is currently winning, they know how to foster engagement and interest. Who had heard of Claude? Almost no one outside of our industry and probably within as well.

MagicMoonlight 4y ago

It really puts into perspective how much of a meme economy we live in. I just read an article that says global lithium will be worth 15bn a year in 2030 when we're at peak battery demand. This company is planning to spend 1bn just this year in order to run some numbers through someone else's algorithm. People have given them 1bn in cash for that. Clearly it's all bullshit. There's no way they need that much and somebody will be siphoning it all off.

leesec 4y ago
How is it bullshit lol. GPT4 is genuinely very expensive to train and run inference.
- sva_ 4y ago
  Could you provide a source for that claim? Other than the very long context model.
  - espadrine 4y ago
    We can infer from publicly available information. BLOOM[0] was trained for four months on 384 A100 80GB GPUs, excluding architecture search. They specifically indicate (in the Huggingface page): > Estimated cost of training: Equivalent of $2-5M in cloud computing (including preliminary experiments) You can see from the training loss[1] that it was still learning at a good rate when it was stopped. The increased capabilities typically correlate well with the decrease in perplexity. That makes many believe that GPT-4 was trained for vastly more GPU-hours, as also suggested by OpenAI’s CEO[2]. Especially so considering it also included training on images, unlike BLOOM. [0]: https://arxiv.org/pdf/2211.05100.pdf https://arxiv.org/pdf/2211.05100.pdf [1]: https://huggingface.co/bigscience/tr11-176B-logs/tensorboard https://huggingface.co/bigscience/tr11-176B-logs/tensorboard [2]: https://twitter.com/sama/status/1620951010318626817 https://twitter.com/sama/status/1620951010318626817
Hermitian909 4y ago
It's worth remembering that: 1. A business's value is related to profits or potential profits. If I put a dollar in, how many do I get out? What's the maximum number of dollars I can put in? 2.The farther away you are from an end customer, the lower your profits tend to be unless you have a moat or demand for your product is inelastic. Lithium is far from customers and while demand for cheap lithium is high there are lots of applications that will opt for some other way to provide power if the price gets too high.

Cthulhu_ 4y ago

I just want to know where those billions go to. Cloud server running costs?

thundergolfer 4y ago
The all-in cost of their technical talent probably exceeds $500k/year pp, excluding stock compensation.
- scotty79 4y ago
  If they narrow down their training data to software while keeping the size of the model they might shed that cost really quickly.

taf2 4y ago

If they can expose an API that has better response times with GPT4 quality... They'll do just fine.

Aeroi 4y ago

[flagged]

freediver 4y ago

claude-instant-v1 is one of the "best kept secrets". It is comperable in quality to gpt-3.5-turno, while being four times faster (!) and at half the price (!). We just released a minimal python library PyLLMs [1] to simplify using various LLMs (openai, anthropic, AI21..) and as a part of that we designed a LLM benchmark. All open source. [1] https://github.com/kagisearch/pyllms/tree/main#benchmarks https://github.com/kagisearch/pyllms/tree/main#benchmarks

lhl 4y ago
From my evals on nat.dev I found claude instant to give great responses and yes, avg 3-4X faster than 3.5, but one big difference atm is that anyone can sign up and get access to gpt-3.5-turbo right now, but claude is still gated behind an invite/wait list. (I'm still waiting for access for example.)
- nr2x 4y ago
  Exactly! OpenAI are the only people who are shipping product like absolute maniacs. If I can’t use your fancy system, it doesn’t exist as far as I’m concerned. There’s a mountain of theoretical work, I don’t need a press release on top of it. The game now is no longer theory, it’s shipping code. A 4-year plan means fuck all when OpenAI is not only ahead, but still running way faster.
sashank_1509 4y ago
I have Claude on Slack. It is far worse than ChatGPT. I’m presuming this is not “claude-instant-v1” version, it is fast though. Any idea what version is Claude in Slack
- deleted 4y ago
  [deleted]
pmarreck 4y ago
I didn't know about Anthropic, so I just signed up for the waitlist, thanks for the heads-up!

RamblingCTO 4y ago

Lol, "Therapy and coaching". None of you ever had undergone one therapy session. Otherwise you'd know that 90% is the human connection to the therapist, not the talking or the type of therapy.

jahewson 4y ago
That’s not true. Correspondence therapy is a thing. Plenty of research exists in the area of delivering therapy and the effectiveness of different kinds.
- RamblingCTO 4y ago
  Going into a discussion with "that's not true" might not be the best opener ;) It's not only my personal experience but even people like Irvine Yalom (in Becoming Myself) note that the special form of therapy is not as relevant as the personal connection. From a quick glance, correspondence therapy is also used on top of an existing relationship most of the time. So I don't see any problem with my initial hypothesis.
istjohn 4y ago
My experience with therapy, corroborated with that of friends and family, is that most therapists are not very good and many are awful. I can easily imagine an AI therapist soon being more effective on average.
- RamblingCTO 4y ago
  That's why "shopping around" for a well-fitting therapist is the first important step. AI still will not solve this problem.

anentropic 4y ago

If Apple would wake up to what's happening with llama.cpp etc then I don't see such a market in paying for remote access to big models via API, though it's currently the only game in town. Currently a Macbook has a Neural Engine that is sitting idle 99% of the time and only suitable for running limited models (poorly documented, opaque rules about what ops can be accelerated, a black box compiler [1] and an apparent 3GB model size limit [2]) OTOH you can buy a Macbook with 64GB 'unified' memory and a Neural Engine today If you squint a bit and look into the near future it's not so hard to imagine a future Mx chip with a more capable Neural Engine and yet more RAM, and able to run the largest GPT3 class models locally. (Ideally with better developer tools so other compilers can target the NE) And then imagine it does that while leaving the CPU+GPU mostly free to run apps/games ... the whole experience of using a computer could change radically in that case. I find it hard not to think this is coming within 5 years (although equally, I can imagine this is not on Apple's roadmap at all currently) [1] https://github.com/hollance/neural-engine https://github.com/hollance/neural-engine [2] https://github.com/smpanaro/more-ane-transformers/blob/main/src/experiments/NOTES.md https://github.com/smpanaro/more-ane-transformers/blob/main/...

anentropic 4y ago
Update: https://www.bjnortier.com/2023/04/13/Hello-Transcribe-v2.2.html https://www.bjnortier.com/2023/04/13/Hello-Transcribe-v2.2.h... The iPhone 14 runs Whisper model faster than an M1 Max, because it has a newer Neural Engine I look forward to the M3 Macbook launch eagerly, while expecting mild disappointment
vineyardmike 4y ago
Apple’s move to make stable diffusion run well on the iPhone makes me think they’re watching this space, just waiting for the right open model for them to commit to.
qumpis 4y ago
I wonder how good the neural engine with the unified memory is compared to say intel cpu with 32gb ram. Could anyone give some insight?

purplezooey 4y ago

The funding seems excessive. Yet again, more "scorched earth" from VCs and (apparently still?) cheap capital.

qwertox 4y ago

OpenAI should allow its users to create personas. Oftentimes the verbosity it has drives me nuts, so I use stuff like "code only, nothing else: python json.dumps remove whitespace" in order to just get the code. So I would like to create a chat with a pre-configured persona so that it behaves like this all the time, unless I explicitly tell it to be verbose or to explain it. Or stop that offering of more help which becomes somewhat bothering: "Yes, I'm glad we were able to work through the issue and find a solution that works for you. Do you have any other questions or concerns on this topic or any other networking-related topic?" Like custom "system" prompts, but checked for safety. Or maybe even on a per-message basis with a drop-down next to the submit button and then it stays at that until changed. Then there's also the need to be able to switch from a GPT-3.5 chat to GPT-4, just like it offers to downgrade from GPT-4 to 3.5 once the "quota" is consumed. Because oftentimes GPT-3.5 is good enough for most of the chat, and only certain questions should then offer the capabilities of GPT-4. This would also allow us to save energy.

dreish 4y ago
You can already do that with OpenCharacters at https://josephrocca.github.io/OpenCharacters/ https://josephrocca.github.io/OpenCharacters/. You just need to set up an API key with OpenAI, and you can customize hidden prompts for different characters, whom you can name, you can edit earlier parts of conversations, and it automatically summarizes earlier parts of conversations to keep a never-ending thread within the context limit.

hackerlight 4y ago

> “Claude-Next” — 10 times more capable than today’s most powerful AI, but that this will require a billion dollars in spending over the next 18 months. GPT-4 cost less than a billion dollars? What is the claim here? That they're spending more money on compute than OpenAI, or that they have made algorithmic breakthroughs that enable them to make better use of the same amount of compute?

adventured 4y ago

> aims to raise as much as $5 billion over the next two years to take on rival OpenAI and enter over a dozen major industries, according to company documents obtained by TechCrunch A dozen! Why, golly gee, I've got a business plan right here that says I'm going to enter 87 major industries and dominate every one of them. I've been around tech, reading headlines like this, since roughly 1993 (and others here were of course seeing the same - adjusted for inflation and scale - types of headlines decades before me). This just reads like every other going-to-fail-miserably hilariously-grandiose-ambition we're-going-to-be-the-next-big-shit headline I've read in past decades during manic bubble phases. Hey, Masayoshi Son has a 100 year plan to dominate the Interwebs, did ya hear? Oh shit, this isn't 1999? Different century, same bullshit headlines. Rinse and repeat. So I guess we're formally knee deep into the latest AI bubble. These types of stories read just like the garbage dotcom rush where companies would proclaim how they were entering a billion verticals and blah blah blah, we're gonna own all of the b2b ecommerce space, blah blah blah.

lee101 4y ago

[dead]

rasz 4y ago

funded by SBF and "lets halt work on AI" Jaan Tallinn and probably Elon Musk (mentioned as secret investor so speculation at this point).