11 ms·

GitHub Copilot loses an average of $20 per user per month

161 points by georgehill 3y ago

ojkelly 3y ago
When I first used copilot a few months after it came out, I thought it was wildly undervalued—I’d pay $100/m for the value and time saved. It’s become such an integral part of my workflow, that until Zed got Copilot integration I couldn’t switch to it. Of course I’d prefer to continue paying $10 a month.
archibaldJ 3y ago
hmm intriguing; I wonder how is Cursor/Tabnine doing in this aspect or are they suffering similiar lose?
nymalt 3y ago
Just one thing I hate about Copilot is how it needs to be updated two times a day.
namelosw 3y ago
The problem is that the industry has't figured out how to properly pricing LLM applications yet. Before LLM, it's perfectly possible to spin up a SaaS on 5$ Digital Ocean VM and charge $4.99 per seat monthly. If you're using low overhead techs like Go and SQLite you might get away pretty far with a decent user base. But LLM is inheirently costly compared to those traditional apps. No matter if you're calling OpenAI or DIY your own GPU cluster it's gonna be way more expensive. Spin your own GPU might ended to be more expensive because utilization problems and upfront costs. The subscription model was kind of the silver-bullet for SaaS but it's probably not going to work well in the AI era. OpenAI, Elevenlabs, Runway, and Midjourney: they have subscription model but the quota is strict and tight. The "unlimited" plan is simply pay-as-you-go. Early wave of LLM products with unlimited subscription models like Github Copilot and Notion AI are probably pricing way too low. $7 or $10 is way too low to support heavy usage. But charging $50 might scare most user away because it exceeded people's expectation for SaaS. And probably still end up losing money. And hobby users may ended up paying too much for the core users - that will lead us back to sophisticated pricing tiers like Elevenlabs and Runway. Are there alternatives? I dunno. Maybe implement bring-your-own-key properly? Like OAuth but for LLMs? It's definitely interesting to see how things will turn out eventually.
skilled 3y ago
From yesterday, GitHub Copilot loses an average of $20 a month per user (https://news.ycombinator.com/item?id=37821756 https://news.ycombinator.com/item?id=37821756) (45 points | 16 comments)
linuxftw 3y ago
Copilot is useless compared to chat-gpt4. There's really no comparison. As far as the costs, it seems like transmitting things repeatedly would be more expensive than the chat model where you just transmit things when you want them.
- tedivm 3y ago
  Copilot is chat gpt4, and has been since roughly April. It's built on top of it.
  - simonw 3y ago
    It's not GPT-4, it's gpt-3.5-turbo (or a variant there-of). Source: I'm sitting in the audience of a talk about it at AI Engineering summit right now, the speaker confirmed it as gpt-3.5-turbo, switched from Codex.
  - tmikaeld 3y ago
    To be fair, before they switched to GPT-4 it was pretty bad..
    - onionisafruit 3y ago
      Interesting that you say that. I had the opposite thought that it doesn’t seem to have improved much over time, but I think my perception might be influenced by my habit of ignoring large suggestions and only looking at the results when it fills the rest of the line I was typing. I’m also using it with intellij instead of vscode, so for all I know I could be using an old version still.
  - TheRoque 3y ago
    Wow, I didn't know there, I must have missed the news. I swear I have seen everywhere that it was using this "Codex" model
    - onionisafruit 3y ago
      I think codex is what early versions of copilot used.
empath-nirvana 3y ago
I would, without hesitation, pay $30/mo for co-pilot. I would even consider paying $100/mo for it.
- arriu 3y ago
  At that rate I’d run my own model
hesdeadjim 3y ago
The moment I can train Copilot on my codebase and improve its predictions, I’d gladly pay more than $20. Even now, I find that I can’t live without it simply as a slightly more intelligent auto-complete.
- reustle 3y ago
  Check out https://cursor.so https://cursor.so - It does this pretty well without requiring much setup.
- bloopernova 3y ago
  Yeah, I really, really want to be able to configure copilot to explicitly include directories to scan in addition to its built-in corpus of knowledge. That would increase its worth to me a lot.
  - visarga 3y ago
    That's what I expected Copilot 2.0 to be, what a bummer.
- grogenaut 3y ago
  I've used some prototypes that tried it. One of the things it does it regurgitate the old patterns you want to stop using and not the new stuff you do. Basically it's a copy pasta tech debt generator.
  - js8 3y ago
    That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders.
    - visarga 3y ago
      They are great information meat grinders, the best. /s Prompting allows you to direct the model into a different way than the training data. If it were not so, LLMs would never solve problems that were not explicitly in their training set.
swexbe 3y ago
They're trying to corner the market and betting on the running costs of these models going down substantially in the future.
- rgbrgb 3y ago
  I think it’s a good bet based on watching inference speed of llama.cpp consistently improving and model ability / size on a similar trajectory. I expect there’s similar room for optimization (probably more) with hosted models. If you don’t care about code/model privacy (I do but it seems like most don’t yet at least) there are even more batching/caching tricks to engineer.
cabirum 3y ago
How came that usually privacy-minded hn crowd uses run-by-someone-else models so cheerfully and carelessly? You do realize any data sent to MS/openai/anyone will be stored forever and probably be used against you in the (near) future?
- thsksbd 3y ago
  Todays HN is as interesting as it was back in 2011. But it is not privacy minded like it once was (was it ever?) at all.
- yjftsjthsd-h 3y ago
  Are you sure it's the same people? That said, even if it is, there's something to be said for intentionally sharing a constrained set of code vs using something that spies on you in real life.
  - cabirum 3y ago
    Could be different sets of people actually! However. It will eventually, chunk by chunk, upload the full source code of the app you are writing. Complete with all typos and mis-pastes. Like youtube blocks videos with just a hint of copyrighted music, they may be able to detect use of "patented" algos or something like that and block you/ send a lawyer/etc.
    - weikju 3y ago
      > upload the full source code of the app you are writing The full code that’s already uploaded via “git push”?
      - cabirum 3y ago
        ... via git push to a trusted origin, which is not github.
        yjftsjthsd-h 3y ago
bloopernova 3y ago
The article says Microsoft wants computers to one day have Neural Processing Units (NPU) like most have GPUs. If that led to faster suggestions for copilot, I'd be all for it. Right now, a GPU is an NPU. Does anyone how an NPU would differ from a graphics card?
- dragontamer 3y ago
  GPUs are a SIMD architecture designed for 4x4 float32 matrix multiplications found in video games. But NPUs (like Googles NPU) are systolic arrays designed for 16x16 or even 256x256 float16 or even int8 matrix multiplications instead. ------- NVidia builds larger matrix multiplications out of the 4x4 float16 base that a SM is designed for. After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications.
  - dmurray 3y ago
    > After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications Isn't it 8 of them? That's why you can't go the other way around and use your 8x8 multiplier to efficiently multiply 4x4s in parallel.
    - uxp8u61q 3y ago
      Yes, it's eight multiplication with the naive algorithm. You can get to seven with a clever trick (Strassen's algorithm). As far as I'm aware, with these precise numbers (8x8 matrix decomposed into four 4x4) it's unknown if we can do better.
    - dragontamer 3y ago
      Yes, it is 8. I was mistaken earlier.
- Squarex 3y ago
  isn't it what neural engine on apple silicon is?
tandav 3y ago
Does anyone know vscode extension that can use OpenAI API to perform code completions like Copilot? There are several ChatGPT-like UIs that you can self host and pay only for API and not for ChatGPT plus. For example, I'm using https://github.com/Yidadaa/ChatGPT-Next-Web https://github.com/Yidadaa/ChatGPT-Next-Web. It would be nice to use Copilot in the same way.
- buremba 3y ago
  In my own testing, Copilot has much better suggestions compared to GPT-4
- 0x008 3y ago
  continue.dev can complete code as well, but it is not a mere code completion plugin like copilot, it is more like an interactive chatbot where you can edit and talk about parts of the code which you have selected. for mere code completion there is this: https://github.com/FarisHijazi/PrivateGitHubCopilot https://github.com/FarisHijazi/PrivateGitHubCopilot but I haven't tested it.
- ntoshev 3y ago
  You can modify this one to do that - or write your own using the same approach: https://github.com/danielgross/localpilot https://github.com/danielgross/localpilot
noodlesUK 3y ago
I feel like the centralisation of server resources for something like copilot doesn’t really make sense. Many (most) professional developers are working on beefy laptops. If ever there were a case to run these models client side, a software developer’s laptop is probably the ideal place. What are the specs that are needed to run inference on these models?
- civilized 3y ago
  There's a huge range of CPU and memory requirements to run these models. The lower end of the range can be run locally but performance is noticeably worse. I'm guessing this will change as more people are looking at efficiency now. Specialized hardware like Apple's Neural Engine on the MacBook Pro may also help.
- KaoruAoiShiho 3y ago
  A lot higher than the beefy laptop... Even a 4090 can't run it well.
- deleted 3y ago
  [deleted]
- insanitybit 3y ago
  I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried experimenting with some local models and: a) They're much slower on my 6GB laptop GPU b) The seem to not be as good, functionally b) I can't make use of larger models I haven't done more than just some experimentation but I can see how this would make sense to put on a server.
  - baq 3y ago
    MacBooks especially pros are in a very privileged position due to their memory architecture. With some dedicated hardware it may be reasonable to run, perhaps even train, useful LLMs on device. I guess we’ll see in a year or two. This must be on everyone’s radar now, Apple won’t be the odd man out.
LeicaLatte 3y ago
GitHub’s CoPilot is the most disappointing of AI products out there.
- bhouston 3y ago
  For some developers for sure, but not all. I am a senior developer with +20 years of experience and I find Github Copilot amazing. It significantly boosts my productivity. I do not expect the world from it, and I do run into limitations, but it helps me tremendously to be productive in languages and tools which I have low/no familiarity. I think Github Copilot paired with ChatGPT fails the most in idiosyncratic code bases where you are doing maintenance or complex refactors. I think Github Copilot with ChatGPT shines in writing green field code in areas where you are will to use the best practices and the open sources libraries that Chat GPT recommends. Basically, if you align with where Chat GPT recommends to go, then it will help you a lot more than if you try to fight it all the way.
  - uxp8u61q 3y ago
    It's also amazing to write documentation or unit tests. Or repetitive editing tasks that are hard to automate (e.g. "invert this if condition").
    - insanitybit 3y ago
      > or unit tests Seriously! I write a documentation comment like: /// Given: blah /// When: blah /// Then: blah and Copilot writes the test for me. I can generate a dozen tests like this and it's awesome. I write a lot more tests because of Copilot.
    - baq 3y ago
      It even does an ok job with ‘rewrite this callback heavy code as async/await’ if you tell it which functions to use instead of ones accepting callbacks.
  - pjc50 3y ago
    Which language? New or maintenance code?
bhouston 3y ago
Thus, the price of Github Copilot is going to go up, 2x to 5x. For all of the criticism of Github Copilot, for a lot of developers (but not all), the value Github Copilot is incredibly high, much more than $20/month. The current rock bottom pricing is low compared to the value provided for those users. As such there is a big opportunity to multiply the price here being charged. Probably an increase between 3x to 5x. The excuse is because they are losing money, but the underlying reason is that the value it is providing is so high in terms of developer productivity.
- insanitybit 3y ago
  I'd probably pay ~2-3x, idk about 5x. I'd consider it though. Hopefully they can cut the costs and meet in the middle.
  - bhouston 3y ago
    At some point, the open source models you can run locally are going to start to get competitive. I suspect the cost problem came from upgrading to ChatGPT 4 over ChatGPT 3 - right there their costs more than 5x.
    - refulgentis 3y ago
      Did they? I'm surprised to hear that. Hard to tell for autocomplete of course. For conversational answers, inference speed is _far_ higher than I'm used to from GPT-4, and lower quality. As in, it matches GPT 3.5/ChatGPT.
      - nithril 3y ago
        Not yet, it is still using GPT 3.5. The access to GitHub Copilot X (GPT-4) is in closed beta [1]. [1] https://github.com/features/preview/copilot-x https://github.com/features/preview/copilot-x
      - laurent_du 3y ago
        Quality is definitely way below GPT4.
bryanlarsen 3y ago
AI is such a huge expectations dichotomy. For those of us used to the continual disappointment that pre-LLM AI was, the current crop of LLM's are amazing, mind blowing things. We start raving about them, so other people take a look expecting that modern LLM's are the greatest thing since sliced bread. They're not quite that, so we get HN comments complaining that AI sucks.
- fennecfoxy 3y ago
  Eh I think people's expectations for things they don't understand are always overblown. One thing I always find funny is the general expectation that machine learning models are both incredibly generalised and designed based on the way biological systems work, but should also be 100% perfect and never be wrong just like a machine and NOT like a biological system, those things are mutually exclusive; even the best, smartest most physically capable humans will still sometimes spill their coffee, yet we expect coffee-bot 2024 not to do this. Certainly machines can be much better at a task than humans are, but if that tasks requires generalisation then it's still gonna fuck up from time to time. Machine learning programs may think a dog is actually a cat sometimes, but afaik they ain't ever called their teacher "Mum" yet.
- _5ygc 3y ago
  I think the problem is that they aren’t mind-blowing, but an improvement. I resent some LLM implementations on principle, but decided to give these code helpers a try. What I found was they’re reasonably bad, and I kept telling them the solution doesn’t work, only to be presented with a little tweak. So I don’t see the point of outsourcing my thinking, I’d rather remain intelligent and do the search/try/tweak on my own, instead of pretending a half-assed LLM is genius. That doesn’t mean they don’t have good use cases, or aren’t an improvement on previous tech. But we definitely should stop calling them mind-blowing. Jaron Lanier had long ago predicted we’d willingly downplay human intelligence to pretend AI was… I.
  - ryanklee 3y ago
    > I think the problem is that they aren’t mind-blowing, but an improvement. You seem to be doing what GP is pointing out. GP's claim is that the relative improvement itself is mind-blowing, not that the tech is mind-blowing in an absolute sense. I tend to agree: much of the detraction hangs on current-state rather than a probable potential-state informed by recent relative advancements. In other words, many proponents are chuffed because of that potential; not because actuality. Likewise, skeptics are reserved because of the actuality, and not because of potential, for whatever reason (there are at least a few main ones, I think).
habosa 3y ago
So the service costs MSFT an average of $30/user/month. So if they want to make a business with decent margins out of this, they're going to need to charge $60/user/month or more. That's not too much if it's adding a lot of value but it's also definitely a lot of money. It would be the most expensive SaaS my team pays for (and we have a lot of subscriptions).
- jetrink 3y ago
  Also, that is assuming that the group willing to pay $60/month includes many who use the service less than average. It's possible that only heavy users would stick around after a 6x price increase.
- yeck 3y ago
  As users' codebases grown over time, the average usage of users will probably continue to grow. By a static price increase alone won't be a great model. They need to make their models and infrastructure more efficient first. Or create a pricing model where the monthly sub covers a certain amount of tokens and then beyond that is pay-as-you-go. Sucks for users, but a scalable model.
baby 3y ago
What’s remarkable to me is that when copilot launched there were a lot of people on HN saying it was not good. Now it seems like everybody has adopted it and can’t live without it.
ackbar03 3y ago
I have a small cloud based Ai image processing service running that's still running (we did it before it was cool). The processing is currently done locally on a computer in my partners closet, but scaling it to that point, whilst ensuring it runs smoothly for multiple users with reasonable processing times was no joke.
deleted 3y ago
[deleted]
ChrisArchitect 3y ago
[dupe]
throwaway4good 3y ago
Maybe the guys doing DDoS attack on Google should give MS/Github a go?
Gijs4g 3y ago
Interesting. What does this say about GPT-4? How much do they lose per User per Month?
monocasa 3y ago
Doesn't copilot slurp up oodles of the code in your workspace for non corporate accounts? Is that data acquisition maybe worth the cost to Microsoft? I know they have github, but tiny scripting glue tasks like copilot excels at aren't the best represented on GitHub.
- jrockway 3y ago
  I use Copilot in Emacs and it doesn't send Github any data other than my prompt.
  - daft_pink 3y ago
    Do you specifically activate it? The main functionality I get out of copilot in vscode is autocomplete on steroids, so it's getting all code as I write it. I don't use the question based generator most of the time.
    - jrockway 3y ago
      I'm probably not using it to the fullest. I mostly use it for boilerplate that I know it's good at, like `// Write a function to run the cpu profiler for 30 seconds and write the result to the filename in variable "foo"`. But I guess if it knows your codebase, then you can say `// Write a test to ensure that Foo accepts a nil FooRequest.` or something. I should figure out how to set that up. All in all it saves me a little bit of typing a couple of times a week.
      - daft_pink 3y ago
        You should try it on vscode for a moment or try to find a plugin for emacs. I'm often writing legal rules in code and from the name of my function, it predicts the way the law is written. It's pretty incredible how well it works. Clearly it's just stealing someone else's work from github, but the autocomplete is very awesome.
  - monocasa 3y ago
    The vast majority of people using copilot aren't using it that way.
siva7 3y ago
I've done my own startup calculations for this. You shouldn't provide a heavily used openai wrapper below 30$/month/user. That will be likely a loss leader. At 46$/month it starts getting profitable for me.
Saline9515 3y ago
This is not surprising, the chat function is mind-blowing. It knows about your code, it is helpful, explains concepts, and is rarely wrong. It is the first time I'm genuinely impressed about an AI tool, and makes learning new frameworks very easy.
SadWebDeveloper 3y ago
I would gladly pay for copilot 5 USD a month (its an specific use case IA), but the current price point 20 month is incredible expensive it only add costs if the developer you are hiring requires it to properly function... thats the only value it adds. IMHO If you are a code monkey copilot makes you a 5x developer, 10x if you have 5+ years of experience on the business Maybe selling tokens/search could be benefitial for copilot, like only pay for what i use will be an interesting approach.
- WillPostForFood 3y ago
  $20 a month is incredibly expensive? If it saves 15 minutes over a month, it pays for itself (assuming your work is ~$100/hr). If it can provide any positive value, it is going to be worth more the $5 a month.
  - SadWebDeveloper 3y ago
    Maybe if you live around the bay area where you get paid absurd money by the hour for a job that can be done overseas for 10 bucks, outside or the rest of the world that 100 usd is per week tops, thats way too much i would pay for service that only barely helps my in my day to day... If you need to constantly use chatgpt/copilot to do your job properly, maybe its time to learn something else because your job will be automated sooner or later, what will be more cheaper a 100/hr or the 20/month service.
    - WillPostForFood 3y ago
      Does it barely help you, or can it completely automate your job? You can't have it both ways. I find it hard to believe there is a scenario where it actually helps you, but only provides $5 of value in a month. What exact kind help is that? If it is able to provide $5 in help, you could likely scale that to $20.
      - SadWebDeveloper 3y ago
        are we talking about copilot or chatgpt? chatgpt4 provides more value than copilot since i can use it for other things, copilot is just for "code" and doesn't provide any value other than autocomplete prediction to me... Maybe if you are website designer... ohh m sorry i should say the modern way website designers call themselves today "frontend engineer", maybe if you are a designer it could help you with all the boilerplate your react/astro/svelte requires just to add a <table> with two columns.
zackmorris 3y ago
Capitalism cracks me up. We live under this fantasy that everything we do can be turned into a profitable business. Even the government should be run like a business, right? Like somehow educating a child for 12+ years should be a profitable venture. Or housing the elderly. Or providing a basic service like water, sanitation or electricity that everyone uses at about the same rate. Surely we can stand on each other to turn a buck. I think that AI will be the biggest loss leader ever invented. We'll buy a robot one time for $10,000 that will perform $1 million of mental and physical labor over its lifetime. Except nobody will pay for that $990,000. There will just be this expectation of something for nothing, so the anticipated payment never comes. And it will happen so quickly, so completely, that we'll wonder how it was ever possible to pay people to do all this free stuff. No, the GitHub Copilot model of charging for AI services isn't going to scale. People are going to open source AI and own it themselves and stop paying for goods and services. Why would they when the AI can provide everything they need? AI is the beginning of the end of money. The headline foretells the end of the era of artificial scarcity, because capitalism can't compete with free.
- esafak 3y ago
  It will not be free; payment will simply shift to the hardware and data used to train the model, etc.
hm-nah 3y ago
I recently cancelled my GitHub CoPilot license. I prefer learning and co-developing with GPT-4.
aryadee 3y ago
GitHub Copilot is not using GPT-4 (yet)! I'm seeing a lot of people under the wrong impression. For code completions it uses an improved version of Codex[1], and for chat beta (part of Copilot X), it uses 3.5-Turbo[2]. However, they're claiming to have "early adoption of OpenAI's GPT-4" for Copilot X on their marketing page[3], which is confusing/deceptive. To be fair, it's still in beta, but they should state outright what model you currently get if you sign up for it. [1]: https://github.blog/2023-07-28-smarter-more-efficient-coding-github-copilot-goes-beyond-codex-with-improved-ai-model/ https://github.blog/2023-07-28-smarter-more-efficient-coding... [2]: https://github.com/orgs/community/discussions/56975#discussioncomment-6086471 https://github.com/orgs/community/discussions/56975#discussi... [3]: https://github.com/features/preview/copilot-x https://github.com/features/preview/copilot-x
_cenw 3y ago
If Copilot loses this much, just imagine how much OpenAI is losing on non-paying users using ChatGPT. Who's gonna pay for this as soon as the VC money runs out?
- tkellogg 3y ago
  the environment
- kyriakos 3y ago
  Maybe by then compute will become cheaper. I am sure they have some plan for the future regardless if it pans out in the end.
- ComputerGuru 3y ago
  Unless even their paid api usage is subsidized (and it very well could be for all I know) they can’t be losing that much. But it doesn’t matter, I’m they never would have gotten the valuations they did if it weren’t for the insane hype around the (free) ChatGPT offering. It more than paid for itself just in that regard.
- treis 3y ago
  They just need to keep it going until GPUs get cheaper. The point is to build/protect the monopoly until that happens. Then costs come down and they start printing an absurd amount of money.
  - VirusNewbie 3y ago
    well, and straight caching. If they know that 80% of people ask the same 10,000 questions without a back and forth dialogue, it's not hard to just write a front end for that.
    - vineyardmike 3y ago
      This problem won’t work for code assistants. No way those queries have high repetition. Not when you’re uploading user files. I assume the best strategy is to shrink the models and tune them more.
tristanb 3y ago
I use Copilot... Or rather I pay for it. I pretty much always end up using ChatGPT4. I honestly haven't had a very good experience with Copilot. It never "learns" anything. It doesn't remember where my imports are, it gives me the wrong paths all the time. Sure occasionally it will write a nice comment for me when I make a new prop, but other than that it's not as good for writing code as GPT4, and its auto suggestions are pretty weak. Wondering if others have different opinions.
jpallen 3y ago
A lot of people are commenting about how gpt4 is better than copilot, both in terms of getting it to do exactly what you want and in the quality of response. I've also found this and so I wrote a simple VS Code coding assistant that uses gpt4 and let's you be direct in asking what you want it to do: https://github.com/jpallen/biggles https://github.com/jpallen/biggles. Maybe it can be useful to some of you too. It also supports voice, which is a fun way to code!
petercooper 3y ago
FWIW, a GitHub employee just said on stage at the AI Engineer Summit that this claim is "not true" and that it's a roughly $100m ARR product (first keynote speaker in this stream: https://www.youtube.com/watch?v=qw4PrtyvJI0 https://www.youtube.com/watch?v=qw4PrtyvJI0)
- prattcmp 3y ago
  Revenue != profit
  - petercooper 3y ago
    Correct, but he also explicitly said the reports that it was losing money were "not true". Given that he's a VP at GitHub, I would consider him to be more authoritative on the matter than the WSJ's "multiple sources."
bluelightning2k 3y ago
Please don't take my Copilot away!
vinni2 3y ago
And it’s free if you’re an educator or a student.
tortoise_in 3y ago
Microsoft should only focus on windows at the end of day and keeps things as they are.

161 points by georgehill 3y ago

When I first used copilot a few months after it came out, I thought it was wildly undervalued—I’d pay $100/m for the value and time saved. It’s become such an integral part of my workflow, that until Zed got Copilot integration I couldn’t switch to it. Of course I’d prefer to continue paying $10 a month.

archibaldJ 3y ago

hmm intriguing; I wonder how is Cursor/Tabnine doing in this aspect or are they suffering similiar lose?

nymalt 3y ago

Just one thing I hate about Copilot is how it needs to be updated two times a day.

namelosw 3y ago

The problem is that the industry has't figured out how to properly pricing LLM applications yet. Before LLM, it's perfectly possible to spin up a SaaS on 5$ Digital Ocean VM and charge $4.99 per seat monthly. If you're using low overhead techs like Go and SQLite you might get away pretty far with a decent user base. But LLM is inheirently costly compared to those traditional apps. No matter if you're calling OpenAI or DIY your own GPU cluster it's gonna be way more expensive. Spin your own GPU might ended to be more expensive because utilization problems and upfront costs. The subscription model was kind of the silver-bullet for SaaS but it's probably not going to work well in the AI era. OpenAI, Elevenlabs, Runway, and Midjourney: they have subscription model but the quota is strict and tight. The "unlimited" plan is simply pay-as-you-go. Early wave of LLM products with unlimited subscription models like Github Copilot and Notion AI are probably pricing way too low. $7 or $10 is way too low to support heavy usage. But charging $50 might scare most user away because it exceeded people's expectation for SaaS. And probably still end up losing money. And hobby users may ended up paying too much for the core users - that will lead us back to sophisticated pricing tiers like Elevenlabs and Runway. Are there alternatives? I dunno. Maybe implement bring-your-own-key properly? Like OAuth but for LLMs? It's definitely interesting to see how things will turn out eventually.

skilled 3y ago

From yesterday, GitHub Copilot loses an average of $20 a month per user (https://news.ycombinator.com/item?id=37821756 https://news.ycombinator.com/item?id=37821756) (45 points | 16 comments)

linuxftw 3y ago

Copilot is useless compared to chat-gpt4. There's really no comparison. As far as the costs, it seems like transmitting things repeatedly would be more expensive than the chat model where you just transmit things when you want them.

tedivm 3y ago
Copilot is chat gpt4, and has been since roughly April. It's built on top of it.
- simonw 3y ago
  It's not GPT-4, it's gpt-3.5-turbo (or a variant there-of). Source: I'm sitting in the audience of a talk about it at AI Engineering summit right now, the speaker confirmed it as gpt-3.5-turbo, switched from Codex.
- tmikaeld 3y ago
  To be fair, before they switched to GPT-4 it was pretty bad..
  - onionisafruit 3y ago
    Interesting that you say that. I had the opposite thought that it doesn’t seem to have improved much over time, but I think my perception might be influenced by my habit of ignoring large suggestions and only looking at the results when it fills the rest of the line I was typing. I’m also using it with intellij instead of vscode, so for all I know I could be using an old version still.
- TheRoque 3y ago
  Wow, I didn't know there, I must have missed the news. I swear I have seen everywhere that it was using this "Codex" model
  - onionisafruit 3y ago
    I think codex is what early versions of copilot used.

empath-nirvana 3y ago

I would, without hesitation, pay $30/mo for co-pilot. I would even consider paying $100/mo for it.

arriu 3y ago
At that rate I’d run my own model

hesdeadjim 3y ago

The moment I can train Copilot on my codebase and improve its predictions, I’d gladly pay more than $20. Even now, I find that I can’t live without it simply as a slightly more intelligent auto-complete.

reustle 3y ago
Check out https://cursor.so https://cursor.so - It does this pretty well without requiring much setup.
bloopernova 3y ago
Yeah, I really, really want to be able to configure copilot to explicitly include directories to scan in addition to its built-in corpus of knowledge. That would increase its worth to me a lot.
- visarga 3y ago
  That's what I expected Copilot 2.0 to be, what a bummer.
grogenaut 3y ago
I've used some prototypes that tried it. One of the things it does it regurgitate the old patterns you want to stop using and not the new stuff you do. Basically it's a copy pasta tech debt generator.
- js8 3y ago
  That's a problem with all LLMs. They will average what they are trained on, without any reflection what is good and what is not. They shouldn't be called intelligent (AI), they're information meatgrinders.
  - visarga 3y ago
    They are great information meat grinders, the best. /s Prompting allows you to direct the model into a different way than the training data. If it were not so, LLMs would never solve problems that were not explicitly in their training set.

swexbe 3y ago

They're trying to corner the market and betting on the running costs of these models going down substantially in the future.

rgbrgb 3y ago
I think it’s a good bet based on watching inference speed of llama.cpp consistently improving and model ability / size on a similar trajectory. I expect there’s similar room for optimization (probably more) with hosted models. If you don’t care about code/model privacy (I do but it seems like most don’t yet at least) there are even more batching/caching tricks to engineer.

cabirum 3y ago

How came that usually privacy-minded hn crowd uses run-by-someone-else models so cheerfully and carelessly? You do realize any data sent to MS/openai/anyone will be stored forever and probably be used against you in the (near) future?

thsksbd 3y ago
Todays HN is as interesting as it was back in 2011. But it is not privacy minded like it once was (was it ever?) at all.
yjftsjthsd-h 3y ago
Are you sure it's the same people? That said, even if it is, there's something to be said for intentionally sharing a constrained set of code vs using something that spies on you in real life.
- cabirum 3y ago
  Could be different sets of people actually! However. It will eventually, chunk by chunk, upload the full source code of the app you are writing. Complete with all typos and mis-pastes. Like youtube blocks videos with just a hint of copyrighted music, they may be able to detect use of "patented" algos or something like that and block you/ send a lawyer/etc.
  - weikju 3y ago
    > upload the full source code of the app you are writing The full code that’s already uploaded via “git push”?
    - cabirum 3y ago
      ... via git push to a trusted origin, which is not github.
      - yjftsjthsd-h 3y ago

bloopernova 3y ago

The article says Microsoft wants computers to one day have Neural Processing Units (NPU) like most have GPUs. If that led to faster suggestions for copilot, I'd be all for it. Right now, a GPU is an NPU. Does anyone how an NPU would differ from a graphics card?

dragontamer 3y ago
GPUs are a SIMD architecture designed for 4x4 float32 matrix multiplications found in video games. But NPUs (like Googles NPU) are systolic arrays designed for 16x16 or even 256x256 float16 or even int8 matrix multiplications instead. ------- NVidia builds larger matrix multiplications out of the 4x4 float16 base that a SM is designed for. After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications.
- dmurray 3y ago
  > After all, a 8x8 matrix multiplication is just four of the smaller 4x4 matrix multiplications Isn't it 8 of them? That's why you can't go the other way around and use your 8x8 multiplier to efficiently multiply 4x4s in parallel.
  - uxp8u61q 3y ago
    Yes, it's eight multiplication with the naive algorithm. You can get to seven with a clever trick (Strassen's algorithm). As far as I'm aware, with these precise numbers (8x8 matrix decomposed into four 4x4) it's unknown if we can do better.
  - dragontamer 3y ago
    Yes, it is 8. I was mistaken earlier.
Squarex 3y ago
isn't it what neural engine on apple silicon is?

tandav 3y ago

Does anyone know vscode extension that can use OpenAI API to perform code completions like Copilot? There are several ChatGPT-like UIs that you can self host and pay only for API and not for ChatGPT plus. For example, I'm using https://github.com/Yidadaa/ChatGPT-Next-Web https://github.com/Yidadaa/ChatGPT-Next-Web. It would be nice to use Copilot in the same way.

buremba 3y ago
In my own testing, Copilot has much better suggestions compared to GPT-4
0x008 3y ago
continue.dev can complete code as well, but it is not a mere code completion plugin like copilot, it is more like an interactive chatbot where you can edit and talk about parts of the code which you have selected. for mere code completion there is this: https://github.com/FarisHijazi/PrivateGitHubCopilot https://github.com/FarisHijazi/PrivateGitHubCopilot but I haven't tested it.
ntoshev 3y ago
You can modify this one to do that - or write your own using the same approach: https://github.com/danielgross/localpilot https://github.com/danielgross/localpilot

noodlesUK 3y ago

I feel like the centralisation of server resources for something like copilot doesn’t really make sense. Many (most) professional developers are working on beefy laptops. If ever there were a case to run these models client side, a software developer’s laptop is probably the ideal place. What are the specs that are needed to run inference on these models?

civilized 3y ago
There's a huge range of CPU and memory requirements to run these models. The lower end of the range can be run locally but performance is noticeably worse. I'm guessing this will change as more people are looking at efficiency now. Specialized hardware like Apple's Neural Engine on the MacBook Pro may also help.
KaoruAoiShiho 3y ago
A lot higher than the beefy laptop... Even a 4090 can't run it well.
deleted 3y ago
[deleted]
insanitybit 3y ago
I don't think most dev laptops have 8GB+ of GPU memory, which (based on ollama requirements) seems like it's on the low-mid end of the requirements. I've tried experimenting with some local models and: a) They're much slower on my 6GB laptop GPU b) The seem to not be as good, functionally b) I can't make use of larger models I haven't done more than just some experimentation but I can see how this would make sense to put on a server.
- baq 3y ago
  MacBooks especially pros are in a very privileged position due to their memory architecture. With some dedicated hardware it may be reasonable to run, perhaps even train, useful LLMs on device. I guess we’ll see in a year or two. This must be on everyone’s radar now, Apple won’t be the odd man out.

LeicaLatte 3y ago

GitHub’s CoPilot is the most disappointing of AI products out there.

bhouston 3y ago
For some developers for sure, but not all. I am a senior developer with +20 years of experience and I find Github Copilot amazing. It significantly boosts my productivity. I do not expect the world from it, and I do run into limitations, but it helps me tremendously to be productive in languages and tools which I have low/no familiarity. I think Github Copilot paired with ChatGPT fails the most in idiosyncratic code bases where you are doing maintenance or complex refactors. I think Github Copilot with ChatGPT shines in writing green field code in areas where you are will to use the best practices and the open sources libraries that Chat GPT recommends. Basically, if you align with where Chat GPT recommends to go, then it will help you a lot more than if you try to fight it all the way.
- uxp8u61q 3y ago
  It's also amazing to write documentation or unit tests. Or repetitive editing tasks that are hard to automate (e.g. "invert this if condition").
  - insanitybit 3y ago
    > or unit tests Seriously! I write a documentation comment like: /// Given: blah /// When: blah /// Then: blah and Copilot writes the test for me. I can generate a dozen tests like this and it's awesome. I write a lot more tests because of Copilot.
  - baq 3y ago
    It even does an ok job with ‘rewrite this callback heavy code as async/await’ if you tell it which functions to use instead of ones accepting callbacks.
- pjc50 3y ago
  Which language? New or maintenance code?

bhouston 3y ago

Thus, the price of Github Copilot is going to go up, 2x to 5x. For all of the criticism of Github Copilot, for a lot of developers (but not all), the value Github Copilot is incredibly high, much more than $20/month. The current rock bottom pricing is low compared to the value provided for those users. As such there is a big opportunity to multiply the price here being charged. Probably an increase between 3x to 5x. The excuse is because they are losing money, but the underlying reason is that the value it is providing is so high in terms of developer productivity.

insanitybit 3y ago
I'd probably pay ~2-3x, idk about 5x. I'd consider it though. Hopefully they can cut the costs and meet in the middle.
- bhouston 3y ago
  At some point, the open source models you can run locally are going to start to get competitive. I suspect the cost problem came from upgrading to ChatGPT 4 over ChatGPT 3 - right there their costs more than 5x.
  - refulgentis 3y ago
    Did they? I'm surprised to hear that. Hard to tell for autocomplete of course. For conversational answers, inference speed is _far_ higher than I'm used to from GPT-4, and lower quality. As in, it matches GPT 3.5/ChatGPT.
    - nithril 3y ago
      Not yet, it is still using GPT 3.5. The access to GitHub Copilot X (GPT-4) is in closed beta [1]. [1] https://github.com/features/preview/copilot-x https://github.com/features/preview/copilot-x
    - laurent_du 3y ago
      Quality is definitely way below GPT4.

bryanlarsen 3y ago

AI is such a huge expectations dichotomy. For those of us used to the continual disappointment that pre-LLM AI was, the current crop of LLM's are amazing, mind blowing things. We start raving about them, so other people take a look expecting that modern LLM's are the greatest thing since sliced bread. They're not quite that, so we get HN comments complaining that AI sucks.

fennecfoxy 3y ago
Eh I think people's expectations for things they don't understand are always overblown. One thing I always find funny is the general expectation that machine learning models are both incredibly generalised and designed based on the way biological systems work, but should also be 100% perfect and never be wrong just like a machine and NOT like a biological system, those things are mutually exclusive; even the best, smartest most physically capable humans will still sometimes spill their coffee, yet we expect coffee-bot 2024 not to do this. Certainly machines can be much better at a task than humans are, but if that tasks requires generalisation then it's still gonna fuck up from time to time. Machine learning programs may think a dog is actually a cat sometimes, but afaik they ain't ever called their teacher "Mum" yet.
_5ygc 3y ago
I think the problem is that they aren’t mind-blowing, but an improvement. I resent some LLM implementations on principle, but decided to give these code helpers a try. What I found was they’re reasonably bad, and I kept telling them the solution doesn’t work, only to be presented with a little tweak. So I don’t see the point of outsourcing my thinking, I’d rather remain intelligent and do the search/try/tweak on my own, instead of pretending a half-assed LLM is genius. That doesn’t mean they don’t have good use cases, or aren’t an improvement on previous tech. But we definitely should stop calling them mind-blowing. Jaron Lanier had long ago predicted we’d willingly downplay human intelligence to pretend AI was… I.
- ryanklee 3y ago
  > I think the problem is that they aren’t mind-blowing, but an improvement. You seem to be doing what GP is pointing out. GP's claim is that the relative improvement itself is mind-blowing, not that the tech is mind-blowing in an absolute sense. I tend to agree: much of the detraction hangs on current-state rather than a probable potential-state informed by recent relative advancements. In other words, many proponents are chuffed because of that potential; not because actuality. Likewise, skeptics are reserved because of the actuality, and not because of potential, for whatever reason (there are at least a few main ones, I think).

habosa 3y ago

So the service costs MSFT an average of $30/user/month. So if they want to make a business with decent margins out of this, they're going to need to charge $60/user/month or more. That's not too much if it's adding a lot of value but it's also definitely a lot of money. It would be the most expensive SaaS my team pays for (and we have a lot of subscriptions).

jetrink 3y ago
Also, that is assuming that the group willing to pay $60/month includes many who use the service less than average. It's possible that only heavy users would stick around after a 6x price increase.
yeck 3y ago
As users' codebases grown over time, the average usage of users will probably continue to grow. By a static price increase alone won't be a great model. They need to make their models and infrastructure more efficient first. Or create a pricing model where the monthly sub covers a certain amount of tokens and then beyond that is pay-as-you-go. Sucks for users, but a scalable model.

baby 3y ago

What’s remarkable to me is that when copilot launched there were a lot of people on HN saying it was not good. Now it seems like everybody has adopted it and can’t live without it.

ackbar03 3y ago

I have a small cloud based Ai image processing service running that's still running (we did it before it was cool). The processing is currently done locally on a computer in my partners closet, but scaling it to that point, whilst ensuring it runs smoothly for multiple users with reasonable processing times was no joke.

deleted 3y ago

[deleted]

ChrisArchitect 3y ago

[dupe]

throwaway4good 3y ago

Maybe the guys doing DDoS attack on Google should give MS/Github a go?

Gijs4g 3y ago

Interesting. What does this say about GPT-4? How much do they lose per User per Month?

monocasa 3y ago

Doesn't copilot slurp up oodles of the code in your workspace for non corporate accounts? Is that data acquisition maybe worth the cost to Microsoft? I know they have github, but tiny scripting glue tasks like copilot excels at aren't the best represented on GitHub.

jrockway 3y ago
I use Copilot in Emacs and it doesn't send Github any data other than my prompt.
- daft_pink 3y ago
  Do you specifically activate it? The main functionality I get out of copilot in vscode is autocomplete on steroids, so it's getting all code as I write it. I don't use the question based generator most of the time.
  - jrockway 3y ago
    I'm probably not using it to the fullest. I mostly use it for boilerplate that I know it's good at, like `// Write a function to run the cpu profiler for 30 seconds and write the result to the filename in variable "foo"`. But I guess if it knows your codebase, then you can say `// Write a test to ensure that Foo accepts a nil FooRequest.` or something. I should figure out how to set that up. All in all it saves me a little bit of typing a couple of times a week.
    - daft_pink 3y ago
      You should try it on vscode for a moment or try to find a plugin for emacs. I'm often writing legal rules in code and from the name of my function, it predicts the way the law is written. It's pretty incredible how well it works. Clearly it's just stealing someone else's work from github, but the autocomplete is very awesome.
- monocasa 3y ago
  The vast majority of people using copilot aren't using it that way.

siva7 3y ago

I've done my own startup calculations for this. You shouldn't provide a heavily used openai wrapper below 30$/month/user. That will be likely a loss leader. At 46$/month it starts getting profitable for me.

Saline9515 3y ago

This is not surprising, the chat function is mind-blowing. It knows about your code, it is helpful, explains concepts, and is rarely wrong. It is the first time I'm genuinely impressed about an AI tool, and makes learning new frameworks very easy.

SadWebDeveloper 3y ago

I would gladly pay for copilot 5 USD a month (its an specific use case IA), but the current price point 20 month is incredible expensive it only add costs if the developer you are hiring requires it to properly function... thats the only value it adds. IMHO If you are a code monkey copilot makes you a 5x developer, 10x if you have 5+ years of experience on the business Maybe selling tokens/search could be benefitial for copilot, like only pay for what i use will be an interesting approach.

WillPostForFood 3y ago
$20 a month is incredibly expensive? If it saves 15 minutes over a month, it pays for itself (assuming your work is ~$100/hr). If it can provide any positive value, it is going to be worth more the $5 a month.
- SadWebDeveloper 3y ago
  Maybe if you live around the bay area where you get paid absurd money by the hour for a job that can be done overseas for 10 bucks, outside or the rest of the world that 100 usd is per week tops, thats way too much i would pay for service that only barely helps my in my day to day... If you need to constantly use chatgpt/copilot to do your job properly, maybe its time to learn something else because your job will be automated sooner or later, what will be more cheaper a 100/hr or the 20/month service.
  - WillPostForFood 3y ago
    Does it barely help you, or can it completely automate your job? You can't have it both ways. I find it hard to believe there is a scenario where it actually helps you, but only provides $5 of value in a month. What exact kind help is that? If it is able to provide $5 in help, you could likely scale that to $20.
    - SadWebDeveloper 3y ago
      are we talking about copilot or chatgpt? chatgpt4 provides more value than copilot since i can use it for other things, copilot is just for "code" and doesn't provide any value other than autocomplete prediction to me... Maybe if you are website designer... ohh m sorry i should say the modern way website designers call themselves today "frontend engineer", maybe if you are a designer it could help you with all the boilerplate your react/astro/svelte requires just to add a <table> with two columns.

zackmorris 3y ago

Capitalism cracks me up. We live under this fantasy that everything we do can be turned into a profitable business. Even the government should be run like a business, right? Like somehow educating a child for 12+ years should be a profitable venture. Or housing the elderly. Or providing a basic service like water, sanitation or electricity that everyone uses at about the same rate. Surely we can stand on each other to turn a buck. I think that AI will be the biggest loss leader ever invented. We'll buy a robot one time for $10,000 that will perform $1 million of mental and physical labor over its lifetime. Except nobody will pay for that $990,000. There will just be this expectation of something for nothing, so the anticipated payment never comes. And it will happen so quickly, so completely, that we'll wonder how it was ever possible to pay people to do all this free stuff. No, the GitHub Copilot model of charging for AI services isn't going to scale. People are going to open source AI and own it themselves and stop paying for goods and services. Why would they when the AI can provide everything they need? AI is the beginning of the end of money. The headline foretells the end of the era of artificial scarcity, because capitalism can't compete with free.

esafak 3y ago
It will not be free; payment will simply shift to the hardware and data used to train the model, etc.

hm-nah 3y ago

I recently cancelled my GitHub CoPilot license. I prefer learning and co-developing with GPT-4.

aryadee 3y ago

GitHub Copilot is not using GPT-4 (yet)! I'm seeing a lot of people under the wrong impression. For code completions it uses an improved version of Codex[1], and for chat beta (part of Copilot X), it uses 3.5-Turbo[2]. However, they're claiming to have "early adoption of OpenAI's GPT-4" for Copilot X on their marketing page[3], which is confusing/deceptive. To be fair, it's still in beta, but they should state outright what model you currently get if you sign up for it. [1]: https://github.blog/2023-07-28-smarter-more-efficient-coding-github-copilot-goes-beyond-codex-with-improved-ai-model/ https://github.blog/2023-07-28-smarter-more-efficient-coding... [2]: https://github.com/orgs/community/discussions/56975#discussioncomment-6086471 https://github.com/orgs/community/discussions/56975#discussi... [3]: https://github.com/features/preview/copilot-x https://github.com/features/preview/copilot-x

_cenw 3y ago

If Copilot loses this much, just imagine how much OpenAI is losing on non-paying users using ChatGPT. Who's gonna pay for this as soon as the VC money runs out?

tkellogg 3y ago
the environment
kyriakos 3y ago
Maybe by then compute will become cheaper. I am sure they have some plan for the future regardless if it pans out in the end.
ComputerGuru 3y ago
Unless even their paid api usage is subsidized (and it very well could be for all I know) they can’t be losing that much. But it doesn’t matter, I’m they never would have gotten the valuations they did if it weren’t for the insane hype around the (free) ChatGPT offering. It more than paid for itself just in that regard.
treis 3y ago
They just need to keep it going until GPUs get cheaper. The point is to build/protect the monopoly until that happens. Then costs come down and they start printing an absurd amount of money.
- VirusNewbie 3y ago
  well, and straight caching. If they know that 80% of people ask the same 10,000 questions without a back and forth dialogue, it's not hard to just write a front end for that.
  - vineyardmike 3y ago
    This problem won’t work for code assistants. No way those queries have high repetition. Not when you’re uploading user files. I assume the best strategy is to shrink the models and tune them more.

tristanb 3y ago

I use Copilot... Or rather I pay for it. I pretty much always end up using ChatGPT4. I honestly haven't had a very good experience with Copilot. It never "learns" anything. It doesn't remember where my imports are, it gives me the wrong paths all the time. Sure occasionally it will write a nice comment for me when I make a new prop, but other than that it's not as good for writing code as GPT4, and its auto suggestions are pretty weak. Wondering if others have different opinions.

jpallen 3y ago

A lot of people are commenting about how gpt4 is better than copilot, both in terms of getting it to do exactly what you want and in the quality of response. I've also found this and so I wrote a simple VS Code coding assistant that uses gpt4 and let's you be direct in asking what you want it to do: https://github.com/jpallen/biggles https://github.com/jpallen/biggles. Maybe it can be useful to some of you too. It also supports voice, which is a fun way to code!

petercooper 3y ago

FWIW, a GitHub employee just said on stage at the AI Engineer Summit that this claim is "not true" and that it's a roughly $100m ARR product (first keynote speaker in this stream: https://www.youtube.com/watch?v=qw4PrtyvJI0 https://www.youtube.com/watch?v=qw4PrtyvJI0)

prattcmp 3y ago
Revenue != profit
- petercooper 3y ago
  Correct, but he also explicitly said the reports that it was losing money were "not true". Given that he's a VP at GitHub, I would consider him to be more authoritative on the matter than the WSJ's "multiple sources."

bluelightning2k 3y ago

Please don't take my Copilot away!

vinni2 3y ago

And it’s free if you’re an educator or a student.

tortoise_in 3y ago

Microsoft should only focus on windows at the end of day and keeps things as they are.