4 ms·

JetBrains ships a local, offline AI auto-complete in PyCharm. Sure it's limited, but it shows it can be done. It's a pity other companies aren't trying the same

by janice1999 2y ago

JetBrains ships a local, offline AI auto-complete in PyCharm. Sure it's limited, but it shows it can be done. It's a pity other companies aren't trying the same, especially this one that boasts about millisecond editing times. Edit: JetBrains, not IntelliJ. Auto-complete details - https://blog.jetbrains.com/blog/2024/04/04/full-line-code-completion-in-jetbrains-ides-all-you-need-to-know/ https://blog.jetbrains.com/blog/2024/04/04/full-line-code-co...

_joel 2y ago
Interesting, have you used it/found it to be usable? I use IntelliJ myself, but it's not known for being a lean beast. With the addition of stuff like test containers and a few plugins, I'd be suprised if my machine didn't melt adding a local LLM too.
- janice1999 2y ago
  I mainly use PyCharm and I found the auto-complete to be good. It doesn't always kick in when I expect but some of the suggestions have been surprisingly complex and correct.
jstummbillig 2y ago
What it shows is that it can be done — in a limited way. Other people might not like those limits and chose to go a different way. I am not sure what's worth lamenting here.
- janice1999 2y ago
  > I am not sure what's worth lamenting here. The normalisation of surrendering all our code to remove AI cloud gods maybe? The other being a super responsive IDE now having major features have network requests delaying them, although HW requirements likely make that faster for most people.
  - jstummbillig 2y ago
    That sounds a little too spooky for my taste, but you do you. What anything beyond that means, in effect, is that you (not necessarily you) want to chose my values for me (not necessarily me). I don't see why I should care to have you do that.
- 0cf8612b2e1e 2y ago
  Zed is trying to position itself as a next gen hackers editor. Text editing with lessons learned from the multitude of experiments in the space. A flagship feature that is online only and requires me to trust a third party is a deal breaker to many. For instance, my employer would not be pleased if I suddenly started sharing code with another. Take the AI out of the conversation: if you told your employer you shared the codebase, that’s an insta-fire kind of move.
  - nprateem 2y ago
    Yeah but so is I sent a message to s random person about our business vs I sent a message to one of our suppliers
    - 0cf8612b2e1e 2y ago
      Is Zed a supplier? Sounds like a random developer can sign up at will without any corporate NDAs or other established legal agreements. Will my employer hash out the NDAs for every developer tool that wants to incorporate cloud AI for the lulz?
raincole 2y ago
I tried it with C#. It's not comparable to Copilot. If I've never used Copilot I might be slightly impressed.
- teaearlgraycold 2y ago
  I don’t see programming getting exponentially harder. So eventually local compute will catch up with programming needs and offline AI will be pretty good.
  - lovethevoid 2y ago
    The local compute problem is a cost problem, and most people do not have the hardware necessary and judging by trends this will continue for another ~decade.
  - __loam 2y ago
    Local compute is also getting progressively harder to make faster. I have a CPU from 6 years ago and checked some benchmarks to see if upgrades would be worth it. It's a 90% increase in performance with a 27% improvement in single threaded. Pretty substantial but not exponential. GPUs are advancing more quickly than CPUs right now but I wouldn't be surprised if they hit a wall soon.
- sangnoir 2y ago
  For those who don't have the latitude of uploading their code to Microsoft servers, smaller, limited but fully local AI is better than nothing.
- rileymichael 2y ago
  You're right, it's not comparable. Jetbrain's code completion actually integrates with the IDE to guarantee the suggested code isn't entirely a hallucination -- for example it checks for unresolved references so that its suggestions don't reference non-existent variables/methods. I've disabled Copilot as it produces garbage way too often. I found myself "pausing" to wait for the suggestion that I'd ultimately ignore because it was completely invalid. I've left Jetbrain's code completion on though because it's basically just a mildly "smarter" autocomplete that I'll occasionally use / I don't find myself relying on.
treesciencebot 2y ago
running a much worse model at a higher latency (since local GPU power is limited) is a worse experience for Zed.
xemoka 2y ago
Zed will hook into ollama as well; so you can run your own locally; just not as part of the same process (yet?).
- vineyardmike 2y ago
  Hooking into standard APIs like Ollama is definitely preferred (at least to me) because it means it’s more composable and modular. Very Unix-y. Lets you use whatever works for you instead of the vendor.
jsheard 2y ago
What's the impact on laptop battery life roughly, when you have a local LLM constantly running in the background of your text editor?
- jeroenhd 2y ago
  IntelliJ is an IDE, though, not a text editor. If you want a text editor with AI, you may need to wait for Microsoft to bring ChatGPT to notepad. It seems to be limited to a single CPU core at a time, so depending on your CPU boosting settings, some but not too much. It's quite snappy despite not using a lot of resources. I tried replicating the effect using a CUDA-based alternative but I could get neither the snappiness nor the quality out of that.
- deleted 2y ago
  [deleted]
nsonha 2y ago
AI autocomplete is just a toy, it has been there since the begining of AI code and I find it pretty useless. I prefer making a proper prompt to get better answers. Also most of the time engineers (use AI to) modify existing codebases rather than making new code.
- rty32 2y ago
  Google's systematic, quantitative evaluation of AI coding assistance tools disagrees with your personal and anecdotal findings (which agrees with many other similar studies): https://research.google/blog/ai-in-software-engineering-at-google-progress-and-the-path-ahead/ https://research.google/blog/ai-in-software-engineering-at-g...
  - lupire 2y ago
    I don't see a comparison to Non-GenAI auto complete there. Eclipse wrote half my code characters 15 years ago.
- simonw 2y ago
  What programming languages do you mostly work in? I've been wondering if the benefits of AI-autocomplete are more material to people who work in languages like Python and JavaScript that are harder to provide IDE-based autocomplete for. If you're a Java or TypeScript developer maybe the impact is reduced because you already have great autocomplete by default.
  - cjonas 2y ago
    I think typescript benefits even more than JavaScript because the type signatures add so much context. After defining the types and signatures copilot will often complete the body within a few changes of perfect
    - zackproser 2y ago
      Yeah +1, the current top LLMs do pretty well with TypeScript
kilohotel 2y ago
sure, but if it was good more people was use it
TechDebtDevin 2y ago
Maybe its because there's literally no point in using a local llm for code-completion. You'd be spending 90% of your time correcting it. Its barely worth it to use co-pilot.
deleted 2y ago
[deleted]
rkharsan64 2y ago
I agree that the completions might not be that great, but for context: this is a 100M model, while most models that people compare this to are atleast 100x bigger. They also focus on single line completions, and ship different models per programming language. All these make it possible to ship a decent completion engine with a very small download size.

_joel 2y ago

Interesting, have you used it/found it to be usable? I use IntelliJ myself, but it's not known for being a lean beast. With the addition of stuff like test containers and a few plugins, I'd be suprised if my machine didn't melt adding a local LLM too.

janice1999 2y ago
I mainly use PyCharm and I found the auto-complete to be good. It doesn't always kick in when I expect but some of the suggestions have been surprisingly complex and correct.

jstummbillig 2y ago

What it shows is that it can be done — in a limited way. Other people might not like those limits and chose to go a different way. I am not sure what's worth lamenting here.

janice1999 2y ago
> I am not sure what's worth lamenting here. The normalisation of surrendering all our code to remove AI cloud gods maybe? The other being a super responsive IDE now having major features have network requests delaying them, although HW requirements likely make that faster for most people.
- jstummbillig 2y ago
  That sounds a little too spooky for my taste, but you do you. What anything beyond that means, in effect, is that you (not necessarily you) want to chose my values for me (not necessarily me). I don't see why I should care to have you do that.
0cf8612b2e1e 2y ago
Zed is trying to position itself as a next gen hackers editor. Text editing with lessons learned from the multitude of experiments in the space. A flagship feature that is online only and requires me to trust a third party is a deal breaker to many. For instance, my employer would not be pleased if I suddenly started sharing code with another. Take the AI out of the conversation: if you told your employer you shared the codebase, that’s an insta-fire kind of move.
- nprateem 2y ago
  Yeah but so is I sent a message to s random person about our business vs I sent a message to one of our suppliers
  - 0cf8612b2e1e 2y ago
    Is Zed a supplier? Sounds like a random developer can sign up at will without any corporate NDAs or other established legal agreements. Will my employer hash out the NDAs for every developer tool that wants to incorporate cloud AI for the lulz?

raincole 2y ago

I tried it with C#. It's not comparable to Copilot. If I've never used Copilot I might be slightly impressed.

teaearlgraycold 2y ago
I don’t see programming getting exponentially harder. So eventually local compute will catch up with programming needs and offline AI will be pretty good.
- lovethevoid 2y ago
  The local compute problem is a cost problem, and most people do not have the hardware necessary and judging by trends this will continue for another ~decade.
- __loam 2y ago
  Local compute is also getting progressively harder to make faster. I have a CPU from 6 years ago and checked some benchmarks to see if upgrades would be worth it. It's a 90% increase in performance with a 27% improvement in single threaded. Pretty substantial but not exponential. GPUs are advancing more quickly than CPUs right now but I wouldn't be surprised if they hit a wall soon.
sangnoir 2y ago
For those who don't have the latitude of uploading their code to Microsoft servers, smaller, limited but fully local AI is better than nothing.
rileymichael 2y ago
You're right, it's not comparable. Jetbrain's code completion actually integrates with the IDE to guarantee the suggested code isn't entirely a hallucination -- for example it checks for unresolved references so that its suggestions don't reference non-existent variables/methods. I've disabled Copilot as it produces garbage way too often. I found myself "pausing" to wait for the suggestion that I'd ultimately ignore because it was completely invalid. I've left Jetbrain's code completion on though because it's basically just a mildly "smarter" autocomplete that I'll occasionally use / I don't find myself relying on.

treesciencebot 2y ago

running a much worse model at a higher latency (since local GPU power is limited) is a worse experience for Zed.

xemoka 2y ago

Zed will hook into ollama as well; so you can run your own locally; just not as part of the same process (yet?).

vineyardmike 2y ago
Hooking into standard APIs like Ollama is definitely preferred (at least to me) because it means it’s more composable and modular. Very Unix-y. Lets you use whatever works for you instead of the vendor.

jsheard 2y ago

What's the impact on laptop battery life roughly, when you have a local LLM constantly running in the background of your text editor?

jeroenhd 2y ago
IntelliJ is an IDE, though, not a text editor. If you want a text editor with AI, you may need to wait for Microsoft to bring ChatGPT to notepad. It seems to be limited to a single CPU core at a time, so depending on your CPU boosting settings, some but not too much. It's quite snappy despite not using a lot of resources. I tried replicating the effect using a CUDA-based alternative but I could get neither the snappiness nor the quality out of that.
deleted 2y ago
[deleted]

nsonha 2y ago

AI autocomplete is just a toy, it has been there since the begining of AI code and I find it pretty useless. I prefer making a proper prompt to get better answers. Also most of the time engineers (use AI to) modify existing codebases rather than making new code.

rty32 2y ago
Google's systematic, quantitative evaluation of AI coding assistance tools disagrees with your personal and anecdotal findings (which agrees with many other similar studies): https://research.google/blog/ai-in-software-engineering-at-google-progress-and-the-path-ahead/ https://research.google/blog/ai-in-software-engineering-at-g...
- lupire 2y ago
  I don't see a comparison to Non-GenAI auto complete there. Eclipse wrote half my code characters 15 years ago.
simonw 2y ago
What programming languages do you mostly work in? I've been wondering if the benefits of AI-autocomplete are more material to people who work in languages like Python and JavaScript that are harder to provide IDE-based autocomplete for. If you're a Java or TypeScript developer maybe the impact is reduced because you already have great autocomplete by default.
- cjonas 2y ago
  I think typescript benefits even more than JavaScript because the type signatures add so much context. After defining the types and signatures copilot will often complete the body within a few changes of perfect
  - zackproser 2y ago
    Yeah +1, the current top LLMs do pretty well with TypeScript

kilohotel 2y ago

sure, but if it was good more people was use it

TechDebtDevin 2y ago

Maybe its because there's literally no point in using a local llm for code-completion. You'd be spending 90% of your time correcting it. Its barely worth it to use co-pilot.

deleted 2y ago

[deleted]

rkharsan64 2y ago

I agree that the completions might not be that great, but for context: this is a 100M model, while most models that people compare this to are atleast 100x bigger. They also focus on single line completions, and ship different models per programming language. All these make it possible to ship a decent completion engine with a very small download size.