sestinj

9 ms·

61.
by sestinj 3y ago
This is a huge deal, congrats! We've had a ton of users asking how to run their own LLMs on Linux and the unfortunate answer was always that the existing options were slightly complicated. Having a single-click to download option is go
62.
Efficient RLHF: Reducing the Memory Usage of PPO(arxiv.org)
2 points by sestinj 3y ago|0 comments
63.
by sestinj 3y ago
Yeah, I asked it to write bubble sort in python, and it did so perfectly, but then decided to add a couple dozen parentheses to the end. I then followed up asking it to remove these, and it didn't seem to grasp that there was a convers
64.
by sestinj 3y ago
We've done work to set Continue up to be a JetBrains plugin—it's a matter of just a) figuring out the basics of putting a webview in a JetBrains plugin and b) implementing this class ( https://github.com/continuedev
65.
by sestinj 3y ago
If you run Ollama locally, the cost is zero. With Replicate, you pay about $0.005 per second, so with an average of say 15 seconds per request to generate the full response, and 25 inputs per day, this comes out to ~$50/month. More exp
66.
by sestinj 3y ago
^exactly, Replicate or Together are only options that are potentially more convenient
67.
by sestinj 3y ago
It's definitely a knowledgable model. I'm seeing a bit of trouble with stopping at the right point and managing chat context, but this is all a matter of prompt engineering. Very promising, and exciting work to be done!
68.
by sestinj 3y ago
Y'all beat me to the punch :) Curious to hear how your experience with Code Llama x Continue turns out
69.
by sestinj 3y ago
You can use Continue for all of this, as easy as highlighting code and making the request. We also support using Code Llama: https://continue.dev/docs/walkthroughs/codellama
70.
by sestinj 3y ago
You can use Continue as a drop-in replacement for Copilot Chat with Code Llama. We've released a short tutorial here: https://continue.dev/docs/walkthroughs/codellama . It should save you a lot of time context
71.
Show HN: Use Code Llama as Drop-In Replacement for Copilot Chat(continue.dev)
187 points by sestinj 3y ago|52 comments
72.
by sestinj 3y ago
We have the user start Ollama themselves on a localhost server, and then can just add ``` models=Models( default=Ollama(model="llama2") ) ``` to the Continue config file. We'll then connect to the Ollama server, so it doesn
73.
by sestinj 3y ago
Yeah, this seems very possible—it will be interesting to see where this goes if the cost of RLHF decreases or, even better, people can choose from a number of RLHF datasets and composably apply them to get their preferred model. And true th
74.
by sestinj 3y ago
Nice! I've been trying out both models for coding (using Ollama + http://github.com/continuedev/continue - disclaimer, author of Continue), and I have to say, it feels like "alignment tax" is real. Uncen
75.
by sestinj 3y ago
Excited to hear all of this! (and appreciate the extensive thinking) Please don't be shy about asking questions in our Discord if you end up wanting to contribute or have more feedback :)
76.
by sestinj 3y ago
Hey there. I realized that the documentation is less clear than it should be on this, so I've made an update: https://continue.dev/docs/customization#local-models-with-gg... The additional thing you need to do is
77.
by sestinj 3y ago
We were actually doing this a couple weeks ago, but due to limitations of VS Code, it was slightly buggy (issues mainly around knowing the terminal window size when setting up a pseudo-terminal). That work still exists here, and we hope to
78.
by sestinj 3y ago
Great question! We plan to make money by helping organizations continuously collect, analyze, and train on their development data. This will let them a) understand ROI of using LLMs for coding and b) use models that are constantly improving
79.
by sestinj 3y ago
We've taken care to decouple our product from VS Code and plan on Jetbrains next, though not for about another month at least
80.
by sestinj 3y ago
I'm very interested to hear this. We are definitely looking to evoke the former.
81.
by sestinj 3y ago
One example is displaying a side-by-side diff as we stream an edit
82.
by sestinj 3y ago
If you mean Meilisearch, we use it to provide a snappy dropdown search experience, like here: https://youtu.be/3Ocrc-WX4iQ?t=86 We chose Meilisearch because it works entirely on your machine (now that analytics are turned o
83.
by sestinj 3y ago
No longer: https://github.com/continuedev/continue/commit/8db5b39170229... Definitely not intending this type of behavior. We want to do everything to keep Continue completely private.
84.
by sestinj 3y ago
It is now :) https://open-vsx.org/extension/Continue/continue
85.
by sestinj 3y ago
Thanks for sharing! We've been recommending something similar with these llama-cpp-python bindings https://github.com/abetlen/llama-cpp-python#web-server , but I'll have to check that out
86.
by sestinj 3y ago
This is fantastic thinking. Actually I'm quite excited about the combination of fault-localization techniques and LLMs; taking a traceback, pruning the call-graph, and seeking out bugs. There's a real future in all of this
87.
by sestinj 3y ago
By default it uses the gpt4 API, but you can optionally use any language model you'd like, including a local model. Can read more here: https://continue.dev/docs/customization#change-the-default-l...
88.
by sestinj 3y ago
Couldn't agree more—it's worth the extra effort to know exactly what enters the prompt. But control isn't mutually exclusive with removing the need to copy/paste. Continue lets you highlight code with perfect precision
89.
by sestinj 3y ago
This is a good catch, thank you
90.
by sestinj 3y ago
This is fair feedback. We will move to packaging it with our extension to avoid running arbitrary code.

More ›