8 ms·
Tabby: Self-hosted AI coding assistant
- thecal 2y agoUnfortunate name. Can you connect Tabby to the OpenAI-compatible TabbyAPI? https://github.com/theroyallab/tabbyAPI https://github.com/theroyallab/tabbyAPI
- mbernstein 2y agoAt least per Github, the TabbyML project is older than the TabbyAPI project.
- mynameisvlad 2y agoAlso, wildly more popular, to the tune of several magnitudes more forks and stars. If anything, this question should be asked of the TabbyAPI project.
- karolist 2y agoI'm not sure what's going on with TabbyAPI's github metrics, but exl2 quants are very popular among nvidia local LLM crowd and TabbyAPI comes in tons of reddit posts of people using it. Might be just my bubble, not saying they're not accurate, just generally surprised such a useful project has under 1k stars. On the flip side, LLMs will hallucinate about TabbyML if you ask it TabbyAPI related questions, so I'd agree the naming is unfortunate.
- Medox 2y agoI though that Tabby, the ssh client [1], got AI capabilities... [1] https://github.com/Eugeny/tabby https://github.com/Eugeny/tabby
- deleted 2y ago[deleted]
- wsxiaoys 2y agoNever imagined our project would make it to the HN front page on Sunday! Tabby has undergone significant development since its launch two years ago [0]. It is now a comprehensive AI developer platform featuring code completion and a codebase chat, with a team [1] / enterprise focus (SSO, Access Control, User Authentication). Tabby's adopters [2][3] have discovered that Tabby is the only platform providing a fully self-service onboarding experience as an on-prem offering. It also delivers performance that rivals other options in the market. If you're curious, I encourage you to give it a try! [0]: https://www.tabbyml.com https://www.tabbyml.com [1]: https://demo.tabbyml.com/search/how-to-add-an-embedding-api-N8Ay31 https://demo.tabbyml.com/search/how-to-add-an-embedding-api-... [2]: https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ [3]: https://www.linkedin.com/posts/kelvinmu_last-week-i-introduced-two-ai-startups-to-activity-7257051111314911232-C3w2 https://www.linkedin.com/posts/kelvinmu_last-week-i-introduc...
- maille 2y agoDo you have a plugin for MSVC?
- tootie 2y agoIs it only compatible with Nvidia and Apple? Will this work with an AMD GPU?
- wsxiaoys 2y agoYes - AMD GPU is supported through vulkan backend: https://github.com/TabbyML/tabby/releases/tag/v0.23.0 https://github.com/TabbyML/tabby/releases/tag/v0.23.0 https://tabby.tabbyml.com/blog/2024/05/01/vulkan-support/ https://tabby.tabbyml.com/blog/2024/05/01/vulkan-support/
- thih9 2y agoAs someone unfamiliar with local AIs and eager to try, how does the “run tabby in 1 minute”[1] compare to e.g. chatgpt’s free 4o-mini? Can I run that docker command on a medium specced macbook pro and have an AI that is comparably fast and capable? Or are we not there (yet)? Edit: looks like there is a separate page with instructions for macbooks[2] that has more context. > The compute power of M1/M2 is limited and is likely to be sufficient only for individual usage. If you require a shared instance for a team, we recommend considering Docker hosting with CUDA or ROCm. [1]: https://github.com/TabbyML/tabby#run-tabby-in-1-minute https://github.com/TabbyML/tabby#run-tabby-in-1-minute docker run -it --gpus all -p 8080:8080 -v $HOME/.tabby:/data tabbyml/tabby serve --model StarCoder-1B --device cuda --chat-model Qwen2-1.5B-Instruct [2]: https://tabby.tabbyml.com/docs/quick-start/installation/apple/ https://tabby.tabbyml.com/docs/quick-start/installation/appl...
- eric-burel 2y agoSide question : open source models tend to be less "smart" than private ones, do you intend to compensate by providing a better context (eg query relevant technology docs to feed context)?
- coder543 2y agogpt-4o-mini might not be the best point of reference for what good LLMs can do with code: https://aider.chat/docs/leaderboards/#aider-polyglot-benchmark-results https://aider.chat/docs/leaderboards/#aider-polyglot-benchma... A teeny tiny model such as a 1.5B model is really dumb, and not good at interactively generating code in a conversational way, but models in the 3B or less size can do a good job of suggesting tab completions. There are larger "open" models (in the 32B - 70B range) that you can run locally that should be much, much better than gpt-4o-mini at just about everything, including writing code. For a few examples, llama3.3-70b-instruct and qwen2.5-coder-32b-instruct are pretty good. If you're really pressed for RAM, qwen2.5-coder-7b-instruct or codegemma-7b-it might be okay for some simple things. > medium specced macbook pro medium specced doesn't mean much. How much RAM do you have? Each "B" (billion) of parameters is going to require about 1GB of RAM, as a rule of thumb. (500MB for really heavily quantized models, 2GB for un-quantized models... but, 8-bit quants use 1GB, and that's usually fine.)
- mjrpes 2y agoWhat is the recommended hardware? GPU required? Could this run OK on an older Ryzen APU (Zen 3 with Vega 7 graphics)?
- wsxiaoys 2y agoCheck https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ to see a local setup with 3090.
- mkl 2y agoThat thread doesn't seem to mention hardware. It would be really helpful to just put hardware requirements in the GitHub README.
- coder543 2y agoThe usual bottleneck for self-hosted LLMs is memory bandwidth. It doesn't really matter if there are integrated graphics or not... the models will run at the same (very slow) speed on CPU-only. Macs are only decent for LLMs because Apple has given Apple Silicon unusually high memory bandwidth, but they're still nowhere near as fast as a high-end GPU with extremely fast VRAM. For extremely tiny models like you would use for tab completion, even an old AMD CPU is probably going to do okay.
- mjrpes 2y agoGood to know. It also looks like you can host TabbyML as an on-premise server with docker and serve requests over a private network. Interesting to think that a self-hosted GPU server might become a thing.
- jslakro 2y agoDuplicated https://news.ycombinator.com/item?id=35470915 https://news.ycombinator.com/item?id=35470915
- leke 2y agoSo does this run on your personal machine, or can you install it on a local company server and have everyone in the company connect to it?
- wsxiaoys 2y agoTabby is engineered for team usage, intended to be deployed on a shared server. However, with robust local computing resources, you can also run Tabby on your individual machine. Check https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ to see a local setup with 3090.
- d--b 2y agoDidn’t you mean to name it Spacey?
- szszrk 2y agoI've heard of tab vs spaces flamewars, but never heard of "space completion" camp. It clearly references to LLM doing a tab completion.
- thedangler 2y agoHow would I tell this to use an api framework it doesn’t know ?
- wsxiaoys 2y agoTabby comes with builtin RAG support so you can add this api framework to it. Example: https://demo.tabbyml.com/search/how-to-configure-sso-in-tabby-wbdVW1 https://demo.tabbyml.com/search/how-to-configure-sso-in-tabb... Settings page: https://demo.tabbyml.com/settings/providers/doc https://demo.tabbyml.com/settings/providers/doc
- SOLAR_FIELDS 2y ago> How to utilize multiple NVIDIA GPUs? | Tabby only supports the use of a single GPU. To utilize multiple GPUs, you can initiate multiple Tabby instances and set CUDA_VISIBLE_DEVICES (for cuda) or HIP_VISIBLE_DEVICES (for rocm) accordingly. So using 2 NVLinked GPU's with inference is not supported? Or is that situation different because NVLink treats the two GPU as a single one?
- wsxiaoys 2y ago> So using 2 NVLinked GPU's with inference is not supported? To make better use of multiple GPUs, we suggest employing a dedicated backend for serving the model. Please refer to https://tabby.tabbyml.com/docs/references/models-http-api/vllm/ https://tabby.tabbyml.com/docs/references/models-http-api/vl... for an example
- SOLAR_FIELDS 2y agoI see. So this is like, I can have tabby be my LLM server with this limitation or I can just turn that feature off and point tabby at my self hosted LLM as any other OpenAI compatible endpoint?
- wsxiaoys 2y agoYes - however, the FIM model requires careful configuration to properly set the prompt template.
- KronisLV 2y agoFor something similar I use Continue.dev with ollama, it’s always nice to see more tools in the space! But as usual, you need pretty formidable hardware to run the actually good models, like the 32B version of Qwen2.5-coder.
- laurentpm 2y ago[dead]
- st3fan 2y agoThe demo on the homepage for the completion of the findMaxElement function is a good example of what is to come. Or maybe where we are at now? The six lines of Python suggested for that function can also be replaced with a simple “return max(arr)”. The suggested code works but is absolute junior level. I am terrified of what is to come. Not just horrible code but also how people who blindly “autocomplete” this code are going to stall in their skill level progress. You may score some story points but did you actually get any better at your craft?
- MaKey 2y agoThe silver lining is that the value of your skills is going up.
- shcheklein 2y agoOn the other hand it might become a next level of abstraction. Machine -> Asm -> C -> Python -> LLM (Human language) It compiles human prompt into some intermediate code (in this case Python). Probably initial version of CPython was not perfect at all, and engineers were also terrified. If we are lucky this new "compiler" will be becoming better and better, more efficient. Never perfect, but people will be paying the same price they are already paying for not dealing directly with ASM.
- MVissers 2y agoYup! No goal to become a programmer– But I like to build programs. Build a rather complex AI-ecosystem simulator with me as the director and GPT-4 now Claude 3.5 as the programmer. Would never have been able to do this beforehand.
- saurik 2y agoI think there is a big difference between an abstraction layer that can improve -- one where you maybe write "code" in prompts and then have a compiler build through real code, allowing that compiler to get better over time -- and an interactive tool that locks bad decisions autocompleted today into both your codebase and your brain, involving you still working at the lower layer but getting low quality "help" in your editor. I am totally pro- compilers and high-level languages, but I think the idea of writing assembly with the help of a partial compiler where you kind of write stuff and then copy/paste the result into your assembly file with some munging to fix issues is dumb. By all means, though: if someone gets us to the point where the "code" I am checking in is a bunch of English -- for which I will likely need a law degree in addition to an engineering background to not get evil genie with a cursed paw results from it trying to figure out what I must have meant from what I said :/ -- I will think that's pretty cool and will actually be a new layer of abstraction in the same class as compiler... and like, if at that point I don't use it, it will only be because I think it is somehow dangerous to humanity itself (and even then I will admit that it is probably more effective)... but we aren't there yet and "we're on the way there" doesn't count anywhere near as much as people often want it to ;P.
- mindcrime 2y agoVery cool. I'm especially happy to see that there is an Eclipse client[1]. One note though: I had to dig around a bit to find the info about the Eclipse client. It's not mentioned in the main readme, or in the list of IDE extensions in the docs. Not sure if that's an oversight or because it's not "ready for prime time" yet or what. [1]: https://github.com/TabbyML/tabby/tree/3bd73a8c59a1c21312e8123aa6817d4f2e096041/clients/eclipse https://github.com/TabbyML/tabby/tree/3bd73a8c59a1c21312e812...
- mlepath 2y agoAwesome project! I love the idea of not sending my data to a big company and trust their TOS. The effectiveness of coding assistant is directly proportional to context length and the open models you can run on your computer are usually much smaller. Would love to see something more quantified around the usefulness on more complex codebases.
- fullstackwife 2y agoI hope for proliferation of 100% local coding assistants, but for now the recommendation of "Works best on $10K+ GPU" is a show stopper, and we are forced to use the "big company". :(
- danw1979 2y agoIt’s not really that bad. You can run some fairly big models on an Apple Silicon machine costing £2k (M4 Pro Mac Mini with 64GB RAM).
- trevor-e 2y agofyi the pricing page has a typo for "Singel Sign-On"
- wsxiaoys 2y agoAppreciated! Fixed
- nbzso 2y agoI will go out on a limb and predict that in the next 10 years AI code assistant will be forbidden:)
- qwertox 2y ago> Toggle IDE / Extensions telemetry Cannot be turned off in the Community Edition. What does this telemetry data contain?
- andypants 2y agostruct HealthState { model: String, chat_model: Option<String>, device: String, arch: String, cpu_info: String, cpu_count: usize, cuda_devices: Vec<String>, version: Version, webserver: Option<bool>, } https://tabby.tabbyml.com/docs/administration/usage-collection/ https://tabby.tabbyml.com/docs/administration/usage-collecti...
- nikkwong 2y agoMaybe a good product but terrible company to interview with. I went through several rounds and was basically ghosted after the 4th with no explanation or follow up. The last interview was to write an blog post for their blog which I submitted and then didn’t hear back until continuously nagging months later. It was pretty disheartening since all of the interviews were some form of a take-home and I spent a combined total of ~10 hours or more.
- NetOpWibby 2y agoWere you at least paid?
- swyx 2y agoyou know that paid interview processes are not the norm, "at least" is unlikely
- nikkwong 2y agoIf I was paid, I probably wouldn't be complaining publicly. :-) It's probably better for both interests if these types of engagements are paid.
- fhd2 2y agoI've worked with paid take home tests for a while, but stopped again. Hiring managers started to make the assignments more convoluted, i.e. stopped respecting the candidate's time. Candidates, on the flip side, always said they don't want to bother with the bureaucracy of writing an invoice and reporting it for their taxes etc., so didn't want to be paid. Now my logic is: If a take home test is designed to take more than two hours, we need to redesign it. Two hours of interviews, two hours of take home test, that ought to suffice. If we're still unsure after that, I sometimes offered the candidate a time limited freelance position, paid obviously. We've ended up hiring everyone who went into that process though.
- 2y ago
- chvid 2y agoAll the examples are for code that would otherwise be found in a library. Some of the code is of dubious quality. LLMs - a spam bot for your codebase?
- leke 2y agoI'm currently investigating a self hosted AI solution for my workplace. I was wondering, how does this company make money? From the pricing there is a free/community/opensource option, but how is the "up to 5 users" monitored? https://www.tabbyml.com/pricing https://www.tabbyml.com/pricing * Up to 5 users * Local deployment * Code Completion, Answer Engine, In-line chat & Context Provider What if we have more than 5 users?
- rirze 2y agoAre you asking on a public forum, on how to get around using a product for a commercial setting by using the non-commercial version of the product?
- leke 2y agoI'm saying I don't understand their open source model. I thought open source meant you could use and modify code and run it yourself without having to pay a license. ie completely independent of the maintainer. So I was confused by this limit of how many were allowed to use something you are running yourself.
- baobun 2y agoYou have it wrong. Neither "open source" nor "free software" imply free-of-cost. Paid software with open and free license is very much a thing. If you want to drill into the details of the licenses: https://github.com/TabbyML/tabby/blob/main/LICENSE https://github.com/TabbyML/tabby/blob/main/LICENSE
- SirMaster 2y agoAll these things that claim to be an alternative to GitHub Copilot, none of them seem to work in VS2022... So how is it really an alternative? All I want is a self-hosted AI assistant for VS2022. VS2022 supports plugins yes, so what gives?
- jimmydoe 2y agoNot using VSCode, would be great to have Sublime Text or Zed support.
- larwent 2y agoI’ve been using something similar called Twinny. It’s a vscode extension that connects to an ollama locally hosted LLM of your choice and works like CoPilot. It’s an extra step to install Ollama, so not as plugnplay as tfa but the license is MIT which makes it worthwhile for me. https://github.com/twinnydotdev/twinny https://github.com/twinnydotdev/twinny