12 ms·

Show HN: Tabby – A self-hosted GitHub Copilot

627 points by wsxiaoys 4y ago

I would like to introduce Tabby, which is a self-hosted alternative to GitHub Copilot that you can integrate into your hardware. While GitHub Copilot has made coding more efficient and less time-consuming by assisting developers with suggestions and completing code, it raises concerns around privacy and security. Tabby is in its early stages, and we are excited to receive feedback from the community. Its Github repository is located here: https://github.com/TabbyML/tabby https://github.com/TabbyML/tabby. We have also deployed the latest docker image to Huggingface for a live demo: https://huggingface.co/spaces/TabbyML/tabby https://huggingface.co/spaces/TabbyML/tabby. Tabby is built on top of the popular Hugging Face Transformers / Triton FasterTransformer backend and is designed to be self-hosted, providing you with complete control over your data and privacy. In Tabby's next feature iteration, you can fine-tune the model to meet your project requirements.

nathancahill 4y ago
This looks exactly like what I need for a project I've been working on. How do you get your own code in to the model? Or is that the future fine-tuning step you're talking about?
- wsxiaoys 4y ago
  Tabby collected the source code from the related repository and built them into a dataset. I already have some proof-of-concept pipelines built [1] and [2], but I still need some time to polish the data pipeline.. [1]: https://github.com/TabbyML/tabby/blob/main/tabby/tools/repository/updater.py https://github.com/TabbyML/tabby/blob/main/tabby/tools/repos... [2]: https://github.com/TabbyML/tabby/blob/main/tabby/tools/trainer/train_lora.py https://github.com/TabbyML/tabby/blob/main/tabby/tools/train...
  - nathancahill 4y ago
    Ok, looks like that makes sense! Is there a prompt token limit like OpenAI models? Codex I believe has a 8k prompt/response limit.
  - skerit 4y ago
    You can train this on all the code in your own repositories? I would assume that makes the completions a lot better?
myin 4y ago
Great to have a self-hosted solution, both for data privacy and quality improvement potentials.
GartzenDeHaes 4y ago
After using Github Copilot for a couple of weeks, it doesn't seem to do much other than Stackoverflow/blog copy paste. Does anyone get much else from it when writing non-boilerplate, non-tutorial, and non-stackoverflow types of code?
- akrymski 4y ago
  I actually still prefer SO as I get to see different solutions to the same problem and related discussion. Co-pilot is great for boilerplate stuff though, which is the majority of lines for most projects these days I'd guess.
  - hn_user2 4y ago
    If you like seeing options, you might be interested in having the Copilot pane open while you code if you are using it. It shows usually about 4 solutions in the panel that it is thinking about for autocomplete, and you can click which one you want. Not necessarily better than SO but I find it nice to see options.
- michaelmior 4y ago
  I haven't used Copilot recently, but I have been using Codium[0] which is similar. I find it often does a reasonably good job at completing similar structures in my own projects where I can't imagine it has seen similar examples. It doesn't always do a great job, but it does save me time. [0] https://codeium.com/ https://codeium.com/
  - dalmo3 4y ago
    I migrated from copilot to codeium as I found it much faster and just as smart. Not sure if the speedup comes from network latency (I'm in New Zealand) or actual model calculations. That was pre GPT4 though, and I'm willing to try copilot again soon.
- phamilton 4y ago
  My favorite usage of it is when writing database migrations. I'll write my `up` case and then it will write the `down` case for me. For migrations adding multiple columns, indexes, triggers, etc. it's good about getting the right order for the down case and generating the right commands. Is it writing meaningful code for me? Nah. Is it helpful? Certainly.
akrymski 4y ago
What LLM is this using? Or did you train your own?
- wsxiaoys 4y ago
  Tabby is tested against various open-source models based on GPT-J/GPT-NeoX, including https://huggingface.co/CarperAI/FIM-NeoX-1.3B https://huggingface.co/CarperAI/FIM-NeoX-1.3B and https://huggingface.co/Salesforce/codegen-350M-multi https://huggingface.co/Salesforce/codegen-350M-multi. In the meantime, I'm training language-specific base models using NeoX blocks. I hope to release them soon. The Huggingface Space demo is running on a derived model of Salesforce/codegen-350M-multi
mska 4y ago
Out of curiosity, how do companies offering self-hosted/on-prem solutions monetize their offerings? Do they rely on legal contracts to prevent customers from using the software for free or modifying it for their own purposes?
- ushakov 4y ago
  VC money, mostly
- sebzim4500 4y ago
  Presumably the same way as every software company before SaaS took over a few years ago.
- wongarsu 4y ago
  Most of them do it by charging money for the product, just like every other software company in the last four decades. If we are talking about open source self-hosted specifically, it's mostly consulting, paid support, and offering managed hosting.
  - passwordoops 4y ago
    Wait a minute... You mean to seriously say people used to pay a one time fee for software? Next you're going to tell me in this magical Utopia companies allowed you to continue using the same version of their software even after they made updates no one asked for
- bonestamp2 4y ago
  A lot of them use a phone home/license key model... so it will stop working if you don't pay for the license. Where they don't, ya it's basically a trust system and most big companies would rather just pay the license fee than expose the legal risk of using software illegally.
syntaxing 4y ago
What sort of resources do we need to run this, particularly VRAM? Also, how does this compare to Fauxpilot?
- wsxiaoys 4y ago
  Tabby's philosophy is to achieve a completion rate comparable to Codex/Copilot by using a model size of less than 1B, with support for BF16/FP16 that reduces VRAM requirements to 2G or less. This may seem impossible, given that the model size is 10 times smaller than that of Codex, but it is definitely achievable, especially in an on-premises environment where customers want to keep the code behind a firewall. Related research works include [1]. (Hints: combine code search and LLM). [1]: RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation https://arxiv.org/abs/2303.12570 https://arxiv.org/abs/2303.12570 This also reveals Tabby's roadmap beyond other OSS work like Fauxipilot :)
  - moffkalast 4y ago
    > philosophy is to achieve That doesn't answer the question, can anyone without more VRAM than sense actually run it as-is or should we wait until they reach their allegedly impossible aspirational goal? The very first line of this sort of post should be the specs required and if the trained model weights are actually available, otherwise it's just straight up clickbait.
    - nacs 4y ago
      The post says it uses "2GB Or less" of VRAM. A 1B parameter transformer model is on the low/tiny-end of model size these days.
    - sp332 4y ago
      Models are usually trained in fp16, meaning 16 bits = 4 bytes per parameter. So a 1B model would take 4GB to begin with, and can be optimised down from there with sparsification and/or quantization. A 50% reduction in size might be noticeably worse than the original, but still useful for boilerplate or other highly predictable patterns.
DanHulton 4y ago
Copilot has so far been pretty useful to me as a "sometimes smarter intellisense". It'll frequently correctly guess the arguments I want for a function (and their types), and every once in a while I'll type `a.map(` and it'll auto-fill the transformation code I was planning on writing. The simpler the task I'm trying to do, the better chance it has of being correct, but that's also the part where I feel I get the most benefit from it, because I already thoroughly understand exactly what I'm writing, why I'm writing it, and what it needs to look like, and Copilot sometimes saves me the 5-30s it takes to write it. Over a day, that adds up and I can move marginally faster. It's definitely not a 100x improvement (or even a 10x improvement), but I'm glad to have it. If this works as well, locally, to escape the privacy issue, I'll be thrilled. Checking it out.
- japhyr 4y ago
  Yes, I went into all of this thinking the tools need to handle complex code in order to really be useful. But that's the code I want to be most careful about, so it's the code I really want to architect myself, and understand thoroughly. Making the simpler code much faster to develop leaves me a lot more time and focus for the complex work. These tools really are amazing, even with their limitations.
- greenie_beans 4y ago
  i feel the same way, but the `a.map(` autofill is hit or miss for me. it'll interrupt my train of thought and it'll take me a minute to remember what i was trying to do, then i'll have to review the code to make sure it's right. and often there's a better way to do it, so i end up refactoring.
- deleted 4y ago
  [deleted]
- jrockway 4y ago
  Yeah, I think it's pretty good at this. I've also done "write a test like the one above but for Bar instead of Foo" and it got the right answer. It would be better to not duplicate code like that in most cases, but sometimes it's OK. I recently needed a script to grab all my github repos from the github api and produce a CSV with the columns URL, name, description, and it could not do that at all. Just hallucinated libraries and APIs that did not exist.
vikp 4y ago
How does this compare to fauxpilot - https://github.com/fauxpilot/fauxpilot https://github.com/fauxpilot/fauxpilot? Fauxpilot also uses Triton with fastertransformers and GPT-J style models (codegen).
- wsxiaoys 4y ago
  Linked my replies to others' questions: [1], [2] [1] https://news.ycombinator.com/item?id=35471882 https://news.ycombinator.com/item?id=35471882 [2] https://news.ycombinator.com/item?id=35471390 https://news.ycombinator.com/item?id=35471390 I hope these can be of help and answer your questions as well!
lfkdev 4y ago
This exakt name is already used by a big open-source projekt, https://github.com/eugeny/tabby https://github.com/eugeny/tabby (50k stars) maybe consider changing the name for better SEO
- wsxiaoys 4y ago
  Thanks, I'm just so into the puns: Tab-by-ML / TabbyML / Tabby
  - boesboes 4y ago
    love it
simonw 4y ago
Anyone know of a quick workaround for this? % docker run \ -it --rm \ -v ./data:/data \ -v ./data/hf_cache:/home/app/.cache/huggingface \ -p 5000:5000 \ -e MODEL_NAME=TabbyML/J-350M \ tabbyml/tabby Unable to find image 'tabbyml/tabby:latest' locally latest: Pulling from tabbyml/tabby docker: no matching manifest for linux/arm64/v8 in the manifest list entries. See 'docker run --help'. I have an M2 Mac. I believe Docker is capable of running images compiled for different architectures using QEMU style workarounds, but is that something I can do with a one-liner or would I need to build a new image from scratch? Previous experiments with Docker and QEMU: https://til.simonwillison.net/docker/emulate-s390x-with-qemu https://til.simonwillison.net/docker/emulate-s390x-with-qemu
- markng 4y ago
  On my M1, --platform linux/amd64 seems to work (I think I needed to turn rosetta on for this for some other stuff, I don't recall.)
  - andrewmunsell 4y ago
    It doesn't seem to work for me on an M1, even with that flag > OSError: [Errno 38] Function not implemented
  - markng 4y ago
    (that said, I am currently downloading this image, and I've had problems running x86-64 stuff in docker sometimes)
nmstoker 4y ago
Is this serious? There's very little to go on in the repo to establish how well thought through this is. I don't want to mark them down for poor language skills but the style of the comments on the TabbyML GitHub profile suggests a rather casual approach, and when combined with a lack of any serious documentation or even basic details beyond a sketched architecture diagram, I kind of wonder... Is there any particular context others can point to that I may be overlooking?
- nmstoker 4y ago
  And to clarify, I'm not trying to be down on the authors of the repo, it's more a surprise this has got so high on HN without much to support it
  - yieldcrv 4y ago
    the interest is in clients side LLMs and similar technologies this is enthusiast level territory that doesn't require professional window dressing at the moment if you’re the kind of person that needs that then you aren’t the target audience yet
    - nmstoker 4y ago
      I don't need marketing and window dressing, I'm interested in evidence of it genuinely working but there's very little here. The idea of client side LLMs is kind of obviously a winner, who wouldn't want that and who hasn't thought of that, so this shouldn't shoot to the top on that basis. The repo looks close to thrown together vapourware. I'm keen not to undermine their efforts since we all aspire to create great things but the fact this has risen so high with so little supporting evidence says more about the current mood here.
      - yieldcrv 4y ago
        its useful client side LLMs that people can potentially package as a commercial offering, and this is an ongoing gap in the market So all sorts of vaporware will fill that, and people will commend other people that seemingly take the initiative instead of just thinking about it
jslakro 4y ago
Considering it's an alpha version I think a VSCode extension is the missing part
- wsxiaoys 4y ago
  There it is: https://marketplace.visualstudio.com/items?itemName=TabbyML.vscode-tabby https://marketplace.visualstudio.com/items?itemName=TabbyML....
wongarsu 4y ago
Trying the demo I got "value is not a valid enumeration member; permitted: 'unknown', 'python', 'javascript'". Trying some clearly identifiable Rust code with language set to 'unknown' I got a completion in Java back. The completion made sense, was properly indented and syntactically correct, it just was the wrong programming language. Is this a limitation of the hosted demo or the chosen model, or do I simply have to wait a bit until my favorite niche language is supported?
uglycoyote 4y ago
I'm confused about the premise here. The power of self-hosting such a thing is presumably that you would be able to train it on your own company's codebase as a corpus of examples to help other people in the company know how to navigate the specifics of your codebase. But there's nothing in the introductory materials about how to train this thing.
- RhodesianHunter 4y ago
  Many of these that I've seen are more focused on keeping company data private than on training the model(s) on company data. Which is pretty entertaining given how many of these companies host their code on Github already.
- sp332 4y ago
  Sometimes it's just about avoiding boilerplate. But the last sentence of the post here says that fine-tuning is coming in the next release.
- PantaloonFlames 4y ago
  Yes, how could I train it on my codebase? Does the team acknowledge this is a thing people would want to do? Or am I misunderstanding the idea here?
  - bonestamp2 4y ago
    I'm not sure either, but I guess one benefit of self hosting is that you skip the $10/mo fee. I don't mind paying for it, but I know I'm not everyone.
- dror 4y ago
  Looking at https://d1muf25xaso8hp.cloudfront.net/https%3A%2F%2Fs3.amazonaws.com%2Fappforest_uf%2Ff1676594147234x637157223107334500%2Fdownload.gif?w=768&h=&auto=compress&dpr=1.25&fit=max https://d1muf25xaso8hp.cloudfront.net/https%3A%2F%2Fs3.amazo... it looks like it's figures out you're using javascript and trains itself on all the libraries you install. I'm guessing there's a way to ask it to look at your specific code as well, but haven't dug into it yet.
moonchrome 4y ago
Anyone dare to guess how fast would GH copilot be if it ran locally ? My main problem with copilot is latency/speed - I would shell out for 4090 if it meant I could use local copilot model that's super fast/low latency/explores deep suggestions.
- atq2119 4y ago
  One of the weaknesses of transformers is that inference is inherently serial, one token at a time, and the entire model needs to be read once per token. This means that inference (after prompt processing) is bounded by memory bandwidth instead of compute. That said, local solutions always tend to be lower latency than cloud ones just because you get to skip the network.
arco1991 4y ago
Very interesting to have a self-hosted version of Copilot. Will definitely try this out.
covi 4y ago
This is awesome, and glad to see that SkyPilot is useful in distributing Tabby on any cloud! https://github.com/TabbyML/tabby/blob/main/deployment/skypilot/README.md https://github.com/TabbyML/tabby/blob/main/deployment/skypil...
zoba 4y ago
My assumption is that this would not be fast enough for practical use on M1/M2 Macbooks. Is that correct?
grudg3 4y ago
Hi, will this work on AMD Gpu? I have plenty of VRAM available
ROFISH 4y ago
Does this come pre-trained, or does it require training? What sources of data are you training on if so? I restrict all usage of AI tools trained from publicly-sourced data because of an unknown copyright restriction, general unease, and lawsuits; however if this can be trained solely on my own codebases that are of clean providence, I can be 100% guaranteed against potential lawsuits. Copilot is a cool tool, but super scary from a legal perspective. And even more heavily regulated industries (that I'm not in) would absolutely need their own firewalled version.
the42thdoctor 4y ago
If not for the wonderful product itself, take my upvote for the "hire" page. It's the first time I see a company presenting a formula where you can input your data and figure out how much you would get paid. Nice! > https://tabbyml.notion.site/Compensation-Sheet-ad61218889ab47249fe35d85fc3c0586 https://tabbyml.notion.site/Compensation-Sheet-ad61218889ab4...
- wsxiaoys 4y ago
  Thank you, I was inspired by the brilliant ideas presented in the employer handbooks of Posthog and GitLab, as acknowledged in the career page.
  - redrove 4y ago
    >location-based pay Good to know they put that out there so people can avoid them.
    - deleted 4y ago
      [deleted]
- brobinson 4y ago
  And 10 years to exercise ISOs after departure. Very nice.
boringuser2 4y ago
It's kinda funny that the same people knocking GPT-3 copilot are going to be losing their absolute minds when copilot X drops. GPT-4 can autogenerate most code a business needs, just need a lone engineer to keep it in check.
- nextworddev 4y ago
  A lone "senior" engineer could, maybe, but definitely not someone who hasn't worked in the problem space
  - boringuser2 4y ago
    Uhh okay that's every dev team downsized to one person.
shaunxcode 4y ago
Do any of the authors of these models disclose a clear list of codebases/discussion sites these models are trained on? Are they at the least avoiding stuff clearly GNU (or such) licensed? Is the hope that there is just no way to figure it out?
unosama 4y ago
I like to use my brain as my copilot
xvilka 4y ago
I hope someone comes with the AI-powered refactoring. I think it has more practical applications than Copilot, at least from my experience in C and C++ test runs of it.
- kateklink 4y ago
  hi! We've actually done it at Refact.ai, would be glad to hear your feedback.
  - xvilka 4y ago
    My two biggest complains from the first sight are: 1) No support for the plain C 2) No support for Vim/NeoVim
Nic0 4y ago
Just in case you didn't know that a project called Tabby exists (it was Terminus). It's a terminal (another one you could say). It's not my project, I'm just a user. https://tabby.sh/ https://tabby.sh/
deleted 4y ago
[deleted]
testing_hn 4y ago
[dead]
funnyfoobar 4y ago
I think that this is a really cool project. Personally, I am working at a financial institution which is regulated, and the legal team + cyber are still evaluating if there would be any problems that could arise with CoPilot. The way the Europe is heading towards AI regulation, Italy has already banned chatGPT. It seems likely that there would be a threat to Copilot as well in the Europe. So these kind of solutions would make a ton of sense wrt organisations adopting this. However as a developer who has used copilot before, here are my 2 cents: The copilot makes a lot of sense, wrt auto completion and code generation, because it understands the current context of code. It would be a friction for the developers if we are expecting them to use the user interface this project comes with. It would be great for this project to go in the direction of the following: - Ability for the developers to train the model with our custom projects, so it can give suggestions wrt to our style of coding - Extensions for popular Editors like VIM, VS Code, IntelliJ etc Happy to share more further info if needed :)
hddqsb 4y ago
I'm glad to see projects like this. I had a look at the demo (https://huggingface.co/spaces/TabbyML/tabby https://huggingface.co/spaces/TabbyML/tabby) and wasn't too impressed with the generated code for the default sample prompt (binary search) -- it recurses infinitely if the item is missing. It would be interesting to compare with Copilot's output. No idea how one would go about fixing this (other than manually add a correct binary search implementation to the training data, which feels like cheating). Request to https://tabbyml-tabby.hf.space/v1/completions https://tabbyml-tabby.hf.space/v1/completions: { "language": "python", "prompt": "def binarySearch(arr, left, right, x):\n mid = (left +" } Response: { "id": "cmpl-...", "created": 1680867355, "choices": [ { "index": 0, "text": " right) >> 1\n if x < arr[mid]:\n return binarySearch(arr, left, mid - 1, x)\n elif x > arr[mid]:\n return binarySearch(arr, mid + 1, right, x)\n else:\n return mid" } ] } Formatted code: def binarySearch(arr, left, right, x): mid = (left + right) >> 1 if x < arr[mid]: return binarySearch(arr, left, mid - 1, x) elif x > arr[mid]: return binarySearch(arr, mid + 1, right, x) else: return mid Manually written test cases: arr = [1, 3, 5, 7] print(binarySearch(arr, 0, len(arr), 5)) # 2 (correct) print(binarySearch(arr, 0, len(arr), 4)) # RecursionError Runnable demo: https://tio.run/##lY/BCoJAEIbvPsUPXVw0yCyCKG@9QB3Fg@maC7bKuII@vc1qdJGg5rIL83/fzDSDKWsdjmMuC9yVTmm4yZSy0k2JfFSyMD5IPUp@enF0wPVUOc5wbQ/e3BSIIgRTVxXocQLjMQeTGbFF0nSkvw6x1jUCO2ZCZDWZon9M1uFZx2fjt6qVC5qzjsMUnxIzEfrY@zgkTkNKG3fp3thFtf0LjgqBFbZws5pIZkb8hu0m7CqzjlpV6wtRTeP4Ag https://tio.run/##lY/BCoJAEIbvPsUPXVw0yCyCKG@9QB3Fg@maC7bKuI...
accelbred 4y ago
Is AMD gpu support on the roadmap? I'm interested in this stuff but I won't be buying Nvidia cards.

627 points by wsxiaoys 4y ago

nathancahill 4y ago

This looks exactly like what I need for a project I've been working on. How do you get your own code in to the model? Or is that the future fine-tuning step you're talking about?

wsxiaoys 4y ago
Tabby collected the source code from the related repository and built them into a dataset. I already have some proof-of-concept pipelines built [1] and [2], but I still need some time to polish the data pipeline.. [1]: https://github.com/TabbyML/tabby/blob/main/tabby/tools/repository/updater.py https://github.com/TabbyML/tabby/blob/main/tabby/tools/repos... [2]: https://github.com/TabbyML/tabby/blob/main/tabby/tools/trainer/train_lora.py https://github.com/TabbyML/tabby/blob/main/tabby/tools/train...
- nathancahill 4y ago
  Ok, looks like that makes sense! Is there a prompt token limit like OpenAI models? Codex I believe has a 8k prompt/response limit.
- skerit 4y ago
  You can train this on all the code in your own repositories? I would assume that makes the completions a lot better?

myin 4y ago

Great to have a self-hosted solution, both for data privacy and quality improvement potentials.

GartzenDeHaes 4y ago

After using Github Copilot for a couple of weeks, it doesn't seem to do much other than Stackoverflow/blog copy paste. Does anyone get much else from it when writing non-boilerplate, non-tutorial, and non-stackoverflow types of code?

akrymski 4y ago
I actually still prefer SO as I get to see different solutions to the same problem and related discussion. Co-pilot is great for boilerplate stuff though, which is the majority of lines for most projects these days I'd guess.
- hn_user2 4y ago
  If you like seeing options, you might be interested in having the Copilot pane open while you code if you are using it. It shows usually about 4 solutions in the panel that it is thinking about for autocomplete, and you can click which one you want. Not necessarily better than SO but I find it nice to see options.
michaelmior 4y ago
I haven't used Copilot recently, but I have been using Codium[0] which is similar. I find it often does a reasonably good job at completing similar structures in my own projects where I can't imagine it has seen similar examples. It doesn't always do a great job, but it does save me time. [0] https://codeium.com/ https://codeium.com/
- dalmo3 4y ago
  I migrated from copilot to codeium as I found it much faster and just as smart. Not sure if the speedup comes from network latency (I'm in New Zealand) or actual model calculations. That was pre GPT4 though, and I'm willing to try copilot again soon.
phamilton 4y ago
My favorite usage of it is when writing database migrations. I'll write my `up` case and then it will write the `down` case for me. For migrations adding multiple columns, indexes, triggers, etc. it's good about getting the right order for the down case and generating the right commands. Is it writing meaningful code for me? Nah. Is it helpful? Certainly.

akrymski 4y ago

What LLM is this using? Or did you train your own?

wsxiaoys 4y ago
Tabby is tested against various open-source models based on GPT-J/GPT-NeoX, including https://huggingface.co/CarperAI/FIM-NeoX-1.3B https://huggingface.co/CarperAI/FIM-NeoX-1.3B and https://huggingface.co/Salesforce/codegen-350M-multi https://huggingface.co/Salesforce/codegen-350M-multi. In the meantime, I'm training language-specific base models using NeoX blocks. I hope to release them soon. The Huggingface Space demo is running on a derived model of Salesforce/codegen-350M-multi

mska 4y ago

Out of curiosity, how do companies offering self-hosted/on-prem solutions monetize their offerings? Do they rely on legal contracts to prevent customers from using the software for free or modifying it for their own purposes?

ushakov 4y ago
VC money, mostly
sebzim4500 4y ago
Presumably the same way as every software company before SaaS took over a few years ago.
wongarsu 4y ago
Most of them do it by charging money for the product, just like every other software company in the last four decades. If we are talking about open source self-hosted specifically, it's mostly consulting, paid support, and offering managed hosting.
- passwordoops 4y ago
  Wait a minute... You mean to seriously say people used to pay a one time fee for software? Next you're going to tell me in this magical Utopia companies allowed you to continue using the same version of their software even after they made updates no one asked for
bonestamp2 4y ago
A lot of them use a phone home/license key model... so it will stop working if you don't pay for the license. Where they don't, ya it's basically a trust system and most big companies would rather just pay the license fee than expose the legal risk of using software illegally.

syntaxing 4y ago

What sort of resources do we need to run this, particularly VRAM? Also, how does this compare to Fauxpilot?

wsxiaoys 4y ago
Tabby's philosophy is to achieve a completion rate comparable to Codex/Copilot by using a model size of less than 1B, with support for BF16/FP16 that reduces VRAM requirements to 2G or less. This may seem impossible, given that the model size is 10 times smaller than that of Codex, but it is definitely achievable, especially in an on-premises environment where customers want to keep the code behind a firewall. Related research works include [1]. (Hints: combine code search and LLM). [1]: RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation https://arxiv.org/abs/2303.12570 https://arxiv.org/abs/2303.12570 This also reveals Tabby's roadmap beyond other OSS work like Fauxipilot :)
- moffkalast 4y ago
  > philosophy is to achieve That doesn't answer the question, can anyone without more VRAM than sense actually run it as-is or should we wait until they reach their allegedly impossible aspirational goal? The very first line of this sort of post should be the specs required and if the trained model weights are actually available, otherwise it's just straight up clickbait.
  - nacs 4y ago
    The post says it uses "2GB Or less" of VRAM. A 1B parameter transformer model is on the low/tiny-end of model size these days.
  - sp332 4y ago
    Models are usually trained in fp16, meaning 16 bits = 4 bytes per parameter. So a 1B model would take 4GB to begin with, and can be optimised down from there with sparsification and/or quantization. A 50% reduction in size might be noticeably worse than the original, but still useful for boilerplate or other highly predictable patterns.

DanHulton 4y ago

Copilot has so far been pretty useful to me as a "sometimes smarter intellisense". It'll frequently correctly guess the arguments I want for a function (and their types), and every once in a while I'll type `a.map(` and it'll auto-fill the transformation code I was planning on writing. The simpler the task I'm trying to do, the better chance it has of being correct, but that's also the part where I feel I get the most benefit from it, because I already thoroughly understand exactly what I'm writing, why I'm writing it, and what it needs to look like, and Copilot sometimes saves me the 5-30s it takes to write it. Over a day, that adds up and I can move marginally faster. It's definitely not a 100x improvement (or even a 10x improvement), but I'm glad to have it. If this works as well, locally, to escape the privacy issue, I'll be thrilled. Checking it out.

japhyr 4y ago
Yes, I went into all of this thinking the tools need to handle complex code in order to really be useful. But that's the code I want to be most careful about, so it's the code I really want to architect myself, and understand thoroughly. Making the simpler code much faster to develop leaves me a lot more time and focus for the complex work. These tools really are amazing, even with their limitations.
greenie_beans 4y ago
i feel the same way, but the `a.map(` autofill is hit or miss for me. it'll interrupt my train of thought and it'll take me a minute to remember what i was trying to do, then i'll have to review the code to make sure it's right. and often there's a better way to do it, so i end up refactoring.
deleted 4y ago
[deleted]
jrockway 4y ago
Yeah, I think it's pretty good at this. I've also done "write a test like the one above but for Bar instead of Foo" and it got the right answer. It would be better to not duplicate code like that in most cases, but sometimes it's OK. I recently needed a script to grab all my github repos from the github api and produce a CSV with the columns URL, name, description, and it could not do that at all. Just hallucinated libraries and APIs that did not exist.

vikp 4y ago

How does this compare to fauxpilot - https://github.com/fauxpilot/fauxpilot https://github.com/fauxpilot/fauxpilot? Fauxpilot also uses Triton with fastertransformers and GPT-J style models (codegen).

wsxiaoys 4y ago
Linked my replies to others' questions: [1], [2] [1] https://news.ycombinator.com/item?id=35471882 https://news.ycombinator.com/item?id=35471882 [2] https://news.ycombinator.com/item?id=35471390 https://news.ycombinator.com/item?id=35471390 I hope these can be of help and answer your questions as well!

lfkdev 4y ago

This exakt name is already used by a big open-source projekt, https://github.com/eugeny/tabby https://github.com/eugeny/tabby (50k stars) maybe consider changing the name for better SEO

wsxiaoys 4y ago
Thanks, I'm just so into the puns: Tab-by-ML / TabbyML / Tabby
- boesboes 4y ago
  love it

simonw 4y ago

Anyone know of a quick workaround for this? % docker run \ -it --rm \ -v ./data:/data \ -v ./data/hf_cache:/home/app/.cache/huggingface \ -p 5000:5000 \ -e MODEL_NAME=TabbyML/J-350M \ tabbyml/tabby Unable to find image 'tabbyml/tabby:latest' locally latest: Pulling from tabbyml/tabby docker: no matching manifest for linux/arm64/v8 in the manifest list entries. See 'docker run --help'. I have an M2 Mac. I believe Docker is capable of running images compiled for different architectures using QEMU style workarounds, but is that something I can do with a one-liner or would I need to build a new image from scratch? Previous experiments with Docker and QEMU: https://til.simonwillison.net/docker/emulate-s390x-with-qemu https://til.simonwillison.net/docker/emulate-s390x-with-qemu

markng 4y ago
On my M1, --platform linux/amd64 seems to work (I think I needed to turn rosetta on for this for some other stuff, I don't recall.)
- andrewmunsell 4y ago
  It doesn't seem to work for me on an M1, even with that flag > OSError: [Errno 38] Function not implemented
- markng 4y ago
  (that said, I am currently downloading this image, and I've had problems running x86-64 stuff in docker sometimes)

nmstoker 4y ago

Is this serious? There's very little to go on in the repo to establish how well thought through this is. I don't want to mark them down for poor language skills but the style of the comments on the TabbyML GitHub profile suggests a rather casual approach, and when combined with a lack of any serious documentation or even basic details beyond a sketched architecture diagram, I kind of wonder... Is there any particular context others can point to that I may be overlooking?

nmstoker 4y ago
And to clarify, I'm not trying to be down on the authors of the repo, it's more a surprise this has got so high on HN without much to support it
- yieldcrv 4y ago
  the interest is in clients side LLMs and similar technologies this is enthusiast level territory that doesn't require professional window dressing at the moment if you’re the kind of person that needs that then you aren’t the target audience yet
  - nmstoker 4y ago
    I don't need marketing and window dressing, I'm interested in evidence of it genuinely working but there's very little here. The idea of client side LLMs is kind of obviously a winner, who wouldn't want that and who hasn't thought of that, so this shouldn't shoot to the top on that basis. The repo looks close to thrown together vapourware. I'm keen not to undermine their efforts since we all aspire to create great things but the fact this has risen so high with so little supporting evidence says more about the current mood here.
    - yieldcrv 4y ago
      its useful client side LLMs that people can potentially package as a commercial offering, and this is an ongoing gap in the market So all sorts of vaporware will fill that, and people will commend other people that seemingly take the initiative instead of just thinking about it

jslakro 4y ago

Considering it's an alpha version I think a VSCode extension is the missing part

wsxiaoys 4y ago
There it is: https://marketplace.visualstudio.com/items?itemName=TabbyML.vscode-tabby https://marketplace.visualstudio.com/items?itemName=TabbyML....

wongarsu 4y ago

Trying the demo I got "value is not a valid enumeration member; permitted: 'unknown', 'python', 'javascript'". Trying some clearly identifiable Rust code with language set to 'unknown' I got a completion in Java back. The completion made sense, was properly indented and syntactically correct, it just was the wrong programming language. Is this a limitation of the hosted demo or the chosen model, or do I simply have to wait a bit until my favorite niche language is supported?

uglycoyote 4y ago

I'm confused about the premise here. The power of self-hosting such a thing is presumably that you would be able to train it on your own company's codebase as a corpus of examples to help other people in the company know how to navigate the specifics of your codebase. But there's nothing in the introductory materials about how to train this thing.

RhodesianHunter 4y ago
Many of these that I've seen are more focused on keeping company data private than on training the model(s) on company data. Which is pretty entertaining given how many of these companies host their code on Github already.
sp332 4y ago
Sometimes it's just about avoiding boilerplate. But the last sentence of the post here says that fine-tuning is coming in the next release.
PantaloonFlames 4y ago
Yes, how could I train it on my codebase? Does the team acknowledge this is a thing people would want to do? Or am I misunderstanding the idea here?
- bonestamp2 4y ago
  I'm not sure either, but I guess one benefit of self hosting is that you skip the $10/mo fee. I don't mind paying for it, but I know I'm not everyone.
dror 4y ago
Looking at https://d1muf25xaso8hp.cloudfront.net/https%3A%2F%2Fs3.amazonaws.com%2Fappforest_uf%2Ff1676594147234x637157223107334500%2Fdownload.gif?w=768&h=&auto=compress&dpr=1.25&fit=max https://d1muf25xaso8hp.cloudfront.net/https%3A%2F%2Fs3.amazo... it looks like it's figures out you're using javascript and trains itself on all the libraries you install. I'm guessing there's a way to ask it to look at your specific code as well, but haven't dug into it yet.

moonchrome 4y ago

Anyone dare to guess how fast would GH copilot be if it ran locally ? My main problem with copilot is latency/speed - I would shell out for 4090 if it meant I could use local copilot model that's super fast/low latency/explores deep suggestions.

atq2119 4y ago
One of the weaknesses of transformers is that inference is inherently serial, one token at a time, and the entire model needs to be read once per token. This means that inference (after prompt processing) is bounded by memory bandwidth instead of compute. That said, local solutions always tend to be lower latency than cloud ones just because you get to skip the network.

arco1991 4y ago

Very interesting to have a self-hosted version of Copilot. Will definitely try this out.

covi 4y ago

This is awesome, and glad to see that SkyPilot is useful in distributing Tabby on any cloud! https://github.com/TabbyML/tabby/blob/main/deployment/skypilot/README.md https://github.com/TabbyML/tabby/blob/main/deployment/skypil...

zoba 4y ago

My assumption is that this would not be fast enough for practical use on M1/M2 Macbooks. Is that correct?

grudg3 4y ago

Hi, will this work on AMD Gpu? I have plenty of VRAM available

ROFISH 4y ago

Does this come pre-trained, or does it require training? What sources of data are you training on if so? I restrict all usage of AI tools trained from publicly-sourced data because of an unknown copyright restriction, general unease, and lawsuits; however if this can be trained solely on my own codebases that are of clean providence, I can be 100% guaranteed against potential lawsuits. Copilot is a cool tool, but super scary from a legal perspective. And even more heavily regulated industries (that I'm not in) would absolutely need their own firewalled version.

the42thdoctor 4y ago

If not for the wonderful product itself, take my upvote for the "hire" page. It's the first time I see a company presenting a formula where you can input your data and figure out how much you would get paid. Nice! > https://tabbyml.notion.site/Compensation-Sheet-ad61218889ab47249fe35d85fc3c0586 https://tabbyml.notion.site/Compensation-Sheet-ad61218889ab4...

wsxiaoys 4y ago
Thank you, I was inspired by the brilliant ideas presented in the employer handbooks of Posthog and GitLab, as acknowledged in the career page.
- redrove 4y ago
  >location-based pay Good to know they put that out there so people can avoid them.
  - deleted 4y ago
    [deleted]
brobinson 4y ago
And 10 years to exercise ISOs after departure. Very nice.

boringuser2 4y ago

It's kinda funny that the same people knocking GPT-3 copilot are going to be losing their absolute minds when copilot X drops. GPT-4 can autogenerate most code a business needs, just need a lone engineer to keep it in check.

nextworddev 4y ago
A lone "senior" engineer could, maybe, but definitely not someone who hasn't worked in the problem space
- boringuser2 4y ago
  Uhh okay that's every dev team downsized to one person.

shaunxcode 4y ago

Do any of the authors of these models disclose a clear list of codebases/discussion sites these models are trained on? Are they at the least avoiding stuff clearly GNU (or such) licensed? Is the hope that there is just no way to figure it out?

unosama 4y ago

I like to use my brain as my copilot

xvilka 4y ago

I hope someone comes with the AI-powered refactoring. I think it has more practical applications than Copilot, at least from my experience in C and C++ test runs of it.

kateklink 4y ago
hi! We've actually done it at Refact.ai, would be glad to hear your feedback.
- xvilka 4y ago
  My two biggest complains from the first sight are: 1) No support for the plain C 2) No support for Vim/NeoVim

Nic0 4y ago

Just in case you didn't know that a project called Tabby exists (it was Terminus). It's a terminal (another one you could say). It's not my project, I'm just a user. https://tabby.sh/ https://tabby.sh/

deleted 4y ago

[deleted]

testing_hn 4y ago

[dead]

funnyfoobar 4y ago

I think that this is a really cool project. Personally, I am working at a financial institution which is regulated, and the legal team + cyber are still evaluating if there would be any problems that could arise with CoPilot. The way the Europe is heading towards AI regulation, Italy has already banned chatGPT. It seems likely that there would be a threat to Copilot as well in the Europe. So these kind of solutions would make a ton of sense wrt organisations adopting this. However as a developer who has used copilot before, here are my 2 cents: The copilot makes a lot of sense, wrt auto completion and code generation, because it understands the current context of code. It would be a friction for the developers if we are expecting them to use the user interface this project comes with. It would be great for this project to go in the direction of the following: - Ability for the developers to train the model with our custom projects, so it can give suggestions wrt to our style of coding - Extensions for popular Editors like VIM, VS Code, IntelliJ etc Happy to share more further info if needed :)

hddqsb 4y ago

I'm glad to see projects like this. I had a look at the demo (https://huggingface.co/spaces/TabbyML/tabby https://huggingface.co/spaces/TabbyML/tabby) and wasn't too impressed with the generated code for the default sample prompt (binary search) -- it recurses infinitely if the item is missing. It would be interesting to compare with Copilot's output. No idea how one would go about fixing this (other than manually add a correct binary search implementation to the training data, which feels like cheating). Request to https://tabbyml-tabby.hf.space/v1/completions https://tabbyml-tabby.hf.space/v1/completions: { "language": "python", "prompt": "def binarySearch(arr, left, right, x):\n mid = (left +" } Response: { "id": "cmpl-...", "created": 1680867355, "choices": [ { "index": 0, "text": " right) >> 1\n if x < arr[mid]:\n return binarySearch(arr, left, mid - 1, x)\n elif x > arr[mid]:\n return binarySearch(arr, mid + 1, right, x)\n else:\n return mid" } ] } Formatted code: def binarySearch(arr, left, right, x): mid = (left + right) >> 1 if x < arr[mid]: return binarySearch(arr, left, mid - 1, x) elif x > arr[mid]: return binarySearch(arr, mid + 1, right, x) else: return mid Manually written test cases: arr = [1, 3, 5, 7] print(binarySearch(arr, 0, len(arr), 5)) # 2 (correct) print(binarySearch(arr, 0, len(arr), 4)) # RecursionError Runnable demo: https://tio.run/##lY/BCoJAEIbvPsUPXVw0yCyCKG@9QB3Fg@maC7bKuII@vc1qdJGg5rIL83/fzDSDKWsdjmMuC9yVTmm4yZSy0k2JfFSyMD5IPUp@enF0wPVUOc5wbQ/e3BSIIgRTVxXocQLjMQeTGbFF0nSkvw6x1jUCO2ZCZDWZon9M1uFZx2fjt6qVC5qzjsMUnxIzEfrY@zgkTkNKG3fp3thFtf0LjgqBFbZws5pIZkb8hu0m7CqzjlpV6wtRTeP4Ag https://tio.run/##lY/BCoJAEIbvPsUPXVw0yCyCKG@9QB3Fg@maC7bKuI...

accelbred 4y ago

Is AMD gpu support on the roadmap? I'm interested in this stuff but I won't be buying Nvidia cards.