5 ms·

Bravo! I'm so glad this is born. I'm curious about its model, SalesForce CodeGen. Does it train on all the public repos on GitHub? Does Copilot have access to p

Bravo! I'm so glad this is born. I'm curious about its model, SalesForce CodeGen. Does it train on all the public repos on GitHub? Does Copilot have access to private repos that CodeGen cannot access? Also, it would be really cool if I can personalize FauxPilot by feeding it with all my repos on GitHub. Sometimes I just need to reimplement a function I've written before but it's really hard to find where my old code is.

pabs3 4y ago
A comment below mentions those details are in the paper: https://news.ycombinator.com/item?id=32328168 https://news.ycombinator.com/item?id=32328168
thejosh 4y ago
Copilot only accesses the private repositories for the user it is authenticated with. I'm fine with them accessing public repositories, as long as they respect the license. Which they probably aren't for private? On a related note, Google apparently treats their free customers in a way where all their data is mixed in for the algorithm, but apparently not the data from accounts who use Google Suite (paid business accounts), instead they have their own recommendations built for them on a per-user basis, then has the public algo mixed in.
- tpxl 4y ago
  > I'm fine with them accessing public repositories, as long as they respect the license. They don't respect the license.
anthropodie 4y ago
> I just need to reimplement a function I've written before There was a project here some time back that allowed to call function using its hash. This avoided replication. It was entirely different paradigm to resolve dependencies. Anyone remembers its name? It was controversial here but it was interesting nonetheless.
- tmp642287479428 4y ago
  Are you talking about Unison? https://www.unison-lang.org/learn/the-big-idea/ https://www.unison-lang.org/learn/the-big-idea/
  - anthropodie 4y ago
    Yeah this is it!
unnouinceput 4y ago
For you and all my sibling comments here. Are you guys forgetting GitHub IS Microsoft? Do you guys actually believe Microsoft is respecting privacy on repositories? Did you read the EULA after the acquisition? C'mon, let's be serious!
moyix 4y ago
It is possible to fine-tune CodeGen using Huggingface Transformers! Then you'd be able to fine-tune it on your own code and use the resulting model. However, training is more expensive -- you'd need an A6000 or better to train the 6B model. Something like the following should work: deepspeed --num_gpus 1 --num_nodes 1 run_clm.py --model_name_or_path=Salesforce/codegen-6B-multi --per_device_train_batch_size=1 --learning_rate 2e-5 --num_train_epochs 1 --output_dir=./codegen-6B-finetuned --dataset_name your_dataset --tokenizer_name Salesforce/codegen-6B-multi --block_size 2048 --gradient_accumulation_steps 32 --do_train --fp16 --overwrite_output_dir --deepspeed ds_config.json Where run_clm.py is this script: https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py https://github.com/huggingface/transformers/blob/main/exampl... It might be doable to set this up on an AWS machine with a beefy GPU or two. I haven't tried it yet though. Once you have a model trained in Huggingface Transformers you'd be able to convert it using this script: https://github.com/moyix/fauxpilot/blob/main/converter/huggingface_gptj_convert.py https://github.com/moyix/fauxpilot/blob/main/converter/huggi...
- bp89 4y ago
  Thank you for sharing the command for finetuning! Is it possible to share your ds_config.json? I tried to finetune the 2B model on A100 (40GB) using your command, but got a CUDA out of memory error. The ds_config I used was the one from huggingface (https://github.com/huggingface/transformers/blob/main/tests/deepspeed/ds_config_zero3.json https://github.com/huggingface/transformers/blob/main/tests/...).
- p1esk 4y ago
  How do I create a dataset?
  - moyix 4y ago
    Have a look at the datasets library [1], but as a shortcut, you can just create a file named "my_code.json" in jsonlines format with one line per source file that looks like: {"text": "contents_of_source_file_1"} {"text": "contents_of_source_file_2"} ... And then pass that my_code.json as the dataset name. [1] https://github.com/huggingface/datasets https://github.com/huggingface/datasets

pabs3 4y ago

A comment below mentions those details are in the paper: https://news.ycombinator.com/item?id=32328168 https://news.ycombinator.com/item?id=32328168

thejosh 4y ago

Copilot only accesses the private repositories for the user it is authenticated with. I'm fine with them accessing public repositories, as long as they respect the license. Which they probably aren't for private? On a related note, Google apparently treats their free customers in a way where all their data is mixed in for the algorithm, but apparently not the data from accounts who use Google Suite (paid business accounts), instead they have their own recommendations built for them on a per-user basis, then has the public algo mixed in.

tpxl 4y ago
> I'm fine with them accessing public repositories, as long as they respect the license. They don't respect the license.

anthropodie 4y ago

> I just need to reimplement a function I've written before There was a project here some time back that allowed to call function using its hash. This avoided replication. It was entirely different paradigm to resolve dependencies. Anyone remembers its name? It was controversial here but it was interesting nonetheless.

tmp642287479428 4y ago
Are you talking about Unison? https://www.unison-lang.org/learn/the-big-idea/ https://www.unison-lang.org/learn/the-big-idea/
- anthropodie 4y ago
  Yeah this is it!

unnouinceput 4y ago

For you and all my sibling comments here. Are you guys forgetting GitHub IS Microsoft? Do you guys actually believe Microsoft is respecting privacy on repositories? Did you read the EULA after the acquisition? C'mon, let's be serious!

moyix 4y ago

It is possible to fine-tune CodeGen using Huggingface Transformers! Then you'd be able to fine-tune it on your own code and use the resulting model. However, training is more expensive -- you'd need an A6000 or better to train the 6B model. Something like the following should work: deepspeed --num_gpus 1 --num_nodes 1 run_clm.py --model_name_or_path=Salesforce/codegen-6B-multi --per_device_train_batch_size=1 --learning_rate 2e-5 --num_train_epochs 1 --output_dir=./codegen-6B-finetuned --dataset_name your_dataset --tokenizer_name Salesforce/codegen-6B-multi --block_size 2048 --gradient_accumulation_steps 32 --do_train --fp16 --overwrite_output_dir --deepspeed ds_config.json Where run_clm.py is this script: https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py https://github.com/huggingface/transformers/blob/main/exampl... It might be doable to set this up on an AWS machine with a beefy GPU or two. I haven't tried it yet though. Once you have a model trained in Huggingface Transformers you'd be able to convert it using this script: https://github.com/moyix/fauxpilot/blob/main/converter/huggingface_gptj_convert.py https://github.com/moyix/fauxpilot/blob/main/converter/huggi...

bp89 4y ago
Thank you for sharing the command for finetuning! Is it possible to share your ds_config.json? I tried to finetune the 2B model on A100 (40GB) using your command, but got a CUDA out of memory error. The ds_config I used was the one from huggingface (https://github.com/huggingface/transformers/blob/main/tests/deepspeed/ds_config_zero3.json https://github.com/huggingface/transformers/blob/main/tests/...).
p1esk 4y ago
How do I create a dataset?
- moyix 4y ago
  Have a look at the datasets library [1], but as a shortcut, you can just create a file named "my_code.json" in jsonlines format with one line per source file that looks like: {"text": "contents_of_source_file_1"} {"text": "contents_of_source_file_2"} ... And then pass that my_code.json as the dataset name. [1] https://github.com/huggingface/datasets https://github.com/huggingface/datasets