moyix

17 ms·

151.
by moyix 4y ago
Please don't do this to me, I don't know if I have it in me to go on ANOTHER big scrape & scan.
152.
by moyix 4y ago
Nope. clang has "-Weverything", and gcc has "-Wextra", both of which go beyond "-Wall". https://stackoverflow.com/questions/11714827/how-can-i-turn-...
153.
by moyix 4y ago
Yeah, unfortunately I have no idea if that was their original goal (which seems unlikely?) or if this is just a minimal example they came up with after tripping over the actual problem in a more realistic setting. I think it suffices to sho
154.
by moyix 4y ago
The gevent issue has an example: https://github.com/gevent/gevent/pull/1820 I haven't examined the code of scipy.stats.skellam.sf so I can't say for sure that it's not converging, but it's
155.
by moyix 4y ago
Nope, it still links in crtfastmath: $ gcc -Ofast -fno-unsafe-math-optimizations -fpic -shared foo.c -o foo.so $ objdump -j .text --disassemble=set_fast_math foo.so foo.so: file format elf64-x86-64 Disassembly of se
156.
by moyix 4y ago
It suppresses output for bad words in both the prompt (your code) and the suggestions.
157.
by moyix 4y ago
Yep, I noticed this last year when they still stored the list client-side and had great fun reverse engineering it: https://twitter.com/moyix/status/1433254293352730628
158.
by moyix 4y ago
FYI, there is in fact a bad word filter in GitHub Copilot. When it was first released, the list was stored client-side in obfuscated form and I had a lot of fun decoding it: https://twitter.com/moyix/status/1433254
159.
by moyix 4y ago
Got it working :) You can now use the 2B models in FauxPilot as well.
160.
by moyix 4y ago
Have a look at the datasets library [1], but as a shortcut, you can just create a file named "my_code.json" in jsonlines format with one line per source file that looks like: {"text": "contents_of_source_file_1
161.
by moyix 4y ago
The largest Python model (codegen-16B-mono) should be pretty competitive! The biggest thing that it lacks is the ability to make use of context after the cursor. Unfortunately, that capability will require retraining the models: https:&#
162.
by moyix 4y ago
Assuming that it's just that hardcoded check I should be able to add it today. But if that check is load-bearing (i.e. if it relies on those assumptions elsewhere in the code) it could be a bit more painful. Edit: sadly, it's no
163.
by moyix 4y ago
The models aren't customized for individual codebases; all of them were trained on most of GitHub.
164.
by moyix 4y ago
Fixed! Didn't expect this to hit HN so fast :)
165.
by moyix 4y ago
It is possible to fine-tune CodeGen using Huggingface Transformers! Then you'd be able to fine-tune it on your own code and use the resulting model. However, training is more expensive -- you'd need an A6000 or better to train the
166.
by moyix 4y ago
QEMU runs just fine on ARM macs :) It will emulate an x86 virtual machine that xv6 runs inside.
167.
by moyix 4y ago
I'm working on integrating it with a VSCode plugin right now :) After that I'll write up a post and try to release it as something easy to use!
168.
by moyix 4y ago
Reposting my commentary: The TL;DR is that this appears to beat both OpenAI Codex (12 billion params) and DeepMind's AlphaCode (1 billion params), despite having only 770M parameters. There is also a paper: https://arxiv.org
169.
AI Coding with CodeRL: Toward Mastering Program Synthesis with Deep RL(blog.salesforceairesearch.com)
66 points by moyix 4y ago|21 comments
170.
Using linear algebra to convert a large code model(gist.github.com)
53 points by moyix 4y ago|3 comments
171.
Using linear algebra to convert a large code model(gist.github.com)
2 points by moyix 4y ago|0 comments
172.
by moyix 4y ago
The TL;DR is that this appears to beat both OpenAI Codex (12 billion params) and DeepMind's AlphaCode (1 billion params), despite having only 770M parameters. There is also a paper: https://arxiv.org/abs/2207.01780
173.
AI Coding with CodeRL: Toward Mastering Program Synthesis with Deep RL(blog.salesforceairesearch.com)
1 points by moyix 4y ago|1 comments
174.
by moyix 4y ago
SVG isn't really possible with the model architecture they're using. The diffusion+upscaling step basically outputs 1024x1024 pixels; at no point does the model have a vector representation. I suppose it's possible that at so
175.
by moyix 4y ago
Haha, I don't know how folks found this, but I actually did make it for a real-world use case. While doing large-scale builds of debian packages [1], occasionally I'd have parts of the build process hang (usually some test that I
176.
by moyix 4y ago
Huh. That one looks like it wasn't able to place any part of the model on the GPU. I know there is currently an issue where if the first layer of the model is too big to fit on the GPU it will put all layers on the CPU/disk (rat
177.
by moyix 4y ago
Just to double-check, are you using the git versions of transformers and accelerate? I believe the BFloat16 support is pretty recent (22 days ago) and may not be in conda/pip: https://github.com/huggingface/acceler
178.
by moyix 4y ago
Yep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even have to sit on disk (SSD). This makes it much slower than if everything were on
179.
by moyix 4y ago
I noticed this too -- I think it's a git LFS issue? The weights themselves are only 330GB, but the .git directory adds another 330GB: $ du -csh pytorch_model*.bin | tail -n 1 329G total $ du -sh .git 329G .git
180.
by moyix 4y ago
You can actually download the weights without an account: git clone https://huggingface.co/bigscience/bloom You will need Git LFS and ~330GB of free space though. It is possible to run inference locally, even wi

More ›