moyix

28 ms·

271.
by moyix 6y ago
I was trying to avoid letting people figure out what id mapped to real/fake but in retrospect this was a mistake, yeah. Sorry about that!
272.
by moyix 6y ago
The “real” code came from these packages: https://moyix.net/~moyix/sample_pkgnames.txt
273.
by moyix 6y ago
This made me worried, so I went and spot-checked 5-6. Using the "cheat sheet" I was always able to guess correctly, so I think the site is working fine. The list of packages the real snippets are drawn from is here (maybe if you w
274.
by moyix 6y ago
Yeah, I trained my own BPE tokenizer for this and it results in pretty good compression. From 1024 BPE tokens you can generate anywhere from 2000-6000 actual characters of text. My guess is that it's a bit more efficient than English-B
275.
by moyix 6y ago
Yep, there could be a lot of noise in there too from people guessing lazily.
276.
by moyix 6y ago
It has entirely memorized a bunch of common open source licenses, a bunch of contributor names/emails, and so on. However when I've tried to locate the actual code it's producing in the training data it's not there.
277.
by moyix 6y ago
Since these are snippets from a random position in the file, it's possible that the code that initialized them was outside the snippet?
278.
by moyix 6y ago
Yep, I trained my own BPE using HuggingFace's tokenizer library. During training I didn't keep the entire dataset in memory because even on an RTX8000 the full dataset + model weights + data used by the optimizer (ADAM) is too big
279.
by moyix 6y ago
Sure, it's here in JSON format: https://moyix.net/~moyix/csrc_dataset_large.json.gz
280.
by moyix 6y ago
It should be equal: there are 1000 real and 1000 generated samples in the database, retrieved via: SELECT id, code, real FROM code ORDER BY random() LIMIT 1
281.
by moyix 6y ago
The generated snippets are prompted with 128 characters from real code (but not code from the training data), so they can often pick up on the name of the project etc.
282.
by moyix 6y ago
There are likely some "tells" but many fewer of them than I expected. I've seen it occasionally generate something malformed like "#includefrom", and like all GPT2 models it has a tendency to repeat things. Yes, I t
283.
by moyix 6y ago
I've tried searching for variable and function names and even bits of comments to see if they're copied from the training data. They're not!
284.
by moyix 6y ago
Hi, author here! Some details on the model: * Trained 17GB of code from the top 10,000 most popular Debian packages. The source files were deduplicated using a process similar to the OpenWebText preprocessing (basically a locality-sensitive
285.
by moyix 6y ago
I remember enjoying this with Fluxbox back in 2003 or so :)
286.
by moyix 6y ago
For large models it does help! The training loop for multiple GPUs with data parallelism is roughly: 1. Split the data up 2. Do a forward and backward pass on each GPU individually 3. Compute the average of the gradients and update the mode
287.
by moyix 6y ago
I went with the NVIDIA Ubuntu repos: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/i... This is nice because staying updated can be done through the usual apt commands, and you can have multiple CUDA
288.
by moyix 6y ago
Linux (Ubuntu 20.04) + Cuda 11.2. For the backend I use PyTorch; Tensorflow has some nice optimizations (like XLA, which uses LLVM to JIT optimized code for the GPU), but I found it very painful to get working reliably, and most of the lang
289.
by moyix 6y ago
I agree. I've been training on 2x3090s connected via NVLink and they're really fast for training language models. I am actually tempted to try and replicate the OP's GPT2 replication using Huggingface, DeepSpeed, and OpenWe
290.
by moyix 6y ago
I think the presence of PS/2 ports is to satisfy gamers – the PS/2 port has a dedicated interrupt line, which (allegedly) gives you lower latency on keypresses than USB, which uses input polling.
291.
by moyix 6y ago
Sure, order a Raspberry Pi 4 and an external USB SSD ;)
292.
by moyix 6y ago
Last time this came up I wrote a bit about the history of Elbrus and why I think it's an interesting platform: https://old.reddit.com/r/hardware/comments/jx60fs/vliw_cpu_i... I very much hope to be
293.
by moyix 6y ago
This is true – but at the same time the security research community doesn't require that every paper do this, only that if your paper has potential for harm that you discuss how it was managed. It definitely comes up in program commi
294.
by moyix 6y ago
The problem is really the opposite – too many organizations slavishly follow the pre-2017 NIST guidance! Per wikipedia: > From 2004, the “NIST Special Publication 800-63. Appendix A,”[2] advised people to use irregular capitalization, sp
295.
by moyix 6y ago
The success of the "trojan defense" is encouraging on this point, I guess: https://en.wikipedia.org/wiki/Trojan_horse_defense#Cases_inv... I'm sure defense lawyers would love it :)
296.
by moyix 6y ago
Has this kind of repudiation ever been tested in the real world? It's hard to imagine a court throwing out email evidence because it lacked a DKIM signature. And on a personal level seeing a chat transcript that had cryptographic non-r
297.
by moyix 6y ago
The AMA section on FP32 seems to imply it's in hardware though, unless I'm missing something – datapath is a hardware term, at least. I mean, I really hope that it's just in the driver so that an enterprising reverse engine
298.
by moyix 6y ago
Where do you see that this is artificial (i.e. in software/firmware) rather than just missing some hardware that's present on the Titan/Quadro cards?
299.
by moyix 6y ago
The main thing isthat it doesn't support system calls, so the vast majority of code won't run. The ability to translate from one architecture to another is super cool though, and if you're curious about it you should check
300.
by moyix 6y ago
I'm not clear on where the pbrt-next branch is, but an early release of PBRTv4 is here: https://github.com/mmp/pbrt-v4

More ›