sanxiyn

31 ms·

541.
by sanxiyn 4y ago
Agreed. Scraping ChatGPT is against OpenAI terms of use and OpenAI is entitled to terminate your access immediately upon notice, but since ChatGPT output is not copyrighted (and copyrightable), output you acquired before termination should
542.
by sanxiyn 4y ago
What overfitting? If anything, LLMs suffer from underfitting, not overfitting. Normally, overfitting is characterized by increasing validation loss while training loss is decreasing, and solved by early stopping (stopping before that happen
543.
by sanxiyn 4y ago
I expected this to include more resources like TPU Research Cloud, but it doesn't mention even TRC itself. Disappointing. https://sites.research.google/trc/about/
544.
by sanxiyn 4y ago
It won't happen. But I agree this would be good for the world, because I expect the merge to make US politics a lot saner. Mexico's population is more than California, Texas, and New York combined, and since US electoral college v
545.
AI Accountability Policy Request for Comment(ntia.gov)
3 points by sanxiyn 4y ago|0 comments
546.
by sanxiyn 4y ago
As the paper explains, optical circuit switches are not new in TPU v4 and not the main topic of this paper. Google was already using it for networking and published about it last year. For details, see https://arxiv.org/abs&
547.
by sanxiyn 4y ago
We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For example, one layer transformer eventually learns to do modular addition completely co
548.
by sanxiyn 4y ago
Let's not be ridiculous. Of course LLMs are planning ahead, not generating words one at a time. You can't even generate correct indefinite articles (a/an) without planning one word (or token) ahead. Surely you are not suggest
549.
by sanxiyn 4y ago
Yes it is, and it was tested. GPT-4 can't solve any of the tests.
550.
by sanxiyn 4y ago
Yes, François Chollet released ARC(Abstraction and Reasoning Corpus) benchmark for this in 2019, and the benchmark can be scored automatically. Humans solve 100% of tests and GPTs solve 0% of tests and GPTs made exactly zero progress from 2
551.
by sanxiyn 4y ago
You know, François Chollet literally did this, and people don't listen. People should listen to Chollet more. https://twitter.com/fchollet/status/1638643323748618240 https://arxiv.org/abs&#x2F
552.
by sanxiyn 4y ago
Okay, I think we are in agreement. Let's say current LLM architectures are clearly capable of developing internal representations, not just learning surface statistics, and it can execute algorithmic computation as complex as computing
553.
by sanxiyn 4y ago
LLMs clearly develop internal representations, this is an empirical fact. For example, one paper cited is https://arxiv.org/abs/2210.13382 , which trained an LLM on Othello games, where an 8 layer model is fed one of 60
554.
by sanxiyn 4y ago
https://arxiv.org/abs/2101.00027 is the paper and it includes 95.16 GiB from GitHub.
555.
by sanxiyn 4y ago
> China can't build a 5 nm chip or a new type of wing or a useful new programming language. You are saying this in a reply to the post to open source RTL of high performance CPU from China? China also can demonstrably build a compet
556.
by sanxiyn 4y ago
Indeed, I see WebGL commented out here: https://github.com/xenova/transformers.js/blob/main/src/mode... .
557.
by sanxiyn 4y ago
This runs on GPU with WebGL, so it will depend on what GPU you have.
558.
by sanxiyn 4y ago
I am pretty sure it exists, but not for you. "Select developers can access the PaLM API ... in Private Preview today, and stay tuned for our waitlist soon".
559.
by sanxiyn 4y ago
This is for "latency-insensitive tasks with batched processing", which LLM integrated into games and browsers is not. Latency sensitive tasks basically have an effective batch size of 1, not 144. So for those tasks, it will take a
560.
by sanxiyn 4y ago
No, it is not RLHF because there is no reward model involved. See also OpenAI's explanation here: https://platform.openai.com/docs/model-index-for-researchers
561.
by sanxiyn 4y ago
Note that Alpaca is NOT using RLHF. It explicitly states it used supervised finetuning.
562.
by sanxiyn 4y ago
Here is an example I just made up: in the network, include a token that is never produced by the tokenizer, and watermark the token's embedding. It won't be touched by normal fine tuning and will remain the same. I am sure there a
563.
by sanxiyn 4y ago
Triton is lower level than this. The post actually mentions Triton, search for it. > Extension mechanisms such as Custom-call enable users to write deep learning primitives with CUDA, HIP, SYCL, Triton and other kernel languages so they
564.
by sanxiyn 4y ago
Sadly, I don't think so. Android already has NNAPI, but this post doesn't mention NNAPI at all. This seems focused on running on servers, instead of running on user devices.
565.
by sanxiyn 4y ago
From Earth's hydrothermal vents, we know sunlight isn't necessary for life although energy is.
566.
by sanxiyn 4y ago
> Attention mechanism inference costs rise with the square of the inference window. This is incorrect. It is correct for the original Transformer, but OpenAI isn't using the original Transformer since GPT-3. Sparse Transformer scale
567.
by sanxiyn 4y ago
Sam graduated in 2022. So his job was being a university student.
568.
by sanxiyn 4y ago
> ... electron beam lithography ... could theoretically be built out of a hacked scanning electron microscope Not only theoretically. Sam hacked his own scanning electron microscope and did electron beam lithography in 2018. http:/
569.
by sanxiyn 4y ago
It's nothing new. Remember Gmail invites?
570.
by sanxiyn 4y ago
Rust is supported in all Debian release architectures. It kinds of has to, it is a build dependency of Firefox (and soon, Linux itself). To be more than a FUD, you should specify which Linux distribution and which architecture. Yes, it was

More ›