Kubuxu

6 ms·

31.
by Kubuxu 1y ago
The question is who does the copyright belong to in this repository. It is both original author and Microsoft (because they took authors code and modified it). So the License file should mention both.
32.
by Kubuxu 2y ago
TME is European as well, and I think Arrow has warehouses in Europe.
33.
by Kubuxu 2y ago
Could you list few? I’ve stopped following media in general because everything I could find was profit maximisation through outrage and sensationalism. So I would really appreciate something that isn’t that.
34.
by Kubuxu 2y ago
I think what would be useful is a histogram of population densities, or for example a percentile based metric: population density covering 99/95% of population.
35.
by Kubuxu 2y ago
Don’t be sad for commercial entity that is not a good player https://github.com/ggerganov/llama.cpp/pull/11016#issuecomme...
36.
by Kubuxu 2y ago
I don't think entire GPU is specialised nor a singular token will use the same expert. I think about it as a gather-scatter operation at each layer. Let's say you have an inference batch of 128 chats, at layer `i` you take the hid
37.
by Kubuxu 2y ago
Not sure about DeepSeek R1, but you are right in regards to previous MoE architectures. It doesn’t reduce memory usage, as each subsequent token might require different expert buy it reduces per token compute/bandwidth usage. If you pl
38.
by Kubuxu 2y ago
I remember reading a paper that showed that giving models even a a few filler tokens before requiring a single phrase/word/number answer significantly increasee accuracy. This is probably similar.
39.
by Kubuxu 2y ago
Chain of Thought was known since 2022 ( https://arxiv.org/abs/2201.11903 ), we just were stuck in a world where we were dumping more data and compute at the training instead of looking at other improvements.
40.
by Kubuxu 2y ago
You can create contract, but contracts require consideration, and I don’t see how you do consideration in a case like this without it being a bribery.
41.
by Kubuxu 2y ago
A lot of people don’t opt-out themselves, but the society opts them out. Be it a bully or stigma in school, not fitting in high-school or uni.
42.
by Kubuxu 2y ago
Yes, depth of focus will be larger, as signified by the larger f-number.
43.
by Kubuxu 2y ago
FFWD later is independent of context size, each processed token passes thought the same weights.
44.
by Kubuxu 2y ago
It is quite ironic to get a cookie banner on tat page.
45.
by Kubuxu 2y ago
Probably forks/duplicates of repos in the dataset.
46.
by Kubuxu 2y ago
It would double the size of the KV cache, which can be significant (multi-GB) at larger context sizes.
47.
by Kubuxu 2y ago
Hmm, 0.8 works well, but let's try setting lower layers to lower initial value. Let's say 0.2. Ok, I need a formula that will go between 0.2 and 0.8, slowly approaching 0.8. Starts fiddling with numbers for 20min, I guess this can
48.
by Kubuxu 2y ago
They thought it was dead end, that is why they released it :P
49.
by Kubuxu 2y ago
Don’t worry they wrote 50-500 more of them.
50.
by Kubuxu 2y ago
AFAIK Intel and AMD have cross-licensing deals for x86 and AMD64 which become invalid in case of acquisition, on the side that gets acquired. This means that you cannot acquire rights to manufacture x64 by acquiring either of them.
51.
WebP: The WebPage Compression Format(purplesyringa.moe)
552 points by Kubuxu 2y ago|203 comments
52.
by Kubuxu 2y ago
You could have an O(N^2) algorithm outperform an O(N) on the scale of 10,000 (or whatever scale you want to imagine). The big-O notation compares only asymptotic behaviour, and sometimes the lower power factors are overwhelming.
53.
by Kubuxu 2y ago
The gap between fine-tuning API and weights-available is much more significant than you give it credit for. You can take the weights and train LoRAs (which is close to fine-tuning), but you can also build custom adapters on top (classificat
54.
by Kubuxu 2y ago
That is partially on PlatformIO. AFAIK, they started pushing hard the scheme "be a partner (meaning pay us) or we won't even accept external contributions to support your platform". ESP had a pretty public dispute with them (
55.
by Kubuxu 2y ago
AB+ is the universal recipient of red blood cells, O- is the universal donor. It is reverse for plasma. The blood rejection is reaction between blood plasma (with cells produced by your immune system) and red blood cells.
56.
by Kubuxu 2y ago
Zero-knowledge cryptography frequently deals with polynomials of degree 2^20 - 2^27.
57.
by Kubuxu 2y ago
FFT is used extensively in curve based zero-knowledge cryptography, not for the purpose of splitting a single large number into smaller ones, but to interpolate and evaluate very large polynomials (for example of the degree 2^27). All of th
58.
by Kubuxu 2y ago
Using image embedding and evaluating 100s billion parameter LLM for OCR is like hunting rabbits using Yamato’s 18in naval gun.
59.
by Kubuxu 2y ago
It keeps the side from curling in and causing the irritation/cut that results in what we call ingrown toenail.
60.
by Kubuxu 3y ago
The 8x is misleading; there are 8 sets of weights (experts) per token and per layer. If it is similar to the previous MoE Mistral models, then two experts get activated per token per layer. This reduces the amount of compute and memory band

More ›