samus

33 ms·

661.
by samus 3y ago
It costs more the more you care about squeaky clean training data. Of course you get a better model in return. ChatGPT used a crawl of the internet and patches things up with alignment and DPO. Big boys like Microsoft might have deals with
662.
by samus 3y ago
Since the GPUs can be rented out afterwards, they amortize very quickly with prices in the order of $1/h.
663.
by samus 3y ago
You're only correct about Qwen's MoE. I presume that Chinese model builders feel more pressure to be efficient about using their GPU time because of sanctions.
664.
by samus 3y ago
It worked so far because our desire to grow never before threatened to exceed the carrying capacity of the whole planet. Growth is not stopped by people, but by environmental boundaries. So far the cost of failure was never the near-extinct
665.
by samus 3y ago
Such sclerotic are vulnerable to competitors though and can only survive if they are entrenched enough to be able sit out any changes with minimal adaptations, like car makers and the weapons industry.
666.
by samus 3y ago
The author referred to a specific problem: the company changing their business strategy not because they have a competitor, not because business is crashing (these were the problems Apple was facing in ~1995), but because they must grow eve
667.
by samus 3y ago
As I indicated above, this seems to be an unsustainable state of affairs and an example of the mentality of kicking the can down the road. Let's hope we can continue living longer and better lives.
668.
by samus 3y ago
Did we really do so well on land? We are already looking at overtaxing our planet's carrying capability. 21st century will be pretty rough as the developing countries' lifestyle will catch up and they won't accept the richer
669.
by samus 3y ago
I fully agree. That's why a stable API is needed. But there might be legitimate concerns to not expose such an API as it could impede development. For example, Java intentionally doesn't offer a stable API to extend the compiler.
670.
by samus 3y ago
Indeed, unskilled labor is a large part of the job market.
671.
by samus 3y ago
The latter is a legit question, for which the answer would be a variation of "it makes future development more difficult for the framework developers" and "please send a proposal for a stable API".
672.
by samus 3y ago
If NextJS vendors that React version and then recommends using those APIs, then it can be argued to becomes part of the public interface . On the other hand, the scary-named property is clearly meant to discourage that in a language wher
673.
by samus 3y ago
Buddhism has a sophisticated landscape of hells :-)
674.
by samus 3y ago
Yeah, that's why I wrote "mostly" :)
675.
by samus 3y ago
The stuff in Google's monorepo is mostly their own code, not snapshots from upstream.
676.
by samus 3y ago
I don't necessarily agree. It requires massive computing power, but running models smaller than 70G parameters is possible on consumer hardware, albeit slowly.
677.
by samus 3y ago
We shouldn't choose LLMs for how many facts they support, but their capability to process human language. There is some overlap between these two though, but an LLM that just doesn't know something can always be augmented with RAG
678.
by samus 3y ago
The output of an autoregressive model is a probability for each token to appear next after the input sequence. Computing these is strictly deterministic from the prior context and the model's weights. Based on that probability distribu
679.
by samus 3y ago
Some newer models trained more recently have been repeatedly shown to have comparable performance as larger models. And the Mixture of Experts architecture makes it possible to train large models that know how to selectively activate only t
680.
by samus 3y ago
Are they though? They are lossy compressing trillions of tokens into a few dozen GB. The decompression action is fuzzy and inefficient though.
681.
by samus 3y ago
The context window is comparable to human short-term memory. LLMs are missing episodic memory and means to migrate knowledge between the different layers and into its weights. Math is mostly impeded by the tokenization, but it would still m
682.
by samus 3y ago
The facts LLMs learned from training are fuzzy, unreliable, and quickly outdated. You actually want retrieval-augmented generation (RAG) where a model queries an external system for facts or to perform calculations and postprocesses the res
683.
by samus 3y ago
Their perception is very likely to be totally different. * They might not perceive some substances at all, others that we don't notice might make it unpalatable. * Some substances might be perceived differently than us, or be indisting
684.
by samus 3y ago
> When I asked how exactly did he calculate it, he simply repeated with different numbers. Now you know how an LLM feels during training!
685.
by samus 3y ago
The abstractions are relatively brittle. If you don't have a powerful GPU, you will be forced to consider how to split the model between CPU and GPU, how much context size you need, whether to quantize the model, and the tradeoffs impl
686.
by samus 3y ago
This skepticism is completely justified since ChatGPT 3.5 is also happily hallucinating things that don't exist. For example how to integrate a different system Python interpreter into pyenv. Though maybe ChatGPT 4 doesn't :)
687.
by samus 3y ago
Do your microservices talk to the same database?
688.
by samus 3y ago
I mean "short" in comparison to the unlimited, but lossy recall that the Mamba blocks provide. Transformers are limited to the context length, while Mamba can carry along state. While it can remember things from a lot farther back
689.
by samus 3y ago
The memory palace is a hack that works because in an evolutionary sense our brain's purpose is to help us navigate our world and be effective in it. To do that, it has to be really good at remembering locations, to plot paths through
690.
by samus 3y ago
Who said that we peaked with transformers? I sure hope we did not . The current focus on them is just institutional inertia. Worst case another AI winter comes, at the end of which a newer, more promising technology would manage to attract

More ›