5 ms·

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on commodity CPUs. If 5-7 tok/s on a single CPU for 100B-class models is reproducible, that's a real milestone for on-device inference. Framework is ready. Now we need someone to actually train the model.

rustyhancock 7mo ago
Yes. I had to read it over twice, it does strike me as odd that there wasn't a base model to work with. But it seems the biggest model available is 10B? Somewhat unusual and does make me wonder just how challenging it will be to train any model in the 100B order of magnitude.
- wongarsu 7mo ago
  Approximately as challenging as training a regular 100B model from scratch. Maybe a bit more challenging because there's less experience with it The key insight of the BitNet paper was that using their custom BitLinear layer instead of normal Linear layers (as well as some more training and architecture changes) lead to much, much better results than quantizing an existing model down to 1.58 bits. So you end up making a full training run in bf16 precision using the specially adapted model architecture
- naasking 7mo ago
  What's unusual about it? It seems pretty standard to train small models to validate an approach, and then show that training scales with model size to 8B to 14B parameter models, which is what they did.
wongarsu 7mo ago
I've also always though that it's an interesting opportunity for custom hardware. Two bit addition is incredibly cheap in hardware, especially compared to anything involving floating point. You could make huge vector instructions on the cheap, then connect it to the fastest memory you can buy, and you have a capable inference chip. You'd still need full GPUs for training, but for inference the hardware would be orders of magnitude simpler than what Nvidia is making
- regularfry 7mo ago
  You only need GPUs if you assume the training is gradient descent. GAs or anything else that can handle nonlinearities would be fine, and possibly fast enough to be interesting.
- monocasa 7mo ago
  These are trits, which provide their own efficiencies. Interestingly, a trit x float multiplier is cheaper than a trit x integer multiplier in hardware if you're willing to ignore things like NaNs. 0 and 1 are trivial, just a mux for identity and zero. But because floats are sign-magnitude, multiply by -1 is just an inverter for the sign bit, where as for integers you need a bitwise inverter and full incrermenter.
  - buo 7mo ago
    Do you know a good reference to learn more about this (quantizing weigths to 1.58 bits, and trit arithmetic)?
    - fc417fc802 7mo ago
      There's lots of literature on quantizing weights (including trits and binary) going back 15+ years. Nothing to hand right now but it's all on arxiv. The relevant trit arithmetic should be on display in the linked repo (I haven't checked). Or try working it out for the uncompressed 2 bit form with a pen and paper. It's quite trivial. Try starting with a couple bitfields (inputs and weights), a couple masks, and see if you can figure it out without any help.
embedding-shape 7mo ago
> Framework is ready. Now we need someone to actually train the model. If Microslop aren't gonna train the model themselves to prove their own thesis, why would others? They've had 2 years (I think?) to prove BitNet in at least some way, are you really saying they haven't tried so far? Personally that makes it slightly worrisome to just take what they say at face value, why wouldn't they train and publish a model themselves if this actually led to worthwhile results?
- gregman1 7mo ago
  Cannot agree more!
- throwaw12 7mo ago
  Because this is Microsoft, experimenting and failing is not encouraged, taking less risky bets and getting promoted is. Also no customer asked them to have 1-bit model, hence PM didn't prioritize it. But it doesn't mean, idea is worthless. You could have said same about Transformers, Google released it, but didn't move forward, turns out it was a great idea.
  - embedding-shape 7mo ago
    > You could have said same about Transformers, Google released it, but didn't move forward, I don't think you can, Google looked at the research results, and continued researching Transformers and related technologies, because they saw the value for it particularly in translations. It's part of the original paper, what direction to take, give it a read, it's relatively approachable for being a machine learning paper :) Sure, it took OpenAI to make it into an "assistant" that answered questions, but it's not like Google was completely sleeping on the Transformer, they just had other research directions to go into first. > But it doesn't mean, idea is worthless. I agree, they aren't, hope that wasn't what my message read as :) But, ideas that don't actually pan out in reality are slightly less useful than ideas that do pan out once put to practice. Root commentator seems to try to say "This is a great idea, it's all ready, only missing piece is for someone to do the training and it'll pan out!" which I'm a bit skeptical about, since it's been two years since they introduced the idea.
cubefox 7mo ago
LLM account
- orbital-decay 7mo ago
  Funny enough I now involuntarily take RTFA as a slight slop signal, because all these accounts dutifully read the article before commenting, unlike most HNers who often respond to headlines.
  - cubefox 7mo ago
    Yeah. It correctly pointed out that the editorialized HN title is wrong, there is no 100B model.
  - yorwba 7mo ago
    Not all of them do: https://news.ycombinator.com/item?id=47335156 https://news.ycombinator.com/item?id=47335156 There are evidently lots of people experimenting with different botting setups. Some do better at blending in than others.
    - PeterHolzwarth 7mo ago
      Interesting - the account you mention, and the GP, are both doing replies that are themselves all about the same length, and also the same length between the two accounts. I get what you mean.
  - vova_hn2 7mo ago
    First they claimed that if you use em dashes you are not human And I did not speak out Because I was not using em dashes Then they claimed that if you're crammar is to gud you r not hmuan And I did not spek aut Because mi gramar sukcs Then they claimed that if you actually read the article that you are trying to discuss you are not human...
    - K0balt 7mo ago
      I’ve been rounded up for things I wrote two decades ago because of my em dashes lol. The pitchfork mentality gives me little hope for how things are going to go once we have hive mind AGI robots pervasive in society.
butILoveLife 7mo ago
>. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. I imagine you got 96gb because you thought you'd be running models locally? Did you not know the phrase Unified Memory is marketing speak?
WithinReason 7mo ago
> a fundamentally different compute profile on commodity CPU In what way? On modern processors, a Fused Multiply-Add (FMA) instruction generally has the exact same execution throughput as a basic addition instruction
- actionfromafar 7mo ago
  Bitnet encoding more information dense per byte perhaps? CPUs have slow buses so would eke out more use of bandwidth?
- ismailmaj 7mo ago
  You drop the memory throughput requirements because of the packed representation of bits so an FMA can become the bottleneck, and you bypass the problem of needing to upscale the bits to whatever FP the FMA instruction needs. typically for 1-bit matmul, you can get away with xors and pop_counts which should have a better throughput profile than FMA when taking into account the SIMD nature of the inputs/outputs.
  - WithinReason 7mo ago
    yes but this is not 1 bit matmul, it's 1.58 bits with expensive unpacking
    - ismailmaj 7mo ago
      The title and the repo uses 1-bit when it means 1.58 bits tertiary values, it doesn't change any of my arguments (still xors and pop_counts).
      - WithinReason 7mo ago
        How do you do ternary matmul with popcnt on 1.58 bit packed data?
        ismailmaj 7mo ago
august11 7mo ago
In their demo they're running 3B model.
webXL 7mo ago
It comes from (intentionally?) misleading docs: https://github.com/microsoft/BitNet/issues/391 https://github.com/microsoft/BitNet/issues/391 (only suggesting that it's intentional because it's been there so long)
- verdverm 7mo ago
  That issue appears to be the one that's wrong. From the technical report > We evaluated bitnet.cpp in terms of both inference speed and energy cost. Comprehensive tests were conducted on models with various parameter sizes, ranging from 125M to 100B. specific configurations for each model are detailed in the Appendix A.
  - webXL 7mo ago
    Thanks for pointing that out. I'll ask the issue creator if they've considered that. Would be nice if the maintainer would handle that (sigh) and link to the actual models used for testing (double sigh).
    - verdverm 7mo ago
      From what I gather, there are no models, this is a framework for running 1bit models, but none have been trained. They are mainly demonstrating the possibility.
      - verdverm 7mo ago
        I also don't expect those with poor MCPs to have any better CLIs or APIs, most of the big companies we want them for are not investing in DX/AX. I suspect i.e. that Intuit, if they had great APIs et al, would see it as a threat to their business. Boy would I love to give my agent access to my Quickbooks. They pushed out an incomplete MCP and haven't touched it since. https://github.com/intuit/quickbooks-online-mcp-server https://github.com/intuit/quickbooks-online-mcp-server
cyanydeez 7mo ago
Check out the new QWEN coder model. Also, isnt there different affinities to 8bit vs 4bit for inferences
RandomTeaParty 7mo ago
> The 1.58-bit approach can we stop already with these decimals and just call it "1 trit" which it exactly is?
- hsbauauvhabzb 7mo ago
  Yeah because THAT won’t confuse the average reader.
riidom 7mo ago
Text is misleading too. 5-7 tok/sec is not reading speed, it's a tad slower. For me, at least, and I am an experienced reader, not especially schooled in quick-reading though. I happened to "live" on 7.0-7.5 tok/sec output speed for a while, and it is an annoying experience. It is the equivalent of walking behind someone slightly slower on a footwalk. I dealt with this by deliberately looking away for a minute until output was "buffered" and only then started reading. For any local setup I'd try to reach for 10 tok/sec. Sacrifice some kv cache and shove a few more layers on your GPU, it's worth it.
DrBazza 7mo ago
> memory bandwidth is always the bottleneck I'm hoping that today's complaints are tomorrow's innovations. Back when 1Mb hard drive was $100,000, or when Gates said 640kb is enough. Perhaps some 'in the (chip) industry' can comment on what RAM manufacturers are doing at the moment - better, faster, larger? Or is there not much headroom left and it's down to MOBO manufacturers, and volume?
- bigyabai 7mo ago
  For larger contexts, the bottleneck is probably token prefill instead of memory bandwidth. Supposedly prefill is faster on the M5+ GPUs, but still a big hurdle for pre-M5 chips.
- fc417fc802 7mo ago
  Chip speed has increased faster than memory speed for a long time now, leaving DRAM behind. GDDR was good for awhile but is no longer sufficient. HBM is what's used now. The last logical step of this process would be figuring out how to mix the CPU transistors with the RAM capacitors on the same chip as opposed to merely stacking separate chips on the same package. A related stopgap is the AI startup (forget which) making accelerators on giant chips full of SRAM. Not a cost effective approach outside of ML.
  - azeirah 7mo ago
    Cerebras?
- joquarky 7mo ago
  It might be advantageous to have a different memory structure altogether, bespoke to the specific task.
- Aerroon 7mo ago
  We have faster memory, it's just all used in data center cards you can't buy (and can't afford to buy). AMD actually used HBM2 memory in their Radeon VII card back in 2019 (!!) for $700. It had 16 GB of HBM2 memory with 1 TB/s throughput. The RTX 5080 in conversion l comparison also has 16 GB of VRAM, but was released in 2025 and has 960 GB/s throughput. The RTX 5090 does have an edge at 1.8 TB/s bandwidth and 32 GB of VRAM but it also costs several times more. Imagine if GPUs had gone down the path of the Radeon VII. That being said, the data center cards from both are monstrous. The Nvidia B200 has 180 GB of VRAM (2x90GB) offering 8.2 TB/s bandwidth (4.1 TB/s x2) released in 2024. It just costs as much as a car, but that doesn't matter, because afaik you can't even buy them individually. I think you need to buy a server system from Nvidia or Dell that will come with like 8 of these and cost you like $600k. AMD has the Mi series. Eg AMD MI325x. 288 GB of VRAM doing 10 TB/s bandwidth and released in 2024. Same story as Nvidia: buy from an OEM that will sell you a full system with 8x of these (and if you do get your hands on one of these you need a special motherboard for them since they don't do PCIe). Supposedly a lot cheaper than Nvidia, but still probably $250k. These are not even the latest and greatest for either company. The B300 and Mi355x are even better. It's a shame about the socket for the Mi series GPUs (and the Nvidia ones too). The Mi200 and Mi250x would be pretty cool to get second-hand. They are 64 GB and 128GB VRAM GPUs, but since they use OAP socket you need the special motherboard to run them. They're from 2021, so in a few years time they will likely be replaced, but as a regular joe you likely can't use them. The systems exist, you just can't have them, but you can rent them in the cloud at about $2-4 per hour per GPU.
deepsquirrelnet 7mo ago
The title being misleading is important as well, because this has landed on the front page, and the only thing that would be the only notable part of this submission. The "new" on huggingface banner has weights that were uploaded 11 months ago, and it's 2B params. Work on this in the repo is 2 years old. The amount of publicity compared to the anemic delivery for BitNet is impressive.
deleted 7mo ago
[deleted]
cat_plus_plus 7mo ago
There are 1 bit average GGUFs of large models, not perfect quality but they will hold a conversation. These days, there is also quantized finetuning to heal the damage.

rustyhancock 7mo ago

Yes. I had to read it over twice, it does strike me as odd that there wasn't a base model to work with. But it seems the biggest model available is 10B? Somewhat unusual and does make me wonder just how challenging it will be to train any model in the 100B order of magnitude.

wongarsu 7mo ago
Approximately as challenging as training a regular 100B model from scratch. Maybe a bit more challenging because there's less experience with it The key insight of the BitNet paper was that using their custom BitLinear layer instead of normal Linear layers (as well as some more training and architecture changes) lead to much, much better results than quantizing an existing model down to 1.58 bits. So you end up making a full training run in bf16 precision using the specially adapted model architecture
naasking 7mo ago
What's unusual about it? It seems pretty standard to train small models to validate an approach, and then show that training scales with model size to 8B to 14B parameter models, which is what they did.

wongarsu 7mo ago

I've also always though that it's an interesting opportunity for custom hardware. Two bit addition is incredibly cheap in hardware, especially compared to anything involving floating point. You could make huge vector instructions on the cheap, then connect it to the fastest memory you can buy, and you have a capable inference chip. You'd still need full GPUs for training, but for inference the hardware would be orders of magnitude simpler than what Nvidia is making

regularfry 7mo ago
You only need GPUs if you assume the training is gradient descent. GAs or anything else that can handle nonlinearities would be fine, and possibly fast enough to be interesting.
monocasa 7mo ago
These are trits, which provide their own efficiencies. Interestingly, a trit x float multiplier is cheaper than a trit x integer multiplier in hardware if you're willing to ignore things like NaNs. 0 and 1 are trivial, just a mux for identity and zero. But because floats are sign-magnitude, multiply by -1 is just an inverter for the sign bit, where as for integers you need a bitwise inverter and full incrermenter.
- buo 7mo ago
  Do you know a good reference to learn more about this (quantizing weigths to 1.58 bits, and trit arithmetic)?
  - fc417fc802 7mo ago
    There's lots of literature on quantizing weights (including trits and binary) going back 15+ years. Nothing to hand right now but it's all on arxiv. The relevant trit arithmetic should be on display in the linked repo (I haven't checked). Or try working it out for the uncompressed 2 bit form with a pen and paper. It's quite trivial. Try starting with a couple bitfields (inputs and weights), a couple masks, and see if you can figure it out without any help.

embedding-shape 7mo ago

> Framework is ready. Now we need someone to actually train the model. If Microslop aren't gonna train the model themselves to prove their own thesis, why would others? They've had 2 years (I think?) to prove BitNet in at least some way, are you really saying they haven't tried so far? Personally that makes it slightly worrisome to just take what they say at face value, why wouldn't they train and publish a model themselves if this actually led to worthwhile results?

gregman1 7mo ago
Cannot agree more!
throwaw12 7mo ago
Because this is Microsoft, experimenting and failing is not encouraged, taking less risky bets and getting promoted is. Also no customer asked them to have 1-bit model, hence PM didn't prioritize it. But it doesn't mean, idea is worthless. You could have said same about Transformers, Google released it, but didn't move forward, turns out it was a great idea.
- embedding-shape 7mo ago
  > You could have said same about Transformers, Google released it, but didn't move forward, I don't think you can, Google looked at the research results, and continued researching Transformers and related technologies, because they saw the value for it particularly in translations. It's part of the original paper, what direction to take, give it a read, it's relatively approachable for being a machine learning paper :) Sure, it took OpenAI to make it into an "assistant" that answered questions, but it's not like Google was completely sleeping on the Transformer, they just had other research directions to go into first. > But it doesn't mean, idea is worthless. I agree, they aren't, hope that wasn't what my message read as :) But, ideas that don't actually pan out in reality are slightly less useful than ideas that do pan out once put to practice. Root commentator seems to try to say "This is a great idea, it's all ready, only missing piece is for someone to do the training and it'll pan out!" which I'm a bit skeptical about, since it's been two years since they introduced the idea.

cubefox 7mo ago

LLM account

orbital-decay 7mo ago
Funny enough I now involuntarily take RTFA as a slight slop signal, because all these accounts dutifully read the article before commenting, unlike most HNers who often respond to headlines.
- cubefox 7mo ago
  Yeah. It correctly pointed out that the editorialized HN title is wrong, there is no 100B model.
- yorwba 7mo ago
  Not all of them do: https://news.ycombinator.com/item?id=47335156 https://news.ycombinator.com/item?id=47335156 There are evidently lots of people experimenting with different botting setups. Some do better at blending in than others.
  - PeterHolzwarth 7mo ago
    Interesting - the account you mention, and the GP, are both doing replies that are themselves all about the same length, and also the same length between the two accounts. I get what you mean.
- vova_hn2 7mo ago
  First they claimed that if you use em dashes you are not human And I did not speak out Because I was not using em dashes Then they claimed that if you're crammar is to gud you r not hmuan And I did not spek aut Because mi gramar sukcs Then they claimed that if you actually read the article that you are trying to discuss you are not human...
  - K0balt 7mo ago
    I’ve been rounded up for things I wrote two decades ago because of my em dashes lol. The pitchfork mentality gives me little hope for how things are going to go once we have hive mind AGI robots pervasive in society.

butILoveLife 7mo ago

>. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. I imagine you got 96gb because you thought you'd be running models locally? Did you not know the phrase Unified Memory is marketing speak?

WithinReason 7mo ago

> a fundamentally different compute profile on commodity CPU In what way? On modern processors, a Fused Multiply-Add (FMA) instruction generally has the exact same execution throughput as a basic addition instruction

actionfromafar 7mo ago
Bitnet encoding more information dense per byte perhaps? CPUs have slow buses so would eke out more use of bandwidth?
ismailmaj 7mo ago
You drop the memory throughput requirements because of the packed representation of bits so an FMA can become the bottleneck, and you bypass the problem of needing to upscale the bits to whatever FP the FMA instruction needs. typically for 1-bit matmul, you can get away with xors and pop_counts which should have a better throughput profile than FMA when taking into account the SIMD nature of the inputs/outputs.
- WithinReason 7mo ago
  yes but this is not 1 bit matmul, it's 1.58 bits with expensive unpacking
  - ismailmaj 7mo ago
    The title and the repo uses 1-bit when it means 1.58 bits tertiary values, it doesn't change any of my arguments (still xors and pop_counts).
    - WithinReason 7mo ago
      How do you do ternary matmul with popcnt on 1.58 bit packed data?
      - ismailmaj 7mo ago

august11 7mo ago

In their demo they're running 3B model.

webXL 7mo ago

It comes from (intentionally?) misleading docs: https://github.com/microsoft/BitNet/issues/391 https://github.com/microsoft/BitNet/issues/391 (only suggesting that it's intentional because it's been there so long)

verdverm 7mo ago
That issue appears to be the one that's wrong. From the technical report > We evaluated bitnet.cpp in terms of both inference speed and energy cost. Comprehensive tests were conducted on models with various parameter sizes, ranging from 125M to 100B. specific configurations for each model are detailed in the Appendix A.
- webXL 7mo ago
  Thanks for pointing that out. I'll ask the issue creator if they've considered that. Would be nice if the maintainer would handle that (sigh) and link to the actual models used for testing (double sigh).
  - verdverm 7mo ago
    From what I gather, there are no models, this is a framework for running 1bit models, but none have been trained. They are mainly demonstrating the possibility.
    - verdverm 7mo ago
      I also don't expect those with poor MCPs to have any better CLIs or APIs, most of the big companies we want them for are not investing in DX/AX. I suspect i.e. that Intuit, if they had great APIs et al, would see it as a threat to their business. Boy would I love to give my agent access to my Quickbooks. They pushed out an incomplete MCP and haven't touched it since. https://github.com/intuit/quickbooks-online-mcp-server https://github.com/intuit/quickbooks-online-mcp-server

cyanydeez 7mo ago

Check out the new QWEN coder model. Also, isnt there different affinities to 8bit vs 4bit for inferences

RandomTeaParty 7mo ago

> The 1.58-bit approach can we stop already with these decimals and just call it "1 trit" which it exactly is?

hsbauauvhabzb 7mo ago
Yeah because THAT won’t confuse the average reader.

riidom 7mo ago

Text is misleading too. 5-7 tok/sec is not reading speed, it's a tad slower. For me, at least, and I am an experienced reader, not especially schooled in quick-reading though. I happened to "live" on 7.0-7.5 tok/sec output speed for a while, and it is an annoying experience. It is the equivalent of walking behind someone slightly slower on a footwalk. I dealt with this by deliberately looking away for a minute until output was "buffered" and only then started reading. For any local setup I'd try to reach for 10 tok/sec. Sacrifice some kv cache and shove a few more layers on your GPU, it's worth it.

DrBazza 7mo ago

> memory bandwidth is always the bottleneck I'm hoping that today's complaints are tomorrow's innovations. Back when 1Mb hard drive was $100,000, or when Gates said 640kb is enough. Perhaps some 'in the (chip) industry' can comment on what RAM manufacturers are doing at the moment - better, faster, larger? Or is there not much headroom left and it's down to MOBO manufacturers, and volume?

bigyabai 7mo ago
For larger contexts, the bottleneck is probably token prefill instead of memory bandwidth. Supposedly prefill is faster on the M5+ GPUs, but still a big hurdle for pre-M5 chips.
fc417fc802 7mo ago
Chip speed has increased faster than memory speed for a long time now, leaving DRAM behind. GDDR was good for awhile but is no longer sufficient. HBM is what's used now. The last logical step of this process would be figuring out how to mix the CPU transistors with the RAM capacitors on the same chip as opposed to merely stacking separate chips on the same package. A related stopgap is the AI startup (forget which) making accelerators on giant chips full of SRAM. Not a cost effective approach outside of ML.
- azeirah 7mo ago
  Cerebras?
joquarky 7mo ago
It might be advantageous to have a different memory structure altogether, bespoke to the specific task.
Aerroon 7mo ago
We have faster memory, it's just all used in data center cards you can't buy (and can't afford to buy). AMD actually used HBM2 memory in their Radeon VII card back in 2019 (!!) for $700. It had 16 GB of HBM2 memory with 1 TB/s throughput. The RTX 5080 in conversion l comparison also has 16 GB of VRAM, but was released in 2025 and has 960 GB/s throughput. The RTX 5090 does have an edge at 1.8 TB/s bandwidth and 32 GB of VRAM but it also costs several times more. Imagine if GPUs had gone down the path of the Radeon VII. That being said, the data center cards from both are monstrous. The Nvidia B200 has 180 GB of VRAM (2x90GB) offering 8.2 TB/s bandwidth (4.1 TB/s x2) released in 2024. It just costs as much as a car, but that doesn't matter, because afaik you can't even buy them individually. I think you need to buy a server system from Nvidia or Dell that will come with like 8 of these and cost you like $600k. AMD has the Mi series. Eg AMD MI325x. 288 GB of VRAM doing 10 TB/s bandwidth and released in 2024. Same story as Nvidia: buy from an OEM that will sell you a full system with 8x of these (and if you do get your hands on one of these you need a special motherboard for them since they don't do PCIe). Supposedly a lot cheaper than Nvidia, but still probably $250k. These are not even the latest and greatest for either company. The B300 and Mi355x are even better. It's a shame about the socket for the Mi series GPUs (and the Nvidia ones too). The Mi200 and Mi250x would be pretty cool to get second-hand. They are 64 GB and 128GB VRAM GPUs, but since they use OAP socket you need the special motherboard to run them. They're from 2021, so in a few years time they will likely be replaced, but as a regular joe you likely can't use them. The systems exist, you just can't have them, but you can rent them in the cloud at about $2-4 per hour per GPU.

deepsquirrelnet 7mo ago

The title being misleading is important as well, because this has landed on the front page, and the only thing that would be the only notable part of this submission. The "new" on huggingface banner has weights that were uploaded 11 months ago, and it's 2B params. Work on this in the repo is 2 years old. The amount of publicity compared to the anemic delivery for BitNet is impressive.

deleted 7mo ago

[deleted]

cat_plus_plus 7mo ago

There are 1 bit average GGUFs of large models, not perfect quality but they will hold a conversation. These days, there is also quantized finetuning to heal the damage.