4 ms·

I am not trying to defend Mozilla doing this, and I don't support sending data to cloud based services like this in a way that users won't understand. But I al

I am not trying to defend Mozilla doing this, and I don't support sending data to cloud based services like this in a way that users won't understand. But I also think that the state of the art in small LLM and user device capabilities aren't there yet to put a "good enough to be actually useful" local-only LLM as a prepackaged thing in a mass market distributed browser. You don't want a browser that takes 10GB of extra RAM (on top of the memory hog that is having just 3 or 4 complex tabs open on its own already) and pegs your CPU at 99% usage for minutes at a time. And not in an era when mass market consumer laptops are still commonly 8GB or 16GB of total system RAM. Many of those with integrated-into-CPU onboard graphics (eg: not a gaming laptop with a discrete GPU on the PCI-E bus). It'll be a catastrophe for laptop battery life, among other resource use problems. And an LLM that fits in under 8 to 10GB of RAM for CPU-only inference is not going to be nearly as capable as an off-device inference system. I wish they had just done this with a very clear up front opt in (not enabled by default) thing that explains what Mistral is, that it's not some big American cloud company but a relatively small startup in France, and that your prompts/LLM interactions will go to their servers. And some documentation on how it will be handled/stored in a supposedly trustworthy manner.

jimmydoe 17d ago
Agree, advocating for Firefox developing features only for rich hobbyists(people who can afford large RAM and GPU) is absurd.
- Barbing 17d ago
  >advocating for Firefox developing features only for rich hobbyists ?: >marketing pages aren't candid enough to clearly explain
  - well_ackshually 17d ago
    The only people having an expectation of translations being done locally is exactly the nerds that keep whining that it's not using a local model. Every single normal person, when presented with a "translate" button either know it's going online, or don't care about it. Begging the purists to run away from Firefox at this point so they can stop wasting everyone's time. Your demands for examplarity and whining about money not going ONLY to firefox and jerking yourselves on Servo was not enough, now you want to restrict the browser to owners of an RTX5080 if they want to use it?
    - pessimizer 17d ago
      You're talking out of your ass. It is so ironic that "normal people don't care about privacy, just you nerds" is an HN meme. Everybody I know cares, often in an extreme way, and I barely hang out with technical people in real life. I've met people so irrational and nontechnical about modern invasions of privacy that they think that it is being driven by demons. The reason this is a meme on HN is because tons of HN posters are people who spend a lot of effort trying to invade people's privacy and to come up with new ways of concealing that fact. Virtually every sleazebag trying to hide things in ToSes, updates, and telemetry has been an HN poster. And they're pretending that they speak for normal people, because they are scumbags, think normal people are animals that will do anything that they can get away with and don't care about any boundary, and that therefore 1) they themselves are normal, and 2) normal people deserve whatever happens to them. This is what comes from making know-nothing Libertarianism/Objectivism a mainstream ideology, ironically at the same moment that Alan Greenspan, a direct Rand acolyte, was admitting that it had failed while the world economy was sliding into the toilet. Intellectual Libertarianism/Objectivism had failed, but the dumb kind had yet to properly rise. I don't know anything except you nerds are worried about nothing, I'm going to do it because I can and nobody is going to tell me what to do, nobody normal cares about this, and if they do let them try and stop me, why do you care anyway... Please just do it and stop talking, you don't have anything to say. I hope all of you end up in cells next to SBF.
      - walrus01 17d ago
        I do not fundamentally disagree with you but it's also an extremely well known phenomenon that average non tech users, in the aggregate of millions of people, will click almost any "yes/I agree/continue/Next" step on a software installer or new user sign up workflow for anything, without reading the ToS. People blithly sign up for all sorts of cloud based things and services without understanding their full ramifications all the time. People such as you are describing and rightfully criticizing are knowingly taking advantage of that. Indeed it's how a lot of malware gets installed too. The person you're responding to is pointing out that a lot of people at the surface level do only appear to care about the results. They put something into google translate, it works, they gets results they are pleased with, they don't put a lot of thought into the fact that the data is going to an external service. That's not an inaccurate description of how a lot of people use their computers these days. The fact that people will click yes/agree/OK on almost anything is how Windows computers got Bonzi Buddy installed on them back in the day, and now it's continued into the cloud-everything era.
- LoganDark 17d ago
  They're advocating for a CHOICE and for the difference to be explained.
paimapi 17d ago
>But I also think that the state of the art in small LLM and user device capabilities aren't there yet to put a "good enough to be actually useful" local-only LLM so then don't add it in as highly advertised feature until it is. doing things right and living up to your core values is a lot to expect from businesses these days but, at minimum, a non-profit foundation should be able to live up to these goals, yes?
nosioptar 17d ago
I wouldn't be pissed at the knuckle dragging fuckwits who run and work at Mozilla if they'd done this how you suggest in the final paragraph.
horsawlarway 17d ago
I'll ding Mozilla directly then - Since your "we can't do this yet" approach is actually the literal thing Chrome is shipping... https://developer.chrome.com/docs/ai/built-in/overview?_gl=1*f13fhs*_up*MQ..*_gs*MQ..&gclid=CjwKCAjw_KjVBhAHEiwAnC0N9BPrOjdOMIaN4cfoB6keFXL1FIQROQe0yKOOtu667ZzVrJP0WWkkZRoC8IgQAvD_BwE&gbraid=0AAAAAC1d8f6PYo2kmg9AmW0JUdL46Cuhf https://developer.chrome.com/docs/ai/built-in/overview?_gl=1...
- odo1242 17d ago
  But the built in AI model in Chrome is 4GB of RAM, 4GB of disk space, and is still only used in a couple places
peri-cl 17d ago
Well, it depends on the task, doesn't it? "running shoes I looked at last week" / "Here's what I found in your browsing history:" doesn't need a 119 billion parameter frontier model; it's a RAG problem for the 0.6 B embedding models. That's an example Mozilla offers. Presumably to explain to their users why it's essential they hand over their last week's browsing history for this convenience (but it isn't! Hardly for that!) I feel it's wrong to tell users that it's important and normal to relinquish all control of their—extremely personal—life history, in bulk, in plaintext, to strangers. I agree wholeheartedly that remote server inference is super useful, and that local inference falls far short on many tasks. (I have no objection at all to Mozilla providing a cloud inference feature). What I don't buy is that we must ask users to redraw their personal boundaries so that their most intimate life details, and remote frontier-model inference, overlap. They do not need to overlap. You can accomplish a lot with private local inference with the smallest of models; and you can accomplish a lot on remote servers which aren't privy to everything. If some convenience is lost by not combining the two, well, so be it. I'm sure most people would agree, if all of this was laid out plainly.
- walrus01 17d ago
  I'm of the opinion that local inference should be done to the greatest extent that is realistically possible, at the earlier time that the hardware/average user platform is capable of doing so. I personally spend a fair bit on kWh extra in my home electrical bill monthly for having a good sized chunk of local inference ability in my house, but that's not a common thing yet. If mozilla is doing things to send users down the path of doing this externally, they need to be much more upfront and transparent with the users about where their data is going, and not bury it in some terms/conditions that only nerds will hunt for.
- graemep 17d ago
  > "running shoes I looked at last week" / "Here's what I found in your browsing history:" Does that need an LLM at all?
  - smsm42 17d ago
    Not really, but it's probably easier to make it on top of LLM than to make specially-purposed tool for it, if we talking in terms of time-to-market effort.
nullc 17d ago
Have you tried Ling-3.0-tiny? It runs fine on CPU-- on a 14700KF gets 40tg/s and 250pp/s and on a ordinary gpu (RTX 4070) does over 200tg/s with no MTP and 7185pp/s. It's certainly not as capable as something that needs a high memory gpu for quick performance, but I was quite impressed with it for what it is. (and fwiw, I had it translate your last paragraph to German, then used google translate back to english: "I wish they had handled this clearly and transparently via an opt-in mechanism—not enabled by default—that explains what Mistral is (not a major American cloud company, but a relatively small French startup) and that your prompts and LLM activities are sent to their servers. I also wish there were documentation explaining how the data is handled and stored in a way that inspires trust.").
- walrus01 17d ago
  How much RAM does it take up in total? I'll have to give that a try on one of my test systems. Looking at a somewhat randomly chose GGUF quantization of it, looks like just under 5GB on disk in Q4, so RAM usage somewhere around 5-6GB? https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF
  - nullc 17d ago
    That sounds about right for Q4. it's an extremely sparse MOE, so there is some odds of acceptable performance using a smaller in-memory cache and the rest on flash. ... I don't have a setup to test that right now. (Of course, if translation is all you want much smaller models will work. Ling-tiny can do summarization, dom manipulation, scripting, etc. too).
- hugodan 17d ago
  Andreessen Horowitz led Mistral's €385M Series A in December 2023.
  - u8080 16d ago
    Is that like 2-4 RTX9000 cards price rn?
ryukoposting 17d ago
I'm worried that the middle could fall out of the computing market across the board. If you can afford to keep up with the upgrade treadmill, you'll get private, local inference capabilities. If you can't afford to stay on the treadmill, you'll be stuck with whatever cloudshit malware Silicon Valley wants to foist on you. I acknowledge that this is already the case, to some extent. The cheapest laptops at Best Buy are crammed with the most preinstalled malware. That's been the case for, what, 25 years? But you've always been able to wipe that cheap laptop and make it into a much more capable, trustworthy machine. Well, assuming LLMs do become a pervasive part of the computing experience, what happens to the cheap laptops? Do all computers get more expensive to accommodate local inference? Does the rift between the everyday user's experience and the savvy user's experience grow even wider than it already is? Neither outcome seems good for the average joe who just needs to check his email.
- walrus01 17d ago
  Remember how in like 1999/2000 Sun was trying to predict that everyone's computer would be some form of thin terminal in the future? Turns out they were very wrong on the part about it running on Sun server back-end infrastructure, but that same general purpose has now been accomplished through other methods where a lot of people do basically EVERYTHING inside a web browser tab to some external cloud service. Now add the need for external inference because very few random consumers are going to buy a $3000 laptop when they can get the $600 laptop at Best Buy, and that trend further escalates.
- tyre 17d ago
  This has always been the case, forever. You have to pay for a product or service. How you do so can be with cash or your data/body/vote/eyeballs/indirect discretionary purchases. The amount of work that can be done funded by foundations and free work is nowhere close to what people want.
  - gremlinunderway 16d ago
    Oh give me a break. This argument that people's objections to advertising comes from some Pollyanna naivety over things being free is nonsense. Tell me the last time you have ever seen a company be upfront and explicitly offer a free tier where they tell you upfront exactly what they are harvesting about you and for what purpose (and no, "improving user experience" isn't being upfront) but also offer you a paid version where they explicitly promise not to do that. "If the product is free then you are the product" is supposed to be a cautionary observation, not an axiomatic proscription. Does anyone actually trust companies that offer paid services to not harvest their data? Whenever I see this trope I think to myself that whatever lack of regulation, oversight and enforcement lead to that being okay would equally allow for them to both take my money and harvest all my of data anyways. This is akin to people who criticise socialised medicine or services and pejoratively characterize supporters as just wanting "free stuff", as if the concept of collective payment is naive or something
repeekad 17d ago
But apple said their AI only falls back to the private cloud when it has to? Just kidding, near every request needs to fallback, because a phone can't actually run a real LLM, no matter how many "neural cores" it has..
smsm42 17d ago
> that it's not some big American cloud company but a relatively small startup in France How it's any better? Small companies can be bought by big companies. French government is as capable of trampling over their citizen's privacy the moment they feel they need it as US government is. And putting a squeeze on a small startup is way easier than on a major cloud company (not that either is particularly hard). Also, OpenAI used to be an idealistic non-profit one day too, then it started to smell trillions and all that went of of the window. > is not going to be nearly as capable as an off-device inference system. I rarely need PhD-level research into my browsing history. I'm not going to solve millennium problems on my bookmarks. The tasks that I will realistically need are well within capacity of most very basic local models. Maybe they'd be a bit slower, who cares.
- mikae1 17d ago
  > How it's any better? Small companies can be bought by big companies. This. I've come to view a startup as a company without a business model, doing everything to get acquired by a mega corp that will finally squeeze the juice out of the userbase (The lack of a viable business model applies to some mega corps too)
- bluebarbet 17d ago
  >How it's any better? Small companies can be bought by big companies. French government is as capable of trampling over their citizen's privacy the moment they feel they need it as US government is. It's better for Europeans because the company in question is European and so will not export their data to New Jersey. And because any success it has will presumably better benefit France and Europe.
  - smsm42 16d ago
    > It's better for Europeans because the company in question is European and so will not export their data to New Jersey There aren't many government institutions in New Jersey that would take a lot of interest in anything Frenchmen are doing. There are a lot of institutions in France that would be interested in anything Frenchmen are doing, if it contradicts what the French government wants to be happening. The main threat to citizen's privacy always comes from the government closest to them, the government overseas has its own citizens to worry about and pays much less attention to foreign citizens on foreign land. There could be exceptions, true, but as a rule, if you look into New Jersey, you'd sooner find mass surveillance of American citizens than mass surveillance of the French. Same goes for commercial interests. If I want to run targeted ads in New Jersey, I want to have profiles of New Jersey people, not French people. So buying data from France in New Jersey would not be a routine occurrence, but buying local data would be.
redox99 17d ago
> But I also think that the state of the art in small LLM and user device capabilities aren't there yet to put a "good enough to be actually useful" local-only LLM as a prepackaged thing in a mass market distributed browser. Then just allow it to be enabled on high end devices? But it must be local only. As hardware advances and people upgrade, more people will be able to turn on the feature.
- deleted 17d ago
  [deleted]
doctorpangloss 17d ago
Every ounce of RAM and spare cycle should be used.
Fnoord 17d ago
> small LLM Read this again, slowly. > You don't want a browser that takes 10GB of extra RAM (on top of the memory hog that is having just 3 or 4 complex tabs open on its own already) and pegs your CPU at 99% usage for minutes at a time. But I do, especially when the choice is either having it locally, remotely, or not at all. I also indeed do nit my CPU used for such, but GPU. I could even run a 8 GB model on a remote (but still local network, on-prem) NPU. There's one caveat though: if you are gaming and browsing.
einpoklum 17d ago
> I am not trying to defend Mozilla doing this Well, you kind of are though. > I wish they had just done this with a very clear up front opt in If a local model is not realistic, then this should not have even been an in-your-face opt-in, but at most some add-on. Of course, their telemetry isn't even opt-out, so even the opt-out for the Mistral thing is kind of disingenuous on their part, since they get a bunch of information from us in other ways. (sigh) Ah, Mozilla has gone down such a dark path over the years. Too bad.
m4rtink 17d ago
If the thing you wan to do is not possible without totally compromising your ideals, maybe don't do it ?
hikashop-nicola 14d ago
[dead]

jimmydoe 17d ago

Agree, advocating for Firefox developing features only for rich hobbyists(people who can afford large RAM and GPU) is absurd.

Barbing 17d ago
>advocating for Firefox developing features only for rich hobbyists ?: >marketing pages aren't candid enough to clearly explain
- well_ackshually 17d ago
  The only people having an expectation of translations being done locally is exactly the nerds that keep whining that it's not using a local model. Every single normal person, when presented with a "translate" button either know it's going online, or don't care about it. Begging the purists to run away from Firefox at this point so they can stop wasting everyone's time. Your demands for examplarity and whining about money not going ONLY to firefox and jerking yourselves on Servo was not enough, now you want to restrict the browser to owners of an RTX5080 if they want to use it?
  - pessimizer 17d ago
    You're talking out of your ass. It is so ironic that "normal people don't care about privacy, just you nerds" is an HN meme. Everybody I know cares, often in an extreme way, and I barely hang out with technical people in real life. I've met people so irrational and nontechnical about modern invasions of privacy that they think that it is being driven by demons. The reason this is a meme on HN is because tons of HN posters are people who spend a lot of effort trying to invade people's privacy and to come up with new ways of concealing that fact. Virtually every sleazebag trying to hide things in ToSes, updates, and telemetry has been an HN poster. And they're pretending that they speak for normal people, because they are scumbags, think normal people are animals that will do anything that they can get away with and don't care about any boundary, and that therefore 1) they themselves are normal, and 2) normal people deserve whatever happens to them. This is what comes from making know-nothing Libertarianism/Objectivism a mainstream ideology, ironically at the same moment that Alan Greenspan, a direct Rand acolyte, was admitting that it had failed while the world economy was sliding into the toilet. Intellectual Libertarianism/Objectivism had failed, but the dumb kind had yet to properly rise. I don't know anything except you nerds are worried about nothing, I'm going to do it because I can and nobody is going to tell me what to do, nobody normal cares about this, and if they do let them try and stop me, why do you care anyway... Please just do it and stop talking, you don't have anything to say. I hope all of you end up in cells next to SBF.
    - walrus01 17d ago
      I do not fundamentally disagree with you but it's also an extremely well known phenomenon that average non tech users, in the aggregate of millions of people, will click almost any "yes/I agree/continue/Next" step on a software installer or new user sign up workflow for anything, without reading the ToS. People blithly sign up for all sorts of cloud based things and services without understanding their full ramifications all the time. People such as you are describing and rightfully criticizing are knowingly taking advantage of that. Indeed it's how a lot of malware gets installed too. The person you're responding to is pointing out that a lot of people at the surface level do only appear to care about the results. They put something into google translate, it works, they gets results they are pleased with, they don't put a lot of thought into the fact that the data is going to an external service. That's not an inaccurate description of how a lot of people use their computers these days. The fact that people will click yes/agree/OK on almost anything is how Windows computers got Bonzi Buddy installed on them back in the day, and now it's continued into the cloud-everything era.
LoganDark 17d ago
They're advocating for a CHOICE and for the difference to be explained.

paimapi 17d ago

>But I also think that the state of the art in small LLM and user device capabilities aren't there yet to put a "good enough to be actually useful" local-only LLM so then don't add it in as highly advertised feature until it is. doing things right and living up to your core values is a lot to expect from businesses these days but, at minimum, a non-profit foundation should be able to live up to these goals, yes?

nosioptar 17d ago

I wouldn't be pissed at the knuckle dragging fuckwits who run and work at Mozilla if they'd done this how you suggest in the final paragraph.

horsawlarway 17d ago

I'll ding Mozilla directly then - Since your "we can't do this yet" approach is actually the literal thing Chrome is shipping... https://developer.chrome.com/docs/ai/built-in/overview?_gl=1*f13fhs*_up*MQ..*_gs*MQ..&gclid=CjwKCAjw_KjVBhAHEiwAnC0N9BPrOjdOMIaN4cfoB6keFXL1FIQROQe0yKOOtu667ZzVrJP0WWkkZRoC8IgQAvD_BwE&gbraid=0AAAAAC1d8f6PYo2kmg9AmW0JUdL46Cuhf https://developer.chrome.com/docs/ai/built-in/overview?_gl=1...

odo1242 17d ago
But the built in AI model in Chrome is 4GB of RAM, 4GB of disk space, and is still only used in a couple places

peri-cl 17d ago

Well, it depends on the task, doesn't it? "running shoes I looked at last week" / "Here's what I found in your browsing history:" doesn't need a 119 billion parameter frontier model; it's a RAG problem for the 0.6 B embedding models. That's an example Mozilla offers. Presumably to explain to their users why it's essential they hand over their last week's browsing history for this convenience (but it isn't! Hardly for that!) I feel it's wrong to tell users that it's important and normal to relinquish all control of their—extremely personal—life history, in bulk, in plaintext, to strangers. I agree wholeheartedly that remote server inference is super useful, and that local inference falls far short on many tasks. (I have no objection at all to Mozilla providing a cloud inference feature). What I don't buy is that we must ask users to redraw their personal boundaries so that their most intimate life details, and remote frontier-model inference, overlap. They do not need to overlap. You can accomplish a lot with private local inference with the smallest of models; and you can accomplish a lot on remote servers which aren't privy to everything. If some convenience is lost by not combining the two, well, so be it. I'm sure most people would agree, if all of this was laid out plainly.

walrus01 17d ago
I'm of the opinion that local inference should be done to the greatest extent that is realistically possible, at the earlier time that the hardware/average user platform is capable of doing so. I personally spend a fair bit on kWh extra in my home electrical bill monthly for having a good sized chunk of local inference ability in my house, but that's not a common thing yet. If mozilla is doing things to send users down the path of doing this externally, they need to be much more upfront and transparent with the users about where their data is going, and not bury it in some terms/conditions that only nerds will hunt for.
graemep 17d ago
> "running shoes I looked at last week" / "Here's what I found in your browsing history:" Does that need an LLM at all?
- smsm42 17d ago
  Not really, but it's probably easier to make it on top of LLM than to make specially-purposed tool for it, if we talking in terms of time-to-market effort.

nullc 17d ago

Have you tried Ling-3.0-tiny? It runs fine on CPU-- on a 14700KF gets 40tg/s and 250pp/s and on a ordinary gpu (RTX 4070) does over 200tg/s with no MTP and 7185pp/s. It's certainly not as capable as something that needs a high memory gpu for quick performance, but I was quite impressed with it for what it is. (and fwiw, I had it translate your last paragraph to German, then used google translate back to english: "I wish they had handled this clearly and transparently via an opt-in mechanism—not enabled by default—that explains what Mistral is (not a major American cloud company, but a relatively small French startup) and that your prompts and LLM activities are sent to their servers. I also wish there were documentation explaining how the data is handled and stored in a way that inspires trust.").

walrus01 17d ago
How much RAM does it take up in total? I'll have to give that a try on one of my test systems. Looking at a somewhat randomly chose GGUF quantization of it, looks like just under 5GB on disk in Q4, so RAM usage somewhere around 5-6GB? https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF
- nullc 17d ago
  That sounds about right for Q4. it's an extremely sparse MOE, so there is some odds of acceptable performance using a smaller in-memory cache and the rest on flash. ... I don't have a setup to test that right now. (Of course, if translation is all you want much smaller models will work. Ling-tiny can do summarization, dom manipulation, scripting, etc. too).
hugodan 17d ago
Andreessen Horowitz led Mistral's €385M Series A in December 2023.
- u8080 16d ago
  Is that like 2-4 RTX9000 cards price rn?

ryukoposting 17d ago

I'm worried that the middle could fall out of the computing market across the board. If you can afford to keep up with the upgrade treadmill, you'll get private, local inference capabilities. If you can't afford to stay on the treadmill, you'll be stuck with whatever cloudshit malware Silicon Valley wants to foist on you. I acknowledge that this is already the case, to some extent. The cheapest laptops at Best Buy are crammed with the most preinstalled malware. That's been the case for, what, 25 years? But you've always been able to wipe that cheap laptop and make it into a much more capable, trustworthy machine. Well, assuming LLMs do become a pervasive part of the computing experience, what happens to the cheap laptops? Do all computers get more expensive to accommodate local inference? Does the rift between the everyday user's experience and the savvy user's experience grow even wider than it already is? Neither outcome seems good for the average joe who just needs to check his email.

walrus01 17d ago
Remember how in like 1999/2000 Sun was trying to predict that everyone's computer would be some form of thin terminal in the future? Turns out they were very wrong on the part about it running on Sun server back-end infrastructure, but that same general purpose has now been accomplished through other methods where a lot of people do basically EVERYTHING inside a web browser tab to some external cloud service. Now add the need for external inference because very few random consumers are going to buy a $3000 laptop when they can get the $600 laptop at Best Buy, and that trend further escalates.
tyre 17d ago
This has always been the case, forever. You have to pay for a product or service. How you do so can be with cash or your data/body/vote/eyeballs/indirect discretionary purchases. The amount of work that can be done funded by foundations and free work is nowhere close to what people want.
- gremlinunderway 16d ago
  Oh give me a break. This argument that people's objections to advertising comes from some Pollyanna naivety over things being free is nonsense. Tell me the last time you have ever seen a company be upfront and explicitly offer a free tier where they tell you upfront exactly what they are harvesting about you and for what purpose (and no, "improving user experience" isn't being upfront) but also offer you a paid version where they explicitly promise not to do that. "If the product is free then you are the product" is supposed to be a cautionary observation, not an axiomatic proscription. Does anyone actually trust companies that offer paid services to not harvest their data? Whenever I see this trope I think to myself that whatever lack of regulation, oversight and enforcement lead to that being okay would equally allow for them to both take my money and harvest all my of data anyways. This is akin to people who criticise socialised medicine or services and pejoratively characterize supporters as just wanting "free stuff", as if the concept of collective payment is naive or something

repeekad 17d ago

But apple said their AI only falls back to the private cloud when it has to? Just kidding, near every request needs to fallback, because a phone can't actually run a real LLM, no matter how many "neural cores" it has..

smsm42 17d ago

> that it's not some big American cloud company but a relatively small startup in France How it's any better? Small companies can be bought by big companies. French government is as capable of trampling over their citizen's privacy the moment they feel they need it as US government is. And putting a squeeze on a small startup is way easier than on a major cloud company (not that either is particularly hard). Also, OpenAI used to be an idealistic non-profit one day too, then it started to smell trillions and all that went of of the window. > is not going to be nearly as capable as an off-device inference system. I rarely need PhD-level research into my browsing history. I'm not going to solve millennium problems on my bookmarks. The tasks that I will realistically need are well within capacity of most very basic local models. Maybe they'd be a bit slower, who cares.

mikae1 17d ago
> How it's any better? Small companies can be bought by big companies. This. I've come to view a startup as a company without a business model, doing everything to get acquired by a mega corp that will finally squeeze the juice out of the userbase (The lack of a viable business model applies to some mega corps too)
bluebarbet 17d ago
>How it's any better? Small companies can be bought by big companies. French government is as capable of trampling over their citizen's privacy the moment they feel they need it as US government is. It's better for Europeans because the company in question is European and so will not export their data to New Jersey. And because any success it has will presumably better benefit France and Europe.
- smsm42 16d ago
  > It's better for Europeans because the company in question is European and so will not export their data to New Jersey There aren't many government institutions in New Jersey that would take a lot of interest in anything Frenchmen are doing. There are a lot of institutions in France that would be interested in anything Frenchmen are doing, if it contradicts what the French government wants to be happening. The main threat to citizen's privacy always comes from the government closest to them, the government overseas has its own citizens to worry about and pays much less attention to foreign citizens on foreign land. There could be exceptions, true, but as a rule, if you look into New Jersey, you'd sooner find mass surveillance of American citizens than mass surveillance of the French. Same goes for commercial interests. If I want to run targeted ads in New Jersey, I want to have profiles of New Jersey people, not French people. So buying data from France in New Jersey would not be a routine occurrence, but buying local data would be.

redox99 17d ago

> But I also think that the state of the art in small LLM and user device capabilities aren't there yet to put a "good enough to be actually useful" local-only LLM as a prepackaged thing in a mass market distributed browser. Then just allow it to be enabled on high end devices? But it must be local only. As hardware advances and people upgrade, more people will be able to turn on the feature.

deleted 17d ago
[deleted]

doctorpangloss 17d ago

Every ounce of RAM and spare cycle should be used.

Fnoord 17d ago

> small LLM Read this again, slowly. > You don't want a browser that takes 10GB of extra RAM (on top of the memory hog that is having just 3 or 4 complex tabs open on its own already) and pegs your CPU at 99% usage for minutes at a time. But I do, especially when the choice is either having it locally, remotely, or not at all. I also indeed do nit my CPU used for such, but GPU. I could even run a 8 GB model on a remote (but still local network, on-prem) NPU. There's one caveat though: if you are gaming and browsing.

einpoklum 17d ago

> I am not trying to defend Mozilla doing this Well, you kind of are though. > I wish they had just done this with a very clear up front opt in If a local model is not realistic, then this should not have even been an in-your-face opt-in, but at most some add-on. Of course, their telemetry isn't even opt-out, so even the opt-out for the Mistral thing is kind of disingenuous on their part, since they get a bunch of information from us in other ways. (sigh) Ah, Mozilla has gone down such a dark path over the years. Too bad.

m4rtink 17d ago

If the thing you wan to do is not possible without totally compromising your ideals, maybe don't do it ?

hikashop-nicola 14d ago

[dead]