3 ms·

Majority of my agentic setup is pi / Claude code where every single Chinese models are not as good except commercial 1T models . Local is a pipe dream . If you

by maxdo 4mo ago

Majority of my agentic setup is pi / Claude code where every single Chinese models are not as good except commercial 1T models . Local is a pipe dream . If you can run it cheap occasionally why commercial companies can’t run it cheaper 24/7 and lower the costs ? The answer is simple. Use cases are more demanding and hence you need more from model not less . Sure if you task is to do a narrow labeling task on 1m records small optimized model is good . If you want to do complex things , it shifts with models advancements

hparadiz 4mo ago
This sounds like something someone at IBM in 1986 would say trying to sell their mainframes. "PCs will never be a thing. No one's gonna want a computer." I'm seeing some impressive results from folks that can afford 10k+ GPUs right now. But those GPUs will all be hand me downs in 10 years. So pipe dream? Hmmm...... that's not how this industry works.
- tyre 4mo ago
  Those are not GPUs available on iPhones. Will we get there eventually? Maybe! Maybe we end up with GPU clusters built on the edge (e.g. cell towers) for offloading, maybe it’s never economical, maybe a different model architecture makes it simpler, who knows. But it doesn’t seem anywhere imminent with our current world state.
  - hparadiz 4mo ago
    My computer is 15,000 times faster and costs in inflation adjusted dollars half that of my computer in 1995. There's zero reason to think that won't happen over the next 30 years again. For whatever reason every generations thinks they are the peak. Naw man. You're just a blip at the bottom of the logarithmic chart.
    - wqaatwt 4mo ago
      It has slowed down massively for CPUs at least. e.g. modern CPUs are hardly more than 3-5x faster than those from 10 years ago. There is zero reason to think won’t happen over the next 10 years again.
      - horsawlarway 4mo ago
        This isn't an crazy statement (cpu performance metrics have mostly stalled their meteoric rise from prior to the 2000s) But it also doesn't capture the entire picture. CPU metrics mostly stalled for two reasons. 1. There wasn't much demand for the extra capacity. Even low end cpus from a decade ago are plenty capable for just browsing the web and typing up documents. It takes a novel use-case to drive demand again (or a desire to do things like play new games). 2. The interest in CPU development shifted in response to mobile. Given point #1 and the state of battery development.... the blocker wasn't "performance". It was "performance per watt". And on that metric you couldn't be more wrong. Since ~2005, MIPS per watt has improved 15x to 30x. Also - fun news is that the traditional CPU pipeline really isn't the bottleneck for AI workloads. So we're going to see incredible interest in things like memory bandwidth and other inference related hardware bottlenecks, which haven't already been optimized.
        BeetleB 4mo ago
        > There wasn't much demand for the extra capacity. Even low end cpus from a decade ago are plenty capable for just browsing the web and typing up documents. It stalled before the rise of PC-as-Internet-portal. I bought a high end PC in 2003, and 5 years later the PCs were not much faster - probably not even 2x. Around 2008-2010 was when most people started using PCs as a way to connect to the Internet. It stalled because scaling got a lot more challenging. Not because of lack of demand.
        horsawlarway 4mo ago
        Yes, but it only stalled along a single dimension - Single core clock speed. I was building gaming machines in the early 2000s, I absolutely remember the 4ghz wall that cpus hit. But it wasn't a real wall... because we then got one of the arguably most influential processors ever in the Core 2 duo. Which... blew the limit away by giving you two processors clocked at 2.93 GHz each. And honestly, even then - it was lack of demand (we could go to 4+ghz, but we didn't want to pay the power bill for the rest of the system - the planned pentium 5 was 7-10ghz on paper, but they canceled the project because keeping it fed and cool was too hard for personal desktop machines). Of Note - we did reach these speeds on consumer hardware (ex - in 2012, Andre Yang hit 8.794Ghz on an AMD FX-8350) So it was never "impossible" to keep scaling. It just wasn't worth it compared to going multi-core. --- And maybe it's because I was in my formative years at this time, but you're off by 5+ years with this: > Around 2008-2010 was when most people started using PCs as a way to connect to the Internet. Gmail was a web only email client released in 2004. Wikipedia was released in 2001. Web browsing was very much one of the "killer" apps for computers by the 2000s. What do you think the damn 2000s dot-com bubble crash was? at the risk of aging myself - I was born in '89, and I literally do not remember a time where we didn't have DSL speeds and above (friends houses often still had dial-up until ~2005, though).
    - sgt101 4mo ago
      For me there are a bunch of questions: - was the pause in model scaling a result of the benefits of RL & SFT being easier to access and quicker than scaling, or was it genuinely the result of scaling being low ROI now? - are power densities necessary to provide high quality on device inference possible? Can the best, technically feasible, architectures accomodate T scale models and run them off batteries that fit in your hand? - will thing slow down enough to allow edge depoloyments to realise value vs. centralised deployments. - do edge use cases drive enough revenue to get this to happen? - can local inference make up for model scale? Does that make sense in a latency/power race with the central infrastructure? Is there a sweet spot here? I am not sure about any of the answers...
iwontberude 4mo ago
Keep working on your agentfu because there is a sweet spot with subagents and parallelizable plans. It’s not about better, it’s about efficiency and picking the right model for the job. You can achieve the same results as frontier models with the right type of planning and context management on local Chinese models.
horsawlarway 4mo ago
Because I have a fixed expenditure on my local machine, and I can be absolutely sure of the costs over a long horizon (5+ years, for low end hardware life, 10+ years with moderate care). Not something that's true for cloud costs. Your argument is actually really similar to an argument around the time Uber started kicking into gear and expanding. It went: --- "Why should I own a car when it's actually cheaper to just Uber for all my rides, compared to the cost of buying, maintaining, and insuring a car?" --- And that wasn't an insane argument at that exact moment. Uber was pricing itself in the range of $5-$7 a ride, was novel and high quality. Except take a look around today... Uber in my area went from ~$5 a ride to ~$27 a ride for the same trip. Uber's quality has also degraded quite a bit. It went from primarily high end, new cars with immaculately clean interiors to "average". So want to make a wager on what's going to happen with cloud costs over the next decade for inference? Because my strong hunch is they're going to follow exactly the same trend. They will stop being subsidized, providers WILL downgrade model quality to improve operating costs (and you'll have no control over this outside of enterprise contracts), and companies will start exploring "additional revenue options"... which means they'll shove ads and sponsored content into your results. Is it worth being ~10-18 months behind the latest and greatest to avoid that entire set of shenanigans? I'd vote yes... I pay one time up front, and get usage limited by my hardware for the cost of electricity over a 10 year timeline. That's a decent deal with no surprises. You're welcome to rent, but renting makes you subject to the whims of the owners. They're being very nice right now to attract all the flies. That's not a mistake, and it's absolutely a trap. --- Side note - if you're only able to do labeling tasks with a local model... you're holding something very, very wrong.
nozzlegear 4mo ago
Depends on what you're doing, of course, but for the small and focused tasks where I'm using agentic AI, local models on my M1 Mac Studio are superb.

by maxdo 4mo ago

hparadiz 4mo ago

This sounds like something someone at IBM in 1986 would say trying to sell their mainframes. "PCs will never be a thing. No one's gonna want a computer." I'm seeing some impressive results from folks that can afford 10k+ GPUs right now. But those GPUs will all be hand me downs in 10 years. So pipe dream? Hmmm...... that's not how this industry works.

tyre 4mo ago
Those are not GPUs available on iPhones. Will we get there eventually? Maybe! Maybe we end up with GPU clusters built on the edge (e.g. cell towers) for offloading, maybe it’s never economical, maybe a different model architecture makes it simpler, who knows. But it doesn’t seem anywhere imminent with our current world state.
- hparadiz 4mo ago
  My computer is 15,000 times faster and costs in inflation adjusted dollars half that of my computer in 1995. There's zero reason to think that won't happen over the next 30 years again. For whatever reason every generations thinks they are the peak. Naw man. You're just a blip at the bottom of the logarithmic chart.
  - wqaatwt 4mo ago
    It has slowed down massively for CPUs at least. e.g. modern CPUs are hardly more than 3-5x faster than those from 10 years ago. There is zero reason to think won’t happen over the next 10 years again.
    - horsawlarway 4mo ago
      This isn't an crazy statement (cpu performance metrics have mostly stalled their meteoric rise from prior to the 2000s) But it also doesn't capture the entire picture. CPU metrics mostly stalled for two reasons. 1. There wasn't much demand for the extra capacity. Even low end cpus from a decade ago are plenty capable for just browsing the web and typing up documents. It takes a novel use-case to drive demand again (or a desire to do things like play new games). 2. The interest in CPU development shifted in response to mobile. Given point #1 and the state of battery development.... the blocker wasn't "performance". It was "performance per watt". And on that metric you couldn't be more wrong. Since ~2005, MIPS per watt has improved 15x to 30x. Also - fun news is that the traditional CPU pipeline really isn't the bottleneck for AI workloads. So we're going to see incredible interest in things like memory bandwidth and other inference related hardware bottlenecks, which haven't already been optimized.
      - BeetleB 4mo ago
        > There wasn't much demand for the extra capacity. Even low end cpus from a decade ago are plenty capable for just browsing the web and typing up documents. It stalled before the rise of PC-as-Internet-portal. I bought a high end PC in 2003, and 5 years later the PCs were not much faster - probably not even 2x. Around 2008-2010 was when most people started using PCs as a way to connect to the Internet. It stalled because scaling got a lot more challenging. Not because of lack of demand.
        horsawlarway 4mo ago
        Yes, but it only stalled along a single dimension - Single core clock speed. I was building gaming machines in the early 2000s, I absolutely remember the 4ghz wall that cpus hit. But it wasn't a real wall... because we then got one of the arguably most influential processors ever in the Core 2 duo. Which... blew the limit away by giving you two processors clocked at 2.93 GHz each. And honestly, even then - it was lack of demand (we could go to 4+ghz, but we didn't want to pay the power bill for the rest of the system - the planned pentium 5 was 7-10ghz on paper, but they canceled the project because keeping it fed and cool was too hard for personal desktop machines). Of Note - we did reach these speeds on consumer hardware (ex - in 2012, Andre Yang hit 8.794Ghz on an AMD FX-8350) So it was never "impossible" to keep scaling. It just wasn't worth it compared to going multi-core. --- And maybe it's because I was in my formative years at this time, but you're off by 5+ years with this: > Around 2008-2010 was when most people started using PCs as a way to connect to the Internet. Gmail was a web only email client released in 2004. Wikipedia was released in 2001. Web browsing was very much one of the "killer" apps for computers by the 2000s. What do you think the damn 2000s dot-com bubble crash was? at the risk of aging myself - I was born in '89, and I literally do not remember a time where we didn't have DSL speeds and above (friends houses often still had dial-up until ~2005, though).
  - sgt101 4mo ago
    For me there are a bunch of questions: - was the pause in model scaling a result of the benefits of RL & SFT being easier to access and quicker than scaling, or was it genuinely the result of scaling being low ROI now? - are power densities necessary to provide high quality on device inference possible? Can the best, technically feasible, architectures accomodate T scale models and run them off batteries that fit in your hand? - will thing slow down enough to allow edge depoloyments to realise value vs. centralised deployments. - do edge use cases drive enough revenue to get this to happen? - can local inference make up for model scale? Does that make sense in a latency/power race with the central infrastructure? Is there a sweet spot here? I am not sure about any of the answers...

iwontberude 4mo ago

Keep working on your agentfu because there is a sweet spot with subagents and parallelizable plans. It’s not about better, it’s about efficiency and picking the right model for the job. You can achieve the same results as frontier models with the right type of planning and context management on local Chinese models.

horsawlarway 4mo ago

Because I have a fixed expenditure on my local machine, and I can be absolutely sure of the costs over a long horizon (5+ years, for low end hardware life, 10+ years with moderate care). Not something that's true for cloud costs. Your argument is actually really similar to an argument around the time Uber started kicking into gear and expanding. It went: --- "Why should I own a car when it's actually cheaper to just Uber for all my rides, compared to the cost of buying, maintaining, and insuring a car?" --- And that wasn't an insane argument at that exact moment. Uber was pricing itself in the range of $5-$7 a ride, was novel and high quality. Except take a look around today... Uber in my area went from ~$5 a ride to ~$27 a ride for the same trip. Uber's quality has also degraded quite a bit. It went from primarily high end, new cars with immaculately clean interiors to "average". So want to make a wager on what's going to happen with cloud costs over the next decade for inference? Because my strong hunch is they're going to follow exactly the same trend. They will stop being subsidized, providers WILL downgrade model quality to improve operating costs (and you'll have no control over this outside of enterprise contracts), and companies will start exploring "additional revenue options"... which means they'll shove ads and sponsored content into your results. Is it worth being ~10-18 months behind the latest and greatest to avoid that entire set of shenanigans? I'd vote yes... I pay one time up front, and get usage limited by my hardware for the cost of electricity over a 10 year timeline. That's a decent deal with no surprises. You're welcome to rent, but renting makes you subject to the whims of the owners. They're being very nice right now to attract all the flies. That's not a mistake, and it's absolutely a trap. --- Side note - if you're only able to do labeling tasks with a local model... you're holding something very, very wrong.

nozzlegear 4mo ago

Depends on what you're doing, of course, but for the small and focused tasks where I'm using agentic AI, local models on my M1 Mac Studio are superb.