5 ms·

I'd love to know how you fit smaller models into your workflow. I have an M4 Macbook Pro w/ 128GB RAM and while I have toyed with some models via ollama, I have

by radlad 7mo ago

I'd love to know how you fit smaller models into your workflow. I have an M4 Macbook Pro w/ 128GB RAM and while I have toyed with some models via ollama, I haven't really found a nice workflow for them yet.

philipkglass 7mo ago
It really depends on the tasks you have to perform. I am using specialized OCR models running locally to extract page layout information and text from scanned legal documents. The quality isn't perfect, but it is really good compared to desktop/server OCR software that I formerly used that cost hundreds or thousands of dollars for a license. If you have similar needs and the time to try just one model, start with GLM-OCR. If you want a general knowledge model for answering questions or a coding agent, nothing you can run on your MacBook will come close to the frontier models. It's going to be frustrating if you try to use local models that way. But there are a lot of useful applications for local-sized models when it comes to interpreting and transforming unstructured data.
- mandeepj 7mo ago
  > I formerly used that cost hundreds or thousands of dollars for a license Azure Doc Intelligence charges $1.50 for 1000 pages. Was that an annual/recurring license? Would you mind sharing your OCR model? I'm using Azure for now, as I want to focus on building the functionality first, but would later opt for a local model.
  - philipkglass 7mo ago
    I took a long break from document processing after working on it heavily 20 years ago. The tools I used before were ABBYY FineReader and PrimeOCR. I haven't tried any of the commercial cloud based solutions. I'm currently using GLM-OCR, Chandra OCR, and Apple's LiveText in conjunction with each other (plus custom code for glue functionality and downstream processing). Try just GLM-OCR if you want to get started quickly. It has good layout recognition quality, good text recognition quality, and they actually tested it on Apple Silicon laptops. It works easily out-of-the-box without the yak shaving I encountered with some other models. Chandra is even more accurate on text but its layout bounding boxes are worse and it runs very slowly unless you can set up batched inference with vLLM on CUDA. (I tried to get batching to run with vllm-mlx so it could work entirely on macOS, but a day spent shaving the yak with Claude Opus's help went nowhere.) If you just want to transcribe documents, you can also try end-to-end models like olmOCR 2. I need pipeline models that expose inner details of document layout because I need to segment and restructure page contents for further processing. The end-to-end models just "magically" turn page scans into complete Markdown or HTML documents, which is more convenient for some uses but not mine.
    - D-Machine 7mo ago
      These are some really great explicit examples and links, much appreciated.
    - naasking 7mo ago
      How does GLM-OCR compare to Qwen 3 VL? I've had good experiences with Qwen for these purposes.
      - philipkglass 7mo ago
        Qwen 3 and 3.5 models are quite capable. Perhaps the greatest benefit of GLM-OCR is speed: it's only a 0.9 billion parameter model, so it's fast enough to run on large volumes of complicated scans even if all you have for inference is an entry level MacBook or a low end Nvidia card. Even CPU based inference on basic laptops is probably tolerable with it for small page volumes.
saltwounds 7mo ago
I use Raycast and connect it to LM Studio to run text clean up and summaries often. The models are small enough I keep them in memory more often than not
Bluecobra 7mo ago
I didn’t realize that you can get 128GB of memory in a notebook, that is impressive!
- AzN1337c0d3r 7mo ago
  Most workstation class laptops (i.e. Lenovo P-series, Dell Precision) have 4 DIMM slots and you can get them with 256 GB (at least, before the current RAM shortages). There's also the Ryzen AI Max+ 395 that has 128GB unified in laptop form factor. Only Apple has the unique dynamic allocation though.
  - the_pwner224 7mo ago
    Yep, I have a 13" gaming tablet with the 128 GB AMD Strix Halo chip (Ryzen AI Max+ 395, what a name). Asus ROG Flow Z13. It's a beast; the performance is totally disproportionate to its size & form factor. I'm not sure what exactly you're referring to with "Only Apple has the unique dynamic allocation though." On Strix Halo you set the fixed VRAM size to 512 MB in the BIOS, and you set a few Linux kernel params that enable dynamic allocation to whatever limit you want (I'm using 110 GB max at the moment). LLMs can use up to that much when loaded, but it's shared fully dynamically with regular RAM and is instantly available for regular system use when you unload the LLM.
    - wilkystyle 7mo ago
      What operating system are you using? I was looking at this exact machine as a potential next upgrade.
      - the_pwner224 7mo ago
        Arch with KDE, it works perfectly out of the box. I configured/disabled RGB lighting in Windows before wiping and the settings carried over to Linux. On Arch, install & enable power-profiles-daemon and you can switch between quiet/balanced/performance fan & TDP profiles. It uses the same profiles & fan curves as the options in Asus's Windows software. KDE has native integration for this in the GUI in the battery menu. You don't need to install asus-linux or rog-control-center. For local AI: set VRAM size to 512 MB in the BIOS, add these kernel params: ttm.pages_limit=31457280 ttm.page_pool_size=31457280 amd_iommu=off Pages are 4 KiB each, so 120 GiB = 120 x 1024^3 / 4096 = 31457280 To check that it worked: sudo dmesg | grep "amdgpu.*memory" will report two values. VRAM is what's set in BIOS (minimum static allocation). GTT is the maximum dynamic quota. The default is 48 GB of GTT. So if you're running small models you actually don't even need to do anything, it'll just work out of the box. LM Studio worked out of the box with no setup, just download the appimage and run it. For Ollama you just `pacman -S ollama-rocm` and `systemctl enable --now ollama`, then it works. I recently got ComfyUI set up to run image gen & 3d gen models and that was also very easy, took <10 minutes. I can't believe this machine is still going for $2,800 with 128 GB. It's an incredible value.
echelon 7mo ago
Shouldn't we prioritize large scale open weights and open source cloud infra? An OpenRunPod with decent usage might encourage more non-leading labs to dump foundation models into the commons. We just need infra to run it. Distilling them down to desktop is a fool's errand. They're meant to run on DC compute. I'm fine with running everything in the cloud as long as we own the software infra and the weights. This is conceivably the only way we could catch up to Claude Code is to have the Chinese start releasing their best coding models and for them to get significant traction with companies calling out to hosted versions. Otherwise, we're going to be stuck in a take off scenario with no bridge.
- girvo 7mo ago
  I run Qwen3.5-plus through Alibaba’s coding plan (Model Studio): incredibly cheap, pretty fast, and decent. I can’t compare it to the highest released weight one though.
  - singpolyma3 7mo ago
    Is that https://www.alibabacloud.com/help/en/model-studio/coding-plan https://www.alibabacloud.com/help/en/model-studio/coding-pla... ? I was a bit confused that it seems to be sized in requests not tokens
    - girvo 7mo ago
      Yeah that's the one. I've not managed to get close to the limits that the cheapest plan has. Though I did get to sign up at $3 a month which has been neat, too, seems that's gone now
  - constantinum 7mo ago
    I also want to try Qwen 3.5 plus. I have a doubt, I see almost same pricing for both Qwen and Claude code(the difference being the highest pro plan looks cheaper), and not for the lower plans. Am I missing something, when you say “cheaper” ??
    - 7mo ago
tempaccount5050 7mo ago
Not OP but I had an XML file with inconsistent formatting for album releases. I wanted to extract YouTube links from it, but the formatting was different from album to album. Nothing you could regex or filter manually. I shoved it all into a DB, looked up the album, then gave the xml to a local LLM and said "give me the song/YouTube pairs from this DB entry". Worked like a charm.
aneyadeng 7mo ago
[flagged]

by radlad 7mo ago

philipkglass 7mo ago

It really depends on the tasks you have to perform. I am using specialized OCR models running locally to extract page layout information and text from scanned legal documents. The quality isn't perfect, but it is really good compared to desktop/server OCR software that I formerly used that cost hundreds or thousands of dollars for a license. If you have similar needs and the time to try just one model, start with GLM-OCR. If you want a general knowledge model for answering questions or a coding agent, nothing you can run on your MacBook will come close to the frontier models. It's going to be frustrating if you try to use local models that way. But there are a lot of useful applications for local-sized models when it comes to interpreting and transforming unstructured data.

mandeepj 7mo ago
> I formerly used that cost hundreds or thousands of dollars for a license Azure Doc Intelligence charges $1.50 for 1000 pages. Was that an annual/recurring license? Would you mind sharing your OCR model? I'm using Azure for now, as I want to focus on building the functionality first, but would later opt for a local model.
- philipkglass 7mo ago
  I took a long break from document processing after working on it heavily 20 years ago. The tools I used before were ABBYY FineReader and PrimeOCR. I haven't tried any of the commercial cloud based solutions. I'm currently using GLM-OCR, Chandra OCR, and Apple's LiveText in conjunction with each other (plus custom code for glue functionality and downstream processing). Try just GLM-OCR if you want to get started quickly. It has good layout recognition quality, good text recognition quality, and they actually tested it on Apple Silicon laptops. It works easily out-of-the-box without the yak shaving I encountered with some other models. Chandra is even more accurate on text but its layout bounding boxes are worse and it runs very slowly unless you can set up batched inference with vLLM on CUDA. (I tried to get batching to run with vllm-mlx so it could work entirely on macOS, but a day spent shaving the yak with Claude Opus's help went nowhere.) If you just want to transcribe documents, you can also try end-to-end models like olmOCR 2. I need pipeline models that expose inner details of document layout because I need to segment and restructure page contents for further processing. The end-to-end models just "magically" turn page scans into complete Markdown or HTML documents, which is more convenient for some uses but not mine.
  - D-Machine 7mo ago
    These are some really great explicit examples and links, much appreciated.
  - naasking 7mo ago
    How does GLM-OCR compare to Qwen 3 VL? I've had good experiences with Qwen for these purposes.
    - philipkglass 7mo ago
      Qwen 3 and 3.5 models are quite capable. Perhaps the greatest benefit of GLM-OCR is speed: it's only a 0.9 billion parameter model, so it's fast enough to run on large volumes of complicated scans even if all you have for inference is an entry level MacBook or a low end Nvidia card. Even CPU based inference on basic laptops is probably tolerable with it for small page volumes.

saltwounds 7mo ago

I use Raycast and connect it to LM Studio to run text clean up and summaries often. The models are small enough I keep them in memory more often than not

Bluecobra 7mo ago

I didn’t realize that you can get 128GB of memory in a notebook, that is impressive!

AzN1337c0d3r 7mo ago
Most workstation class laptops (i.e. Lenovo P-series, Dell Precision) have 4 DIMM slots and you can get them with 256 GB (at least, before the current RAM shortages). There's also the Ryzen AI Max+ 395 that has 128GB unified in laptop form factor. Only Apple has the unique dynamic allocation though.
- the_pwner224 7mo ago
  Yep, I have a 13" gaming tablet with the 128 GB AMD Strix Halo chip (Ryzen AI Max+ 395, what a name). Asus ROG Flow Z13. It's a beast; the performance is totally disproportionate to its size & form factor. I'm not sure what exactly you're referring to with "Only Apple has the unique dynamic allocation though." On Strix Halo you set the fixed VRAM size to 512 MB in the BIOS, and you set a few Linux kernel params that enable dynamic allocation to whatever limit you want (I'm using 110 GB max at the moment). LLMs can use up to that much when loaded, but it's shared fully dynamically with regular RAM and is instantly available for regular system use when you unload the LLM.
  - wilkystyle 7mo ago
    What operating system are you using? I was looking at this exact machine as a potential next upgrade.
    - the_pwner224 7mo ago
      Arch with KDE, it works perfectly out of the box. I configured/disabled RGB lighting in Windows before wiping and the settings carried over to Linux. On Arch, install & enable power-profiles-daemon and you can switch between quiet/balanced/performance fan & TDP profiles. It uses the same profiles & fan curves as the options in Asus's Windows software. KDE has native integration for this in the GUI in the battery menu. You don't need to install asus-linux or rog-control-center. For local AI: set VRAM size to 512 MB in the BIOS, add these kernel params: ttm.pages_limit=31457280 ttm.page_pool_size=31457280 amd_iommu=off Pages are 4 KiB each, so 120 GiB = 120 x 1024^3 / 4096 = 31457280 To check that it worked: sudo dmesg | grep "amdgpu.*memory" will report two values. VRAM is what's set in BIOS (minimum static allocation). GTT is the maximum dynamic quota. The default is 48 GB of GTT. So if you're running small models you actually don't even need to do anything, it'll just work out of the box. LM Studio worked out of the box with no setup, just download the appimage and run it. For Ollama you just `pacman -S ollama-rocm` and `systemctl enable --now ollama`, then it works. I recently got ComfyUI set up to run image gen & 3d gen models and that was also very easy, took <10 minutes. I can't believe this machine is still going for $2,800 with 128 GB. It's an incredible value.

echelon 7mo ago

Shouldn't we prioritize large scale open weights and open source cloud infra? An OpenRunPod with decent usage might encourage more non-leading labs to dump foundation models into the commons. We just need infra to run it. Distilling them down to desktop is a fool's errand. They're meant to run on DC compute. I'm fine with running everything in the cloud as long as we own the software infra and the weights. This is conceivably the only way we could catch up to Claude Code is to have the Chinese start releasing their best coding models and for them to get significant traction with companies calling out to hosted versions. Otherwise, we're going to be stuck in a take off scenario with no bridge.

girvo 7mo ago
I run Qwen3.5-plus through Alibaba’s coding plan (Model Studio): incredibly cheap, pretty fast, and decent. I can’t compare it to the highest released weight one though.
- singpolyma3 7mo ago
  Is that https://www.alibabacloud.com/help/en/model-studio/coding-plan https://www.alibabacloud.com/help/en/model-studio/coding-pla... ? I was a bit confused that it seems to be sized in requests not tokens
  - girvo 7mo ago
    Yeah that's the one. I've not managed to get close to the limits that the cheapest plan has. Though I did get to sign up at $3 a month which has been neat, too, seems that's gone now
- constantinum 7mo ago
  I also want to try Qwen 3.5 plus. I have a doubt, I see almost same pricing for both Qwen and Claude code(the difference being the highest pro plan looks cheaper), and not for the lower plans. Am I missing something, when you say “cheaper” ??
  - 7mo ago

tempaccount5050 7mo ago

Not OP but I had an XML file with inconsistent formatting for album releases. I wanted to extract YouTube links from it, but the formatting was different from album to album. Nothing you could regex or filter manually. I shoved it all into a DB, looked up the album, then gave the xml to a local LLM and said "give me the song/YouTube pairs from this DB entry". Worked like a charm.

aneyadeng 7mo ago

[flagged]