5 ms·

I've been working on a utility that lets me "see through" app windows on macOS [1] (I was a dev on Apple's Xcode team and have a strong understanding of how to

I've been working on a utility that lets me "see through" app windows on macOS [1] (I was a dev on Apple's Xcode team and have a strong understanding of how to do this efficiently using private APIs). I wondered how Claude Code would approach the problem. I fully expected it to do something most human engineers would do: brute-force with ScreenCaptureKit. It almost instantly figured out that it didn't have to "see through" anything and (correctly) dismissed ScreenCaptureKit due to the performance overhead. This obviously isn't a "frontier" type problem, but I was impressed that it came up with a novel solution. [1]: https://imgur.com/a/gWTGGYa https://imgur.com/a/gWTGGYa

stavros 7mo ago
What was the solution?
- LatencyKills 7mo ago
  Well, I'm not going to share either solution as this is actually a pretty useful utility that I plan on releasing, but the short answer is: 1) don't use ScreenCaptureKit, and 2) take advantage of what CGWindowListCreateImage() offers through the content server. This is a simple IPC mechanism that does not trigger all the SKC limitations (i.e., no multi-space or multi-desktop support). In fact, when using SKC, the user cannot even minimize the "watched" window. Claude realized those issues right from the start. One of the trickiest parts is tracking the window content while the window is moving - the content server doesn't, natively, provide that information.
  - stavros 7mo ago
    Huh, Claude one-shotted it out of a single message from me. Man, LLMs have gotten good.
    - LatencyKills 7mo ago
      No it didn't. Like I said... it may have gotten something that worked but there is no way Claude got it to work while supporting multi-spaces, multi-desktops, and using under 2% cpu utilization. My solution can display app window content even when those windows are minimized, which is not something the content server supports. My point was that Claude realized all the SKC problems and came up with a solution that 99% of macOS devs wouldn't even know existed.
      - TeMPOraL 7mo ago
        > it may have gotten something that worked but there is no way Claude got it to work while supporting multi-spaces, multi-desktops, and using under 2% cpu utilization. Maybe, but that's the magic of LLMs - they can now one-shot or few-shot (N<10) you something good enough for a specific user. Like, not supporting multi-desktops is fine if one doesn't use them (and if that changes, few more prompts about this particular issue - now the user actually knows specifically what they need - should close the gap).
      - jacquesm 7mo ago
        And now it does.
        LatencyKills 7mo ago
        Sorry, "now it does", what?
        TeMPOraL 7mo ago
        The things it didn't, which you then helpfully spelled out.
        LatencyKills 7mo ago
        Do you believe my brief overview of the problem will help Claude identify the specific undocumented functions required for my solution? Is that how you think data gets fed back into models during training?
    - RileyJames 7mo ago
      It one-shotted what exactly? Because LatencyKills is clearly describing a broader set of requirements related to their solution.
skc 7mo ago
That's actually pretty cool. What made you think of doing this in the first place?
- LatencyKills 7mo ago
  Thanks! I've been doing a lot of work on a laptop screen (I normally work on an ultrawide) and got tired of constantly switching between windows to find the information I need. I've also added the ability to create a picture-in-picture section of any application window, so you can move a window to the background while still seeing its important content. I'll probably do a Show HN at some point.
saagarjha 7mo ago
Why is ScreenCaptureKit a bad choice for performance?
- LatencyKills 7mo ago
  Because you can't control what the content server is doing. SCK doesn't care if you only need a small section of a window: it performs multiple full window memory copies that aren't a problem for normal screen recorders... but for a utility like mine, the user needs to see the updated content in milliseconds. Also, as I mentioned above, when using SCK, the user cannot minimize or maximize any "watched" window, which is, in most cases, a deal-breaker. My solution runs at under 2% cpu utilization because I don't have to first receive the full window content. SCK was not designed for this use case at all.
  - saagarjha 7mo ago
    It's been a while since I looked at this but I'm not entirely sure I agree with this. ScreenCaptureKit vends IOSurfaces which don't have copies besides the one that happens to fill the buffer during rendering. I'm not entirely sure what other options you have that are better besides maybe portal views.
    - LatencyKills 7mo ago
      > but I'm not entirely sure I agree with this I worked on the AVC team and built the original SCK Instruments plugin for performance monitoring. I'm assuming you aren't talking about ring-0 (which is where the performance hit occurs). That said, if you want your users to be able to minimize/maximize any "watched" window, ScreenCaptureKit is a non-starter. The OBS team has been asking Apple to remove that restriction for years. Here's a more real-world scenario [0] where Seymour has to handle more than a single window. I can cycle through the entire z-order at 60 fps while capturing all of the content windows. In fact, Seymore can display the contents of minimized windows, which the content server doesn't even support natively. BTW, this quick demo was done using a debug build. The release build can run at < 4% cpu utilization with a dozen windows active and has full multi-monitor and multi-space support. Also, remember that SCK pays no attention to windows that are hidden; something that Seymour has to do constantly. Here's something else you can't do with SCK: picture-in-picture windows [1] that can exist even when the source window is hidden. This is super helpful when watching builds or app logs on larger monitors. No more command+tabbing to babysit things. [0]: https://imgur.com/a/cfgrD0y https://imgur.com/a/cfgrD0y [1]: https://imgur.com/a/jiR3GQ0 https://imgur.com/a/jiR3GQ0
Yizahi 7mo ago
Was it a novel solution for you or for everyone? Because that's a pretty big difference. A lot stuff novel for me would be something someone had been doing for decades somewhere.
- LatencyKills 7mo ago
  Unless you worked on the macOS content server directly you’d have no idea that my solution was even possible. That fact that Claude skipped over all the obvious solutions is why I used the word novel.
  - gf000 7mo ago
    How confident are you that this knowledge was not part of the training data? Was there no stackoverflow questions/replies with it, no tech forum posts, private knowledge bases, etc? Not trying to diminish its results, just one should always assume that LLMs have a rough memory on pretty much the whole of the internet/human knowledge. Google itself was very impressive back then in how it managed to dig out stuff interesting me (though it's no longer good at finding a single article with almost exact keywords...), and what makes LLMs especially great is that they combine that with some surface level transformation to make that information fit the current, particular need.
    - aspenmartin 7mo ago
      Do you think AlphaGo is regurgitating human gameplay? No it’s not: it’s learning an optimal policy based on self play. That is essentially what you’re seeing with agents. People have a very misguided understanding of the training process and the implications of RL in verifiable domains. That’s why coding agents will certainly reach superhuman performance. Straw/steel man depending on what you believe: “But they won’t be able to understand systems! But a good spec IS programming!” also a bad take: agents absolutely can interact with humans, interpret vague deseridata, fill in the gaps, ask for direction. You are not going to need to write a spec the same way you need to today. It will be exactly like interacting with a very good programmer in EVERY sense of the word

stavros 7mo ago

What was the solution?

LatencyKills 7mo ago
Well, I'm not going to share either solution as this is actually a pretty useful utility that I plan on releasing, but the short answer is: 1) don't use ScreenCaptureKit, and 2) take advantage of what CGWindowListCreateImage() offers through the content server. This is a simple IPC mechanism that does not trigger all the SKC limitations (i.e., no multi-space or multi-desktop support). In fact, when using SKC, the user cannot even minimize the "watched" window. Claude realized those issues right from the start. One of the trickiest parts is tracking the window content while the window is moving - the content server doesn't, natively, provide that information.
- stavros 7mo ago
  Huh, Claude one-shotted it out of a single message from me. Man, LLMs have gotten good.
  - LatencyKills 7mo ago
    No it didn't. Like I said... it may have gotten something that worked but there is no way Claude got it to work while supporting multi-spaces, multi-desktops, and using under 2% cpu utilization. My solution can display app window content even when those windows are minimized, which is not something the content server supports. My point was that Claude realized all the SKC problems and came up with a solution that 99% of macOS devs wouldn't even know existed.
    - TeMPOraL 7mo ago
      > it may have gotten something that worked but there is no way Claude got it to work while supporting multi-spaces, multi-desktops, and using under 2% cpu utilization. Maybe, but that's the magic of LLMs - they can now one-shot or few-shot (N<10) you something good enough for a specific user. Like, not supporting multi-desktops is fine if one doesn't use them (and if that changes, few more prompts about this particular issue - now the user actually knows specifically what they need - should close the gap).
    - jacquesm 7mo ago
      And now it does.
      - LatencyKills 7mo ago
        Sorry, "now it does", what?
        TeMPOraL 7mo ago
        The things it didn't, which you then helpfully spelled out.
        LatencyKills 7mo ago
        Do you believe my brief overview of the problem will help Claude identify the specific undocumented functions required for my solution? Is that how you think data gets fed back into models during training?
  - RileyJames 7mo ago
    It one-shotted what exactly? Because LatencyKills is clearly describing a broader set of requirements related to their solution.

skc 7mo ago

That's actually pretty cool. What made you think of doing this in the first place?

LatencyKills 7mo ago
Thanks! I've been doing a lot of work on a laptop screen (I normally work on an ultrawide) and got tired of constantly switching between windows to find the information I need. I've also added the ability to create a picture-in-picture section of any application window, so you can move a window to the background while still seeing its important content. I'll probably do a Show HN at some point.

saagarjha 7mo ago

Why is ScreenCaptureKit a bad choice for performance?

LatencyKills 7mo ago
Because you can't control what the content server is doing. SCK doesn't care if you only need a small section of a window: it performs multiple full window memory copies that aren't a problem for normal screen recorders... but for a utility like mine, the user needs to see the updated content in milliseconds. Also, as I mentioned above, when using SCK, the user cannot minimize or maximize any "watched" window, which is, in most cases, a deal-breaker. My solution runs at under 2% cpu utilization because I don't have to first receive the full window content. SCK was not designed for this use case at all.
- saagarjha 7mo ago
  It's been a while since I looked at this but I'm not entirely sure I agree with this. ScreenCaptureKit vends IOSurfaces which don't have copies besides the one that happens to fill the buffer during rendering. I'm not entirely sure what other options you have that are better besides maybe portal views.
  - LatencyKills 7mo ago
    > but I'm not entirely sure I agree with this I worked on the AVC team and built the original SCK Instruments plugin for performance monitoring. I'm assuming you aren't talking about ring-0 (which is where the performance hit occurs). That said, if you want your users to be able to minimize/maximize any "watched" window, ScreenCaptureKit is a non-starter. The OBS team has been asking Apple to remove that restriction for years. Here's a more real-world scenario [0] where Seymour has to handle more than a single window. I can cycle through the entire z-order at 60 fps while capturing all of the content windows. In fact, Seymore can display the contents of minimized windows, which the content server doesn't even support natively. BTW, this quick demo was done using a debug build. The release build can run at < 4% cpu utilization with a dozen windows active and has full multi-monitor and multi-space support. Also, remember that SCK pays no attention to windows that are hidden; something that Seymour has to do constantly. Here's something else you can't do with SCK: picture-in-picture windows [1] that can exist even when the source window is hidden. This is super helpful when watching builds or app logs on larger monitors. No more command+tabbing to babysit things. [0]: https://imgur.com/a/cfgrD0y https://imgur.com/a/cfgrD0y [1]: https://imgur.com/a/jiR3GQ0 https://imgur.com/a/jiR3GQ0

Yizahi 7mo ago

Was it a novel solution for you or for everyone? Because that's a pretty big difference. A lot stuff novel for me would be something someone had been doing for decades somewhere.

LatencyKills 7mo ago
Unless you worked on the macOS content server directly you’d have no idea that my solution was even possible. That fact that Claude skipped over all the obvious solutions is why I used the word novel.
- gf000 7mo ago
  How confident are you that this knowledge was not part of the training data? Was there no stackoverflow questions/replies with it, no tech forum posts, private knowledge bases, etc? Not trying to diminish its results, just one should always assume that LLMs have a rough memory on pretty much the whole of the internet/human knowledge. Google itself was very impressive back then in how it managed to dig out stuff interesting me (though it's no longer good at finding a single article with almost exact keywords...), and what makes LLMs especially great is that they combine that with some surface level transformation to make that information fit the current, particular need.
  - aspenmartin 7mo ago
    Do you think AlphaGo is regurgitating human gameplay? No it’s not: it’s learning an optimal policy based on self play. That is essentially what you’re seeing with agents. People have a very misguided understanding of the training process and the implications of RL in verifiable domains. That’s why coding agents will certainly reach superhuman performance. Straw/steel man depending on what you believe: “But they won’t be able to understand systems! But a good spec IS programming!” also a bad take: agents absolutely can interact with humans, interpret vague deseridata, fill in the gaps, ask for direction. You are not going to need to write a spec the same way you need to today. It will be exactly like interacting with a very good programmer in EVERY sense of the word