3 ms·

> If your AI scaling statement is accurate then the problem will eventually solve itself as organizations that mandated AI usage will start to fall behind their

> If your AI scaling statement is accurate then the problem will eventually solve itself as organizations that mandated AI usage will start to fall behind their non-AI mandating peers. All things being equal, I would agree. Things are not equal though. The slow down can manifest as: needing more developers for the same productivity, lots of new projects to do things like "break the AI monolith into microservices", all the things that a company needs to do when growing from 50 employees to 200 employees. Having a magicly different architecture is kinda just a different reality, too much chaos to always say that one approach would really be different. One thing though, it does often take 2 to 5 years before knowing whether the chosen approach was 'bad' or not (and why). Companies that are trying to scale - almost no two are alike. So it'll be difficult to do a peer-to-peer comparison, it won't be apples to apples (and if so, the sample size is absurdly small). Did architecture kill a company, or bad team cohesion? Did good team cohesion save the company despite bad architecture? Did AI slop wind up slowing things down so much that the company couldn't grow revenue? Very hard to make peer-to-peer comparisons when the problem space is so complex and chaotic. It's also amazing what people and companies can do with just sheer stubbornness. Facebook has over (I hear) 1000+ engineers just for their mobile app. > My experience so far is that if you architect your systems properly AI continues to scale very well with code base size. It's worth noting that the architecture to support sustained AI velocity improvement may not be the architecture that some human architects have previously grown comfortable with as their optimal architecture for human productivity in their organization I fear this is the start of a no-true-scotsman argument. That aside, what is the largest code base size you have reached so far? Would you mind providing some/any insight into the architecture differences for an AI-first codebase? Are there any articles or blog posts that I could read? I'm very interested to learn more where certain good architectures are not good for AI tooling.

CuriouslyC 1y ago
Regarding architecture considerations: AI likes modular function grammars with consistent syntax and interfaces. In practice this means you want a monolithic service architecture or a thin function-as-a-service architecture with a monolithic imported function library. Microservices should be avoided if at all possible. The goal there is to enable straightforward static analysis and dependency extraction. With all relevant functions and dependencies defined in a single codebase or importable module, you can reliably parse the code and determine exactly which parts need to be included in context for reasoning or code generation. LLMs are bad at reasoning across service boundaries, and even if you have OpenAPI definitions the language shift tends to confuse them (and I think they're just less well trained on OpenAPI specs than other common languages). Additionally, to use LLMs for debugging you want to have access to a single logging stream, where they can see the original sources of the logging statements in context. If engineers have to collect logs from multiple locations and load them into context manually, and go repo hopping to find the places in the code emitting those logging statements, it kills iteration speed. Finally, LLMs _LOVE_ good documentation even more than humans, because the humans usually have the advantage of having business/domain context from real world interactions and can use that to sort of contextually fumble their way through to an understanding of code, but AI doesn't have that, so that stuff needs to be made as explicit in the code as possible. The largest individual repo under my purview currently is around 250k LoC, my experience (with Gemini at least) is that you can load up to about 10k LoC functionally into a model at a time, which should _USUALLY_ be enough to let you work even in huge repos, as long as you pre-summarize the various folders across the repo (I like to put a README.md in every non-trivial folder in a repo for this purpose). If you're writing pure, functional code as much as possible you can use signatures and summary docs for large swathes of the repo, combined with parsed code dependencies for stuff actively being worked on, and instruct the model to request to get full source for modules as needed, and it's actually pretty good about it.
- seadan83 1y ago
  The note on LLMs loving documentation is a solid gem, amongst others - thank you for the response.

CuriouslyC 1y ago

Regarding architecture considerations: AI likes modular function grammars with consistent syntax and interfaces. In practice this means you want a monolithic service architecture or a thin function-as-a-service architecture with a monolithic imported function library. Microservices should be avoided if at all possible. The goal there is to enable straightforward static analysis and dependency extraction. With all relevant functions and dependencies defined in a single codebase or importable module, you can reliably parse the code and determine exactly which parts need to be included in context for reasoning or code generation. LLMs are bad at reasoning across service boundaries, and even if you have OpenAPI definitions the language shift tends to confuse them (and I think they're just less well trained on OpenAPI specs than other common languages). Additionally, to use LLMs for debugging you want to have access to a single logging stream, where they can see the original sources of the logging statements in context. If engineers have to collect logs from multiple locations and load them into context manually, and go repo hopping to find the places in the code emitting those logging statements, it kills iteration speed. Finally, LLMs _LOVE_ good documentation even more than humans, because the humans usually have the advantage of having business/domain context from real world interactions and can use that to sort of contextually fumble their way through to an understanding of code, but AI doesn't have that, so that stuff needs to be made as explicit in the code as possible. The largest individual repo under my purview currently is around 250k LoC, my experience (with Gemini at least) is that you can load up to about 10k LoC functionally into a model at a time, which should _USUALLY_ be enough to let you work even in huge repos, as long as you pre-summarize the various folders across the repo (I like to put a README.md in every non-trivial folder in a repo for this purpose). If you're writing pure, functional code as much as possible you can use signatures and summary docs for large swathes of the repo, combined with parsed code dependencies for stuff actively being worked on, and instruct the model to request to get full source for modules as needed, and it's actually pretty good about it.

seadan83 1y ago
The note on LLMs loving documentation is a solid gem, amongst others - thank you for the response.