3 ms·

> Writing both the client- and server-side for a PDF annotation editor would have taken 60 hours, maybe more. How do you know? Seems to me you’re making the ex

by latexr 1y ago

> Writing both the client- and server-side for a PDF annotation editor would have taken 60 hours, maybe more. How do you know? Seems to me you’re making the exact same estimation mistake of the people in the study. > Instead, a combination Copilot, DeepSeek, Claude, and Gemini yielded a working prototype in under 6 hours Six hours for a prototype using four LLMs? That is not impressive, it sounds insane and a tremendous mess that will take so long to dig out of the prototype stage it’ll effectively require a rewrite. And why are you comparing an LLM prototype to a finished product “by hand” (I surely hope you’re not suggesting such a prototype would take sixty hours)? That is disingenuous and skewing the numbers.

thangalin 1y ago
> How do you know? Seems to me you’re making the exact same estimation mistake of the people in the study. I have over 20 years of web development experience and 40 years of general experience writing software. I wrote the authors and they confirmed my thoughts: "I totally believe it! Per the paper abstract, we find many factors driving results - and one of the factors is how experienced the developers are on the codebase, and how big/complex the codebases are. "Given that this was a new and unfamiliar domain and new codebase, I would expect there to be much more speedup than the domain we studied!" > Six hours for a prototype using four LLMs? They have limits on the number of queries, so I used four different LLMs in tandem to circumvent query limits. I didn't write it four times using four different LLMs. > it sounds insane and a tremendous mess I posted the code. It's well organized, has few (if any) encapsulation violations, sticks to OOP quite well, works, and---if I knew the PDF.js API---would be fairly easy to maintain. Yes, I stand by my claim that writing this annotation editor (PHP, HTML, CSS, and JS) would take me about 60 hours by hand and about 6 hours using the LLMs.

by latexr 1y ago

thangalin 1y ago

> How do you know? Seems to me you’re making the exact same estimation mistake of the people in the study. I have over 20 years of web development experience and 40 years of general experience writing software. I wrote the authors and they confirmed my thoughts: "I totally believe it! Per the paper abstract, we find many factors driving results - and one of the factors is how experienced the developers are on the codebase, and how big/complex the codebases are. "Given that this was a new and unfamiliar domain and new codebase, I would expect there to be much more speedup than the domain we studied!" > Six hours for a prototype using four LLMs? They have limits on the number of queries, so I used four different LLMs in tandem to circumvent query limits. I didn't write it four times using four different LLMs. > it sounds insane and a tremendous mess I posted the code. It's well organized, has few (if any) encapsulation violations, sticks to OOP quite well, works, and---if I knew the PDF.js API---would be fairly easy to maintain. Yes, I stand by my claim that writing this annotation editor (PHP, HTML, CSS, and JS) would take me about 60 hours by hand and about 6 hours using the LLMs.