anotherpaulg

9 ms·

91.
Sonnet Seems as Good As Ever(aider.chat)
4 points by anotherpaulg 2y ago|0 comments
92.
by anotherpaulg 2y ago
An aider community member made a neovim plugin. It provides the aider style pair programming chat UX. Not the cursor/copilot AI autocomplete function. https://github.com/joshuavial/aider.nvim
93.
by anotherpaulg 2y ago
LLMs produce lower quality code if they’re asked to return it as part of a structured JSON response. This seems to be true for many top models, including those with specialized support for JSON. Benchmarks show that models struggle with syn
94.
LLMs are bad at returning code in JSON(aider.chat)
8 points by anotherpaulg 2y ago|2 comments
95.
by anotherpaulg 2y ago
Not much. I spent $25 on Anthropic in July.
96.
by anotherpaulg 2y ago
As a comparison, I use aider every day to develop aider. Aider wrote 61% of the new code in its last release. It’s been averaging about 50% since the new Sonnet came out. Data and graphs about aider’s contribution to its own code base: htt
97.
by anotherpaulg 2y ago
I’m the author of aider, an ai pair programming tool. I use aider to develop aider, and it keeps track of how much of its own code that it writes. Aider wrote 58% of the code in the last release, and >40% of the previous few. The release
98.
by anotherpaulg 2y ago
The interactive help has the full context of the user's current aider session: what they're working on in the coding chat, active config settings, their OS & shell, etc. Aider can tailor the help response to exactly their ques
99.
by anotherpaulg 2y ago
I feel this pain as a solo maintainer of a somewhat popular open source project. More users brings more questions in GitHub issues and Discord. That’s one reason I added interactive help [0] to aider. Users can type `/help <question
100.
by anotherpaulg 2y ago
The main aider leaderboard includes gemini models. I benchmarked gemma-2-27b-it, but it did so badly that it wasn't worth adding to the leaderboard. https://aider.chat/docs/leaderboards/
101.
Coding with Llama 3.1, New DeepSeek Coder and Mistral Large(aider.chat)
32 points by anotherpaulg 2y ago|7 comments
102.
by anotherpaulg 2y ago
Five noteworthy models have been released in the last few days, with a wide range of code editing capabilities. Here are their results from aider’s code editing leaderboard with Sonnet and GPT-3.5 included for scale. 77% claude-3.5-sonn
103.
by anotherpaulg 2y ago
Llama 3.1 405B instruct is #7 on aider's leaderboard, well behind Claude 3.5 Sonnet & GPT-4o. When using SEARCH/REPLACE to efficiently edit code, it drops to #11. https://aider.chat/docs/leaderboards/
104.
by anotherpaulg 2y ago
Thanks for the reply. I really did like the txtai approach. I am working with markdown files. I think that required me to use Tika & Java based on this note in your docs [0]? Note: BeautifulSoup4 only supports HTML documents, anything
105.
by anotherpaulg 2y ago
I did some prototyping with txtai for the RAG used in aider’s interactive help feature [0]. This lets users ask aider questions about using aider, customizing settings, troubleshooting, using LLMs, etc. I really liked the simplicity of txta
106.
by anotherpaulg 2y ago
The recommended workflow is to just use /undo to revert any edits that you don’t like.
107.
by anotherpaulg 2y ago
The repo map is computed dynamically, based on the current contents of the coding chat. So "importance" is relative to that, and will pull out the parts of each file which are most relevant to the task at hand.
108.
by anotherpaulg 2y ago
I agree that many AI coding tools have rushed to adopt naive RAG on code. Have you done any quantitative evaluation of your wiki style code summaries? My first impression is that they might be too wordy and not deliver valuable context in a
109.
Sonnet Is the Opposite of Lazy(aider.chat)
4 points by anotherpaulg 2y ago|1 comments
110.
by anotherpaulg 2y ago
Claude 3.5 Sonnet represents a step change in AI coding. It is incredibly industrious, diligent and hard working. Unexpectedly, Sonnet’s amazing work ethic caused it to regularly hit the 4k output token limit -- stoping its coding in mid-st
111.
by anotherpaulg 2y ago
This is really interesting. They fine-tune on instances of this sort of task: Do a task using the list of dictionaries below. Dictionary [1] {122: 765, 4548: 1475, 4818: 4782} Dictionary [2] {526: 290, 9205: 9318, 9278: 1565} ... Di
112.
by anotherpaulg 2y ago
I’ve been hoping for awhile to train or fine tune an LLM to reliably output diffs. It’s really fundamental to code editing. https://aider.chat/2023/12/21/unified-diffs.html#unified-dif...
113.
by anotherpaulg 2y ago
No. I just benchmarked against the API at deepseek.com, which doesn't appear to serve the lite model.
114.
Deepseek Coder V2 Beats GPT-4o, Opus(aider.chat)
28 points by anotherpaulg 2y ago|6 comments
115.
by anotherpaulg 2y ago
The new Deepseek Coder V2 model [0] is now atop aider’s code editing leaderboard! It’s worth noting that it is only capable of using aider’s “whole” edit format. This means it returns a modified full copy of each file when it makes changes.
116.
by anotherpaulg 2y ago
Very interesting approach. The folks at rift [0] worked on LSP as an integration point for AI coding. But I think they are focusing on other things now. Do you think the LSP abstraction can support interactions beyond copilot style autocomp
117.
Aider Is SOTA for Both SWE Bench and SWE Bench Lite(aider.chat)
2 points by anotherpaulg 2y ago|1 comments
118.
by anotherpaulg 2y ago
Aider scored 18.9% on the main SWE Bench benchmark, achieving a state-of-the-art result. The current top leaderboard entry is 13.8% from Amazon Q Developer Agent. The best result reported elsewhere seems to be 13.9% from Devin. This result
119.
by anotherpaulg 2y ago
Mainly via existing features for static code analysis, reliable LLM code editing, and pragmatic UX for AI pair programming. Without RAG, vector search, LLM tools, web search and other strongly agentic behaviors.
120.
How Aider Scored SOTA 26.3% on SWE Bench Lite(aider.chat)
4 points by anotherpaulg 2y ago|1 comments

More ›