3 ms·

We need an Eval Leaderboard for LLM assisted Agentic IDEs. The space is getting crowded: New Editors: - Firebase Studio - Zed - OpenHands (OSS Devin Clone)

We need an Eval Leaderboard for LLM assisted Agentic IDEs. The space is getting crowded: New Editors: - Firebase Studio - Zed - OpenHands (OSS Devin Clone) VS Code Forks: - Cursor - Windsurf Editor - Void VS Code Extensions: - Gemini Code Assist - Continue.dev - GitHub Copilot Agent Mode - Cline - RooCode - Kilo Code (RooCode + Cline Fork) - Windsurf Plugin - Kodu.ai Claude Coder (not claude code!) Terminal Agents: - Aider - Claude Code - OpenAI codex Issue Fixing Agents: - SWE-agent

paradite 1y ago
Missing OpenAI Codex cli Also missing a class of non-IDE desktop apps like 16x Prompt and Repo Prompt.
- underlines 1y ago
  Thanks. I added codex. Though, since I specifically mentioned agentic, I wanted to exclude non-agentic tools like prompt builders and context managers that you linked. :) Reason being: my idea of agents is to generalize well enough, so the need for workflow based apps isn't needed anymore. During discovery and planning phase, the agents should traverse the code base with a retrieval strategy packaged as a tool (embedded search, code-graphs, ...) and then add that new knowledge to the plan before executing the code changes.
  - paradite 1y ago
    I don't think it's a black and white distinction between agentic and non-agentic tools. Not to mention tools are constantly evolving and changing. For example, Cursor a year ago was not agentic at all. GitHub Copilot only recently added agentic features. I also think the end game for an agentic tool would not an IDE, because IDE was designed for human workflows, not agents. I wrote about this topic a while ago and made a classification that's probably a bit outdated, but still relevant: https://prompt.16x.engineer/blog/ai-coding-l1-l5 https://prompt.16x.engineer/blog/ai-coding-l1-l5
pzo 1y ago
There is also Trae - vscode fork from bytedance
insane_dreamer 1y ago
JetBrains Junie

paradite 1y ago

Missing OpenAI Codex cli Also missing a class of non-IDE desktop apps like 16x Prompt and Repo Prompt.

underlines 1y ago
Thanks. I added codex. Though, since I specifically mentioned agentic, I wanted to exclude non-agentic tools like prompt builders and context managers that you linked. :) Reason being: my idea of agents is to generalize well enough, so the need for workflow based apps isn't needed anymore. During discovery and planning phase, the agents should traverse the code base with a retrieval strategy packaged as a tool (embedded search, code-graphs, ...) and then add that new knowledge to the plan before executing the code changes.
- paradite 1y ago
  I don't think it's a black and white distinction between agentic and non-agentic tools. Not to mention tools are constantly evolving and changing. For example, Cursor a year ago was not agentic at all. GitHub Copilot only recently added agentic features. I also think the end game for an agentic tool would not an IDE, because IDE was designed for human workflows, not agents. I wrote about this topic a while ago and made a classification that's probably a bit outdated, but still relevant: https://prompt.16x.engineer/blog/ai-coding-l1-l5 https://prompt.16x.engineer/blog/ai-coding-l1-l5

pzo 1y ago

There is also Trae - vscode fork from bytedance

insane_dreamer 1y ago

JetBrains Junie