5 ms·
My startup builds agents for penetration testing, and this is the bet we have been making for over a year when models started getting good at coding. There was
by samuelknight 11mo ago
My startup builds agents for penetration testing, and this is the bet we have been making for over a year when models started getting good at coding. There was a huge jump in capability from Sonnet 4 to Sonnet 4.5. We are still internally testing Opus 4.5, which is the first version of Opus priced low enough to use in production. It's very clever and we are re-designing our benchmark systems because it's saturating the test cases.
- VladVladikoff 11mo agoI have a hotel software startup and if you are interested in showing me how good your agents are you can look us up at rook like the chess piece, hotel dot com
- karlgkk 11mo agoIs it rookhotel.com?
- dboreham 11mo agoI've had similar experience using LLMs for static analysis of code looking for security vulnerabilities, but I'm not sure it makes sense for me to found a start up around that "product". Reason being that the technology with the moat isn't mine -- it belongs to Anthropic. Actually it may not even belong to them, probably it belongs to whoever owns the training data they feed their models. Definitely not me though. Curious to hear your thoughts on that. Is the idea to just try for light speed and exit before the market figures this out?
- apercu 11mo agoThat’s 100% why I haven’t done this - we’ve seen the movie where people build a business around someone else’s product and then the api gets disabled or the prime uses your product as market research and replaces you.
- tharkun__ 11mo agoDoes that matter as long as you've made a few millions and just move on to do other fun stuff?
- pavel_lishin 11mo agoAssuming you make those few millions.
- ryanjshaw 11mo agoThere are armies of people at universities, Code4rena and Sherlock who do this full-time. Oh and apparently Anthropic too. Tough game to beat if you have other commitments.
- apercu 11mo agoI don't entirely fit in to modern capitalism - my values are a little old fashioned - quality, customer service, value, honesty, integrity, sustainability.
- deleted 11mo ago[deleted]
- micromacrofoot 11mo agowild that so many companies these days consider the exit before they've even entered
- blitzar 11mo agothe exit is the business
- rajamaka 11mo agoEvery company evaluates potential risks before starting.
- davidw 11mo agoDepending on how much of a bubble it is. When things really heat up it's sometimes more like "just send it, bro".
- micromacrofoot 11mo agothat's not what this is though, the "exit" is often viewed as "get rich so I don't have to do it anymore"
- NortySpock 11mo agoIt is considered prudent to write a business plan and do some market research if possible before starting a business.
- micromacrofoot 11mo agoyes but traditionally how often was the original business plan "get acquired"? this seems like a new phenomenon?
- Ekaros 11mo agoIsn't that first step? Consider either sustainability or the exit. You start a business either to make a living or make a profit to make a living. At least in most cases. Thus thinking of can you sustain this for reasonable period at least a few years. Or can you flip it at end should be big considerations. Unless it is just a hobby and you do not care about losing time and/or money.
- vngzs 11mo agoHow do you manage to coax public production models into developing exploits or otherwise attacking systems? My experience has been extremely mixed, and I can't imagine it boding well for a pentesting tools startup to have end-users face responses like "I'm sorry, but I can't assist you in developing exploits."
- ceejayoz 11mo agoPoetry? https://news.ycombinator.com/item?id=45991738 https://news.ycombinator.com/item?id=45991738
- aussieguy1234 11mo agoof the adversarial variety
- embedding-shape 11mo agoDivide the steps into small enough steps so the LLMs don't actually know the big picture of what you're trying to achieve. Better for high-quality responses anyways. Instead of prompting "Find security holes for me to exploit in this other person's project", do "Given this code snippet, is there any potential security issues?"
- computerthings 11mo ago[dead]
- paranoidrobot 11mo agoTheir security protections are quite weak. A few months ago I had someone submit a security issue to us with a PoC that was broken but mostly complete and looked like it might actually be valid. Rather than swap out the various encoded bits for ones that would be relevant for my local dev environment - I asked Claude to do it for me. The first response was all "Oh, no, I can't do that" I then said I was evaluating a PoC and I'm an admin - no problems, off it went.
- 11mo ago
- carsoon 11mo agoYeah this latest generation of models (Opus 4.5 GPT 5.1 and Gemini Pro 3) are the biggest breakthrough since gpt-4o in my mind. Before it felt like they were good for very specific usecases and common frameworks (Python and nextjs) but still made tons of mistakes constantly. Now they work with novel frameworks and are very good at correcting themselves using linting errors, debugging themselves by reading files and querying databases and these models are affordable enough for many different usecases.
- justanotherunit 11mo agoIs it the models tho? With every release (mutlimodal etc) its just a well crafted layer of business logic between the user and the LLM. Sometimes I feel like we lose track of what the LLM does, and what the API before it does.
- NitpickLawyer 11mo agoIt's 100% the models. Terminal bench is a good indication for this. There the agents get "just a terminal tool", and yet they still can solve lots and lots of tasks. Last year you needed lots of glue, and two years ago you needed monstrosities like langchain that worked maybe once in a blue moon, if you didn't look funny at it. Check out the exercise from the swe-agent people who released a mini agent that's "terminal in a loop" and that started to get close to the engineered agents this year. https://github.com/SWE-agent/mini-swe-agent https://github.com/SWE-agent/mini-swe-agent
- ACCount37 11mo agoIt's the models. "A well crafted layer of business logic" just doesn't exist. The amount of "business logic" involved in frontier LLMs is surprisingly low, and mostly comes down to prompting and how tools like search or memory are implemented. Things like RAG never quite took off in frontier labs, and the agentic scaffolding they use is quite barebones. They bet on improving the model's own capabilities instead, and they're winning on that bet.
- 11mo ago