samuelknight

8 ms·

91.
by samuelknight 9mo ago
I am working on vulnetic.ai, an agentic penetration testing platform.
92.
by samuelknight 9mo ago
I do this all the time in my Claude code workflow: - Claude will stumble a few times before figuring out how to do part of a complex task - I will ask it to explain what it was trying to do, how it eventually solved it, and what was missing
93.
by samuelknight 10mo ago
These complaints are about technical limitations that will go away for codebase-sized problems as inference cost continues its collapse and context windows grow. There are literally hundreds of engineering improvements that we will see alon
94.
by samuelknight 10mo ago
You're Absolutely Right!
95.
by samuelknight 10mo ago
Switching from my 8-core ryzen minipc to an 8-core ryzen desktop makes my unit tests run way faster. TDP limits can tip you off to very different performance envelopes in otherwise similar spec CPUs.
96.
by samuelknight 10mo ago
It's good to be skeptical of new ideas as long as you don't box yourself in with dogmatism. If you're young you do this by looking at the world with fresh eyes. If you are experienced you do it by identifying assumptions and
97.
by samuelknight 10mo ago
This is an interesting experiment that we can summarize as "I gave a smart model a bad objective", with the key result at the end "...oh and the app still works, there's no new features, and just a few new bugs." No
98.
by samuelknight 11mo ago
My startup builds agents for penetration testing, and this is the bet we have been making for over a year when models started getting good at coding. There was a huge jump in capability from Sonnet 4 to Sonnet 4.5. We are still internally t
99.
by samuelknight 11mo ago
I use .md to tell the model about my development workflow. Along the lines of "here's how you lint", "do this to re-generate the API", "this is how you run unit tests", "The sister repositories are cl
100.
by samuelknight 11mo ago
"Gemini 3 Pro Preview" is in Vertex
101.
by samuelknight 11mo ago
I look at LLMs with an engineering mindset. It is an intelligence black box that goes in a tool box with the old classical algorithms and frameworks. In order to use it in a solution I need to figure out: 1) Whether I can give it informatio
102.
by samuelknight 11mo ago
My startup is building agents for automating pentesting. We started experimenting with Llama 3.1 last year. Pentesting with agents started getting good around Sonnet 3.5 v1. The switch from Sonnet 4 to 4.5 was a huge step change. One of our
103.
by samuelknight 11mo ago
That's what I thought when starting and it functions so poorly that I think they should remove it from their docs. You can enforce a schema by creating a tool definition with json in the exact shape you want the output, then set "
104.
by samuelknight 11mo ago
LLM 'neurons' are not single input/single output functions. Most 'neurons' are Mat-Vec computations that combine the products of dozens or hundreds of prior weights. In our lane the only important question to ask is
105.
by samuelknight 1y ago
You can address the issue by putting the report and the code base in a sandbox with an agent that tries to reproduce it. If it can't reproduce it then that should be a strike against the reporter. OSS projects should absolutely ban acc
106.
by samuelknight 1y ago
Microcenter does this too. When you are in the checkout you can see the cashier using a DOS interface to ring up the items.
107.
by samuelknight 1y ago
(1) JSON requires lots of escape characters that mangle the strings + hex escapes and (2) it's much easier for model attention to track when a semantic block begins and ends when it's wrapped by the name of that section <instru
108.
by samuelknight 1y ago
90% as good as Sonnet 4 or 4.5? Openrouter just started reporting, and it's saying Haiku is 2x as fast (60tps vs 125tps) and 2-3x less latent (2-3s vs 1s)
109.
by samuelknight 1y ago
Sonnet 4.5 is an excellent model for my startup's use case. Chatting to Haiku it looks promising too, and it may be great drop in replacement for some of inference tasks that have a lot of input tokens but don't require 4.5-level
110.
by samuelknight 1y ago
No. I can prototype in 20 mins things that would have taken me a day before.
111.
by samuelknight 1y ago
There is a deep literature on this in the High Performance Computing (HPC) field, where researchers traditionally needed to design simulations to run on hundreds to thousands of nodes with up to hundreds of CPU threads each. Computation can
112.
by samuelknight 1y ago
I disagree with this model because it assumes processing occurs at a point and memory is (optimally) distributed across space around it in every direction in an analog to a Von Neumann CPU architecture. However it is entirely possible to di
113.
by samuelknight 1y ago
I opened the CLI for about 10 seconds, the "Auto Update" status flashed. Then I restarted and it was version 2.0
114.
by samuelknight 1y ago
You are vastly underestimating the price decline. To cherrypick one article; in the first two years since GPT 3.5, inference price for the same amount of intelligence has decreased 10x per year according to a study by Andreessen Horowitz