3 ms·
Leaderboards are not that useful for measuring real-life effectiveness of the models atleast in my day-today usage. I am currently struggling to diagnose an ip
by manishsharan 2y ago
Leaderboards are not that useful for measuring real-life effectiveness of the models atleast in my day-today usage.
I am currently struggling to diagnose an ipv6 mis-configuration in my enormous aws cloudformation yaml code. I gave the same input to Claude Opus, Gemini and ChatGPT ( o1 and 4o).
4o was the worst. verbose and waste of my time.
Claude completely went off-tangent and began recommending fixes for ipv4 while I specifically asked for ipv6 issues
o1 made a suggestion which I tried out and it fixed it. It literally found a needle in the haystack. The solution is working well now.
Gemini made a suggestion which almost got it right but it was not a full solution.
I must clarify diagnosing network issues on AWS VPC is not my expertise and I use the LLMs to supplement my knowledge.
- blastbking 2y agoSonnet 3.5 as of today is superior to Opus, curious if sonnet could have solved your problem
- nunodonato 2y agoYes. I find it a bit funny how much people care about leaderboards. I see models going up and down, winning this or that benchmark and yet, for me, Sonnet 3.5 still beats the crap out of all of them.
- manishsharan 2y agoI too was puzzled by the response from Claude. I am using the Anthropic workbench with claude-3-5-sonnet-20241022 (latest) But it think it has to do more with the freshness of training data. AWS IPV6 Egress is a new technology from AWS which was introduced only recently. Previously, we had to deploy NAT gateway which supported IPV4. I am assuming claude-3-5-sonnet-20241022 (latest) was not trained on this data.