14 ms·

Frontier AI has broken the open CTF format

• not_a9 5mo ago
  I’m interested in finding out how attack-defense style CTFs are affected by slopping. ENOWARS skorbor will probably significantly differ from the last time around.
• walletdrainer 5mo ago
  >I started playing CTFs in 2021 >and the old game is not coming back For many people the CTF scene was already dead in 2021 because it had turned into something unrecognisable. In reality it’s just different.
  • lukan 5mo ago
    Well, I had to google what CTF means (capture the flag, a hacking competition), so surely cannot judge here, but the text indicates that with AI some things are very different today: "That makes open CTFs pay-to-win. The more tokens you can throw at a competition, the faster you can burn down the board. Specialised cybersecurity models like alias1 by Alias Robotics are becoming less relevant compared to general frontier LLMs. The competition is turning into "who can afford to run enough agents, with enough context, for long enough.""
    • mock-possum 5mo ago
      Isn’t that the bitter lesson in a nutshell? “Specialised cybersecurity models … are becoming less relevant compared to general frontier LLMs.”
    • walletdrainer 5mo ago
      There are two different schools of thought: 1) It’s OK to do just about anything to win a CTF, including installing malware on the organisers computers months before the actual event so you’ll have an easy time stealing the flags. 2) It’s not ok to try and win the CTF with a solution the authors did not intend. Recently the #2 crowd has been winning because the hacking scene has turned corporate and boring. People started to partake in CTFs in the hopes of landing a job(!) CTFs are indeed ruined for those people, I personally don’t mind. For the people in group #1 LLMs change little. Attacking the challenges directly was always a last resort.
      • Karrot_Kream 5mo ago
        Yeah I remember running a few CTFs in school and was always scared (in a good way) about what the players would do to the game's servers. For this reason we also only ran the CTF on the school's network and IT even floated running in an isolated VLAN. The fact that CTFs became a sort of SAT score for getting a security job made me lose interest very early on.
  • deleted 5mo ago
    [deleted]
  • Grimburger 5mo ago
    >Learning about eternal September in May 2026 Hits different doesn't it
  • Retr0id 5mo ago
    I started playing in 2015 or so and had mostly stopped by 2020. Not because I felt it was "dead" exactly but it just wasn't hitting the same for me. By then it wasn't "the winner has the most LLMs", but "the winner has the most members on their team". I merged into one of the mega-teams and it just wasn't fun any more.
• deafpolygon 5mo ago
  Unrelated, but does anyone find this site incredibly hard to read?
  • walletdrainer 5mo ago
    Bizarre font and poor contrast, yep. The text itself being exceedingly long for no obvious reason doesn’t help.
    • lukan 5mo ago
      Poor contrast? White on black? And if you think it was too long, what part would you have shortened? I never knew about the scene and found it interesting to read this personal take on it.
      • swiftcoder 5mo ago
        > White on black? According to Pikka, the paragraph text is Taupe Grey (#92908a) on a Liquorice (#111110) background. That's... pretty far from black and white.
• tromp 5mo ago
  https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurity) https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit... still has no mention of AI, but that will likely change as they increasingly dominate competition.
  • sumeno 5mo ago
    Using AI on CTF is like using a car to get better at the 100 yard dash
    • Choco31415 5mo ago
      Except some people want to win and don't care about their own personal performance.
    • low_tech_love 5mo ago
      Yeah but for the brief time window (which is probably coming to an end right now) where getting your name on the leaderboards was still worth it anyway, because people had not yet realized that the game is over, players will use the car anyway. Now that the game is over, leaderboards are meaningless, so we will figure out ways to move past that (like playing with people who care). But that will change the game in unavoidable ways: the tiny, fragmented scale will give less incentive for creators to come up with massive intricate complex challenges (it would be like hoping Bethesda would make Skyrim for a handful of die hard players). And soon, maybe on a few years, people will invariably start questioning whether it even makes sense to waste their time with this hobby if learning CTF skills is basically useless in an AI dominated world. There are still people learning Assembly for fun, but almost nobody does Assembly programming challenges as a hobby.
• vasco 5mo ago
  My first ever was Stripe CTF in 2012 I think, I still wear the shirt I got (now super fainted) from passing some challenges. I was a student in portugal and remember receiving the shirt for it and thinking, maybe those Americans aren't any better than me and I can compete at the same level. I never got super into security but it gave me the confidence to play in the same field and lose the stupid aura I had that somehow "rich americans" would be better than me at everything because they had better universities or because of Hollywood or something. Sad that another cool thing is lost to AI but I guess kids will learn in other ways.
• chvid 5mo ago
  What is CTF? And why is the cyber security world filled with silly gaming references?
  • throwa356262 5mo ago
    https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurity) https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit... Its a war game reference I guess?
  • mort96 5mo ago
    Capture The Flag is a cybersecurity game where the organizers set up a bunch of intentionally vulnerable computer systems with a "flag" on them, a string that's "supposed to be" secret but is accessible through exploiting the vulnerabilities. This may be a line in /etc/password, a string in memory, a field in a database, whatever. The goal of the game is to hack into the computer systems, find ("capture") the flag, then copy/paste it into the organiser's scoreboard website to prove that you solved that particular challenge. It's pretty fun. Or at least it was, back when you had some sense that your competitors were competing on an even playing field and just beat you because they were better than you. I wouldn't say the name is a "gaming reference", it's just a descriptive name for a game.
• monarx 5mo ago
  used to see some really good CTF videos show up on youtube and now nothing like that shows up on the feed
• susam 5mo ago
  I have normally found any sort of timed technical competition intimidating. Even so, about 6 or 7 years ago, after being persuaded by a colleague, I participated in a few CTFs. I am glad I did, back when this type of thing still meant something. I have kept a screenshot from one of the CTFs that I am quite fond of: https://susam.net/files/blog/ctf-2019.png https://susam.net/files/blog/ctf-2019.png
• deleted 5mo ago
  [deleted]
• eecc 5mo ago
  “solve”, why not solution? Like “spend” and not expenditure, why use the verb as a noun and not care about grammar?
  • deleted 5mo ago
    [deleted]
  • iainmerrick 5mo ago
    They’re shorter. Why so pedantic?
  • sheept 5mo ago
    These examples that you're calling "verbs as a noun" are standard grammar. You can't just invent simplified rules about a language and declare it wrong when the rules fall apart.
  • msm_ 5mo ago
    In addition to what others have said, this usage is very common in the CTF world. "The challenge has no solves", "We just got the first solve" etc are very idiomatic. It would actually look weird to me if this was "solution".
  • tripzilch 5mo ago
    I don't understand the complainiture, it's an improve
• kevinsimper 5mo ago
  You could make it offline and with provided laptops only, just like with the competitive CS2 scene.
  • eastbound 5mo ago
    Since real-life situations involve AI, banning AI would make CTFs just a simple game, not a demonstration of capabilities and talent.
    • loeg 5mo ago
      They always were just a game?
    • mort96 5mo ago
      What do you mean? Solving a CTF challenge demonstrates way more capabilities and talent than just asking a chat bot to solve a CTF challenge.
  • hsbauauvhabzb 5mo ago
    Ctfs need preparation and unconstrained internet, even if you block domains it’s possible to tunnel out
    • belabartok39 5mo ago
      Use jumpbox to access CTF. Disable all wireless for the playing hall.
      • hsbauauvhabzb 5mo ago
        I think you’re forgetting hotspots, or laptops with inbuilt 4/5g
        • swiftcoder 5mo ago
          Faraday cages exist. Finally a use for all those damn SCIFs tech companies were building in the late 2010's...
• rurban 5mo ago
  I don't do CTF's but took part at the security workshop for fun ~2 years with my Android phone only. I was first with the first simple challenge, but then couldnt continue because my phone was just too limited. But I watched what the others did. And a young Indian guy did everything with ChatGPT then. I found it silly, but amusing, because he actually got second. There was no Codex nor Claude then. Nowadays it must be dead for real, because I would solve everything with my agents, as I do in the real world.
• Grimburger 5mo ago
  Very impressed that OP has gone from starting university in 2021 to becoming a Senior Security Engineer. It's an incredibly exciting time in security research in my humble old man opinion. Think the cadence of new exploits is perhaps a good measure of that rather than subjective thoughts by anyone regardless of experience.
  • toraway 5mo ago
    Okay, but none of that is actually responsive to what the article is discussing, which is competitive CTFs. There's not a single criticism of using AI for actual security research in anything they wrote and they mention being a heavy user of GPT-5.5 and GPT-5.5 Pro so belittling the author's experience to defend LLMs wasn't actually necessary.
• himata4113 5mo ago
  I was writing an obfuscator recently, I just had the model deobfuscate and optimize the code back to original and I kept improving the obfuscator until it couldn't. The funny thing is that after all this I also ended up with a really strong deobfuscator and optimizer which is probably more capable than most commercial tools. The solution is just to make CTFs harder, but when do CTFs become too hard? Maybe the problem is that 'hard' CTFs are fundementally too 'simple' where it's just a logic chain and an exhaustive bruteforce towards a solution since there really are limited ways to express a solution in plain sight. Or maybe human creativity has been exhausted and we're not so limitless as we thought. Only time will tell. I had another idea spring to mind: we could hide two flags, one that could only be found by ai agents and not humans or tools written by humans.
  • koolala 5mo ago
    A portion could require astral projection and computers can't do that. Or maybe just a VR mini-game like the 90s always imagined.
    • himata4113 5mo ago
      bringing CTF solutions into the real world is a really good idea! I didn't even think of this until you mentioned it. we have very powerful simulation tools so something like "project a pattern at these angles" wouldn't really work as you could simulate that. I guess something cool is that we can make simulating the solution very expensive, but in real world it would be free since it's analog... As long as simulations take longer than it takes for a human to find a solution it would be a pretty good way to deal with it. I am sure people smarter than me can come up with something. Maybe I was too early to dismiss human creativity.
      • dguest 5mo ago
        Maybe CTF is dead, but there are plenty of fun problems in the real world -- ask any scientist, engineer, or medical researcher. There are a million places where a computer can interact with a non-digital system in a loop. - Tune an FPGA, or a whole data-center, or just a physical computer. - Make a drone fly somewhere. - Design a selective toxin (or anti-toxin). Or, you know, get more people to click on adds. All totally possible to automate.
• amingilani 5mo ago
  I don’t think CTFs are dead, they’ll just evolve. The difficulty level will need to be increased or the rules locked down. Just like sports and racing persist despite the existence of performance enhancing drugs and rocket technology. I just did a CTF where I was in the top 10. It was the first CTF I completed and I used AI because the rules permitted it. That said, I couldn’t solve all challenges. But yes, it was significantly easier now than I last attempted one. Even manually solving with AI assisted assembly interpretation was much easier.
  • mort96 5mo ago
    Increasing the difficulty level is a terrible solution. The problem with CTFs isn't that they're too easy. Making them harder just makes them even less accessible to people who don't cheat. It'd be like seeing people who put hidden electric motors in their bikes during Tour de France and conclude, "oh we just need longer distances and steeper hills".
    • StrauXX 5mo ago
      LLMs don't tend to help much when solving challenges beyond their skill level. Either they one-shot a challenge, or thei are almost useless as a companion for them.
    • viccis 5mo ago
      Exactly. The whole point of CTFs is that you could start on a simple one (CSAW was usually my go to one to recommend) as a complete novice who'd never done a second of computer security work and, after a few days of 8+ hours of running into concepts you hadn't encountered, googling, reading tutorial, practicing, overcoming the challenges to get a flag, etc., you'd come out the other end knowing a solid bit of security practitioner basics and likely whether you'd like to continue. Then you could keep going upwards and onwards. I went from 0 knowledge to a nice job in the field in a year. Raising the difficulty only matters for the (imo) less important part: the dick measuring competition between the very top teams. The actual point of CTFs was usually to keep your skills sharp and stay learning. Eventually you build your own challenges, thereby completing the "have it taught to me, then do it myself, then teach another person" three step process towards mastering concepts. You can just say "let the people who want to learn from it do so" but honestly the entire culture of learning in the US at least is DEAD. We turned "education" into a rote system of maximizing incentives to the extent that that's all the youth know it as, and (increasingly) all educators can do. It's just gone without some kind of major reckoning, and we all know things will just collapse before that happens. The ball is in the court of whatever country can learn how to force its youth to learn the real way and use AI productively only AFTER learning the concepts it's being used to accelerate.
• raphman 5mo ago
  Interesting and well written article that mirrors/foreshadows how LLMs do and will change other scenes. As I don't know much about the CTF scene, I looked for other takes on this topic. Here's an article from 2015 about how tool-assistance already changed CTFs: > Individual skill will undoubtedly be a factor next year. But, I'm left wondering whether next year's DEFCON CTF will tell us anything more than how well-developed each team's tools are (and how well they can interpret the results). https://fuzyll.com/2015/ctf-is-dead-long-live-ctf/ https://fuzyll.com/2015/ctf-is-dead-long-live-ctf/ But there are quite a few recent (2026) articles with the same core message as in the original article, e.g., https://blog.includesecurity.com/2026/04/ctfs-in-the-ai-era/ https://blog.includesecurity.com/2026/04/ctfs-in-the-ai-era/ or https://k3ng.xyz/blog/ctf-is-dead https://k3ng.xyz/blog/ctf-is-dead And here's someone explaining how Claude Max allowed them to win CTFs: > I had always been interested in CTF as one of the only ways people could compete and show off their skill in coding/problem solving on a global scale. It was just too difficult and didn't make sense for me to learn the fundamentals as an electrical engineer. As time went on, I got better and better, and it was hard to tell whether it was because of experience or if it was because of improvements in AI. > I accomplished my goals, and for that reason I'm quitting CTF, at least for now. [...] I'd like to think I highlighted the problem before it became a bigger issue. So, how do we fix this? Teams and challenge authors losing motivation is not good. CTF dying is not good. AI bad. Or is it? https://blog.krauq.com/post/ctf-is-dying-because-of-ai https://blog.krauq.com/post/ctf-is-dying-because-of-ai The only article that saw LLMs as a non-negative force for CTFs was this one. Fittingly, it sounds like LLM output ("Let's be honest", "This is where things get interesting.") and only contains hallucinated references. https://caverav.cl/posts/ctfs-not-dead/ctfs-not-dead/ https://caverav.cl/posts/ctfs-not-dead/ctfs-not-dead/
• utopiah 5mo ago
  Right, the same way that car racing has "broken" jogging. This is so dumb. /s The whole point of competitions is to provide a safe environment thanks to a set of rules all participants AGREE on in order to progress together. If new tools "break" the competition, we change the rules and that's A-OK. CTF isn't a natural phenomenon, if tools change, rules change, simple.
  • swiftcoder 5mo ago
    The only way this actually works is if you move CTF to in-person only. There's no other way to reasonably prevent the whole leaderboard being taken up by whoever spent the most on tokens.
    • utopiah 5mo ago
      Sure, I don't know how to make it work. I just know that DeepBlue didn't kill competitive chess. We simply have at least 3 different rule sets, namely - no computer assistance, which does also mean no mobile on competition, human only - advanced chess with assistance - computer only, no human assistance and arguably chess itself is not doing worst since.
      • swiftcoder 5mo ago
        I think the big difference here, is that organisers of chess tournaments don’t have to design multiple entirely new board games for each competition. When AI can one-shot CTF challenges, you have to develop new challenges in secret for every competition, and they are single-use.
      • metroholografix 5mo ago
        Official chess competitions are taking place under stringent monitoring conditions and even then, with professional reputations on the line, there have been multiple high profile cheating incidents. Amateur online chess on the other hand is besieged by cheaters that use engines, even in casual non-ranked games where there's absolutely nothing to gain besides a pat on the ego. This has drastically changed how the game is played today with lot of players gravitating towards speed chess (bullet and blitz) to compensate. That will thin the herd of cheaters but one still runs into engines on a weekly basis. This is also the tip of the iceberg, with the true scale of the problem being orders of magnitude worse, as someone dedicated enough can use an engine to cheat in a way that's essentially undetectable.
• rqd3 5mo ago
  tldr; adapters took my elo
• deleted 5mo ago
  [deleted]
• r4indeer 5mo ago
  I'm conflicted on the use of AI in CTFs. On the one hand, they are supposed to mirror real-life scenarios, so of course you should be able to use any tool that would be available to you in real life. On the other hand, CTFs are fundamentally a game and a competition which are supposed to be fun and compare and improve ones skill. So when I let an LLM generate the entire solution for me, what's the point anymore? I did not learn anything. I did not work for that place on the leaderboard, I just copied the solution. And worst of all, I did not have any fun. It's boring. So how does using AI as a solver not feel like cheating?
  • deleted 5mo ago
    [deleted]
• zzvimercm 5mo ago
  [flagged]
• baq 5mo ago
  Replace ‘CTF’ with ‘high school’ or ‘university’ and you’ve described the total slow motion collapse of education; the only saving grace is that most of it requires in person presence. We’ve figured out the human replacement pipeline it seems, but we haven’t figured out the eduction part. LLMs can be wonderful teachers, but the temptation to just tell it ‘do it for me’ is almost impossible to resist.
  • daniel3303 5mo ago
    [flagged]
  • daymanstep 5mo ago
    Wonderful teachers that give unreliable information with total confidence?
    • k__ 5mo ago
      Anti-intellectualism is at it again, hu?
    • Bawoosette 5mo ago
      To be fair, that was much of my actual experience with human professors in university.
      • IshKebab 5mo ago
        Yeah one of my teachers was able to identify which high school I had come from due to something I had been mistaught.
      • renticulous 5mo ago
        Veritasium proved that in a difficult challenge. A Physics Prof Bet Me $10,000 I'm Wrong https://www.youtube.com/watch?v=yCsgoLc_fzI https://www.youtube.com/watch?v=yCsgoLc_fzI
    • victorbjorklund 5mo ago
      Like humans.
• SoylentOrange 5mo ago
  Great article, well written, and good analogy to chess. I’ve been playing competitive chess most of my adult life and I think that the solution lies in how chess dealt with this problem: Explicit ELO measurements with some cheating detection. AI assistance wholly banned. As you climb the ELO ladder, detection gets more onerous. At top level during online events, anti cheating teams require the use of both monitoring software and multiple cameras. Idea is that you can cheat pretty easily at the lowest levels but it gets less easy the higher you go. This allows for better feeding into the truly elite competitions. I think chess’s very firm stance that AI is never allowed in competition (neither online nor in person), rather than CTF’s acceptance, was the right call.
  • salt4034 5mo ago
    Yes, chess has been dealing with AI for decades at this point, and it's amusing/frustrating that so many other communities are deciding to re-discover everything from scratch, rather than just learn from the chess experience. If CTF is a player-vs-player event, then AI should just be banned outright, otherwise it will devolve into AI-vs-AI, which is just not an interesting competition format, as we learned in chess. Compared to FIDE top events (which bans AI), only a tiny niche audience actually watches the Top Chess Engine Championship (AI-centered). It turns out what we care about is not whether chess can be solved by any means available, but what are the limits of the human mind in learning chess. Pretty much all chess coaches/educators also warn against relying heavily on AI during learning; engines only give you an illusion of understanding.
• 3qw128 5mo ago
  The article is the thickest of AI slop. Don't believe anything.
  • sevindob 5mo ago
    ikr, if bro can't be bothered to write an article himself then anything he says is automatically suspect
• hoyd 5mo ago
  «That feedback loop is breaking. If the visible scoreboard is dominated by teams using AI, a beginner is pushed toward using AI before they have built the instincts the AI is replacing. That is an anti-pattern. It prevents active learning, and active struggle is the bit that actually teaches you. It is also completely demotivating to put in real effort and see no visible progress because the ladder above you has been automated.» This stands out to me, and speaks perhaps broader than the article itself? I’m sure this has been in the spotlight before, but well put for many areas I think.
  • black_knight 5mo ago
    I see this with beginner programming students at university. They get AI to help them with assignments, with the intention of learning, but ultimately they do not get the understanding they would have if they had done the assignment themselves. Then they are at a deficit for learning more advanced topics. My fear is that they never get to the level they need to be at to create good software even with the help of AI. So, although an expert with AI can create great software, that is not where we end up. In stead we will have vibe coded messes by people who barely have any grasp of what is going on.
• chrismorgan 5mo ago
  Meta: this was submitted with the article’s title “The CTF scene is dead” which I found very easy to understand. It has just been updated to use the subtitle’s first sentence, “Frontier AI has broken the open CTF format”. I find that much harder to grasp, rather like a garden-path sentence. My immediate thoughts were that “Frontier” was a company name, and that there was some file format named CTF. If you don’t know about Capture The Flag contests, the change doesn’t help. If you do, I think the change makes it worse.
  • IanCal 5mo ago
    If it helps I understand the second much better and feels less clickbaity and includes more info. I do agree with the points you made about the confusion although I find frontier a term used in this area a lot, “frontier AI models have” would probably resolve that.
    • jofzar 5mo ago
      Imo frontier is too niche and specific, if you know what a frontier model means then it's fine, but if you don't then it's negative/detrimental to the title. "new" does the same thing and is probably just a better descriptor then frontier
      • rockskon 5mo ago
        But then you're not acting as a billboard promoting AI. Isn't that partly the point?
      • jack_pp 5mo ago
        if you are on HN and have no idea what "frontier model" would mean maybe it's time you found out.
        • hbbio 5mo ago
          I also misread the updated title. "Frontier models break the open CTF format" is good "Frontier AI..." means wtf is Frontier AI. Because of course it exists (just googled it): https://frontierai.company/ https://frontierai.company/
• motbus3 5mo ago
  I think soon there will be ways to trick this models and I think when it happens it will be yet another layer like aslr These models seems completely unbeatable only in the ads. There are 100+ times way someone puts Hindi Yoda talk In Morse Code and it goes nuts. The reason they are going to hard for PR Marketing on this is because they know it is a matter of time.
  • Avamander 5mo ago
    The more you obfuscate a topic against LLMs the lower the educational value of a challenge. The only things that works is novelty and obscurity. LLMs still suck with things mentioned in the footnotes of datasheets and manuals, things that deviate in subtle ways, unique constructions that alter something very very common. It's hard for LLMs to avoid common pitfalls in terms of making assumptions, while staying on track.
• SirHumphrey 5mo ago
  Competitive programming scene always included offline competition and with AI they are becoming more important (and in general they were more fair even before). If CTFs are to survive, they should probably try to adopt this strategy. You could even go so far that anything loaded on your computer is fair game, but not more than that (certain competitive programming competition for example allow unlimited amount of paper material - for CTFs you probably need much more than that, therefore electronic).
• virtualritz 5mo ago
  Chess and Go are not dead just because Ai got better than humans at these games. What am I missing here?
  • hnlmorg 5mo ago
    You aren’t allowed to use tools to play competitive Chess / Go but that are required for solving CTF.
  • jofzar 5mo ago
    These have very strong anti cheats and in person is very stringent on no electronics. Its not really a good comparison
  • lugu 5mo ago
    Read the article.
    • artninja1988 5mo ago
      https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html "Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that"."
    • virtualritz 5mo ago
      I read the article. Their chess section makes no sense as in "why this wouldn't work for CTF". But I don't know enough that's why I asked. I imagine one could do CTF in public, machines you work on vetted/prepared to some spec, yada yada. If chess and Go can do it why can't CTF? That was my question when I wrote "what am I missing here".
  • aymenfurter 5mo ago
    Chess banned engines from competition. CTFs can't really do that because you need internet access and tooling to play.
• vagab0nd 5mo ago
  This left a strange feeling. The article reads as extremely bleak. But from a different perspective this is extremely bullish for AI.
  • Avamander 5mo ago
    LLMs managing the "coloring book" equivalent of something is not bullish for the "art" version of something. The intent for most CTFs is to provide a meaningful challenge that concerns a single topic without introducing noise that wastes time. Of course a training exercise is easier to complete for an LLM.
  • kangalioo 5mo ago
    I agree. The article mourns the death of pentesting as an art form due to automation. But you could also celebrate the death of pentesting as an arduous necessary evil due to automation
• lokrian 5mo ago
  Is AI also superior to humans at black box challenges and attacking actual targets on the internet? That seems like a really important question.
  • Avamander 5mo ago
    No, the search space is much more vast and the feedback loop almost nonexistent. The reason LLMs can do CTFs so well is partially because the challenges are usually designed to avoid wasting time and to introduce a single concept without noise.
• slurpyb 5mo ago
  How to motivate cybersec best outcome reddit 2026 no mythos
• mikehuntt 5mo ago
  [dead]
• parasti 5mo ago
  I can't help but draw parallels with video games. Aimbots in competitive multiplayer games is a well defined issue: it's considered cheating and frowned upon, players caught cheating are banned from the game. Tool-assisted speedruns (TAS) where a player attempts a world record at completion in a single-player game is another face of the same concept (computers help you win), but one that is socially accepted as long as runs are clearly labelled as TAS.
  • ViscountPenguin 5mo ago
    The biggest difference would be the fact that you can discover video game cheating through some kind of trace. Speed running communities go pretty hardcore on that kind of thing nowadays. It's a lot harder to detect cheating when your only trace is how fast someone submitted the string CTF{DUck1e_Pwned}
  • justanotherjoe 5mo ago
    Sure if the goal is entertainment and sports, you're right. However, unlike chess or counter strike it's downstream from a real needed utility. Like, is there a point to do it anymore? (ofc there is, but still, it's been devalued from the perspective of the 'real utility')
    • nrabulinski 5mo ago
      It’s literally not. The most interesting and satisfying CTFs have never been grounded in reality, it’s just been an expression of mastery, both from players and authors, with a few notable exceptions. But they’re that, exceptions, not the rule.
  • tptacek 5mo ago
    Aimbots in competitive multiplayer games are (almost always) game-breaking abuses. CTFs have always rewarded tooling and automation. They're different cultures.
• copx 5mo ago
  >If adaptation means accepting that the scoreboard is now an AI orchestration benchmark, then we should say that honestly instead of pretending the old competition still exists. This is like someone complaining that making machine parts has been ruined: Skillful craftsmen used to make them by hand using manual tools! Nowadays the CAD/CAM/CNC cheaters have almost completely automated the whole thing. How is the next generation of craftsmen going to learn how to craft a gear by hand when the process of gear making has been reduced to pressing start on a CNC machine?! See what I mean? Sorry, I think this article is just Luddite. I can empathize with the pain of your beloved craft basically being rendered obsolete by new technology, but the process can neither be stopped nor is it bad in general. The manual skills you trained with CTF puzzles are now simply no longer relevant . (Field-specific) "AI orchestration" is the new cyber securtiy skill if LLMs really have become so good at this, and what the author used to do manually then has the same value as being able to craft a gear by hand.
  • raddan 5mo ago
    The way I read the post is that the author is disappointed that the community is gone. The CTF was just a reason for a number of like-minded people to organize around an activity. Indeed, in the real world, plenty of people organize to do formerly-skillful tasks together. I have not personally crafted a gear by hand, but I have built a house in a long-abandoned style with a group of people only using hand tools. There _is_ a danger that society forgets how to do these things. During that house-building exercise, there were many tricks of the trade that, while likely documented somewhere in a book, would have been difficult to reproduce without seeing a demonstration. From the standpoint of “does it matter?” it depends on what you care about. We absolutely do not need cruck-framed houses with scribed joints. Modern construction is faster and cheaper and lasts long enough. But it would sadden me greatly if practices like this faded from memory, because it’s one of those things that makes you gasp “wow!” when you see it. And your appreciation only deepens when you try it yourself.
  • deleted 5mo ago
    [deleted]
  • toraway 5mo ago
    Just parachuting in to reflexively throw the "Luddite" label at someone lamenting the decline of a niche community they've enjoyed participating in and contributing to is certainly ... a choice. Within the framework of your analogy, it's like responding to someone active in DIY maker groups suddenly dealing with an influx of influencers in meetups showing off Chinese junk from Etsy to post on Tiktok, and accusing them of being a Luddite blinded by their zealous hatred of mass production -- both strangely abrasive and also fairly nonsensical except as a "mass production supporter" social signifier. Not to mention, in the article they specifically describe themselves as a heavy user of frontier models for security research ever since the release of Opus 4.5, calling them "useful within the field". In fact I don't see any actual criticism of AI/LLMs anywhere whether for security research, programming or anything else, except for making competitive CTFs no longer viable. What does it take to avoid the "Luddite" brand? Using AI themselves and praising AI as useful (to the point of having a lopsided advantage over humans) isn't enough? Do they also need to say "I haven't written a line of code in 6 months/it's easily a 100x multiplier for my job" every time they mention it too?
• spacedcowboy 5mo ago
  The first paragraph on anything with an acronym in it should explain the bloody acronym. I assumed CTF was an encryption standard, given the headline. It was only coming here and reading the comments that made me realise it's a game-format ("Capture The Flag").
  • jaffa2 5mo ago
    Capture the flag the only expansion of CTF that i know but even if it is capture the flag this still doesnt make any sense. Like Quake CTF?
    • arm 5mo ago
      https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurity) https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit...
      • jaffa2 5mo ago
        thanks for the context! Any of which was completely absent from the article or the comments when I posted originally.
  • msm_ 5mo ago
    I don't know what to tell you. If you don't know what "CTF" is you're not the target of this blog post. It's like stumbling upon article "What's new in HTTP/2" and complaining that "HTTP" acronym is not explained. I don't mean that everyone must know what CTF is, but sometimes it's OK to write things just for your community (CTF community in this case), not for general population.
    • spacedcowboy 5mo ago
      Ok, so picture the situation: 1) You see a headline on HN about some open format being broken by frontier-level AI. You don't recognise the acronym. 2) You visit the site, you read the first few paragraphs, you still have no effing clue what the site is talking about 3) You come back to HN and read the comments to figure out WTF is going on. Oh, it's just some game style, so not a Cryptographic Trust File, or something you have to care about after all. The point is you can't know what some opaque acronym is about until you visit the source of something that will hopefully explain this opaque acronym. Leaving the site being still none-the-wiser is a failure of the site, in my view. If you don't agree, that's fine, we're adults, we can differ, but it seems like a valid complaint to me. FWIW (this means "for what it's worth" :) I'm not railing against acronyms in general, and HTTP is probably one of the most-used ones on the internet so I'm not sure it really applies as a good counterpoint. Using CTF without an explanation is more like using SSTP (Secure socket tunneling protocol) without one, IMHO (this means In My Humble Opinion :) ...
• jimnotgym 5mo ago
  You can still do competitions. But you'll all need to fly to the same place and work on laptops with a fresh install of Linux. 1 hour to install tooling then Internet off, challenge revealed. Not as easy logistically...
• xiphias2 5mo ago
  ,,a beginner is pushed toward using AI before they have built the instincts the AI is replacing. That is an anti-pattern.'' The same article talks about CTF skills as a way to learn about security best practices and separately a sport. In reality it was all about learning an extremely important skillset (securing/attacking software and systems) that is getting automated. The real thing the author seems to be frustrated about is AGI is coming in computationally verifiable domains first, and lot of his skillset was taken over in a big part.
• TrackerFF 5mo ago
  Question: Was this website made with Claude? I've seen that exact font and color scheme a dozen of times the past weeks.
  • deleted 5mo ago
    [deleted]
• saidnooneever 5mo ago
  Do CTFs like Lan parties or factor in new tooling avalable to people. change is not death. or death is not an end. either way, people will enjoy applying and showing off their skill. competing with eachother on a human level,.with or without ai tools.
• JackSlateur 5mo ago
  No relationship with the CTF (Common Trace Format) format ..
• petterroea 5mo ago
  I helped arrange my country's longest living CTF this year. Our CTF is *made for amateurs*, but we always have challenges for intermediate to skilled players and the top of the scoreboard is usually topped by them. It is the compromise we have - amateurs get so many tasks they struggle to solve them all, and the pro's get to win. Our goal is to nerdsnipe people who are curious into trying our CTF by offering easy beginner tasks, and then get them hooked enough to stick around for the intermediate ones, even if it takes them a day to solve one. This year, multiple groups on the top of the leaderboard were clearly abusing LLMs. You can tell because they know nothing of what a CTF is nor the terminology, nor really the fields the challenges were about when they were talked to. They were obviously amateurs. It was pretty depressing to hear how unaware they were of how obviously they did not fit in to the type that usually is on the top of the leaderboard. It seems they seriously think they were under the radar. If it was one group it could be a freak incident - some times someone just shows up and curbstomps competition. But there were many groups like this this year. They also had a certain smugness to it - one staff reported that a group was hinting to other teams about their "super weapon". Another group credited their "secret third team member they didn't want to talk about". I use LLM frequently and experiment with it a lot, both at work and on my free time. Nowadays they are good enough to have value and I am interested in learning more about that. They let me spend more time on hard problems and avoid spending the day on simple CRUD. I say this to say that LLM doesnt have to equal bad, it is a tool, that's all. However, I generally avoid LLM communities because many LLM fans are lazy and unskilled people who are just happy they can feel they are worth something even if they have no skill. They don't really have much to provide of conversation. If anything, from reading the CTF crowd this year, the rise of LLMs has just meant more of these people can stomp on and harvest the CTF scene for self validation. This is not me trying to gatekeep who can play CTF. Anyone is welcome, but there is one condition: You are here to learn and have fun. The conclusion many I talk to has come to is that nowadays, it is harder to learn to put in hard work and become good at something because there are just too many ways to cheat and take shortcuts. I suspect in the future there will be a shortage of useful people - the kind that have critical thought and know the value of doing something properly. This doesn't mean "Not using LLM", but as said by many on HN before you need a certain seniority before LLMs are useful augmentations to your skills and not just stopping you from learning yourself. I agree with the article. Anything but physical competitions with strong security - think professional e-sports with organizer-provided PCs, is over. But I think one of the most interesting things to take away from my CTF experience is that the bottom of the leaderboard was still full of amateurs slowly working their way up - it is a few rotten apples that ruin the fun for most, and there are still plenty of people who want to learn and deep-dive.
• phoebe_builds 5mo ago
  [flagged]
• Gathering6678 5mo ago
  I thought a company called Frontier broke a file format CTF.
• dostick 5mo ago
  Unable to find what “CTF” means, since it doesnt look like referring to Capture The Flag gaming
  • yc-kraln 5mo ago
    It does--but a particular form of Capture The Flag where there is a computer system and the "capturing" is breaking in or exploiting a security issue in that system.
• bornfreddy 5mo ago
  I guess this is very similar to what happened to demo scene, in some way. The limits are what makes these problems interesting, and once we have better machines / tools, the incredible skill is no longer prerequisite, making everything less interesting for participants. Sad, but - such is life...
• yk 5mo ago
  There's something funny about complaining about cheating in a hacking competition. Well actually I get it. In cycling motor doping, putting a hidden engine into the bike, seems more offensive than regular doping. I think this is because there is a continuum from eating well to taking supplements to injecting stuff, but having a engine breaks a fundamental idea about cycling. Similar hacking is about cleverly abusing the rules.
• tkel 5mo ago
  Pretty ironic that this article was also written using LLMs. It has all the LLM-isms.
  • qassiov 5mo ago
    Surprised to not see more discussing this. It's so grating, and nobody noticing (or believing others that say it's AI generated) makes me feel like I'm going crazy
• lachiflippi 5mo ago
  The "CTF for fun" aspect has been dead ever since the winning teams had thousands of dollars of rewards waiting for them. Of course people are going to use anything that's not explicitly forbidden by the rules to win. Introducing what amounts to an "I win" button that both can't be prevented by rules and is accessible to anyone didn't "break the format" anymore than the epidemic of giant merger teams did a couple years ago, it just broke the community because you now don't have to actually talk to other people to cheat anymore. Many CTFs have switched to a dual-leaderboard format recently, one for "agentic teams," one for the rest. If all you care about is "learning" and imaginary internet points, you can just participate as a human team and adblock the AI scoreboard, and maybe lobby CTFTime into splitting their rankings as well.
• tommy29tmar 5mo ago
  [flagged]
• tardedmeme 5mo ago
  When I did my first CTF, it was close to the deadline and I thought I had the extracted the flag from the program and the rest of the program was just filler, so I entered the flag, and it told me it was not the flag. It turns out the program multiplies the input by a pseudorandom matrix before comparing it against the flag, so I had to implement a matrix inversion and then get the flag. That's not the story though. The matrix was always the same and the challenge was clearly designed so that the point was being able to read anything at all, not knowing how to invert a matrix, so I asked the creator what was up. He told me that there were tools that would trace input values until they reached a comparison instruction, then print what they were compared against. Therefore it was necessary for every deobfuscation challenge to scramble the input in some way too complex for these tools to undo, before comparing it. Hence the multiplication by a pseudorandom matrix. The point is, cheating tools aren't new.
  • mpeg 5mo ago
    Yes but you can't compare some ollydbg script that would maybe be useful in a super specific challenge to LLMs which trivialise absolutely every challenge in a ctf and are de facto necessary to compete now
• simonTrace 5mo ago
  AI-generated phishing is the scariest development in cybersecurity right now. Click rates on AI-written phishing emails are 54% compared to 12% for traditional attacks. Automated real-time detection is the only scalable answer at this point
  • brandonwindson 5mo ago
    Click rate is higher because AI removes the broken English and generic scams. But detection is not the only answer. Most phishing comes from a small set of hosting providers that ignore reports. The real fix is making abuse reports actionable. Fix that, and you cripple the economics of phishing. AI detection is reactive; killing the source is proactive.
• simonTrace 5mo ago
  [flagged]
• 3vo-ai 5mo ago
  [flagged]
• Michael666 5mo ago
  [dead]
• JoshGG 5mo ago
  CTF = Capture The Flag https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurity) https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit...
• lmeyerov 5mo ago
  It's tough. We run botsbench.com , which tracks AI progress on a top CTF, and I gave a talk at CCC a few months ago on our own results doing AI speed runs, so I think about this a lot. In our own trainings we give (AI agents for security, and a graph masterclass), we ended up leaning into it. For example, we ship with a skills bundle. There are plus sides, like less code-forward participants can go further and are appreciating that, and less of a gap between high-level concepts and successful hands-on. But at the same time, manual work does build a lot of intuition & knowledge that gets missed in auto modes.
  • nine_k 5mo ago
    Will this bring back the age of LAN parties, where the LAN is disconnected from the internet, and mobile connectivity is blocked?
    • lmeyerov 5mo ago
      I think that ship has sailed as well -- botsbench.com shows Sonnet 4.5+ with Claude Code harness does pretty well, and Sonnet roughly tracks the edge of what self-hosted models do on the upper tier of affordable GPUs, like running 1-2 DGX Sparks and waiting 6mo for oss to catch up a bit
• lg5689 5mo ago
  This is happening to other forms of competitive programming too. The most recent AIs have problem solving skills rivaling top humans, and so if AI can't be easily banned, the competition is dominated by AI agents. I thought code golf would take longer for AIs because there's so little training data (it's more niche), but we're seeing AIs starting to match expert humans there too. Sucks because golf has been my favorite type of programming puzzle. It's crazy how far AIs have come in problem solving ability.
  • Legend2440 5mo ago
    Code golf is well-suited for AI because you have a easily verified objective (minimize code size while passing tests) and can run an LLM in a loop to churn away at it.
• notepad0x90 5mo ago
  You can introduce canaries, and ban auto-pwning in general. that's usually banned anyways. Some challenges just can't be solved by a human in under a certain period of time. Another idea is deep red herrings. solves that lead to more solves, on and on, except only if the previous solves were solved quickly. The effect will be that participants who solve things quickly will keep finding things to solve. they can't know that the path they're on will lead to victory, even if they artificially slow down, unless they consistently slow down just as a human would. It will eliminate the speed advantage. For the skill advantage, other than having another LLM procedurally generate challenges, I don't know of a good solution. There are always things like captchas. or the good 'ol honor system. A person can spend only so much for things that have no financial reward in the end, only clout. --- Alright, all that said, i think i really do have a good solution for this, as well as academic exams. Or I think I do, because it's so simple, I've been scratching my head as to why everyone isn't doing it already. Require screen sharing/recording. LLMs can't fake that well enough. Have another LLM audit the video for mouse, key stroke, window movement and other details to see if it looks human-generated or not. If a student has an essay assignment, have them record their screen as they research, and actually type out the whole thing. In the extreme, require anti-cheat proctoring software installed, as is done in remote examination. In an even more high-stakes and extreme scenario, have them share their face. Their eye and face movement, correlated with the screen-share, and correlated with the activity observed on the server end, should be pretty hard to beat, even in the next ~5 years of LLM advances.
• ChiperSoft 5mo ago
  Neither the article nor the comments in this thread explain which of the many meanings the acronym CTF is being applied to...
  • mr_mph 5mo ago
    In this context, it stands for capture-the-flag: A type of computer security competition, usually in a 'jeopardy' style, where challenges that fewer teams have solved are worth more points.
• hemlock4593 5mo ago
  I feel the post. For me AI has ruined both, playing CTFs and also building CTFs challenges. The most annoying thing to me is the "yeah idk but here is the flag" mentality. Before when playing CTFs with my mates was usually sitting there for hours tackling a challenge until some other mate joined, had some look together and solved it with you together in 30 minutes which is the most rewarding learning experience. Nowadays mate joins in throws the clanker on it and solved it in 5 minntes. Asking on how it worked you always get the "yeah idk what it did, but who cares, here is the flag" response. Same for creating challenges. Whenever I ask for writeups or if some people solved it differently I usually get the "yeah idk, clanker solved that one" response taking the fun out of it. So yep, this CTF format is definitely dead. Mainly because the strong competitiveness and prices. This encourages people to cheese challenges and sometimes solving them differently was fine as you still had a creative out-of-the-box thinking moment, but nowadays with AI there is no brainpower needed, no cheesing needed, no human needed. As you mentioned, it's pay to win. My two cents is that the 24/7 CTFs will get more attraction as the scoreboard doesn't matter there and simply doesn't give you any price.
  • gmm1990 5mo ago
    I don’t know like chess engines didn’t kill chess. You could just play with people that don’t use the “engine”
    • hemlock4593 5mo ago
      Yea, but chess adapted to it and is restricting use of engines. When you play a tournament you are banned from using a phone and will be disqualified if you do so. Online tournaments don't have a prize money for that reason, so there is no real benefit for cheating. Lichess and chess.com additionally add rankings for bots and have a strict anticheat if you use bots for regular games. For me it feels like this is not really possible for live CTFs. In contrast to chess you can't ban AI, as live CTFs are about breaking things by design, so they'll always try to circumvent an AI ban.
    • low_tech_love 5mo ago
      It’s different, unfortunately. I wish you were right. The problem is that creating interesting and fun CTF challenges is a very active, time consuming, creativity-heavy task. A chess board is always the same, and always will be, but every CTF competition is unique. There is little to no incentive anymore to spend time creating the challenges. You might say “well create the challenges and share them with people who care and who want to play honestly” which is probably the right answer here, and might happen at a smaller scale. I picture CTF in the future almost like a tabletop RPG experience, one where a small amount of people will share with close friends who they trust. But the usual “open” CTF scene (as mentioned by op) is probably over for good, if we’re being honest.
• charcircuit 5mo ago
  >Imagine giving every competitive chess player the best chess engine and letting them use it freely during matches. Would that be considered fair? Imagine every competitive chess player being allowed to video call with a hundred other people to help them make a move. CTF have never been fair, nor has it ever been effectively structured for learning.
• nektro 5mo ago
  easy, CTFs should ban it. then it'd be more like the chess community
  • archi42 5mo ago
    The article addresses this: > Rules that ask people not to use LLMs are ignored and almost impossible to enforce in open online events. It's quite sad to see CTFs dying. I never had the time do seriously participate in CTFs, but I always respected those who did, as well as the people organizing these events.
    • legacynl 5mo ago
      > Rules that ask people not to use LLMs are ignored and almost impossible to enforce in open online events. That's such a non-reason. If your competition cannot enforce the rules of the competition, then what's the point? Does the CTFs specifically need to be 'open'?
      • archi42 5mo ago
        I don't get your reasoning? You're agreeing with the author but are not? The author argues that open CTFs are done for because of rampant cheating. You're agreeing with that, don't you? The title is "AI has broken the open CTF format". If the format is "open CTF" then it is very specifically open. As to your second question: Yeah, I believe having open CTFs was a good idea.
        • legacynl 5mo ago
          ah my bad. You're right, the author specifically states 'OPEN CTFs'. I think that keyword slipped my mind by the time I was at the end of the article. So my question then becomes, what will realistically be lost if CTFs move to a form that requires teams and individuals to sign up?
• Nifty3929 5mo ago
  Must I beg to have an acronym spelled out a least once, the first time it's used? Even if you assume 90% of readers already know, the other 10% (including me, in this case) will thank you, it doesn't take much effort, and it expands the reach of your communication or idea. Exceptions for cases where the acronym is just so well known that a lot of people don't even know what it stands for even though they know the concept well. I recall one corporate training I was sitting through and they used the term "Border Gateway Protocol" and it took me a half beat to think through "oh, you mean BGP?" Thanks!
  • bawolff 5mo ago
    Which acronym do you mean? CTF? I think that acronym, just like BGP, is more well known by itself than what it stands for. More generally, not every piece of writing is meant for every audience. Like if someone writes a blog post about CTFs aimed at people who like CTFs, nobody in the target audience needs to have CTF explained to them. Ultimately HN is a link aggregator, but sometimes its a bit like eavesdropping on a conversation. When you are just listening in you don't get the full context sometimes.
    • deleted 5mo ago
      [deleted]
    • doublescoop 5mo ago
      Best practice in writing about technical concepts is to spell out acronyms like this on their first use. There is a ton of stuff I learn about here on HN that I didn't know anything about before. It doesn't help that the linked article never bothers to explain this either.
      • MobiusHorizons 5mo ago
        Does spelling it out help? From memory, it is a security competition where participants compete to gain certain objectives. I think capture the flag may explain how scoring is kept, but it wouldn’t help me find out what it is, given that capture the flag is also just the name of a game people play outside by running, or in laser tag or in certain video games.
• codemog 5mo ago
  We’re in an age where, to be possibly a bit rude but blunt, pseudo-intellectuals are obsolete. A pseudo-intellectual prided themselves on being able to efficiently solve closed, man made problems such as leetcode, CTF problems, or even math Olympiad problems. They could do good in school by memorizing a rote technique and applying it to some test. They typically don’t have any real creativity and if you put them to work on a problem you can’t Google or isn’t a fake man made one, they fall apart incredibly fast. They may as well be the human equivalent to what LLMs currently are. I do not mourn these people, as they’re usually the most arrogant types. I hope for their sake they adapt.
• Dzugaru 5mo ago
  It's not only CTFs. I strongly believe being a programmer at a gamejam like Ludum Dare, or hackathons is pretty much over.
  • wasmperson 5mo ago
    Ludum Dare 59 just wrapped up last week, and both first and second place were won by developers using "Agentic" coding tools, something the community there is still discussing: https://ldjam.com/events/ludum-dare/59/setidream/about-ai-arguments https://ldjam.com/events/ludum-dare/59/setidream/about-ai-ar... For what it's worth, the non-AI-coded entries were still quite good relative to the winners, so it's not so obvious that AI use confers an unbeatable advantage.
    • Dzugaru 5mo ago
      I did vibe code jam 59 entry with friends, the spirit of the rules there's a lot more lax. We didn't even get to top 100, but that's mostly due to gamedesign errors, not tech. This is the first entry in years which was vibecoded 100%, and I have very mixed feelings about it. It's no doubt anymore - 1.5x-2x speedup, which makes not using it (if allowed) a complete no-go. But psychologically it's tough losing control, and changing workflow to managerial one substantially, it diminishes the craft.
• s3p 5mo ago
  Don't hate me, I do agree with the premise of the article (I really do!) but I can’t help but notice: >The issue was never that AI could help. proceeds to write the next 3 sentences about how the problem IS in fact ai help >Teams that refused to use AI were not just missing a convenience; they were playing a slower version of the competition. >CTFs were not just a set of puzzles. They were a ladder. >The claim is not that every challenge is solved. The claim is that... >The loss is not just a scoreboard. It is the ladder from Guys I'm so sorry I just can't stop noticing stuff like this. Anyone else?
  • Yenrabbit 5mo ago
    I got some AI writing vibes too, but looking closer, I think it might be human-written (or at least partly so) - perhaps just picking up some AI conversation styles? FWIW, Pangram gives it a mixed but mostly-human score too. Maybe AI is not just changing the way we speak; it's changing the way we perceive all writing ;)
• tptacek 5mo ago
  A big fraction of the comments on this thread are about the impact of cheating on competitive games. It's important to understand that automating CTF challenges isn't usually cheating. It's normally part of CTF culture. The better teams have toolboxes ready to shred the early challenges; it's not a level playing field and was never intended to be. (The author of the piece understands this; I think they're broadly right, though I think these games will find other ways to incentivize participation without the now-meaningless leaderboards.)
  • viccis 5mo ago
    This is already addressed in the blog post about the fast that frontier LLMs have moved to being able to solve the kind of problem you'd expect a talented amateur or mid-level pro to do (aka top level CTF problems)
• atleastoptimal 5mo ago
  >The competition is turning into "who can afford to run enough agents, with enough context, for long enough." This will basically become true for everything.
  • low_tech_love 5mo ago
    It’s unavoidable, but really sad, isn’t it? Thinking about the incredible creativity and hard work that went into creating such challenges and now it’s probably history. I feel something similar with free-to-play gacha video games. The gacha mechanics are slowly creeping into every type of game, and where you once had very clear and obvious slot machines, now you get them transparently mixed in with beautiful and fun games (e.g. ZZZ, Where Winds Meet, Genshin, etc) in a way that sounds like in the near future no company will have any incentive to not have gacha mechanics in any game.
• 3we 5mo ago
  [flagged]
• SebFender 5mo ago
  Yes you're right - But just like many other stuff things change - CTF Veteran for more than 3 decades I find lots of fun figuring out how to use some of my agents and new tools to find vulnerabilities - The goal is the same / tools change and that's good.
• somesortofthing 5mo ago
  I have no experience in the CTF scene so I'm curious - why not lean in and design the puzzles with an AI harness like the one top teams use in the loop and use the(presumably) expert skills of the designers to patch up the holes until the AI can't find them? Do you just end up designing ~perfectly secure systems that no human can break without finding monetizable 0days?
  • a_t48 5mo ago
    I think that misses the point - it's a little bit like asking why FPS game developers don't lean into aimbot usage. You could, but by default it's a bit boring, and a different type of game.
• SadWebDeveloper 5mo ago
  Kinda FUD article... the reality is that common problems are going to be easy because the solution is probably inside the training dataset, the challenge should be adapted to make LLM's useless for example once at Defcon CTF the problems were for an unknown CPU architecture based on octal that required to write even your own disassembler... this are the kind of things that will probably be hard for frontier LLM's
  • netsec_burn 5mo ago
    Speaking from experience, the LLM agents adapt fairly well to these contexts too. It's not at all FUD, you're at a significant disadvantage if you don't compete with AI now. I went to a CTF recently against teams I have won against every year, and within 10 minutes of the event starting they had solved every challenge. They have an agent loop and it solves everything immediately, so they won. Anyone attempting to solve the challenges on their own has no chance, even if you think "maybe this is too out of the box for LLMs". Furthermore, the DEFCON CTF you're referring to has quals, and if you don't qualify you don't get those challenges in the finals. Quals has mainly binary exploitation challenges which Opus (and others) solve as long as you hold the gas pedal down on your API bill. I don't believe it's hyperbole to say CTF is dead, as a competitor.
• alisideas 5mo ago
  [dead]
• rmac 5mo ago
  i asked hackathon judges about this - the net is More than ever teams of one are winning I guess this goes in parallel with the whole building for one narrative And while I have ideas to excite and promote LLM use in these style games I've still not been able to crack the human collaboration component that is at the forefront of all of this change
• legacynl 5mo ago
  > The issue was never that AI could help. CTF players have always used tools. [...] Teams that refused to use AI were not just missing a convenience; they were playing a slower version of the competition. So the obvious solution is to fully ban AI and AI generated tools? To destroy your own hobby just because AI can semantically be considered a tool, seems very stupid to me. If the point of these CTFs is to practice and measure your skill, what becomes the point of the competition once everyone uses AI?
  • nicce 5mo ago
    Many existing challenges are purely based on knowing the existing precise tool. It has been the difference of "very easy" and "insane" challenge, whether you knew the tool or not. So, I would not start banning the tools. They always been there. We just need to fine tune the challenges where bar goes beyond things and you really need to use AI as tool. Maybe the definition of insane starts to be custom kernel fork with planted bugs and you need to use AI to find the bugs, and use some exploit chains against that kernel with specific web server. That is the real world right now, I guess.
• electr1cBugaloo 5mo ago
  This article hits really close to home for me. I have just recently started doing CTFs and feel like I am already a somewhat decent intermediate player. Much of the motivation for me is learning but also comparing/succeeding vs. my peers. Asking AI for learning / explaining purposes is absolutely fine in my book - but I have absolutely no motivation to set up AI to solve challenges. Without AI you can't really compete successfully. So AI is really taking motivation away for me which in turn again prevents me from learning more. I am not sure that this is solvable.
• richardfey 5mo ago
  I have a different take on this: how many people using a rifle do you see at composite bow tournaments? We might just move this kind of activities to sandboxes and implement more strict requirements for participation. It might make them niche or indeed disappear.