4 ms·

Here are my thoughts: I managed to scrape through level 4 so saw them all - this CTF was fun and educational. Level 4 in particular felt like a real education (

Here are my thoughts: I managed to scrape through level 4 so saw them all - this CTF was fun and educational. Level 4 in particular felt like a real education (for me at least), and a great way to learn. Unlike many others I was fond of level 0 because it brought up questions that even beginner programmers should address - how to organise your data so that you can search through it easily, which was in some ways a preparation for the other levels. For the more expert than me it probably felt tedious or inconsequential, but it was quick to solve, and I thought inviting for beginners and open ended. Lessons learned for me Distributed consensus was really interesting; I'd never heard of it and enjoyed reading about Paxos and Raft, even if I don't fully understand them yet. Distributed searches can actually be faster than searches in one process - I didn't believe this but as resources were limited on the server it was true in some cases (depends how fast you can make your search, mine wasn't as fast as Henrik's :). Big scala projects take an aaage to compile - is it normally this bad in real projects? Anyone care to comment? The remote was pretty slow, so that might be why. Golang needs package versioning! On level 4 there was a versioning bug with differing versions of the go-raft library being fetched locally and on the server which led to problems setting the heartbeat timeout of raft. That was confusing and annoying. A few notes for the CTF authors The dramatic variation between resources on remote and locally was a problem in tests - I felt like that needed to be adjusted for as the local test results were meaningless compared to the server. This particularly made level 3 and 4 difficult. Code examples are great to get you started, but it'd be nice if there was also a clearly specified API for each one which you were required to conform to, to make it easier to use another language, and if the build scripts simply launched one binary at all times - level 3 had two different styles of starting the servers for remote and local for example, that was frustrating. The varying test cases were sometimes frustrating, and were too varied and randomly chosen. I completely agree with Henrik that you need small tests. Each run should have say 5 tests which can be run individually (for speed), or as a suite. The downloading of tests broke a few times for me and left me stranded and constantly re-downloading new test data/tests seemed a bit pointless. Proper logging output when tests fail is really important - level3 was particularly bad for this as the diffs were useless; I had to edit the test harness to get decent output. The use of the JVM on level 3 was pretty painful for me - compile times on remote were in the minutes, and I couldn't even get it to compile locally even after installing another JVM, nor on remote without compiling all the sbt stuff - this one was probably my fault, and I just rewrote with golang in the end. Explicit language independence for all the problems would be great. Unix sockets felt like an unnecessary distraction which caused pain on level 4, it would have been nice if it used IPs, but perhaps for reasons to do with Octopus that wasn't possible, and on a deeper level, being given broken code to fix just felt a bit wrong and distracting - having slow code you have to speed up is one thing, but having code which feels like it has been sabotaged is another. Don't bother changing the deadline even if people whine - the contest isn't remotely fair anyway or an objective test, so it's better to have a deadline and stick to it. Oh, and thanks again Stripe, it was a blast. The original code and some more writups: https://github.com/ctfs/write-ups/tree/master/stripe-ctf3 https://github.com/ctfs/write-ups/tree/master/stripe-ctf3 http://blog.joneisen.me/post/75008410654 http://blog.joneisen.me/post/75008410654

necubi 13y ago
The reason the build times on level 3 were so bad was due to time spent downloading libraries. Stripe seems to have set things up to use a different ivy cache for each user, so every time you deployed code it had to redownload all of the dependencies. For me, this tended to take a couple of minutes. Actually compiling the Scala was pretty fast (< 10s for me). So yes, Scala is generally slow to compile, but don't blame scalac for this one :). Personally, I didn't quite manage to finish 4. I had a working solution locally, but didn't have the hours to debug why their remote environment differed so much from my local environment. My submissions also failed randomly about 4/5 times as the deadline got closer, presumably due to overload on their servers. And the unix domain sockets indeed felt unnecessary; I probably spent more than an hour just dealing with issues related to that. The format (particularly on 3 and 4) of being given pretty broken code also sort of messed with my normal approach to code reading, which assumes that the code is more or less ok. This was particularly problematic for me on 4, since I had never before used Go and had a harder time figuring out what was wrong and what was just Go weirdness. All around, though, I had a pretty fun time. Even if it did suck up way more time this week than I intended. Note to Stripe: I would gladly pay $10 next time around if it could buy more/better ec2 instances.
- grey-area 13y ago
  Stripe seems to have set things up to use a different ivy cache for each user, so every time you deployed code it had to redownload all of the dependencies. Thank you. So normally you'd have the dependencies prebuilt, and it wouldn't take long at all to deploy? That sounds more sane. The build time for golang was < 1 second for comparison, and at first I couldn't get rid of the sbt build due to confusion over start_server/start_servers being used to launch, which was a bit annoying. I couldn't get their solution to build locally so I just gave up on Scala, which is a shame, perhaps I'll try again sometime. l4 was pretty hard, even just integrating raft in the time available, I started adapting what they had, then switched to inserting raft - I was still left with some issues, and it was humbling and awesome to see people scoring so highly on it for me with things like in-memory dbs, but also great just to read about the problem space, which isn't one I'd normally touch at all. So an education even if you didn't pass, which was pretty chancy anyway in my experience. Note to Stripe: I would gladly pay $10 next time around if it could buy more/better ec2 instances. I wondered if that was part of the challenge, particularly on l3, it meant it was harder to solve with just one instance, but yes it did cause frustration, particularly because a working solution locally would then not work remotely and scores were all over the place, even for the same code, due to contention and the random tests loading remotely. It's probably really hard to set this stuff up, and I look forward to the infrastructure posts to see how they did it.

necubi 13y ago

The reason the build times on level 3 were so bad was due to time spent downloading libraries. Stripe seems to have set things up to use a different ivy cache for each user, so every time you deployed code it had to redownload all of the dependencies. For me, this tended to take a couple of minutes. Actually compiling the Scala was pretty fast (< 10s for me). So yes, Scala is generally slow to compile, but don't blame scalac for this one :). Personally, I didn't quite manage to finish 4. I had a working solution locally, but didn't have the hours to debug why their remote environment differed so much from my local environment. My submissions also failed randomly about 4/5 times as the deadline got closer, presumably due to overload on their servers. And the unix domain sockets indeed felt unnecessary; I probably spent more than an hour just dealing with issues related to that. The format (particularly on 3 and 4) of being given pretty broken code also sort of messed with my normal approach to code reading, which assumes that the code is more or less ok. This was particularly problematic for me on 4, since I had never before used Go and had a harder time figuring out what was wrong and what was just Go weirdness. All around, though, I had a pretty fun time. Even if it did suck up way more time this week than I intended. Note to Stripe: I would gladly pay $10 next time around if it could buy more/better ec2 instances.

grey-area 13y ago
Stripe seems to have set things up to use a different ivy cache for each user, so every time you deployed code it had to redownload all of the dependencies. Thank you. So normally you'd have the dependencies prebuilt, and it wouldn't take long at all to deploy? That sounds more sane. The build time for golang was < 1 second for comparison, and at first I couldn't get rid of the sbt build due to confusion over start_server/start_servers being used to launch, which was a bit annoying. I couldn't get their solution to build locally so I just gave up on Scala, which is a shame, perhaps I'll try again sometime. l4 was pretty hard, even just integrating raft in the time available, I started adapting what they had, then switched to inserting raft - I was still left with some issues, and it was humbling and awesome to see people scoring so highly on it for me with things like in-memory dbs, but also great just to read about the problem space, which isn't one I'd normally touch at all. So an education even if you didn't pass, which was pretty chancy anyway in my experience. Note to Stripe: I would gladly pay $10 next time around if it could buy more/better ec2 instances. I wondered if that was part of the challenge, particularly on l3, it meant it was harder to solve with just one instance, but yes it did cause frustration, particularly because a working solution locally would then not work remotely and scores were all over the place, even for the same code, due to contention and the random tests loading remotely. It's probably really hard to set this stuff up, and I look forward to the infrastructure posts to see how they did it.