8 ms·
Gossip Glomers: Fly.io Distributed Systems Challenges
- paxys 4y agoSo this is some new elaborate hiring challenge? But then when I open the linked jobs page the company seems to have exactly one open position. Is it that hard to hire one infrastructure engineer that you need to do...all this?
- andai 4y agoIt is also marketing (by virtue of the fact we are talking about it!)
- packetslave 4y ago[flagged]
- sieabahlpark 4y ago[dead]
- tptacek 4y agoNo. We contracted Kyle to help us build a hiring challenge, and he and Ben overdelivered dramatically, so we turned it into a public thing. They're just programming challenges. If you're not interested in programming distributed systems, this won't be interesting to you.
- andrewfromx 4y agohehe I'm literally in the middle of your actual hiring challenge with "Fix the Glitch" 6pn networking. But I've hit a wall with it. Maybe I'll switch over to these :)
- specialp 4y agoI completed that exercise along with a Nomad one w ELK stack. And submitted it. And never got a response ever besides we will look at it ASAP. (3 months ago.). It was billed as a "2 hour" exercise, but to even get close to that you would need to be completely fluent with Fly, Nomad, ELK, Wireguard, Linux networking stack. I thought it was a great take home but you would at least expect a response for completing it successfully.
- sosodev 4y agoThat's even worse than my experience https://news.ycombinator.com/item?id=34523227 https://news.ycombinator.com/item?id=34523227 I'm pretty skeptical of these hiring challenges at this point. It seems that they often require more time than the traditional model and provide worse outcomes for the candidates. For me it was particularly frustrating because I can typically write off the traditional model as irrational but I was drawn to fly.io because they make it seem so much better than that.
- whiplash451 4y agoIf I were looking for a job, I would trade the usual stupid 1h leetcode interview for this take-home in an eyeblink. This one is much more time-consuming _up front_, but at least it gives you a sense of the job to be done _and_ doing the exercise lets you think through whether you really want that job after all.
- paxys 4y agoWell in that 1 hour leetcode session at least the interviewer is wasting the same amount of time as you. In this case you can spend all weekend working on the exercise and never even get a response.
- deleted 4y ago[deleted]
- 4y ago
- eatonphil 4y agoLove it. Thanks for putting this together! The actual challenges here [0]. Though I'm curious: are these different from the chapters in the Maelstrom documentation [1]? There seems to be a bit of overlap anyway. [0] https://fly.io/dist-sys/ https://fly.io/dist-sys/ [1] https://github.com/jepsen-io/maelstrom#documentation https://github.com/jepsen-io/maelstrom#documentation
- benbjohnson 4y agoThanks, Phil! Good question. Yes, there is some overlap with the Maelstrom docs but most of the challenges either have different goals or they are new (Kafka, Totally-Available Transactions). These challenges also are aimed at people who are familiar with distributed systems and want to test their ability as opposed to the Maelstrom docs where they are meant to walk folks through the issue. We also contributed a Go implementation of a Maelstrom client library and we were able to fix a handful of bugs in Maelstrom during the process. Kyle uses Maelstrom for teaching classes as well so work we did on these challenges were also contributed back into the repo. The fun part about these challenges is there isn't one right way to do them. Many of the later challenges have several approaches and they each have their own trade-offs. Some of the challenge goals are performance targets so you can continue to optimize your implementation to get better numbers.
- newaccount2021 4y ago[flagged]
- ignoramous 4y agoOr stability...
- rvz 4y agoHe's right you know, just make sure that you don't become insolvent in the next 3 months before you raise more money to find another year in still losing more money.
- Reventlov 4y agoThe json from the first echo example is not actual json, there is a trailing comma that shouldn't be there :p
- benbjohnson 4y agoAh, you solved the first challenge! Noticing my typos. :) I fixed up and validated all the JSON. The fixed version will be up on the web site in a couple minutes. Thanks for letting me know.
- laweijfmvo 4y agoI think the reply JSON for Challenge #2 should say "msg_id", not "id"?
- gsanderson 4y ago".. represented by two separate yet equally important groups ...". Cue the Law and Order intro in my head. I like it.
- neom 4y ago[flagged]
- tptacek 4y ago1. Not really a recruiting thing. 2. You'd get disciplined at Fly.io for non-ironically using the word "thought leadership". We're a company full of message board nerds working on something super fun. We're going to write about it. Best get used to it!
- tough 4y agoSounds cool, keep up the great work
- monksy 4y agoIs the discipline shots of malort?
- tptacek 4y agoNot even I would come up with a disciplinary scheme requiring employees to ingest poison.
- sosodev 4y agoHow is this not a recruiting thing? It's promoting your supposedly superior hiring process.
- tptacek 4y agoWell, for starters, we've paused hiring for a few months on the role it pertains to? Once again: this started as a hiring challenge, and Ben and Kyle overperformed and made it something more interesting. I guess we could have waited a few months and ran this at a time when we could metabolize more platform candidates, but, once again: this isn't a recruiting thing.
- 4y ago
- babelfish 4y agoA few weeks ago, someone at Fly threatened to disable a customer's account over Twitter (Alex Graveley, the creator of Github Copilot) because he complained about a minor-version breaking API change. Did anything ever come of that? I like Fly as a company, but overall the impression I get from their employees (even in this thread) is just...highly defensive and immature. Edit: Here is the link to the thread. Alex's original tweets were deleted. https://twitter.com/alexgraveley/status/1619117645932150784 https://twitter.com/alexgraveley/status/1619117645932150784
- mbStavola 4y agoIt's hard to pass judgement when the first couple of tweets are deleted, but this didn't read to me as Fly.io kicking anyone off the platform. Here is what my interpretation was of this interaction: > Customer: Your service is bad and I had a bad experience > Vendor: Okay > C: I've given you multiple chances but you're still not up to my standards > V: (assuming C was done with the product) Sorry to hear that, but here is a refund Am I missing something here?
- deleted 4y ago[deleted]
- cldellow 4y agoI don't see it in as clearly bad a light as babelfish does, but nor do I see it as positively as you do. My rubric: - CEOs shouldn't publicly call their customers dicks - if a customer thinks the CEO is threatening to kick them off the platform and asks for clarification about it, they should get a clear response (Graveley's last tweet) Both of these points are about professionalism of the vendor. Perhaps the Fly CEO realized Twitter wasn't the right place to have this conversation and answered Graveley's question in a separate channel.
- ryanjshaw 4y ago(1) @mrkurt: ... you can go ahead and find yourself another vendor. (2) @alexgraveley: [I'm] Trying! [to find another vendor] Kurt thinks Alex is choosing to leave due to (2): @mrkurt: ok good luck [switching vendors by your own choice]. I'll refund all your payments. Alex thinks Kurt was the one telling him to go due to (1): @alexgraveley: I'm sorry, you're kicking me off your platform ... ? Typical human stuff. Maybe this is a great opportunity for an LLM extension - have it give people the benefit of the doubt before you respond nastily to them, by checking your response. I've tried asking ChatGPT a few times to explain responses people have made to me on Twitter that I didn't rate highly but which received likes. It wasn't particularly insightful, but its unbiased perspective had a surprisingly calming effect. EDIT: I fed all but the last line of this conversation into ChatGPT, then asked it to evaluate 'my' proposed reply (the final line) with the following prompt: "What is the presupposition behind my response? Can you point out any potential mistakes I've made in my assumptions?" It correctly identified the ambiguity in the statements, and made a helpful suggestion on how to proceed.
- LAC-Tech 4y agoReading a stack of MIT PhD dissertations may be a good Friday night I'm happy if I can read through a single MIT PhD dissertation in a stack of days! First I've heard of Maelstrom. Very cool.
- mohon 4y agome too, always learn something new everyday. thanks HN!
- bogomipz 4y ago[flagged]
- tptacek 4y agoWe hire people to write distributed systems, and for those people, distributed systems is extremely relevant. We also hire people to do low-level Linux kernel stuff, and for those people, it's less relevant. I'm having trouble figuring out what it is you think you've found to dunk on here.
- bogomipz 4y ago[flagged]
- tptacek 4y agoProbably for the same reason that practically every other team that builds distributed systems engages Kyle. But I can't speak for them, I guess. Later (This comment was rewritten after I replied to it; it previously expressed contempt for the idea that a company that supposedly works on distributed systems would retain any of the services of Kyle Kingsbury.)
- deleted 4y ago[deleted]
- biorach 4y agoI don't know why you're being a total dick about something that's actually pretty neat.
- yodelshady 4y agoDockerfile would be a really good idea. Followed instructions to letter, got : main.go:7:5: no required module provides package github.com/jepson-io/maelstrom/demo/go $GOPATH is correctly set, I can see the package there, I've spent half an hour looking through incorrect internet answers and frankly I have better things to do than debug build scripts. Actually, no I don't. But I still normally get paid.
- deleted 4y ago[deleted]
- benbjohnson 4y agoI tried reproducing the error and I think I know what happened. There's an instruction to run "go mod tidy" in Go's "go mod init" command but that could be more clear in the instructions on our site. I'll update that. Thanks for letting me know.
- benbjohnson 4y agoI pushed up the "go mod tidy" instruction so hopefully that helps other folks that run across that issue in the future. Thanks again for catching that. As for Docker, that's probably a good idea for bundling Maestrom but it'd be tricky getting the Go dev environment bundled up in Docker as it utilizes several different build caches to make it fast on rebuilds. Mapping those onto Docker volumes or utilizing Docker's cache is going to be a different can of worms that probably wouldn't be any more enjoyable.
- rbjorklin 4y agoI've got a blog post for you: https://vsupalov.com/buildkit-cache-mount-dockerfile/ https://vsupalov.com/buildkit-cache-mount-dockerfile/
- spmurrayzzz 4y agoI dig this, a lot. Seems like theres an intentional selection bias here in attracting people that love distributed systems, and I think that's cool. As an aside, I think a conversational version of these challenges work well as a middle ground for anyone that think the take home coding approach may be too heavy-handed or not time efficient. That's how I do it in my funnel (sharding a database from scratch) and it only takes about an hour of face-to-face time. Most of the time you don't even need a whiteboard.
- benbjohnson 4y agoThanks! These challenges aren't part of our hiring process. We do an at-home code challenge and then we do a short "workday" on our Slack instead. We wrote up some details about the process a while back: https://fly.io/docs/hiring/hiring/ https://fly.io/docs/hiring/hiring/ Our workday is very much conversational -- it's hard to have whiteboards in Slack! We're fully distributed so Slack workdays are a good way to gauge how comfortable people are in that form of collaboration.
- ryanschneider 4y agoTotal aside but thank you for discussing the term “sequentially consistent” that’s the exact thing I was searching for just a couple weeks ago but didn’t know what to call it (I was searching for monotonically consistent). Gonna have to give Maelstrom a look over tomorrow!
- d-z-m 4y agoLooks awesome, can't wait to give these a try.
- nazka 4y agoAwesome stuff! Just to share more in a course I know in the same area: open source training courses about distributed database and distributed systems by PingCap. https://github.com/pingcap/talent-plan https://github.com/pingcap/talent-plan
- ryanschneider 4y agoAnother aside: after going through the first two exercises I got to thinking: is there anything out there that's like "Caddy For Distributed Systems"? By which I mean is there a framework (ideally for Go) where I could "just" implement my client-facing API and something like your Malestrom package's `Node.Handle` function but all the rest like node discovery, updates, etc is handled for me? Or put another way: I wish there was a turn key solution to releasing an API around something like etcd or Consul, but I'm having trouble finding anything like that. Let's say I actually wanted to deploy my own distributed version of the grow-only counter challenge, it seems like there's a lot of non-trivial operations and such I need to get right _in addition_ to my message handling logic, and that feels like a gap that if filled would allow a lot more developers to get into distributed systems development, much like how Caddy has made it order of magnitudes easier for developers to implement custom reverse proxy logic.
- belmont_sup 4y agoIsn’t based around Consul or etcd, but the encore folks https://encore.dev/docs https://encore.dev/docs have been building a framework that makes working with microservices much more straight forward. I haven’t used it much, but the tool is able to generate all the code for services that want to speak to one another. This to me is enough service discovery.