4 ms·

I wish I had enough spare cash to take you up on a bet that you could implement this in less than a weekend. Maybe HN can crowdfund paying you for two days at $

I wish I had enough spare cash to take you up on a bet that you could implement this in less than a weekend. Maybe HN can crowdfund paying you for two days at $1000/day to replicate, versus your time and a $3000 rebate if you can't.

josephg 9y ago
My loved ones are out of town, so you're on! I'll do it for the fun of it and the experience. And because I still haven't built anything on top of kafka, but there's no time like now to fix that. The challenge is: Meet the main requirements of r/place: 1000x1000 image, a web based editor, 333+ edits / second, with an architecture that can scale to 100k simultaneous users (although that part will be hard to actually test). I won't implement mobile support or notifications, and I won't implement any sort of user access control (thats out of scope). The challenge is to do it & get it hosted online before I go to bed on Saturday (so, I'm allowing myself some slop there). But its already 1:30pm on Friday, so I think that easily qualifies as "less than a weekend". As a stretch I'm going to write all the actual code in nodejs while aiming for well into the thousands / tens of thousands of writes per second territory. I'm willing to accept some fun stakes if anyone wants to propose them (ice bucket challenge level stuff). I'll live-tweet progress here - https://twitter.com/josephgentle https://twitter.com/josephgentle (Edit: fixed days)
- SomeStupidPoint 9y ago
  Wait... Where are you that it's Saturday already? I thought I knew timezones... But that's 40+ hours ahead of me.
  - josephg 9y ago
    Errr... oh, that thing is happening where its a public holiday so I'm a bit unmoored from reality. I'll get it done by Saturday night, not Sunday night.
- xapata 9y ago
  You can simulate the users for load testing. Have them random walk from an arbitrary start point, changing a pixel every 10-20 minutes.
  - josephg 9y ago
    Simulating hundreds to thousands of writes per second via post requests will be easy. Simulating 100k simultaneous readers of an eventstream will be much harder. I don't know of any tools which can load test that many readers. Do you have any ideas?
    - scrollaway 9y ago
      locust can spawn arbitrary scripts; you'd have to write your own eventstream reader, and simulating 100k of them will take a few machines concurrently load-testing, but i think this should be the easiest way to do it.
- employee8000 9y ago
  Unless you stick with a single Kafka partition (which might not withstand the load) I don't think you can consume the events in exact chronological order.
  - josephg 9y ago
    A single kafka partition should be able to handle the load - it only needs to write 333 ops/second, and I could coalesce several edits together over half second windows if I wanted to make that more efficient. And I'll only have as many kafka consumers as I have frontend servers. So we're talking about 333 ops/second and like, 5 consumers of those operations. Somehow I think a single kafka partition should be able to manage the load :) But there's no reason I couldn't use multiple partitions if I had to. To make it fully consistent I'd just need a consistent way to resolve conflicting concurrent writes on different partitions. Either adding a server timestamp to the messages, or using the partition number to pick a winner would work fine for this. It would just make the server logic a little more complicated.
- didibus 9y ago
  Yes!! Love me a good show. I'll be following your tweets.
- developer2 9y ago
  >> I won't implement mobile support Mobile support was a huge factor for /r/place. To reimplement the project while completely ignoring mobile is basically taking out maybe 30% of the effort required. It's a bit of a copout to claim "can implement in a weekend", while discounting one of the main reasons why it is nearly impossible to hammer out solo in a single weekend. The other "big ticket item" is integrating such a system into an existing live stack. It's very convenient that you get to pick the tools you think are most efficient for the job from scratch; it's quite another to restrict yourself to an existing set of tech. >> an architecture that can scale to 100k simultaneous users (although that part will be hard to actually test) Also this. Reddit had a single opportunity to deploy live. You can't throw out many days of required dev/staging benchmarking for a system that must work on first deploy. You're not "done" after a weekend unless your first deploy both works 100% and scales as required. tldr; Were I to do what you are doing, it would be out of interest and to put it live for others to use. What seems disingenuous is to do so with the primary goal of proving that "see, child's play - it's not difficult at all!". The goal should be to build something that works, not to rush the job as some way to show off and take something away from Reddit's efforts.
  - josephg 9y ago
    Absolutely! And I'm not going to claim otherwise. I said above I think I can only implement it in a weekend if I can work to my technical strengths. Doing otherwise would mean learning, which is super slow compared to typing. Just implementing to a closed spec cuts out maybe half of the work of something like this, because you don't need to figure out the requirements or waste time going down blind alleys. From my post above: > But I couldn't implement r/place that quickly if reddit didn't already do all the work deciding on the scope of the problem, and what the end result should look like.
    - developer2 9y ago
      Upvoted you. I'm a very cynical person, and thus I focused on the "weekend" aspect as being more of an attempt to refute the claim that Reddit had to put in quite a bit of effort to accomplish what they did, rather than you simply limiting how much time you're willing to sink into it. If anything, this only increases my motivation to replicate the project myself, whether it's during a weekend or two full weeks. It's interesting enough and at the right level of complexity - kind of simple, but not too simple - to make it a fun side project.
- josephg 9y ago
  12 hours in and the site is up: https://josephg.com/sp/ https://josephg.com/sp/ Polish and perf tuning tomorrow. Code: https://github.com/josephg/sephsplace https://github.com/josephg/sephsplace (warning: contains evil)
  - always_good 9y ago
    12 hour amphetamine binge can produce some good work!
  - renlo 9y ago
    Very cool man. For the real-time updates you keep the browser-client requests open, right? Any reason you did it that way?
    - josephg 9y ago
      Well, the server needs to message the client with updates somehow. Either the client keeps a connection open or it has to periodically poll the client. Keeping a connection open is faster and it uses less system resources.
  - josephg 9y ago
    Complete and working. I haven't tested the load for real, but judging by the performance numbers it should be able to handle 100k connections fine with 2 beefy backend servers. (40% of one of my laptop's cores handles 15k connections with 400 edits per second. So about 8 cores should handle 100k users no sweat.) The bottleneck is fanout, so until kafka starts hitting subscribe limits it should scale linearly with frontend servers. (And kafka is a long way from hitting that limit because I've added write coalescing.) I'll post a writeup about it all once I've eaten some real food and had a rest. But for now, enjoy!
  - socmag 9y ago
    I like what you are doing a lot. It doesn't matter that it isn't a one to one comparison, the point is you are taking on the challenge and enjoying it. Probably actually you are learning some stuff along the way as well. It would be kind of neat if the place experiment could become something of a micro-benchmark for online customer facing distributed platforms. Keep up the good work man, it's looking good. Don't forget to hydrate and consume pizza.

josephg 9y ago

My loved ones are out of town, so you're on! I'll do it for the fun of it and the experience. And because I still haven't built anything on top of kafka, but there's no time like now to fix that. The challenge is: Meet the main requirements of r/place: 1000x1000 image, a web based editor, 333+ edits / second, with an architecture that can scale to 100k simultaneous users (although that part will be hard to actually test). I won't implement mobile support or notifications, and I won't implement any sort of user access control (thats out of scope). The challenge is to do it & get it hosted online before I go to bed on Saturday (so, I'm allowing myself some slop there). But its already 1:30pm on Friday, so I think that easily qualifies as "less than a weekend". As a stretch I'm going to write all the actual code in nodejs while aiming for well into the thousands / tens of thousands of writes per second territory. I'm willing to accept some fun stakes if anyone wants to propose them (ice bucket challenge level stuff). I'll live-tweet progress here - https://twitter.com/josephgentle https://twitter.com/josephgentle (Edit: fixed days)

SomeStupidPoint 9y ago
Wait... Where are you that it's Saturday already? I thought I knew timezones... But that's 40+ hours ahead of me.
- josephg 9y ago
  Errr... oh, that thing is happening where its a public holiday so I'm a bit unmoored from reality. I'll get it done by Saturday night, not Sunday night.
xapata 9y ago
You can simulate the users for load testing. Have them random walk from an arbitrary start point, changing a pixel every 10-20 minutes.
- josephg 9y ago
  Simulating hundreds to thousands of writes per second via post requests will be easy. Simulating 100k simultaneous readers of an eventstream will be much harder. I don't know of any tools which can load test that many readers. Do you have any ideas?
  - scrollaway 9y ago
    locust can spawn arbitrary scripts; you'd have to write your own eventstream reader, and simulating 100k of them will take a few machines concurrently load-testing, but i think this should be the easiest way to do it.
employee8000 9y ago
Unless you stick with a single Kafka partition (which might not withstand the load) I don't think you can consume the events in exact chronological order.
- josephg 9y ago
  A single kafka partition should be able to handle the load - it only needs to write 333 ops/second, and I could coalesce several edits together over half second windows if I wanted to make that more efficient. And I'll only have as many kafka consumers as I have frontend servers. So we're talking about 333 ops/second and like, 5 consumers of those operations. Somehow I think a single kafka partition should be able to manage the load :) But there's no reason I couldn't use multiple partitions if I had to. To make it fully consistent I'd just need a consistent way to resolve conflicting concurrent writes on different partitions. Either adding a server timestamp to the messages, or using the partition number to pick a winner would work fine for this. It would just make the server logic a little more complicated.
didibus 9y ago
Yes!! Love me a good show. I'll be following your tweets.
developer2 9y ago
>> I won't implement mobile support Mobile support was a huge factor for /r/place. To reimplement the project while completely ignoring mobile is basically taking out maybe 30% of the effort required. It's a bit of a copout to claim "can implement in a weekend", while discounting one of the main reasons why it is nearly impossible to hammer out solo in a single weekend. The other "big ticket item" is integrating such a system into an existing live stack. It's very convenient that you get to pick the tools you think are most efficient for the job from scratch; it's quite another to restrict yourself to an existing set of tech. >> an architecture that can scale to 100k simultaneous users (although that part will be hard to actually test) Also this. Reddit had a single opportunity to deploy live. You can't throw out many days of required dev/staging benchmarking for a system that must work on first deploy. You're not "done" after a weekend unless your first deploy both works 100% and scales as required. tldr; Were I to do what you are doing, it would be out of interest and to put it live for others to use. What seems disingenuous is to do so with the primary goal of proving that "see, child's play - it's not difficult at all!". The goal should be to build something that works, not to rush the job as some way to show off and take something away from Reddit's efforts.
- josephg 9y ago
  Absolutely! And I'm not going to claim otherwise. I said above I think I can only implement it in a weekend if I can work to my technical strengths. Doing otherwise would mean learning, which is super slow compared to typing. Just implementing to a closed spec cuts out maybe half of the work of something like this, because you don't need to figure out the requirements or waste time going down blind alleys. From my post above: > But I couldn't implement r/place that quickly if reddit didn't already do all the work deciding on the scope of the problem, and what the end result should look like.
  - developer2 9y ago
    Upvoted you. I'm a very cynical person, and thus I focused on the "weekend" aspect as being more of an attempt to refute the claim that Reddit had to put in quite a bit of effort to accomplish what they did, rather than you simply limiting how much time you're willing to sink into it. If anything, this only increases my motivation to replicate the project myself, whether it's during a weekend or two full weeks. It's interesting enough and at the right level of complexity - kind of simple, but not too simple - to make it a fun side project.
josephg 9y ago
12 hours in and the site is up: https://josephg.com/sp/ https://josephg.com/sp/ Polish and perf tuning tomorrow. Code: https://github.com/josephg/sephsplace https://github.com/josephg/sephsplace (warning: contains evil)
- always_good 9y ago
  12 hour amphetamine binge can produce some good work!
- renlo 9y ago
  Very cool man. For the real-time updates you keep the browser-client requests open, right? Any reason you did it that way?
  - josephg 9y ago
    Well, the server needs to message the client with updates somehow. Either the client keeps a connection open or it has to periodically poll the client. Keeping a connection open is faster and it uses less system resources.
- josephg 9y ago
  Complete and working. I haven't tested the load for real, but judging by the performance numbers it should be able to handle 100k connections fine with 2 beefy backend servers. (40% of one of my laptop's cores handles 15k connections with 400 edits per second. So about 8 cores should handle 100k users no sweat.) The bottleneck is fanout, so until kafka starts hitting subscribe limits it should scale linearly with frontend servers. (And kafka is a long way from hitting that limit because I've added write coalescing.) I'll post a writeup about it all once I've eaten some real food and had a rest. But for now, enjoy!
- socmag 9y ago
  I like what you are doing a lot. It doesn't matter that it isn't a one to one comparison, the point is you are taking on the challenge and enjoying it. Probably actually you are learning some stuff along the way as well. It would be kind of neat if the place experiment could become something of a micro-benchmark for online customer facing distributed platforms. Keep up the good work man, it's looking good. Don't forget to hydrate and consume pizza.