3 ms·

I noticed this as well. There was no integration testing? No manual testing? I don’t pay for Lichess, love the UX, and I know it’s a labor of love, but come on

I noticed this as well. There was no integration testing? No manual testing? I don’t pay for Lichess, love the UX, and I know it’s a labor of love, but come on. Surely Lichess refactoring aren’t so urgent and user critical that they should just go straight to prod?

_carbyau_ 4y ago
Yeah ... but nah. Yeah, there should be some kind of shakedown test. But Nah, not every issue can be tested for within the manpower/$ budget. Issues of load being one of them. A valid strategy is: send it!, work with the beast a bit, if too unruly roll it back. In this case, they tamed the new beast. Not like Lichess is going to lose billions...
- bravetraveler 4y ago
  Load testing isn't quite the expensive proposition it seems! Think about it - clients or requests are really just ephemeral ports... There are something like 30 thousand of them available by default with Linux If you use sockets you can have as many connections as your system allows for open file descriptors. The upward limit is... $ sysctl fs.file-max fs.file-max = 9223372036854775807 Spawn a ton of concurrent requests to hit the service/API! apachebench is a decent example of the idea; imagine some proprietary sauce with it It gets cheaper (and in some ways, more effective) if you actually remove the network -- latency goes down You can also rate limit it in software, but it's really not analogous in some ways (eg: net.core.somaxconn limitations/realities) Development time is totally understandable though. It's important I leave my green field of comfort
  - boarquantile 4y ago
    Load testing would have to be quite advanced to find this, doing more than just simulating users for the most common features. It took more than 24 hours of real-world usage to visit sufficiently many niche code paths to fill the code cache.
    - bravetraveler 4y ago
      Advanced in that it's POST instead of GET, and some parameters to boot I'm not saying it doesn't take effort, I'm just saying it can and is regularly done Regression testing with speed. If you test your features as they're added, niche-ness doesn't matter - you know about it and test it A regression test quickly turns into a load test if you do them all at once To be clear, I'm not trying to twist the knife. I understand why it wouldn't be done. I just won't pretend it can't I've been trying to avoid this 'hindsight is 20/20' thing... I'm not trying to lay blame - just offer thoughts towards improvement, and encouragement against some defeatist thinking patterns A lot can be done with very little equipment/time investment
snhly 4y ago
Lichess being a project that is 85% written by one guy, and all other changes approved and read by him, he may have felt confident (until now) that he could maintain all the complexity of the software in his own head, and recall what to do and where to do it whenever he made changes. Seems bonkers to most of us writing software in teams, but compare this choice of workflow to ones made by others in similar situations: programmers in academia, who often work entirely alone, or Linus Torvalds in earlier versions of the kernel. Skipping testing is a common choice, especially when working alone and with a typed language. I could see Thibault bringing tests in now however. They would not have caught much in this case, because it seems like the problem came to light only when the software started running with real users.
- bravetraveler 4y ago
  While I do agree, these folks deserve a break... see my other comment in this thread RE: load testing It's not particularly expensive or difficult, but it does need investment

_carbyau_ 4y ago

Yeah ... but nah. Yeah, there should be some kind of shakedown test. But Nah, not every issue can be tested for within the manpower/$ budget. Issues of load being one of them. A valid strategy is: send it!, work with the beast a bit, if too unruly roll it back. In this case, they tamed the new beast. Not like Lichess is going to lose billions...

bravetraveler 4y ago
Load testing isn't quite the expensive proposition it seems! Think about it - clients or requests are really just ephemeral ports... There are something like 30 thousand of them available by default with Linux If you use sockets you can have as many connections as your system allows for open file descriptors. The upward limit is... $ sysctl fs.file-max fs.file-max = 9223372036854775807 Spawn a ton of concurrent requests to hit the service/API! apachebench is a decent example of the idea; imagine some proprietary sauce with it It gets cheaper (and in some ways, more effective) if you actually remove the network -- latency goes down You can also rate limit it in software, but it's really not analogous in some ways (eg: net.core.somaxconn limitations/realities) Development time is totally understandable though. It's important I leave my green field of comfort
- boarquantile 4y ago
  Load testing would have to be quite advanced to find this, doing more than just simulating users for the most common features. It took more than 24 hours of real-world usage to visit sufficiently many niche code paths to fill the code cache.
  - bravetraveler 4y ago
    Advanced in that it's POST instead of GET, and some parameters to boot I'm not saying it doesn't take effort, I'm just saying it can and is regularly done Regression testing with speed. If you test your features as they're added, niche-ness doesn't matter - you know about it and test it A regression test quickly turns into a load test if you do them all at once To be clear, I'm not trying to twist the knife. I understand why it wouldn't be done. I just won't pretend it can't I've been trying to avoid this 'hindsight is 20/20' thing... I'm not trying to lay blame - just offer thoughts towards improvement, and encouragement against some defeatist thinking patterns A lot can be done with very little equipment/time investment

snhly 4y ago

Lichess being a project that is 85% written by one guy, and all other changes approved and read by him, he may have felt confident (until now) that he could maintain all the complexity of the software in his own head, and recall what to do and where to do it whenever he made changes. Seems bonkers to most of us writing software in teams, but compare this choice of workflow to ones made by others in similar situations: programmers in academia, who often work entirely alone, or Linus Torvalds in earlier versions of the kernel. Skipping testing is a common choice, especially when working alone and with a typed language. I could see Thibault bringing tests in now however. They would not have caught much in this case, because it seems like the problem came to light only when the software started running with real users.

bravetraveler 4y ago
While I do agree, these folks deserve a break... see my other comment in this thread RE: load testing It's not particularly expensive or difficult, but it does need investment