22 ms·

It takes $420k per year to run Lichess

298 points by i0exception 5y ago

jka 5y ago
I was wondering what the cost per game is on lichess, and fortunately the linked spreadsheet includes the answer: lichess currently runs at a cost of $0.00022 per game (4545 games to the dollar).
- shric 5y ago
  That's good to know. I've played 12,316 games on lichess so almost $3 worth. Might donate $10 or so soon.
  - DarylZero 5y ago
    Do you computer-analyze your games? One stockfish computer analysis must cost more than merely playing 1,000 unanalyzed games.
    - altvali 5y ago
      You're downloading the engine from the Lichess server but the analysis is done on your computer, with your resources.
      - kmm 5y ago
        Computer analyses are done on a distributed network called fishnet[0], not locally 0: https://github.com/niklasf/fishnet https://github.com/niklasf/fishnet
        altvali 5y ago
        Except fishnet is used to analyze broadcasted games, puzzle discovery and others. Not used when analyzing your own games. You can easily test this by opening your task manager, monitoring your CPU usage, then opening a game you played on Lichess and toggling the aptly named "local evaluation". It's the toggle bar on the upper right, next to the text "Stockfish 14+ NNUE in local browser". You will see your CPU usage increase while the evaluation is on.
        wishawa 5y ago
        That is true when you’re playing in your browser. On the mobile app (at least the iOS version) you can request computer analysis for your game done on the server.
        ornicar 5y ago
        You can request server analysis for your own games, up to 40 per day. They use fishnet.
        kmm 5y ago
        No, that is an analysis of a single position, not a game. The big button under every game which says "Request Computer Analysis" is done on fishnet as well, it literally says "Stockfish 14.1 server analysis".
        fho 5y ago
        <sarcasm> So ... time to implement fishcoins?</sarcasm> Analyzing a chess game or two is not exactly easily verifiable, but it should be somehow possible to shoehorn that into a proof of work thing.
      - agalunar 5y ago
        Only if you toggle local-evaluation in the analysis board. When you "request a computer analysis", the computation is done by fishnet [1] on volunteers' hardware. [1] https://github.com/niklasf/fishnet https://github.com/niklasf/fishnet
        altvali 5y ago
        But can you request a computer analysis with fishnet anymore? I have a non-patron account and I can't find that option anywhere. Old forum posts seem to talk about it working by pressing Z and being limited, but when I press Z I get a message that it is disabled by default and a link to enable it. When pressing to enable, nothing happens.
        agalunar 5y ago
        It's been a while since I used Lichess anonymously, but I suspect you do need an account to request a server analysis – that way they can limit the number of analyses per day for a user [1]. [1] 40 a day per https://lichess.org/features https://lichess.org/features
        shric 5y ago
        You don't need to be a patron, you just need an account. One of the many amazing things about lichess is it's 100% free and donators get no extra privileges except a little logo next to their user
      - deleted 5y ago
        [deleted]
    - Kranar 5y ago
      Stockfish runs in your own browser using your own CPU.
      - llimllib 5y ago
        Not if you do a game analysis
    - mileycyrusXOXO 5y ago
      I think Stockfish usually runs clientside but it does look like they do some server side too https://lichess.org/blog/YOCx7hIAACUAgsUo/stockfish-14-has-arrived https://lichess.org/blog/YOCx7hIAACUAgsUo/stockfish-14-has-a...
    - jb_s 5y ago
      For this reason I wish I could run something locally to generate the same analysis UI and therefore not put any load on the Lichess servers.
    - shric 5y ago
      I do run a computer analysis on maybe 10% of games, but I also participate in fishnet (explained elsewhere in this thread) on a local machine so that hopefully offsets it
booleandilemma 5y ago
So, 4 developer salaries for a service used worldwide. Not bad at all.
tommek4077 5y ago
- michaelcampbell 5y ago
  > Maybe stop using aws or google cloud? Or, maybe start. Would at least be worth looking at to see if [the equivalents of] spot or reserved instances, Lambda's, Cloud Run, or other serverless etc. might save some money.
- WastingMyTime89 5y ago
  You need at least three admins to have 24h availability. I highly doubt you can save money by selfhosting at Lichess size.
  - woadwarrior01 5y ago
    Hosting on the cloud does not magically make your availability issues go away.
    - theshrike79 5y ago
      No, but they make many availability issues Someone Else's Problem. When EC2 goes down, I need to maybe wake up and log in to see what happened. I see that all of us-east-1 is down (again), notify everyone involved: "It's down can't do anything, it'll come back up when AWS engineers get to it" and go back to bed. I can check up on it again during office hours. If this was a colocated server, I'd need to drag my ass out of bed and drive to the site to see what's up.
      - greedo 5y ago
        Drive to a colo? Most colos have 'remote hands' where you can have an onsite tech do stuff for you. There's also OOB management tools you can use (assuming the hardware isn't toast). AWS is really expensive if you have the skills to managed hardware yourself.
wodenokoto 5y ago
Where does funding come from?
- shric 5y ago
  Donations
  - aloisdg 5y ago
    You can also buy some swag: https://lichess-org.myspreadshop.com/ https://lichess-org.myspreadshop.com/
BossingAround 5y ago
For those completely unaware (like I was), lichess [1] seems to be a chess server. Not sure whether they need AWS scale and couldn't cut costs down from 420k a year to something cheaper. Naively, this seems like something MS could sponsor by providing long-term discount. That said, I have no idea how much cheaper this would be on other cloud providers, or whether they even have the need to cut down costs. For all I know, they might be making 10x of that in donations. [1] http://lichess.org http://lichess.org
- shric 5y ago
  If you look at the cost breakdown, that includes several people's salaries and other things than just hosting. As for scale they do almost 100 million games per month and keep every game available for download and online access. I suspect donations ~= costs, which is why the salaries are quite modest to say the least.
- tyingq 5y ago
  A lot of the server costs have datacenter names that look like OVH ones to me. And the server costs seem to be ~$68k/year of the total.
_zooted 5y ago
> $68,600 in Servers
- pessimizer 5y ago
  > Candles $3,600 That being said, it's probably a lot cheaper than rewriting it to run on less.
  - whalesalad 5y ago
    i will never financially recover from this
devit 5y ago
It's the minority of the total spend, but it seems the hosting costs are too high. For example, they are paying $266/month for 16 thread 128GB 3.8TB SSD, but you can rent a better machine for 99$/month from Hetzner (AX61-NVME). They are also spending $40k/year on "data protection services" (?!?) and 60k for "site moderation" (they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate).
- kevingadd 5y ago
  "they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate" at that point, you might as well not run the service. moderation isn't optional if you want a healthy community.
  - hffftz 5y ago
- Tenoke 5y ago
  >$40k/year on "data protection services" (?!?) I'm guessing this is paying an employee/consultant to ensure they are compliant with everyone's policies. At that scale you have to care about every jurisdiction and letting the one dev guess at compliance might not be ideal. > 60k for "site moderation" (they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate). Can you even just not moderate? There's some stuff you legally have to due to say NetzDG for German users, and compliance with everyone else.
- cillian64 5y ago
  Everybody complains when someone gets inexplicably/stupidly banned by an AI and there are no humans to review it. Spending money on moderation by actual skilled people is how you solve the problem. This is one place I’m very happy to see them not skimping.
- iKlsR 5y ago
  > (they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate) Have you ever used the site? How does that work by not moderating? When I report someone for being racist, spamming or submit a game where I think someone cheated... who handles that, what forum or open service doesn't have some type of moderation and in this case were you want a seamless experience with your app as they do with chat, blog, forum and multiple daily tournaments.
whalesalad 5y ago
Lots of pets here, not many cattle. When you get to this scale, tools like Kubernetes make more sense. Then you can just think about your system in terms of "how much cpu/ram/storage do I need total?" Unless every one of those servers is running at or near capacity, there is a lot of cash being wasted there. There is also a maintenance cost, too. If one of these vital boxes goes down, what is the downtime implication and restoration cost? I am not saying any of this to be critical of Lichess. There are different ways to solve these problems, and their way is clearly working. This also happens slowly over many years, so it is hard/impossible to see the end state until you are there. The app is very quick and responsive. I got my ass handed to me on my first anon game :) My feedback is more for the community here in the context of using this as a byte-sized case study. At the end of the day we are reading and writing 1's and 0's to a network device, or a disk. Have to imagine you can run and persist chess games with a lot less resources.
- Sebb767 5y ago
  > When you get to this scale, tools like Kubernetes make more sense. If you look at the details[0], servers are really "just" 6k$ per month. A lot of that goes towards the databases, which you can't optimize much with k8s, so the optimization potential overall is already rather limited. If you then need a consultant or a developer to do the migration, you'll quickly outspend any potential savings. I'm a fan of k8s, but this is really a showcase for hardware being (comparatively) cheap and the scale that you can go to with conventional hardware. [0] https://docs.google.com/spreadsheets/d/1Si3PMUJGR9KrpE5lngSkHLJKJkb0ZuI4/preview https://docs.google.com/spreadsheets/d/1Si3PMUJGR9KrpE5lngSk...
  - whalesalad 5y ago
    Why is kube and conventional hardware mutually exclusive? It’s just a different way to utilize resources. I don't see an issue with the costs - I see an issue with this massive list of inventory that needs to be carefully managed. Managing 30 boxes is not easy when they are all funny shapes and sizes. Are they all running the same OS? Same version? How do you roll out security updates? With kube, you just bring a new node online and then kill the old one. Done. My argument still remains - what happens when a vital box dies? How quickly can a new one be brought online? Is that automated? What about in the meantime, is service degraded or completely down? What about when the ES boxes shit the bed and you realize you actually needed a cluster of 5 nodes? Do you buy 5 new boxes that are carefully sized, or do you just scale your mega cluster from 4 to 5 and add a few new pods? Kube is not nearly as big and scary as this community makes it out to be.
amelius 5y ago
Main developer gets a monthly salary of $4705, is that normal in Europe? (Assuming Europe because they pay French taxes).
- DC-3 5y ago
  It's somewhat low but they've probably made a concious choice to accept a lower salary to work for a project they are passionate about.
- throwbacktictac 5y ago
  I don't know how much time he dedicates to running the service at this point but the salary seems light if it's full time work.
  - Scarblac 5y ago
    It's his hobby project that he turned into this, it runs completely by his philosophy, and it's decent money for the country he's in. I think he's won.
- a_humean 5y ago
  Maybe a bit low for someone this experienced, but actually its just above the median salary for a developer in Paris. Developer salaries in Europe are not the astronomic salaries you see in the US in the major cities. Probably the highest paying in Europe is London in the UK, and even then its still less than 50% of what developers in SF make at probably around 75k USD per year/6-7k USD per month in London.
  - laurentb 5y ago
    worth baring in mind that there's a lot of extra paid by the employer on top of the "gross" salary someone gets in Europe (especially in France) that someone in the US would have to cover by himself (hence why i't typical to see double that amount for someone in the US).
    - 5y ago
tholman 5y ago
Lots of people saying "You can improve costs here!" without factoring in the developer time/learning/upkeep dollars it would take to do that. Sometimes the best solution is the one you know that will be easiest for you.
marcrosoft 5y ago
5k in French taxes to run a non profit site, why?
- henvic 5y ago
  Because the state was never about making things easier for people.
- RaitoBezarius 5y ago
  Non profit does not mean that it do not have to pay taxes on the money it manipulates. Non profit orgs receive services from the state and they have a cost to fund social security and such. Other non profits which do not have a very nice situation can receive help from the state, so why not share the bits of success of Lichess to everyone else?
- pyb 5y ago
  It's the norm in France for employers to pay another euro of tax for each euro their employees earn.
  - newaccount74 5y ago
    I assume with "tax" you mean "tax + health insurance + unemployment insurance + retirement savings etc"?
    - ta988 5y ago
      Yes and it was a mistake in my opinion. It make employees feel that the company is paying for these benefits and it makes the company believe that they are spending money for things they should not (I've heard so many CEOs complaining about having to pay "all those taxes for their employees"). I prefer a way where the employee is responsible for all costs with a deduction at the source to make sure you don't get nasty surprises at the end of the year. In term of numbers that's all the same, not in term of psychology.
      - duxup 5y ago
jeremyjh 5y ago
In this thread: people with zero knowledge who can cut costs drastically without impacting the service (that they have never used and don't understand).
- deleted 5y ago
  [deleted]
- dang 5y ago
  Putting others down doesn't help. If you know more than others, the thing to do is to share some of what you know, so the rest of us can learn. Then you're addressing the underlying ignorance as well as contributing to the ecosystem, instead of poisoning it further. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=share%20know%20learn%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - jeremyjh 5y ago
    It's not a put down to say that there are people without knowledge of Lichess's operations or infrastructure. I'm one of those people. I don't presume to have any knowledge of effective cost cutting measures. Others do.
    - dang 5y ago
      It's a putdown to snarkily sneer at others in the community, which is why the site guidelines specifically ask you not to: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. I understand the temptation to do this—we all feel it. That's why that guideline is in there. Such comments reliably lead to lower-quality threads, because they don't add anything that anyone can learn from. They just assert the commenter as somehow different from / better than others. That's not to say the comments you were criticizing aren't bad. Of course, shallow dismissals are bad. But it only makes things worse to respond with a different sort of shallow dismissal.
dmurray 5y ago
- qeternity 5y ago
  Haha yeah I think those specs would be 10x on AWS. GRA, RBX, etc - OVH about as cheap as it gets.
- yunohn 5y ago
  > bottom-of-the-barrel Indian devs with a handful of Western project managers It's really disappointing to see such casual racism on HN. Surely you could think of a better adjective than "bottom-of-the-barrel" to describe humans (Indians) that are intentionally hired at the lowest possible wages, by Western managers to maximize profits for a Western owned & led company?
  - malteof 5y ago
    You can also read the post as “out of all Indian devs (who on average tend to be cheaper than devs from most other countries), hire the bottom of the barrel”. I doubt they meant that all Indian devs are bottom of the barrel.
    - lillecarl 5y ago
      I try to assume good intent, but I found it hard here considering the phrasing
      - Rapzid 5y ago
        I found it easy.
        PieUser 5y ago
        I found it hard as well.
        5y ago
leros 5y ago
For reference $420k is about the cost of 2 employees at a typical tech company.
- _fnqu 5y ago
  Which typical tech company outside of Silicon Valley would that be? I’ve never seen wages at $210k for a „typical tech company“
  - jlack 5y ago
    An employee costs more than just their salary.
  - dijonman2 5y ago
    My company isn’t based out of SV, we pay 130-220k base depending on team and experience. Senior devs are 200k+
    - yurishimo 5y ago
      Hey, it’s me, your future employee!
  - graton 5y ago
    A rough estimate is that the cost of an employee is 1.25x - 1.4x the salary of an employee in the US. With health insurance, Social Security taxes, 401K matching, office expenses, and more.
    - blip54321 5y ago
      That seems exceedingly low. I've generally seen 2x. It's a bit less than that as salary goes up, but I can't imagine hitting 1.25x.
  - nojito 5y ago
    Wages are just part of the total cost of a employee.
rickdeveloper 5y ago
The author did an AMA on Reddit 9 months ago that I think is relevant here: https://old.reddit.com/r/chess/comments/mpasyl/i_started_lichess_ask_me_anything/gu9azgf/ https://old.reddit.com/r/chess/comments/mpasyl/i_started_lic... In particular: How they remain free: https://www.reddit.com/r/chess/comments/mpasyl/i_started_lichess_ask_me_anything/gu8mbuy/ https://www.reddit.com/r/chess/comments/mpasyl/i_started_lic... On his salary: https://www.reddit.com/r/chess/comments/mpasyl/i_started_lichess_ask_me_anything/gu8nv6o/ https://www.reddit.com/r/chess/comments/mpasyl/i_started_lic...
oneepic 5y ago
...for 12 seconds. https://www.youtube.com/watch?v=jHgZh4GV9G0&t=28s https://www.youtube.com/watch?v=jHgZh4GV9G0&t=28s
bluecalm 5y ago
It only 420k because they pay themselves minimum salaries. The founder/lead dev is easily worth mid 6 figures on the market and yet his salary is not even 60k EUR. It's a great project, great quality and run at a low cost but it's all that because it's run by people who sacrifice their financial situation to make their idea happen.
- ta988 5y ago
  And I wish they would be rewarded by their users even more.
  - systemvoltage 5y ago
    They should charge. And I would pay.
    - kdmytro 5y ago
      They have users because the service is accessible. If they charge, many would playe elsewhere. There is a lot of competition.
c4m 5y ago
I love Lichess's feature comparison page (https://lichess.org/features https://lichess.org/features), which helps users decide whether to upgrade to a premium account.
iqanq 5y ago
JVM and then probably Cassandra as DB. Had they used something other than Java, it would've costed 10% of what it does. A rewrite may make sense. I don't use Lichess. If I did, I would have donated. But after knowing that they throw money down the sink like this, I would be hesitant.
- OrvalWintermute 5y ago
  I too looked at the infrastructure list and wondered. While I know many people are saying don't look at the cloud costs, when I look at those numbers it makes me think of CAPEX vs OPEX trades, and the OPEX here is non-trivial, but potentially a variable offset by some labor costs. I think some of this is being driven by a dated tech stack and technical debt. Now, I am not one to trivialize the work that may need to go into getting this efficient, but I think you are right, we should consider investing into modernization of the stack.
- xunn0026 5y ago
  Yeah, sure. Maybe you could rewrite it in assembler over the weekend and run it on a Raspberry Pi, right? I worked on Java codebases that absolutely squeezed every bit of performance from a machine and that would be very hard to do in other languages.
- wishawa 5y ago
  Tell us, this language 10x more efficient than Java, what is it?
- ksec 5y ago
  Considering Java and JVM power some of the highest performance server and HFT on the planet. Tell me about the language or ecosystem that give you 90% cost savings?
  - vt85 5y ago
- deleted 5y ago
  [deleted]
altvali 5y ago
As someone who is working on a similar project (a chess mmo) and looking to validate a business model in order to get funding, how are they covering the costs? The answer usually is "donations", but there are only 402 patrons on https://lichess.org/patron https://lichess.org/patron and Thibault said on twitter that the average donation is $8. That's $3216/mo, less than a tenth of monthly costs. Is the rest coming from the swag store? From coaches? How much do the donations surpass the costs? Lots of missing pieces from the revenue puzzle.
- llimllib 5y ago
  That’s definitely not every patron
- ornicar 5y ago
  These are just the most recent. ATM there are 10,722 active patron accounts.
- Jommi 5y ago
  Maybe you should look at a model where your chess mmo runs on top of another platform? That way a lot of your costs would go down and you could focus on the core parts that you want to build. E.g. you dont really want to spend time paying for or optimizing for hundreds of simultaneous users from multiple countries.
anyfactor 5y ago
> Accountancy - 5,160 > Independent Audit - 6,000 I studied accounting at university and no one plans to do anything but auditing or consulting. I still love accounting, gives the most straight answer of how a business is run. Case in point.
freewilly1040 5y ago
It's truly amazing that a product with the reach of Lichess can be run so cheaply. Comments here criticizing the hosting costs extremely myopic - dev time isn't free, and doing a rewrite to chase down hosting savings isn't necessarily a good call. I just signed up for a monthly donation. I complain about the ad supported internet all the time, but had never donated to LiChess. This thread is a good reminder of how far dollars to support projects like this can go.
- lhorie 5y ago
  FWIW, in the earlier days of lichess, IIRC when it had barely broken the Alexa top 10,000 threshold, lichess ran on like 2 boxes despite serving millions of users. There's a good chance that many AWS slinging devs here have not actually had to deal with the volume of traffic that lichess sees. Half a mil yearly to run the entire lichess infrastructure, including dev salary and providing stockfish analysis for every user for free takes an insanely efficient setup that most big tech companies could only dream of.
  - Tenoke 5y ago
    > providing stockfish analysis for every user for free Edit: As discussed in another subthread most of the analysis happens in the user's browser, the rest is run via a fishnet by volunteers.
    - lhorie 5y ago
      While we're in the topic of impressive technological achievements, it's probably also worth mentioning how crazy it is that just a few decades ago, Deep Blue running on a supercomputer was the state of art in chess analysis, and nowadays we can just casually say, oh lichess runs stockfish on users' phones via wasm to cut on server costs.
  - epolanski 5y ago
    Most engineers in the world never saw such daily traffic. I doubt you can't make Alexa top 10k with a million of daily unique visitors.
chubot 5y ago
I love lichess! In addition to the web interface, it's got a great iPad interface too.
wenbin 5y ago
I’m actually surprised how cheap the server cost is , given the scale of linchess… Great job! Recent years, I have a feeling that HN commenters would assume server cost should always approach $0, not matter how big, or how different a real world online service is different from their single page web app side project :) For any server cost sharing posts, we’ll see people making comments like “I can run this under $10/month” . Ha
faangiq 5y ago
He should take a better salary and figure out a more viable business model. Running it this way is a risk to its future.
anthk 5y ago
And freechess.org ?
killingtime74 5y ago
I’m surprised French taxes are so high for a non profit
sharno 5y ago
I wonder google doesn't show Lichess in the first page when I search "lichess"?

298 points by i0exception 5y ago

jka 5y ago

I was wondering what the cost per game is on lichess, and fortunately the linked spreadsheet includes the answer: lichess currently runs at a cost of $0.00022 per game (4545 games to the dollar).

shric 5y ago
That's good to know. I've played 12,316 games on lichess so almost $3 worth. Might donate $10 or so soon.
- DarylZero 5y ago
  Do you computer-analyze your games? One stockfish computer analysis must cost more than merely playing 1,000 unanalyzed games.
  - altvali 5y ago
    You're downloading the engine from the Lichess server but the analysis is done on your computer, with your resources.
    - kmm 5y ago
      Computer analyses are done on a distributed network called fishnet[0], not locally 0: https://github.com/niklasf/fishnet https://github.com/niklasf/fishnet
      - altvali 5y ago
        Except fishnet is used to analyze broadcasted games, puzzle discovery and others. Not used when analyzing your own games. You can easily test this by opening your task manager, monitoring your CPU usage, then opening a game you played on Lichess and toggling the aptly named "local evaluation". It's the toggle bar on the upper right, next to the text "Stockfish 14+ NNUE in local browser". You will see your CPU usage increase while the evaluation is on.
        wishawa 5y ago
        That is true when you’re playing in your browser. On the mobile app (at least the iOS version) you can request computer analysis for your game done on the server.
        ornicar 5y ago
        You can request server analysis for your own games, up to 40 per day. They use fishnet.
        kmm 5y ago
        No, that is an analysis of a single position, not a game. The big button under every game which says "Request Computer Analysis" is done on fishnet as well, it literally says "Stockfish 14.1 server analysis".
      - fho 5y ago
        <sarcasm> So ... time to implement fishcoins?</sarcasm> Analyzing a chess game or two is not exactly easily verifiable, but it should be somehow possible to shoehorn that into a proof of work thing.
    - agalunar 5y ago
      Only if you toggle local-evaluation in the analysis board. When you "request a computer analysis", the computation is done by fishnet [1] on volunteers' hardware. [1] https://github.com/niklasf/fishnet https://github.com/niklasf/fishnet
      - altvali 5y ago
        But can you request a computer analysis with fishnet anymore? I have a non-patron account and I can't find that option anywhere. Old forum posts seem to talk about it working by pressing Z and being limited, but when I press Z I get a message that it is disabled by default and a link to enable it. When pressing to enable, nothing happens.
        agalunar 5y ago
        It's been a while since I used Lichess anonymously, but I suspect you do need an account to request a server analysis – that way they can limit the number of analyses per day for a user [1]. [1] 40 a day per https://lichess.org/features https://lichess.org/features
        shric 5y ago
        You don't need to be a patron, you just need an account. One of the many amazing things about lichess is it's 100% free and donators get no extra privileges except a little logo next to their user
    - deleted 5y ago
      [deleted]
  - Kranar 5y ago
    Stockfish runs in your own browser using your own CPU.
    - llimllib 5y ago
      Not if you do a game analysis
  - mileycyrusXOXO 5y ago
    I think Stockfish usually runs clientside but it does look like they do some server side too https://lichess.org/blog/YOCx7hIAACUAgsUo/stockfish-14-has-arrived https://lichess.org/blog/YOCx7hIAACUAgsUo/stockfish-14-has-a...
  - jb_s 5y ago
    For this reason I wish I could run something locally to generate the same analysis UI and therefore not put any load on the Lichess servers.
  - shric 5y ago
    I do run a computer analysis on maybe 10% of games, but I also participate in fishnet (explained elsewhere in this thread) on a local machine so that hopefully offsets it

booleandilemma 5y ago

So, 4 developer salaries for a service used worldwide. Not bad at all.

tommek4077 5y ago

michaelcampbell 5y ago
> Maybe stop using aws or google cloud? Or, maybe start. Would at least be worth looking at to see if [the equivalents of] spot or reserved instances, Lambda's, Cloud Run, or other serverless etc. might save some money.
WastingMyTime89 5y ago
You need at least three admins to have 24h availability. I highly doubt you can save money by selfhosting at Lichess size.
- woadwarrior01 5y ago
  Hosting on the cloud does not magically make your availability issues go away.
  - theshrike79 5y ago
    No, but they make many availability issues Someone Else's Problem. When EC2 goes down, I need to maybe wake up and log in to see what happened. I see that all of us-east-1 is down (again), notify everyone involved: "It's down can't do anything, it'll come back up when AWS engineers get to it" and go back to bed. I can check up on it again during office hours. If this was a colocated server, I'd need to drag my ass out of bed and drive to the site to see what's up.
    - greedo 5y ago
      Drive to a colo? Most colos have 'remote hands' where you can have an onsite tech do stuff for you. There's also OOB management tools you can use (assuming the hardware isn't toast). AWS is really expensive if you have the skills to managed hardware yourself.

wodenokoto 5y ago

Where does funding come from?

shric 5y ago
Donations
- aloisdg 5y ago
  You can also buy some swag: https://lichess-org.myspreadshop.com/ https://lichess-org.myspreadshop.com/

BossingAround 5y ago

For those completely unaware (like I was), lichess [1] seems to be a chess server. Not sure whether they need AWS scale and couldn't cut costs down from 420k a year to something cheaper. Naively, this seems like something MS could sponsor by providing long-term discount. That said, I have no idea how much cheaper this would be on other cloud providers, or whether they even have the need to cut down costs. For all I know, they might be making 10x of that in donations. [1] http://lichess.org http://lichess.org

shric 5y ago
If you look at the cost breakdown, that includes several people's salaries and other things than just hosting. As for scale they do almost 100 million games per month and keep every game available for download and online access. I suspect donations ~= costs, which is why the salaries are quite modest to say the least.
tyingq 5y ago
A lot of the server costs have datacenter names that look like OVH ones to me. And the server costs seem to be ~$68k/year of the total.

_zooted 5y ago

> $68,600 in Servers

pessimizer 5y ago
> Candles $3,600 That being said, it's probably a lot cheaper than rewriting it to run on less.
- whalesalad 5y ago
  i will never financially recover from this

devit 5y ago

It's the minority of the total spend, but it seems the hosting costs are too high. For example, they are paying $266/month for 16 thread 128GB 3.8TB SSD, but you can rent a better machine for 99$/month from Hetzner (AX61-NVME). They are also spending $40k/year on "data protection services" (?!?) and 60k for "site moderation" (they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate).

kevingadd 5y ago
"they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate" at that point, you might as well not run the service. moderation isn't optional if you want a healthy community.
- hffftz 5y ago
Tenoke 5y ago
>$40k/year on "data protection services" (?!?) I'm guessing this is paying an employee/consultant to ensure they are compliant with everyone's policies. At that scale you have to care about every jurisdiction and letting the one dev guess at compliance might not be ideal. > 60k for "site moderation" (they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate). Can you even just not moderate? There's some stuff you legally have to due to say NetzDG for German users, and compliance with everyone else.
cillian64 5y ago
Everybody complains when someone gets inexplicably/stupidly banned by an AI and there are no humans to review it. Spending money on moderation by actual skilled people is how you solve the problem. This is one place I’m very happy to see them not skimping.
iKlsR 5y ago
> (they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate) Have you ever used the site? How does that work by not moderating? When I report someone for being racist, spamming or submit a game where I think someone cheated... who handles that, what forum or open service doesn't have some type of moderation and in this case were you want a seamless experience with your app as they do with chat, blog, forum and multiple daily tournaments.

whalesalad 5y ago

Lots of pets here, not many cattle. When you get to this scale, tools like Kubernetes make more sense. Then you can just think about your system in terms of "how much cpu/ram/storage do I need total?" Unless every one of those servers is running at or near capacity, there is a lot of cash being wasted there. There is also a maintenance cost, too. If one of these vital boxes goes down, what is the downtime implication and restoration cost? I am not saying any of this to be critical of Lichess. There are different ways to solve these problems, and their way is clearly working. This also happens slowly over many years, so it is hard/impossible to see the end state until you are there. The app is very quick and responsive. I got my ass handed to me on my first anon game :) My feedback is more for the community here in the context of using this as a byte-sized case study. At the end of the day we are reading and writing 1's and 0's to a network device, or a disk. Have to imagine you can run and persist chess games with a lot less resources.

Sebb767 5y ago
> When you get to this scale, tools like Kubernetes make more sense. If you look at the details[0], servers are really "just" 6k$ per month. A lot of that goes towards the databases, which you can't optimize much with k8s, so the optimization potential overall is already rather limited. If you then need a consultant or a developer to do the migration, you'll quickly outspend any potential savings. I'm a fan of k8s, but this is really a showcase for hardware being (comparatively) cheap and the scale that you can go to with conventional hardware. [0] https://docs.google.com/spreadsheets/d/1Si3PMUJGR9KrpE5lngSkHLJKJkb0ZuI4/preview https://docs.google.com/spreadsheets/d/1Si3PMUJGR9KrpE5lngSk...
- whalesalad 5y ago
  Why is kube and conventional hardware mutually exclusive? It’s just a different way to utilize resources. I don't see an issue with the costs - I see an issue with this massive list of inventory that needs to be carefully managed. Managing 30 boxes is not easy when they are all funny shapes and sizes. Are they all running the same OS? Same version? How do you roll out security updates? With kube, you just bring a new node online and then kill the old one. Done. My argument still remains - what happens when a vital box dies? How quickly can a new one be brought online? Is that automated? What about in the meantime, is service degraded or completely down? What about when the ES boxes shit the bed and you realize you actually needed a cluster of 5 nodes? Do you buy 5 new boxes that are carefully sized, or do you just scale your mega cluster from 4 to 5 and add a few new pods? Kube is not nearly as big and scary as this community makes it out to be.

amelius 5y ago

Main developer gets a monthly salary of $4705, is that normal in Europe? (Assuming Europe because they pay French taxes).

DC-3 5y ago
It's somewhat low but they've probably made a concious choice to accept a lower salary to work for a project they are passionate about.
throwbacktictac 5y ago
I don't know how much time he dedicates to running the service at this point but the salary seems light if it's full time work.
- Scarblac 5y ago
  It's his hobby project that he turned into this, it runs completely by his philosophy, and it's decent money for the country he's in. I think he's won.
a_humean 5y ago
Maybe a bit low for someone this experienced, but actually its just above the median salary for a developer in Paris. Developer salaries in Europe are not the astronomic salaries you see in the US in the major cities. Probably the highest paying in Europe is London in the UK, and even then its still less than 50% of what developers in SF make at probably around 75k USD per year/6-7k USD per month in London.
- laurentb 5y ago
  worth baring in mind that there's a lot of extra paid by the employer on top of the "gross" salary someone gets in Europe (especially in France) that someone in the US would have to cover by himself (hence why i't typical to see double that amount for someone in the US).
  - 5y ago

tholman 5y ago

Lots of people saying "You can improve costs here!" without factoring in the developer time/learning/upkeep dollars it would take to do that. Sometimes the best solution is the one you know that will be easiest for you.

marcrosoft 5y ago

5k in French taxes to run a non profit site, why?

henvic 5y ago
Because the state was never about making things easier for people.
RaitoBezarius 5y ago
Non profit does not mean that it do not have to pay taxes on the money it manipulates. Non profit orgs receive services from the state and they have a cost to fund social security and such. Other non profits which do not have a very nice situation can receive help from the state, so why not share the bits of success of Lichess to everyone else?
pyb 5y ago
It's the norm in France for employers to pay another euro of tax for each euro their employees earn.
- newaccount74 5y ago
  I assume with "tax" you mean "tax + health insurance + unemployment insurance + retirement savings etc"?
  - ta988 5y ago
    Yes and it was a mistake in my opinion. It make employees feel that the company is paying for these benefits and it makes the company believe that they are spending money for things they should not (I've heard so many CEOs complaining about having to pay "all those taxes for their employees"). I prefer a way where the employee is responsible for all costs with a deduction at the source to make sure you don't get nasty surprises at the end of the year. In term of numbers that's all the same, not in term of psychology.
    - duxup 5y ago

jeremyjh 5y ago

In this thread: people with zero knowledge who can cut costs drastically without impacting the service (that they have never used and don't understand).

deleted 5y ago
[deleted]
dang 5y ago
Putting others down doesn't help. If you know more than others, the thing to do is to share some of what you know, so the rest of us can learn. Then you're addressing the underlying ignorance as well as contributing to the ecosystem, instead of poisoning it further. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sort=byDate&type=comment&query=share%20know%20learn%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- jeremyjh 5y ago
  It's not a put down to say that there are people without knowledge of Lichess's operations or infrastructure. I'm one of those people. I don't presume to have any knowledge of effective cost cutting measures. Others do.
  - dang 5y ago
    It's a putdown to snarkily sneer at others in the community, which is why the site guidelines specifically ask you not to: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html. I understand the temptation to do this—we all feel it. That's why that guideline is in there. Such comments reliably lead to lower-quality threads, because they don't add anything that anyone can learn from. They just assert the commenter as somehow different from / better than others. That's not to say the comments you were criticizing aren't bad. Of course, shallow dismissals are bad. But it only makes things worse to respond with a different sort of shallow dismissal.

dmurray 5y ago

qeternity 5y ago
Haha yeah I think those specs would be 10x on AWS. GRA, RBX, etc - OVH about as cheap as it gets.
yunohn 5y ago
> bottom-of-the-barrel Indian devs with a handful of Western project managers It's really disappointing to see such casual racism on HN. Surely you could think of a better adjective than "bottom-of-the-barrel" to describe humans (Indians) that are intentionally hired at the lowest possible wages, by Western managers to maximize profits for a Western owned & led company?
- malteof 5y ago
  You can also read the post as “out of all Indian devs (who on average tend to be cheaper than devs from most other countries), hire the bottom of the barrel”. I doubt they meant that all Indian devs are bottom of the barrel.
  - lillecarl 5y ago
    I try to assume good intent, but I found it hard here considering the phrasing
    - Rapzid 5y ago
      I found it easy.
      - PieUser 5y ago
        I found it hard as well.
        5y ago

leros 5y ago

For reference $420k is about the cost of 2 employees at a typical tech company.

_fnqu 5y ago
Which typical tech company outside of Silicon Valley would that be? I’ve never seen wages at $210k for a „typical tech company“
- jlack 5y ago
  An employee costs more than just their salary.
- dijonman2 5y ago
  My company isn’t based out of SV, we pay 130-220k base depending on team and experience. Senior devs are 200k+
  - yurishimo 5y ago
    Hey, it’s me, your future employee!
- graton 5y ago
  A rough estimate is that the cost of an employee is 1.25x - 1.4x the salary of an employee in the US. With health insurance, Social Security taxes, 401K matching, office expenses, and more.
  - blip54321 5y ago
    That seems exceedingly low. I've generally seen 2x. It's a bit less than that as salary goes up, but I can't imagine hitting 1.25x.
- nojito 5y ago
  Wages are just part of the total cost of a employee.

rickdeveloper 5y ago

The author did an AMA on Reddit 9 months ago that I think is relevant here: https://old.reddit.com/r/chess/comments/mpasyl/i_started_lichess_ask_me_anything/gu9azgf/ https://old.reddit.com/r/chess/comments/mpasyl/i_started_lic... In particular: How they remain free: https://www.reddit.com/r/chess/comments/mpasyl/i_started_lichess_ask_me_anything/gu8mbuy/ https://www.reddit.com/r/chess/comments/mpasyl/i_started_lic... On his salary: https://www.reddit.com/r/chess/comments/mpasyl/i_started_lichess_ask_me_anything/gu8nv6o/ https://www.reddit.com/r/chess/comments/mpasyl/i_started_lic...

oneepic 5y ago

...for 12 seconds. https://www.youtube.com/watch?v=jHgZh4GV9G0&t=28s https://www.youtube.com/watch?v=jHgZh4GV9G0&t=28s

bluecalm 5y ago

It only 420k because they pay themselves minimum salaries. The founder/lead dev is easily worth mid 6 figures on the market and yet his salary is not even 60k EUR. It's a great project, great quality and run at a low cost but it's all that because it's run by people who sacrifice their financial situation to make their idea happen.

ta988 5y ago
And I wish they would be rewarded by their users even more.
- systemvoltage 5y ago
  They should charge. And I would pay.
  - kdmytro 5y ago
    They have users because the service is accessible. If they charge, many would playe elsewhere. There is a lot of competition.

c4m 5y ago

I love Lichess's feature comparison page (https://lichess.org/features https://lichess.org/features), which helps users decide whether to upgrade to a premium account.

iqanq 5y ago

JVM and then probably Cassandra as DB. Had they used something other than Java, it would've costed 10% of what it does. A rewrite may make sense. I don't use Lichess. If I did, I would have donated. But after knowing that they throw money down the sink like this, I would be hesitant.

OrvalWintermute 5y ago
I too looked at the infrastructure list and wondered. While I know many people are saying don't look at the cloud costs, when I look at those numbers it makes me think of CAPEX vs OPEX trades, and the OPEX here is non-trivial, but potentially a variable offset by some labor costs. I think some of this is being driven by a dated tech stack and technical debt. Now, I am not one to trivialize the work that may need to go into getting this efficient, but I think you are right, we should consider investing into modernization of the stack.
xunn0026 5y ago
Yeah, sure. Maybe you could rewrite it in assembler over the weekend and run it on a Raspberry Pi, right? I worked on Java codebases that absolutely squeezed every bit of performance from a machine and that would be very hard to do in other languages.
wishawa 5y ago
Tell us, this language 10x more efficient than Java, what is it?
ksec 5y ago
Considering Java and JVM power some of the highest performance server and HFT on the planet. Tell me about the language or ecosystem that give you 90% cost savings?
- vt85 5y ago
deleted 5y ago
[deleted]

altvali 5y ago

As someone who is working on a similar project (a chess mmo) and looking to validate a business model in order to get funding, how are they covering the costs? The answer usually is "donations", but there are only 402 patrons on https://lichess.org/patron https://lichess.org/patron and Thibault said on twitter that the average donation is $8. That's $3216/mo, less than a tenth of monthly costs. Is the rest coming from the swag store? From coaches? How much do the donations surpass the costs? Lots of missing pieces from the revenue puzzle.

llimllib 5y ago
That’s definitely not every patron
ornicar 5y ago
These are just the most recent. ATM there are 10,722 active patron accounts.
Jommi 5y ago
Maybe you should look at a model where your chess mmo runs on top of another platform? That way a lot of your costs would go down and you could focus on the core parts that you want to build. E.g. you dont really want to spend time paying for or optimizing for hundreds of simultaneous users from multiple countries.

anyfactor 5y ago

> Accountancy - 5,160 > Independent Audit - 6,000 I studied accounting at university and no one plans to do anything but auditing or consulting. I still love accounting, gives the most straight answer of how a business is run. Case in point.

freewilly1040 5y ago

It's truly amazing that a product with the reach of Lichess can be run so cheaply. Comments here criticizing the hosting costs extremely myopic - dev time isn't free, and doing a rewrite to chase down hosting savings isn't necessarily a good call. I just signed up for a monthly donation. I complain about the ad supported internet all the time, but had never donated to LiChess. This thread is a good reminder of how far dollars to support projects like this can go.

lhorie 5y ago
FWIW, in the earlier days of lichess, IIRC when it had barely broken the Alexa top 10,000 threshold, lichess ran on like 2 boxes despite serving millions of users. There's a good chance that many AWS slinging devs here have not actually had to deal with the volume of traffic that lichess sees. Half a mil yearly to run the entire lichess infrastructure, including dev salary and providing stockfish analysis for every user for free takes an insanely efficient setup that most big tech companies could only dream of.
- Tenoke 5y ago
  > providing stockfish analysis for every user for free Edit: As discussed in another subthread most of the analysis happens in the user's browser, the rest is run via a fishnet by volunteers.
  - lhorie 5y ago
    While we're in the topic of impressive technological achievements, it's probably also worth mentioning how crazy it is that just a few decades ago, Deep Blue running on a supercomputer was the state of art in chess analysis, and nowadays we can just casually say, oh lichess runs stockfish on users' phones via wasm to cut on server costs.
- epolanski 5y ago
  Most engineers in the world never saw such daily traffic. I doubt you can't make Alexa top 10k with a million of daily unique visitors.

chubot 5y ago

I love lichess! In addition to the web interface, it's got a great iPad interface too.

wenbin 5y ago

I’m actually surprised how cheap the server cost is , given the scale of linchess… Great job! Recent years, I have a feeling that HN commenters would assume server cost should always approach $0, not matter how big, or how different a real world online service is different from their single page web app side project :) For any server cost sharing posts, we’ll see people making comments like “I can run this under $10/month” . Ha

faangiq 5y ago

He should take a better salary and figure out a more viable business model. Running it this way is a risk to its future.

anthk 5y ago

And freechess.org ?

killingtime74 5y ago

I’m surprised French taxes are so high for a non profit

sharno 5y ago

I wonder google doesn't show Lichess in the first page when I search "lichess"?