5 ms·
> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our arti
by dig1 1y ago
> We are already underway on the arXiv CE ("Cloud Edition") project... replace the portion of our backends still written in perl and PHP...re-architect our article processing to be fully asynchronous,.. we can deploy via Kubernetes or services like Google Cloud Run...improve our monitoring and logging facilities
Why do I smell someone from G was there and sold them fancy cloud story (or they wanted VMs and reseller sold them CloudRun)? Anyway, goodbye simplicity and stability, hello exorbitant monthy costs for the same/less service quality. Would love to be wrong.
- elif 1y agoI think this is more likely a move to prevent Cornell funding from getting tied up with dictations about what gets published. But in all likelihood someone was probably just like "we're tired of doing ops on 2 decades old stacks"
- johann8384 1y agoIt sounds like they are just using GCP. It doesn't really change that.
- elif 1y agoYea but GCP was state of the art 20 years ago, php+perl was already crufty
- ordersofmag 1y agoSo state of the art it wasn't even available yet (preview launch was in 2008).
- elif 1y agoSounds about right. I remember hearing about it first in a talk being given by Doug Crockford at my university around that time. It blew my mind. I thought it was like gcc for the Internet. It's kind of wild that in the interim we have experienced the complete rise and fall of mongodb, node.js, even today the react paradigm are all expressions of this tiny little functional scripting language..
- whatever1 1y agoGoogle cloud can easily move the the instance to a region that is not science/free speech hostile
- londons_explore 1y agoThis.... I bet arXiv was run on server hardware costing under $10k before... And now it'll end up costing $10k per month (with free credit from Google which will eventually go away and then arXiv will shut down or be forced to go commercial)
- falcor84 1y agoI assume they would still have the serving code they use now and if they do choose to go back to maintaining it on their own hardware they'll always have that option. It seems they just don't want that anymore.
- gapan 1y agoThat will be several years down the line, when everything has bit-rotten to death and nobody in the team remembers how the old setup worked.
- londons_explore 1y agoThat's why they're being encouraged to use cloud run and all the other cloud functionality so a migration back becomes very hard.
- perihelions 1y agoarXiv budgets about $88,000/year for server costs as of 2019, (pdf) https://info.arxiv.org/about/reports/arXiv_CY19_midyear.pdf https://info.arxiv.org/about/reports/arXiv_CY19_midyear.pdf
- deleted 1y ago[deleted]
- sightbroke 1y ago> This is a project to re-home all arXiv services from VMs at Cornell to a cloud provider (Google Cloud). They are already using VMs but one of the things it'll do is: > containerize all, or nearly all arXiv services so we can deploy via Kubernetes or services like Google Cloud Run And further state: > The modernization will enable: - arXiv to expand the subject areas that we cover - improve the metadata we collect and make available for articles, adding fields that the research community has requested such as funder identification - deal with the problem of ambiguous author identities - improve accessibility to support users with impairments, particularly visual impairments - improve usability for the entire arXiv community
- notpushkin 1y agoContainers part I can understand. Why not spin up a tiny Docker Swarm (or k3s/k0s) cluster instead of straight out going to Google though?
- kelnos 1y agoBecause those other things require more maintenance effort to run. Getting creative is often just a pain in the ass. Doing the standard things, walking the well-trod path, is generally easier to do, even if it may not be the cheapest or most hardware/software-efficient thing to do.
- johann8384 1y agoMoving to K8s, adding in additional instrumentation, just sounds like some new folks took over or joined the project and are doing some renovations. Seems like pretty standard stuff, doesn't really seem as sinister as you make it to be.
- jstummbillig 1y agoI noticed that while everyone on hn is quite clever, we are regularly not clever enough to assume that other people in similar settings are just as clever, and recognize when they probably spent a lot more time thinking about an issue we just skim the headline of.
- wordofx 1y ago[flagged]
- fuckbrownpeople 1y ago[dead]
- kadushka 1y agoThe people on both ends of that conversation (google vs cornell) are clever but the result will probably be enshittification.
- keepamovin 1y agoImpossible to argue with HN's regression to entropy cynicism hehehe :)
- jeffbee 1y agoI'm torn on flagging comments that throw out "enshittification". Do you feel that this stands in for actual thought?
- kadushka 1y ago[flagged]
- Analemma_ 1y agoI'm going to fling that response right back at you. "Enshittification" is not a generic term for "update I don't like", it describes a specific dynamic that happens when a company inserts itself as a middleman into a two-sided market. arXiv could get worse in other ways, but enshittification in particular can't happen to it, that's a category error. I think you should offer better thoughts instead of mad-libbing in buzzwords where they don't apply. Enshittification is actually a useful concept, and I don't want it to go the way of "FUD", which had a similar trajectory in the later years of Slashdot where people just reduced it to a useless catch-all phrase whenever Microsoft said anything about anything.
- specialp 1y agoI don't think it is that. I work for an org with close ties to arXiv, and just like us they are getting a lot more demand due to AI crawling. As a primary source of information there is a lot of traffic. They do have technical issues from time to time due to this demand, and I think their stability is just due to the exceptional amount of effort they take to keep it going. They are also getting more submissions and interest. Kubernetes does add complexity but it does add a lot of good things too. Auto scaling, cycling of unhealthy pods, and failover of failed nodes are some of them. I know there is this feeling here sometimes that cloud services and orchestrated containers are too much for many applications, but if you are running a very busy site like arXiv I can't see how running on bare metal is going to be better for your staff and experience. I don't think they are naive and got conned into GCP as the OP alludes to. They are smart people that are dealing with scaling and tech debt issues just like we all end up with at some point in our careers.
- CharlieDigital 1y agoSound like all they needed was a CDN if the problem is AI crawlers. Adding auto-scaling compute just increases costs faster.
- specialp 1y agoCDN is one part of strategy to deal with load. But it is not the only solution unless your site is exclusively static content. Their search, APIs, submission pipelines, duplicate detectors and a lot of other things are not going to be powered by CDNs.
- motorest 1y agoThank you for the insight. It's very easy to prescribe simple solutions when we are oblivious to the actual problems being solved.
- coliveira 1y agoAll these services can be throttled to deal with AI. I don't see this as a justification. The idea that a service like arXiv should be run as a startup is, simply put, foolish.
- moralestapia 1y ago>we can deploy via Kubernetes Oh noes ... they got scammed