4 ms·

WordPress Plugin Mirror Downloader (Proof of Concept)

83 points by rob 2y ago

debacle 2y ago
What's that pg quote about the internet routing around such and such?
- lolinder 2y ago
  > The Net interprets censorship as damage and routes around it. Apparently [0] John Gilmore [1] said it. [0] https://quoteinvestigator.com/2021/07/12/censor/?amp=1 https://quoteinvestigator.com/2021/07/12/censor/?amp=1 [1] https://en.m.wikipedia.org/wiki/John_Gilmore_(activist) https://en.m.wikipedia.org/wiki/John_Gilmore_(activist)
  - sneak 2y ago
    Unfortunately, with the sizes of data these days, this isn’t remotely true, if it even ever was.
    - vntok 2y ago
      P2P networks are made for this.
indigodaddy 2y ago
centminmod itself is pretty great if you want a stable easy lemp stack for EL. And makes WP management easy as well
3eb7988a1663 2y ago
Based on Cloudflare R2 S3 object storage costs for 33GB of WordPress plugins, for a private WordPress plugin mirror would cost less than US$0.35/month though will rise as new plugin versions get downloaded and stored and if under generous R2 Class A/B write and read operation free monthly quota, the R2 operation costs would be free. The Cloudflare Worker costs based on past 24hrs metrics was 765K Worker requests at median of 2.1ms CPU time. So with 100K Worker requests/day free, cost would essentially be the $5/month Cloudflare Worker subscription fee as overages won't apply under their generous paid US$5/month subscription plan includes up to 10 million requests per month at average 10ms average CPU time. All up, it would be US$5.35/month starting costs which will rise as number of WordPress plugins and Worker usage increases as outlined here. Cloudflare CDN bandwidth is free of charge, so no need to worry about egress bandwidth fees. Shockingly affordable.
- compootr 2y ago
  > Cloudflare CDN bandwidth is free of charge free of charge, until sales shows up: https://news.ycombinator.com/item?id=40481808 https://news.ycombinator.com/item?id=40481808
  - viraptor 2y ago
    Yeah, it's very much a "try it, first one is free" type of business.
  - vundercind 2y ago
    Also, unless something's changed, the free tier only technically covers "web content". Going outside HTML and related material directly supporting the display of ordinary web pages (javascript referenced by your HTML, some light image serving, CSS) puts you outside what's covered by the TOS. You can't count on it for that at all. IIRC that also goes for the other "self serve" plans, so includes paid plans. You need an enterprise plan (probably $5k+/m) to distribute non-Web-page-related files without violating the TOS. People often get away with it as long as they don't go too wild, but you're in even riskier territory than one already is using non-enterprise Cloudflare.
    - esdf 2y ago
      Cloudflare does allow R2 (& some other "Developer Platform" services) for non-HTML content. They made some TOS changes in the past to be more explicit about it.
    - yawnxyz 2y ago
      wait a minute the docs say (https://developers.cloudflare.com/r2 https://developers.cloudflare.com/r2) You can use R2 for multiple scenarios, including but not limited to: Storage for cloud-native applications Cloud storage for web content Storage for podcast episodes Data lakes (analytics and big data) Cloud storage output for large batch processes, such as machine learning model artifacts or data sets
      - 2y ago
sneak 2y ago
I feel like the solution to wordpress-the-org being mismanaged is not to build better replacement infra around wordpress, but to build better publishing tools that don’t burn a mysql connection and require php get executed on every pageview.
- jzb 2y ago
  WordPress has literally millions of Web sites in production, tons of add-on software, and an ecosystem of companies and people providing solutions for it. Suggesting that the solution is "well, just start from scratch with something else" is unlikely to get you very far. It is far easier and more likely that the community around WordPress would rally around replacing WP.org than starting over from scratch with a new toolkit. Some people might flee WP altogether, but the majority of people and companies just want to get things done - not seeing them having a lot of tolerance for or interest in having to start again from square one.
  - Dalewyn 2y ago
    >Suggesting that the solution is "well, just start from scratch with something else" is unlikely to get you very far. To be fair, the first and best solution most people in these parts have is "Fork it, I'm going to make my own Linux distro." or something along those lines. No, they don't get very far but I will credit their consistency.
  - neya 2y ago
    Bad software gets popularised all the time, there is nothing wrong with having alternatives and having more choices. I strongly think we need some good alternatives to Wordpress, as someone who has been in the publishing industry in the last decade. Also, Wordpress doesn't scale well (without getting expensive), the code is bad, it is poorly managed, security issues are all over the place. It's about time we got some alternatives.
    - etchalon 2y ago
      There are already an unfathomable number of WordPress alternatives.
mediumsmart 2y ago
Maybe they could just host the newsletter and contact form plugins to cover 97% of WordPress sites and email other plugin updates to the remaining 3% on a weekly basis?
safety1st 2y ago
This also exists - https://github.com/bluehost/pluginmirror https://github.com/bluehost/pluginmirror Bluehost operated a plugin mirror for a while, and published all their code.
akshat 2y ago
We have also published our tool to mirror the WordPress repository. We took a different approach. Our tool, Morpheus, mirrors the exact WordPress API. It fetches and maintains only the latest versions of plugins, themes, and the WordPress core. This limits disk space considerably. There’s also a plugin that connects to WordPress.org by default but falls back to Morpheus if needed. We’re still working on improvements. Right now, we only support English-language updates, but we’re adding more languages and features. Our goal is to support the entire API. https://github.com/blogvault/morpheus https://github.com/blogvault/morpheus
deleted 2y ago
[deleted]
pbowyer 2y ago
If WordPress users are going to start using mirrors then WordPress needs to add and support package signing. Otherwise we've added another way to backdoor WordPress sites, using a malicious mirror.
- vbtechguy 2y ago
  +1 that would be very useful. This mirror implementation at least mirrors the official wordpress.org plugin json metadata and checksum files so you can verify as well https://github.com/centminmod/wordpress-plugin-mirror-poc#how-it-works https://github.com/centminmod/wordpress-plugin-mirror-poc#ho... Though still limiting as you can only checksum files after plugin installation. You probably need a staging WordPress install to verify mirror plugins before rolling out for production usage

83 points by rob 2y ago

debacle 2y ago

What's that pg quote about the internet routing around such and such?

lolinder 2y ago
> The Net interprets censorship as damage and routes around it. Apparently [0] John Gilmore [1] said it. [0] https://quoteinvestigator.com/2021/07/12/censor/?amp=1 https://quoteinvestigator.com/2021/07/12/censor/?amp=1 [1] https://en.m.wikipedia.org/wiki/John_Gilmore_(activist) https://en.m.wikipedia.org/wiki/John_Gilmore_(activist)
- sneak 2y ago
  Unfortunately, with the sizes of data these days, this isn’t remotely true, if it even ever was.
  - vntok 2y ago
    P2P networks are made for this.

indigodaddy 2y ago

centminmod itself is pretty great if you want a stable easy lemp stack for EL. And makes WP management easy as well

3eb7988a1663 2y ago

Based on Cloudflare R2 S3 object storage costs for 33GB of WordPress plugins, for a private WordPress plugin mirror would cost less than US$0.35/month though will rise as new plugin versions get downloaded and stored and if under generous R2 Class A/B write and read operation free monthly quota, the R2 operation costs would be free. The Cloudflare Worker costs based on past 24hrs metrics was 765K Worker requests at median of 2.1ms CPU time. So with 100K Worker requests/day free, cost would essentially be the $5/month Cloudflare Worker subscription fee as overages won't apply under their generous paid US$5/month subscription plan includes up to 10 million requests per month at average 10ms average CPU time. All up, it would be US$5.35/month starting costs which will rise as number of WordPress plugins and Worker usage increases as outlined here. Cloudflare CDN bandwidth is free of charge, so no need to worry about egress bandwidth fees. Shockingly affordable.

compootr 2y ago
> Cloudflare CDN bandwidth is free of charge free of charge, until sales shows up: https://news.ycombinator.com/item?id=40481808 https://news.ycombinator.com/item?id=40481808
- viraptor 2y ago
  Yeah, it's very much a "try it, first one is free" type of business.
- vundercind 2y ago
  Also, unless something's changed, the free tier only technically covers "web content". Going outside HTML and related material directly supporting the display of ordinary web pages (javascript referenced by your HTML, some light image serving, CSS) puts you outside what's covered by the TOS. You can't count on it for that at all. IIRC that also goes for the other "self serve" plans, so includes paid plans. You need an enterprise plan (probably $5k+/m) to distribute non-Web-page-related files without violating the TOS. People often get away with it as long as they don't go too wild, but you're in even riskier territory than one already is using non-enterprise Cloudflare.
  - esdf 2y ago
    Cloudflare does allow R2 (& some other "Developer Platform" services) for non-HTML content. They made some TOS changes in the past to be more explicit about it.
  - yawnxyz 2y ago
    wait a minute the docs say (https://developers.cloudflare.com/r2 https://developers.cloudflare.com/r2) You can use R2 for multiple scenarios, including but not limited to: Storage for cloud-native applications Cloud storage for web content Storage for podcast episodes Data lakes (analytics and big data) Cloud storage output for large batch processes, such as machine learning model artifacts or data sets
    - 2y ago

sneak 2y ago

I feel like the solution to wordpress-the-org being mismanaged is not to build better replacement infra around wordpress, but to build better publishing tools that don’t burn a mysql connection and require php get executed on every pageview.

jzb 2y ago
WordPress has literally millions of Web sites in production, tons of add-on software, and an ecosystem of companies and people providing solutions for it. Suggesting that the solution is "well, just start from scratch with something else" is unlikely to get you very far. It is far easier and more likely that the community around WordPress would rally around replacing WP.org than starting over from scratch with a new toolkit. Some people might flee WP altogether, but the majority of people and companies just want to get things done - not seeing them having a lot of tolerance for or interest in having to start again from square one.
- Dalewyn 2y ago
  >Suggesting that the solution is "well, just start from scratch with something else" is unlikely to get you very far. To be fair, the first and best solution most people in these parts have is "Fork it, I'm going to make my own Linux distro." or something along those lines. No, they don't get very far but I will credit their consistency.
- neya 2y ago
  Bad software gets popularised all the time, there is nothing wrong with having alternatives and having more choices. I strongly think we need some good alternatives to Wordpress, as someone who has been in the publishing industry in the last decade. Also, Wordpress doesn't scale well (without getting expensive), the code is bad, it is poorly managed, security issues are all over the place. It's about time we got some alternatives.
  - etchalon 2y ago
    There are already an unfathomable number of WordPress alternatives.

mediumsmart 2y ago

Maybe they could just host the newsletter and contact form plugins to cover 97% of WordPress sites and email other plugin updates to the remaining 3% on a weekly basis?

safety1st 2y ago

This also exists - https://github.com/bluehost/pluginmirror https://github.com/bluehost/pluginmirror Bluehost operated a plugin mirror for a while, and published all their code.

akshat 2y ago

We have also published our tool to mirror the WordPress repository. We took a different approach. Our tool, Morpheus, mirrors the exact WordPress API. It fetches and maintains only the latest versions of plugins, themes, and the WordPress core. This limits disk space considerably. There’s also a plugin that connects to WordPress.org by default but falls back to Morpheus if needed. We’re still working on improvements. Right now, we only support English-language updates, but we’re adding more languages and features. Our goal is to support the entire API. https://github.com/blogvault/morpheus https://github.com/blogvault/morpheus

deleted 2y ago

[deleted]

pbowyer 2y ago

If WordPress users are going to start using mirrors then WordPress needs to add and support package signing. Otherwise we've added another way to backdoor WordPress sites, using a malicious mirror.

vbtechguy 2y ago
+1 that would be very useful. This mirror implementation at least mirrors the official wordpress.org plugin json metadata and checksum files so you can verify as well https://github.com/centminmod/wordpress-plugin-mirror-poc#how-it-works https://github.com/centminmod/wordpress-plugin-mirror-poc#ho... Though still limiting as you can only checksum files after plugin installation. You probably need a staging WordPress install to verify mirror plugins before rolling out for production usage