3 ms·

Good question. The first step would be to just write 'censored' into the 'content' file and commit. That would remove it from the current view. Does it have t

Good question. The first step would be to just write 'censored' into the 'content' file and commit. That would remove it from the current view. Does it have to be removed from history? If the history is not public, probably not. Otherwise.. hmm.. the normal way to rewrite history would probably be too expensive. And would not play well with users who already have a fork of the repo. How about locating the file in the .git/objects/ directory and truncate it, or write 'censored' into the file? That would be a very cheap operation. Would that mess up anything?

LinuxBender 3y ago
If the history is not public, probably not. I suspect this could be risky. If someone reports that illegal content was uploaded and it is known the site operates from git then it would be known that the contents would still be in git. The reason I mention this is that people who would view this site as their competitor game the legal system by uploading illegal material via proxies and compromised machines and automatically reporting it to authorities, server providers and dns registrars to get the nodes, domains and admins cancelled. This is how many small sites have been shut down and why many small forums have gone private. The competitors are anyone with a vested financial interest in the existing platforms. I think I would want all traces of the content purged in a timely manor to mitigate this risk if I were running the site. Another tricky part of this is how git deals with binary blobs. The repos are going to get massive rather fast. I suspect truncating the binaries without making git aware may lead to errors. Even if they can be ignored it may get harder with time to troubleshoot the repo. I suppose you could make a prototype and load test it to see what happens when you upload a few TB of files and start truncating some of them.
- MrThoughtful 3y ago
  I think one can also run a git repo without history. That would solve the "dirty stuff in the history" problem. I don't think binary blobs should be included at the beginning. The "content" file should be just a text file.
  - LinuxBender 3y ago
    In my opinion by implementing git without history and linking to binaries is just a reinvention of phpBB [1] which already has powerful moderation capabilities, a very advanced permissions systems and file storage that has been battle hardened for a couple decades. The UI can be made to look modern using modular styles and by uploading modern emoji which phpBB already supports. The most work would be working on a style to make it look modern. phpBB can use Postgres or MySQL for the metadata and it links to file storage for attachments. There is also a very advanced permissions system with limits per file type. If the site grew big enough the file storage could be scaled with Ceph. The only thing they leave out is some tuning/hardening of php.ini disabling unused classes and functions [2] and implementing good outbound firewall rules to mitigate most 0-day's. Those are fairly straight forward to implement. [Edit] FWIW I tried vouching your submission that was flagged but then it was re-flagged. [1] - https://www.phpbb.com/ https://www.phpbb.com/ [2] - https://www.cyberciti.biz/faq/linux-unix-apache-lighttpd-phpini-disable-functions/ https://www.cyberciti.biz/faq/linux-unix-apache-lighttpd-php...
rektide 3y ago
Are they good questions though? It feels like github/gitlab/gitea solve 90% of these questions at scale fine. Personally these questions came off as a rude routine to me, one that prefigure whenever anyone suggests networked social systems. Healthy skepticism is good, just but people who spread disbelief dont rank high in my estimation. As for rewriting history, maybe use a month by month git repo, so if old content needs to be rebased the blast zone is contained. Job done. Then git submodules to collect each month.
- LinuxBender 3y ago
  It feels like github/gitlab/gitea solve 90% of these questions at scale fine. Used as a social platform vs code repository? Surely Github have enterprise class storage and networking so if the question is around hosting the platform on Github then maybe that works at a technical level assuming one is willing to take the risk of Microsoft cancelling their social platform when someone complains about a comment and Microsoft don't want whatever divisive content is on their platform. If self hosting I would be surprised if a single server or single VM could scale git to millions of people making comments, tens of thousands of comments being edited and thousands of comments being purged. My experience with managing large git servers is that it gets exponentially slower with mass manipulation of the repositories and that was on fairly beefy servers and that was just Jira, Fisheye, Crucible and Confluence and less than 10k developers, not millions of individuals. That is unless by self hosting we are talking about using automation to spin up many instances and even then it is not clear to me that git can scale horizontally like that. Even then that is not really self hosting, that is getting VC funding and starting a new company which draws even more attention and requires even more staffing to manage the content. I think there would need to be some simulated load testing that attempts to replicate real world social platform behavior to see where git starts to fall over otherwise we would just be speculating in circles. I suspect if this were attempted there would be multiple layers of caching nodes not doing git and trying to minimize or optimize edits and purges to the storage and that may not even be enough.

LinuxBender 3y ago

If the history is not public, probably not. I suspect this could be risky. If someone reports that illegal content was uploaded and it is known the site operates from git then it would be known that the contents would still be in git. The reason I mention this is that people who would view this site as their competitor game the legal system by uploading illegal material via proxies and compromised machines and automatically reporting it to authorities, server providers and dns registrars to get the nodes, domains and admins cancelled. This is how many small sites have been shut down and why many small forums have gone private. The competitors are anyone with a vested financial interest in the existing platforms. I think I would want all traces of the content purged in a timely manor to mitigate this risk if I were running the site. Another tricky part of this is how git deals with binary blobs. The repos are going to get massive rather fast. I suspect truncating the binaries without making git aware may lead to errors. Even if they can be ignored it may get harder with time to troubleshoot the repo. I suppose you could make a prototype and load test it to see what happens when you upload a few TB of files and start truncating some of them.

MrThoughtful 3y ago
I think one can also run a git repo without history. That would solve the "dirty stuff in the history" problem. I don't think binary blobs should be included at the beginning. The "content" file should be just a text file.
- LinuxBender 3y ago
  In my opinion by implementing git without history and linking to binaries is just a reinvention of phpBB [1] which already has powerful moderation capabilities, a very advanced permissions systems and file storage that has been battle hardened for a couple decades. The UI can be made to look modern using modular styles and by uploading modern emoji which phpBB already supports. The most work would be working on a style to make it look modern. phpBB can use Postgres or MySQL for the metadata and it links to file storage for attachments. There is also a very advanced permissions system with limits per file type. If the site grew big enough the file storage could be scaled with Ceph. The only thing they leave out is some tuning/hardening of php.ini disabling unused classes and functions [2] and implementing good outbound firewall rules to mitigate most 0-day's. Those are fairly straight forward to implement. [Edit] FWIW I tried vouching your submission that was flagged but then it was re-flagged. [1] - https://www.phpbb.com/ https://www.phpbb.com/ [2] - https://www.cyberciti.biz/faq/linux-unix-apache-lighttpd-phpini-disable-functions/ https://www.cyberciti.biz/faq/linux-unix-apache-lighttpd-php...

rektide 3y ago

Are they good questions though? It feels like github/gitlab/gitea solve 90% of these questions at scale fine. Personally these questions came off as a rude routine to me, one that prefigure whenever anyone suggests networked social systems. Healthy skepticism is good, just but people who spread disbelief dont rank high in my estimation. As for rewriting history, maybe use a month by month git repo, so if old content needs to be rebased the blast zone is contained. Job done. Then git submodules to collect each month.

LinuxBender 3y ago
It feels like github/gitlab/gitea solve 90% of these questions at scale fine. Used as a social platform vs code repository? Surely Github have enterprise class storage and networking so if the question is around hosting the platform on Github then maybe that works at a technical level assuming one is willing to take the risk of Microsoft cancelling their social platform when someone complains about a comment and Microsoft don't want whatever divisive content is on their platform. If self hosting I would be surprised if a single server or single VM could scale git to millions of people making comments, tens of thousands of comments being edited and thousands of comments being purged. My experience with managing large git servers is that it gets exponentially slower with mass manipulation of the repositories and that was on fairly beefy servers and that was just Jira, Fisheye, Crucible and Confluence and less than 10k developers, not millions of individuals. That is unless by self hosting we are talking about using automation to spin up many instances and even then it is not clear to me that git can scale horizontally like that. Even then that is not really self hosting, that is getting VC funding and starting a new company which draws even more attention and requires even more staffing to manage the content. I think there would need to be some simulated load testing that attempts to replicate real world social platform behavior to see where git starts to fall over otherwise we would just be speculating in circles. I suspect if this were attempted there would be multiple layers of caching nodes not doing git and trying to minimize or optimize edits and purges to the storage and that may not even be enough.