12 ms·

I had to decom a major wiki in a very large company not long ago. I can give you four reasons why these things happen, possibly without notice. 1) The wiki sof

I had to decom a major wiki in a very large company not long ago. I can give you four reasons why these things happen, possibly without notice. 1) The wiki software and database have been abandoned for years. There is no maintenance and no further release. 2) It will stop working shortly. Like, it doesn't start on Ubuntu > 14 or current MySql at all. 3) It already stopped working and/or the content is already lost. Can be an accidental deletion or the 10 year old server passed away. 4) Security vulnerability. Remote code execution / SQL injection in the wiki software. That can't be fixed because point one. I wrote a longer blog post on software death cycle in companies https://thehftguy.com/2019/12/10/why-products-are-shutdown-the-case-of-verizon-and-yahoo-groups/ https://thehftguy.com/2019/12/10/why-products-are-shutdown-t...

tempestn 7y ago
It shouldn't be hard to save a static copy of the actual HTML and replace the dynamic site with that though. Or at least give others enough time to do the same.
- dontbeunethical 7y ago
  Not that hard? You'd need to process each page then data mine it.
  - eof 7y ago
    A company that can build a game engine should probably be able to crawl a site and save an html dump?
  - biggestdecision 7y ago
    Just archive.org every page on the wiki. Their api will let you do 15/minute.
  - smacktoward 7y ago
    HTTrack (https://www.httrack.com/ https://www.httrack.com/) makes tasks like this trivial.
    - imtringued 7y ago
      It doesn't run a full browser engine so it won't work with the vast majority of websites.
- user5994461 7y ago
  It's not hard, except when it is and it takes weeks of work. From my own experience: 1) httrack wasn't able to crawl anything for some reasons. wget only with special flags available in the latest version. 2) All the links are broken. Wiki have their own linking system between pages that are processed on the fly. The archive with all links hardcoded to wikidump.example.com/wikidump/page really doesn't work in place. 3) More challenges with saving pictures and attachments. 4) Unicode characters in some pages, that broke the page and/or the crawler. 5) Infinite crawling and a ton of useless pages. Consider that every URL with a query string is a separate page. /page?version=50&comparewith=49 6) Crawling large amount of documents takes forever. Could be an hour wasted on each try. Consider tens or hundreds of thousands of unique URLs to save, see point five. Really wish the crawler could parallelize and run on the same host as the wiki.
  - tempestn 7y ago
    Good points. Perhaps I should have said, "It should be very possible," or similar instead. I'm sure there would be challenges, but I would expect them to be surmountable.

tempestn 7y ago

It shouldn't be hard to save a static copy of the actual HTML and replace the dynamic site with that though. Or at least give others enough time to do the same.

dontbeunethical 7y ago
Not that hard? You'd need to process each page then data mine it.
- eof 7y ago
  A company that can build a game engine should probably be able to crawl a site and save an html dump?
- biggestdecision 7y ago
  Just archive.org every page on the wiki. Their api will let you do 15/minute.
- smacktoward 7y ago
  HTTrack (https://www.httrack.com/ https://www.httrack.com/) makes tasks like this trivial.
  - imtringued 7y ago
    It doesn't run a full browser engine so it won't work with the vast majority of websites.
user5994461 7y ago
It's not hard, except when it is and it takes weeks of work. From my own experience: 1) httrack wasn't able to crawl anything for some reasons. wget only with special flags available in the latest version. 2) All the links are broken. Wiki have their own linking system between pages that are processed on the fly. The archive with all links hardcoded to wikidump.example.com/wikidump/page really doesn't work in place. 3) More challenges with saving pictures and attachments. 4) Unicode characters in some pages, that broke the page and/or the crawler. 5) Infinite crawling and a ton of useless pages. Consider that every URL with a query string is a separate page. /page?version=50&comparewith=49 6) Crawling large amount of documents takes forever. Could be an hour wasted on each try. Consider tens or hundreds of thousands of unique URLs to save, see point five. Really wish the crawler could parallelize and run on the same host as the wiki.
- tempestn 7y ago
  Good points. Perhaps I should have said, "It should be very possible," or similar instead. I'm sure there would be challenges, but I would expect them to be surmountable.