5 ms·

Building a legacy search engine for a legacy protocol

137 points by benjojo12 9y ago

MrRadar 9y ago
Do you still have the raw crawl data? If so, have you considered uploading it to the Internet Archive to help preserve it for the future?
- benjojo12 9y ago
  Nice idea, I've rolled up all of my data and crawl database and put it on the internet archive: https://archive.org/details/gopher-may-2017.tar https://archive.org/details/gopher-may-2017.tar
  - MrRadar 9y ago
    Thanks! It's amusing how the entirety of Gopher fits into a 1.6 GB archive compared to the petabytes (exabytes?) it would take to archive the entirety of the WWW.
    - bastawhiz 9y ago
      The internet is many, many, many exabytes large at this point. Hell, I'd bet YouTube alone is a few hundred exabytes at minimum.
  - snakeanus 9y ago
    Could someone please help seed the torrent? It has been stuck at 99.9% for the past day.
    - MrRadar 9y ago
      The Internet Archive should be seeding all the torrents they host. Have you tried restarting your torrent client, forcing it to recheck your local data, or re-downloading and re-adding the torrent file?
edward_rolf 9y ago
Very cool and a nice write-up, thank you. I know absolutely nothing about the AltaVista software so this was interesting. I'm wondering, were there limitations in scale? Did the search engine have other types of technical limitations? If you had to choose from modern software, what would have been your choice?
mysterydip 9y ago
I actually thought about doing this a couple months ago but never got past the "cool idea in my head" phase. Great to see this!
mkup 9y ago
Windows 98 is a flaky OS, IIRC it used to crash with BSoDs multiple times a day even on real hardware. That AltaVista standalone crawler software probably could run much better under NT 4.0 or Windows 2000. And NTFS allows for much larger datasets than FAT32. NTFS hasn't much changed since that time, by the way.
- digi_owl 9y ago
  Maybe i am lucky, but what most "BSODs" on 9x amounted to was a "the program is unresponsive".
- derefr 9y ago
  Old Windows versions crashed so much because hardware manufacturers at the time slapped together poor drivers, and Microsoft couldn't do much about that if they wanted people's computers to work at all. The #1 thing that as improved since then, was simply that more of the drivers are now written by Microsoft themselves. The #2 thing is Microsoft getting enough power over hardware makers to force driver quality-assurance and signing on them. Which is all to say: if you run Windows 98 on hardware that has good drivers written for it—or especially on virtual hardware whose "drivers" are just paravirtualized calls into a modern OS kernel—your copy of Windows 98 won't be BSoDing any time soon.
  - Macha 9y ago
    The OP's issues seem to indicate that this is not the case, the system would just crash under hdd load in those virtualized systems.
  - TazeTSchnitzel 9y ago
    There's also the thing where, uh, Windows NT has full memory protection. On Windows 9x it was trivial for any app to hose the OS, because critical memory regions were unprotected.
- benjojo12 9y ago
  Yup, 98 is trash for this use, This is why it makes is so exciting to try and run a "production service" on!
mempko 9y ago
Wow, this really brings me back. Well done.
digi_owl 9y ago
Gopher strikes me as something of a file explorer for the net.
DamonHD 9y ago
AltaVista brings back memories! And in the archive I was able to find some of our old text from '92! It's alive I tell you. I didn't use gopher much if at all when I had the chance back in mumble mumble '80s/'90s so I'm pleased to finally use it. Surely it'll be back in fashion soon?
deleted 9y ago
[deleted]
e12e 9y ago
Fantastic article. Wonder if the altavista software would run under reactos?
anigbrowl 9y ago
Gopher was awesome back in its day, and I'm sad that the semantic web efforts (its nearest modern aspirant) have done so poorly.
Keyframe 9y ago
Anyone remembers back in 93/94 hearing about HTTP and saying 'gopher, but with images? Neat! Oh, but I need a graphics terminal... damn'.
spc476 9y ago
Wow ... weird to think I'm running one of the 370 gopher servers left in the world: gopher://gopher.conman.org/
rurban 9y ago
But still, I would rather use xapian over AltaVista. Same technology (inverted index), but stable and much better. Win98 as service is just asking for too much trouble. And no, not ElasticSearch. No java in the house, C++ is enough.

137 points by benjojo12 9y ago

Do you still have the raw crawl data? If so, have you considered uploading it to the Internet Archive to help preserve it for the future?

benjojo12 9y ago
Nice idea, I've rolled up all of my data and crawl database and put it on the internet archive: https://archive.org/details/gopher-may-2017.tar https://archive.org/details/gopher-may-2017.tar
- MrRadar 9y ago
  Thanks! It's amusing how the entirety of Gopher fits into a 1.6 GB archive compared to the petabytes (exabytes?) it would take to archive the entirety of the WWW.
  - bastawhiz 9y ago
    The internet is many, many, many exabytes large at this point. Hell, I'd bet YouTube alone is a few hundred exabytes at minimum.
- snakeanus 9y ago
  Could someone please help seed the torrent? It has been stuck at 99.9% for the past day.
  - MrRadar 9y ago
    The Internet Archive should be seeding all the torrents they host. Have you tried restarting your torrent client, forcing it to recheck your local data, or re-downloading and re-adding the torrent file?

edward_rolf 9y ago

Very cool and a nice write-up, thank you. I know absolutely nothing about the AltaVista software so this was interesting. I'm wondering, were there limitations in scale? Did the search engine have other types of technical limitations? If you had to choose from modern software, what would have been your choice?

mysterydip 9y ago

I actually thought about doing this a couple months ago but never got past the "cool idea in my head" phase. Great to see this!

mkup 9y ago

Windows 98 is a flaky OS, IIRC it used to crash with BSoDs multiple times a day even on real hardware. That AltaVista standalone crawler software probably could run much better under NT 4.0 or Windows 2000. And NTFS allows for much larger datasets than FAT32. NTFS hasn't much changed since that time, by the way.

digi_owl 9y ago
Maybe i am lucky, but what most "BSODs" on 9x amounted to was a "the program is unresponsive".
derefr 9y ago
Old Windows versions crashed so much because hardware manufacturers at the time slapped together poor drivers, and Microsoft couldn't do much about that if they wanted people's computers to work at all. The #1 thing that as improved since then, was simply that more of the drivers are now written by Microsoft themselves. The #2 thing is Microsoft getting enough power over hardware makers to force driver quality-assurance and signing on them. Which is all to say: if you run Windows 98 on hardware that has good drivers written for it—or especially on virtual hardware whose "drivers" are just paravirtualized calls into a modern OS kernel—your copy of Windows 98 won't be BSoDing any time soon.
- Macha 9y ago
  The OP's issues seem to indicate that this is not the case, the system would just crash under hdd load in those virtualized systems.
- TazeTSchnitzel 9y ago
  There's also the thing where, uh, Windows NT has full memory protection. On Windows 9x it was trivial for any app to hose the OS, because critical memory regions were unprotected.
benjojo12 9y ago
Yup, 98 is trash for this use, This is why it makes is so exciting to try and run a "production service" on!

mempko 9y ago

Wow, this really brings me back. Well done.

digi_owl 9y ago

Gopher strikes me as something of a file explorer for the net.

DamonHD 9y ago

AltaVista brings back memories! And in the archive I was able to find some of our old text from '92! It's alive I tell you. I didn't use gopher much if at all when I had the chance back in mumble mumble '80s/'90s so I'm pleased to finally use it. Surely it'll be back in fashion soon?

deleted 9y ago

[deleted]

e12e 9y ago

Fantastic article. Wonder if the altavista software would run under reactos?

anigbrowl 9y ago

Gopher was awesome back in its day, and I'm sad that the semantic web efforts (its nearest modern aspirant) have done so poorly.

Keyframe 9y ago

Anyone remembers back in 93/94 hearing about HTTP and saying 'gopher, but with images? Neat! Oh, but I need a graphics terminal... damn'.

spc476 9y ago

Wow ... weird to think I'm running one of the 370 gopher servers left in the world: gopher://gopher.conman.org/

rurban 9y ago

But still, I would rather use xapian over AltaVista. Same technology (inverted index), but stable and much better. Win98 as service is just asking for too much trouble. And no, not ElasticSearch. No java in the house, C++ is enough.