5 ms·
I'll argue any civilized programmer should have a Wikipedia dump downloaded onto their machine. They're surprisingly small, and it saves you from having to use
by marginalia_nu 3y ago
I'll argue any civilized programmer should have a Wikipedia dump downloaded onto their machine. They're surprisingly small, and it saves you from having to use slow and unreliable APIs to do these types of basic processing tasks.
They also let you do less basic processing tasks that would have been too expensive to expose over API.
- teaearlgraycold 3y agoI learned how expensive hashmaps and hashsets are through Wikipedia dumps. I did some analysis of the most linked-to pages. Countries were among the highest. Hash sets for holding outgoing edges in the link graph ended up causing my program to exceed my laptop’s memory. Plain old lists (Python) were fine, though. And given there aren’t a crazy number of links per page using lists is fine performance wise.
- angrais 3y agoWhy did lists require less memory? Was it because you only held a subset of keys in the lists?
- teaearlgraycold 3y agoLists in Python have a integer for their size and a pointer for each element. Sets presumably have some number of buckets that are used to put pointers in, but many more buckets are allocated than get used in small sets.
- wenyuanyu 3y agoSort them, and use a vector of vectors for the adjacency list... Or better still use a graph processing library or graph database to manage that for you...
- marginalia_nu 3y agoThis is a fairly large data set indeed. The memory overhead (which is probably something like 4-8x for hash maps?) can start to become fairly noticeable at those sizes. Since Wikipedia posts already have a canonical numeric ID, if map semantics are important, I'd probably load that mapping into memory and use something like roaringbitmap for compressed storage of relations.
- kuschku 3y agoHow'd the hashset exceed your laptop memory, if the whole dump is just 22GB? You should be able to fit the entire dataset in RAM.
- marginalia_nu 3y agoThat is a compressed dump you are looking at. The uncompressed data is much larger. Link graphs in general can grow quite big. Also, not every laptop has 32 GB RAM.
- johnnyanmac 3y agoI'm still sticking with 16GB on my laptop so that would exceed my current RAM. That may also cut close for a 32GB machine anyway, since the OS and other programs may not let you access all your physical RAM.
- teaearlgraycold 3y agoI think my laptop had 8GB at the time
- downboots 3y agoDoes the download dump include edit history?
- marginalia_nu 3y agoWell there are full database dumps, but they're quite a bit too big to be of much practical use. I'm usually working with the text-only OpenZim version, which cuts out most of the cruft.
- aragonite 3y agoRelatedly: to drastically improve Wikipedia loading speed for personal browsing purposes, do not stay logged in to your Wikipedia account. The reason as explained here (see top reply by baowolff) https://news.ycombinator.com/item?id=36114477 https://news.ycombinator.com/item?id=36114477
- rafram 3y agoTo be honest, three-tenths of a second per page load just doesn’t matter very much to me. Logging in and out all the time will take much longer.
- sneed_chucker 3y agoWell, I specifically stay logged in so that I can opt out of the redesign they dropped a year or so ago. Never made an account before that.
- hiAndrewQuinn 3y ago1. Download http://static.wiki/ http://static.wiki/. 2. Run it locally on https://datasette.io/ https://datasette.io/. 3. ??? 4. Profit?
- marginalia_nu 3y agoI built my own at encyclopedia.marginalia.nu, but basically, yes.