9 ms·
In search of the least viewed article on Wikipedia
- onos 4y agoGreat sleuthing. Is there a convenient alternative algorithm that might be used for random article that would also continue to work fine as more articles are added or removed?
- stevage 4y agoSeems like it would be pretty easy to just maintain an alternative index of articles, with an integer ID counting from 1 to N. Then just pick a random number from 1 to N. It could even be maintained entirely outside the Wikimedia servers, relying on database dumps.
- bawolff 4y agoI disagree that anything is wrong with the current algorithm. However there was a brief time period where randomPage used elasticsearch to get the random article instead.
- bombcar 4y agoAdded is easy. It’s removed that is the tricky part, though perhaps you could reuse IDs somehow. Maybe a hashing function could work.
- zarzavat 4y agoYou can make a binary tree where each node counts the total (recursive) number of children. Updating these counts is log(N). To insert a node, generate a random bit string: descend the tree and when the counts are equal then take the branch corresponding to that position in the bit string. When the counts are unequal, take the branch with the smallest count. To remove a node just remove it from the tree and update the counts up along its path to the root. This assumes that articles are added as often as they are removed. To sample, just generate another random bit string and traverse the tree according to it. But I agree with the sibling comment that the current non-uniform algorithm is fine for a cute “Random article” button.
- HALtheWise 4y agoIf the articles have some reasonably dense small IDs associated with them, then an easy algorithm is to simply pick a random ID between 0 and the max, check if it's a still-existing article, and repeat if not. There are plenty of distributed queue designs capable of distributing IDs to servers so they can be given to new articles, and re-using the id from a deleted article is fine to keep things denser.
- anewhnaccount2 4y agoThe database itself may internally store a more dense identifier than an autoincrementing primary key. For e.g. ctid in Postgres or, under certain conditions rowid in sqlite. These should be fully dense after a vacuum, and rejection sampling can be used to paper over tombstones generated between vacuums.
- blackboxlogic 4y agoFor the smallest change with the fairest result, I would reseed the random IDs on a schedule.
- amai 4y agoTo get a more equal spacing between the articles maybe one can make use of quasirandom numbers instead of random numbers: https://en.wikipedia.org/wiki/Low-discrepancy_sequence https://en.wikipedia.org/wiki/Low-discrepancy_sequence
- softgrow 4y agoShould be titled differently, all of the least viewed articles are now going to be viewed, lots. I'm really having trouble myself not visiting them. Mind you it's time some obscure moth species articles got some improvement :)
- simonh 4y agoIt's not as if 'not being visited' is some special protected status we need to preserve for posterity.
- thematrixturtle 4y agoThis is the interesting number paradox: the least interesting number in the world ceases to be boring once it has the interesting property of being the least interesting attached to it. https://en.wikipedia.org/wiki/Interesting_number_paradox https://en.wikipedia.org/wiki/Interesting_number_paradox
- McBeige 4y agoDoesn't that just mean the least interesting number definitely exists but is simply unknowable?
- puzzledobserver 4y agoOff the top of my head, I can think of two ways to resolve the paradox: 1. Reject the intuitive definition of "interesting"ness as being ill-formed. We need an objective definition non-self-referential definition of the word interesting. 2. Accept that there are interesting or uninteresting numbers, but their status as objects of interest is tied to a specific time. For example, 1729 might have been an uninteresting number until the conversation between Ramanujan and Hardy. Several of the examples on the Wikipedia page have such temporal limits on when they were / were not interesting.
- beowulfey 4y agoI would instead say it has more to do with “interestingness” being an impermanent quality. As an odd extension of this idea (I don’t know of an English equivalent), if this were Spanish we would probably use “estar” instead of “ser” to affix the description.
- bawolff 4y agoI don't find the random gap argument convincing. There are so many articles i think the gaps are going to be very small and reasonably close in size as to not matter (i have not done any analysis to back this up). Sure there will of course be some unlucky articles, but does that actually matter?
- IfOnlyYouKnew 4y agoIf you randomly draw z numbers from a bucket of numbers from 0 to 10000 times x, the gaps will differ because there is no mechanism that would make some number with close neighbors less or more likely to be drawn than any other remaining number. It’s only once your supply of numbers runs low that differences will start to equalize, reaching 1 when you have exhausted your supply. Besides, the author isn’t really making an argument. They are giving you actual data showing the differences to the next lowest number. It’s hard to argue with that.
- bawolff 4y agoRight, they said the absolute worst case outlier was a gap about 100x smaller than average. Which i guess depends on your views, but i would say that's not bad. > the gaps will differ because there is no mechanism that would make some number with close neighbors less or more likely to be drawn than any other remaining number. When a new article is inserted, there is a higher probability it will be inserted in a large gap than a small gap, so it should balance out. I suppose you're right,i am responding to an implied criticism to the randomness method that the author didn't make. They just offered it as explanation.
- IfOnlyYouKnew 4y ago> When a new article is inserted, there is a higher probability it will be inserted in a large gap than a small gap, so it should balance out. That’s a good point and I’m not entirely sure why it (appears to) not work that way. Maybe it’s because that interval has a higher likelihood, but there is no preference for numbers towards the middle, that would dissect it into (roughly) equal parts?
- notorandit 4y agoEasy answer: moths!
- _the_inflator 4y agoThis is good stuff! Besides finding hidden gems or undervalued articles, I would really like to see traction tracked. Now that rarely viewed articles are somewhat in the spotlight and get promoted, a before and after comparison would be interesting. Fun anyway.
- thelamest 4y agoDo you mean pageview stats for the mentioned articles? You can add them in this for example. Data for today are not collated yet, should be visible by tomorrow. https://pageviews.wmcloud.org/?project=en.wikipedia.org&platform=all-access&agent=user&redirects=0&range=latest-90&pages=DMZ//38 https://pageviews.wmcloud.org/?project=en.wikipedia.org&plat...
- bryanrasmussen 4y agoSince they have links to all of these least viewed articles they will soon cease to be the least viewed articles.
- inopinatus 4y agoc.f. https://en.wikipedia.org/wiki/Observer_effect_(physics) https://en.wikipedia.org/wiki/Observer_effect_(physics)
- yesenadam 4y agoAlso seems related to the interesting number paradox: "the smallest uninteresting number is itself interesting because it is the smallest uninteresting number, thus producing a contradiction." https://en.wikipedia.org/wiki/Interesting_number_paradox https://en.wikipedia.org/wiki/Interesting_number_paradox Incidentally, the last line of that article is: > The mathematician and philosopher Alex Bellos suggested in 2014 that a candidate for the lowest uninteresting number would be 247 because it was, at the time, "the lowest number not to have its own page on Wikipedia".
- inopinatus 4y agoperhaps resolvable by moving the self-reference to the definition e.g. "the smallest number not interesting for any other reason that its uninterestingness"
- kreetx 4y agoWonder what the least viewed articles were without the "random article" function?
- exo-pla-net 4y agoYou'd have a big stack of bot-generated 0 or 1 view articles about moths.
- jl6 4y agoSo Wikipedia, just like the real world, is filled with staggering quantities of different types of bugs.
- simonh 4y agoMay brain parsed that as if you were talking about software bugs for a second.
- IAmEveryone 4y agoNote that page view data is also available at https://pageviews.wmcloud.org/?project=en.wikipedia.org&platform=all-access&agent=user&redirects=0&range=latest-20&pages=List_of_longest_fish https://pageviews.wmcloud.org/?project=en.wikipedia.org&plat... (I didn't see this linked in the post. Apologies if I missed it)
- mgdlbp 4y agoAnd a quick way to get there is the Page information link in the sidebar. On the info page for each page[0] is a count and plot of views in the past 30 days, and at the bottom is a link to external tools like the one above. [0] example: https://en.wikipedia.org/w/index.php?title=Weimer_Township&action=info https://en.wikipedia.org/w/index.php?title=Weimer_Township&a...
- longrod 4y agoThe most interesting thing here is that all of the 6 million articles on Wikipedia have been seen by 1 person at least once. So if we take all humans and ask them to search 1 unique thing on Wikipedia, would there be any articles that would get 0 hits?
- tsimionescu 4y agoWell, presumably the person who contributed the article is guaranteed to see it, I don't think an article can exist with 0 views.
- zufallsheld 4y agoThere are many articles created by bots that may never have been seen by a human. https://wap.business-standard.com/article/pti-stories/wikipedia-bot-writes-10-000-articles-a-day-114071600987_1.html https://wap.business-standard.com/article/pti-stories/wikipe...
- smitop 4y agoNote that Lsjbot is only active on certain language editions; it doesn't edit English Wikipedia. There are a lot less bot-created articles on enwiki.
- enkid 4y agoThe article was about annual views, so not necessarily.
- deleted 4y ago[deleted]
- DavidSJ 4y agoInterestingly, two of these — one of the disambiguation pages, and one of the least-viewed non-disambiguation pages — were created by the same user, Carlossuarez46, and are both about a location in Iran. Furthermore, the user Ruigeroeland has contributed to three of the insect pages. So I guess these users have the distinction of having contributed to multiple of the least-seen articles on Wikipedia. (They probably also contributed to many widely-seen articles!)
- shmageggy 4y agoThis is addressed at the end of the article. Many are probably made with automated tools > For example, the 12-word stub Pottallinda (5 views last year) was created on 18 January 2011 by User:Ser Amantio di Nicolao, who happens to be the most active editor in all of Wikipedia (as measured by number of edits). Within 60 seconds of creating this page, the same editor also created Polmalagama, Polommana, Polpitiya, Polwatta, and dozens of other substantially identical articles.
- iamacyborg 4y agoYeah, he’s likely using AutoWikiBrowser for this, it’s a great tool for automating MediaWiki tasks.
- dmix 4y agoI noticed moths came up a couple times, a brief guess why: Wikipedia says it's "one of most speciose orders" (besides flies and beetles). But maybe it has the most pages because they are so easy to catch with a light in your backyard, that it's far easier to name them all than something parasitoid wasps [1]? Although this same paper says that "more species of beetles (>350,000) have been described than any other order of animal, insect or otherwise" [1] https://www.biorxiv.org/content/10.1101/274431v1.full.pdf https://www.biorxiv.org/content/10.1101/274431v1.full.pdf Just speculation...
- DonaldFisk 4y agoMother here. The examples listed in the article aren't of much use, as there's not much there you can use to identify species. If you're interested in moths, you'll visit a specialist web site, and if you aren't, you won't visit the Wikipedia pages either. The web site I use is https://www.ukmoths.org.uk https://www.ukmoths.org.uk, which has photographs of each (UK) species, as well as its range, food plant, and flying time. Regarding the number of beetle species, JBS Haldane (probably) said that "God is incredibly fond of beetles" (or words to that effect).
- beowulfey 4y agoAs an aside, is mother actually the term? My first readthrough of your comment had me envisioning an incredibly knowledgeable parent with a cool hobby before I realized what you meant!
- Taywee 4y agoAccording to Wiktionary, it's been used since at least 2001. I was similarly confused.
- myfonj 4y agohttps://en.wiktionary.org/wiki/moth-er https://en.wiktionary.org/wiki/moth-er is filed with hyphen ("moth-er"); hyphenless spelling ("mother") is listed as an alternative form. https://en.wiktionary.org/wiki/mother https://en.wiktionary.org/wiki/mother does not mention that fact, though. Nevertheless, what a nice new ambiguous word to spice up the context!
- Gigachad 4y agoLooking up “fox tossing” on Wikipedia ended up being just as barbaric as I imagined.
- Veen 4y agoThat page leads down an unpleasant Wikipedia rabbit hole. It's amazing to me that people thought "Goose pulling" was a pleasant pastime—tie a goose to a frame, ride towards it at a gallop, and grab it by the neck so its head comes off. https://en.wikipedia.org/wiki/Goose_pulling https://en.wikipedia.org/wiki/Goose_pulling
- deleted 4y ago[deleted]
- bombcar 4y agoAnyone who has had significant experience with Canada geese would understand this pastime.
- onychomys 4y ago> there’s unfortunately no easy way to sort out the least viewed pages, short of a very slow linear search for the needle in the haystack So this sentence made me wonder why he didn't actually just go do it, 6M pages isn't really all that big of a data set. Turns out that it's a problem of how the data is arranged. The raw files are divided by year/month/day/second[0], and then each of those seconds is a zipped file of about 500MB in size, where pages are listed like this: en.m Alcibiades_(character) 1 0 en.m Alcibiades_DeBlanc 2 0 en.m Alcibiades_the_Schoolboy 1 0 en.m Alcide_De_Gasperi 2 0 en.m Alcide_Herveaux 1 0 en.m Alcide_Laurin 1 0 en.m Alcide_de_Gasperi 1 0 en.m Alcides_Escobar 3 0 en.m Alcimus_(mythology) 1 0 with en.m getting a separate listing from the desktop em, and all the other country codes getting their own listings too. So just collating the data would be a huge job. The API also doesn't offer a specific list of all pages by time, so you'd have to go and make a separate call for each of the 6M pages for a given year and then collate that data too. [0]for example, https://dumps.wikimedia.org/other/pageviews/2019/2019-01/ https://dumps.wikimedia.org/other/pageviews/2019/2019-01/
- melony 4y agoMapReduce and dump everything into something like DuckDB.
- deleted 4y ago[deleted]
- t0astbread 4y agoWhy was the random function done this way? Is it not possible to index into a database table at a random position?
- bombcar 4y agoI assume it's for performance reasons, a "select next greater than X" being faster than "get a random row" - not sure if databases even have the concept of returning a random row, or how performant it is.
- K2L8M11N2 4y agoI can't think of a reason why getting an article with ID of randomIntBetween(0, count(articles)) would be slow (maybe UUID primary keys?)
- Sohcahtoa82 4y agoEven if you used UUIDs, couldn't you just "select count(id) from articles" to get the list of articles, generate a random number between 0 and the count, then "select id from articles offset {index}"?
- farleyflip 4y agoI wouldn't equate the least viewed with unpopular. They may not show up well in search for lack of content. I've come across plenty of articles which were low in views and only have a handful of lines of content. I would love to expand these articles, but the "culture" of Wikipedia actively works against this.
- Kaze404 4y ago> I would love to expand these articles, but the "culture" of Wikipedia actively works against this. Can you expand on this? I've only had positive experiences editing for Wikipedia as a newbie.