13 ms·
Rappers, Sorted by Size of Vocabulary
- konceptz 12y agoWhat I would like to see, is this same comparison done against album sales with the implication of mainstream vs. underground.
- danielsf 12y agoI kept this focused on vocab so that the data viz was very straightforward and easy to digest/draw insights from. I've had many requests for album sales to be added, and I plan to as soon as possible :)
- Aqueous 12y agoGreatly enjoyed the analysis but while I was reading it I felt a lot like this guy: https://www.youtube.com/watch?v=GKlDBi0cyIA https://www.youtube.com/watch?v=GKlDBi0cyIA
- ballstothewalls 12y agoThis is a great graph, but I think it would be neat if a y-axis was thrown in. My first thought was album sales or some other metric of popularity that help you find specific rappers quick instead of going through the huge bunch of little pics.
- allan_ 12y agowhere is KRS-ONE?
- twic 12y ago4585 unique words.
- camus2 12y agoI would love the same chart but sorted by vulgarity.
- selimthegrim 12y agoMaybe this will help me answer that nagging question at the back of my brain: What does DJ Khaled actually _do_?
- devindotcom 12y agoCouldn't find Aceyalone - I thought he'd be in the top 10, I guess he wasn't included.
- crusty 12y agoTop 10? I would put money on him being at least #2 and giving Aesop Rock a run for his money for #1 (depending on which albums his 35,000 words fall on - he's got a solid discography). I just put on A Book of Human Language(https://www.youtube.com/watch?v=lwVNp42l3Xo https://www.youtube.com/watch?v=lwVNp42l3Xo - full album or https://www.youtube.com/watch?v=GnsCO0Fxw3A https://www.youtube.com/watch?v=GnsCO0Fxw3A - solid song selection). Along with that, Wu Tang as the only crew analyzed? how about Freestyle Fellowship (Aceyalone's crew), Quannum Collective, Jurassic 5. I'd also like to see Mos Def on the list, along with everyone from Quannum, and the Soulsides and SoundBombing volumes. Also, (unique words : total words) might be an interesting scoring method, and would allow comparison over entire works regardless of their individual volumes of output. Or choosing a random sample of # of words as opposed to first # of words, as someone who started publishing as a young buck may take a hit for early immaturity.
- officialjunk 12y agoMos def is in there
- crusty 12y agothanks, must have missed him
- russelluresti 12y agoAceyalone would be a great addition. I'd also like to see Chali 2na and the whole Jurassic5 crew.
- Ryanmf 12y agoOP: Did your analysis of MF DOOM include his work alongside Madlib as Madvillian or his various other pseudonyms (King Geedorah, Viktor Vaughn, etc.)? I find it a little hard to believe he's not at least in the Wu Tang/Canibus/KK cluster, if not #1 overall.
- Tycho 12y agoYeah I would have though Doom would be very high. But the density of his lyrics perhaps stem more from allusions/references and humour than from the words themselves.
- airfoil 12y agoAgree. I was surprised not to see DOOM as well. Another MC I think would score pretty highly is Chino XL.
- sizzle 12y agoI can't take this list seriously until DOOM is at the top, I agree with you guys. Daniel Dumile is on his own level no doubt.
- joefkelley 12y agoSeriously, DOOM is in his own league. At one point in "All Outta Ale", he rhymes "3-4-methylenedioxymethamphetamine" with "oxyacetaline." Also, probably my favorite individual rhyme of all time, from "Meat Grinder": "Borderline schizo, sort of fine tits though"
- rthomas6 12y agoThis infographic doesn't take into account other rappers possibly copying earlier really influential artists, making the earlier influential artists rank lower. More generally, it would be cool to see this chart ranked by the amount of original words present in the first 35,000 lyrics that were not present yet at the albums' time of publication.
- jarnix 12y agoHow many words in "fo shizzle ma nizzle" ? 4 or 0 ?
- danielsf 12y ago4!
- oakaz 12y agoWhy Jedi Mind Tricks is not counted? He'd be the first in this list; https://www.youtube.com/watch?v=TlZgiK6FiO0 https://www.youtube.com/watch?v=TlZgiK6FiO0
- VeejayRampay 12y agoJedi Mind Tricks is not one person. It's two rappers (Vinnie Paz and Jus Allah).
- pandler 12y agoJus Allah has been in and out of JMT. http://en.wikipedia.org/wiki/Jedi_Mind_Tricks http://en.wikipedia.org/wiki/Jedi_Mind_Tricks
- oakaz 12y agoIt doesn't matter, they are still rappers and I bet they'd be one of the top
- VeejayRampay 12y agoYou're right. And I agree, they most likely would. Army of the Pharaohs would be up there as well.
- Tycho 12y agoI've been wanting to do some NLP on rap genius's corpus for ages. This is a great analysis. What I had thought of is write a program to detect ghostwriting. Rappers probably have some sort of lyrical 'DNA' in the construction of their verses. How often they use certain words, number of words per line, number of unique words per song, ratio of adjectives to nouns, that kind of thing. You could probably unmask some ghost-writing secrets. Looking at the analysis here, it's interesting to see some clustering in the results. IMO the second cluster is the sweet spot: Wu Tang's excessive invention of vocabulary is cool but probably detracts from the poetic effect. Meanwhile rappers like 2Pac are just kind of boring IMO, at least going by their lyrics alone.
- seizethecheese 12y agoMany here seem to be interpreting vocabulary size as a signal for quality. When it comes to rap I completely disagree. Firstly, the repetition is rap's main ingredient. I read an article a while ago where researchers found that listening to a spoken phrase that is looped activates the same part of the brain as music, which helps explain this phenomenon. Personally, if I want food for thought I read. Rap is not an intellectual pursuit. I've been perusing rappers on this list, and the top artists have not been good at all to my ears. It seems that the best rappers are in the middle, and being on either extreme is a negative signal.
- TheCoelacanth 12y ago> It seems that the best rappers are in the middle, and being on either extreme is a negative signal. Objectively, I don't think that is an accurate assessment. There are highly respected artists all across the chart, including the extremes. For instance, Wu-Tang Clan and its members at the upper extreme and Kanye, Snoop and Tupac at the lower extreme.
- seizethecheese 12y agoYeah I agree, that statement was definitely a stretch. There are well respected rappers across the spectrum, which is my main point: that the interpretation of the chart as a proxy for quality is wrong.
- pandler 12y ago> Firstly, the repetition is rap's main ingredient. Is it though? I don't dispute that sheer vocabulary size isn't a sign of quality, but that seems like a very ignorant generalization of rap.
- TheCoelacanth 12y agoRepetition is a key ingredient in all genres of music. You would be hard pressed to find many significant pieces of music that don't use repetition.
- 12y ago
- dnautics 12y agoHow about a 2d visualization with a sliding 10000 word window, with the y axis as unique words out of 10k and the x aaxis time. Are there cultural trends that are time dependent? Did young mc and Del use more words than contemporary artists? Did their trends as artists follow the global trend over time?
- ignacioelola 12y agoI would love to see the same analysis across different music styles. How compare vocabulary size of Madonna, Bob Dylan and Justin Bieber?
- danielsf 12y agoI only have rap data, sadly :(
- ignacioelola 12y agoMatt, I can very easily gives you corpus for other artists, I'll send you an email
- zerohm 12y agoYeah, I'd like to see where The Clash rank.
- ff10 12y agoReally surprised MF Doom is not ranked higher – are his side projects included?
- zopticity 12y agoLil Jon should be at the bottom with 7 words: "Yeah!", "Okay!", "Shots!" and "Turn down for what?"
- axx 12y agoHe's not, because you forgot: "WHAAAAAT?", "SKIT", "SKIT" and "SKIT!".
- m_mueller 12y agoI'd be interested in how Nerdcore rappers compare to this, such as MC Frontalot or Professor Elemental.
- thegasman 12y agoNo mention of MF Doom? Metalface? Doom? Victor Vaughan? (All the same gentleman from LA)
- xkarga00 12y agoActually he is from UK
- joshschreuder 12y agoMF DOOM is on there pretty much right on the Shakespeare line. It's not clear if it includes his work as Danger / Villain / etc.
- b3b0p 12y agoWas it mentioned where the data was sourced from? I'm not seeing anything and I went back and checked. Did I miss it?
- mryan 12y ago> All lyrics are provided by Rap Genius, but are only current to 2012. My lack of recent data prevented me from using quite a few current artists.
- loso 12y agoI enjoyed reading this chart but I hope it doesn't reinforce the bias that some fans have that word complexity is the only way to tell if a rapper is good or not. There are several ways to judge the strength and weaknesses of a rapper. Complexity is one of them, flow is another. Story telling ability is also another very strong in indicator. The best rappers are able to bring a mix while some are just so strong in one area that they explode no matter if they are really weak in other areas.
- harryh 12y agoTo fully understand rap, we must first be fluent with its meter, rhyme, and figures of speech. Then ask two questions: One, how artfully has the objective of the song been rendered, and two, how important is that objective. Question one rates the song's perfection, question two rates its importance. And once these questions have been answered, determining a song's greatest becomes a relatively simple matter. If the song's score for perfection is plotted along the horizontal of a graph, and its importance is plotted on the vertical, then calculating the total area of the song yields the measure of its greatness.
- hatu 12y agoIt's kind of like putting book authors in this kind of a list. Using more words and more complex ones doesn't really say anything about the quality of the writing. Pretty interesting data-set for fun though.
- throwaway13qf85 12y agoWilliam Faulkner once criticised Ernest Hemingway by saying that he "had never been known to use a word that might send the reader to the dictionary." Hemingway responded by saying, "Poor Faulkner. Does he really think big emotions come from big words? He thinks I don't know the ten-dollar words. I know them all right. But there are older and simpler and better words, and those are the ones I use."
- jomtung 12y agoKillah Priest should be grouped with Wu-Tang.
- danielsf 12y agoOP: he's an associate, not a member
- prg318 12y agoThank you based god!!
- Totient 12y agoI wonder where things like classic rock / broadway musicals / opera / etc. fits on this spectrum. I really appreciate including Shakespeare and Moby Dick on the spectrum, but I'd still like some more perspective. For that matter, I wonder how many unique words I use every day.
- kenjackson 12y agoThis is an interesting analysis. I love the fact that E-40 is about on par with Shakespeare. I'm sure he would take it as a compliment to be called the modern day Shakespeare.
- danielsf 12y agoauthor here: thanks!
- deleted 12y ago[deleted]
- skylan_q 12y agoKool Keith should be exempt from this list. He's not from any of the 4 regions listed, but from Jupiter.
- dylanz 12y agoI checked for him immediately once I saw this chart. Between Doc Oc, Dr. Dooom and Black Elvis, he has some insanely weird lyrics. We stuck together when one of my parakeets died You broke down and cried, for the love of animals I used to always cut the legs off a roach See if he'll stay there on a piece of tissue And give him a piece of toast That morning, he would wake up and be gone What, the insect had a ambulance?"
- jboggan 12y agoI had never heard this particular Kool Keith rap before but as soon as I read the lyrics I could hear his very distinctive delivery pattern. I listened to the track on YouTube and was not surprised it was just as I have imagined. How distinctive.
- meepmorp 12y agoAnd, does his Dr. Octagon work count as Kool Keith or it's own thing? Is his work in Ultramagnetic MCs counted or just solo stuff?
- deleted 12y ago[deleted]
- christiangenco 12y agoFor those who didn't get the reference: > Dr. Octagon is a persona created and used by American rapper Keith Matthew Thornton, better known as Kool Keith...Dr. Octagon is an extraterrestrial time traveling gynecologist and surgeon from the planet Jupiter. [1] 1. http://en.wikipedia.org/wiki/Dr._Octagon http://en.wikipedia.org/wiki/Dr._Octagon
- sizzle 12y agoFrom the song 'Earth People' " First patient, pull out the skull, remove the cancer Breaking his back, chisel necks for the answer Supersonic bionic robot voodoo power Equator ex my chance to flex skills on Ampex With power meters and heaters gauge anti-freeze Octagon oxygen, aluminum intoxicants More ways to blow blood cells in your face React with four bombs and six fire missiles Armed with seven rounds of space doo-doo pistols You may not believe, living on the Earth planet My skin is green and silver, warhead looking mean Astronauts get played, tough like the ukelele As I move in rockets, overriding, levels Nothing's aware, same data, same system" [...] Hook x4 Earth People, New York and California Earth People, I was born on Jupiter. yup, Jupiter indeed. God I love this song.
- orblivion 12y agoThis looks at the first so many lyrics in each rapper's career. Aesop Rock came out with some weird stuff right off the bat. I wonder if some of these other rappers became more sophisticated over time. Maybe an average per song would be better, or average uniques per word, would be better.
- iLoch 12y agoI agree, perhaps the 35,000 most recent words would be better.
- sfrank2147 12y agoThe problem with average per song is that you "use up" words in every new song, so all things being equal each marginal song has progressively fewer new words.
- plorg 12y agoI bet you could get something insightful from plotting "unique words" versus "total words" - That might give a good idea of the amount of repetition over time, the length or quantity of output, and the total vocabulary.
- danielsf 12y agohere's what this looks like. ugly as sin as useless for comparing rappers. http://www.mdaniels.com/vocab/scatter.png http://www.mdaniels.com/vocab/scatter.png love your other ideas – hopefully can do them later.
- htk 12y ago"Each word is counted once, so pimps, pimp, pimping, and pimpin are four unique words" So much for the modern Shakespeares on the list.
- VMG 12y agoAs well as the old Shakespeare who was ranked according to the same rules.
- htk 12y agoI haven't read all his work, but I guess Shakespeare didn't use "hangn" and "hangin" as an alternative to "hanging". The author could validate the words against a dictionary, but it would still be flawed due to conjugations being counted as different words.
- GFK_of_xmaspast 12y agoWhat about conjugations like hang'd vs hanged to fit the meter?
- octo_t 12y agounlike shakespeare who was so high class and never made 'your mom' jokes or used any toilet humour or anything like that.
- crdoconnor 12y agoor jokes about "nothing".
- redacted 12y agoAnd literally made pimp jokes: http://poetry.rapgenius.com/William-shakespeare-hamlet-act-2-scene-2-annotated#note-1567189 http://poetry.rapgenius.com/William-shakespeare-hamlet-act-2...
- riggins 12y agoI find it hilarious that DMX is dead last. I've now got empirical evidence of what I always thought. I think DMX rhymes words with themselves more than any rapper I've ever heard.
- jeffandersen 12y agoX gon' Give it to ya? Nope.
- poink 12y agoI'm pretty sure this fails to take into account DMX's rich canine vocabulary.
- ch4s3 12y agoI said out loud before clicking the link that DMX would likely be dead last.
- aw3c2 12y agoSampling bias.
- ziziyO 12y agoI think Rick Ross would give DMX a run for his money. I've heard him rhyme a word with the same word before (Atlantic).
- andybak 12y agoI would have been rather surprised not to see Aesop Rock fairly high up the list. I was reading the Rap Genius pages for a few of his tracks the other week and the sheer density of wordplay was fairly overwhelming. It is rap for geeks though ;)
- deleted 12y ago[deleted]
- benihana 12y agoI'd really like to see this broken down by established vocabulary and made up vocabulary. I think that would really start to show who were the best lyricists on both ends. Rappers with a lot of made up words might be on the far left, and rappers with a lot of unique words that aren't made up would be on the far right. Both sides of the scale would show rapping talent on different dimensions. Influential rappers like E-40 who add new words to the vocabulary, and wordy rappers like Aes on the right who use a really dense and descriptive vocabulary.
- Blahah 12y agoWould also be nice to see size of the corpus controlled for. What's the rate at which they use new words?
- TheCoelacanth 12y agoThis chart already does that. The corpus for each artist is their first 35,000 words of lyrics.
- coherentpony 12y ago>Rappers with a lot of made up words might be on the far left It's interesting that you think that. I'd recommend you look at this: http://www.shakespeare-online.com/biography/wordsinvented.html http://www.shakespeare-online.com/biography/wordsinvented.ht... At what point do 'made up' words become 'established'. After they've been published? If so, every made up word in a song should be considered 'established'.
- benihana 12y agoI didn't mean to imply that being on the far left or being made up meant 'worse' in any way. Quite the opposite in fact - I think that if someone can create a word or phrase that becomes normal in everyday usages, it's a sign of... something. Maybe influence, maybe genius. Like you said, Shakespeare first used words that we consider common. I have no idea at what point a word becomes 'established' though.
- unfunco 12y agoThis is fascinating. I'm only a recent listener of hip-hop (primarily because of Earl Sweatshirt and Odd Future) and I'm in awe of the vernacular. And similarly, as a boredom exercise a few weeks ago I did some lexical analysis of the song Timber (the monstrosity was being constantly played on the radio at the time) and here's what I came out with: "83.1% of the words in the lyrics are five letters or less, 58.9% are four letters or less. The lexical density (the number of unique words divided by the total number of words, multiplied by one-hundred) is 29.1%. There is only one word in the song which has three or more syllables. Eleven people were involved with the writing of the song, each of them capable of producing just nine unique words each."
- philtar 12y agoYou must be using big words in case someone does the analysis on HN comments. Definitions: Vernacular: ?? Lexical analysis (in this case): ratio of unique words to non unique words Lexical density: What persent of the words is unique?
- unfunco 12y agoThe paragraph in quotes is copied and pasted from when I wrote that a few weeks ago, there's a definition in parentheses following lexical density. Analysis is a word that should not need a definition attached to it (you have used it yourself in your comment.) Vernacular is commonly used in the United Kingdom. Google will provide you with a definition.
- orbitur 12y ago> Eleven people were involved with the writing of the song, each of them capable of producing just nine unique words each. I'm not sure why this is notable when you consider that lyrics are probably the least important aspect of a song intended for the top 40. If you take a moment to listen to the melody and production, you'd probably see why it's credited to 10+ people. That song is a well-oiled machine.
- unfunco 12y ago
- nmac 12y agoIts a nice touch including portmanteaus and 'incorrect' ebonics on the list (like "ery'day"), since authors like shakespeare, joyce and others took the same liberties with language. Arguably, that's how language develops and makes it interesting to study and think about. The OP could have easily stuck to words in the OED, kudos.
- danielsf 12y agoop here: thanks yo!
- dmourati 12y agoI'm a big fan of the project and the way it is presented. Not sure why Wu-Tang features so prominently but I guess I'm okay with that. Kool Keith should be broken down further into his constituent parts. I also would have thought the Beastie Boys would have run higher.
- thinkpad20 12y agoIs Del tha Funkee Homosapien on this list? I'd be curious, since he has pretty non-standard lyrics.
- dbrian 12y agoThe author promised Reddit that he would be added. http://www.reddit.com/r/Music/comments/24omhw/rappers_sorted_by_size_of_vocabulary/ch974ju http://www.reddit.com/r/Music/comments/24omhw/rappers_sorted...
- Xcelerate 12y agoI'd never heard of Aesop Rock before and decided to check out some of his music. He sounds a lot like Del.
- benihana 12y agoI think I remember reading Aesop say Del influenced him a great deal. If you like Aesop, check out El-P, Cannibal Ox, Illogic, and basically anyone who was associated with Def-Jux.
- thrownaway2424 12y agoThey are also on the same same label. Definitive Jux has some quality product. If you're looking for expansive vocabularies you should consider exploring other dorky rappers like Scroobius Pip.
- dnlserrano 12y agoAwesome. This guy should definitely work for RapGenius.com.
- danielsf 12y agoauthor here: the data is straight from RP, so I have been working with them :)
- shaggyfrog 12y agoIncredibly, a list about rapper vocabulary is missing anyone associated with nerdcore. I'm interested to see where the likes of MC Frontalot, Wordburglar, YTCracker, etc. rank on that scale...
- redacted 12y agoThe author addresses the "Why is $RAPPER not on the list?" on the larger reddit threads: http://www.reddit.com/r/hiphopheads/comments/24neym/rappers_sorted_by_size_of_vocabulary/ http://www.reddit.com/r/hiphopheads/comments/24neym/rappers_... http://www.reddit.com/r/dataisbeautiful/comments/24nw9p/rappers_sorted_by_size_of_vocabulary_xpost/ http://www.reddit.com/r/dataisbeautiful/comments/24nw9p/rapp... My gut feeling is that the nerdcore guys would be decidedly middle-of-the-road, but I'd love to see the result!
- danielsf 12y agoauthor here: indeed. thanks for adding this to the thread.
- shaggyfrog 12y agoDownvoters, explain please?
- GFK_of_xmaspast 12y agoI don't see why novelty acts would be relevant.
- shaggyfrog 12y agoWhy do you dismiss nerdcore as "novelty"?
- Oxxide 12y agobecause most of the artists are gimmick acts.
- 12y ago
- Aardwolf 12y ago> Shakespeare’s vocabulary: across his entire corpus, he uses 28,829 words, suggesting he knew over 100,000 words Why does that suggest he knew over 100k words? Maybe it means he knew 28,829 and used all of them? Would he really know over 70,000 words he never used in his works? What would those 70,000 words be? Probably very obscure ones. How can you know that many obscure ones?
- mbillie1 12y agoVocabulary has for a while been considered in terms of 'receptive' and 'productive' capacity, with the assumption being that ones 'receptive' vocabulary can be larger, since it is easier to hear/read/understand a word than it is to use it correctly in reading/writing (this is not necessarily the popular opinion anymore [http://www.readingconnect.net/web/FILES/english-language-and-literature/elal_LSWP_Vol_4_Pignot_Shahov.pdf http://www.readingconnect.net/web/FILES/english-language-and...] but may provide the context for the claim about Shakespeare). The notion is that you are able to understand more words than you commonly use in your speech/writing, which is on some level intuitive, although of course it is an empirical question.
- maaku 12y agoAssumptions tend to break down at the extremes. E.g. Shakespeare actually invented a lot of the obscure words he used.
- sroerick 12y agoThe 28K figure is acheived by counting multiple spellings of the same word. Shakespeare lived before dictionaries, so there was never a single standard way to spell a word.
- Crito 12y agoI'm curious if words that Shakespeare invented count. There are many words that we see first used by Shakespeare, though some of them were probably words invented during his time by others with him merely being the first to record (in documents that survived until today).
- 1ris 12y agoShouldn't that be adjusted to the size of the text corpus?
- jemfinch 12y agoThe text corpus was normalized to the first 35k words for each rapper.
- moron4hire 12y agoThis might be the best-made infographic I've ever seen.
- danielsf 12y agoauthor here: oh man. flattered.
- tokipin 12y agoJust a note, those artists don't necessarily use all their vocabulary. Eminem for example clearly holds back on his vocabulary. Rap is as much an art as anything can be so there are all sorts of factors. Be careful what you might want to draw here other than curiousity.
- x3ro 12y ago> Eminem for example clearly holds back on his vocabulary. What makes you say this?
- likeclockwork 12y agoI'm sure the answer to this question is "He looks more articulate than the average rapper."
- omegaham 12y agoGibes regarding racism aside, some people seem more articulate due to the fact that they carry themselves differently during interviews. Of course, many musicians will keep a persona going during interviews as well, so it's still not a very reliable metric. The most extreme example I've seen was Marilyn Manson, but there are plenty of musicians who rap / sing about really inane stuff and then show that they're way smarter than the way they present themselves with their music.
- matwood 12y agothere are plenty of musicians who rap / sing about really inane stuff and then show that they're way smarter than the way they present themselves with their music. As mentioned in the article, Jay-Z even raps about doing just that.
- deleted 12y ago[deleted]
- richardlblair 12y agoHe would spend hours studying the dictionary.
- sbierwagen 12y agoCool to see Canibus so high in the rankings. It'd also be cool to add the members of AOTP to the analysis.
- danielsf 12y agoauthor here: hit me up with questions you've got.
- just1dude 12y agoan artist that meets the requirements for the data, AZ. He's got 5+ albums, some of which were gold and it'd be interesting as I believe he may also be the highest selling solo artist in the top 10 aside from Ghostface
- evan_ 12y agoDid you remove hooks/choruses from the artist's 35,000 word count?
- danielsf 12y agonope. they are in there. if it's on the rap genius lyrics page, I included it.
- kenshiro_o 12y agoBefore making this study, what were your predictions? Would have have expected Wu-Tang and GZA to be near the top? What did you expect the average to be? It would be very interesting to do something similar for rockers too.
- danielsf 12y agoI only have hip hop data, sadly. I didn't expect wu-tang to be on the top since it would go across 11 different vocabs. I felt like the law of averages would basically play out.
- pandler 12y agoAny chance you would release your code for this? I'd love to run an analysis on some lesser known rappers and play around with some of the filters. Awesome project btw. EDIT: The reason I ask is that I assume you don't have the time or desire to add every rapper every person asks you for.
- quux 12y agoI wonder where Weird Al Yankovic would come in on this ranking.
- coherentpony 12y agoWeird Al's songs are not articulate masterpieces, but cheap parodies of other rap songs and rappers. He's probably somewhere around the 5,000 mark with the other artists. A cursory google on the size of the average vocabulary [1] yields an interesting fact. I'm not sure how watertight it is. I realise it's probably unfair to compare the size of the average vocabulary to that of a series of songs. Songs being shorter for one. Still, it's interesting.
- Moto7451 12y agoNot sure if that's fair to Weird Al. The people he parodies wouldn't really agree either [1]. It's not like he's doing the cheap morning show tactic of swapping clean words for dirty words or bad puns but leaving the rest of the song intact. They maintain a consistent theme which is really tough. [1] http://www.weirdalforum.com/viewtopic.php?t=5673 http://www.weirdalforum.com/viewtopic.php?t=5673
- coherentpony 12y agoYeah, perhaps I was a little harsh. Don't get me wrong, I like Weird Al. I probably could have phrased my comment a little better. I should have said something along the lines of, "In my experience listening to Weird Al, it doesn't feel like he explores a lot of the English language." Your link is cool, thanks for sharing that.
- NAFV_P 12y agoAll the rappers listed seem to be American. Whack this through your Bowers and Wilkins: https://www.youtube.com/watch?v=p_SQEUZomug https://www.youtube.com/watch?v=p_SQEUZomug
- krick 12y agoReally interesting, but not as representative as it should be. It's not clear why some have larger vocabulary than others. It could be using words like "zeitgeist" (in case of Aesop Rock) or some clever wordplay (I don't know much about hip-hop, so I can't find example for some artist from the list right off the bat, but I remember Marilyn Manson using word "gloominati" for instance) or pretty meaningless made up words like "schizzle" (in case of Snoop Dogg) or usual derivatives like "fuckedy fuck". Moreover, in many transcripts for hip-hop people write down words as they are pronounced, which can be pretty much distorted for some artists (which of course ideally shouldn't count as a "new word", but that's complicated, yeah). While Aeson Rock and DMX are clearly extreme and not surprising at all, it's not that clear for some guys in the middle. So, first off, for every data project sources should be provided, or at least more specific definition, how text was processed, tokenized, analyzed. Second, several more "data slices" should be provided, for instance 100 most used words which are unique for that artist compared to other artist in the list.
- duney 12y agoThe example you used for clever wordplay, "gloominati," is actually considered a portmanteau word. It's the result of combining multiple words to create a new word. (I say this not to be a pedant, but because I learned the term recently and was amused that we actually have a word for it.)
- krick 12y agoYes, portmanteaus are exactly what I was meaning by "clever wordplay". English isn't my primary language so it's hard to remember the right word sometimes, sorry for that. :)
- danielsf 12y agoOP here. Do I really need to provide all of this to satisfy the reader's ability to grasp the basic premise of the site? this isn't a thesis or academic pursuit, just comparing some rappers for fun. I used plain NLTK token analysis on rap genius lyrics. in terms of several more data slices...I agree that there should be more cuts of the data, but you must understand the amount of time that it took me to put this together.
- blahbl4hblahtoo 12y agoAesop "really do" Rock is the fucking business.
- habosa 12y agoNot surprised to see Wu Tang at the top and Drake at the bottom. Started from the bottom ... still there.
- pandler 12y agoHaha I was thinking that as you move left on the scale the more likely you are to see rappers that people tend to mock.
- sarreph 12y agoWe might all be self-confessed hackers, but we'll never explicitly confess our adoration for the gloriousness of the genre that is gangster rap.
- antocv 12y agoWhat do you mean never explicitly confess? The fuck? Gangster rap is awesome as well as most of hip hop. But as the chart shows CunninLynguists rules all
- coherentpony 12y agoThis comment is nothing more than a cheap attempt to inject musical elitism. Please, if you have nothing critical to say that doesn't also inspire well-meaning debate, then take your business to Slashdot.
- msutherl 12y agoI would love to see this analysis without filters. Who is the rapper with the largest vocabulary? What does the distribution look like at the top? Surely Antipop Consortium or MF DOOM have larger vocabularies than Aesop for instance.
- Fishkins 12y agoMF DOOM is on the list. He's above average, but well below Aes or most of Wu-Tang. I've listened to a fair number of rappers, and I was pretty confident Aes would be at the top of this list. I agree it would be cool to see a list of all rappers, though. I was surprised not to see Del, and maybe there is some more obscure rapper I'm not thinking of with a broader vocabulary than Aes.
- S_A_P 12y agoI think the only problem I see is that some rap groups are listed as rappers. For instance beastie boys, de la soul and wu tang are listed. So there is some collective vocabulary being compared to single rappers. That said this is cool and pretty telling. From what I could see it is probably loosely couple to the intelligence of the rappers listed. I will echo the sentiments about DMX here. Looks like some shock jock rappers definitely are low on the list (too short).
- rmk2 12y ago> "Wu-Tang Clan at #6 is fucking impressive given that 10 members, with vastly different styles, are equally contributing lyrics. Add the fact that GZA, Ghostface, Raekwon, and Method Man's solo works are also in the top 20 – notably, GZA at #2. Perhaps their countless hours of studio time together (and RZA’s mentorship) exposed each rapper’s vocabulary to one another." At least in the case of the Wu-Tang Clan, this seems to be done on purpose and suggests that there is a strong correspondence between the individual members' repertoire on one hand and the group's vocabulary as a whole, with a presumed exchange in both direction (i.e. both as a deductive and an inductive process).
- randomdrake 12y agoFor those who aren't familiar with Aesop Rock, I'd invite you to give him a listen sometime. His earlier albums, in particular, have been very influential to me in many ways. Both in my artistic and professional careers. From comments on the conditions of the working man and the condition of feeling trapped in a "j-o-b"[1]: "Now we the American working population Hate the fact that eight hours a day Is wasted on chasing the dream of someone that isn't us And we may not hate our jobs But we hate jobs in general That don't have to do with fighting our own causes We the American working population Hate the nine-to-five day-in day-out When we'd rather be supporting ourselves By being paid to perfect the pastimes That we have harbored based solely on the fact That it makes us smile if it sounds dope" To storytelling masterpieces regarding living and dreaming[2]: "Look, I've never had a dream in my life Because a dream is what you wanna do, but still haven't pursued I knew what I wanted and did it till it was done So I've been the dream that I wanted to be since day one!" Aesop Rock takes language and linguistics to entirely different levels than one might expect from the single genre that is hip-hop. He even challenges himself and the listeners, playing fantastic word games, for instance re-using the letters L, S, and D in odd and rhythmical ways after a mention[3]: "Lazy summer days Like some decrepit landshark dumb luck squad dog lurks sicker deluded Last sturdy domino lean's secluded Don't let stupid delusions lesson super-duty labor students Dragnet lifer solutions Daddy loved sloppy dimensions like son-daughter links Such determinated lepers, successfully disheveled Little soliders developed like serpents despite life sentence ducking Lemmings Some don't like sobriety's dirty lenses Some do" And then there are just incredible gems that stick with you like[4]: "I don't flick neeedles like my sick friend I don't march like Beetle Bailey through a quick trend I don't frequent church's steeples on my weekend And I don't comment if you formulate a weak Zen" There's a lot to explore from Aesop Rock. Should you find this type of hip-hop interesting, a decent place to start is with the label you can find these songs on, Definitive Jux[5]. Incredible talent has been on and off that label over the years. So much good stuff. [1] - "9-5ers Anthem" - http://rapgenius.com/Aesop-rock-9-5ers-anthem-lyrics http://rapgenius.com/Aesop-rock-9-5ers-anthem-lyrics [2] - "No Regrets" - http://rapgenius.com/Aesop-rock-no-regrets-lyrics http://rapgenius.com/Aesop-rock-no-regrets-lyrics [3] - "The Greatest Pac-Man Victory in History" - http://rapgenius.com/Aesop-rock-the-greatest-pac-man-victory-in-history-lyrics http://rapgenius.com/Aesop-rock-the-greatest-pac-man-victory... [4] - "Save Yourself" - http://rapgenius.com/Aesop-rock-save-yourself-lyrics http://rapgenius.com/Aesop-rock-save-yourself-lyrics [5] - http://en.wikipedia.org/wiki/Definitive_Jux http://en.wikipedia.org/wiki/Definitive_Jux
- bretthopper 12y agoLooked for Canibus near the top and wasn't surprised to find him 4th. If anyone hasn't heard of him, highly suggest listening to his older stuff such as his first Can-I-Bus, 2000 BC and Mic Club. He raps about science and space all the time which is cool. Here's an example of his ridiculous lyrics: http://rapgenius.com/Canibus-poet-laureate-infinity-lyrics http://rapgenius.com/Canibus-poet-laureate-infinity-lyrics
- shawnz 12y agoAdditionally, many HN users have probably already heard Canibus rapping even if they don't know it, since he wrote the Office Space theme song. :)
- Oxxide 12y agoAlways loved that song. My personal favorite Canibus track is Master Thesis, though.
- StevenNunez 12y agoYey! I'm not the only Canibus fan! Seriously though Mic Club is awesome.
- coherentpony 12y agoMaybe this is just me, but it's a little unfair to compare to literary texts. Humour me for a moment. When an artist writes a song, he (or she) has constraints. Most rappers would like to rhyme the ends of their sentences. I know sometimes they don't (like poetry), but it's certainly pleasing to the ear to have that constraint. Artists endeavour to make their songs catchy, that's highly correlated with the gross sales of the product. When an artist writes a novel, this constraint is not weighted quite as highly. I know Shakespeare wrote poetry, too, and to call me out on this comparison is entirely fair. That said, there's also an argument to be made for eye rhymes. Shakespeare used these a lot. Eye rhymes are words that don't rhyme aurally, but do rhyme visually. It's the story that pleases the reader, not necessarily its aural 'catchiness'. I probably made that word up. But Shakespeare made words up too. The point is, you knew what I meant. At the end of the day these comparisons, while certainly interesting, should be taken with a pinch of salt. While I'm at it, this advice can easily be extrapolated to any dataset. Always understand there may be unknown correlations.
- danielsf 12y agoOP here: the shakespeare thing is really just a hook, food for thought rather than an academic/cultural judgement. I also had several suggestions to use shakespeare's sonnets rather than plays, which I should have done. and yes, this is all just pinch of salt barbership discussion :)
- sareon 12y agoThis reminds me of a PyCon talk from this year in analyzing rap lyrics with some basic NLP techniques http://pyvideo.org/video/2658/analyzing-rap-lyrics-with-python http://pyvideo.org/video/2658/analyzing-rap-lyrics-with-pyth... The author was trying to see if rappers are considered more hateful towards women by their usage of "bitch per song". The results are quite interesting.
- simonster 12y agoThe estimate of vocabulary size here is based on the number of unique words used. This seems like it is strongly biased: if two artists have the same size vocabulary, but one has released more albums and thus used more words, that artist will probably have used more unique words. To underscore this point, the number of unique words used by Aesop Rock is half of the estimated vocabulary size of the average college student, although to be fair that estimate is the number of words that an individual can recognize, not the number of words they use. (Edit: the bias is somewhat mitigated by the fact that the same number of words is used to estimate the vocabulary for each artist, but the bias is not dependent on sample size alone but also upon the size of the artist's underlying vocabulary; see my comments below.) The underlying problem is one of estimating the cardinality of a multinomial distribution given a fixed number of samples. In isolation this problem is ill-posed, since it is always possible that there is a word in a given lyricist's vocabulary that he uses with very low frequency and that is unlikely to appear in any sample, but with appropriate prior information it may be possible to obtain an accurate estimate. This is not my field, but a brief Google Scholar search shows that there are several papers on estimating vocabulary size, or equivalently, estimating the number of species based on sampling. There is a somewhat dated review (http://cvcl.mit.edu/SUNSeminar/BungeFitzpatrick_1993.pdf http://cvcl.mit.edu/SUNSeminar/BungeFitzpatrick_1993.pdf) that details some methods of estimation (in this case, I believe we are in the domain of "infinite population, multinomial sample" with unequal class sizes). The paper notes that there is no unbiased estimator available without assumptions on the distribution of word use frequencies, but some of the proposed estimators may be more accurate than the naive estimate used here.
- deleted 12y ago[deleted]
- vadskye 12y agoSample size is already controlled for. See the second paragraph, immediately before the graphic: "I used each artist’s first 35,000 lyrics. That way, prolific artists, such as Jay-Z, could be compared to newer artists, such as Drake."
- superuser2 12y agoThis is why it's "number of unique words used within the artist's first 35,000 lyrics." Sample size is held constant. (Except, maybe, those who haven't yet written 35k words?)
- zeppelinnn 12y agoThis is awesome. Reminds me of all the data viz they are doing on rapgenius. You forgot Atmosphere though (Slug)
- Mikeb85 12y agoNot particularly surprised at the list. Aesop Rock, the whole Wu-tang Clan, and guys like Nas, Wale, all near the top. DMX and Too Short at the bottom... Definitely comes out in their music...
- oinksoft 12y ago> Definitely comes out in their music... That's right, Too $hort's music is laser-focused!
- thrownaway2424 12y agoGotta wonder about the garbage-in factor of Rap Genius. From one randomly selected Aesop Rock cut: "Please I want to donate my brain to the monstrous Panasonic profit" I guess it could be. I always heard it as "monstrous Panasonic prophet." It would be in keeping with the previous lyric "Television, all hail grand pixelated god of fantasy."
- thrownaway2424 12y agoThere's some other howlers in the same song like "canope" instead of "canopy", "intervenes" instead of "intravenous".
- nightpool 12y agofixed. This song has no credited transcripter, so it was one of the originals added to "bootstrap" the site, and hence has a lot lower quality then the later songs added and cleaned up by users.
- tehwebguy 12y agoIn that instance my guess is that he probably used a homonym on purpose. Perhaps another interesting breakdown would account (and give extra points) for homonyms since it's sort of 2 words.
- nightpool 12y agoProfit makes more sense to me, because it ties into the directly proceeding line "Please take me, please calm me, please make me a zombie". Aes is clearly talking about advertising and (broadly) media here, and while the double meaning of "prophet" does contribute to the line, the surface meaning is that he's donating his brain (attention) to the Panasonic profit (from advertising). However, that said, that page (http://rapgenius.com/Aesop-rock-basic-cable-lyrics http://rapgenius.com/Aesop-rock-basic-cable-lyrics) does need some love. You should help out! Use the "suggestions" box on individual annotations to leave things for Editors (trusted members of the community, like myself) to integrate into the existing annotations.
- bladecatcher 12y agoI would like to see Dälek included in the study. I'd be surprised if they didn't show up on the far right on the scale.
- DigitalSea 12y agoMakes me very happy to see Aesop Rock in the number #1 spot. He isn't as underground as many people assume, still relatively unknown in the mainstream, but well known enough to sell records and sell-out shows. I wasn't a big fan of his 2012 release Skelethon, but the way he structures his lyrics and the meaning behind them means he never writes a bad lyric. Interestingly Eminem whom I would have thought would rank pretty highly for his clever method of word bending and enunciation is only in the middle of the scale. Still a whole lot better than some of his counterparts, but still surprising. Another interesting thing to note is Eminem being grouped in the same league as the likes of Jay-Z, Rakim and Lupe Fiasco. With only a couple of hundred unique words separating them from one another.
- xentronium 12y agoI always thought eminem was famous for his clever wordplay, not his vocabulary diversity. FWIW, as a non-native speaker I can gather most of his verses. Aesop Rock, on the other hand, is totally indecipherable for me without printed lyrics.
- tps12 12y agoSo funny comparing this to the same graph they did for pop lyricists.
- gfody 12y agoI'm pretty sure E-40 scored so high because of all the made up words. He's highly regarded for being innovative and influential but you know for every piece of slang that stuck there's like ten that didn't.
- koala_advert 12y agoI keep getting this error, in Firefox and Chrome: <Error> <Code>AccessDenied</Code> <Message>Access Denied</Message> <RequestId>3CB1F41D7DFDC794</RequestId> <HostId> wHCPzEYPDsmkMJX+YIgjU40YPrGYytHrk5B44dApi7663NkQQI0RKx9A/6EX7Iph </HostId> </Error>
- pinkskip 12y agoWoah so awesome!
- Grue3 12y agoWould be interesting to see how they compare to rock bands like Titus Andronicus, Fucked Up or Bad Religion.
- snarfy 12y agoI wasn't surprised to see Canibus and Outkast up there.
- ryan1234567890 12y agoTo put some perspective on this: ryan@3G08:~/Desktop/bleh$ pdftotext David-Foster-Wallace-Infinite-Jest-v2.0.pdf ryan@3G08:~/Desktop/bleh$ python dfw.py size of vocabulary: 30725 The man passed Shakespeare by 1,896 words with that book. code: import nltk from nltk.stem import * import string raw = open("/home/ryan/Desktop/bleh/David-Foster-Wallace-Infinite-Jest-v2.0.txt",'rU').read() exclude = set(string.punctuation) raw = ''.join(ch for ch in raw if ch not in exclude) raw = raw.lower() tokens=nltk.word_tokenize(raw) stemmer = PorterStemmer() stemmed_tokens = set() for token in tokens: stemmed_tokens.add(stemmer.stem(token)) print "size of vocabulary:", len(set(stemmed_tokens))
- jmt7les 12y agoI'd love to see Immortal Technique also.
- deleted 12y ago[deleted]