14 ms·

Rap Genius is Back on Google

242 points by tomlemon 13y ago

Soviet 13y ago
I wonder how long it would take for any other website without 15 mln backing and direct line to Google to return.
- ashray 13y ago
  As someone who knows folks who got wrongly blacklisted.. Well, you pretty much never get back. No matter what you do. You're lucky if you get a decent reply from customer support to begin with.. Your best bet is to change domains, fill in a redirect and try to build up backlinks again. It'll take a year or two and hopefully no competitor pops up in your space. Or, you can try and get Andreesen-Horowitz interested in your business.
  - adidash 13y ago
    I have helped sites with over 2 million backlinks and got them out of manual penalty in under 3 months in very competitive verticals like travel & finance. It entirely depends upon the scope of penalty. Unless you are completely deindexed from Google search results, you can always recover from a penalty. Feel free to email me if you have any specific need.
kenshiro_o 13y ago
Looks like this involved quite a bit of work on Rapgenius' part but I am glad they cleaned up their mess and apologized to Google. Moreover, this article feels pretty neutral/humble and techy, a far cry from the haughty attitude their founders usually display in interviews.
LukeB_UK 13y ago
They have the word 'lyrics' at the end of the slug for that post and a number of others which don't have any relevance to lyrics. Surely that's still bad SEO practice?
- jarnix 13y ago
  Like lyrics of Franz Kafka, did not notice before reading your post. http://poetry.rapgenius.com/Franz-kafka-le-nouvel-avocat-lyrics http://poetry.rapgenius.com/Franz-kafka-le-nouvel-avocat-lyr...
- MattBearman 13y ago
  Well spotted, it looks like all of their news article URLs end in '-lyrics', even those when it's not even slightly relevant eg: http://news.rapgenius.com/Snapchat-find-friends-abuse-lyrics http://news.rapgenius.com/Snapchat-find-friends-abuse-lyrics More shady SEO practices from Rap Genius, disappointed but not surprised.
- dmn757 13y ago
  Very observant... I wonder what Google would say about this. I would almost find this more egregious than their original offense due to the posts having nothing to do with lyrics.
- knapp 13y ago
  All of the links in the poetry section I can see have "lyrics" appended to the URL. And some of those are not even poems: http://poetry.rapgenius.com/Herman-melville-bartleby-the-scrivener-a-story-of-wall-street-lyrics http://poetry.rapgenius.com/Herman-melville-bartleby-the-scr... This one is particularly weird / egregious: http://poetry.rapgenius.com/Engineering-genius-understanding-gears-with-lego-lyrics http://poetry.rapgenius.com/Engineering-genius-understanding...
- elwell 13y ago
  Not really bad SEO practice, just inflexible software.
- tomlemon 13y ago
  Tom Lehman here (Rap Genius co-founder) The fact that all our text and song page URLs end in "-lyrics" is a relic of a time when we hosted almost exclusively lyrics. Making the changes required to host all kinds of annotated texts has been a big project and making the URLs make sense is something we haven't gotten around to. The reason it's not a totally trivial change is that both users and texts share the same top-level namespaces, and so right now we use the "-lyrics" suffix to differentiate song and text pages from user profile pages. We want to change the suffix to "-annotated" for non-music pages and will hopefully get around to in the next few weeks. To our knowledge the existing situation doesn't help with SEO (no one is searching "bartleby the scrivener lyrics"), and it's confusing, so we 100% agree that the current situation is bad. For more info on the "Worse is Better" philosophy that caused the current jank situation to persist for so long, see this talk: https://www.youtube.com/watch?v=X45YY97FmL4 https://www.youtube.com/watch?v=X45YY97FmL4
deleted 13y ago
[deleted]
kanamekun 13y ago
Pretty interesting decision to include a lengthy "technical digression" in their SEO apology: << Technical Digression: How to scrape 178k URLs in Ruby in less than 15 minutes - Ok, so you have 178k URLs in a postgres database table. You need to scrape and analyze all of them and write the analysis back to the database. Then, once everything’s done, generate a CSV from the scraped data. >>
- r0h1n 13y ago
  As I see it, the "technical digression" serves two purposes: - first, they use it with Google to show how sincere they were about "cleaning up" - second, and perhaps more important, it gives Google a public excuse to justify why they restored RG so quickly in rankings. As in, "Oh they did a very thorough job of complying by our standards that we had no option but to restore their status". Unfortunately in this case Google's response speed comes across as a bad thing. Because it applies to only a very select group of sites that are either very rich (aka large advertisers) or very well connected. [Edit: clarified imagined comment]
- rebelde 13y ago
  My favorite is that they seemed to have spent a few hours (more than 3!) improving their code so that it would run in 15 minutes instead of three hours.
jdmitch 13y ago
Interesting closing paragraph: >Though Google is an extremely important part of helping people discover and navigate Rap Genius, we hope that this ordeal will make fans see that Rap Genius is more than a Google-access-only website. The only way to fully appreciate and benefit from Rap Genius is to sign up for an account and use Rap Genius – not as a substitute for Wikipedia or lyrics sites, but as a social network where curious and intelligent people gather to socialize and engage in close reading of text. ie, you should really just become a member rather than relying on google... Also, where was bing in all of this?!? ;)
- eruditely 13y ago
  Other search sites dropped the ball on this one
bushido 13y ago
This is great for Rap Genius. But this truly saddens me as well. I cannot for the life of me understand how they were able to fix ten-of-thousands of backlinks and other shady practices and get back in the good graces of Google so darn quickly. Frankly I do not know a single other legitimate(+with good traffic) site who was able to get back in the good graces of Google after algorithm updates without spending many many months researching any possible reasons to be penalized and fixing it themselves after repeatedly asking google for help to no avail. Could it be that Rap Genius was special because it has Andreessen Horowitz? Sure seems that way. I am not going to stop using Google, but I am inclined to use them lesser now. I would like to hear why, how rap genius was fixed this and what this means for other people who get penalized from Matt Cutts though!!
- sillysaurus2 13y ago
  They're back on Google, but keep in mind that they're seemingly not in a great position. Googling "justin beiber lyrics" doesn't show Rapgenius. Neither does "lady gaga lyrics" or "tupac lyrics". In all three cases, azlyrics.com has the top spot.
  - deleted 13y ago
    [deleted]
  - bushido 13y ago
    Edit: the reply is to the now reacted comment: > They're a startup, and this was a problem which was either going to kill them or be surmounted quickly. Naturally, they surmounted it quickly. Google laid out a clear plan of action for them. Is it hard to believe they finished it quickly? Yeah, most of the other sites I refer to were also start-ups by almost any definition. They did not quite finish their action plan. They got away on reasonable doubt. This is the first time I am seeing this defense considered by Google. I willing to bet any other non-vc funded bootstrapped start-up would never be afforded this consideration by Google. This rant is not a judgement on Rap Genius, I am actually glad they maneuvered around this. I am judging Google. They have killed other start-ups for reasons such as other sites farming and duplication their content more vehemently. Reasonable doubt has never been considered for anybody else and most likely will never be hence forth.
jarnix 13y ago
This post is complete bullshit. Of course they have relations. Maybe they cleaned their backlinks, but still I cannot believe how they got back so fast in SERP. Anyway, Franz Kafka would die again if he saw his text on this site (http://poetry.rapgenius.com/Franz-kafka-a-dream-lyrics http://poetry.rapgenius.com/Franz-kafka-a-dream-lyrics), so let's just forget about this site and don't use it.
- tinco 13y ago
  Don't be so spiteful. It might very well be that they have relations, don't make it sound like that's a bad thing. Besides that they also seem to just have cleaned up their act very thoroughly. And about Franz Kafka, I think he'd turn in his grave if he heard about how petty people like yourself criticize a website that tries to make literary criticism of his work digestible and enjoyable for the masses.
  - ronaldx 13y ago
    > It might very well be that they have relations, don't make it sound like that's a bad thing. It is a bad thing here: it shows that Google is apparently willing to act preferentially to favoured businesses, undermining the idea that search results are provided with neutrality.
    - tinco 13y ago
      No it doesn't. It means that Google employees can do stuff manually, when asked by a friend or colleague. It says nothing about Google as a business favouring Rap Genius as a business.
      - ashray 13y ago
        The fact that you can request a google employee in a position to help you out to do so displays enormous leverage. Imagine a business in Germany, Vietnam, or even.. Texas having to do the same? It's pretty much impossible. This is what happens to everyone else out there. Which is what the parent is referring to when he says that google results aren't as neutral as they claim to be. They clearly favored someone with leverage in this instance. But this is the only instance that we know of. So far.
uladzislau 13y ago
Probably the story is missing the help of a highly skilled SEO consultant/company but it's impressive anyway. To get it done so quickly they had to work 24/7. Kudos!
benjaminwootton 13y ago
I've followed SEO on and off for years and I wouldn't have thought that their strategy of allowing people to generate and embed links back to the site was shady at all. I even recommended something similar a few years ago in good faith. They're useful, relevant, and it just seems like good marketing. These things will spread organically and drive traffic even if you remove the SEO benefits. Could you say that Scribd or Slideshare are doing something similar by allowing people to embed documents in their websites along with a no follow link back to their site?
- thirsteh 13y ago
  "nofollow" is the key.
- JohnTHaller 13y ago
  Allowing people to create links to your website isn't shady. Paying people to link back to your website on high-value keywords like 'justin beiber lyrics' from blog posts on other sites to artificially inflate your pagerank is 100% blackhat SEO. The payment in this case was a Twitter post about the link provider's website.
h1fra 13y ago
The tech part is obvsiouly here to calm down people from Hacker News and make them say "Oh this part was good, after all they are not so bad". It's just a shitty scraper but they talk about it like it was a revolutionnary script... There is no way they could get back so fast and so easily on google.
ghostdiver 13y ago
Disavow tool actions take months to bring any effect(if any).
- hippich 13y ago
  after a year I still see "unnatural" inbound links in webmaster tool. So I would say it have no effect.
OoTheNigerian 13y ago
I do not understand why people are offended that they came back quickly AFTER doing a bit of penance. Should their investors not have made calls? However, it should be noted that they did their part. One of my favorite quotes is this: "To save a drowning man, he must first give you his hand" If the Rap Genius guys were incompetent and did not do their part in removing the links so quickly (see main story for technical details) there would be little or nothing their investors would have been able to do about that. BTW, it also helped that people really found Rap Genius useful. I'm sure many searched had rapgenius appended to it. So the lesson is, make your app so good that when Google delists you, it will make them look bad. Thankfully?, there is also a precedence for other offenders to use. if you can detail you have atoned for your sins and Google insists your you must do a certain time, you know where to turn to. BTW, this is Hackernews, and I look forward to the discussion of the (de)merits of how they scrapped and analyzed 177,000 links and not espousing of anger that they survived the punishment. I'm sure Mahbod (my favorite) and co have learned that you do not go 'daaawging' when you are in a hole. I wish them success.
- sbreaking6 13y ago
  Ehh, the story is the double standard google is showcasing. The earlier story about Expedia and this outcome tell it well. This is rapgenius. If it had to cross the line why on something so materially insignificant.
- holyjaw 13y ago
  Just to hijack on one of your points: "it also helped that people really found Rap Genius useful." This is painfully true, and I know it has been discussed in prior discussions. Rap Genius is without a doubt THE BEST lyrics site available, hands down. Even if the other sites weren't ad-ridden scammy buggy shitty websites, if they got their acts together and actually TRIED, Rap Genius would still be top dog. I don't care for the attitudes or personalities of the people in charge, and I am absolutely disgusted that they felt the need to scam their Google rank, but I am willing to see that some infractions can be rectified through penance, and am happy to know that they will not be forced to shut down because of this fiasco.
grimlck 13y ago
This really shows that even in Silicon Valley, your success is driven as much by the connections you have and the people you know, rather than pure talent and merit.
- theodoracharl 13y ago
  What it really shows is that Y-Combinator's original ambition to support hackers in hacking capitalism is over. Successful startups, like mainstream corporations, are determined by 1. Public relations stunts, getting press, marketing bullshit 2. Connections, Privilege, Cronyism, Nepotism, Harvard, Yale, Stanford Paul Graham sold out and is now just like every other capitalist. Exploiting privilege to get richer. ycombinator's biggest successes are marketing successes--their success is defined by PR and Cronyism. This isn't Steve Wozniak--the tech genius--creating the Apple II. This is fat headed fratboys using PR stunts and cronies to rake in money by squeezing out the competition. The rich will get richer. The children of the rich will get richer. PR darlings and frat douchebags will get richer. Hackers will be ignored by YCombinator--Hackers aren't as profitable.
tszming 13y ago
People should worry why Google has so much power nowadays, rather than why they unban Rap Genius so fast.
- kenjackson 13y ago
  This really is the thing that struck me. All this talk about companies being destroyed and jobs lost and hardly a mention of worry about the power Google yields. We need more competition in this space. Sites need a search engine with real market share to compete against Google.
  - diydsp 13y ago
    yup. just adding duck duck go to this laptop now. thanks for the reminder. https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ssl/?src=search https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ss...
  - diydsp 13y ago
    yup. just adding duck duck go to this laptop now. thanks for the reminder. https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ssl/?src=search https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ss...
hippich 13y ago
I have to note, you have to be big to make comeback like this. Google provided disavow tool, but from my experience it simply not working. So it is manual action for sure.
virtualwhys 13y ago
Complete and utter bullshit. That they blatantly employed blackhat SEO techniques and are back on Google within a matter of days is sickening. Think of all the sites that are penalized in various ways for similarly shady SEO practices and have to spend _months_ to regain their previous position. Hopefully Bing et al can capture some marketshare and bring some degree of neutrality to the search engine landscape (i.e. not have every website owner on the planet beholden to the beast).
danso 13y ago
The "technical digression" with Github code on how to do a mass scrape is cool, though not surprising (er...how else would you accomplish such a big data task...though I'm glad I'm not the only one who uses Typhoeus, a great but underused gem). But what's surprising to me is that a company that knows how to automate things and work with data...why the f--- were they manually soliciting bloggers to post links? That seems seriously inefficient and comically backwards...as if Sergey Brin spent his early Google days modifying SERPs by editing and uploading Excel spreadsheets to the server. It seems seriously inefficient to me because at this point, Rap Genius has a lot of traffic and a lot of linkage already. So did the few backlinks, per day, that RapGenius got, through lame blogmasters who would participate in such a thing...really have an effect on their SEO rank, i.e. is it so easy to still game Google? Or, was RG just doing it because they thought it would have a worthwhile effect (keep in mind that this is one of their co-founders who was individually emailing people), i.e. RG was being naively optimistic?
jstalin 13y ago
Is it a sign of getting older when black background web sites get harder and harder to look at for longer periods of time?
NicoJuicy 13y ago
Actually, it's understandable that Google set this on High Priority... #1: it's a high traffic website, that is a startup and the whole deal got a lot of attention of us (HN Readers) + Matt even saw this here. So it was good to respond fast (no choice) #2: RapGenius is dependant on Google (although they try to let you see it as a social network, but it's not), they would have gone bankrupt ( i suppose) without Google. #3: Without a quick response and/or measure to fix their shit on their end, investors wouldn't invest any more money in RapGenius and Google would probably hurt a lot of future investments (how is this startup dependent on Google, ...) BMW once got a penalty from Google, they didn't got easily off. But their business wasn't dependent on Google. Either way, there are both pro and cons to each action. But overall, i think they handled if fairly well.
undoware 13y ago
I'm waiting for their next SEO single. "Oops, I did it again!"
wengzilla 13y ago
Infinite scroll (or some type of weird JS) won't let me get to the bottom of the page... Ain't got ish to do with this but I just thought that I should mention.
programminggeek 13y ago
One thing I know is that RapGenius is good about making a big stink, being in the news, and then having things work out in their favor, even if they were the ones doing a dumb thing in the first place. They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku? They did some incredibly dumb SEO things because they are greedy and are willing to cheat to win. That is on RapGenius, but once they got banned it is somehow Google's fault and they should be reincluded? I don't get it. RapGenius is great at one thing - creating controversy and getting press for it. I guess that works for them, but it seems like a very selfish way to grow a business. Instead of focusing on building value themselves, they are willing to tear down their business partners in public to get what they want. I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.
- jmduke 13y ago
  They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku? Are you arguing that what Heroku was doing wasn't wrong? I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate. Comments on every single RapGenius post on HN have been filled with vitriol toward the founders, their attitude, and the culture they foster.
sbreaking6 13y ago
Clear as day there is a double standard. That's what should be discussed.
sbreaking6 13y ago
Well atleast now AH partners can get back to posting rap lyrics on twitter. We wouldn't want anything to stop that bit of entertainment.
friscobob 13y ago
God this pisses me off to no end. So my site is targeted by negative SEO. (Yeah, I know, I know, negative SEO doesn't exist. Fuck off.) So who do I call? I've been penalized for months and there is nothing I can do about it. Nobody can help. But these fuckers are able to make a call and recover from a SELF INFLICTED wound in less than a week. I am being targeted by crap that is COMPLETELY OUT OF MY CONTROL and it's tough luck little buddy. The double standard here is fucking sickening.
- argonaut 13y ago
  Disavow?
  - friscobob 13y ago
    I've tried the disavow. Seen no indication that it had any effect. There were 31 thousand new backlinks added on December 31st (Holy shit, I just noticed that. WTF!). Another 1300 added on the 1st of January. It's normally 100 or 200 a day. Not sure how I'm supposed to manage the disavow process when I get that many new spammy inbound links a day.
    - aerolite 13y ago
      maserati problem
      - friscobob 13y ago
        Why would you say that? This is impacting my Google search results TODAY. If I turn off Adwords, my leads drop to virtually nothing...
- MitziMoto 13y ago
  This scares the crap out of me. There are some really shady people in my industry and I'm sure they wouldn't think twice about launching a negative SEO campaign against me. I wish Google and Matt Cutts would address this for real.
rasengan 13y ago
I am sure they went through many sleepless nights during the holidays. Hella sucks. I think Google is doing the right thing and I am sure not just related to seo that rap genius learned a lot about themselves and will become stronger... as well will the overall tech community.
mgulaid 13y ago
what does not make sense to me is that after removing all the back-linked urls, should not that drop Rap Genius's overall ranking even lower than before, since the artificial boosting has been removed. Why they are back on the top. googling "lyrics", RP is number #6 in the first page. I think RP has received special treatment here. Google search team needs to clarify this.
dhorowirtz 13y ago
http://m.entrepreneur.com/article/230631 http://m.entrepreneur.com/article/230631 But now, Rap Genius co-founder Ilan Zechory tells us the site has returned to Google's search results. "It takes a couple days for Google to re-index everything, so search results are a little wonky right now, but we are officially reinstated," Zechory said in an email. Somewhat ironically, Zechory served as a project manager at Google for two years prior to founding Rap Genius in 2009. >> Wow that is troubling. So site that gets banned gets unbanned quickly and founder just happens to be ex google project manager
JohnTHaller 13y ago
Honestly, this is just saddening. Google needs to have a consistent policy towards these blackhat SEO offenders and enforce it for everyone. RapGenius bought backlinks to artificially inflate their pagerank. They knew they were cheating but they did it anyway. They got caught. And they were hit with a manual action on Google. Sounds good so far. But that's where it stops. RapGenius, using the connections they have from their multi-million dollars in VC funding, got back on Google and was able to avoid paying the price for their cheating. Compare this to regular websites. Lots of smaller sites will end up paying a "whitehat SEO" firm to work on a "link strategy". These firms will claim up down and sideways that what they do is legal, ethical and follows Google's rules. But, what they actually do is either (1) create networks of fake sites to provide backlinks on certain terms to artificially boost the site, (2) place spam comments using bots on legitimate sites to do the same [not that this will thankfully no longer work well due to the latest Google algorithm update], or (3) pay legit sites to place backlinks to artificially transfer pagerank the same way that RapGenius did. Now, these other sites, when they get caught, they get a manual action or a smackdown. The difference? They have to actually pay the penalty. Arguing that they didn't know usually doesn't work. The penalty is LONG. They can't call on their VC firm to make calls at Google to give them a get out of jail free card. I'd like to call on Google to create a public policy on how they handle these manual actions with some clearly defined penalties (example: 3 month manual action of 6 PR drop, etc) and to consistently enforce them across the board. That way a mom and pop site that pays a 'whitehat SEO firm' and gets caught doesn't have a worse time than a site like RapGenius that purposely engages in blackhat SEO who can use their VC connections to get out of having to pay a penalty in under 2 weeks. It's also sad that Google gave this get out of jail free card to a site whose entire business model is based around other people's copyrighted works which RapGenius doesn't have a license for, doesn't pay for, and publishes illegally.
- michaeldunworth 13y ago
  This.
- sheetjs 13y ago
  Google is a business, and there is a very good chance that they financially benefit from Rap Genius being restored. Other companies that suffer don't have that financial link, and maybe the best course is for them to raise VC funds (and posture themselves so that Google stands to lose if they penalize) I agree with you in theory, but in practice it sounds like you expect google to act as if they care about things like fairness. They are a company. The bottom line is what matters. If you think it's unfair, then try to hit the bottom line by using other services and convincing your friends and family to do the same
McKittrick 13y ago
Its interesting that despite their founders outspokeness, the influx of heavy weight VC, tech press expose's on their offices, etc., RapGenius is really no more or no less than a standard, multi-multi-page SEO site, no different than the countless others just like it. No brand, no user affinity, no direct traffic to speak of. Just a bunch of drift net pages crawled by Google waiting to pick off unsuspecting search traffic. My favorite line of this post was the last: "Much love. iOS app dropping next week!" - unless you are searching that term in google, Im not sure who that is addressed to.
rburhum 13y ago
Snapchat could learn a thing or two of how to do PR from this post.
myzerox 13y ago
This is a relief. RG certainly gives the best search results for a lot of queries. This whole story made me fear that Google was choosing power over user experience. And that should never be the case, no matter how offending the founders of RG might appear sometimes. Don't be evil!
- dclara 13y ago
  Is there a smell of corruption? Because it is in favor of power, connections instead of user voting, just like in favor of dictation over democracy. Isn't it?
  - myzerox 13y ago
    Fair voters = valid pages linking to RapGenius Bought voters = manipulated SEO spam blog posts Fair voters >> bought voters Not reacting to the SEO manipulation would have meant accepting a fraction of bought voters. Banning RG completely would have meant to ignore the even bigger fraction of valid voters. Google's actual decision seems to be the most democratic: Forcing RG to decrease the number of bought voters while not dictating over the voice of the valid voters.
    - dclara 13y ago
      You have an interesting point. But it's hard to say: Fair voters always greater greater than bought voters. When you buy backlinks from the SEO consulting service, they gave you hundreds or thousands everyday, just like some people complained in the comments. "Google's actual decision seems to be the most democratic: Forcing RG to decrease the number of bought voters while not dictating over the voice of the valid voters." The problem is: how can they tell which is which? I doubt this type of criteria is reliable. I collected some information from the user comments along with other research and wrote a blog post here: http://bingobo.info/blog/bingobo/what-are-the-current-seo-problems.jsp http://bingobo.info/blog/bingobo/what-are-the-current-seo-pr... Maybe you will also take a look at the possible solutions here: http://bingobo.info/blog/bingobo/how-bingobo-can-help-business-solve-their-seo-problems.jsp http://bingobo.info/blog/bingobo/how-bingobo-can-help-busine...
buro9 13y ago
Were the 178k URLs that they scraped on 178k unique domains or a very few domains? The tool they wrote looks like it would have been a DoS attack if the latter were the case. There appears to be no thought or consideration for the sites they scraped that had linked to them, no attempt to read robots.txt and see whether they should fetch the pages. What they did was wrong in the first place, but they could try and do things better in their attempt to repair things. Instead they continue to act in a way that externalises any cost to them and continues to treat their service as somehow privileged enough to deserve such breaks. How many of the 53k failures to pull and scrape pages were to do with the fact that the servers in question were under a DoS attack by RapGenius?
tzm 13y ago
"tis better to apologize later, than ask for permission first."
notastartup 13y ago
Certainly a double standard exists at Google. I originally thought finally Google is taking a firm stance, but this reversal only does harm to Google's image imho. Take home message is, blackhat SEO is forgiven if you are funded by connected people. Don't have funding or connections but make the slightest infraction in SEO? Feel the google girth.
bjoernbu 13y ago
So which hit should google show for a query like "the fox lyrics"? Of course, it should be a page that contains the lyrics, but which one? The most popular one? The one shown by google will be the most popular one. From an IR perspective, search for lyrics is somewhat strange. I don't think it's easy to say if some hit is more relevant than the other. Sure, more popular sites may offer better UX (or simply less annoying ads), but at the end of the day it's a search for facts. I wouldn't be surprised if google will serve those lyrics themselves some day. Freebase (and hence the google knowledge graph) already contains tons and tons of information on pratically any popular song (organized in a confusion way via recordings, canonical versions and whatnot) - artist, genre, album, release dates, length of a recording, etc. I don't think adding the lyrics as well is totally out of question. After all multible KBs are user curated and users provides lyrics. Just like they serve "fact-hits" from their knowledge graph for queries like "movies by spielberg", google should then be able to serve the lyrics directly. While this is bad for lyrics sites, imho it is convenient for users and follows the same prinicple like "movies by spielberg" or "wife tom cruise"
naner 13y ago
Rap Genius appears to have offset their drop in search results traffic with all the publicity they've been getting from being penalized by Google.
rodly 13y ago
If Google were to have a year long ban on RG stuff to leave a precedent, Would people start to view Google as a sub-par search engine because Bing returns "high quality" results like RG still? It's also rather weird how a lot of people seem to view Google as this public service for the people by the people etc, it's not. They can do whatever the hell they want and if you don't like it, simply stop using their service to send a message.
JeremyMorgan 13y ago
Money wins every time.
deleted 13y ago
[deleted]
deleted 13y ago
[deleted]
mathewsimonton 13y ago
The SEO strats / growth hacks weren't entirely misguided. Rap Genius just didn't perfect their activity and didn't let it exist entirely behind closed doors. If Rap Genius wanted to risk it and engage in blackhat-like activity, it should have at least kept things more quiet and less spammy. Rap Genius should have kept the finer details, at least, to a phone call with the blogger. Google can't see a local business handing their customers/client a $25 gift certificate in-person for a random positive Google+ review. Along the same thread, Google would also not be able to see Rap Genius flying out a community outreach rep to meet with a prolific music blogger at a bar to go over an exchange for a positive mention and authoritative link. Just saying.
davemel37 13y ago
I think it will take time for them to regain their rankings for non-branded keywords. I just searched for justin Bieber lyrics and they are sitting in middle of page 3 on google. They are ranking for branded searches, but not money keywords. Although long term, I see this being a net positive for them due to the thousands of backlinks they got from all the press around this story. I see it taking them several months to regain rankings...especially if the links they disavowed were artificially propping them up.
jfoster 13y ago
While we're on this topic, are GroupOn's API Branding Requirements in violation of Google's webmaster policies? http://www.groupon.com/pages/api-branding-requirements http://www.groupon.com/pages/api-branding-requirements They make it a requirement of using the API that "nofollow" is not used: "Don’t ‘nofollow’ your links to Groupon."
joelotz 13y ago
I give them a lot of credit for being transparent with their learning mistake, taking ownership, and describing what they did to fix it.

242 points by tomlemon 13y ago

Soviet 13y ago

I wonder how long it would take for any other website without 15 mln backing and direct line to Google to return.

ashray 13y ago
As someone who knows folks who got wrongly blacklisted.. Well, you pretty much never get back. No matter what you do. You're lucky if you get a decent reply from customer support to begin with.. Your best bet is to change domains, fill in a redirect and try to build up backlinks again. It'll take a year or two and hopefully no competitor pops up in your space. Or, you can try and get Andreesen-Horowitz interested in your business.
- adidash 13y ago
  I have helped sites with over 2 million backlinks and got them out of manual penalty in under 3 months in very competitive verticals like travel & finance. It entirely depends upon the scope of penalty. Unless you are completely deindexed from Google search results, you can always recover from a penalty. Feel free to email me if you have any specific need.

kenshiro_o 13y ago

Looks like this involved quite a bit of work on Rapgenius' part but I am glad they cleaned up their mess and apologized to Google. Moreover, this article feels pretty neutral/humble and techy, a far cry from the haughty attitude their founders usually display in interviews.

LukeB_UK 13y ago

They have the word 'lyrics' at the end of the slug for that post and a number of others which don't have any relevance to lyrics. Surely that's still bad SEO practice?

jarnix 13y ago
Like lyrics of Franz Kafka, did not notice before reading your post. http://poetry.rapgenius.com/Franz-kafka-le-nouvel-avocat-lyrics http://poetry.rapgenius.com/Franz-kafka-le-nouvel-avocat-lyr...
MattBearman 13y ago
Well spotted, it looks like all of their news article URLs end in '-lyrics', even those when it's not even slightly relevant eg: http://news.rapgenius.com/Snapchat-find-friends-abuse-lyrics http://news.rapgenius.com/Snapchat-find-friends-abuse-lyrics More shady SEO practices from Rap Genius, disappointed but not surprised.
dmn757 13y ago
Very observant... I wonder what Google would say about this. I would almost find this more egregious than their original offense due to the posts having nothing to do with lyrics.
knapp 13y ago
All of the links in the poetry section I can see have "lyrics" appended to the URL. And some of those are not even poems: http://poetry.rapgenius.com/Herman-melville-bartleby-the-scrivener-a-story-of-wall-street-lyrics http://poetry.rapgenius.com/Herman-melville-bartleby-the-scr... This one is particularly weird / egregious: http://poetry.rapgenius.com/Engineering-genius-understanding-gears-with-lego-lyrics http://poetry.rapgenius.com/Engineering-genius-understanding...
elwell 13y ago
Not really bad SEO practice, just inflexible software.
tomlemon 13y ago
Tom Lehman here (Rap Genius co-founder) The fact that all our text and song page URLs end in "-lyrics" is a relic of a time when we hosted almost exclusively lyrics. Making the changes required to host all kinds of annotated texts has been a big project and making the URLs make sense is something we haven't gotten around to. The reason it's not a totally trivial change is that both users and texts share the same top-level namespaces, and so right now we use the "-lyrics" suffix to differentiate song and text pages from user profile pages. We want to change the suffix to "-annotated" for non-music pages and will hopefully get around to in the next few weeks. To our knowledge the existing situation doesn't help with SEO (no one is searching "bartleby the scrivener lyrics"), and it's confusing, so we 100% agree that the current situation is bad. For more info on the "Worse is Better" philosophy that caused the current jank situation to persist for so long, see this talk: https://www.youtube.com/watch?v=X45YY97FmL4 https://www.youtube.com/watch?v=X45YY97FmL4

deleted 13y ago

[deleted]

kanamekun 13y ago

Pretty interesting decision to include a lengthy "technical digression" in their SEO apology: << Technical Digression: How to scrape 178k URLs in Ruby in less than 15 minutes - Ok, so you have 178k URLs in a postgres database table. You need to scrape and analyze all of them and write the analysis back to the database. Then, once everything’s done, generate a CSV from the scraped data. >>

r0h1n 13y ago
As I see it, the "technical digression" serves two purposes: - first, they use it with Google to show how sincere they were about "cleaning up" - second, and perhaps more important, it gives Google a public excuse to justify why they restored RG so quickly in rankings. As in, "Oh they did a very thorough job of complying by our standards that we had no option but to restore their status". Unfortunately in this case Google's response speed comes across as a bad thing. Because it applies to only a very select group of sites that are either very rich (aka large advertisers) or very well connected. [Edit: clarified imagined comment]
rebelde 13y ago
My favorite is that they seemed to have spent a few hours (more than 3!) improving their code so that it would run in 15 minutes instead of three hours.

jdmitch 13y ago

Interesting closing paragraph: >Though Google is an extremely important part of helping people discover and navigate Rap Genius, we hope that this ordeal will make fans see that Rap Genius is more than a Google-access-only website. The only way to fully appreciate and benefit from Rap Genius is to sign up for an account and use Rap Genius – not as a substitute for Wikipedia or lyrics sites, but as a social network where curious and intelligent people gather to socialize and engage in close reading of text. ie, you should really just become a member rather than relying on google... Also, where was bing in all of this?!? ;)

eruditely 13y ago
Other search sites dropped the ball on this one

bushido 13y ago

This is great for Rap Genius. But this truly saddens me as well. I cannot for the life of me understand how they were able to fix ten-of-thousands of backlinks and other shady practices and get back in the good graces of Google so darn quickly. Frankly I do not know a single other legitimate(+with good traffic) site who was able to get back in the good graces of Google after algorithm updates without spending many many months researching any possible reasons to be penalized and fixing it themselves after repeatedly asking google for help to no avail. Could it be that Rap Genius was special because it has Andreessen Horowitz? Sure seems that way. I am not going to stop using Google, but I am inclined to use them lesser now. I would like to hear why, how rap genius was fixed this and what this means for other people who get penalized from Matt Cutts though!!

sillysaurus2 13y ago
They're back on Google, but keep in mind that they're seemingly not in a great position. Googling "justin beiber lyrics" doesn't show Rapgenius. Neither does "lady gaga lyrics" or "tupac lyrics". In all three cases, azlyrics.com has the top spot.
- deleted 13y ago
  [deleted]
- bushido 13y ago
  Edit: the reply is to the now reacted comment: > They're a startup, and this was a problem which was either going to kill them or be surmounted quickly. Naturally, they surmounted it quickly. Google laid out a clear plan of action for them. Is it hard to believe they finished it quickly? Yeah, most of the other sites I refer to were also start-ups by almost any definition. They did not quite finish their action plan. They got away on reasonable doubt. This is the first time I am seeing this defense considered by Google. I willing to bet any other non-vc funded bootstrapped start-up would never be afforded this consideration by Google. This rant is not a judgement on Rap Genius, I am actually glad they maneuvered around this. I am judging Google. They have killed other start-ups for reasons such as other sites farming and duplication their content more vehemently. Reasonable doubt has never been considered for anybody else and most likely will never be hence forth.

jarnix 13y ago

This post is complete bullshit. Of course they have relations. Maybe they cleaned their backlinks, but still I cannot believe how they got back so fast in SERP. Anyway, Franz Kafka would die again if he saw his text on this site (http://poetry.rapgenius.com/Franz-kafka-a-dream-lyrics http://poetry.rapgenius.com/Franz-kafka-a-dream-lyrics), so let's just forget about this site and don't use it.

tinco 13y ago
Don't be so spiteful. It might very well be that they have relations, don't make it sound like that's a bad thing. Besides that they also seem to just have cleaned up their act very thoroughly. And about Franz Kafka, I think he'd turn in his grave if he heard about how petty people like yourself criticize a website that tries to make literary criticism of his work digestible and enjoyable for the masses.
- ronaldx 13y ago
  > It might very well be that they have relations, don't make it sound like that's a bad thing. It is a bad thing here: it shows that Google is apparently willing to act preferentially to favoured businesses, undermining the idea that search results are provided with neutrality.
  - tinco 13y ago
    No it doesn't. It means that Google employees can do stuff manually, when asked by a friend or colleague. It says nothing about Google as a business favouring Rap Genius as a business.
    - ashray 13y ago
      The fact that you can request a google employee in a position to help you out to do so displays enormous leverage. Imagine a business in Germany, Vietnam, or even.. Texas having to do the same? It's pretty much impossible. This is what happens to everyone else out there. Which is what the parent is referring to when he says that google results aren't as neutral as they claim to be. They clearly favored someone with leverage in this instance. But this is the only instance that we know of. So far.

uladzislau 13y ago

Probably the story is missing the help of a highly skilled SEO consultant/company but it's impressive anyway. To get it done so quickly they had to work 24/7. Kudos!

benjaminwootton 13y ago

I've followed SEO on and off for years and I wouldn't have thought that their strategy of allowing people to generate and embed links back to the site was shady at all. I even recommended something similar a few years ago in good faith. They're useful, relevant, and it just seems like good marketing. These things will spread organically and drive traffic even if you remove the SEO benefits. Could you say that Scribd or Slideshare are doing something similar by allowing people to embed documents in their websites along with a no follow link back to their site?

thirsteh 13y ago
"nofollow" is the key.
JohnTHaller 13y ago
Allowing people to create links to your website isn't shady. Paying people to link back to your website on high-value keywords like 'justin beiber lyrics' from blog posts on other sites to artificially inflate your pagerank is 100% blackhat SEO. The payment in this case was a Twitter post about the link provider's website.

h1fra 13y ago

The tech part is obvsiouly here to calm down people from Hacker News and make them say "Oh this part was good, after all they are not so bad". It's just a shitty scraper but they talk about it like it was a revolutionnary script... There is no way they could get back so fast and so easily on google.

ghostdiver 13y ago

Disavow tool actions take months to bring any effect(if any).

hippich 13y ago
after a year I still see "unnatural" inbound links in webmaster tool. So I would say it have no effect.

OoTheNigerian 13y ago

I do not understand why people are offended that they came back quickly AFTER doing a bit of penance. Should their investors not have made calls? However, it should be noted that they did their part. One of my favorite quotes is this: "To save a drowning man, he must first give you his hand" If the Rap Genius guys were incompetent and did not do their part in removing the links so quickly (see main story for technical details) there would be little or nothing their investors would have been able to do about that. BTW, it also helped that people really found Rap Genius useful. I'm sure many searched had rapgenius appended to it. So the lesson is, make your app so good that when Google delists you, it will make them look bad. Thankfully?, there is also a precedence for other offenders to use. if you can detail you have atoned for your sins and Google insists your you must do a certain time, you know where to turn to. BTW, this is Hackernews, and I look forward to the discussion of the (de)merits of how they scrapped and analyzed 177,000 links and not espousing of anger that they survived the punishment. I'm sure Mahbod (my favorite) and co have learned that you do not go 'daaawging' when you are in a hole. I wish them success.

sbreaking6 13y ago
Ehh, the story is the double standard google is showcasing. The earlier story about Expedia and this outcome tell it well. This is rapgenius. If it had to cross the line why on something so materially insignificant.
holyjaw 13y ago
Just to hijack on one of your points: "it also helped that people really found Rap Genius useful." This is painfully true, and I know it has been discussed in prior discussions. Rap Genius is without a doubt THE BEST lyrics site available, hands down. Even if the other sites weren't ad-ridden scammy buggy shitty websites, if they got their acts together and actually TRIED, Rap Genius would still be top dog. I don't care for the attitudes or personalities of the people in charge, and I am absolutely disgusted that they felt the need to scam their Google rank, but I am willing to see that some infractions can be rectified through penance, and am happy to know that they will not be forced to shut down because of this fiasco.

grimlck 13y ago

This really shows that even in Silicon Valley, your success is driven as much by the connections you have and the people you know, rather than pure talent and merit.

theodoracharl 13y ago
What it really shows is that Y-Combinator's original ambition to support hackers in hacking capitalism is over. Successful startups, like mainstream corporations, are determined by 1. Public relations stunts, getting press, marketing bullshit 2. Connections, Privilege, Cronyism, Nepotism, Harvard, Yale, Stanford Paul Graham sold out and is now just like every other capitalist. Exploiting privilege to get richer. ycombinator's biggest successes are marketing successes--their success is defined by PR and Cronyism. This isn't Steve Wozniak--the tech genius--creating the Apple II. This is fat headed fratboys using PR stunts and cronies to rake in money by squeezing out the competition. The rich will get richer. The children of the rich will get richer. PR darlings and frat douchebags will get richer. Hackers will be ignored by YCombinator--Hackers aren't as profitable.

tszming 13y ago

People should worry why Google has so much power nowadays, rather than why they unban Rap Genius so fast.

kenjackson 13y ago
This really is the thing that struck me. All this talk about companies being destroyed and jobs lost and hardly a mention of worry about the power Google yields. We need more competition in this space. Sites need a search engine with real market share to compete against Google.
- diydsp 13y ago
  yup. just adding duck duck go to this laptop now. thanks for the reminder. https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ssl/?src=search https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ss...
- diydsp 13y ago
  yup. just adding duck duck go to this laptop now. thanks for the reminder. https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ssl/?src=search https://addons.mozilla.org/en-US/firefox/addon/duckduckgo-ss...

hippich 13y ago

I have to note, you have to be big to make comeback like this. Google provided disavow tool, but from my experience it simply not working. So it is manual action for sure.

virtualwhys 13y ago

Complete and utter bullshit. That they blatantly employed blackhat SEO techniques and are back on Google within a matter of days is sickening. Think of all the sites that are penalized in various ways for similarly shady SEO practices and have to spend _months_ to regain their previous position. Hopefully Bing et al can capture some marketshare and bring some degree of neutrality to the search engine landscape (i.e. not have every website owner on the planet beholden to the beast).

danso 13y ago

The "technical digression" with Github code on how to do a mass scrape is cool, though not surprising (er...how else would you accomplish such a big data task...though I'm glad I'm not the only one who uses Typhoeus, a great but underused gem). But what's surprising to me is that a company that knows how to automate things and work with data...why the f--- were they manually soliciting bloggers to post links? That seems seriously inefficient and comically backwards...as if Sergey Brin spent his early Google days modifying SERPs by editing and uploading Excel spreadsheets to the server. It seems seriously inefficient to me because at this point, Rap Genius has a lot of traffic and a lot of linkage already. So did the few backlinks, per day, that RapGenius got, through lame blogmasters who would participate in such a thing...really have an effect on their SEO rank, i.e. is it so easy to still game Google? Or, was RG just doing it because they thought it would have a worthwhile effect (keep in mind that this is one of their co-founders who was individually emailing people), i.e. RG was being naively optimistic?

jstalin 13y ago

Is it a sign of getting older when black background web sites get harder and harder to look at for longer periods of time?

NicoJuicy 13y ago

Actually, it's understandable that Google set this on High Priority... #1: it's a high traffic website, that is a startup and the whole deal got a lot of attention of us (HN Readers) + Matt even saw this here. So it was good to respond fast (no choice) #2: RapGenius is dependant on Google (although they try to let you see it as a social network, but it's not), they would have gone bankrupt ( i suppose) without Google. #3: Without a quick response and/or measure to fix their shit on their end, investors wouldn't invest any more money in RapGenius and Google would probably hurt a lot of future investments (how is this startup dependent on Google, ...) BMW once got a penalty from Google, they didn't got easily off. But their business wasn't dependent on Google. Either way, there are both pro and cons to each action. But overall, i think they handled if fairly well.

undoware 13y ago

I'm waiting for their next SEO single. "Oops, I did it again!"

wengzilla 13y ago

Infinite scroll (or some type of weird JS) won't let me get to the bottom of the page... Ain't got ish to do with this but I just thought that I should mention.

programminggeek 13y ago

One thing I know is that RapGenius is good about making a big stink, being in the news, and then having things work out in their favor, even if they were the ones doing a dumb thing in the first place. They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku? They did some incredibly dumb SEO things because they are greedy and are willing to cheat to win. That is on RapGenius, but once they got banned it is somehow Google's fault and they should be reincluded? I don't get it. RapGenius is great at one thing - creating controversy and getting press for it. I guess that works for them, but it seems like a very selfish way to grow a business. Instead of focusing on building value themselves, they are willing to tear down their business partners in public to get what they want. I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate.

jmduke 13y ago
They freaked out at Heroku with decent reason because it was costing them money, but if they ran their own infrastructure, that would have been a total non issue and at the scale they are running, why ARE they using Heroku? Are you arguing that what Heroku was doing wasn't wrong? I'm not sure why this behavior hasn't put them in the same category as Zynga, Groupon, Swoopo, and all the other companies the tech industry loves to hate. Comments on every single RapGenius post on HN have been filled with vitriol toward the founders, their attitude, and the culture they foster.

sbreaking6 13y ago

Clear as day there is a double standard. That's what should be discussed.

sbreaking6 13y ago

Well atleast now AH partners can get back to posting rap lyrics on twitter. We wouldn't want anything to stop that bit of entertainment.

friscobob 13y ago

God this pisses me off to no end. So my site is targeted by negative SEO. (Yeah, I know, I know, negative SEO doesn't exist. Fuck off.) So who do I call? I've been penalized for months and there is nothing I can do about it. Nobody can help. But these fuckers are able to make a call and recover from a SELF INFLICTED wound in less than a week. I am being targeted by crap that is COMPLETELY OUT OF MY CONTROL and it's tough luck little buddy. The double standard here is fucking sickening.

argonaut 13y ago
Disavow?
- friscobob 13y ago
  I've tried the disavow. Seen no indication that it had any effect. There were 31 thousand new backlinks added on December 31st (Holy shit, I just noticed that. WTF!). Another 1300 added on the 1st of January. It's normally 100 or 200 a day. Not sure how I'm supposed to manage the disavow process when I get that many new spammy inbound links a day.
  - aerolite 13y ago
    maserati problem
    - friscobob 13y ago
      Why would you say that? This is impacting my Google search results TODAY. If I turn off Adwords, my leads drop to virtually nothing...
MitziMoto 13y ago
This scares the crap out of me. There are some really shady people in my industry and I'm sure they wouldn't think twice about launching a negative SEO campaign against me. I wish Google and Matt Cutts would address this for real.

rasengan 13y ago

I am sure they went through many sleepless nights during the holidays. Hella sucks. I think Google is doing the right thing and I am sure not just related to seo that rap genius learned a lot about themselves and will become stronger... as well will the overall tech community.

mgulaid 13y ago

what does not make sense to me is that after removing all the back-linked urls, should not that drop Rap Genius's overall ranking even lower than before, since the artificial boosting has been removed. Why they are back on the top. googling "lyrics", RP is number #6 in the first page. I think RP has received special treatment here. Google search team needs to clarify this.

dhorowirtz 13y ago

http://m.entrepreneur.com/article/230631 http://m.entrepreneur.com/article/230631 But now, Rap Genius co-founder Ilan Zechory tells us the site has returned to Google's search results. "It takes a couple days for Google to re-index everything, so search results are a little wonky right now, but we are officially reinstated," Zechory said in an email. Somewhat ironically, Zechory served as a project manager at Google for two years prior to founding Rap Genius in 2009. >> Wow that is troubling. So site that gets banned gets unbanned quickly and founder just happens to be ex google project manager

JohnTHaller 13y ago

Honestly, this is just saddening. Google needs to have a consistent policy towards these blackhat SEO offenders and enforce it for everyone. RapGenius bought backlinks to artificially inflate their pagerank. They knew they were cheating but they did it anyway. They got caught. And they were hit with a manual action on Google. Sounds good so far. But that's where it stops. RapGenius, using the connections they have from their multi-million dollars in VC funding, got back on Google and was able to avoid paying the price for their cheating. Compare this to regular websites. Lots of smaller sites will end up paying a "whitehat SEO" firm to work on a "link strategy". These firms will claim up down and sideways that what they do is legal, ethical and follows Google's rules. But, what they actually do is either (1) create networks of fake sites to provide backlinks on certain terms to artificially boost the site, (2) place spam comments using bots on legitimate sites to do the same [not that this will thankfully no longer work well due to the latest Google algorithm update], or (3) pay legit sites to place backlinks to artificially transfer pagerank the same way that RapGenius did. Now, these other sites, when they get caught, they get a manual action or a smackdown. The difference? They have to actually pay the penalty. Arguing that they didn't know usually doesn't work. The penalty is LONG. They can't call on their VC firm to make calls at Google to give them a get out of jail free card. I'd like to call on Google to create a public policy on how they handle these manual actions with some clearly defined penalties (example: 3 month manual action of 6 PR drop, etc) and to consistently enforce them across the board. That way a mom and pop site that pays a 'whitehat SEO firm' and gets caught doesn't have a worse time than a site like RapGenius that purposely engages in blackhat SEO who can use their VC connections to get out of having to pay a penalty in under 2 weeks. It's also sad that Google gave this get out of jail free card to a site whose entire business model is based around other people's copyrighted works which RapGenius doesn't have a license for, doesn't pay for, and publishes illegally.

michaeldunworth 13y ago
This.
sheetjs 13y ago
Google is a business, and there is a very good chance that they financially benefit from Rap Genius being restored. Other companies that suffer don't have that financial link, and maybe the best course is for them to raise VC funds (and posture themselves so that Google stands to lose if they penalize) I agree with you in theory, but in practice it sounds like you expect google to act as if they care about things like fairness. They are a company. The bottom line is what matters. If you think it's unfair, then try to hit the bottom line by using other services and convincing your friends and family to do the same

McKittrick 13y ago

Its interesting that despite their founders outspokeness, the influx of heavy weight VC, tech press expose's on their offices, etc., RapGenius is really no more or no less than a standard, multi-multi-page SEO site, no different than the countless others just like it. No brand, no user affinity, no direct traffic to speak of. Just a bunch of drift net pages crawled by Google waiting to pick off unsuspecting search traffic. My favorite line of this post was the last: "Much love. iOS app dropping next week!" - unless you are searching that term in google, Im not sure who that is addressed to.

rburhum 13y ago

Snapchat could learn a thing or two of how to do PR from this post.

myzerox 13y ago

This is a relief. RG certainly gives the best search results for a lot of queries. This whole story made me fear that Google was choosing power over user experience. And that should never be the case, no matter how offending the founders of RG might appear sometimes. Don't be evil!

dclara 13y ago
Is there a smell of corruption? Because it is in favor of power, connections instead of user voting, just like in favor of dictation over democracy. Isn't it?
- myzerox 13y ago
  Fair voters = valid pages linking to RapGenius Bought voters = manipulated SEO spam blog posts Fair voters >> bought voters Not reacting to the SEO manipulation would have meant accepting a fraction of bought voters. Banning RG completely would have meant to ignore the even bigger fraction of valid voters. Google's actual decision seems to be the most democratic: Forcing RG to decrease the number of bought voters while not dictating over the voice of the valid voters.
  - dclara 13y ago
    You have an interesting point. But it's hard to say: Fair voters always greater greater than bought voters. When you buy backlinks from the SEO consulting service, they gave you hundreds or thousands everyday, just like some people complained in the comments. "Google's actual decision seems to be the most democratic: Forcing RG to decrease the number of bought voters while not dictating over the voice of the valid voters." The problem is: how can they tell which is which? I doubt this type of criteria is reliable. I collected some information from the user comments along with other research and wrote a blog post here: http://bingobo.info/blog/bingobo/what-are-the-current-seo-problems.jsp http://bingobo.info/blog/bingobo/what-are-the-current-seo-pr... Maybe you will also take a look at the possible solutions here: http://bingobo.info/blog/bingobo/how-bingobo-can-help-business-solve-their-seo-problems.jsp http://bingobo.info/blog/bingobo/how-bingobo-can-help-busine...

buro9 13y ago

Were the 178k URLs that they scraped on 178k unique domains or a very few domains? The tool they wrote looks like it would have been a DoS attack if the latter were the case. There appears to be no thought or consideration for the sites they scraped that had linked to them, no attempt to read robots.txt and see whether they should fetch the pages. What they did was wrong in the first place, but they could try and do things better in their attempt to repair things. Instead they continue to act in a way that externalises any cost to them and continues to treat their service as somehow privileged enough to deserve such breaks. How many of the 53k failures to pull and scrape pages were to do with the fact that the servers in question were under a DoS attack by RapGenius?

tzm 13y ago

"tis better to apologize later, than ask for permission first."

notastartup 13y ago

Certainly a double standard exists at Google. I originally thought finally Google is taking a firm stance, but this reversal only does harm to Google's image imho. Take home message is, blackhat SEO is forgiven if you are funded by connected people. Don't have funding or connections but make the slightest infraction in SEO? Feel the google girth.

bjoernbu 13y ago

So which hit should google show for a query like "the fox lyrics"? Of course, it should be a page that contains the lyrics, but which one? The most popular one? The one shown by google will be the most popular one. From an IR perspective, search for lyrics is somewhat strange. I don't think it's easy to say if some hit is more relevant than the other. Sure, more popular sites may offer better UX (or simply less annoying ads), but at the end of the day it's a search for facts. I wouldn't be surprised if google will serve those lyrics themselves some day. Freebase (and hence the google knowledge graph) already contains tons and tons of information on pratically any popular song (organized in a confusion way via recordings, canonical versions and whatnot) - artist, genre, album, release dates, length of a recording, etc. I don't think adding the lyrics as well is totally out of question. After all multible KBs are user curated and users provides lyrics. Just like they serve "fact-hits" from their knowledge graph for queries like "movies by spielberg", google should then be able to serve the lyrics directly. While this is bad for lyrics sites, imho it is convenient for users and follows the same prinicple like "movies by spielberg" or "wife tom cruise"

naner 13y ago

Rap Genius appears to have offset their drop in search results traffic with all the publicity they've been getting from being penalized by Google.

rodly 13y ago

If Google were to have a year long ban on RG stuff to leave a precedent, Would people start to view Google as a sub-par search engine because Bing returns "high quality" results like RG still? It's also rather weird how a lot of people seem to view Google as this public service for the people by the people etc, it's not. They can do whatever the hell they want and if you don't like it, simply stop using their service to send a message.

JeremyMorgan 13y ago

Money wins every time.

deleted 13y ago

[deleted]

deleted 13y ago

[deleted]

mathewsimonton 13y ago

The SEO strats / growth hacks weren't entirely misguided. Rap Genius just didn't perfect their activity and didn't let it exist entirely behind closed doors. If Rap Genius wanted to risk it and engage in blackhat-like activity, it should have at least kept things more quiet and less spammy. Rap Genius should have kept the finer details, at least, to a phone call with the blogger. Google can't see a local business handing their customers/client a $25 gift certificate in-person for a random positive Google+ review. Along the same thread, Google would also not be able to see Rap Genius flying out a community outreach rep to meet with a prolific music blogger at a bar to go over an exchange for a positive mention and authoritative link. Just saying.

davemel37 13y ago

I think it will take time for them to regain their rankings for non-branded keywords. I just searched for justin Bieber lyrics and they are sitting in middle of page 3 on google. They are ranking for branded searches, but not money keywords. Although long term, I see this being a net positive for them due to the thousands of backlinks they got from all the press around this story. I see it taking them several months to regain rankings...especially if the links they disavowed were artificially propping them up.

jfoster 13y ago

While we're on this topic, are GroupOn's API Branding Requirements in violation of Google's webmaster policies? http://www.groupon.com/pages/api-branding-requirements http://www.groupon.com/pages/api-branding-requirements They make it a requirement of using the API that "nofollow" is not used: "Don’t ‘nofollow’ your links to Groupon."

joelotz 13y ago

I give them a lot of credit for being transparent with their learning mistake, taking ownership, and describing what they did to fix it.