7 ms·
Robots.txt
- phillgeorge 14y agoBmac27, what kind of enterprise seo toolset do you recommend ? I am currently evaluating Skycara, Brightedge and SEOcatalyst but wanted something else to try and look at. I'm runnign a site with 15,000 pages, 10,000 errors and about 20k keywords
- deleted 14y ago[deleted]
- artursapek 14y agoI thought he was gonna ask for a job.
- deleted 14y ago[deleted]
- gcatalfamo 14y agome too. in that case it would have been cool.
- explicitlyme 14y agoHa! I should have... Maybe in the next version...
- hornbaker 14y agoAn SEO consultant complaining about Google's frequent algo changes is like a mechanic complaining that cars break down too often.
- RyanMcGreal 14y agoIf I was feeling uncharitable, I would say it's a bit like a burglar complaining that the homeowner keeps installing better locks.
- devcpp 14y agoAnd the house would be the White House (in the sense that SEOs aren't just working against Google but against its billion users).
- aidenn0 14y agoMore like an arsonist complaining about updates to building code.
- cryowaffle 14y agoBingo
- explicitlyme 14y agoActually you are more right than you think. Frankly I would rather do something else than SEO. Like real marketing. (I am the author of said site)
- tlrobinson 14y agoIt's more like new cars being introduced without service manuals, forcing the mechanic to continually figure out how to fix them.
- drzaiusapelord 14y agoExcept SEO isn't a typical use case. It like complaining that newer tires made for better grip in the wet aren't as good as cutting people off during rush hour.
- mixedbit 14y agoI don't think this is a good analogy. Google's job is to make sure good content is ranked high and crap is ranked low. You should just create content that is useful to others and let Google figure out how to discover it and rank it high (they are good at it). To create useful content, you don't need to know Google internals.
- dohertyjf 14y agoAnd we would just do this if Google actually made good on their promises to do this, but I see an awful lot of crap still ranking in the SERPs especially because of hacked sites and link networks. Full disclosure - I work in SEO.
- illuminate 14y agoThe solution to no-content black hattery is not to promote your no-content site with black hat techniques. Nobody here is criticizing site tuning and at the same time defending content farms. Why defend misbehavior as a justification for misbehavior? Note that I am not saying that you yourself perform any of this in your role. Our anger is not at site tuning and placement but the techniques used to do so, which you appear to be unnecessarily defending as essential for any site's survival.
- dohertyjf 14y agoYour comment doesn't make any sense. Of course the solution to something bad isn't promoting it with something bad. That's just ridiculous and I would never argue that. You're automatically assuming that SEO is bad, or is all blackhat, or is just about promoting crap. I would argue vehemently that good SEOs only want to promote good content. I turn down many freelance gigs because the company is shit. I think you're confusing legitimate SEO with crap affiliate marketing. Big difference there, buddy.
- drzaiusapelord 14y agoOr whining about "too many computers in cars nowadays." Thanks but no thanks. I'll take the improved efficiency and reliability over your laziness to retrain.
- illuminate 14y agoA legit complaint would be the miniaturization making it difficult to work on your own car without a more complete disassembly, engine removal, etc, I rarely hear "too many computers" from actual mechanics or people who used to do self-repairs in the past. I tend to hear these complaints from less-knowledgeable consumers rationalizing why their cars are breaking down.
- bunderbunder 14y agoNot entirely - after all, to his clients an SEO consultant isn't just a black hole they throw money into. They do expect to see results in the form of higher search ranking, and Google's algorithm changes make it harder to deliver on that. I think I'd mirror others' analogies. It's more that the complaint is futile - and kind of sad. One comes away with the impression that the author is naive enough about the nature of his profession to genuinely believe that, on this subject, he and Google should be buddies and not adversaries.
- illuminate 14y ago"to his clients an SEO consultant isn't just a black hole they throw money into. They do expect to see results in the form of higher search ranking, and Google's algorithm changes make it harder to deliver on that." Wouldn't it then be harder to demonstrate objective returns if Google's algorithms are so obfuscated? If they're not afraid to game Google for their customer's benefit, they're certainly not going to be afraid to game customer expectations and fudge their results for a living.
- mattmaroon 14y agoNot at all. It's like complaining they don't break down enough.
- dkhenry 14y agoKeep up the good work Google. I hope this serves as encouragement for them to continue to screw over those who want to turn searching for relevant information into a war for who can cheat the system the best.
- buster 14y agoExactly my thoughts.
- morganwilde 14y agoExactly what I thought, SEO should not have ever become a "thing" if Google did its job right.
- illuminate 14y ago"SEO should not have ever become a "thing" if Google did its job right." SEO will always be a "thing" as long as people are not always interested in providing relevant information. It will always be an asymmetric battle between people who want to prioritize their noise over useful signal.
- pbhjpbhj 14y ago>SEO will always be a "thing" as long as people are not always interested in providing relevant information. // SEO is [also] about bridging the gap between machine-based abstraction of the data in your site and human searchers. Most of the on-page/on-site optimisations are simply ensuring that the SE can abstract the right information so that users searching for the content you're providing can find it. SEO will always be a thing as long as Googlebot fails to perfectly abstract the data content of a site and perfectly rank sites according to user needs.
- illuminate 14y agoThe discrepancy here is probably that "SEO" is mostly used as a pejorative versus the intentions you describe.
- buster 14y agoAm i the only one who thinks that SEO is more a plague then it is a real profession? Google making it hard for SEO is exactly the right thing to do. Making it impossible for SEO should be the ultimate _goal_ of a good search engine.
- Caligula 14y agoMy opinion of google improved reading this article. I equate the quality of Google's search as the inverse of the happiness of "seo gurus".
- explicitlyme 14y agoSometimes, that statement is fairly accurate, then in other times when google boosts its own commercial properties to increase their ad revenue, it wouldn't be.
- Osmium 14y agoI think this is a common view, and I share it: I really don't like SEO. The thing is, SEO does provide real value, but it provides value because the system is broken: your website should reach the top of search results based on merit, not based on some arcane black magic. I think it's somewhat akin to lobbyists: I don't blame them for making money off and taking advantage off the political system, and they certainly provide something of value to their customers, but I dislike the fact that they can exist at all. But that's not the lobbyist's fault, it's the system. In this respect, I applaud Google for keeping people on their toes and making SEO less relevant. That's their job after all.
- dpeck 14y agoIn short, don't hate the player hate the game? I understand where you're coming from, but that's crap. Just because there is always someone willing to do ethically questionable things that fall within the letter of the law, doesn't mean that those types of people shouldn't be treated with disdain by the rest of us.
- moccajoghurt 14y agoI wonder how this post got to the top even though the sentiment in the comments is that this posts sucks.
- grecy 14y agoI think you'll find only a very tiny fraction of HNers bother to comment, so it's not a representative view of people's opinions on a given article.
- michaelhoffman 14y agoWe can't downvote posts.
- deleted 14y ago[deleted]
- Terretta 14y agoBut we can flag them.
- michaelhoffman 14y agoYes, but why would we?
- illuminate 14y agoRight, I'd only do so if it was an outright scam.
- deleted 14y ago[deleted]
- oldpatricka 14y agoI find the perspective of the SEO guy interesting, even if I find what he does kind of skeezy and annoying.
- webwanderings 14y agoI think SEO is practically dead, and for good. The Social Web is supposedly lot more democratic and apparently a better choice.
- dublinben 14y agoI think any good social marketers (as they've now rebranded themselves) are fully involved in subverting The Social Web. Any site that reaches a certain size (Digg, Reddit, etc.) will become a ripe target for less than organic posts.
- homosaur 14y agoI can give plenty of counter examples, but the one that immediately jumps to mind is orgs like Pissed Consumer, who are essentially using subdomains to hijack other companies' Google juice. However, it's also hard for me to suggest that those links are not relevant to queries if you look for something like "_____ reviews"
- eksith 14y agoSEO Won't be any more dead than health fads (which really started in the late 1800's). As long as Snake Oil exists, people will buy it. They'll probably take on a different guise now, like calling it "Social Media Promotion" or some other malarkey. Plenty of companies want to be the talk of the town on FB and Twitter.
- ErikAugust 14y agoThese days I'd argue to a client to spend their money hiring PR people. PR people obtain high quality backlinks the legit, organic way.
- illuminate 14y ago"I think SEO is practically dead, and for good. The Social Web is supposedly lot more democratic and apparently a better choice." Subverting "The Social Web" occurs to feed users to their advertising or other attempts at monetization, they don't get paid to Tweet.
- 14y ago
- napolux 14y agoThat robots.txt is so big that even with comments maybe google will drop it after trying to parse it for a while....
- fein 14y agoI would hope that google, of all companies, would be capable of writing a crawler that ignores robots.txt comments. Other than that the guy only has one rule.
- zalew 14y agorobots this size aren't that unusual http://www.bbc.co.uk/robots.txt http://www.bbc.co.uk/robots.txt
- chayesfss 14y agowow, just realized this is a great way to find information about websites
- deleted 14y ago[deleted]
- thenextcorner 14y agoI think you just proved the point here: SEO can be very valuable. Any good SEO would point out to the BBC that having similar rules for User-agent: Googlebot-Mobile and User-agent: * is not very efficient.
- signifiers 14y agoI bet they are fairly liberal with their size limits: http://www.amazon.com/robots.txt http://www.amazon.com/robots.txt http://aws.amazon.com/robots.txt http://aws.amazon.com/robots.txt http://www.google.com/robots.txt http://www.google.com/robots.txt http://store.apple.com/robots.txt http://store.apple.com/robots.txt
- martin-adams 14y ago
- daigoba66 14y agoI always thought SEO was supposed to be a cat/mouse game. The search engine tries to balance the search results fairly. The SEO consultant tries to game the system and get certain links to appear ranked higher, whether fair or not. Isn't that just how this works?
- perlgeek 14y agogaming the system is blackhat SEO. There's also supposed to be the SEO business where you actually improve the site's quality, indexability and speed.
- woohoo 14y agoNobody else read this as a joke? I read it as a joke (and a slightly racist one at that).
- illuminate 14y agoEntitled SEOs tend to be more common than exceptional. While I didn't see anything so hamfisted that it had to be parody, but I can see what you mean. From the topics on their default page and his other communications, it seems to be sincere. https://twitter.com/rishil/status/299185628321566720 https://twitter.com/rishil/status/299185628321566720
- allons-y 14y agoI was wondering if it was satire... especially once I got to the designer watch bit. If it is, it's really dry.
- CKKim 14y agoI read it as a joke by the end but, worryingly, didn't automatically recontextualise it as racist in light of that. Which, I think I agree with you, it is, if it's a joke and all the East Africa stuff is made up for "comedy value".
- Apocryphon 14y agoI thought he was genuinely from Kenya or Ethiopia or somewhere.
- nirvanatikku 14y ago....? where did the idea of "The East Africa stuff" being made up come from?
- CKKim 14y agoNowhere. It was speculation on my part extrapolated from the premise that the piece was written as a joke and nothing in it could be taken at face value. A little research on the guy suggests I was reading too much into it. Here he is http://www.linkedin.com/in/rishilakhani http://www.linkedin.com/in/rishilakhani and his languages are listed as "Hindi, Swahili, Gujarati, Punjabi" - we don't need much more than that to connect the dots.
- bengillies 14y agoIt seems that most people here would like it very much if SEO died a horrible death as quickly as possible. Let me try and paint a (probably somewhat sketchy) picture of why it should exist (I have nothing to do with SEO consulting btw): 1. You have a business. 2. Said business has a website. 3. Said website is bringing in no discernable revenue for your business. 4. Poking around for reasons, you discover that your website ranks really lowly on Google. 5. You have no idea why. 6. You pay an SEO consultant. They take a look at your web site, and your overall web presence, to try and find out what's going wrong. 7. They discover that your markup is terrible, the wrong things are enclosed in header tags, etc. Also, you have little to no visibility outside of your website. 8. They advise some sensible changes to your markup, and force you to start posting things on Google+, Twitter, Facebook, etc. 9. Your ranking improves. As does your revenue. Which bit of that process is evil and wrong? Of course, there will always be a shady side to SEO (people taking it too far, etc), but there's a valid reason reason it exists: many people simply do not know enough to get the basic components right. SEO consultants help them do that, and improve revenue as a result.
- cs02rm0 14y agoI think the general contention is that the process you describe is just using someone who's going to write sensible HTML not SEO. The shady side of actually modifying code not for easy-to-read, browser-efficiency, etc. is SEO.
- RBerenguel 14y agoI have to disagree. I consider part of my job to be some SEO (not my primary task and neither the most important, but I take care of it at my company) and its definitely the first part of what you sketch. And trying to clean all the second part stupid stuff someone else may have done before. I don't see it as a game against Google... I try to make our sites as useful for humans as possible, and hope Google agrees with a decent enough ranking so humans can visit it and enjoy the content. But when it doesn't, then the game is between me and the "other SEOs" who may not see the point of making something useful for readers, not machines.
- tibbon 14y agoI still strongly feel that the best way to achieve excellent SEO is through building human-accessible pages. How is your site on a screen reader? If its easy to navigate by braille keyboard or voice output, then it probably is well enough that Google should be able to make sense of it. Better yet, make it so that your CODE is readable and understandable easily by someone who doesn't have perfect vision. If you achieve this and create great, relevant content, then Google will snap it up and the customers will appear. No other tricks needed.
- explicitlyme 14y ago10 people targeting the same keyword. Only 10 spots on page one of the SERPS, if the on site content and site is well built, then who gets the top spot? The machine isn't clever enough yet to get the perfect answer, and till it is, humans and marketers will continue the manipulation of visibility in pursuit of traffic and revenue. It's sad, but it's a fact.
- addlepate 14y agoI think discouraging SEOs was the whole point. Find a new way to make money without improving anything.
- mixedbit 14y agoSEO has made a lot of damage to the Internet. 10 years ago links really expressed people preferences, and it was easy to get linked and ranked high when you've published useful stuff. Today, almost all large sites that people use to share links to original content use nofollow links. Because of this, it is common that when you Google for some original content, you will find links to aggregators (Stack Overflow, Reddit, HN, etc.) but the original page will be very far on the result list.
- handzhiev 14y agoWell, actually Google did this first, because they started using links as "votes". Then everyone started hunting for links instead of trying build do good content.
- deleted 14y ago[deleted]
- daGrevis 14y agoYou should use `humans.txt`. :) http://humanstxt.org/ http://humanstxt.org/
- Igalze 14y agoVery cool stuff but I have to disagree with the general idea. I've been doing SEO for 8 years and I have to say that I came for the fun and money but I stayed for the challenge and algo changes. I remember how, 8 years ago, it was all about "5% density" and link stuffing. Today is content quality, social signals, link profile (not just quantity), contextual co-relations and so much much more... I think that I only truly fallen in love with SEO when Google started using rel=canonical. This is when it really became a game of poker and not a "hungy hungy hippos" with links in it.