4 ms·
"Sufficiently advanced spam is indistinguishable from content"
- Gormo 17y agoIf we presume that any algorithmic, procedural, or structural system built by one party can be reverse-engineered and understood by another party, the concept of Optimization by Proxy, and the more general Goodhart's law, form a pretty compelling argument against designing optimized systems as solutions to problems in general. Maybe in some cases keeping a system convoluted and inconsistent can actually help ensure stability and durability?
- moultano 17y agoI work on search-quality at Google. This is my life.
- MikeCapone 17y agoTell us more!
- moultano 17y agoI wish I could. :) All of the fascinating things about signals are confidential for all of the reasons listed in the article, and Google has been sued so many times by sites that think they should rank better than they do that I can't really give examples. I think it's safe to say though that there are a lot of people worried about and thinking hard about what the web is turning into and how to rank it appropriately. Most of the content is no longer written by devoted hobbyists, people no longer link as often to things they like, and much of the content on the front pages of reddit, digg (and sometimes even hackernews) was put there by people trying to make your search results worse.
- whopa 17y agoI feel like you missed listing a big change: with the rise of UGC sites, the boosting of results of anything on an "important" domain doesn't seem valid anymore. Just because something is on twitter.com doesn't mean it's highly relevant, since anyone could've posted it. But it's fairly easy to google bomb someone's name by just registering a Twitter account and listing their name as owning it. Similarly, stackoverflow.com doesn't have very good answers for a number of technical topics, but it's often on the first 10 hits, even when the answers are useless and there's much better answers ranked lower (like project mailing list archives).
- lsc 16y agoI think that a mailing list only search engine done well and broadly would be a very useful technical resource. Much of the internet, at least for SysAdmins and Systems Programmers still happens on mailing lists.
- _delirium 16y agoYeah, I often find myself wanting an "exclude answers sites and wikis that aren't Wikipedia" checkbox.
- vannevar 16y agoThe days of a central search engine 'indexing' the Web are nearing an end---indeed, it's already over, people just don't realize it yet. New material now appears much faster than it can possibly be indexed, and unless Google figures out a way to beat Einstein, it's only going to fall further and further behind. People will discover relevant material through their network, in a massively parallel fashion. Everyone will be their own Google.
- alexandros 17y agoHi, I am the author of that. Would you say the depiction in the article is more-or-less accurate? I am asking as I wrote this purely from an outside/theoretical perspective.
- moultano 17y agoIt's very true for some things, and not for others. There's information asymmetry in both directions, things site owners know that Google doesn't, and things that Google knows that site owners don't. The summary of the history seems pretty accurate to my perception of it, but I don't think it's hopeless from here. :)
- jodrellblank 17y agoMy life is going from using a Google that used to give me useful results to one where "tar up website" returns the top result: "Deep-sea ice crystals stymie Gulf oil leak fix - Yahoo! News 8 May 2010 ... thick blobs of tar began washing up on Alabama's white sand beaches. ... platform at the Deep Sea Horizon oil spill site in the Gulf" At least a result from 4 days ago is an improvement on when I'd get usenet or mailing list results from 1999-2004 whenever I searched for anything linuxy. :/
- BoppreH 17y agoInternal Server Error And sufficiently advanced errors are indistinguishable from pages made for pure irony.
- moultano 17y agoGoogle cache: http://webcache.googleusercontent.com/search?sourceid=chrome&ie=UTF-8&q=cache:http://lesswrong.com/lw/28r/is_google_paperclipping_the_web_the_perils_of/ http://webcache.googleusercontent.com/search?sourceid=chrome...
- BoppreH 17y agoThanks, but it's back again.
- deleted 17y ago[deleted]
- diN0bot 17y agoabsolutely....sometimes i mark as "spam" conversations that i'm personally not interested in, even if the author is "legitimately" spamming me. (eg a mis-guided friend's mass email...or more likely the dozens of mis-guided reply-all's)
- alextp 17y agoI think this is a valid use case of spam filters. I have trained more than one to detect my father's powerpoint emails and bad chain-mail jokes and separate them from his personal messages that I actually want to read.
- Tichy 17y agoAlso all the newsletters companies feel entitled to send just because you bought a toothpick from them 10 years ago.
- RyanMcGreal 17y agoFascinating essay, but I'm not quite sure whether it's a problem that sufficiently advanced spam is indistinguishable from content. After all, Demand Media does produce real, editorially vetted content from real human writers. The payment system encourages what I'll call extreme efficiency of research and writing, but that simply optimizes it for the handy-reference domain of search results (e.g. How to fillet a smallmouth bass), which may not be "high quality" as such but does provide direct, clearly written and reasonably valid responses to the search queries that elicit them.
- moultano 17y agoI've seen a lot of pages where I couldn't tell if it was written by a markov-model or a human. Many of the people who get paid for $1 content don't speak English natively.
- klenwell 17y agoOn that topic, I present Emily Chimpinson, a little project I've been playing with: http://twitter.com/chimpinson http://twitter.com/chimpinson She's a Markov-based script inspired by the public domain works of a certain poet. All of her incantations are checked to make sure they don't repeat verbatim her model. Every once in a while she comes up with something inspired.
- kingsley_20 17y agoMost english speakers in the world learnt it as a second language, and it only becomes a problem to me when they're not good at it and they're trying to sell me a service predicated on their language skills.
- byrneseyeview 17y agohttp://www.ehow.com/how_2066890_get-bees-out-house.html http://www.ehow.com/how_2066890_get-bees-out-house.html
- duskwuff 17y agoI'd put a finer point on it: paid writing encourages the creation of content which appears superficially relevant (especially through the eyes of a search engine), but doesn't actually convey any substantial information.
- Tichy 17y agoHaven't read it all, but I am just wondering: by now data dumps of people's connections are probably making the rounds in the dark channels? I think sending spam that appears to be from your friends could be a big "improvement", and should be child's play with the data that is already freely available. Maybe that could become one of the first privacy disasters, when people realize they made their email unusable by publishing their connections.
- adriand 17y agoIt's interesting that PageRank's measure of quality is entirely dependent on there being a community that recognizes the quality of the content first, before the search engine. Without a community, you're not going to get incoming links. In other words, work produced by lonely geniuses is quite likely to go unnoticed. For all we know, the content that is being produced by companies like Demand Media has already been produced by thoughtful people, writing at length about subjects they love on obscure websites that no one ever links to. What a shame that would be!
- dejb 17y ago> In other words, work produced by lonely geniuses is quite likely to go unnoticed. I think this is something that has happened throughout history. The web probably makes it easier for the their work to be uncover than before but they are still at a disadvantage.
- moultano 17y agoI've actually seen that happen to one of the lead engineers in search quality at Google. He'd written a great guide to ultralight backpacking that until I linked to it, wasn't indexed by any major search engines. http://eric-and-april.com/Ultralight/index.html http://eric-and-april.com/Ultralight/index.html
- ovi256 17y agoIs this a meta-post ? Did you just link to it for the first time ?
- ugh 17y agoIn other words, work produced by lonely geniuses is quite likely to go unnoticed. It’s not quite as depressing as that. I recently made a quaint little site for a band and it has exactly zero other sites linking to it. It’s the first result when you search for the name of the band (which is town-name+generic-term-used-in-bandnames). This only works with stuff that’s rare on the web, though. If there were other bands with the same name and if someone linked to them my little website would probably get swamped. (The same would presumably happen if someone were to write a blog post about the band – say, a scathing review of their last gig – and if that one post gets only a handful of links. Hm, so getting a few links seems at least like a good defense in such cases. Luckily many of the band’s target demographic aren’t actually all that internet savvy :)
- pook 17y agoHah, I ranted a bit about this evolutionary arms race years ago, from a different angle. http://hamstermotor.motime.com/post/683104/the-future-of-spam-an-information-theoretic-argument-%252Arestored%252A http://hamstermotor.motime.com/post/683104/the-future-of-spa...
- alexandros 17y agoThat was very interesting. I added a pointer to your article to the OP.
- samg 17y agoJust ask Calacanis!
- randfish 17y agoMoultano - I have a strange request, but one I hope you'll take seriously. I think this issue is very important - to Google, to web searchers, to businesses seeking to be found by Google and even to less scrupulous web operators. I'd love the opportunity to engage in 20-30 minute written chat with you and publish it (anywhere on the web you'd like). As background, I've worked for years as an SEO consultant, founded a community and company in the space (SEOmoz.org), and have been spending the last few years developing and launching search marketing software. I certainly respect your background and beliefs, but I think there's some flawed logic in your assumptions and arguments that I'd love to dig into, talk about and maybe even have some of my own perceptions changed. I would not ask you to disclose anything that's confidential - I'm much more interested in the theory and logic behind web spam, SEO and search relevancy. You can reach me via email - rand@seomoz.org. Would love to hear from you!
- moultano 17y agoSorry, I'm not equipped for that sort of public discussion. Talk to Matt. ;)
- randfish 17y agoIt could be anonymous on your end?... I like Matt a lot, too, but his opinions (at least, those he publicly offers) are well known and well publicized. It would be great to hear other voices. If not, how about if/when you ever leave the team. Happy to be patient :-)