12 ms·
Youporn.com is now a 100% Redis Site
- antirez 15y agoThat's porn for database geeks: 100 million pageviews per day, 300k requests per second against Redis. The way they use MySQL is also interesting IMHO: they populate a relational database in order to be able to build new indexes in the Redis side, using the relational DB for the stuff it is best at, generating new "views" of the data easily. (Relational DBs are also good to do a zillion more things of course.)
- badboy 15y agoYou know your software is successful when it's used by porn sites :D
- pud 15y agoI remember in the 90's having a conversation with my friend. We were discussing Linux vs Windows for web hosting. My friend said "tell me if you can find a single porn site that's hosted on Windows." He was right, I couldn't find any.
- mhurron 15y agoThey do exist. I turned down the chance to interview at one about 6 months ago.
- leviathant 15y agoSure, but IIS in 2011 is a completely different beast than IIS in the 90s.
- junto 15y agoHow long did you spend on this "research" out of interest? :-)
- vacri 15y agoHard to say. The research was done in short bursts.
- traskjd 15y agoThey do exist. I run a .NET developer tools company and had a customer purchase our data access products to use with several of their porn sites. Never did manage to see the final product though!
- mahmud 15y agoYou prude!
- wonster 15y agoWorked for one that was at least in the top 5 porn sites back around 1998-99. All Windows servers running IIS and we were using NetApp appliances for storage.
- d3x 15y agoAEBN was and might still be the largest VOD service online and they run Windows or at least they did while I was there.
- brettz 15y agoSpankwire is alexa top400 and runs in .NET
- thezilch 15y agoVHS, Linux, and now Redis; the webscale race is over!!
- pellias 15y agoI was afraid to see what youporn is (at work now), so i guess it is really a porn site.
- gaius 15y agoThe online porn industry were the first people to really deal with scalability (of managing and serving web content, anyway), the first people to deal with online payments, the first people to come under really heavy, sustained hacking attempts, etc. Don't underestimate how much of what you take for granted technologically in 2012, they pioneered in the mid-90s.
- mun2mun 15y agoRelevant http://www.pcworld.com/article/155745/thank_you_porn_12_ways_the_sex_trade_has_changed_the_web.html http://www.pcworld.com/article/155745/thank_you_porn_12_ways...
- GrumpySimon 15y agoIt's not just now either, the porn industry was hugely influential in causing the adoption and spread of video (you know, on casette), movies and photography.
- jgw 15y agoIndeed - I've heard it argued that the real reason that VHS won out over the technically superior Betamax format, was that Sony refused to license the Betamax technology to adult video vendors. Never dug into it, but it seems plausible.
- saraid216 15y agoHaving just one deciding factor seems implausible. Straw that broke the camel's back? Sure. But calling it "the real reason" completely ignores every other relevant factor.
- anamax 15y ago
- itmag 15y agoPorn and war drive all innovation. Or so they say.
- roc 15y agoSex and war would seem more accurate; porn being just a subset of sex. And I suppose one could argue that war is often just a manifestation of a desire for sex or power/money as an intermediary to sex.
- rev087 15y agoWhich make the concept of war even more obscene.
- roc 15y agoIf you feel that sex is obscene, perhaps. Edit: Otherwise I don't see what the goal does to make the butchery of your fellow man any more or less obscene than it inherently is.
- fluidcruft 15y agoI assumed he was gesturing at the creation/destruction discord inherent in sex/war with war being the referent for obscene. Of course, a full account of sex more than kisses and pleasantries, it also involves competition.
- itmag 15y agoThe (oftentimes sublimated) drive to procreate is one of the most powerful drivers of all human society and action. Perhaps even THE "primus motor" behind it all. In Prometheus Rising RAW mentions that the reason monks take a vow of chastity is because it instantly frees them from one of the major hamster-wheels of the human condition. Of course, I'm not planning to become a monk :) But food for thought...
- Dylan16807 15y ago
- thezilch 15y agoWe do the same thing, where we populate a mysql store as a secondary storage for business intelligence -- our Founder, PM, AdOps, etc know enough SQL to be dangerous if we build out more traditional, relational models. It amused me that you were testing a porn site -- the homepage / potentially-misused zrange-issue.
- deleted 15y ago[deleted]
- ibadaxx 15y agoCongrats! BIG F*CKING CONGRATS! :)
- gnarfz 15y agoredis made with/for love :)
- Mikushi 15y agoYep, i've been a Redis lobbyist for over a year at this company (i use to be the Lead Software Engineer on Youporn/Pornhub), and i'm really happy to see them using it and using it properly.
- hello_moto 15y agoHow do you store the videos? I remember seeing YouTube storing videos using some sort of specialized hardware from NetApp or something and they still got hit by the limitation of Linux filesystem (number of nodes?)
- alanh 15y agoIt was number of files in a directory, and those were the garage days.
- StavrosK 15y ago300k requests/sec? Surely that can't be on a single Redis server, is it? Is Redis cluster out yet, or did they implement their own?
- sanswork 15y agoIt would be mostly reads so could be done with lots of slaves and sharding without redis cluster.
- StavrosK 15y agoVery true, but now I wonder how many reads/sec each of those servers pushed.
- thezilch 15y agoRedis can run anywhere from 50-200K IOPs, on a single core, dependent on CPU; you can find benchmarks at the following: * http://code.google.com/p/redis/wiki/Benchmarks * http://redis.io/topics/benchmarks At the point where multi-core sharding/ring setups are involved, you begin to have NIC/network saturation. It's feasible to achieve 300K on a single box.
- awj 15y agoThe OP actually addresses this. They're doing some form of clustering and had to add more nodes due to network saturation.
- shykes 15y agoJust to clarify, they're using regular master-slave, as opposed to "clustering" which is an unreleased feature. I love how they use sInterStore as a replacement for a fat SQL query.
- StavrosK 15y agoThat's amazing, thank you. This would make me want to use redis for everything, if I weren't already using it for everything.
- johnx123-up 15y agoDo they use any sort of wrapper or architecture around it? Is it in Ruby, BTW?
- pavelkaroukin 15y agoAFAIK, it is Perl project. And I remember seeing it showcased at Catalyst website, so it likely built on top of Catalyst Framework
- povilas 15y agoNo. It is PHP. https://groups.google.com/d/msg/symfony-devs/WGMooRPjmEY/hwWxacpaTQYJ https://groups.google.com/d/msg/symfony-devs/WGMooRPjmEY/hwW...
- pavelkaroukin 15y agoThanks. Tomorrow our site, with over 100 million page views per day, will be relaunching running on Symfony2. This is a complete rewrite. Prior to that it was running the Catalyst Engine written in Perl.
- webda2l 15y agoSymfony2 rocks
- chubot 15y ago100M page views per day is 1157 page views per second. I wonder how that's being blown up by a factor of 300? I suppose if each video's metadata causes a redis lookup and there are 300 videos displayed per page that would do it, but you'd think you could batch those lookups. It seems a fairly inefficient. Either that or they have some background jobs causing redis lookups that aren't a result of page views.
- memoryfault 15y agoPerhaps the 300k/s figure was at peak load.
- n_time 15y ago...
- eric-hu 15y agoThere's 30 videos on the front page. Mousing over the image preview of a video displays a series of preview images. There were 76 requests made as observed from the net tab in Firebug. The majority of these requests seemed to be for those video images. Note that I'm using an adblocker and refuse to check out the site without it. Note also that they said 300k queries. There could be constantly-running background jobs doing conversion, pulling those preview frames, and crunching numbers on the metadata.
- rhizome 15y agoSounds like Redis is being put to the role of reader boxes in the older split-head MySQL (or *SQL) architecture "write to master, read from slaves" with a conversion shim between MySQL & Redis..
- jarin 15y agoI'm doing something similar with Dirty Hot Productions (one of my clients). We use MySQL on a separate CMS Rails app to store everything, and all of the websites communicate with the CMS via an API. We use Redis to cache all of the API responses, so effectively everything except the first read (or the first "dirty" read) on a particular query is coming straight from Redis. It works amazingly well, and it doesn't use nearly the amount of RAM I thought it would.
- siculars 15y agoDon't travel to Iran.
- mvzink 15y agoYou know what this means? Salvatore Sanfilippo could be executed if he visits Iran.
- Iroiso 15y agoAbsolutely... Sucks right..
- RandallBrown 15y agoFor some reason I always imagine porn sites to be run by guys in cheap suits and lots of hair gel. Then I realize that in order to run a site with the kind of traffic that some of them have, you really need to have some decent engineers on staff.
- antirez 15y agoSure, not only there is huge traffic, they also generate tons of money, so porn sites are definitely not second citizens in the web from a technical point of view.
- jwr 15y agoYou'd be surprised if you went to meetings with porn business companies in Japan. Stylish offices, elegant suits, great manners, no hint of actual porn anywhere. In fact, you could easily confuse the office with an investment bank.
- arethuza 15y agoI once had a surreal meeting with a hotel chain where they mentioned they had detailed management reports on stuff like the average time films are watched for and that they had a graph showing how long porn movies are watched for. This was all being discussed in the context of a fairly formal business meeting.
- geoffw8 15y agoI used to work with a company who provide most of the network/software/hardware for hotels/airports/tran stations/lobbies around the world. Some very interesting insights from the hotels side of things.
- deleted 15y ago[deleted]
- DavidChouinard 15y agoCare to share?
- exim 15y agoHow are you getting the actual content?
- glogla 15y agoI would guess lot of nginxes (nginxii?) on top of SAN / really big NAS. But I'm curious about that too. Unless you ask how they get porn, that is.
- maw 15y agoIt'd surely be nginces (for n. and a., anyway).
- vegardx 15y agoIf you look at the http headers you'll see that they use varnish to cache the content and nginx as webservers. They probably have a pretty decent CDN to deliver the content to users across the world. Side-note: I also heard that the founders of Akamai started up after seeing how the porn industry handled all the traffic from geographical different places.
- eapen 15y agoGreat way to advertise the website to one of your target groups. ;) But seriously, I am quite surprised by the number of visits the site gets.
- visarg 15y agoGoogle missed an opportunity here - having YouTube and not YouPorn. Porn is huge.
- keithvan 15y agoWhen will they unleash the Python?
- SafdarIqbal 15y ago"After the switchover we had to add some additional Redis nodes but not because Redis was overworked but because the network cards couldn't keep up with Redis." Wow!
- EmielMols 15y agoThat doesn't seem very special or unique to me though. In almost all settings where some in-memory data is pushed on the network stack, you can expect stuff to be network bound.
- throwaway_uporn 15y agoHopefully someone from YouPorn is reading this: YouPorn totally stopped working on my Android about two weeks ago. I used to "monitor" that site daily, and it suddenly stopped working. I wouldn't doubt if it coincided with this change. I can see "still pictures" of the videos, but can't play them. This is true for the Android Motorola Droid 2 and the Kindle Fire. So I've had to monitor other sites since then.
- ultimatewarrior 15y agoWhy not Hadoop?
- scorpion032 15y agoPorn industry is generally among the first to adopt new technologies and promising architectures. Among other things, this indicates Redis' impending adoption as a standard infrastructure component across the industry in general and as a replacement to standard databases in some of those places.
- nvk 15y agoAnd you thought people were spending time on Facebook eh....
- ksajadi 15y agoproblem is you can't tell your boss: "well youporn uses it" when he looks dubious about using redis for his trading system.
- awj 15y agoThat depends entirely on your boss. Mine would probably hear the reasoning out ... then ask for a less "unsavory" example he can use when talking with everyone else.
- deleted 15y ago[deleted]
- leanucci 15y agoWhat is youporn?
- joering2 15y agodoes anyone know anything about their $ #s?? revenue model, revenue/profits/costs #s ??
- hercynium 15y agoI'm currently looking into redis for similar needs. I'd be using it from perl and wondered how will it would scale... (IIRC (from people I know who have worked there) youporn/pornhub runs on Catalyst, an MVC-based web framework written in perl.)
- draegtun 15y agoTheir certainly regarded as probably being one of the busiest Catalyst sites and they were employing Perl/Catalyst developers back in 2009: http://news.ycombinator.com/item?id=565152 http://news.ycombinator.com/item?id=565152 NB: The HN article link hits an expired job post. Here is original job ad via WayBackMachine: http://web.archive.org/web/20090418113245/http://jobs.perl.org/job/10462 http://web.archive.org/web/20090418113245/http://jobs.perl.o...
- Mikushi 15y agoYouporn used to work on Catalyst, it's been rewritten using Symfony 2. As for Pornhub it's running a custom PHP framework.
- zolfaghari 15y agowww.canadinimmigration.ca
- alexeykleptsin 15y agohttp://alexeykleptsin.ru/ http://alexeykleptsin.ru/
- nilsbunger 15y agoThe perfect alibi for watching porn at work: investigating redis.
- ypcx 15y agoProbably a better question for the Redis group, but I'll throw it here anyway. Can someone explain why there's no Ordered Map in Redis? Basically to achieve and ordered index, one has to use Sorted Set and a classic Hash Map, and pair these two when ordered data are needed, which is an extra level of indirection and wasted space. Am I missing something?
- Jayasimhan 15y agoDamn society, why can't you let these guys have a company blog where they can blog about this.