9 ms·
How I Made Porn Video Streaming More Efficient with Python and C
- lmm 13y agoIt's easy to underestimate the power of switching to a better language by just doing it - guess at the syntax until it works, then refactor as you start to understand the language and its culture more. In fact I've found I learn faster this way than any other.
- kbenson 13y agoI love reading articles about technical issues and solutions in the Porn industry. It's like getting a peek inside a youtube scale company as they grow.
- niggler 13y agoAre any of the porn sites larger than youtube?
- mseebach 13y agoThe question isn't unambigiously answered, but there's a top of good stuff here (first link in the article): http://www.extremetech.com/computing/123929-just-how-big-are-porn-sites http://www.extremetech.com/computing/123929-just-how-big-are... "In short, porn sites cope with astronomical amounts of data. The only sites that really come close in term of raw bandwidth are YouTube or Hulu, but even then YouPorn is something like six times larger than Hulu."
- deleted 13y ago[deleted]
- magikbum 13y agoI would rather learn if there was a porn site bigger than Netflix.. as Netflix currently uses around 30% of US bandwith [1] 1. http://www.pcmag.com/article2/0,2817,2395372,00.asp http://www.pcmag.com/article2/0,2817,2395372,00.asp
- skriticos2 13y agoProbably. Netflix has the big constraint of availability that only extends to the US. Porn is global. Even though the US has a above-average bandwidth usage (guesstimated) it's still only a small(ish) drop in the global network.
- toptalgergely 13y agoEU traffic in our case about ~1.2x the US traffic.
- rorrr2 13y agoNo way. One year ago Youtube users uploaded an hour of video every second.
- christopheraden 13y agoDisregarding the morality and ethics that are often center-stage, they are faced with really interesting technical hurdles. It's an old article, but I was fascinated to hear about some of the challenges they face as well: http://highscalability.com/blog/2012/4/2/youporn-targeting-200-million-views-a-day-and-beyond.html http://highscalability.com/blog/2012/4/2/youporn-targeting-2...
- toptal 13y agoHey this is awesome! Though the admin/moderator changed the title for some unknown reason from the title of the blogpost to their own.
- wodow 13y agoThis happens quite a lot / has become standard on HN (since sometimes in the last year).
- vsh 13y agoOf course because of snobbery. NH visitors should be protected from words, which starts from "p" and ends with "orn".
- niggler 13y ago"NH visitors should be protected from words, which starts from "p" and ends with "orn"." popcorn? preworn?
- mezeek 13y agoThis title and the article's title don't quite match...
- crm416 13y agoSubmitted title matched the page, but it's been changed by the mods
- xSwag 13y ago>The aggregated bandwidth of the clusters was around 50 Gbps, from which they used around 10 Gbps while at peak load. Now that is a lot a porn. Also, the illustrations look really good! How did you make them?
- OtherPlanet 13y agoVery glad you like those illustrations, first we have created couple of the sketches on the paper and then recreated that in the Photoshop.
- deleted 13y ago[deleted]
- tarikozket 13y agoWhat service do you use for flowcharts?
- i5rider 13y agoGreat work. I worked with RTMP for some time and I recall the pain to reverse-engineer their protocol. One other comment, maybe the title should be more like: "How I improved a slow/inefficient RTMP video streaming service by 20x". Just a thought.
- hackerboos 13y agoWould he get the same CTR if he changed the title?
- jallmann 13y agoAgreed that RTMP is an abomination that needs to be exorcised from the Earth as soon as possible. Unfortunately, it is probably here to stay until something like WebRTC gains critical mass. It's not clear what the article means by "repackaging" a stream or "pointers" to tags (especially in the diagram that shows tag pointers being transported between users). While RTMP is cumbersome, shoving media data (tags) under a per-session protocol header is essentially the standard way of moving data from one session to another. So I'm not really following this. Is this cutting out the RTMP entirely for receiving clients, and instead sending the FLV down via another transport, like HTTP or whatever? Or is it more of, "I wrote my own RTMP streaming server in C and Python", along with some implementation details which I'm not understanding? (Not that there's anything wrong with doing so. Options are limited in the streaming-server space.)
- toptalgergely 13y agoAuthor of the original post here. >It's not clear what the article means by "repackaging" a stream or "pointers" to tags (especially in the diagram that shows tag pointers being transported between users). By repackaging I meant extracting the FLV tags which pretty much travel in the same format as in an FLV stream (.flv file) if memory serves well. Pointers to tags refer to the internal implementation. I took the FLV tags out of the RTMP stream, which resulted in an almost complete FLV stream. With the proper header prepended to it you could save it as a file and play it or stream it and play it. That's just what I did, created a header for every new user, and after that was sent I could just stream the FLV tags from a common buffer. The users had pointers that pointed in this buffer so after the header is sent it was true multicasting. > So I'm not really following this. Is this cutting out the RTMP entirely for receiving clients, and instead sending the FLV down via another transport, like HTTP or whatever? Or is it more of, "I wrote my own RTMP streaming server in C and Python", along with some implementation details which I'm not understanding? (Not that there's anything wrong with doing so. Options are limited in the streaming- server space.) Yes. From the source RTMP stream I extract the FLV tags, which I could use to multicast. Sending the same RTMP stuff to every user would not work, but I can easily send the FLV tag stream over HTTP if I send the crafted header first. I hope that helped
- danso 13y agoI sympathize with why the mods changed the title from "Porn" to "Video", but now you've made it so that HN users are unwittingly clicking into an article that talks about porn, in such a way that could be detected by a workplace firewall. That's not an ideal usability decision. Why not use good ol square brackets to maintain the integrity of the title: How I Made [Porn] Video Streaming 20x More Efficient with Python and C
- FiloSottile 13y agoOr the other way around How I Made Porn [Video Streaming] 20x More Efficient with Python as technically the editor notes are bracketed.
- obviouslygreen 13y agoI think your sympathy is overapplied here... that's very clearly an important bit of information for anyone considering clicking on this link. Thanks for putting the warning up for those of us who would not otherwise have known thanks to a poor title edit.
- coldtea 13y ago>but now you've made it so that HN users are unwittingly clicking into an article that talks about porn, in such a way that could be detected by a workplace firewall. If you get heat about things like that in your workplace, then run, don't walk.
- peterhajas 13y agoWhy do mods change titles of posts? This is frequently complained about on HN, and seems unfortunate.
- philh 13y agoMostly they make things better, and nobody notices. Sometimes they make things worse, and people complain.
- enraged_camel 13y ago
- hardik988 13y agoI appreciate the effort the mods take in curating titles, I really do - but please spare articles like this at least? I clicked on this link at work (part-time at grad-school) and now I have a "how to run a pornographic website faster" link logged in my name.
- brokenparser 13y agoSaid Mr. Hard Dick.
- sc00ter 13y agoOh, please. The article uses the word porn exactly three times in the introductory paragraphs. Four if you count the title. The comments here have double that, including your own use. If you were genuinely that paranoid about being "logged" for having visited a page that used the word porn (really?!) I doubt you'd be using it yourself.
- hardik988 13y agoI apologize if it sounded that way, but I wasn't insinuating that I'm going to get into trouble. I'm just saying that it could get someone into trouble. As far as the "logging" goes, it had more to do with the word porn in the title of the page, because many content-filters just parse the title for blacklisted words.
- jeremyx 13y agoWhat is the association with this site toptal.com? Has anyone here ever worked with them before?
- toptal 13y agoHey Jeremy, this is our engineering blog.
- jeremyx 13y agoThanks. This is the only article on the blog? I was actually intrigued by the site and was wondering if anyone on HN has used this service as a dev or a client....
- crm416 13y agoOP here, and Director of Community at Toptal. This is the first live article, but we've got loads more in progress covering a bunch of different projects and technologies.
- jeremyx 13y agoBallsy to lead with an article about porn. I'm glad you did because I know they must face scaling challenges that only a handful of services have to deal with, but rarely do I see articles about how they overcome the challenges. Thanks.
- toptalgergely 13y agoI'm glad you liked it, it was a lot of fun. I also implemented on the fly transcoding, but that's probably for a different article. For the tech, well really at that place we used pretty much what everyone else did. Not a lot of big secrets, as far as I know the porn driving the tech revolution thing is ridiculous.
- toptal 13y agoJeremy, also to note we'll be putting out many in-depth articles about a variety of engineering related topics in the comings weeks.
- ckdarby 13y agoNothing open source in this article which was the biggest drawback of me reading this.
- toptalgergely 13y agoUnfortunately that's the case. I heard really good things about wowza these days. Not open source but a lot of things are open and flexible. Also back then red5 was also an ok alternative, not sure if that's still the case.
- atechnerd 13y agoMore efficiently? For two minutes of video? I could see if you were trying to stream Zombieland or Spaceballs... but porn?! I guess whatever inspires people to innovate is okay with me.
- toptalgergely 13y agoOriginal writer here. Actually this was a website which streamed online broadcasts live from people, like ustream does but with adult content. Some broadcasts could be hours long.
- dangayle 13y agoI got hit with a firewall and a "This event will be reported". Thanks, HN Moderator.
- jneal 13y agoI know others already said this, but I clicked on this link from work and immediately became appalled when realizing it was about porn and quickly backed out. I'd gladly read this from the comfort of my own home and I'm sure the content itself is SFW but still the point is I was mislead. Can we please [mods] not change titles in cases like this?
- Myrmornis 13y agoDon't understand. You can browse articles for leisure at work. So what was the problem with this article?
- buster 13y agoStrange how 3/4 of the comments are more concerned with the title link, instead of the actual content. Very nice post, was an interesting read!
- tekacs 13y agoTo be fair I clicked on the article and immediately came back to the comments expecting the absolutely inevitable sprawl of comments about the title edit. :P So not strange, methinks. :P
- proland 13y agoIf you're going to change the title, at least add NSFW to it...
- specialp 13y agoHow is it NSFW? It is an article that is about video streaming on scale. Yes it does mention it was for a porn site, but it really does not have much to do with the article. If you work somewhere where this would be viewed as inappropriate I am not sure what can be viewed at that employer.
- Myrmornis 13y agoThere's no reason anyone should cater to the consequences of you working at a prudish and illiberal organization.
- ConceitedCode 13y agoIs there an open source project that accomplishes something similar?
- bochi 13y agoThere is on open source a Nginx module that streams RTMP and creates HLS segments that looks promising: https://github.com/arut/nginx-rtmp-module https://github.com/arut/nginx-rtmp-module There are also red5 (http://www.red5.org/ http://www.red5.org/) and rtmpd (http://www.rtmpd.com/ http://www.rtmpd.com/)
- chopsueyar 13y agoIs this similar to what Wowza (wowzamedia.com) does? In terms of licensing, it is $55/mo/instance or $995 for a one-time license. There are also EC2 instances that start at 15 cents per hour. Much less expensive than FMS
- mtrimpe 13y agoYou also have Red5 which is open source and rocks very much. It is just a -tad- less reliable than Wowza which you can't really sell to paying customers, but it's much more fun and customizable. And to answer your question, Wowza is just a cheaper FMS which also supports other platforms next to RTMP. This article is basically taking the more performant HTTP download mechanism for static content (like YouTube uses) and then hacking it to put a live stream in it instead.
- ericcholis 13y agoThe network graphics are pretty sharp, anybody familiar with where they came from? Or, are they custom?
- coldtea 13y agoI'm astonished by some of the responses here, that could be summarised thusly: >"I work on workplace, in a modern western society, not some theocratic backwater, that monitors my web activity and would frown if I visited an article with the word porn in it. This on 2013. I find this OK, and won't quit my job or raise hell protesting this degrading treatment, but would rather complain for HN titles". In an age where people fight for LGBT rights, this is what the American workplace has come to?
- sillysaurus 13y agoIt's not that simple when you have a family to provide for. The job you have might be the best job for many miles. I don't agree with the decision not to protest, and I don't think HN should really care about catering to that demographic, but I understand their motive.
- danso 13y agoSince when did workplaces become bastions of principles, rather than bureaucracies that have a business to run and have to, at some level, apply policies that suspend an employee's worktime freedom as part of the agreement that involves paying that employee's salary? I work at a small company where no one would care. They also don't care if I spend time on Facebook, Twitter, etc. etc. But not everyone is in a company like that. Big legacy companies hire from the HN crowd too. Whether or not their policies are fair is not always a simplistic argument. However, what is as sure as rain is that trivial violations of those policies can be used as leverage to punish people, when the office politics get dirty and desperate. It's best to let the worker -- the HN reader in this case -- make the decision whether he/she wants to pick that fight, rather than have them accidentally stumble into it.
- coldtea 13y ago>Since when did workplaces become bastions of principles, rather than bureaucracies that have a business to run and have to, at some level, apply policies that suspend an employee's worktime freedom as part of the agreement that involves paying that employee's salary? Since people let them get away with it. The "agreement that involves paying that employee's salary" does not meant they should get away with treating him less than a civilized society accept. Signing a contract to work on some place doesn't give the employer any inherent rights over the employee, besides those that the society is willing to accept. Hitting an employee was once tolerated. Not so anymore. One time not hiring or paying blacks less was allowed. Not so anymore (not explicitly at least). Child labour was allowed. Not anymore. Racial or sexist slurs were allowed. Not anymore. Lax safety at work was tolerated. Not anymore. So it's not like there is some undeniable inherent right of an employeer to "have the employee pee in a cup" or to "check his Facebook profile". It's just that people haven't protested enough to make it into law that it's not his fucking business what the employee does at his own time. >It's best to let the worker -- the HN reader in this case -- make the decision whether he/she wants to pick that fight, rather than have them accidentally stumble into it. Sure, but at least some anger should be directed against businesses having those policies, not just on HN titles, as if the policies are OK.
- nacho2sweet 13y agoLOL at everyone who works at places that fear that the company firewall saw them read an article that had the word "Porn" in it. Hopefully you get in trouble and fired, it will help your life in the long run.
- serginho 13y agoI've never expected that the story would be about porn.
- ianhawes 13y agoIf anyone is interested in an alternative to the usual RTMP servers (FMS, Red5, and Wowza), I highly recommend EvoStream (http://www.evostream.com http://www.evostream.com). Compared to the alternatives, EvoStream is much more efficient. I believe TinyChat published a whitepaper discussing their transition from Red5 to EvoStream, which resulted in a decline in the number of required servers. EvoStream is a highly scalable streaming media server written in C++ based off of the open source RTMPD (http://www.rtmpd.com http://www.rtmpd.com). The commercial company, also called EvoStream, is a relatively new startup and they do great custom work for those not familiar with streaming media/RTMP.
- deleted 13y ago[deleted]
- coherentpony 13y agoCongratulations to whoever messed with the title; I clicked this at work.
- mariuolo 13y agoWhy is the type of content relevant? Perhaps there's something peculiar about porn users?
- protomyth 13y agoBecause, like it or not, the porn industry dealt with a lot of the technical issues of selling video, pictures, and video streams on the internet before anyone else. They needed it to be profitable / have actual revenue.
- Killswitch 13y agoNothing. I can talk about my work on a blog or to a technical person without mentioning once that it's porn related and I've been a programmer in the porn industry for over 10 years. They used it as a way to gain viewers.
- latchkey 13y agoAs someone who built the infrastructure for serving porn for Kink.com, I'd say that this was a total waste of time. Spend the money on a third party CDN and serve from there.
- swalsh 13y agoI've always been curious, how is working for a porn company affected your social life? does it at all?
- latchkey 13y agoI worked behind the camera, not in front of it. Sorry, I'm not interested in satisfying your curiosities.
- iso8859-1 13y agoWith these amounts of bandwidth, does that still make sense? Sounds like this was a one-man job that worked.
- latchkey 13y agoI'm not really sure what you mean, but absolutely using a CDN product is the way to go when you want to sell your content on a global scale. Otherwise, you are going to deal with having to build / maintain your own and the cost of doing that is far higher than just letting other experts deal with the issues that surround it.
- kibibu 13y agoThe article suggests that this is for live streaming shows. Would a CDN based approach work in this use case?
- latchkey 13y agoTake a look at KinkLive.com. We were the first porn company to do live streaming in HD using a CDN (Bitgravity), all paid for, by the minute, with a micro currency system (kinks) that we built.
- deleted 13y ago[deleted]