16 ms·
Why did OKCupid write their own web server?
- ErrantX 16y agoThe questioned is phrased very much in the now e.g. Isn't the technology stack basically a commodity at this point? and And the answer highlights why you should never retrospectively "judge" design choices several years after the fact.
- cheez 16y agoI've had an opportunity to chat with some people tangentially related to OKC and I think that they have definitely done some very cool stuff. There are some things that the OKWS architecture does very well. As I understand it, there is a bit of "Rails envy" but they seem to copy good ideas very quickly. That being said, I think that if they were to start again, they would try and use commodity technology. But, these guys are really fucking brilliant and productive. Immensely... I feel like a chump in comparison.
- cschep 16y agoYes. How often do I say, "I wish this thing ____" or.. "oh that's just not possible." They just decided to write a bunch of c++ to make it possible. Awesome!
- rubashov 16y agoWell these days writing an http server in C++ is about 30 lines of code using POCO or boost.asio. If your business model was "serve a shit-ton of dynamic requests super cheap" it might make just as much sense as ever to build the whole thing in C++ embedded directly in a C++ webserver.
- gdulli 16y agoOKCupid is the only web site I can think of that has a regular, not occasional, pattern of being very slow and simply not responding to an unusual number of requests. Hitting F5 to reload a page just to get it to show up instead of the Firefox server unavailable message is a regular part of my usage of the site. Even though all sites have bugs, broken links, what have you, I don't know any other site that's given me such an expectation that it will be unresponsive for a significant number of page views for any given session over a long term period. Even the sites that started development circa 2003.
- tobias3 16y agoIf you have a small team and everybody has much C++ experience you can pull this off. Otherwise one person who doens't have the discipline to do the manual memory management right can crash the whole server. Don't try it at home ;) use an VM-language instead which can recover from such errors. (I wrote some C++ webapps myself)
- maxtaco 16y agoManual memory management is for the birds. Use something like Boost's shared pointers and you never need to worry about it.
- api 16y agoYou can also often make clever use of RAII and STL container classes and do memory management that way.
- Peaker 16y agoThat's not entirely honest. A) You still have to worry about it when interfacing with libraries that use plain pointers B) Shared pointers incur runtime penalties (larger data pushes things off the cache, spurious inc/dec-refs messing said cache. If you don't care about that, why do you use a language like C++ in the first place? C) Reference counting is a poor form of automatic memory management, you still have to worry about cycles, and use weak references or such to break the cycles.
- Johngibb 16y agoI'd think they'd begin transitioning to a higher level language now that there are many options available. It's gotta be a burden at this point to be (1) maintaining their own web server and (2) developing new features in C++. I'd way rather use ruby/python (or even .Net) and fall to C++ for the really performance intensive stuff. (Disclaimer: I interviewed @ OKCupid in 2007)
- dustingetz 16y ago"general rpc servers for solving specific problems using in memory data structures (e.g., who qualifies for a match search given dozens of constraints and millions of users; what your match score is with 10,000 qualifying people, given you've all answered hundreds of different questions each on average) ... Great tech is available now, serving is cheaper, and you probably don't have the computational workload OkCupid does." I know people who have used OKC before. OKC users in my social class (male, white, educated) ignore the match percentages, because the SNR is really low. They just plow through all the search results of people to find good pictures and interesting profiles. So, I'd speculate that match-percentages are a marketing thing, and that they know they made a weak business decision which required lots of computation and now they're stuck with it. I'm probably wrong. Maybe the long-tale users pay attention to match-percentage.
- gfunk911 16y agoI don't ignore the match percentages at all. Obviously they aren't perfectly correlated, but they're pretty solid. People with very low match numbers are basically 100% duds.
- starpilot 16y agoYou probably are wrong, and you imply you've never used okc. Match percentages matter to me highly since they certainly indicate users I'd get along with, based on my interactions with other users. Their data also show that messages response rates correlate strongly with match percentages.
- neild 16y agoI signed up for OKC, lurked for a while, and then sent a message to the person at the top of my match list. We're still together, five years later. I'm probably an outlier, but hey--match percentage works some of the time!
- DrStalker 16y agoI think of the match percentage as the equivalent of having HR look at tech resumes before forwarding them on to me; it's not perfect, but it saves me from reading through completely irrelevant profiles.
- tseabrooks 16y agoMy only real complaint about the nature of HN is the (as perceived by me) strong anti-C++ bent, like the ones in these comments, from what seems like people who don't have lots of experience using it in the work place. Am I imagining this anti-C++ bias? Am I wrong and all the people bashing C++ have tons of C++ experience? I'll be the first to point out the numerous flaws C++ has but it just feels like folly to make fun of the ugly chick at the party without realizing everyone at the party is covered in warts... (The is a metaphor for all programming languages having problems)
- tptacek 16y ago(I'm a recovering C++ dev). C++ has problems unique to C++: a deceptive illusion of abstraction (what Spolsky would call "leaky" abstractions) makes a bunch of idioms in the language dangerous, including virtually all "smart" pointers, iterators, exceptions, allocators, and arrays. It is uniquely difficult to write reliable code in C++. Add to that the superficial but more practical and common complaints against C++: the ghastly compile times, the header file dependency hell that forces every class into a kabuki dance of "pImpls" and nested classes, the error messages rendered in ancient Sumerian... you know I could go on, but you get the idea. C++ is just not a very good language. C is a fine language. If you are building a system for which C's abstractions are inadequate --- and I'll stipulate that such systems exist --- you're better off with C and a very good glue language (Lisp and Lua are two good, popular choices) than you are with a uniform C++ implementation.
- btrask 16y ago"C++ is a powerful language with flexible solutions to problems that other languages don't have."
- statictype 16y ago"C++: An octopus formed by nailing extra legs to a dog"
- TillE 16y ago> C++ is just not a very good language. C is a fine language. This is the attitude that I really, truly do not get. If I want a linked list or any other common data structure in C++, I include the appropriate STL header and declare an object with the template syntax. It's two easy lines of code. If I want a data structure in C, I have to write it myself. From scratch. Every time. C++ is a powerful high-level language. C is a low-level language where everything is explicit; I know this is "elegant" to many programmers, but really, I'm just trying to get things done, and C++ has served me far better in that practical role than C ever could. I don't love C++; it has tons of flaws (many of which disappear if you use shared_ptr for everything). But I loathe the tedium of working in C.
- ibejoeb 16y agoWe've actually had this conversation before. It sounds ridiculous, but the story is actually a little different: OKWS was built prior to OKCupid, and they inherited it. See http://pdos.csail.mit.edu/~max/docs/okws.pdf http://pdos.csail.mit.edu/~max/docs/okws.pdf.
- mhp 16y agoThat doesn't sound like the answer the founder of OKCupid posted here: http://answers.onstartups.com/questions/21323/why-would-okcupid-write-their-own-web-server/21377#21377 http://answers.onstartups.com/questions/21323/why-would-okcu...
- ibejoeb 16y agoWell, I wasn't there, so I don't really know. But here is the previous thread, where I joined in dismissing the premise: http://news.ycombinator.com/item?id=2078210 http://news.ycombinator.com/item?id=2078210. See the two responses for details.
- nostrademons 16y agoI could swear I remember, back when they were TheSpark, them claiming that the TheSpark ran on a custom webserver written in C++. That would be consistent with the idea that OKWS was done as Max's Harvard thesis project and then inherited by TheSpark and OKCupid later. There are a variety of reasons why startup founders may want to bend the truth with their public statements.
- jacques_chester 16y agoI remember this coming up at reddit a few months ago[1]. At the time I downloaded and read the paper on the design[2]. It all made sense to me because my own thinking had been heading in the same direction. OKWS is less a web server than it is an architecture of servers. It's the difference between sendmail and qmail/postfix. It has nice security and performance properties because each service is run as a separate user, with a separate process. Logging is handled by an independent daemon. Request demultiplexing is handled by a simple daemon that binds to port 80. Actual HTTP parsing is handled by a shared library that services link to. [1] http://www.reddit.com/r/programming/comments/exkk3/ok_webserver_from_ok_cupid_the_devs_at_a_dating/ http://www.reddit.com/r/programming/comments/exkk3/ok_webser... [2] http://pdos.csail.mit.edu/~max/docs/okws.pdf http://pdos.csail.mit.edu/~max/docs/okws.pdf
- guelo 16y agoI imagine one big downside of this custom stack is that they will have a hell of a time doing any sort of integration with Match.com. Hiring and training is also probably more difficult, though that has got be a huge boon to OKCupid engineers since Match cannot afford to lose them.
- mjs 16y agoHa, it gets better: the web server "was partially funded by the DARPA Composable High Assurance Trusted Systems program." http://www.okws.org/doku.php?id=okws:sponsorship http://www.okws.org/doku.php?id=okws:sponsorship
- mkjones 16y agoWait, the OKWS web site runs on apache?
- starpilot 16y agoThat was explained in a FAQ previously on the site. OKWS is meant for serving highly dynamic content, while they feel Apache is better at static and more modest dynamic content. Their original philosophy was that segregating dynamic and static serving improved security, stability, and speed, though that's mostly been nullified by cheaper hosting.
- itsnotvalid 16y agoAny languages could possibility be made to work. They did make it to work with things like SFSLite[1] which looked like coroutines or fibers (actually Stratified JavaScript, but that is not something common) would solve for async callbacks. However, those are something that is acting as extensions to the core language. One of the biggest problem C++ has is the fact that core language has too many stuff but still lacking things that people really want to use. It's certainly workable, and the results are fast since it is compiled very well. However 'workable' does not mean 'a pleasure to work'. [1]: http://www.okws.org/doku.php?id=sfslite:tame2:tutorial http://www.okws.org/doku.php?id=sfslite:tame2:tutorial , http://www.okws.org/doku.php?id=sfslite http://www.okws.org/doku.php?id=sfslite