22 ms·
Disqus: Scaling the World’s Largest Django Application (2010) [video]
- hahainternet 14y agoWhat struck me instantly was the use of Slony. I haven't listened to the whole thing yet but I am interested in their justification. Perhaps they just haven't moved to 9.2 yet.
- d0ugal 14y agoI don't think they could move to 9.2 in 2010 ;)
- sugarcode 14y agoslony offers some compelling advantages over streaming replication - even on new databases we setup, we still like slony for several reasons: * Version upgrades (streaming replication requires the same PG version between master/slave, slony does not) * Logical replication gives us finer-grained control over how data is replicated across the cluster * Ability to create additional indexes on slaves slony isn't perfect and it has caused us some headaches, but its flexibility makes it our go-to replication tool for postgres.
- lewispb 14y agoLargest? Is Disqus really 'larger' than Instagram?
- simonw 14y agoI think it was in 2010 when they gave this talk.
- saevarom 14y agoYeah these figures are really outdated. This one from PyCon 2011 is newer: http://www.slideshare.net/zeeg/pycon-2011-scaling-disqus-7251315 http://www.slideshare.net/zeeg/pycon-2011-scaling-disqus-725...
- tangue 14y agoInteresting : According to these slides [14] there is no difference between Apache + mod_wsgi and Nginx + uWSGI
- mibbitier 14y agoAlso "large" is fairly meaningless when you don't include a measure of complexity. Sharing images is a fairly simple task.
- stef25 14y agoPainful spam in the disqus comments at the bottom of that page. Don't they have something in place against this?
- beaumartinez 14y ago"+1"s are hardly spam, they're just low quality comments. If they filtered those, the web would be a very commentless place
- d0ugal 14y ago+1
- kordless 14y agoAt least they haven't translated the slug into Russian. Every time I get email from them about a comment on my post, it shows up translated into Russian.
- zerop 14y agoAll with apache and haproxy? No nginx or uwsgi/gunicorn or Redis? How old is this article?
- ihsw 14y agoSept. 7 2010, according to their slideshare link. http://www.slideshare.net/zeeg/djangocon-2010-scaling-disqus?from=ss_embed http://www.slideshare.net/zeeg/djangocon-2010-scaling-disqus...
- recuter 14y agoIn the newer (2011) slides they mention growth by ~100 Million comments in 6 months. Most comments are tiny, I'd be surprised if this whole dataset was much larger then ~100GB. I don't really understand why Disqus and Reddit for that matter don't just switch to redis. There has got to be a good reason because I can't think of one. Sounds like you could run the whole thing from just one box and only have slaves for redundancy. Why isn't vertical scaling more in vogue?
- lamby 14y agoBecause people like things like transactions and are generally risk-adverse when choosing storage technologies?
- bretthoerner 14y agoIf you really think that's the case then you don't understand Disqus/Reddit OR Redis.
- recuter 14y agoWell I opened with "I don't understand". I'm not saying that they could, I'm saying I don't understand why they can't. I'd love for you to elaborate. As far as I can tell they are not using PostgreSQL as a relational database, but rather, as a column store. So why not use Cassandra or even Redis (if the amount of data can totally fit into RAM easily, maybe it can't)? In fact I think Reddit moved to Cassandra... anyway, I am not an expert, I'm asking.
- Kilimanjaro 14y agoUsing django for a distributed commenting system? Hammer and screws.
- deleted 14y ago[deleted]
- legutierr 14y agoJust wondering (I'm not necessarially objecting to your statement): * what would you see as the right they of app for Django, if not this? Obviously Django is handling it, so what's the issue? * what is the right tool to build a distributed comment system?
- Kilimanjaro 14y agoWhat is a distributed commenting system? An HTML script tag to load a static file that will ping an api that will serve you comments. What is django? A framework with strong routing, OR-mapping, templates and admin modules. You need none to build it. Your main concern shouldn't be a coding framework, but load balancing, caching, failover control, more sysadmin stuff than code. Any language could do. Framework? not needed. Now, for the backend system to control that monster, then yes, you may use django. So, use django for complex apps that need routing, data management and UI presentation, plus a powerful admin module. * I am a python/django developer.
- tkaemming 14y ago> What is a distributed commenting system? An HTML script tag to load a static file that will ping an api that will serve you comments. This is how Disqus works. Something still has to power the API that serves the data for it and we happen to like Django a lot so that's what we use. There are also a lot of parts of Disqus that are not the embed (moderation panel, account management, etc.)
- Kilimanjaro 14y agoExactly, that's my point. Not meant to demerit your great work. The admin/crud stuff, moderation panel, account management, etc, that's what django was developed for. Great choice. If you ask me to start Disqus from scratch again, I'd probably use django too, for the admin part, but for the API I'd go commando, closer to the metal, without framework at all. No need to load a 100MB routing/modeling/templating monster just to perform an invisible API call. Highly optimized plain python scripts would work better. Sometimes frameworks are more like handcuffs. Just sometimes.
- TommyDANGerous 14y agoGreat watch.
- zeeg 14y agoAs these slides are very old, here's some updates: * We use Flask and nginx in various areas now (the main app is still Django). Our realtime app, for example, is powered off of uwsgi and Flask. * There are nearly 1b monthly uniques across the network serviced by the platform. * ~300 servers * Still Postgres (with Slony, multiple clusters), Redis, Memcache, and some Cassandra for newer things (not comments). Also mostly confident we're still the "largest Django app" in terms of traffic.
- recuter 14y agoInteresting. Care to update on the other stats like requests at peak? Also, I wonder - if you're at 1b uniques is there anywhere to grow from here other then more engagement? Oh, also: Would you say the 1% contribute, 9% curate, 90% consume heuristic holds true in your experience at this scale or is it even more skewed?
- reinhardt 14y agoVirtual servers or bare metal? What do you use for herding them?
- lucian303 14y agoI second that. Would be interesting to know a bit more about the systems setup. Also why the decision to use an ORM? To manage the partitions?
- zeeg 14y agoEventually you need abstraction. Python is fast enough, as is Django, that the abstraction costs us less than the value it provides us. That said, we do manage our partitions (currently) using an ORM layer (at the application level). We do want to break this out into a proxy middleware at some point though.
- lucian303 14y agoRight, just interested in the choose of such an anti pattern. What proxies are you considering?