4 ms·
(My background: I work for Google, I did a real-time web prototype using the client libraries for GChat back in 2009 when real-time search was all the rage, my
by nostrademons 13y ago
(My background: I work for Google, I did a real-time web prototype using the client libraries for GChat back in 2009 when real-time search was all the rage, my Noogler mentor at Google was the frontend tech lead for the eventual real-time search product we launched, and before Google I'd worked in financial software, where real-time responsiveness really is required.)
I think that the folks currently building prototypes in Meteor dramatically underestimate the difficulty of scaling up real-time software to production-grade quality.
The problem is that if a single component in your stack blocks, you are no longer real-time. Any time one client writes into the database and another reads it, you have to poll, since the DB won't give you notifications. (Exception: PostGres gives you PQnotifies, Oracle gives you the User Messaging Service, MySQL it's theoretically possible with triggers and user-defined stored procedures that make a network call, and MongoDB you can break the DB abstraction and tail the oplog. Good luck plumbing any of these up through your language DB driver and ORM, though.) If you have business logic in a middle-tier server that's request-response only, then that logic becomes a synchronization bottleneck, and you have to constantly update that server and poll it with requests. If your algorithms require complete state snapshots, you're out of luck unless you build a service to manage and update that state consistently while triggering the algorithms whenever it changes. If your algorithms can't run in soft-realtime time guarantees (dozens to hundreds of milliseconds, usually), you're still out of luck. You need to figure out sharding of state and message notifications yourself. You need to figure out message recovery protocols - most real-time systems have odd consistency problems when messages get dropped due to overload, network failures, or software errors.
Google's real-time search ended up polling every 15 seconds with simple AJAX calls, because when the lag for a post to go through the indexing & serving pipeline is a minute or two (itself a major accomplishment), an additional 15 seconds isn't going to be noticeable to the user.
People on HN love to hate on Twitter engineering, but one thing they've done really well is scale a system that actually is soft real-time and has a lot of potential producers and consumers. This is far from the trivial exercise that someone who picked up Meteor in a weekend might think it is.
- recuter 13y agoI think the twitter-hate was usually because Twitter started with a naive Rails approach on Joyent and would fail whale constantly in the early years. Just recently they blogged how they finally completed a rewrite that improved per node performance to 100,000 messages per second instead of a few hundred - a better than expected result. Its just a bit OCD inducing how they had to solve problems of their own creation for a long time instead of "doing it right from the start". I offer no opinion whether or not they were being smart and pragmatic or incompetent hipsters in the beginning. Edit - Found it, here's the post: https://blog.twitter.com/2013/new-tweets-per-second-record-and-how https://blog.twitter.com/2013/new-tweets-per-second-record-a...
- matthewmacleod 13y agoIts just a bit OCD inducing how they had to solve problems of their own creation for a long time instead of "doing it right from the start" That's a really, really silly thing to get annoyed about, and flies straight in the face of lean, agile and MVP. I'd wager that if Twitter had "done it right from the start" then they wouldn't be around at all today.
- recuter 13y agoI don't disagree.
- rbanffy 13y ago> and flies straight in the face of lean, agile and MVP It all depends on how complex is doing something right from the start, and how right must it be done for you to stay out of trouble. Twitter had a really naïve approach when it started. Had they started with a slightly better (not necessarily harder) approach, they'd have had far less problems at a much later time. MVP has a "V" in the middle. Twitter only survived because their V was extremely flexible.
- siliconc0w 13y agoDjango's event system lets you avoid polling the DB. Here is a plugin that offers some real-time capability (through a separate process of course): http://telegraphy.readthedocs.org/en/latest/django-telegraphy.html http://telegraphy.readthedocs.org/en/latest/django-telegraph... Seems to be the best thing on offer that is off the shelf but you may be better rolling your own node, websocket, redis pub/sub, django integration yourself(which actually isn't that hard to do and may give you better flexibility).
- nostrademons 13y agoIf I understand the Telegraph docs correctly, it offers some event-based capability if you interact the app through your Django models exclusively. If, say, you've got a crawler that's out there discovering new content on the web, or a chatbot that listens on IRC, or another app that interacts with the DB via a different framework, you're pretty much out of luck, unless you also write those components to use Telegraph and Django models. That's my point about every part of your system needing to be async-aware. Most off-the-shelf open-source components that people use to build websites now are not. Ironically I think we were in a better position in this respect in the early 2000s with JMS, Jini, JavaSpaces, CORBA, RMI, etc. but all of those fell by the wayside because of justifiable concerns about programming complexity. Now the latest generation of web developers is discovering async soft real-time programming again, and soon we will again discover those concerns about programming complexity.
- est 13y ago> Django's event system lets you avoid polling the DB. Django's signal system must be run on the same instance, synchronously, before returning any response to client. This means if a signal blocks, the response will never reach the client. It's technically impossible to handle post-response events in WSGI. http://dirtsimple.org/2011/07/wsgi-is-dead-long-live-wsgi-lite.html http://dirtsimple.org/2011/07/wsgi-is-dead-long-live-wsgi-li... For example, user requests a blog post, you can't first return response then increase your view counter by 1. You just can't. Unless you use some container specific hooks (like uWSGI, Tornado, etc.)
- 13y ago
- joshowens 13y agoScaling anything is hard to achieve, especially when you are also trying to build a business around an idea. Scaling is usually a problem you solve after you become more popular. My original article about Meteor and Rails wasn't about handling twitter-esque type traffic.
- sylvinus 13y agoFrom what I understood, Meteor does tail the MongoDB oplog.
- jafaku 13y agoHow do you know someone works for Google? They will tell you.
- grovulent 13y agoThis is unnecessarily snarky. Sure, there is probably a status element in the declaration - but it also confers authority to their claims, whether you like it or not. This is handy for folks skimming comments to look for quality contributions.
- hmsimha 13y agoThe grandparent wasn't even referencing working at Google for a status boost, but merely because it directly related to their diatribe (their experience with implementing a Google feature and how that relates to scalability issues that Meteor fails to address)