4 ms·

Just to get on your lawn for a moment. For Gridspy, we store data into our server from sensors and allow customers to watch a 1-second live data stream. http:

by gridspy 15y ago

Just to get on your lawn for a moment. For Gridspy, we store data into our server from sensors and allow customers to watch a 1-second live data stream. http://your.gridspy.co.nz/demo/ http://your.gridspy.co.nz/demo/ The key from us was to switch from a "polling" model to a "push" model. We don't poll the database, our backend pushes the new data through the stack. In our case I plan to actively rebuild assets and store them to disk rather than rebuild and cache on demand. But you can see the live updates on the site now. More details (a bit dated now): http://blog.gridspy.co.nz/2009/10/realtime-data-from-sensors-to-browsers.html http://blog.gridspy.co.nz/2009/10/realtime-data-from-sensors... You might consider an approach such as http://socket.io/ http://socket.io/ and lots of logic in the page as a great way to avoid the "everyone polling the backend as fast as they can" model. By the way, your lawn is very nice. Well maintained. Perhaps you need a zen garden.

bradleyland 15y ago
I appreciate the thoughtful response :) Polling actually works really well for us though. It's like rudimentary time-division multiplexing for sockets. Push means you have to keep an open socket to clients, which has its own set of scaling problems. Our challenges are related to cache design complexity, fast cache rebuild, serialization, and queuing & distribution of tasks without sacrificing timeliness of response. There's a lot of pressure for us to deliver a specific subset of the data in the same request/response cycle, which can be challenging. These are all pretty well known problems in computer science, and we feel pretty confident that we've got solutions for them. I'm definitely going to keep an eye on your blog though. It's nice to keep in touch with companies that face similar challenges.
- gridspy 15y ago
  Cool! I agree on the "good to know people with similar problems" I can see why aggregation of the data can be such a challenge for you. My suggestion revolves around a smart client that can do aggregation through (for eg) javascript on in a browser. Our system is actually polling under the hood (that is how "Comet" works) but our backend never sees the polls. You have a good point about open sockets being another problem. Fortunately it is thoroughly solved by messaging servers, for instance http://www.rabbitmq.com/ http://www.rabbitmq.com/ - it handles tons of connections so your backend doesn't have to. Your requirements are a great fit for a single threaded server such as node.js, twisted or so on that uses that whole polling model rather than many threads. You could have (say) a twisted server that receives messages from the backend on rabbitmq, interprets and aggregates and then forwards to your clients. The send to clients happens via another publish of a message to RabbitMq on a different channel and is un-affected by the number of users watching. But like you said, solved problems are not worth a whole bunch of engineering effort when things are still running just fine. Just throw more hardware at it and move on to selling what you have. :)
- StavrosK 15y ago
  There are services you can use to push data (I've used Pusher a bit and like them, no affiliation), and it sounds like it will solve most of your problems, since it will get your request rate way, way down. The main benefit, though, is that your problem is trying to emulate pushing data via polling, hence all the cache problems, etc. Imagine the reformulated problem: You have bidders sending you their bids, and you send N notifications 0.5 times a second. That sounds like a much easier problem, rather than get N requests a second and having to give them correct data. You can integrate Pusher in five minutes or so. Give it a shot, and if you like it/it solves your problem, you owe me a beer. However, I only know three paragraphs' worth of your problem, so it's just a suggestion.

by gridspy 15y ago

bradleyland 15y ago

I appreciate the thoughtful response :) Polling actually works really well for us though. It's like rudimentary time-division multiplexing for sockets. Push means you have to keep an open socket to clients, which has its own set of scaling problems. Our challenges are related to cache design complexity, fast cache rebuild, serialization, and queuing & distribution of tasks without sacrificing timeliness of response. There's a lot of pressure for us to deliver a specific subset of the data in the same request/response cycle, which can be challenging. These are all pretty well known problems in computer science, and we feel pretty confident that we've got solutions for them. I'm definitely going to keep an eye on your blog though. It's nice to keep in touch with companies that face similar challenges.

gridspy 15y ago
Cool! I agree on the "good to know people with similar problems" I can see why aggregation of the data can be such a challenge for you. My suggestion revolves around a smart client that can do aggregation through (for eg) javascript on in a browser. Our system is actually polling under the hood (that is how "Comet" works) but our backend never sees the polls. You have a good point about open sockets being another problem. Fortunately it is thoroughly solved by messaging servers, for instance http://www.rabbitmq.com/ http://www.rabbitmq.com/ - it handles tons of connections so your backend doesn't have to. Your requirements are a great fit for a single threaded server such as node.js, twisted or so on that uses that whole polling model rather than many threads. You could have (say) a twisted server that receives messages from the backend on rabbitmq, interprets and aggregates and then forwards to your clients. The send to clients happens via another publish of a message to RabbitMq on a different channel and is un-affected by the number of users watching. But like you said, solved problems are not worth a whole bunch of engineering effort when things are still running just fine. Just throw more hardware at it and move on to selling what you have. :)
StavrosK 15y ago
There are services you can use to push data (I've used Pusher a bit and like them, no affiliation), and it sounds like it will solve most of your problems, since it will get your request rate way, way down. The main benefit, though, is that your problem is trying to emulate pushing data via polling, hence all the cache problems, etc. Imagine the reformulated problem: You have bidders sending you their bids, and you send N notifications 0.5 times a second. That sounds like a much easier problem, rather than get N requests a second and having to give them correct data. You can integrate Pusher in five minutes or so. Give it a shot, and if you like it/it solves your problem, you owe me a beer. However, I only know three paragraphs' worth of your problem, so it's just a suggestion.