3 ms·

I laughed, but this is exactly how many of the answers here sound to me. Answers like this (even though I'm assuming yours was sarcastic) exemplify everything

I laughed, but this is exactly how many of the answers here sound to me. Answers like this (even though I'm assuming yours was sarcastic) exemplify everything that I outlined above. The "you" stated below is in the general form, not you specifically. It is the "you" intended to represent people who would seriously respond to a question like this. You don't even know if a database is our bottleneck (it's not), you don't know what database/datastore we're using (it might be redis), you represent no experience in our problem domain, you have no detailed information about our current performance, but you have no hesitation about positing an answer. Sorry to be so blunt, but uh... "Get off my lawn, you damn kids!" :)

gridspy 15y ago
Just to get on your lawn for a moment. For Gridspy, we store data into our server from sensors and allow customers to watch a 1-second live data stream. http://your.gridspy.co.nz/demo/ http://your.gridspy.co.nz/demo/ The key from us was to switch from a "polling" model to a "push" model. We don't poll the database, our backend pushes the new data through the stack. In our case I plan to actively rebuild assets and store them to disk rather than rebuild and cache on demand. But you can see the live updates on the site now. More details (a bit dated now): http://blog.gridspy.co.nz/2009/10/realtime-data-from-sensors-to-browsers.html http://blog.gridspy.co.nz/2009/10/realtime-data-from-sensors... You might consider an approach such as http://socket.io/ http://socket.io/ and lots of logic in the page as a great way to avoid the "everyone polling the backend as fast as they can" model. By the way, your lawn is very nice. Well maintained. Perhaps you need a zen garden.
- bradleyland 15y ago
  I appreciate the thoughtful response :) Polling actually works really well for us though. It's like rudimentary time-division multiplexing for sockets. Push means you have to keep an open socket to clients, which has its own set of scaling problems. Our challenges are related to cache design complexity, fast cache rebuild, serialization, and queuing & distribution of tasks without sacrificing timeliness of response. There's a lot of pressure for us to deliver a specific subset of the data in the same request/response cycle, which can be challenging. These are all pretty well known problems in computer science, and we feel pretty confident that we've got solutions for them. I'm definitely going to keep an eye on your blog though. It's nice to keep in touch with companies that face similar challenges.
  - gridspy 15y ago
    Cool! I agree on the "good to know people with similar problems" I can see why aggregation of the data can be such a challenge for you. My suggestion revolves around a smart client that can do aggregation through (for eg) javascript on in a browser. Our system is actually polling under the hood (that is how "Comet" works) but our backend never sees the polls. You have a good point about open sockets being another problem. Fortunately it is thoroughly solved by messaging servers, for instance http://www.rabbitmq.com/ http://www.rabbitmq.com/ - it handles tons of connections so your backend doesn't have to. Your requirements are a great fit for a single threaded server such as node.js, twisted or so on that uses that whole polling model rather than many threads. You could have (say) a twisted server that receives messages from the backend on rabbitmq, interprets and aggregates and then forwards to your clients. The send to clients happens via another publish of a message to RabbitMq on a different channel and is un-affected by the number of users watching. But like you said, solved problems are not worth a whole bunch of engineering effort when things are still running just fine. Just throw more hardware at it and move on to selling what you have. :)
  - StavrosK 15y ago
    There are services you can use to push data (I've used Pusher a bit and like them, no affiliation), and it sounds like it will solve most of your problems, since it will get your request rate way, way down. The main benefit, though, is that your problem is trying to emulate pushing data via polling, hence all the cache problems, etc. Imagine the reformulated problem: You have bidders sending you their bids, and you send N notifications 0.5 times a second. That sounds like a much easier problem, rather than get N requests a second and having to give them correct data. You can integrate Pusher in five minutes or so. Give it a shot, and if you like it/it solves your problem, you owe me a beer. However, I only know three paragraphs' worth of your problem, so it's just a suggestion.
moe 15y ago
Actually I wasn't sarcastic. Having built similar sites in the past I'd be really curious what your bottleneck is, if not the database?
- bradleyland 15y ago
  So, I should just throw Redis at it? Then it will be ROFLScale and I can sleep at night? I'm trying not to be a complete prick about this, but how could you even begin to make the assumption that our database is our bottleneck without knowing a single thing about our application. How do you even know we're not using Redis already? Our bottleneck is the computational complexity of our datasets. The nature of reverse auctions is that the dataset changes frequently. The fact that every bidder is ranked on a per line-item basis means there are cache interdependencies, which cause tidal waves of cache expiration when bids are placed. Adding difficulty, there is a lot of pressure to deliver a subset of the data in the same request/response cycle, so that bidders aren't waiting in limbo for their rank. This helps maintain bidder momentum. We don't currently have any scaling issues that we haven't solved. That's not the point. The point is that suggesting blanked solutions to problems you don't understand is major Dunning–Kruger territory.
  - moe 15y ago
    So, I should just throw Redis at it? Then it will be ROFLScale and I can sleep at night? Yes. but how could you even begin to make the assumption that our database is our bottleneck Because everything else in your app is parallelizable. How do you even know we're not using Redis already? Because you just asked about it a line above. tidal waves, subset of the data in the same request/response, dataset changes frequently, cache interdependencies And because of that. If you were using redis for state then none of that would be worth mentioning.

gridspy 15y ago

Just to get on your lawn for a moment. For Gridspy, we store data into our server from sensors and allow customers to watch a 1-second live data stream. http://your.gridspy.co.nz/demo/ http://your.gridspy.co.nz/demo/ The key from us was to switch from a "polling" model to a "push" model. We don't poll the database, our backend pushes the new data through the stack. In our case I plan to actively rebuild assets and store them to disk rather than rebuild and cache on demand. But you can see the live updates on the site now. More details (a bit dated now): http://blog.gridspy.co.nz/2009/10/realtime-data-from-sensors-to-browsers.html http://blog.gridspy.co.nz/2009/10/realtime-data-from-sensors... You might consider an approach such as http://socket.io/ http://socket.io/ and lots of logic in the page as a great way to avoid the "everyone polling the backend as fast as they can" model. By the way, your lawn is very nice. Well maintained. Perhaps you need a zen garden.

bradleyland 15y ago
I appreciate the thoughtful response :) Polling actually works really well for us though. It's like rudimentary time-division multiplexing for sockets. Push means you have to keep an open socket to clients, which has its own set of scaling problems. Our challenges are related to cache design complexity, fast cache rebuild, serialization, and queuing & distribution of tasks without sacrificing timeliness of response. There's a lot of pressure for us to deliver a specific subset of the data in the same request/response cycle, which can be challenging. These are all pretty well known problems in computer science, and we feel pretty confident that we've got solutions for them. I'm definitely going to keep an eye on your blog though. It's nice to keep in touch with companies that face similar challenges.
- gridspy 15y ago
  Cool! I agree on the "good to know people with similar problems" I can see why aggregation of the data can be such a challenge for you. My suggestion revolves around a smart client that can do aggregation through (for eg) javascript on in a browser. Our system is actually polling under the hood (that is how "Comet" works) but our backend never sees the polls. You have a good point about open sockets being another problem. Fortunately it is thoroughly solved by messaging servers, for instance http://www.rabbitmq.com/ http://www.rabbitmq.com/ - it handles tons of connections so your backend doesn't have to. Your requirements are a great fit for a single threaded server such as node.js, twisted or so on that uses that whole polling model rather than many threads. You could have (say) a twisted server that receives messages from the backend on rabbitmq, interprets and aggregates and then forwards to your clients. The send to clients happens via another publish of a message to RabbitMq on a different channel and is un-affected by the number of users watching. But like you said, solved problems are not worth a whole bunch of engineering effort when things are still running just fine. Just throw more hardware at it and move on to selling what you have. :)
- StavrosK 15y ago
  There are services you can use to push data (I've used Pusher a bit and like them, no affiliation), and it sounds like it will solve most of your problems, since it will get your request rate way, way down. The main benefit, though, is that your problem is trying to emulate pushing data via polling, hence all the cache problems, etc. Imagine the reformulated problem: You have bidders sending you their bids, and you send N notifications 0.5 times a second. That sounds like a much easier problem, rather than get N requests a second and having to give them correct data. You can integrate Pusher in five minutes or so. Give it a shot, and if you like it/it solves your problem, you owe me a beer. However, I only know three paragraphs' worth of your problem, so it's just a suggestion.

moe 15y ago

Actually I wasn't sarcastic. Having built similar sites in the past I'd be really curious what your bottleneck is, if not the database?

bradleyland 15y ago
So, I should just throw Redis at it? Then it will be ROFLScale and I can sleep at night? I'm trying not to be a complete prick about this, but how could you even begin to make the assumption that our database is our bottleneck without knowing a single thing about our application. How do you even know we're not using Redis already? Our bottleneck is the computational complexity of our datasets. The nature of reverse auctions is that the dataset changes frequently. The fact that every bidder is ranked on a per line-item basis means there are cache interdependencies, which cause tidal waves of cache expiration when bids are placed. Adding difficulty, there is a lot of pressure to deliver a subset of the data in the same request/response cycle, so that bidders aren't waiting in limbo for their rank. This helps maintain bidder momentum. We don't currently have any scaling issues that we haven't solved. That's not the point. The point is that suggesting blanked solutions to problems you don't understand is major Dunning–Kruger territory.
- moe 15y ago
  So, I should just throw Redis at it? Then it will be ROFLScale and I can sleep at night? Yes. but how could you even begin to make the assumption that our database is our bottleneck Because everything else in your app is parallelizable. How do you even know we're not using Redis already? Because you just asked about it a line above. tidal waves, subset of the data in the same request/response, dataset changes frequently, cache interdependencies And because of that. If you were using redis for state then none of that would be worth mentioning.