3 ms·
A couple things strike me about the rebuttals here: The illusion of simplicity No one is saying this specifically, but the spirit of what is being said follow
by bradleyland 15y ago
A couple things strike me about the rebuttals here:
The illusion of simplicity
No one is saying this specifically, but the spirit of what is being said follows the line of thinking that because it's easy to get a server stack set up and running, it's a "simple task"". Getting a stack running is easy. Making it scale beyond a certain threshold is not simple. That threshold tends to be a brick wall in my experience.
Underestimation of time spent
It's easy to sink a lot of time in to these administrative tasks. You have to remember what business you're in. If you're a sysadmin, great. This is time well spent, but most of us are not sysadmins. We're in some other business. It is good to understand how a web hosting stack works from a high level. It helps you make good decisions. One of those "good decisions" is recognizing where your time is best spent. If your business is anything other than system administration, I doubt that relentlessly benchmarking your stack to see how many reqs/s you can host is a good use of your time.
The assumption that "I" know a solution
The only people with enough knowledge to challenge Patrick's decisions are those who have met this challenge. My WP hosting works fine, but that's because 2,500 visitors a day is a big day for me. I wouldn't even begin to suggest that he didn't make a sound decision until I had met this challenge myself.
I have some small idea of how difficult it must be, because our company's software does "real time" online procurement events (reverse auctions). Sounds like boring stuff, right? Consider one of the basic tenants of scaling: serve from cache and avoid re-building on each request. Now, consider that in a reverse auction, you might have 10 bidders placing bids on 500 line items spread out over 8 lots. Bid rates regularly top 30 bids per minute. Each bidder has a unique object cache because of the nature of reverse auctions (they each have their own data sets to track). Sometimes we implement weighting, where the bids are multiplied by a factor, have some fixed value added to them, or are calculated as a markup/markdown percentage against a fixed value. Then there's the observer screen, which has views for overview, line item summary, and line item detail, each of which have their own view states. Every time someone places a bid, there are a number of objects that are completely invalidated and must be rebuilt. Everyone involved (observers and bidders) in the live event is polling the server at the fastest rate we can afford.
As much as I appreciate input, I'm not asking for solutions to our scaling challenges. We have plenty of good ideas, and we've stayed ahead of the curve (mostly) on what could very generously be called a shoestring budget. Even having done that, I can't question Patrick's decision here. If I suspect anything, it's that he made a wise choice. Scaling is hard.
My point is that you'll never know if you can meet a challenge until you do it yourself. If you have met the challenge, great, but at what cost? Consider your own bias and take a serious look at the time investment. If you still think you've got all the answers and you can do it in trivial time, please send me your business card, because I'll hire you on the spot.
- moe 15y agoYour answer is redis. But I don't have business cards.
- bradleyland 15y agoI laughed, but this is exactly how many of the answers here sound to me. Answers like this (even though I'm assuming yours was sarcastic) exemplify everything that I outlined above. The "you" stated below is in the general form, not you specifically. It is the "you" intended to represent people who would seriously respond to a question like this. You don't even know if a database is our bottleneck (it's not), you don't know what database/datastore we're using (it might be redis), you represent no experience in our problem domain, you have no detailed information about our current performance, but you have no hesitation about positing an answer. Sorry to be so blunt, but uh... "Get off my lawn, you damn kids!" :)
- gridspy 15y agoJust to get on your lawn for a moment. For Gridspy, we store data into our server from sensors and allow customers to watch a 1-second live data stream. http://your.gridspy.co.nz/demo/ http://your.gridspy.co.nz/demo/ The key from us was to switch from a "polling" model to a "push" model. We don't poll the database, our backend pushes the new data through the stack. In our case I plan to actively rebuild assets and store them to disk rather than rebuild and cache on demand. But you can see the live updates on the site now. More details (a bit dated now): http://blog.gridspy.co.nz/2009/10/realtime-data-from-sensors-to-browsers.html http://blog.gridspy.co.nz/2009/10/realtime-data-from-sensors... You might consider an approach such as http://socket.io/ http://socket.io/ and lots of logic in the page as a great way to avoid the "everyone polling the backend as fast as they can" model. By the way, your lawn is very nice. Well maintained. Perhaps you need a zen garden.
- bradleyland 15y agoI appreciate the thoughtful response :) Polling actually works really well for us though. It's like rudimentary time-division multiplexing for sockets. Push means you have to keep an open socket to clients, which has its own set of scaling problems. Our challenges are related to cache design complexity, fast cache rebuild, serialization, and queuing & distribution of tasks without sacrificing timeliness of response. There's a lot of pressure for us to deliver a specific subset of the data in the same request/response cycle, which can be challenging. These are all pretty well known problems in computer science, and we feel pretty confident that we've got solutions for them. I'm definitely going to keep an eye on your blog though. It's nice to keep in touch with companies that face similar challenges.