5 ms·

All the new shiny. From experience I can tell you a) most companies deal with small amounts of data. Small can mean dozens of megabytes to dozens or hundreds o

All the new shiny. From experience I can tell you a) most companies deal with small amounts of data. Small can mean dozens of megabytes to dozens or hundreds of gigabytes. A single well provisioned server will typically be able to handle that very well. Also an SQL database can do a great deal if you know what you're doing. b) inappropriately used big data frameworks are expensive performance killers. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-... for example. c) Good quality programming, as in understanding the machine, memory layout and why it matters, and a good understanding of algorithms (and a hefty dose of common sense), will often yield you more speedup than buying almost any number of new machines. c) Hiring is often driven by fads and companies often don't like being told 'you don't need this roomful of servers', they like to waste money, so maybe do learn them (the profligacy with money is likely to be coming to an end with the economic damage of covid). Takeaway: brainpower will get you much further than horsepower

nisa 6y ago
Yes. Also if your foundation is already flaky designed and not well understood turning it into a distributed system likely makes everything more error-prone and more likely to fail. > The first rule of distributed systems is don't distribute your system until you have an observable reason to.
- throwaway_pdp09 6y ago
  Hah. Well said. Along the same lines: “You can have a second computer once you’ve shown you know how to use the first one.” –Paul Barham
techsin101 6y ago
I'm OP, and if I could I'd use Postgres everywhere, I want to know exactly when other stuff is useful and necessary, the trade offs, so I can make judgement call for myself and team.. That yes we don't need this, regular sql server would do with some pub/sub.
- throwaway_pdp09 6y ago
  I can't tell you when to use it, I can clearly tell you when not to (as has the other guy here). If 1 machine postgres suffices, stick with it. If it does not, would buying more processors/memory/disk fix that? If so, stick with it. The big data stuff is when you really have too much data to deal with in one box and literally cannot get round that. The tradeoffs are (my limited experience): cost in terms of buying a stack of boxes. Cost in terms of learning a massive new software stack. Cost in terms of "it scales almost infinitely but each node processes so much less". I gave a link before, I hope that showed something useful. Here's another, see article and paper https://html.duckduckgo.com/html?q=scalability%20%22at%20what%20cost%22 https://html.duckduckgo.com/html?q=scalability%20%22at%20wha... Does that provide (at least the beginnings of) an answer?
  - throwaway_pdp09 6y ago
    (that last line could come across as rather snotty; was not intended that way. Meant to imply I'm happy to try to answer any questions to the very limited extent that I can).
collyw 6y ago
A classical LAMP stack with server side rendering will be fine for 90+ % of companies out there. And will save a load of effort in development and maintenance.
firstprimate 6y ago
So much this. The original question has the cart before the horse. First understand the problem you're trying to solve, then solve THAT problem. Not the one you wish you had. Or, in the words of John Gall: "A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system."

nisa 6y ago

Yes. Also if your foundation is already flaky designed and not well understood turning it into a distributed system likely makes everything more error-prone and more likely to fail. > The first rule of distributed systems is don't distribute your system until you have an observable reason to.

throwaway_pdp09 6y ago
Hah. Well said. Along the same lines: “You can have a second computer once you’ve shown you know how to use the first one.” –Paul Barham

techsin101 6y ago

I'm OP, and if I could I'd use Postgres everywhere, I want to know exactly when other stuff is useful and necessary, the trade offs, so I can make judgement call for myself and team.. That yes we don't need this, regular sql server would do with some pub/sub.

throwaway_pdp09 6y ago
I can't tell you when to use it, I can clearly tell you when not to (as has the other guy here). If 1 machine postgres suffices, stick with it. If it does not, would buying more processors/memory/disk fix that? If so, stick with it. The big data stuff is when you really have too much data to deal with in one box and literally cannot get round that. The tradeoffs are (my limited experience): cost in terms of buying a stack of boxes. Cost in terms of learning a massive new software stack. Cost in terms of "it scales almost infinitely but each node processes so much less". I gave a link before, I hope that showed something useful. Here's another, see article and paper https://html.duckduckgo.com/html?q=scalability%20%22at%20what%20cost%22 https://html.duckduckgo.com/html?q=scalability%20%22at%20wha... Does that provide (at least the beginnings of) an answer?
- throwaway_pdp09 6y ago
  (that last line could come across as rather snotty; was not intended that way. Meant to imply I'm happy to try to answer any questions to the very limited extent that I can).

collyw 6y ago

A classical LAMP stack with server side rendering will be fine for 90+ % of companies out there. And will save a load of effort in development and maintenance.

firstprimate 6y ago

So much this. The original question has the cart before the horse. First understand the problem you're trying to solve, then solve THAT problem. Not the one you wish you had. Or, in the words of John Gall: "A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system."