6 ms·

I find this argument to be pretty weak, and seems to be a stand-in for, "I don't want to use a new technology because it'd require me to learn something, so I'm

by dimino 7y ago

I find this argument to be pretty weak, and seems to be a stand-in for, "I don't want to use a new technology because it'd require me to learn something, so I'm going to shoehorn something I feel more comfortable with, even though it's not the best tool for the job." Edit: I'm rate limited so to elaborate a bit. Queueing technologies are not "new interesting bit of technology", they're tailored solutions to solve a specific problem. Using a RDBMS for queueing is not what it was built to do, and you will run into issues doing so (I have). By trying to use one tool for everything, you lose out on all kinds of optimizations, features, and performance enhancements that are specific to the problem you're trying to solve. It's bad engineering to try and force Postgres into the role of queue when vastly superior technologies exist. You're actively hurting the engineers you work with, and the company you work for, if you force the same tech into use cases it isn't optimal for. I cannot stress this enough; fear of learning is anathema to software, and trying to hide it behind a veneer of caution is not only disingenuous but potentially malicious as well, maximizing exclusively for the benefit of the individual against the interests of the group.

mlyle 7y ago
Fewer pieces in the technology stack is better, because each piece is operationally expensive (it is its own set of work in maintenance, upgrades, migrations, etc). There are also the mentioned costs (you can't atomically do things across divergent pieces of infrastructure, at least not very easily at all). Fighting the tendency of everyone wanting to draw in each new interesting bit of technology is important. Every piece of complexity you incur should be proven as required, and you should build things to just somewhat larger than near term projected scale... and not expect that you are a temporarily embarrassed unicorn with loads of unexpected demand showing up. If you run out of performance, you have some headroom from tuning; from that point, you can then choose whether it makes sense to retrofit to a higher scale technology or throw larger hardware at the problem (or both).
- threeseed 7y ago
  These days it's pretty trivial to have a cloud managed component e.g. Redis that is maintained, upgraded and supported. And then you have a component that is designed for the job instead of trying to use a database as a poor man's queue.
  - mlyle 7y ago
    > These days it's pretty trivial to have a cloud managed component e.g. Redis that is maintained, upgraded and supported. It's still another moving part. Thing should be as simple as they can be, but no simpler. > trying to use a database as a poor man's queue. Of course, it's not really a "poor man's" queue-- it's got some superior capabilities. It just loses on top-end performance. (Of course, using those capabilities is dangerous, because it creates some degree of lock-in, so go into it open-eyed). For as much as you accuse others of looking down their nose / not willing to seriously consider other technologies... you seem to be inclined that way yourself. Redis is great. But if you have Postgres already, and modest to moderate queuing requirements, why add another piece to your stack? Postgres-by-default is not a bad technology sourcing strategy.
    - dimino 7y ago
      It is very much a bad technology sourcing strategy, and you will kill your business if you attempt it. Period. It comes from a place of ignorance, and you're promoting ignorance. Learn why technologies exist and make an informed decision about tradeoffs, instead of being lazy and incompetent by blindly choosing technology based on what's a very locally maxima for you personally. It's selfish and damaging. Edit since I'm rate limited: The folks landing on Postgres are not landing there after due consideration, they're landing on a technology they're familiar with because they don't know how to learn. It is lazy, ignorant and selfish, and while of course people don't like having truth spoken to them, that's what it is; truth. Edit 2: Making these poor technology choices will kill your business because you won't have the agility that using a specific message broker technology gives you. You won't have built-in solutions to common queueing problems, you won't have dedicated specific logging/metrics to monitor, you won't have the libraries in your preferred language to directly tackle your problem, you won't have the support community available to you (it will be much smaller), you won't be able to pivot onto related patterns as your needs change, your scaling will always be more complicated because fewer people are doing it and the tool you use isn't tailored for your use case; the list goes on. Your competitors will swallow you because they move faster than you do, and your business will die. This is annoying to me because I've been in situations where I've had to maintain and write features against technology that was a poor fit for its use, but people like those in this comment sections bitched and moaned about a "new thing" existing in our stack. The reality was they didn't want to learn anything, and were fine pushing the hard work off onto the developers, so they could safely continue to do as little as they possibly could get away with. Do your job, learn technology that actually fits your use case, and stop trying to push work off onto other people.
- ukd1 7y ago
  So much love for this; 'Every piece of complexity you incur should be proven as required', plus 'Fewer pieces in the technology stack is better, because each piece is operationally expensive'. Also, this totally ignores (as these, imho are enough in themselves) the actual practical differences between using a single thing and two - synchronizing state; most of the cases here guarantee atomicnes individually, but not as a pair - you (the engineer) have to deal with that - which is non-trivial.
ukd1 7y ago
It's not bad-engineering at all; how many engineers have to operate at that level in your stack, ever. Close to zero, as normally they just use an abstraction on top of it - Rails for instance is ActiveJob. No one has to care what's behind it, except folks running production - and for them it's a trade off; it's simpler - one less moving part, but also more load on one single part. Is that worth it, or not? Depends. I can't stress this enough; claiming anything is "vastly superior", especially with no examples, isn't useful - things aren't this black and white in reality. Most tech-choices like this are somewhere between trade-offs, preferences, ignorance or wrongly held opinions about something being vastly superior > something else.
- dimino 7y ago
  Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing. You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics. I cannot stress this enough; you are flat wrong if you think Postgres is appropriate to use for queueing.
  - ukd1 7y ago
    An array can be appropriate for queuing for some things. You're wrong - as you are trying to state "a problem" has a specific solution, yet you don't know the problem. Yes, there are situations when using a dedicated message broker would be objectively better; there are many when that's totally wrong. Nothing is likely be correct about your arguments if you don't know what you are trying to fix or the sitation, which should also be blatantly obvious. You're applying some unstated subjective situation in your head, then dumping out something you've heard / done before. LOL. Also "if you've ever used both" - I have, which is why I know. It's gray; sometimes you should, sometimes you shouldn't. You're assuming I haven't as I have differing point of view to you.
    - dimino 7y ago
      No, I'm assuming you haven't otherwise you'd know that redis AND postgres are inferior message brokers, both of them, and the fact that you haven't corrected me means you're not well versed at all in this problem domain.
outworlder 7y ago
> I cannot stress this enough; fear of learning is anathema to software, and trying to hide it behind a veneer of caution is not only disingenuous but potentially malicious as well, maximizing exclusively for the benefit of the individual against the interests of the group. This is wrong. You are ultimately getting paid to solve the company's problems. If those problems actually require deployment of a new technology to satisfy the requirements, then it is fine. Note that I said satisfy requirements, not exceed them. But more often than not, * that's not the case *. If your queue requirements can be served with PostgreSQL, and you already have it, then why not? To do otherwise would be over engineering. Because you may know a new and fancy technology that would be objectively better. Cool. Now, do you have enough know-how in the company to use it effectively? Do you know what the best practices are? Can you deploy, monitor, audit and patch this in production? If you have answered NO to any of those questions, you'll either hire people or you shouldn't deploy. Period. And if you have only one person answering YES to the previous questions, you still shouldn't deploy it. Because one day that person will leave, and you are now left the company in a worse situation than it was before. For no reason other than satisfying your engineering itch. THAT is anathema to working, reliable production software. Toys, you can do whatever. If you want to introduce something new, you can. But you need to do it responsibly. It needs a reason to exist - a valid reason, not "it's better". How does it being "better" help the company? Will it reduce costs? Maintenance? Future development work will happen faster? Will it make a faster user experience? Scale to the projected company growth? It will have a cost that will have to be accounted for – including opportunity costs. > By trying to use one tool for everything, you lose out on all kinds of optimizations, features, and performance enhancements that are specific to the problem you're trying to solve. You only care about this if you have a reason to care about this. Otherwise it's all irrelevant.
- dimino 7y ago
  Sorry, but what you call a "toy", I call a highly reliable technology that's used in production widely across the majority of Fortune 500 companies. No one here is suggesting anything remotely like what you're strawmanning, but it's clear the straw man is the only argument you can take down. The reason you are completely wrong and a toxic member of your team is that your logic can be used to justify all kinds of terrible, short term wins that sacrifice any kind of long term/sustainable design. You'd rather make a bad technology play so your boss gets off your back, than think through your decision and make a long term investment into a sustainable, maintainable, improvable, and agile technology that's actually suited to solve the real problem you have. OF COURSE you're there to accomplish the tasks your business needs to succeed. That's a given. What's not a given is that you need to slap together the shittiest piece of software that'll do that job right then and there. Your attitude is what keeps companies from solving problems, and forces them to pay dollar after dollar to fix the same issue over and over again. You're selfish, you're greedy, and you cost your company insane amounts of money. You should not be employed in this industry if what you wrote here is how you think, period.

by dimino 7y ago

mlyle 7y ago

Fewer pieces in the technology stack is better, because each piece is operationally expensive (it is its own set of work in maintenance, upgrades, migrations, etc). There are also the mentioned costs (you can't atomically do things across divergent pieces of infrastructure, at least not very easily at all). Fighting the tendency of everyone wanting to draw in each new interesting bit of technology is important. Every piece of complexity you incur should be proven as required, and you should build things to just somewhat larger than near term projected scale... and not expect that you are a temporarily embarrassed unicorn with loads of unexpected demand showing up. If you run out of performance, you have some headroom from tuning; from that point, you can then choose whether it makes sense to retrofit to a higher scale technology or throw larger hardware at the problem (or both).

threeseed 7y ago
These days it's pretty trivial to have a cloud managed component e.g. Redis that is maintained, upgraded and supported. And then you have a component that is designed for the job instead of trying to use a database as a poor man's queue.
- mlyle 7y ago
  > These days it's pretty trivial to have a cloud managed component e.g. Redis that is maintained, upgraded and supported. It's still another moving part. Thing should be as simple as they can be, but no simpler. > trying to use a database as a poor man's queue. Of course, it's not really a "poor man's" queue-- it's got some superior capabilities. It just loses on top-end performance. (Of course, using those capabilities is dangerous, because it creates some degree of lock-in, so go into it open-eyed). For as much as you accuse others of looking down their nose / not willing to seriously consider other technologies... you seem to be inclined that way yourself. Redis is great. But if you have Postgres already, and modest to moderate queuing requirements, why add another piece to your stack? Postgres-by-default is not a bad technology sourcing strategy.
  - dimino 7y ago
    It is very much a bad technology sourcing strategy, and you will kill your business if you attempt it. Period. It comes from a place of ignorance, and you're promoting ignorance. Learn why technologies exist and make an informed decision about tradeoffs, instead of being lazy and incompetent by blindly choosing technology based on what's a very locally maxima for you personally. It's selfish and damaging. Edit since I'm rate limited: The folks landing on Postgres are not landing there after due consideration, they're landing on a technology they're familiar with because they don't know how to learn. It is lazy, ignorant and selfish, and while of course people don't like having truth spoken to them, that's what it is; truth. Edit 2: Making these poor technology choices will kill your business because you won't have the agility that using a specific message broker technology gives you. You won't have built-in solutions to common queueing problems, you won't have dedicated specific logging/metrics to monitor, you won't have the libraries in your preferred language to directly tackle your problem, you won't have the support community available to you (it will be much smaller), you won't be able to pivot onto related patterns as your needs change, your scaling will always be more complicated because fewer people are doing it and the tool you use isn't tailored for your use case; the list goes on. Your competitors will swallow you because they move faster than you do, and your business will die. This is annoying to me because I've been in situations where I've had to maintain and write features against technology that was a poor fit for its use, but people like those in this comment sections bitched and moaned about a "new thing" existing in our stack. The reality was they didn't want to learn anything, and were fine pushing the hard work off onto the developers, so they could safely continue to do as little as they possibly could get away with. Do your job, learn technology that actually fits your use case, and stop trying to push work off onto other people.
ukd1 7y ago
So much love for this; 'Every piece of complexity you incur should be proven as required', plus 'Fewer pieces in the technology stack is better, because each piece is operationally expensive'. Also, this totally ignores (as these, imho are enough in themselves) the actual practical differences between using a single thing and two - synchronizing state; most of the cases here guarantee atomicnes individually, but not as a pair - you (the engineer) have to deal with that - which is non-trivial.

ukd1 7y ago

It's not bad-engineering at all; how many engineers have to operate at that level in your stack, ever. Close to zero, as normally they just use an abstraction on top of it - Rails for instance is ActiveJob. No one has to care what's behind it, except folks running production - and for them it's a trade off; it's simpler - one less moving part, but also more load on one single part. Is that worth it, or not? Depends. I can't stress this enough; claiming anything is "vastly superior", especially with no examples, isn't useful - things aren't this black and white in reality. Most tech-choices like this are somewhere between trade-offs, preferences, ignorance or wrongly held opinions about something being vastly superior > something else.

dimino 7y ago
Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing. You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics. I cannot stress this enough; you are flat wrong if you think Postgres is appropriate to use for queueing.
- ukd1 7y ago
  An array can be appropriate for queuing for some things. You're wrong - as you are trying to state "a problem" has a specific solution, yet you don't know the problem. Yes, there are situations when using a dedicated message broker would be objectively better; there are many when that's totally wrong. Nothing is likely be correct about your arguments if you don't know what you are trying to fix or the sitation, which should also be blatantly obvious. You're applying some unstated subjective situation in your head, then dumping out something you've heard / done before. LOL. Also "if you've ever used both" - I have, which is why I know. It's gray; sometimes you should, sometimes you shouldn't. You're assuming I haven't as I have differing point of view to you.
  - dimino 7y ago
    No, I'm assuming you haven't otherwise you'd know that redis AND postgres are inferior message brokers, both of them, and the fact that you haven't corrected me means you're not well versed at all in this problem domain.

outworlder 7y ago

> I cannot stress this enough; fear of learning is anathema to software, and trying to hide it behind a veneer of caution is not only disingenuous but potentially malicious as well, maximizing exclusively for the benefit of the individual against the interests of the group. This is wrong. You are ultimately getting paid to solve the company's problems. If those problems actually require deployment of a new technology to satisfy the requirements, then it is fine. Note that I said satisfy requirements, not exceed them. But more often than not, * that's not the case *. If your queue requirements can be served with PostgreSQL, and you already have it, then why not? To do otherwise would be over engineering. Because you may know a new and fancy technology that would be objectively better. Cool. Now, do you have enough know-how in the company to use it effectively? Do you know what the best practices are? Can you deploy, monitor, audit and patch this in production? If you have answered NO to any of those questions, you'll either hire people or you shouldn't deploy. Period. And if you have only one person answering YES to the previous questions, you still shouldn't deploy it. Because one day that person will leave, and you are now left the company in a worse situation than it was before. For no reason other than satisfying your engineering itch. THAT is anathema to working, reliable production software. Toys, you can do whatever. If you want to introduce something new, you can. But you need to do it responsibly. It needs a reason to exist - a valid reason, not "it's better". How does it being "better" help the company? Will it reduce costs? Maintenance? Future development work will happen faster? Will it make a faster user experience? Scale to the projected company growth? It will have a cost that will have to be accounted for – including opportunity costs. > By trying to use one tool for everything, you lose out on all kinds of optimizations, features, and performance enhancements that are specific to the problem you're trying to solve. You only care about this if you have a reason to care about this. Otherwise it's all irrelevant.

dimino 7y ago
Sorry, but what you call a "toy", I call a highly reliable technology that's used in production widely across the majority of Fortune 500 companies. No one here is suggesting anything remotely like what you're strawmanning, but it's clear the straw man is the only argument you can take down. The reason you are completely wrong and a toxic member of your team is that your logic can be used to justify all kinds of terrible, short term wins that sacrifice any kind of long term/sustainable design. You'd rather make a bad technology play so your boss gets off your back, than think through your decision and make a long term investment into a sustainable, maintainable, improvable, and agile technology that's actually suited to solve the real problem you have. OF COURSE you're there to accomplish the tasks your business needs to succeed. That's a given. What's not a given is that you need to slap together the shittiest piece of software that'll do that job right then and there. Your attitude is what keeps companies from solving problems, and forces them to pay dollar after dollar to fix the same issue over and over again. You're selfish, you're greedy, and you cost your company insane amounts of money. You should not be employed in this industry if what you wrote here is how you think, period.