4 ms·

I know of a Famous Large Company that used email as their message queue to synchronize the data across two of their large systems. It's a perfectly apt message

I know of a Famous Large Company that used email as their message queue to synchronize the data across two of their large systems. It's a perfectly apt message queue, just a bit heavyweight. But if it's "light enough", it comes "for free" with many OSes.

KaiserPro 3y ago
A famous financial news paper used an FTP server as its kafka "fan in" node. It was so old that the 1u case started to droop in the middle. (it was an sparc something or other with real SCSI ultra 320 drives in them)
doubled112 3y ago
Email is a message queue in the most literal of senses, isn't it?
- xena 3y ago
  Yes, it's actually somewhat of a decent (if cursed) message queue for many usecases too. Not to mention the debuggability (you already know how to use an email client).
  - Ozzie_osman 3y ago
    You could use Slack, too.
    - flir 3y ago
      Email keeps everything on-prem.
  - marcosdumay 3y ago
    > if cursed As long as you keep it internal-only, get a reliable server that doesn't do any anti-spam shit (AKA, nothing from MS or any other large company), doesn't use some unreliable proprietary database (again no MS), and have a reliable journaled disk, your only important failure mode will be losing disks. That's an easy failure mode to deal with.
    - alduin32 3y ago
      I love email, and to me it is a very important part of our infrastructure, but it is quite cursed as an arbitrary transport. The semantics are very complex, and any transported data will generally need to be encoded as base64, which is very inefficient. You also need to add misbehaving clients on your list of things not to use with a message-queue, just opening the queue's folder on Outlook to inspect it may be enough to completely corrupt it (Outlook will happily reorder, even rewrite, header fields of incoming messages, replicate messages to infinity, or who knows what else). I'd be worried to have a message queue that is compatible with such a cursed ecosystem and transport protocol.
c54 3y ago
Ha! I think i know what company you’re referring to
- SteveNuts 3y ago
  Oracle came to my mind but I'm sure it's in common use in many places
EvanAnderson 3y ago
Active Directory is (was?) capable of running replication traffic (for very particular use cases) over SMTP. I always thought that was ingenious.
lucianbr 3y ago
Using SMTP to transfer messages seems just plain common sense. Far different than IPv6-over-S3.
vidarh 3y ago
I co-founded a webmail provider in '99, and when we needed a message queue reaching for our heavily customized Qmail setup was a relatively natural choice. I've mentioned it here before. Provides all the routing and retries we needed, and made debugging trivial (e-mail from our desktop clients to the queues worked; cc:'ing a real mailbox with copies worked; checking out queues with POP3 worked...)
ahefner 3y ago
I always thought using e-mail in this would be good for systems that occasionally need a human in the loop. Normally one service processes and mails results to the next, occasionally something is forwarded to a human, who can make some decision, edit the mail, then forward it along back into the automated path.
simonjgreen 3y ago
Before there was REST and other http based standards ebXML was a hotness and tools like bizspark etc. You could see the logical progression through: I fill in the order form and post it. They mail back an invoice. I fax the order form, they fax back an invoice. My computer sends the order form in a structured way over email. My computer sends the order form in a structured way over http. And that's why many large telcos and banks still use ebXML in their B2B transactions. It's fundamentally the same business process and logic it's always been, with glacially slow improvement in performance over time.
anotherevan 3y ago
One of my customers used email (a gmail account, no less) as a message queue between their front end site and the back-office processor. This worked quite well for close to a decade I think. It basically evolved from when applications from their original customer facing site were emailed and manually entered into the back-office system by a human. They were looking to automate this with a minimum of changes to too many of the moving parts at once, so I reformatted the sent email to contain an XML payload so the new back-office automation could read and process it (and in a pinch, a human could still review any problem applications) using Java's mail APIs. Things evolved, the front-end web site got replaced with a Wordpress site, but the email message queue kept working for a long time. In the last year or so it was getting more and more onerous though. Reconciling information between the front-end and the mail box showed not all emails were being delivered, and authentication to gmail was becoming more and more of a burdensome moving target. I just recently replaced the whole thing with an API call made from the back-office to the Wordpress site to access the stored data. (The original site didn't store, just emailed, which was why this was not an option historically.)
- schwartzworld 3y ago
  I wrote a static site generator with comments that used email as an ingestion cue.
  - klysm 3y ago
    Queue?
    - schwartzworld 3y ago
      Gesundheit

KaiserPro 3y ago

A famous financial news paper used an FTP server as its kafka "fan in" node. It was so old that the 1u case started to droop in the middle. (it was an sparc something or other with real SCSI ultra 320 drives in them)

doubled112 3y ago

Email is a message queue in the most literal of senses, isn't it?

xena 3y ago
Yes, it's actually somewhat of a decent (if cursed) message queue for many usecases too. Not to mention the debuggability (you already know how to use an email client).
- Ozzie_osman 3y ago
  You could use Slack, too.
  - flir 3y ago
    Email keeps everything on-prem.
- marcosdumay 3y ago
  > if cursed As long as you keep it internal-only, get a reliable server that doesn't do any anti-spam shit (AKA, nothing from MS or any other large company), doesn't use some unreliable proprietary database (again no MS), and have a reliable journaled disk, your only important failure mode will be losing disks. That's an easy failure mode to deal with.
  - alduin32 3y ago
    I love email, and to me it is a very important part of our infrastructure, but it is quite cursed as an arbitrary transport. The semantics are very complex, and any transported data will generally need to be encoded as base64, which is very inefficient. You also need to add misbehaving clients on your list of things not to use with a message-queue, just opening the queue's folder on Outlook to inspect it may be enough to completely corrupt it (Outlook will happily reorder, even rewrite, header fields of incoming messages, replicate messages to infinity, or who knows what else). I'd be worried to have a message queue that is compatible with such a cursed ecosystem and transport protocol.

c54 3y ago

Ha! I think i know what company you’re referring to

SteveNuts 3y ago
Oracle came to my mind but I'm sure it's in common use in many places

EvanAnderson 3y ago

Active Directory is (was?) capable of running replication traffic (for very particular use cases) over SMTP. I always thought that was ingenious.

lucianbr 3y ago

Using SMTP to transfer messages seems just plain common sense. Far different than IPv6-over-S3.

vidarh 3y ago

I co-founded a webmail provider in '99, and when we needed a message queue reaching for our heavily customized Qmail setup was a relatively natural choice. I've mentioned it here before. Provides all the routing and retries we needed, and made debugging trivial (e-mail from our desktop clients to the queues worked; cc:'ing a real mailbox with copies worked; checking out queues with POP3 worked...)

ahefner 3y ago

I always thought using e-mail in this would be good for systems that occasionally need a human in the loop. Normally one service processes and mails results to the next, occasionally something is forwarded to a human, who can make some decision, edit the mail, then forward it along back into the automated path.

simonjgreen 3y ago

Before there was REST and other http based standards ebXML was a hotness and tools like bizspark etc. You could see the logical progression through: I fill in the order form and post it. They mail back an invoice. I fax the order form, they fax back an invoice. My computer sends the order form in a structured way over email. My computer sends the order form in a structured way over http. And that's why many large telcos and banks still use ebXML in their B2B transactions. It's fundamentally the same business process and logic it's always been, with glacially slow improvement in performance over time.

anotherevan 3y ago

One of my customers used email (a gmail account, no less) as a message queue between their front end site and the back-office processor. This worked quite well for close to a decade I think. It basically evolved from when applications from their original customer facing site were emailed and manually entered into the back-office system by a human. They were looking to automate this with a minimum of changes to too many of the moving parts at once, so I reformatted the sent email to contain an XML payload so the new back-office automation could read and process it (and in a pinch, a human could still review any problem applications) using Java's mail APIs. Things evolved, the front-end web site got replaced with a Wordpress site, but the email message queue kept working for a long time. In the last year or so it was getting more and more onerous though. Reconciling information between the front-end and the mail box showed not all emails were being delivered, and authentication to gmail was becoming more and more of a burdensome moving target. I just recently replaced the whole thing with an API call made from the back-office to the Wordpress site to access the stored data. (The original site didn't store, just emailed, which was why this was not an option historically.)

schwartzworld 3y ago
I wrote a static site generator with comments that used email as an ingestion cue.
- klysm 3y ago
  Queue?
  - schwartzworld 3y ago
    Gesundheit