7 ms·

This might be a stupid question, but surely no one thinks of RabbigMQ as a database right? I’ve used it from 2012 to 2018 extensively, including using things l

This might be a stupid question, but surely no one thinks of RabbigMQ as a database right? I’ve used it from 2012 to 2018 extensively, including using things like shovels to build hub spoke topologies, however not once did I think of it as anything but a message broker. Did I miss something huge?

henryfjordan 6y ago
RabbitMQ stores your data, right? Then it's a database! That's pretty much all it takes. A flat file, memory-store, SQL DB, Document store, any of them can be databases if that's where you stick your data! But also no, RabbitMQ and Kafka and the like are clearly message buses and though they might also technically qualify as a DB it would be a poor descriptor.
- ceocoder 6y ago
  Ah I see, we are going with “well technically it stores something therefore it is database joke”. Now I’m fully onboard :) Back when I worked in LA my CTO used to joke that most places use Microsoft Outlook as a database and Excel as BI tool.
  - goatinaboat 6y ago
    well technically it stores something therefore it is database joke Confluent, the company behind Kafka, are 100% serious about Kafka being a database. It is however a far better database than MongoDB.
  - m0zg 6y ago
    You laugh, but I bet Excel produces orders of magnitude more real "business intelligence" than all other "BI" tools combined.
    - capsulecorp 6y ago
      You bet, but I'd really love to see data that supports that.
    - GuB-42 6y ago
      Here is an anecdote. I had to work on a tool that shows what's wrong with an assembly line: missing parts, delays, etc... So that management can take corrective action. Typical "BI" stuff but in a more industrial setting. The company went all out on new technologies. Web front-end, responsive design, "big data", distributed computing, etc... My job was to use PySpark to extract indicators from a variety of data sources. Nothing complex, but the development environment was so terrible it turned the most simple task into a challenge. One day, the project manager (sorry, "scrum master") came in, opened an excel sheet, imported the data sets, and in about 5 minutes, showed me what I had to do. It took me several days to implement... So basically, my manager with Excel was hundreds of times more efficient than I was with all that shiny new technology. That experience made me respect Excel and people who know how to use it a lot more, and modern stacks a lot less. I am fully aware that Excel is not always the right tool for the job, and that modern stacks have a place. For example, Excel does not scale, but there are cases where you don't need scalability. An assembly line isn't going to start processing 100x more parts anytime soon, and one that does will be very different. There are physical limits.
      - yomly 6y ago
        Well in these situations, the implicit ask of your company (I've been there myself) is to basically rebuild excel but replace some of the power/flexibility of excel for safety and to remove the risk of error away from front end users (aka move the risk to the back end developers) Unfortunately which specific features of Excel are acceptable to remove are unknown until you have already way over invested into the project. The best I've seen this done is having Excel as a client for your data store. Where read access is straightforward and write can be done via csv upload (and heavy validation and maybe history rollback). That way the business can self-service every permutation of dashboard/report they need and only when a very specific usecase arises do you need to start putting engineering effort behind it. I suppose you can also supplement the Excel workflow with a pared down CRUD interface for the inevitable employee allergic to excel.
        tjalfi 6y ago
        I posted elsewhere[0] in this thread about my employer's successful practice of replacing shared spreadsheets with web applications. Here is another option that we use instead of CSV import. Our applications support custom reports and custom fields. Users can define new reports and run them on demand. They can also define custom field types with validation, data entry support, etc. This combination provides some of the extensibility of Excel while retaining the advantages of an application. Edited for wording changes. [0] https://news.ycombinator.com/item?id=23292374 https://news.ycombinator.com/item?id=23292374
    - jtdev 6y ago
      ...And orders of magnitude more wasted time and capital due to inaccurate and isolated data.
      - tjalfi 6y ago
        People use what they know to solve the problems they have. You can complain about their solution or see it as an opportunity. I posted elsewhere[0] in this thread about my employer's practice of replacing shared spreadsheets with web applications. This approach works quite well for us and I would encourage you to consider it as an option. [0] https://news.ycombinator.com/item?id=23292374 https://news.ycombinator.com/item?id=23292374
  - nieve 6y ago
    If memory serves the original EToys.com code treated the filesystem as tree-structured database using atomic operations (though no transactions). It worked just fine, then the rewrite with an RDBMS that should have been stabler and faster resulted in the famous meltdowns. Admittedly this is cheating a bit since you can name folders & files with semi-arbitrary or internally structured string keys. By 1997 standards pure disk access without having to walk the filesystem heirarchy was blazingly fast compared to many of the databases I was using. [Source: I was friends with the guy who wrote it as well as other EToys employees. God that was a trainwreck.]
    - __Joker 6y ago
      Interesting, is there a blog around discussing this in detail ? If not would be kind enough to go more into detail.
      - nieve 6y ago
        I don't think anyone posted about their particular system, but it's not unknown now. If you google "filesystem as a database" there are some relevant hits. One super simple and probably not ideal, but at least balanced version uses a hash of some primary key like customer row id as the file index, then partitions the items into directories with all permutations at each level (or only populated ones) based on successive parts of the hash. For example an item key that hashes to a32c4214585e9cb7a55474133a5fc986 would be located somewhere like this: a32c/4214/585e/9cb7/a554/74133a5fc986 a32c/ 4214/ 585e/ 9cb7/ a554/ 74133/a5fc986 The advantage of this kind of structure is that you never need to manually scan a directory since you know exactly what path you're trying to open. You still incur the OS lookup time for the inode-equivalent in the directory entry, but a deeper heirarchy keeps that faster. You can trade off time to traverse the heirarchy versus number of entries in the final directories by adjusting the length of the hash chunk you use at each level. Two characters will put vastly fewer entries at a given level, but vastly increase your directory depth. Basically if you're manually scanning the heirarchy for anything but a consistency check or garbage collection you've already lost.
  - tjalfi 6y ago
    Excel can be an excellent source of new line-of-business applications. Many of my employer's applications started out as a shared spreadsheet or Access database. Our development team worked with the users and built a web application to solve the same problem. This approach has a lot of advantages: * The market exists and has an incumbent. There's a lower risk of a write-off. * The users are open to process changes. You still have to migrate people off of the spreadsheet, though. * It's easy to add value with reporting, error checking, concurrent access, and access control. * You can import the existing data to make the transition easier. This will require a lot of data cleaning. Edited to add the following text from another post. You can cover most of the requirements with a set of fixed fields. The last 10% to 20% of the use cases requires custom reports and custom fields. Users should be able to define their own reports and run them without your involvement. They should also be able to define custom field types with validation, data entry support, etc. If your web application has these two features and other advantages then you should be able to replace Excel.
- gopalv 6y ago
  > Kafka and the like are clearly message buses and though they might also technically qualify as a DB ksqldb is actually a database on top of this. The thing is that they have an incrementally updated materialized view that is the table, while the event stream is similar to a WAL ("ahead of write logs?" in this case). Because eventually you can't just go over your entire history for every query.
  - deleted 6y ago
    [deleted]
- makach 6y ago
  Oh ho ho ho. What weird things we use as a databases. I remember when I first started out as a consultant developer we were using a CMS as our data repository because someone thought that was a good idea. (It wasn't). The vendor was flown in from the states to help troubleshooting. I will never forget how he looked at me when I had to explain to him why we made so many nodes in the content tree, it was because we were using the CMS as a repository.
twic 6y ago
I once worked on a system for notifying customers of events by posting to their APIs. Events came in on a Rabbit queue and got posted. If a customer's API was down, the event would go back on the queue with a header saying to retry it after some time. You can do some sort of incantation to specifically retrieve messages with a suitable header value, to find messages which are ready to retry. We used exponential backoff, capped at one day, because the API might be down for a week. I didn't think of RabbitMQ as a database when I started that work, but it looked a lot like it by the time I finished.
- sida 6y ago
  lol api could be down for a week? What?
  - csours 6y ago
    Not everything is or needs to be webscale.
- layoutIfNeeded 6y ago
  Sounds like delay line memory.
Hamuko 6y ago
>This might be a stupid question, but surely no one thinks of RabbigMQ as a database right? Arguably the world's most popular database is Microsoft Excel.
- deleted 6y ago
  [deleted]
- mr_toad 6y ago
  It’s definitely popular, that much is inarguable.
detaro 6y ago
I interpret it as they'd probably not call it a database, but they might use it in places where a database would be better suited, and effectively store data in it.
danpalmer 6y ago
It's both. It's best used when it's being used as a message broker, but any sufficiently advanced message broker will need many of the features of a database – durability of messages, querying in various ways, etc. I think it's reasonable to think of it as a very specialised database.

henryfjordan 6y ago

RabbitMQ stores your data, right? Then it's a database! That's pretty much all it takes. A flat file, memory-store, SQL DB, Document store, any of them can be databases if that's where you stick your data! But also no, RabbitMQ and Kafka and the like are clearly message buses and though they might also technically qualify as a DB it would be a poor descriptor.

ceocoder 6y ago
Ah I see, we are going with “well technically it stores something therefore it is database joke”. Now I’m fully onboard :) Back when I worked in LA my CTO used to joke that most places use Microsoft Outlook as a database and Excel as BI tool.
- goatinaboat 6y ago
  well technically it stores something therefore it is database joke Confluent, the company behind Kafka, are 100% serious about Kafka being a database. It is however a far better database than MongoDB.
- m0zg 6y ago
  You laugh, but I bet Excel produces orders of magnitude more real "business intelligence" than all other "BI" tools combined.
  - capsulecorp 6y ago
    You bet, but I'd really love to see data that supports that.
  - GuB-42 6y ago
    Here is an anecdote. I had to work on a tool that shows what's wrong with an assembly line: missing parts, delays, etc... So that management can take corrective action. Typical "BI" stuff but in a more industrial setting. The company went all out on new technologies. Web front-end, responsive design, "big data", distributed computing, etc... My job was to use PySpark to extract indicators from a variety of data sources. Nothing complex, but the development environment was so terrible it turned the most simple task into a challenge. One day, the project manager (sorry, "scrum master") came in, opened an excel sheet, imported the data sets, and in about 5 minutes, showed me what I had to do. It took me several days to implement... So basically, my manager with Excel was hundreds of times more efficient than I was with all that shiny new technology. That experience made me respect Excel and people who know how to use it a lot more, and modern stacks a lot less. I am fully aware that Excel is not always the right tool for the job, and that modern stacks have a place. For example, Excel does not scale, but there are cases where you don't need scalability. An assembly line isn't going to start processing 100x more parts anytime soon, and one that does will be very different. There are physical limits.
    - yomly 6y ago
      Well in these situations, the implicit ask of your company (I've been there myself) is to basically rebuild excel but replace some of the power/flexibility of excel for safety and to remove the risk of error away from front end users (aka move the risk to the back end developers) Unfortunately which specific features of Excel are acceptable to remove are unknown until you have already way over invested into the project. The best I've seen this done is having Excel as a client for your data store. Where read access is straightforward and write can be done via csv upload (and heavy validation and maybe history rollback). That way the business can self-service every permutation of dashboard/report they need and only when a very specific usecase arises do you need to start putting engineering effort behind it. I suppose you can also supplement the Excel workflow with a pared down CRUD interface for the inevitable employee allergic to excel.
      - tjalfi 6y ago
        I posted elsewhere[0] in this thread about my employer's successful practice of replacing shared spreadsheets with web applications. Here is another option that we use instead of CSV import. Our applications support custom reports and custom fields. Users can define new reports and run them on demand. They can also define custom field types with validation, data entry support, etc. This combination provides some of the extensibility of Excel while retaining the advantages of an application. Edited for wording changes. [0] https://news.ycombinator.com/item?id=23292374 https://news.ycombinator.com/item?id=23292374
  - jtdev 6y ago
    ...And orders of magnitude more wasted time and capital due to inaccurate and isolated data.
    - tjalfi 6y ago
      People use what they know to solve the problems they have. You can complain about their solution or see it as an opportunity. I posted elsewhere[0] in this thread about my employer's practice of replacing shared spreadsheets with web applications. This approach works quite well for us and I would encourage you to consider it as an option. [0] https://news.ycombinator.com/item?id=23292374 https://news.ycombinator.com/item?id=23292374
- nieve 6y ago
  If memory serves the original EToys.com code treated the filesystem as tree-structured database using atomic operations (though no transactions). It worked just fine, then the rewrite with an RDBMS that should have been stabler and faster resulted in the famous meltdowns. Admittedly this is cheating a bit since you can name folders & files with semi-arbitrary or internally structured string keys. By 1997 standards pure disk access without having to walk the filesystem heirarchy was blazingly fast compared to many of the databases I was using. [Source: I was friends with the guy who wrote it as well as other EToys employees. God that was a trainwreck.]
  - __Joker 6y ago
    Interesting, is there a blog around discussing this in detail ? If not would be kind enough to go more into detail.
    - nieve 6y ago
      I don't think anyone posted about their particular system, but it's not unknown now. If you google "filesystem as a database" there are some relevant hits. One super simple and probably not ideal, but at least balanced version uses a hash of some primary key like customer row id as the file index, then partitions the items into directories with all permutations at each level (or only populated ones) based on successive parts of the hash. For example an item key that hashes to a32c4214585e9cb7a55474133a5fc986 would be located somewhere like this: a32c/4214/585e/9cb7/a554/74133a5fc986 a32c/ 4214/ 585e/ 9cb7/ a554/ 74133/a5fc986 The advantage of this kind of structure is that you never need to manually scan a directory since you know exactly what path you're trying to open. You still incur the OS lookup time for the inode-equivalent in the directory entry, but a deeper heirarchy keeps that faster. You can trade off time to traverse the heirarchy versus number of entries in the final directories by adjusting the length of the hash chunk you use at each level. Two characters will put vastly fewer entries at a given level, but vastly increase your directory depth. Basically if you're manually scanning the heirarchy for anything but a consistency check or garbage collection you've already lost.
- tjalfi 6y ago
  Excel can be an excellent source of new line-of-business applications. Many of my employer's applications started out as a shared spreadsheet or Access database. Our development team worked with the users and built a web application to solve the same problem. This approach has a lot of advantages: * The market exists and has an incumbent. There's a lower risk of a write-off. * The users are open to process changes. You still have to migrate people off of the spreadsheet, though. * It's easy to add value with reporting, error checking, concurrent access, and access control. * You can import the existing data to make the transition easier. This will require a lot of data cleaning. Edited to add the following text from another post. You can cover most of the requirements with a set of fixed fields. The last 10% to 20% of the use cases requires custom reports and custom fields. Users should be able to define their own reports and run them without your involvement. They should also be able to define custom field types with validation, data entry support, etc. If your web application has these two features and other advantages then you should be able to replace Excel.
gopalv 6y ago
> Kafka and the like are clearly message buses and though they might also technically qualify as a DB ksqldb is actually a database on top of this. The thing is that they have an incrementally updated materialized view that is the table, while the event stream is similar to a WAL ("ahead of write logs?" in this case). Because eventually you can't just go over your entire history for every query.
- deleted 6y ago
  [deleted]
makach 6y ago
Oh ho ho ho. What weird things we use as a databases. I remember when I first started out as a consultant developer we were using a CMS as our data repository because someone thought that was a good idea. (It wasn't). The vendor was flown in from the states to help troubleshooting. I will never forget how he looked at me when I had to explain to him why we made so many nodes in the content tree, it was because we were using the CMS as a repository.

twic 6y ago

I once worked on a system for notifying customers of events by posting to their APIs. Events came in on a Rabbit queue and got posted. If a customer's API was down, the event would go back on the queue with a header saying to retry it after some time. You can do some sort of incantation to specifically retrieve messages with a suitable header value, to find messages which are ready to retry. We used exponential backoff, capped at one day, because the API might be down for a week. I didn't think of RabbitMQ as a database when I started that work, but it looked a lot like it by the time I finished.

sida 6y ago
lol api could be down for a week? What?
- csours 6y ago
  Not everything is or needs to be webscale.
layoutIfNeeded 6y ago
Sounds like delay line memory.

Hamuko 6y ago

>This might be a stupid question, but surely no one thinks of RabbigMQ as a database right? Arguably the world's most popular database is Microsoft Excel.

deleted 6y ago
[deleted]
mr_toad 6y ago
It’s definitely popular, that much is inarguable.

detaro 6y ago

I interpret it as they'd probably not call it a database, but they might use it in places where a database would be better suited, and effectively store data in it.

danpalmer 6y ago

It's both. It's best used when it's being used as a message broker, but any sufficiently advanced message broker will need many of the features of a database – durability of messages, querying in various ways, etc. I think it's reasonable to think of it as a very specialised database.