5 ms·

Thats not really what this article is about. Their problem wasn't throughput. What's the size of all the data in your MongoDB instance? And what's the latency i

Thats not really what this article is about. Their problem wasn't throughput. What's the size of all the data in your MongoDB instance? And what's the latency in your reads? In the big data world the "complexity" of the data doesn't really mean much. It's just bytes.

lmilcin 5y ago
> What's the size of all the data in your MongoDB instance? 3x12TB > In the big data world the "complexity" of the data doesn't really mean much. Oh how wrong you are. It is much easier to deal with data when the only thing you need to do is to just move it from A to B. Like "find who should see this message, make sure they see it". It is much different when you have large, rich domain model that runs tens of thousands of business rules on incoming data and each entity can have very different processing depending on its state and the event that came. I am writing whole applications just to data-mine our processing flow just to be able to understand a little bit of what is happening there. At that traffic you can't even log anything for each of the transactions. You have to work indirectly through various metrics, etc.
- phailhaus 5y ago
  Again, the article is not about throughput. How fast can you search across all historical trades?
  - lmilcin 5y ago
    As an example, there are bitemporal queries like "for the given population of trades specified by following rules, find the set of trades that met the rules at a particular point in time, based on our knowledge at another given point in time". Also trades are versioned (are a stream of business events from trading system), then have amendments (each event may be amended in the future but the older version must be preserved). Our system can also amend the data (for example to add some additional data to the trade later). All this causes trades to be a tree of immutable versions you need to comb through. A trade can have anywhere from 1 to 30k versions. This takes about 20 seconds. The process opens about 200 connections to the cluster and transfers data at about 2-4GB/s.
    - phailhaus 5y ago
      How many simultaneous queries of that nature can the system handle?
      - fao_ 5y ago
        Are you not sure that financial data "with hundreds of fields" is more complex than chat data which has a relatively linear threading and only a handful of fields?
        lmilcin 5y ago
        Actually, our threading is quite simple. There is exactly as many threads (that do anything) as CPU cores.
        gpderetta 5y ago
        I think parent meant threading as in message threads.
        lmilcin 5y ago
        Just recently Symphony switched order in which I have received two messages from a colleague. This completely changed the meaning of the message and we got in an argument that was only resolved after I have posted screenshot of what he wrote. It seems, the threading might not be that simple after all.
        fao_ 5y ago
        I meant threading in as much as the connections or links between message data All discord needs to store is: { channel_id, message_id, user_id, content, reply_to, datetime } these days they added an extra thread_id field, sure. But the data itself is blisteringly uncomplex and there is only a single way to display it (ordered by time, i.e. the 'thread')
        phailhaus 5y ago
- kevinsundar 5y ago
  Nice that's a pretty decent size, curious on the latency still. Thats the primary problem for a real time chat app. Complexity of data and running business rules on it is not a data store problem though, that's a compute problem. It's highly parallelizable and compute is cheap. For reference, my team runs transformations on about 1 PB of (uncompressed) data per day with 3 spark clusters, each with 50 nodes. We've got about 70ish PB of (compressed) data queryable. All our challenges come from storage, not compute.
  - lmilcin 5y ago
    The latency is a complex topic. In order to be able to run so much stuff on MongoDB, we almost never run single queries to the database. If I fetch or insert trade data, I probably run a query for 10 thousand trades at the same time. So what happens is, as data comes from multiple directions it is being batched (for example 1-10 thousand at a time), split into groups that can be processed together in a roughly similar process, and then travels the pipeline as a single batch which is super important as it allows amortizing some of the costs. Also the processing pipeline has many, many steps in it. A lot of them have buffers inbetween so that steps don't get starved for data. All this causes latency. I try to keep it subsecond but it is a tradeoff between throughput and latency. It could have been implemented better, but the implementation would be complex and inflexible. I think having clear, readable, flexible implementation is worth a little bit of tradeoff in latency. As to storage being source of most woes, I fully agree. In our case it is trying to deal with bloat of data caused by business wanting to add this or that. All this data causes database caches to be less effective, requires more network throughput, more CPU for parsing/serializing, needs to be replicated, etc. So half the effort is constantly trying to figure out why they want to add this or that and is it really necessary or can be avoided somehow.
    - fao_ 5y ago
      I'm not sure why you're being downvoted when you're a domain expert talking about your craft? People have weird hangups on hacker news, it seems
      - 5y ago

lmilcin 5y ago

> What's the size of all the data in your MongoDB instance? 3x12TB > In the big data world the "complexity" of the data doesn't really mean much. Oh how wrong you are. It is much easier to deal with data when the only thing you need to do is to just move it from A to B. Like "find who should see this message, make sure they see it". It is much different when you have large, rich domain model that runs tens of thousands of business rules on incoming data and each entity can have very different processing depending on its state and the event that came. I am writing whole applications just to data-mine our processing flow just to be able to understand a little bit of what is happening there. At that traffic you can't even log anything for each of the transactions. You have to work indirectly through various metrics, etc.

phailhaus 5y ago
Again, the article is not about throughput. How fast can you search across all historical trades?
- lmilcin 5y ago
  As an example, there are bitemporal queries like "for the given population of trades specified by following rules, find the set of trades that met the rules at a particular point in time, based on our knowledge at another given point in time". Also trades are versioned (are a stream of business events from trading system), then have amendments (each event may be amended in the future but the older version must be preserved). Our system can also amend the data (for example to add some additional data to the trade later). All this causes trades to be a tree of immutable versions you need to comb through. A trade can have anywhere from 1 to 30k versions. This takes about 20 seconds. The process opens about 200 connections to the cluster and transfers data at about 2-4GB/s.
  - phailhaus 5y ago
    How many simultaneous queries of that nature can the system handle?
    - fao_ 5y ago
      Are you not sure that financial data "with hundreds of fields" is more complex than chat data which has a relatively linear threading and only a handful of fields?
      - lmilcin 5y ago
        Actually, our threading is quite simple. There is exactly as many threads (that do anything) as CPU cores.
        gpderetta 5y ago
        I think parent meant threading as in message threads.
        lmilcin 5y ago
        Just recently Symphony switched order in which I have received two messages from a colleague. This completely changed the meaning of the message and we got in an argument that was only resolved after I have posted screenshot of what he wrote. It seems, the threading might not be that simple after all.
        fao_ 5y ago
        I meant threading in as much as the connections or links between message data All discord needs to store is: { channel_id, message_id, user_id, content, reply_to, datetime } these days they added an extra thread_id field, sure. But the data itself is blisteringly uncomplex and there is only a single way to display it (ordered by time, i.e. the 'thread')
      - phailhaus 5y ago
kevinsundar 5y ago
Nice that's a pretty decent size, curious on the latency still. Thats the primary problem for a real time chat app. Complexity of data and running business rules on it is not a data store problem though, that's a compute problem. It's highly parallelizable and compute is cheap. For reference, my team runs transformations on about 1 PB of (uncompressed) data per day with 3 spark clusters, each with 50 nodes. We've got about 70ish PB of (compressed) data queryable. All our challenges come from storage, not compute.
- lmilcin 5y ago
  The latency is a complex topic. In order to be able to run so much stuff on MongoDB, we almost never run single queries to the database. If I fetch or insert trade data, I probably run a query for 10 thousand trades at the same time. So what happens is, as data comes from multiple directions it is being batched (for example 1-10 thousand at a time), split into groups that can be processed together in a roughly similar process, and then travels the pipeline as a single batch which is super important as it allows amortizing some of the costs. Also the processing pipeline has many, many steps in it. A lot of them have buffers inbetween so that steps don't get starved for data. All this causes latency. I try to keep it subsecond but it is a tradeoff between throughput and latency. It could have been implemented better, but the implementation would be complex and inflexible. I think having clear, readable, flexible implementation is worth a little bit of tradeoff in latency. As to storage being source of most woes, I fully agree. In our case it is trying to deal with bloat of data caused by business wanting to add this or that. All this data causes database caches to be less effective, requires more network throughput, more CPU for parsing/serializing, needs to be replicated, etc. So half the effort is constantly trying to figure out why they want to add this or that and is it really necessary or can be avoided somehow.
  - fao_ 5y ago
    I'm not sure why you're being downvoted when you're a domain expert talking about your craft? People have weird hangups on hacker news, it seems
    - 5y ago