5 ms·
As an example, there are bitemporal queries like "for the given population of trades specified by following rules, find the set of trades that met the rules at
by lmilcin 5y ago
As an example, there are bitemporal queries like "for the given population of trades specified by following rules, find the set of trades that met the rules at a particular point in time, based on our knowledge at another given point in time". Also trades are versioned (are a stream of business events from trading system), then have amendments (each event may be amended in the future but the older version must be preserved). Our system can also amend the data (for example to add some additional data to the trade later). All this causes trades to be a tree of immutable versions you need to comb through. A trade can have anywhere from 1 to 30k versions.
This takes about 20 seconds. The process opens about 200 connections to the cluster and transfers data at about 2-4GB/s.
- phailhaus 5y agoHow many simultaneous queries of that nature can the system handle?
- fao_ 5y agoAre you not sure that financial data "with hundreds of fields" is more complex than chat data which has a relatively linear threading and only a handful of fields?
- lmilcin 5y agoActually, our threading is quite simple. There is exactly as many threads (that do anything) as CPU cores.
- gpderetta 5y agoI think parent meant threading as in message threads.
- lmilcin 5y agoJust recently Symphony switched order in which I have received two messages from a colleague. This completely changed the meaning of the message and we got in an argument that was only resolved after I have posted screenshot of what he wrote. It seems, the threading might not be that simple after all.
- fao_ 5y agoI meant threading in as much as the connections or links between message data All discord needs to store is: { channel_id, message_id, user_id, content, reply_to, datetime } these days they added an extra thread_id field, sure. But the data itself is blisteringly uncomplex and there is only a single way to display it (ordered by time, i.e. the 'thread')
- phailhaus 5y agoI'm asking about how your system scales to the number of queries, but you seem to be taking every question personally. You seem to really want to make sure everyone knows that you think Discord's problems are easy to solve. I'm not saying Discord is more complicated, but you're not really giving enough information to prove that Discord's problems are a subset of yours. Do you support more simultaneous queries than Discord?
- fao_ 5y agoI think you need to recheck my username, you have mistaken me with another poster in the thread.
- michael_j_ward 5y agoThanks for all the great insight. Did you consider an explicitly bitemporal database like crux[0]? [0] https://github.com/juxt/crux https://github.com/juxt/crux?
- lmilcin 5y agoOur application predates crux. As to crux, I can't comment because I did not know about it when I was actively searching for a product to do this.