bluestreak

15 ms·

1. 61.▲
   by bluestreak 6y ago

   QuestDB is quite intensive on cores. For example it is not a good idea to put two threads on hyperthreded cores that share the same physical core. Also it isn't good idea to put threads on cores that belong to different physical CPUs o
2. 62.▲
   by bluestreak 6y ago

   I wonder what does this query compile to? In terms of C-code equivalent? SELECT cab_type, count(*) FROM trips GROUP BY cab_type; From execution time it seems to me that this is a straight sum() of 32-bit integers. "cab_ty
3. 63.▲
   by bluestreak 6y ago

   Thanks! This is something we built in house. In fact, ‘mpsq’ did all of that. All credit should go to him!
4. 64.▲
   by bluestreak 6y ago

   if you can send all three in the same message, Influx Line Protocol for example, we will store them as 3 columns in one table. Does this help?
5. 65.▲
   by bluestreak 6y ago

   Our best performance currently is in C++ code and LLVM is something we are considering using to compute expressions, such as predicates and select clause items. This is most likely to be way faster than what we can currently do in Java. Wha
6. 66.▲
   by bluestreak 6y ago

   OLTP is not a good fit, if your workflow consists from INSERT/UPDATE/DELETE statements
7. 67.▲
   by bluestreak 6y ago

   :) Eventually both. We are starting with baby steps, e.g. get data from A to B quickly and reliably. Replication/HA will be first of course. Then we want to scale queries across multiple hosts. Since all nodes have the same data - they
8. 68.▲
   by bluestreak 6y ago

   The strategy is to multicast data to several nodes simultaneously. Data packets are sequence to allow receiver identify data loss. When loss is detected receiver finds breathing space to send a NACK. The packet and the nack would identify m
9. 69.▲
   by bluestreak 6y ago

   Over the network streaming is not yet available. Someone has mentioned Kafka support, how useful would that be to stream processed (aggregated) values and/or actual table changes?
10. 70.▲
    by bluestreak 6y ago

    Thank you for the kind words!
11. 71.▲
    by bluestreak 6y ago

    This is incredibly useful, thank you! It would be awesome if we could chat more about your use cases at some point. Drop us a line on hello at questdb.io or join our slack. Whichever is easier for you.
12. 72.▲
    by bluestreak 6y ago

    We store "dimensions" as table columns with no artificial limits on column count. If you able to send all dimensions in the same message, they will be stored on one row of data. If dimensions are sent as separate messages, current
13. 73.▲
    by bluestreak 6y ago

    Thank you for the kind words and constructive feedback. We are here to build on feedback like this. Grafana plugin is coming soon.
14. 74.▲
    by bluestreak 6y ago

    We are working on building a solid PostgreSQL support insofar as allowing ODBC driver to execute this type of query from Excel. This is work in progress with not that much left on it.
15. 75.▲
    by bluestreak 6y ago

    There will be two different flavors of replication: - TCP-based replication for WAN - UDP-based replication for LAN and high traffic environments We are currently building foundation elements of this replication, such as column-first and pa
16. 76.▲
    by bluestreak 6y ago

    It has been asked here: https://github.com/questdb/questdb/issues/261 . Definitely on our road map. It would be good if you could share your story why you need arrow?
17. 77.▲
    by bluestreak 6y ago

    Thank you! We have quite active slack and we try to listen to and help our community there. Feel free to join!
18. 78.▲
    by bluestreak 6y ago

    thank you! - replication is in the works, this is going to be both TCP and UDP based, column-first, very fast. - yes, benchmarks are indeed are done on second pass over the mmaped pages. First pass would trigger IO, which is OS-driven and d
19. 79.▲
    by bluestreak 6y ago

    thank you for sharing! The core of memory management is abstracted away. All of the query execution logic is unaware of the source of memory pointer. That said we are still learning and really appreciate your feedback. There are some places
20. 80.▲
    by bluestreak 6y ago

    Java was the starting point. Back in the day Rust wasn't a thing and C++ projects were quite expensive to maintain. What Java does for us is IDE support, instant compilation time and super easy test coverage. For things that does requi
21. 81.▲
    by bluestreak 6y ago

    "normal" database does not preserve the order of data as it comes in. To get the data out you have to constantly rely on indexes or "order by" constantly to get chronological order back. Time series database should maint
22. 82.▲
    by bluestreak 6y ago

    thanks! What differentiates us from other time series databases is the performance. Both for ingestion and queries. For example we can ingest application performance metrics via Influx Line Protocol and query them via SQL and both should fa
23. 83.▲
    by bluestreak 6y ago

    thank you, life is an interesting experience :) I used to work with Fabio, Vortexa CEO and had to turn down an offer of being first employee there to focus on QuestDB. They are an absolute awesome bunch of guys and deserve every bit of thei
24. 84.▲
    by bluestreak 6y ago

    http://try.questdb.io:9000/
25. 85.▲
    Launch HN: QuestDB (YC S20) – Fast open source time series database

    357 points by bluestreak 6y ago|167 comments
26. 86.▲
    by bluestreak 6y ago

    SQL optimiser works out that this is a cross join and renders execution path as such. Cross join implementation is streaming with pre-determined result set size. We would not be able to render this much data this quickly without optimiser :
27. 87.▲
    by bluestreak 6y ago

    the use of where clause right now turns off vectorization completely is falls back on simple single threaded execution. This is the reason why the query is slow right now. We are going to vectorize 'where' really soon.
28. 88.▲
    by bluestreak 6y ago

    multi-key aggregation is next on our list to optimise. It will perform similar to single key one in most cases.
29. 89.▲
    by bluestreak 6y ago

    This demo is showing raw database performance. None of the results are cached or pre-calculated.
30. 90.▲
    by bluestreak 6y ago

    Thank you for the kind words. Databases can be daunting but for a variety of reasons. In my experience, it's not so much because of the sheer amount of features and function that need to be implemented. It's because you have to ma