zX41ZdbW

19 ms·

1. 301.▲
   by zX41ZdbW 2y ago

   It is very easy to get this dataset directly from HN API. Let me just post it here: Table definition: CREATE TABLE hackernews_history ( update_time DateTime DEFAULT now(), id UInt32, deleted UInt8,
2. 302.▲
   by zX41ZdbW 2y ago

   Thank you! Let's take it as a comment about comparing styles in different database CLIs. Some have \-commands, and some have .-commands, which I found hacky - because it is a micro language notation in addition to the actual language.
3. 303.▲
   by zX41ZdbW 2y ago

   I'm the author of ClickHouse, and this topic is interesting to me.
4. 304.▲
   by zX41ZdbW 2y ago

   I think the right way to avoid this problem is to avoid using ABI at runtime or build time. At runtime, it means - don't use shared libraries. At build time, it means - build every library from the source, don't use pre-built arti
5. 305.▲
   by zX41ZdbW 2y ago

   Umbra in ClickBench: https://github.com/ClickHouse/ClickBench/pull/161 The initial submission didn't reproduce successfully due to a segmentation fault in an attempt to restart it after data loading. But
6. 306.▲
   by zX41ZdbW 2y ago

   In fact, the opposite is true. While small datasets can be handled on GPU (although there are no good GPU databases comparable in performance to ClickHouse), large datasets don't fit, and unless there is a large amount of computation p
7. 307.▲
   by zX41ZdbW 2y ago

   Interesting if there are benefits over MergeTree (ClickHouse's data format).
8. 308.▲
   by zX41ZdbW 3y ago

   I use this technique in a few of my projects: https://reversedns.space/ (a map of the Internet) - Morton curve is used as a visualization tool; https://adsb.exposed/ (a visualizer of air traffic) - Morton c
9. 309.▲
   by zX41ZdbW 3y ago

   Interesting about "Timeouts (only 30s???)" - most likely, this is a limitation configured explicitly for a user on your server. You can set it up with the `max_execution_time`, and by default, it is unlimited. For example, I'
10. 310.▲
    by zX41ZdbW 3y ago

    Yes, it is possible to connect to the database directly and run an arbitrary query. Example: $ clickhouse-client --user website --host kvzqttvc2n.eu-west-1.aws.clickhouse-staging.com clickhouse-cloud :) SELECT t, desc, count() AS
11. 311.▲
    by zX41ZdbW 3y ago

    Could you please clarify the details? I took extra care to provide all the needed attributions and credits and I believe it is complete and sufficient. If I missed something, please describe it at https://github.com/ClickHou
12. 312.▲
    by zX41ZdbW 3y ago

    You can find the attribution and the link to the details at the bottom right corner of the main page. It links to the documentation: https://github.com/ClickHouse/adsb.exposed , which provides the full details. Addition
13. 313.▲
    by zX41ZdbW 3y ago

    There are two data sources: adsb.lol and adsbexchange. The first is updated each day from https://github.com/adsblol/globe_history_2024 . The second should be updated each month (they provide only sample data for the fi
14. 314.▲
    by zX41ZdbW 3y ago

    We bought clickhouse.com from the previous owner 2.5 years ago, who used it for some ad network - it was in some block lists, but I hope we managed to clean up most of them.
15. 315.▲
    by zX41ZdbW 3y ago

    I keep the dashboard open :) So far ok with ~2000 QPS.
16. 316.▲
    Show HN: ADS-B visualizer(adsb.exposed)

    339 points by zX41ZdbW 3y ago|76 comments
17. 317.▲
    by zX41ZdbW 3y ago

    A visualizer and analytics app for air traffic data: https://adsb.exposed/ Although it is based on another app, also made over a weekend: https://reversedns.space/ But then it took a few months before I man
18. 318.▲
    by zX41ZdbW 3y ago

    Sounds totally redundant to me. You can write all location updates into ClickHouse, and the problem is solved. As a demo, I've recently implemented a tool to browse 50 billion airplane locations: https://adsb.exposed/
19. 319.▲
    by zX41ZdbW 3y ago

    Many ideas look like they were influenced by ClickHouse, and some are direct copies. I'm surprised they didn't provide references to ClickHouse, where the implementations are proven in production in the first place.
20. 320.▲
    by zX41ZdbW 3y ago

    In our repository, I've set up a few hard limits: each translation unit cannot spend more than a certain amount of memory for compilation and a certain amount of CPU time, and the compiled binary has to be not larger than a certain siz
21. 321.▲
    by zX41ZdbW 3y ago

    I want to make it updateable, so we can see how the DNS records change over time. Also, I want to map various scans, similar to https://blog.benjojo.co.uk/post/scan-ping-the-internet-hilbe...
22. 322.▲
    by zX41ZdbW 3y ago

    Thank you for posting! There is also a presentation about it: https://presentations.clickhouse.com/meetup85/app/
23. 323.▲
    by zX41ZdbW 3y ago

    I did a similar thing recently: https://twitter.com/tbragin/status/1749580473800556779 And it is entirely trivial: ITEMS=$(clickhouse-local --query "SELECT arrayStringConcat(items, ',') FROM u
24. 324.▲
    by zX41ZdbW 3y ago

    I've recently implemented a map of the Internet using a similar technique: https://reversedns.space/ In ClickHouse, we have almost every mentioned technique: H3 and S2, geohashes, and indexing by space-filling curves.
25. 325.▲
    by zX41ZdbW 3y ago

    I created this table on the European server, but forgot to enable replication to the US.
26. 326.▲
    by zX41ZdbW 3y ago

    The website has a picture https://arbitrand.com/fpga.png but it is difficult to understand what it represents. Needs clarification.
27. 327.▲
    by zX41ZdbW 3y ago

    There is a constantly updated table in ClickHouse with no login required for querying: https://play.clickhouse.com/play?user=play#U0VMRUNUIGNvdW50K...
28. 328.▲
    by zX41ZdbW 3y ago

    I have a project dedicated to this topic: https://github.com/ClickHouse/ClickBench It is important to explain the limitations of a benchmark, provide a methodology, and make it reproducible. It also has to be simple en
29. 329.▲
    by zX41ZdbW 3y ago

    ClickHouse can work with S3 as a main storage. This is possible because a table is a set of immutable data parts. Data parts can be written once and deleted, possibly as a result of a background merge operation. S3 API is almost enough, exc
30. 330.▲
    by zX41ZdbW 3y ago

    Observable integrates really well with ClickHouse using its REST API, like in this example: https://observablehq.com/@stas-sl/github-issues-survival-ana... But I didn't try the new Observable Framework - interesti