3 ms·

Tangential, but I'm curious about the installation command: curl https://clickhouse.com/ | sh It seems like it just detects curl('s user agent?) and serve

by notamy 4y ago

Tangential, but I'm curious about the installation command: curl https://clickhouse.com/ | sh It seems like it just detects curl('s user agent?) and serves up the script, serving HTML otherwise (ex. `wget -O- ...` would print HTML). I wonder what the design decisions here were.

afandian 4y ago
curl -vvv https://clickhouse.com/ ... > user-agent: curl/7.79.1
- notamy 4y ago
  Yes, that's obvious. The question is more WHY -- why serve curl detection from the root domain? Why only detect curl? ...
  - slt2021 4y ago
    easier to remember download link?
    - hk__2 4y ago
      There’s nothing in that link that says "I’m a download link" and I don’t think you would remember that the site you have open in your browser also serves as the download link if you curl it with the right user-agent.
zX41ZdbW 4y ago
Yes, it is detecting curl by the user agent - by using the "page rules" in Cloudflare. Cloudflare is good for traffic management rules. I've created this distribution channel as an experiment and for my own need - I use it to download and install ClickHouse without the need of package manager. So I don't need to run "apt update" to install ClickHouse. It works if the package manager is broken, and it works well on very old servers, and rare Linux distributions. It is using the fresh build from the master branch. I have also created a website for my team to monitor the status of the continuous integration system: https://aretestsgreenyet.com/ https://aretestsgreenyet.com/ We also have deb, rpm, tgz and Docker as wrappers for the release builds.
- notamy 4y ago
  > I've created this distribution channel as an experiment and for my own need - I use it to download and install ClickHouse without the need of package manager. So I don't need to run "apt update" to install ClickHouse. It works if the package manager is broken, and it works well on very old servers, and rare Linux distributions. Makes perfect sense! Thanks for the explanation :D
- CoolCold 4y ago
  Interesting - for me broken package manager is indicator of that server to be fixed or reprovisioned, I'd be wondered to see some DB running on such kind of server - highly likely it's missing updates as well
  - zX41ZdbW 4y ago
    Totally agree. But it's usually more complicated. For example, MongoDB supports Ubuntu 20.04, but not 22.04: https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/ https://www.mongodb.com/docs/manual/tutorial/install-mongodb... Or another example - folks in a large company successfully running Ubuntu 12.04 in production for 10 years and cannot upgrade it because some Perl dependencies of their amalgamation of microservices working slightly differently.
- cheeaun 4y ago
  Hmm, how do you create this "page rule" in Cloudflare? I can't seem to figure it out.
deleted 4y ago
[deleted]
goodpoint 4y ago
That's really sketchy. https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...
- mananaysiempre 4y ago
  Sketchy would be detecting bash and serving a different page depending on that; this is only a bit hackish. Now that I’m thinking about it, normal browsers usually ask for text/html or text/* as a preference don’t they? While curl and wget just say Accept: *. Then content negotiation could achieve the same result per the actual HTTP semantics, without any hacks involving the User-Agent header.

by notamy 4y ago

afandian 4y ago

curl -vvv https://clickhouse.com/ ... > user-agent: curl/7.79.1

notamy 4y ago
Yes, that's obvious. The question is more WHY -- why serve curl detection from the root domain? Why only detect curl? ...
- slt2021 4y ago
  easier to remember download link?
  - hk__2 4y ago
    There’s nothing in that link that says "I’m a download link" and I don’t think you would remember that the site you have open in your browser also serves as the download link if you curl it with the right user-agent.

zX41ZdbW 4y ago

Yes, it is detecting curl by the user agent - by using the "page rules" in Cloudflare. Cloudflare is good for traffic management rules. I've created this distribution channel as an experiment and for my own need - I use it to download and install ClickHouse without the need of package manager. So I don't need to run "apt update" to install ClickHouse. It works if the package manager is broken, and it works well on very old servers, and rare Linux distributions. It is using the fresh build from the master branch. I have also created a website for my team to monitor the status of the continuous integration system: https://aretestsgreenyet.com/ https://aretestsgreenyet.com/ We also have deb, rpm, tgz and Docker as wrappers for the release builds.

notamy 4y ago
> I've created this distribution channel as an experiment and for my own need - I use it to download and install ClickHouse without the need of package manager. So I don't need to run "apt update" to install ClickHouse. It works if the package manager is broken, and it works well on very old servers, and rare Linux distributions. Makes perfect sense! Thanks for the explanation :D
CoolCold 4y ago
Interesting - for me broken package manager is indicator of that server to be fixed or reprovisioned, I'd be wondered to see some DB running on such kind of server - highly likely it's missing updates as well
- zX41ZdbW 4y ago
  Totally agree. But it's usually more complicated. For example, MongoDB supports Ubuntu 20.04, but not 22.04: https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/ https://www.mongodb.com/docs/manual/tutorial/install-mongodb... Or another example - folks in a large company successfully running Ubuntu 12.04 in production for 10 years and cannot upgrade it because some Perl dependencies of their amalgamation of microservices working slightly differently.
cheeaun 4y ago
Hmm, how do you create this "page rule" in Cloudflare? I can't seem to figure it out.

deleted 4y ago

[deleted]

goodpoint 4y ago

That's really sketchy. https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...

mananaysiempre 4y ago
Sketchy would be detecting bash and serving a different page depending on that; this is only a bit hackish. Now that I’m thinking about it, normal browsers usually ask for text/html or text/* as a preference don’t they? While curl and wget just say Accept: *. Then content negotiation could achieve the same result per the actual HTTP semantics, without any hacks involving the User-Agent header.