11 ms·

Microsoft cURLs too

268 points by TXCSwe 9y ago

oblio 9y ago
Either the Curl developers are at fault somewhere, which I somehow doubt, or distributions are really special snowflakes, which I also doubt, or software distribution in the Open Source world is, in my opinion, flawed: > Finally, I’d like to add that like all operating system distributions that ship curl (macOS, Linux distros, the BSDs, AIX, etc) Microsoft builds, packages and ships the curl binary completely independently from the actual curl project. Why would everyone rebuild it? There are some security considerations (matching source and binary; disabling "dangerous" stuff) and some feature considerations (disable stuff you don't need to reduce resource usage - maybe), but conceptually this seems so wrong to me. Conceptually I'd want downstream packagers to talk to upstream developers so that upstream has reasonable defaults and settings and I'd want packagers to just package and make the package follow distribution conventions. But rebuild seems overkill. Maybe I'm missing something obvious?
- lwf 9y ago
  Almost all Linux distributions rebuild upstream software from source -- this ensures everything is built from the same toolchain (gcc/libc etc), and that the binaries distributed match the source. It also allows for ease of patching in a stable release -- generally it's preferred to just fix specific high-impact bugs rather than moving to a new upstream version, which might introduce regressions. (Context: I'm a Debian developer and on the Ubuntu MOTU team)
  - digi_owl 9y ago
    And then there is the whole dependencies shitstorm, were far too many upstreams have a bad habit of breaking APIs etc as they see fit. There are ways to work around it, but it gets messy quickly. And rather than clean up their act they start championing things like Flatpak, that is basically a throwback to the DOS days of everything living in their own folder tree with a bit of souped up chroot thrown on top. I really expect that if the likes of flatpak becomes mainstream in the Linux world having some flaw being found in a lib somewhere will produce a stampede of updates because every damn project crammed in a copy to make sure it was present.
  - gerdesj 9y ago
    "Almost all Linux distributions rebuild upstream software from source" On Gentoo et al, the end user does the building. OK, yes the ebuilds are recipes but I've lost count of the times I've used epatch_user (https://wiki.gentoo.org/wiki//etc/portage/patches https://wiki.gentoo.org/wiki//etc/portage/patches). You have a near infinite choice of ways to destroy your system, what with USE flags, mixed stable/unstable and all the other crazy stuff. Despite that, my Gentoo systems have been surprisingly stable. In winter an update world session on a laptop keeps you (very) warm 8) (wrt "Context": Ta for your work)
- kogepathic 9y ago
  > Maybe I'm missing something obvious? There are many different package managers depending on which Linux you're running. Debian and derivatives use apt, Red Hat uses yum or dnf, SuSE uses yast, Gentoo has emerge, Arch has pacman... So each package manager needs to build their own package and it's easier to recompile from source than slice and dice a binary. Also distributions will install the binaries, shared libraries, man pages, etc to different locations (some to /usr, some to /use/local, etc) which is also easier to define at configuration since most autoconf/make files support this already. Finally they might want to add patches for distribution specific features or quirks. Or maybe they compile with ulibc instead of glibc. There are many valid reasons why distributions would and should take the upstream source and build/package it themselves.
- 0x0 9y ago
  Especially in the Linux world, you can't expect upstream to supply binaries for all possible architectures and configuration options. For example, you might be running on armv5 with libressl. Or you may be running on sparc64 with openssl. Or you may be running on 32bit windows with WinSSL. An upstream is unlikely to have access to build all possible configurations and provide binaries every time a security patch is announced. Also, as a distro provider, you will want to be sure you can build the application yourself, because you might want to ship an updated library dependency that is ABI-incompatible and so you must be able to rebuild the consumers of these libraries. For example curl, in the case of openssl.
- pjc50 9y ago
  Rebuild is necessary for things as trivial as changing the default install path. It's absolutley standard; a Linux distro that doesn't rebuild packages somewhere would almost not count as a distro.
  - chungy 9y ago
    Hmm. All the various Ubuntu derivatives (eg, Hanna Montana Edition or Christian Edition, or even Kubuntu and MATE Editions) that change some defaults without rebuilding packages... don't count as distributions? This is somewhat tongue-in-cheek, but it's actually a question I don't have a solid yes or no answer to. I can see it both ways.
    - Spivak 9y ago
      If your users are pulling software direct from the upstream distribution then you're not a separate distribution of Linux, you're a 'spin', 'edition', or an installer. This does get a little murky when some of the packages are distributed directly but others are pulled from upstream like Antergos but the changes are so minor that I would still consider it an Arch spin.
- dfox 9y ago
  Apart from various technical reasons the workflow of essentially all linux package managers is built on the notion that the binary package is built by some well defined (and repeatable, at least in the "does same thing" sense) process from some kind of source package. Also the souce package format usually contains mechanism that allows distributions to comply with copy-left licenses while also cleanly documenting which modifications were made to the package from upstream version (it is sort of special purpose versioning system).
- geofft 9y ago
  In order to make sure you actually have the freedom to modify the software, you want packagers to have the ability to rebuild the package from source - and preferably using just their OS as a build environment, not a black-box build environment from upstream like a Docker configured just so and make sure they have an equivalent binary. Even if there's nothing to patch today, there might be something to patch tomorrow. I have seen many times Debian packagers try to build something from upstream and find that it just does not build anywhere other than the maintainer's computer. The fact that Debian requires that every package it ships can be built buy anyone in a generic environment is immensely valuable to free software, even if nobody used the binaries that Debian built. (And to be clear, other distros do the same, I'm just most familiar with Debian.) I'd agree that in an ideal world, all the patches would be upstream and the binary would not just be equivalent but bit-for-bit reproducible. Some practical reasons why it wouldn't be are that various dependencies are of slightly different versions (e.g., one distro manages to get a new libc uploaded a little bit before another), that downstream conventions are different in different distros (e.g., Red Hat-style distros use /lib64 and Debian-style distros use /lib/x86_64-linux-gnu), or that a dependency has some shortcoming which many but not all distros patch in the same way, and the patch impacts things that use the dependency (e.g., upstream OpenSSL <1.1 does not have symbol versions, but most Linux distros patch them in). Yes, in an ideal world, all these things would be resolved, but there are going to be so many tiny things like this that come up that having infrastructure to accommodate them is the right plan.
- bartkmq 9y ago
  I can think of a few reasons (1) Use of a different libc (alpine linux with musl) (2) most builds are not reproducible, rebuilds are needed for security reasons (3) non-rolling release distros (-> most distros) fork the upstream projects to backport fixes for their older releases (4) Different filesystem structure (Gobo Linux) (6) Most distros want to use the build system associated with their own package manager There are probably many more than that. Most distros don't even try to stay close the upstream repo and instead maintain a lot of patches. In the future we will most likely have an distro-specific basic system build and container apps (snap, appimage, flatpak) build directly by upstream on non-server systems.
  - digi_owl 9y ago
    > In the future we will most likely have an distro-specific basic system build and container apps (snap, appimage, flatpak) build directly by upstream on non-server systems. I sure hope not. A better system would be something like NIX/GUIX or even Gobolinux, that give us a single package as today, but with the option of installing multiple versions in parallel if upstream has screwed up the API (again). Flatpak and like will just be and excuse for upsteam to bundle everything and the kitchen sink, resulting in bloat and having to update a mass of paks rather than individual libs in case a flaw is found.
    - bartkmq 9y ago
      You've got good points and I admit I'm split on this issue. In theory i would rather have everything handled by the package manager but container apps provide a few useful advantages: - sandboxing/isolation out of the box - being able to report bugs directly to upstream - It's easier to distribute a small app to all distributions (for some value of all)* - much easier to distribute proprietary applications (subjective advantage) - much easier to install old versions of an application (sometimes needed) - it's possible to record the bandwidth/ram/cpu usage; with a standard package that's quiet difficult
  - laumars 9y ago
    7) CPU architecture, 8) output binary format (though these days people generally only use ELF or PE), 9) compile time options (eg some packages will allow you to choose which TLS library to use at compile time) 10) hardware specific optimisations Basically just a plethora of reasons
- megous 9y ago
  Have you ever been an upstream? What you're suggesting is awfully wasteful. You'd get endless requests for "please add build for XXX" if you ever distribute any binary. Where XXX is anything from arbitrary versions of arbitrary distribution, or any kind of non-linux OSes. And all that for i686, x86_64, various ARMs,... Who has time for that? If you state you only distribute source code and perhaps binary for Windows, you will not get bothered ever again. Better let the building be done by people who are doing building all day en masse for a single arch, or have infrastructures set up to do builds for multiple archs easilly, than to expect every upstream to have this setup.
- zAy0LfpBZLC8mAC 9y ago
  Really, why would everyone not rebuild it? People have pointed out tons of reasons why distros do their own builds, but really I don't understand what would possibly be a reason not to!? The build system itself also is just a piece of software that gets distributed along with the source of the software to be built, and just as you can download and run curl and get a predictable result (the download of some URL), you can download and run the curl build system and get a predictable result (a cURL binary). So, in that regard, what does it matter what execution of the cURL build system your cURL binary came from? In particular with the trend towards reproducible builds, where the build result will be bit-identical between different runs of the build system if it's using the same compiler (version), I just don't see why you care!? Yeah, distros shouldn't just modify the software they package willy-nilly, but that's completely orthogonal to whether they should do their own builds. There are many reasons for applying small changes to enable integration with the distro, and in particular it's just unreasonable to expect that everyone of the thousands of upstream authors or Debian packages, say, operate machines of all ten CPU architectures that are currently supported by Debian so they can provide binaries for all architectures. So, if you want to be able to distribute software written by people who don't happen to have a System z or a MIPS machine in Debian, maintainers have to prepare packages in such a way that binaries for those architectures can be built--and if they do that, it's trivial to also build all the other architectures from the same source. Adding special cases for when a binary is already available for some architecture from upstream would be just completely pointless complexity.
- int_19h 9y ago
  For Windows, at least, the big reason to build the binaries would be to sign them with the Microsoft certificate, so that you know that the binary is authentic.
colemannugent 9y ago
Cool, now I don't need to remember the arcane incantation to download a file with Powershell. Do you think we'll see things like the good old "curl <some url> | bash" for Windows now? They still have no package manager worth using.
- ygra 9y ago
  Hasn't that been popular as iwr <some url> | iex for a while?
- pjc50 9y ago
  > curl <some url> | bash That's still not a package manager worth using.
- temp-dude-87844 9y ago
  Windows 10 has a "package manager", of course, that's quite good, but both user culture, developer disinterest, and years and years of inertia keeps it from being used for enough things to cross the network-effect side of some treshold of "worthwhile". The 'download | execute' paradigm, on the other hand, is the complete antithesis of package management, to where your desire to obtain and execute the code trumps your willingness and patience to wait until it has been vetted by your preferred package manager, and installed in a less haphazard way. I fail to see your point.
  - sjellis 9y ago
    If you are referring to the package management stuff in PowerShell, that's just a unified interface to several different package management systems, each of which handles specific types of packages. Windows has MSI, VSIX, PowerShell modules and probably some others as well, all of which are separate from Microsoft Update.
- Analemma_ 9y ago
  Yeah, “iwr” sure was hard to remember.
  - 9y ago
orf 9y ago
Curl is alright, and congratulations on this massive and very impressive step forwards, but the CLI not exactly very user friendly. httpie[1] is a great tool if you find curl invocation somewhat arcane. 1. https://httpie.org/ https://httpie.org/
- unkoman 9y ago
  curl is much more than http.
- Something1234 9y ago
  HTTPie is great, but sometimes you need curl. Curl supports a lot more protocols.
  - chrisper 9y ago
    Yes, but MS seems to disable many of them.
  - Rapzid 9y ago
    Plus, it's easy to forget that httpie is SLOW. Sometimes when testing an API the timings are worrying, until you remember and switch over to cURL to confirm everything is right with the world.
    - nicolaslem 9y ago
      I don't know why you are getting downvoted. That's true, starting a Python interpreter is a relatively expensive operation. There is a noticeable lag with most CLIs written in Python. There has been some work recently to make the interpreter start faster. I hope we will see the result in Python 3.7.
- andrew_mason1 9y ago
  curl may not have the greatest CLI, but it's a huge step up from doing the equivalent tasks with powershell. not to mention this brings windows closer and closer to nix. windows making itself more familiar to nix increases the number of ops folks that are willing to transition from nix shops to windows, increasing windows share in the server market plus, as others have pointed out, curl handles protocols besides http.
frik 9y ago
Please keep builds of Win32/Win64 plus source on your website, built by you. Every M$ exe nowadays contain some telemetry phone home stuff, extra annoying.
lettergram 9y ago
One thing I am always sure to share with collegues when we discuss curl, is the fact from the command line you can generate the underlying C code. This is pretty useful when creating a CLI for pretty much any app, and I've used it regularly to generate a CLI for an app. My post on how to do it: http://austingwalters.com/export-a-command-line-curl-command-to-an-executable/ http://austingwalters.com/export-a-command-line-curl-command...
- ww520 9y ago
  That's amazing. Never knew that. Learn something today.
- andrestc 9y ago
  This is amazing! Thanks for sharing
- 0xFFFE 9y ago
  Brilliant, thanks for sharing.
- henvic 9y ago
  Amazing tip!
- innagadadavida 9y ago
  There is one for generating goalang code as well: https://github.com/mholt/curl-to-go https://github.com/mholt/curl-to-go Also, it does not depend on libcurl.
- aplorbust 9y ago
  It wasnt until 2006 that curl added HTTP/1.1 pipelining support. Hence I always used netcat instead of curl because I utilised pipelining heavily for text/html retrieval (IME, most servers supported it). Imagine something like this with curl: curl << eof http://example.com/a.htm http://example.com/b.htm eof where curl only opens a single connection. Alas, AFAIK, pipelining is still not enabled in the curl binary. As I understand it, the --libcurl option only generates code for what is possible with the curl binary, e.g., curl_easy_init(), curl_easy_setopt(), etc. As such, it will not generate code using curl_multi_init(), curl_multi_setopt(), etc. I have to automate the code generation myself.
ape4 9y ago
Now, just ~1000 other commands to go
- shitloadofbooks 9y ago
  But they're _all_ already there via the WLSS (Bash on Ubuntu on Windows) if you want them. Not to mention that the PowerShell equivalents of a lot of *nix commands are _much_ better. "Everything is an object" is a brilliant philosophy and it's a joy to use.
  - frou_dh 9y ago
    Regardless of its merits, they blew it in terms of marketing. PowerShell is destined to be a deadend technology only used by Windows sysadmins.
    - Spivak 9y ago
      I don't think MS envisioned PS as anything other than a sysadmin tool. If they did it doesn't seem to come though with the design.
      - frou_dh 9y ago
        Doubt that. You think in the PowerShell design phase, the team would have shrugged if some MS decision maker somehow mentioned "BTW, 10 years from now, for developers, we're going to be promoting installing a Linux emulation layer and using Bash"? I bet the PS folks would have been dismayed.
- jgome 9y ago
  Just install busybox.exe from https://frippery.org/busybox/ https://frippery.org/busybox/ to some directory in your %PATH% or in C\: and run 'busybox sh'.
gregmac 9y ago
For debugging on remote servers this is pretty handy (at least, one it makes its way into the server versions). Now I have hope they'll put in a text editor that understands unix line endings.
- martin-adams 9y ago
  It's like they're trying to get a record by not doing that. For years I've been baffled why Notepad at the very least couldn't understand unix line endings.
  - eat_veggies 9y ago
    It's possible they lost the source code and literally can't. They've done it before: https://www.bleepingcomputer.com/news/microsoft/microsoft-appears-to-have-lost-the-source-code-of-an-office-component/ https://www.bleepingcomputer.com/news/microsoft/microsoft-ap...
    - nwellnhof 9y ago
      Equation Editor was developed by a third party and MS probably never had the source code in the first place: https://support.microsoft.com/en-us/help/4057882/error-when-editing-an-equation-in-office https://support.microsoft.com/en-us/help/4057882/error-when-...
    - userbinator 9y ago
      Notepad is literally a window with the standard Windows Edit control inside it, so they certainly have the source code. My guess as to why they don't care to support \n-only is that there's been very little need to; anyone who needs more advanced editing isn't going to use Notepad anyway. As the sibling comment mentions, WordPad (which is similar but with a RichEdit control) does support \n-only.
      - TazeTSchnitzel 9y ago
        If you want notepad-with-LF-support, ReactOS, the open-source Windows NT clone, has this.
chrisper 9y ago
What would be the reason behind disabling all those protocols?
- syncsynchalt 9y ago
  Probably to reduce exposure to security issues. The more code you ship, the more code you're responsible for keeping secure. Looking briefly at the list at https://curl.haxx.se/docs/security.html https://curl.haxx.se/docs/security.html I see issues for FTP (x2), IMAP, and TFTP in 2017 alone. These protocols which are outside of curl's core competency of http are likely to have less scrutiny and more bugs. While FTP shouldn't be removed from curl I don't think a protocol like TFTP or gopher is crucial, and I wouldn't mind too much if it got the axe in a distribution I used
  - pritambaral 9y ago
    HTTP is NOT curl's core competency, "transferring data with URLs" is. HTTP just happens to be the most often used in the world, and thus in curl.
nodesocket 9y ago
> They ship 7.55.1, while 7.57.0 was the latest version at the time. That’s just three releases away so I consider that pretty good. Lots of distros and others ship (much) older releases. Indeed. I am running latest High Sierra and: curl --version curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
qwerty456127 9y ago
Cool. I've been installing wget on every Windows PC in my authority since the days of Windows 98SE. Every net-enabled operating system is to have such a tool installed by default.
ComodoHacker 9y ago
On a side note, for all my daily download tasks (other than debugging some web API) I've settled with aria2[0]. It seems to support every protocol used in modern Internet and has plenty of flexibility (connection multiplexing, bandwidth control etc.). It can even serve as 24/7 torrent client managed via remote API. 0. https://aria2.github.io/ https://aria2.github.io/
- ghews 9y ago
  I also think so specilly the featrue supporting every protocol in modern Internet.
ghews 9y ago
I don't like Microsoft ,but I still wish Linux on Windows will have a brighter future.
qwerty456127 9y ago
BTW another program (and a library) that should, IMHO, be made a standard component of every modern OS (except those that would choose to exclude it for a practical reason, e.g. some of extremely-lightweight and heavily specialized embedded ones) is SQLite3

268 points by TXCSwe 9y ago

Either the Curl developers are at fault somewhere, which I somehow doubt, or distributions are really special snowflakes, which I also doubt, or software distribution in the Open Source world is, in my opinion, flawed: > Finally, I’d like to add that like all operating system distributions that ship curl (macOS, Linux distros, the BSDs, AIX, etc) Microsoft builds, packages and ships the curl binary completely independently from the actual curl project. Why would everyone rebuild it? There are some security considerations (matching source and binary; disabling "dangerous" stuff) and some feature considerations (disable stuff you don't need to reduce resource usage - maybe), but conceptually this seems so wrong to me. Conceptually I'd want downstream packagers to talk to upstream developers so that upstream has reasonable defaults and settings and I'd want packagers to just package and make the package follow distribution conventions. But rebuild seems overkill. Maybe I'm missing something obvious?

lwf 9y ago
Almost all Linux distributions rebuild upstream software from source -- this ensures everything is built from the same toolchain (gcc/libc etc), and that the binaries distributed match the source. It also allows for ease of patching in a stable release -- generally it's preferred to just fix specific high-impact bugs rather than moving to a new upstream version, which might introduce regressions. (Context: I'm a Debian developer and on the Ubuntu MOTU team)
- digi_owl 9y ago
  And then there is the whole dependencies shitstorm, were far too many upstreams have a bad habit of breaking APIs etc as they see fit. There are ways to work around it, but it gets messy quickly. And rather than clean up their act they start championing things like Flatpak, that is basically a throwback to the DOS days of everything living in their own folder tree with a bit of souped up chroot thrown on top. I really expect that if the likes of flatpak becomes mainstream in the Linux world having some flaw being found in a lib somewhere will produce a stampede of updates because every damn project crammed in a copy to make sure it was present.
- gerdesj 9y ago
  "Almost all Linux distributions rebuild upstream software from source" On Gentoo et al, the end user does the building. OK, yes the ebuilds are recipes but I've lost count of the times I've used epatch_user (https://wiki.gentoo.org/wiki//etc/portage/patches https://wiki.gentoo.org/wiki//etc/portage/patches). You have a near infinite choice of ways to destroy your system, what with USE flags, mixed stable/unstable and all the other crazy stuff. Despite that, my Gentoo systems have been surprisingly stable. In winter an update world session on a laptop keeps you (very) warm 8) (wrt "Context": Ta for your work)
kogepathic 9y ago
> Maybe I'm missing something obvious? There are many different package managers depending on which Linux you're running. Debian and derivatives use apt, Red Hat uses yum or dnf, SuSE uses yast, Gentoo has emerge, Arch has pacman... So each package manager needs to build their own package and it's easier to recompile from source than slice and dice a binary. Also distributions will install the binaries, shared libraries, man pages, etc to different locations (some to /usr, some to /use/local, etc) which is also easier to define at configuration since most autoconf/make files support this already. Finally they might want to add patches for distribution specific features or quirks. Or maybe they compile with ulibc instead of glibc. There are many valid reasons why distributions would and should take the upstream source and build/package it themselves.
0x0 9y ago
Especially in the Linux world, you can't expect upstream to supply binaries for all possible architectures and configuration options. For example, you might be running on armv5 with libressl. Or you may be running on sparc64 with openssl. Or you may be running on 32bit windows with WinSSL. An upstream is unlikely to have access to build all possible configurations and provide binaries every time a security patch is announced. Also, as a distro provider, you will want to be sure you can build the application yourself, because you might want to ship an updated library dependency that is ABI-incompatible and so you must be able to rebuild the consumers of these libraries. For example curl, in the case of openssl.
pjc50 9y ago
Rebuild is necessary for things as trivial as changing the default install path. It's absolutley standard; a Linux distro that doesn't rebuild packages somewhere would almost not count as a distro.
- chungy 9y ago
  Hmm. All the various Ubuntu derivatives (eg, Hanna Montana Edition or Christian Edition, or even Kubuntu and MATE Editions) that change some defaults without rebuilding packages... don't count as distributions? This is somewhat tongue-in-cheek, but it's actually a question I don't have a solid yes or no answer to. I can see it both ways.
  - Spivak 9y ago
    If your users are pulling software direct from the upstream distribution then you're not a separate distribution of Linux, you're a 'spin', 'edition', or an installer. This does get a little murky when some of the packages are distributed directly but others are pulled from upstream like Antergos but the changes are so minor that I would still consider it an Arch spin.
dfox 9y ago
Apart from various technical reasons the workflow of essentially all linux package managers is built on the notion that the binary package is built by some well defined (and repeatable, at least in the "does same thing" sense) process from some kind of source package. Also the souce package format usually contains mechanism that allows distributions to comply with copy-left licenses while also cleanly documenting which modifications were made to the package from upstream version (it is sort of special purpose versioning system).
geofft 9y ago
In order to make sure you actually have the freedom to modify the software, you want packagers to have the ability to rebuild the package from source - and preferably using just their OS as a build environment, not a black-box build environment from upstream like a Docker configured just so and make sure they have an equivalent binary. Even if there's nothing to patch today, there might be something to patch tomorrow. I have seen many times Debian packagers try to build something from upstream and find that it just does not build anywhere other than the maintainer's computer. The fact that Debian requires that every package it ships can be built buy anyone in a generic environment is immensely valuable to free software, even if nobody used the binaries that Debian built. (And to be clear, other distros do the same, I'm just most familiar with Debian.) I'd agree that in an ideal world, all the patches would be upstream and the binary would not just be equivalent but bit-for-bit reproducible. Some practical reasons why it wouldn't be are that various dependencies are of slightly different versions (e.g., one distro manages to get a new libc uploaded a little bit before another), that downstream conventions are different in different distros (e.g., Red Hat-style distros use /lib64 and Debian-style distros use /lib/x86_64-linux-gnu), or that a dependency has some shortcoming which many but not all distros patch in the same way, and the patch impacts things that use the dependency (e.g., upstream OpenSSL <1.1 does not have symbol versions, but most Linux distros patch them in). Yes, in an ideal world, all these things would be resolved, but there are going to be so many tiny things like this that come up that having infrastructure to accommodate them is the right plan.
bartkmq 9y ago
I can think of a few reasons (1) Use of a different libc (alpine linux with musl) (2) most builds are not reproducible, rebuilds are needed for security reasons (3) non-rolling release distros (-> most distros) fork the upstream projects to backport fixes for their older releases (4) Different filesystem structure (Gobo Linux) (6) Most distros want to use the build system associated with their own package manager There are probably many more than that. Most distros don't even try to stay close the upstream repo and instead maintain a lot of patches. In the future we will most likely have an distro-specific basic system build and container apps (snap, appimage, flatpak) build directly by upstream on non-server systems.
- digi_owl 9y ago
  > In the future we will most likely have an distro-specific basic system build and container apps (snap, appimage, flatpak) build directly by upstream on non-server systems. I sure hope not. A better system would be something like NIX/GUIX or even Gobolinux, that give us a single package as today, but with the option of installing multiple versions in parallel if upstream has screwed up the API (again). Flatpak and like will just be and excuse for upsteam to bundle everything and the kitchen sink, resulting in bloat and having to update a mass of paks rather than individual libs in case a flaw is found.
  - bartkmq 9y ago
    You've got good points and I admit I'm split on this issue. In theory i would rather have everything handled by the package manager but container apps provide a few useful advantages: - sandboxing/isolation out of the box - being able to report bugs directly to upstream - It's easier to distribute a small app to all distributions (for some value of all)* - much easier to distribute proprietary applications (subjective advantage) - much easier to install old versions of an application (sometimes needed) - it's possible to record the bandwidth/ram/cpu usage; with a standard package that's quiet difficult
- laumars 9y ago
  7) CPU architecture, 8) output binary format (though these days people generally only use ELF or PE), 9) compile time options (eg some packages will allow you to choose which TLS library to use at compile time) 10) hardware specific optimisations Basically just a plethora of reasons
megous 9y ago
Have you ever been an upstream? What you're suggesting is awfully wasteful. You'd get endless requests for "please add build for XXX" if you ever distribute any binary. Where XXX is anything from arbitrary versions of arbitrary distribution, or any kind of non-linux OSes. And all that for i686, x86_64, various ARMs,... Who has time for that? If you state you only distribute source code and perhaps binary for Windows, you will not get bothered ever again. Better let the building be done by people who are doing building all day en masse for a single arch, or have infrastructures set up to do builds for multiple archs easilly, than to expect every upstream to have this setup.
zAy0LfpBZLC8mAC 9y ago
Really, why would everyone not rebuild it? People have pointed out tons of reasons why distros do their own builds, but really I don't understand what would possibly be a reason not to!? The build system itself also is just a piece of software that gets distributed along with the source of the software to be built, and just as you can download and run curl and get a predictable result (the download of some URL), you can download and run the curl build system and get a predictable result (a cURL binary). So, in that regard, what does it matter what execution of the cURL build system your cURL binary came from? In particular with the trend towards reproducible builds, where the build result will be bit-identical between different runs of the build system if it's using the same compiler (version), I just don't see why you care!? Yeah, distros shouldn't just modify the software they package willy-nilly, but that's completely orthogonal to whether they should do their own builds. There are many reasons for applying small changes to enable integration with the distro, and in particular it's just unreasonable to expect that everyone of the thousands of upstream authors or Debian packages, say, operate machines of all ten CPU architectures that are currently supported by Debian so they can provide binaries for all architectures. So, if you want to be able to distribute software written by people who don't happen to have a System z or a MIPS machine in Debian, maintainers have to prepare packages in such a way that binaries for those architectures can be built--and if they do that, it's trivial to also build all the other architectures from the same source. Adding special cases for when a binary is already available for some architecture from upstream would be just completely pointless complexity.
int_19h 9y ago
For Windows, at least, the big reason to build the binaries would be to sign them with the Microsoft certificate, so that you know that the binary is authentic.

colemannugent 9y ago

Cool, now I don't need to remember the arcane incantation to download a file with Powershell. Do you think we'll see things like the good old "curl <some url> | bash" for Windows now? They still have no package manager worth using.

ygra 9y ago
Hasn't that been popular as iwr <some url> | iex for a while?
pjc50 9y ago
> curl <some url> | bash That's still not a package manager worth using.
temp-dude-87844 9y ago
Windows 10 has a "package manager", of course, that's quite good, but both user culture, developer disinterest, and years and years of inertia keeps it from being used for enough things to cross the network-effect side of some treshold of "worthwhile". The 'download | execute' paradigm, on the other hand, is the complete antithesis of package management, to where your desire to obtain and execute the code trumps your willingness and patience to wait until it has been vetted by your preferred package manager, and installed in a less haphazard way. I fail to see your point.
- sjellis 9y ago
  If you are referring to the package management stuff in PowerShell, that's just a unified interface to several different package management systems, each of which handles specific types of packages. Windows has MSI, VSIX, PowerShell modules and probably some others as well, all of which are separate from Microsoft Update.
Analemma_ 9y ago
Yeah, “iwr” sure was hard to remember.
- 9y ago

orf 9y ago

Curl is alright, and congratulations on this massive and very impressive step forwards, but the CLI not exactly very user friendly. httpie[1] is a great tool if you find curl invocation somewhat arcane. 1. https://httpie.org/ https://httpie.org/

unkoman 9y ago
curl is much more than http.
Something1234 9y ago
HTTPie is great, but sometimes you need curl. Curl supports a lot more protocols.
- chrisper 9y ago
  Yes, but MS seems to disable many of them.
- Rapzid 9y ago
  Plus, it's easy to forget that httpie is SLOW. Sometimes when testing an API the timings are worrying, until you remember and switch over to cURL to confirm everything is right with the world.
  - nicolaslem 9y ago
    I don't know why you are getting downvoted. That's true, starting a Python interpreter is a relatively expensive operation. There is a noticeable lag with most CLIs written in Python. There has been some work recently to make the interpreter start faster. I hope we will see the result in Python 3.7.
andrew_mason1 9y ago
curl may not have the greatest CLI, but it's a huge step up from doing the equivalent tasks with powershell. not to mention this brings windows closer and closer to nix. windows making itself more familiar to nix increases the number of ops folks that are willing to transition from nix shops to windows, increasing windows share in the server market plus, as others have pointed out, curl handles protocols besides http.

frik 9y ago

Please keep builds of Win32/Win64 plus source on your website, built by you. Every M$ exe nowadays contain some telemetry phone home stuff, extra annoying.

lettergram 9y ago

One thing I am always sure to share with collegues when we discuss curl, is the fact from the command line you can generate the underlying C code. This is pretty useful when creating a CLI for pretty much any app, and I've used it regularly to generate a CLI for an app. My post on how to do it: http://austingwalters.com/export-a-command-line-curl-command-to-an-executable/ http://austingwalters.com/export-a-command-line-curl-command...

ww520 9y ago
That's amazing. Never knew that. Learn something today.
andrestc 9y ago
This is amazing! Thanks for sharing
0xFFFE 9y ago
Brilliant, thanks for sharing.
henvic 9y ago
Amazing tip!
innagadadavida 9y ago
There is one for generating goalang code as well: https://github.com/mholt/curl-to-go https://github.com/mholt/curl-to-go Also, it does not depend on libcurl.
aplorbust 9y ago
It wasnt until 2006 that curl added HTTP/1.1 pipelining support. Hence I always used netcat instead of curl because I utilised pipelining heavily for text/html retrieval (IME, most servers supported it). Imagine something like this with curl: curl << eof http://example.com/a.htm http://example.com/b.htm eof where curl only opens a single connection. Alas, AFAIK, pipelining is still not enabled in the curl binary. As I understand it, the --libcurl option only generates code for what is possible with the curl binary, e.g., curl_easy_init(), curl_easy_setopt(), etc. As such, it will not generate code using curl_multi_init(), curl_multi_setopt(), etc. I have to automate the code generation myself.

ape4 9y ago

Now, just ~1000 other commands to go

shitloadofbooks 9y ago
But they're _all_ already there via the WLSS (Bash on Ubuntu on Windows) if you want them. Not to mention that the PowerShell equivalents of a lot of *nix commands are _much_ better. "Everything is an object" is a brilliant philosophy and it's a joy to use.
- frou_dh 9y ago
  Regardless of its merits, they blew it in terms of marketing. PowerShell is destined to be a deadend technology only used by Windows sysadmins.
  - Spivak 9y ago
    I don't think MS envisioned PS as anything other than a sysadmin tool. If they did it doesn't seem to come though with the design.
    - frou_dh 9y ago
      Doubt that. You think in the PowerShell design phase, the team would have shrugged if some MS decision maker somehow mentioned "BTW, 10 years from now, for developers, we're going to be promoting installing a Linux emulation layer and using Bash"? I bet the PS folks would have been dismayed.
jgome 9y ago
Just install busybox.exe from https://frippery.org/busybox/ https://frippery.org/busybox/ to some directory in your %PATH% or in C\: and run 'busybox sh'.

gregmac 9y ago

For debugging on remote servers this is pretty handy (at least, one it makes its way into the server versions). Now I have hope they'll put in a text editor that understands unix line endings.

martin-adams 9y ago
It's like they're trying to get a record by not doing that. For years I've been baffled why Notepad at the very least couldn't understand unix line endings.
- eat_veggies 9y ago
  It's possible they lost the source code and literally can't. They've done it before: https://www.bleepingcomputer.com/news/microsoft/microsoft-appears-to-have-lost-the-source-code-of-an-office-component/ https://www.bleepingcomputer.com/news/microsoft/microsoft-ap...
  - nwellnhof 9y ago
    Equation Editor was developed by a third party and MS probably never had the source code in the first place: https://support.microsoft.com/en-us/help/4057882/error-when-editing-an-equation-in-office https://support.microsoft.com/en-us/help/4057882/error-when-...
  - userbinator 9y ago
    Notepad is literally a window with the standard Windows Edit control inside it, so they certainly have the source code. My guess as to why they don't care to support \n-only is that there's been very little need to; anyone who needs more advanced editing isn't going to use Notepad anyway. As the sibling comment mentions, WordPad (which is similar but with a RichEdit control) does support \n-only.
    - TazeTSchnitzel 9y ago
      If you want notepad-with-LF-support, ReactOS, the open-source Windows NT clone, has this.

chrisper 9y ago

What would be the reason behind disabling all those protocols?

syncsynchalt 9y ago
Probably to reduce exposure to security issues. The more code you ship, the more code you're responsible for keeping secure. Looking briefly at the list at https://curl.haxx.se/docs/security.html https://curl.haxx.se/docs/security.html I see issues for FTP (x2), IMAP, and TFTP in 2017 alone. These protocols which are outside of curl's core competency of http are likely to have less scrutiny and more bugs. While FTP shouldn't be removed from curl I don't think a protocol like TFTP or gopher is crucial, and I wouldn't mind too much if it got the axe in a distribution I used
- pritambaral 9y ago
  HTTP is NOT curl's core competency, "transferring data with URLs" is. HTTP just happens to be the most often used in the world, and thus in curl.

nodesocket 9y ago

> They ship 7.55.1, while 7.57.0 was the latest version at the time. That’s just three releases away so I consider that pretty good. Lots of distros and others ship (much) older releases. Indeed. I am running latest High Sierra and: curl --version curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0

qwerty456127 9y ago

Cool. I've been installing wget on every Windows PC in my authority since the days of Windows 98SE. Every net-enabled operating system is to have such a tool installed by default.

ComodoHacker 9y ago

On a side note, for all my daily download tasks (other than debugging some web API) I've settled with aria2[0]. It seems to support every protocol used in modern Internet and has plenty of flexibility (connection multiplexing, bandwidth control etc.). It can even serve as 24/7 torrent client managed via remote API. 0. https://aria2.github.io/ https://aria2.github.io/

ghews 9y ago
I also think so specilly the featrue supporting every protocol in modern Internet.

ghews 9y ago

I don't like Microsoft ,but I still wish Linux on Windows will have a brighter future.

qwerty456127 9y ago

BTW another program (and a library) that should, IMHO, be made a standard component of every modern OS (except those that would choose to exclude it for a practical reason, e.g. some of extremely-lightweight and heavily specialized embedded ones) is SQLite3