8 ms·
XCurl
- genocidicbunny 3y agoThe mentioned security requirements likely have to do with certification requirements for games released for the Xbox. You couldn't just make network connections willy-nilly, there was some amount of having to declare ahead of time what ports you would be using and for what. The normal socket API is also severely limited, and largely cannot even be used without obtaining a special exception from Microsoft for your games. I was actually at one point (years ago) tasked with determining if we could use curl in a project I was working on. There was indeed much surgery needed to ensure that the code even compiled, much less worked correctly. We ended up just using WinHTTP directly instead. As for getting access to the source code for xCurl, that will likely not be possible without actually signing up as a dev with Microsoft and going through the whole process. And even then, you'd be under a NDA. (My information is several years old though, so it might be somewhat outdated)
- pavlov 3y agoIt sounds like xCurl is more like a libcurl-compatible wrapper API on top of WinHTTP rather than an actual fork of curl.
- genocidicbunny 3y agoThat is highly likely to be the case. I no longer have access to take a look, but from my experience, it would be silly to do otherwise. There were a handful of other xSomething libraries that they had that were effectively wrappers or simplified versions of libSomething.
- SiempreViernes 3y agoQuoted in the article: > xCurl differs from libCurl in that xCurl is implemented on top of WinHttp and automatically follows all the extra Microsoft Game Development Kit (GDK) requirements and best practices. While libCurl itself doesn’t meet the security requirements for the Microsoft Game Development Kit (GDK) console platform, use xCurl to maintain your same libCurl HTTP implementation across all platforms while changing only one header include and library linkage.
- pavlov 3y agoYet the author of the article (and curl itself) seems to think it's a fork: "With large invasive changes of this kind we can certainly hope that the team making it has invested time and spent serious effort on additional testing, both before release and ongoing." "...since I can’t find the source code I cannot really get a grip of exactly how much and how invasive Microsoft has patched this." It's not a patch of libcurl if they just implement a mostly compatible API from scratch.
- hobofan 3y agoThe presence of the license strongly suggests that it's a fork. "Fork" and "API compatible wrapper" are also not two mutually exclusive things. Often, one of the easiest ways to build an API compatible wrapper is to start by forking the original project, scoop out the internals and replace them with what you are trying to wrap, and then discard all the dangling code (e.g. features that you couldn't wire up because your custom internals don't support it).
- pavlov 3y ago> ”The presence of the license strongly suggests that it's a fork.” It could just mean that Microsoft copied over the header file from libcurl without using any of the implementation.
- jeremyjh 3y agoThis was my assumption reading it; it very nearly has to be exactly that to work as described by Microsoft. Which IMO makes all this a complete nothing burger.
- dontlaugh 3y agoThis sort of reason is also why gRPC isn’t usable in games that also release on consoles. It comes with its own HTTP implementation which you can’t use and even if you wanted to reimplement it, it requires HTTP/2 which isn’t supported by the “blessed” console libraries.
- imglorp 3y ago> certification requirements for games released for the Xbox Could it be simply that WinHTTP collects usage telemetry and MS wants to ensure their valuable feed by forcing all traffic through their library?
- meibo 3y agoI doubt it. If you've ever worked on a console game, you'll know that these requirements are no joke - they're pretty hard to consistently comply with, and using APIs from the manufacturer is often much much easier than to do your own thing. It will also be much cheaper, because failing certification is unimaginably expensive - in the worst case, it can even lead to cancellation. A lot of studios that ship on multiple platforms have abstraction layers that can internally be switched out to every platform specific API.
- genocidicbunny 3y ago> A lot of studios that ship on multiple platforms have abstraction layers that can internally be switched out to every platform specific API. Exactly this. At one point, I was the man responsible for that abstraction layer. Even most of our engine devs never even saw those low layers of the console APIs -- the single exception being the graphics peeps who by necessity had to work at the same layer I was. I basically wrote a Xbox/Playstation/Nintendo/PC libc/runtime that the rest of our engine was based on. I Imagine a lot of other studios that shipped cross-platform had some level of this as well.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- yugarinn 3y agoAs impressive as I find the curl project, I find it almost equally remarkable that Daniel had the courage to introduce its own license.
- deleted 3y ago[deleted]
- girvo 3y agoFrom the sounds of this blog post, in retrospect I think Daniel finds it remarkable too! > It is an MIT license that I was unclever enough to slightly modify many years ago (emphasis mine)
- hk__2 3y agoHow is changing a few words in a licence file courageous?
- WesolyKubeczek 3y agoBecause now, see, you need an army of very expensive lawyers to very carefully analyze the changes to determine if you can still be using the thing. The answer is always tricky, boils down to “it depends, now pls give us more of those sweet hourly fees and your own project description in detail so we may determine if maybe probably it’s ok in your particular case. Not guaranteed, of course, gotta have a court case to guarantee anything.” So legally tricky, much legal busywork. Very courageous.
- bawolff 3y agoHe's hardly the first person to do this. Often when people do this it causes real problems, as most people aren't lawyers and sometimes their edits have unintended consequences. It also tends to mean that reusers have to re-evaluate if the trivial change affects them, possibly get separate legal approval if they are in a company, etc. Tends to just cause a huge headache. Anyways, I'm sure Daniel meant no harm with this, but i still think people should be strongly discouraged from doing that sort of thing.
- deleted 3y ago[deleted]
- hgo 3y agoIf you excuse some speculation: (clarifications appreciated of course.) If i remember correctly, ages ago (Windows Server 2003?) Microsoft introduced in-kernel HTTP-handling for the IIS web-server. I think it was for performance improvements with less copying between kernel- and user-space memory. I suspect the for me unknown WinHTTP apis mentioned here could use these optimizations? Maybe that's why they mention security requirements? (Which would obviously be needed when doing parsing in the kernel.)
- genocidicbunny 3y agoThe security requirements are more due to trying to keep nasty things from being downloaded in the context of a game running on the Xbox. Games tend to run at fairly high privilege levels on Xbox, so Microsoft is probably worried mainly about jailbreaks.
- teh_klev 3y agoThe thing you're thinking about is http.sys and it was introduced in Windows 2008/IIS7 to support the new "Integrated Mode" request pipeline. WinHTTP doesn't have anything to do with http.sys it just listens for HTTP requests and then hands them off to the right bits inside IIS. WinHTTP is essentially a HTTP stack for client services running on Windows Server to allow them to make HTTP requests. It has a sibling API named WinINet which is aimed at use in desktop environments. I think the threading models are main differentiator (I been a while since I looked at this). WinHTTP is fairly well documented: https://learn.microsoft.com/en-us/windows/win32/winhttp/about-winhttp https://learn.microsoft.com/en-us/windows/win32/winhttp/abou...
- rvz 3y agoeXtend.
- Karellen 3y agoIs that a reference to EEE? I feel like it is, but the capitalisation is confusing me, and now I'm wondering if I'm missing the actual intended meaning.
- eequah9L 3y agoI think it is. The capitalization is like that because the project name is xCurl, with x. Maybe they should have called it eCurl?
- Crestwave 3y agoI believe that they're relating the X in XCurl to eXtend.
- deleted 3y ago[deleted]
- russtrotter 3y agoi'm thinking two themes here: the X in xcurl and the famous MS-ism "embrace, extend, extinguish"
- rvz 3y agoYes.
- Karellen 3y agoMicrosoft is certainly within the rights afforded to them by the curl license to do this, but: > The provided functionality is certainly a very stripped down and limited version of the libcurl API. A fun detail is that the quite bluntly just link to the libcurl API documentation to describe how xCurl works. is IMO a major dick move. It seems inevitable that doing this will cause users with questions about xCurl to seek out curl maintainers/forums for support with xCurl (as with the email that prompted this article), even though they have no insight at all into how xCurl works. And some of those people might not be very happy with the response "we have no idea - try contacting Microsoft". Now, that might not have been Microsoft's deliberate intent, but I don't see how it's anything but completely forseeable, and should have been avoided. (Yes, Microsoft need to credit curl with the copyright info and license, but the place for that is the "Credits" and/or "License" parts of the documentation, not the "Usage"/"API reference" parts.)
- fsniper 3y agoOh that's a huge dick move. And this behaviour you mention, "people seeking help from libcurl for xcurl" is an already documented and detrimental problem. You can see why Sqlite temp files has an extension of "etilqs". That's a grandiose example of this. That was not even a documentation link! https://github.com/mackyle/sqlite/blob/3cf493d/src/os.h#L52-L66 https://github.com/mackyle/sqlite/blob/3cf493d/src/os.h#L52-...
- bawolff 3y agoAnother common example is people seeking help at upstream projects for packages provided by their linux distro that have subtle changes which upstream knows nothing about. Honestly though, that's life. I don't think its that big a deal, if you've ever done open source user support you'll know that people ask about all sorts of crazy things. You're lucky if they are even asking about software and not how to wash a car.
- otikik 3y agoWow, fun one, thanks!
- 3y ago
- INTPenis 3y agoObvious question, why didn't they just implement the security features necessary? They could have contacted Daniel to get his cooperation. Edit: Answer is in the comments. So they should have just implemented a wrapper ontop of libcurl. But I guess the real effort here is to promote WinHttp as a replacement for libcurl. And since the developer community at large is so used to libcurl they made a wrapper for WinHttp instead. Oh lord.
- postmodest 3y agoEmbrace. Extend. Extinguish. ...standard MS behavior.
- hducy4 3y agoAs far as triple E goes this embrace seems pretty limp.
- johannes1234321 3y agoThe purpose is to restrict HTTP features and where it can connect to and proxy usage to restrict "wrong" doing. (for some definition of wrong) WinHttp itself is relatively high level, comparable to curl. Most likely this is using curl headers, but a complete custom implementation, nothing upstreamable. (Especially as Daniel probably won't like the burden of supporting a proprietary platform he doesn't have access to)
- kuschkufan 3y agoSo in summary, this seems like it could be a security issue waiting to happen for games that use that xcurl lib from Microsoft, since it seems they don't keep up with upstream and will miss important patches. How that improves "Microsoft security requirements" is beyond me, really.
- taspeotis 3y agoMy interpretation is this is an implementation of the interface, basically wrapping WinHttp for existing code that uses curl. So security vulnerabilities in curl are not necessarily present here, since the two implementations would have matching forward declarations but little in common in terms of implementation.
- cxr 3y ago> this seems like it could be a security issue waiting to happen for games that use that xcurl lib from Microsoft, since it seems they don't keep up with upstream and will miss important patches And at the same time, missing out on the continual stream of bugs that are being put in. (I found it interesting how Daniel was able to give such a positive spin to the comment, "We merge bugfixes at a rate of around three bugfixes per day.")
- freedomben 3y agoThis was my thought as well. Software that is as mature and well-defined as curl, I would be shocked to see three bug fixes per day. That means it is getting lots of new features / new code. That's not a bad thing necessarily, but most of the people that I know who use curl have been using it the exact same way for many years. A version that is a few years old would definitely not need three bug fixes a day
- matsemann 3y agoWhile doing these code things are fine, is the reuse of the name ok? That's what I would be concerned about.
- Aachen 3y agoRelevant: https://curl.se/docs/thename.html https://curl.se/docs/thename.html (page is linked from the copyright page linked in the article) Their position on a previous name collision involving an overseas trademark sounds like "we are bigger, nobody ever confuses us for you, but we'll be sure to redirect anyone who we notice is looking for your project!" Not sure what that says about the inverse, but since the post mentions nothing about trademark concerns, I imagine the author isn't too worried now either
- Tabular-Iceberg 3y agoGiven that their own implementation of curl itself answers with "curl : The remote name could not be resolved: '--help'" just for trying to figure out how to use the damn thing, I wouldn't count on XCurl to be implemented with any degree of competence either.
- myfonj 3y agoSpeaking of strange Microsoft CURL variants, have you ever faced problem with HTTP2 in practically any network client (for example Eclipse strangely failing all attempts to fetch from new HTTP2-enabled Maven) that could be traced to C:\Windows\System32\curl.exe failing handshakes with such HTTP2 servers, whilst any other CURL (from e.g. git-for-windows, even with lover version) could connect to the same server correctly?
- colejohnson66 3y agoThe Windows provided curl is not for user consumption IIRC. If any application is using it, that’s a bug.
- ugexe 3y agoDo you have any evidence of this? IIRC it is completely expected for users to consume it — https://learn.microsoft.com/en-us/virtualization/community/team-blog/2017/20171219-tar-and-curl-come-to-windows https://learn.microsoft.com/en-us/virtualization/community/t...
- myfonj 3y agoFrom what I recall from aforementioned incident, I was not able to reliably prove that client app (Eclipse) really used Windows' curl (I would have guessed that such software would have it's own internal stack for network connections, but my knowledge in this area is non-existent), only that it exhibited suspiciously similar problems. So I came to conclusion that it either uses that CURL or CURL that is similarly broken as the Windows' one. I was unable (or more precisely haven't even tried) to swap Windows curl with the "better" one to see if it fixes Eclipse Maven. Result back then was only that HTTP2 was disabled on the Maven server and all (windows) clients (both of them) were happy. (Other might become less happy since it presumably slowed something down.)
- masto 3y agoThe voice of the battle-weary maintainer comes through clearly in this one. I can't imagine the burden of being responsible for something as widely used as curl. Ok, I can kind of imagine it, or I wouldn't be writing this comment. That feeling of "oh no, someone has just created a support nightmare for me" is familiar, even if I've only experienced it at a minuscule scale in comparison.
- progre 3y agoI'm surprised that they don't just use Invoke-Webrequest from PowerShell. That has to be good enough since Microsoft insists on having "curl" as an alias for Invoke-Webrequest.
- uxp8u61q 3y agoThis hasn't been the case for years. Only in old versions of windows powershell. Also, this is about the library libcurl, not a command line program that uses the library.
- progre 3y agoIt's still there in the powershell 5 witch is still the default in Windows 10
- debugnik 3y agoWindows' Powershell is in fact an old version that they still ship because it's the last one compatible with .NET Framework. Newer versions must be installed separately (so they've ironically had more adoption in Linux). But curl.exe should still bring the actual curl, I think.
- uxp8u61q 3y agoThe tone of this article is really strange. It somehow sounds like the author is upset that someone forked his project. (There are several indicators of that throughout the post.) That's the whole point of free software. If you didn't want that, you shouldn't have chosen a free software license. Also, as explained in other comments, it's unlikely this is actually a fork of curl. It's more likely it's just a wrapper around winhttp with a curl-like API.
- progre 3y ago> It somehow sounds like the author is upset that someone forked his project. I didn't get that at all. It reads like he is slightly annoyed that he is getting detail requests for a half baked fork though.
- vultour 3y agoHe got an email requesting support for software with the same name as his, that points to his documentation, while having zero access to the software itself. I'd say being mildly annoyed at that is warranted, especially since we're talking about a gigantic corporation on the other side.
- deleted 3y ago[deleted]
- charonn0 3y agoThis isn't even the first time[1] Microsoft has done something like this with the curl name. And, then as now, people bother Daniel with support requests. [1]: https://daniel.haxx.se/blog/2016/08/19/removing-the-powershell-curl-alias/ https://daniel.haxx.se/blog/2016/08/19/removing-the-powershe...
- 0xbadcafebee 3y agoAs a cynical Enterprise person, my take is that some team within M$ wanted to force some other team to no longer use Curl/Libcurl for "reasons", so they had to gut it and replace pieces with some other team's cruft. There is no interest in upstreaming, supporting, extending, documenting, releasing, etc this monstrosity as it was probably an ugly kludge to meet an internal bureaucratic requirement. This is why the MIT license is great. If it was GPL'd or something, they would have to rip out curl entirely and develop a completely in-house library, which takes more time, costs more, adds bugs, etc. On top of adapting any programs that normally use libcurl to use their custom thing, or creating a shim library (either way more work). MIT-licensed code allows corporations to still build on top of open source, and monkey-patch their own shittiness in the process. Many of us OSS devs choose MIT for this reason: we just want more people to be able to use it.
- doikor 3y agoIt is purely due to fact that developers on Xbox (where GDK is mostly used) do not have free access to sockets but instead have to clear every network connection with Microsoft. This is much easier to do if every http call goes through WinHttp which they can control/monitor easily on the console.
- shp0ngle 3y ago> I have not been able to figure out how to download xCurl in any form I think it will be somewhere in here. It will probably get installed on your PC with this thing. https://github.com/microsoft/GDK https://github.com/microsoft/GDK edit: you can see references to xurl.dll and xurl.h here, which is GDK examples https://github.com/search?q=repo%3Amicrosoft%2FXbox-GDK-Samples%20xcurl&type=code https://github.com/search?q=repo%3Amicrosoft%2FXbox-GDK-Samp... so the xcurl.dll and xurl.h is really probably in the installer.
- HackerThemAll 3y agoYeah those corporate kindlings deeming themselves "senior developers" or whatnot have those ideas all the time. In this case the schizophrenia is that on one hand Microsoft is using unmodified curl in Windows (used by "billions"), and it's okay. But on the other hand for their game blahblah SDK the "requirements" are higher so they do need to have to maintain separate fork. That's not curable I think.
- justtocomment 3y agoI'm aware that this is utterly childish, but still wondering how far one would get, forking Visual Studio Code and calling it XVisual Studio Code.
- mmh0000 3y agoTo be fair, there is vscodium[1] which is only a few letters off vscode: https://vscodium.com/ https://vscodium.com/