4 ms·

> For example, installing a pkg non-interactively may fail, but return success. Or, it may return failure, but the package actually installed successfully. New

> For example, installing a pkg non-interactively may fail, but return success. Or, it may return failure, but the package actually installed successfully. News to me. Should I just take your word for it? > There had been a bug about it in the FreeBSD tracker about it for a couple of years when I ran into it...I don't know if it's been fixed by now. This was during the 7.0 release cycle. Do you have a link to that? Anyway, bugs happen. For instance, some RPM signing functionality is broken [1]. There's also the fact that killing a simple yum info command can leave the rpmdb in an inconsistent state [2] -- and I'd call that a "really scary quirk" if it wasn't just a bug. Should I claim these are some kind of inherent failings of RPMs compared to FreeBSD ports, the way you claimed a bug was some kind of inherent failing of the FreeBSD ports system? > I've already mentioned the dependency issues I've run into. I've run into dependency issues with DEB and RPM based tools, too. > And, at the end of it all, the user can run "apt-get update; apt-get upgrade" and our packages get updated along with everything else. I asked around. It is absolutely possible to set up a third-party repository for FreeBSD ports. If you want to, you can even set up a second, completely separate ports tree, and a handful of lines of sh script can be used to as a wrapper around your ports management tool of choice to automate the process of selecting a ports tree from which to get what you need to manage a particular port if it isn't in the default tree -- if that's the route you want to take. I poked around in manpages for about three minutes before coming up with the basics for such a system, then decided before putting more effort into it to ask on the freebsd-questions list whether a third-party ports source could be used with standard ports management tools; the first response gave me an idea how to do this, and it pointed to exactly the sort of thing I had found in my manpage browsing. Hire someone who uses (and likes) FreeBSD if you're having that much trouble finding what took me about three minutes to find. > And, we have a nice UI in the product itself that can query the native package manager to display a list of available updates and such. Getting that information out of the ports tree is trivial. > I found pkg_info during my Trail of FreeBSD Tears, but I don't believe it tells you about the configuration of the installed port, does it? I don't remember exactly what information I needed and couldn't get, as it's been a couple of years since I was banging on it seriously, but I do recall there were details I couldn't get making it difficult to figure out the actual state of the system. Well, sorry -- I can't tell you how to find something if you don't know what you need to find. I suspect it's there to be found, though, if it's not completely unreasonable. > I hate our current distribution and installation process on FreeBSD, and having the ability to use native packages with native update mechanisms would be great. I'd recommend starting with references to environment variables in the pkgtools.conf file if you want to get where I was going in short order. You can also get some hints from this response to my freebsd-questions query [3]. He talked a little more about mirroring than I had intended with my question, but a lot of the information is very generalizable. > I want less. I want One True Package Manager on the system and I want it to be able to manage and update all the packages. Adding more software to an already large stack is exactly the opposite of what I want to do. Admirable. > yum/rpm makes this trivially easy. apt-get makes it reasonably easy. FreeBSD makes it pretty much impossible as far as I can tell. But, hey, if you get it figured out, I'd be more than happy to be wrong. Supporting FreeBSD has been a pain in my ass for years now, and it'll continue to be so as long as installation is such a mess. If you're still having problems with this a year from now, you could always check back with me and see if my curiosity has driven me to look into it enough to write up a howto (or to find and test any that might have been written by others, since that would probably be preferable to duplicating effort). > our FreeBSD users tend to be better educated about their systems than our CentOS users, on average; which makes sense, given that CentOS is the "standard" in hosting, and you'd have to go seek out FreeBSD Have you considered throwing the question to your FreeBSD users to see if they can offer any help or suggestions to make things easier for themselves? --- edit: I don't know how long I'll be able to keep up with this discussion here. I'm not entirely clear on how long HN keeps comments open, and the interface for my comments list makes it difficult to keep up with older discussions. If you need to keep in further touch, try my contact page at blogstrapping [4]. --- notes: [1] https://bugzilla.redhat.com/show_bug.cgi?id=624585 https://bugzilla.redhat.com/show_bug.cgi?id=624585 [2] https://bugzilla.redhat.com/show_bug.cgi?id=557741 https://bugzilla.redhat.com/show_bug.cgi?id=557741 [3] http://lists.freebsd.org/pipermail/freebsd-questions/2010-October/222433.html http://lists.freebsd.org/pipermail/freebsd-questions/2010-Oc... [4] http://blogstrapping.com http://blogstrapping.com

SwellJoe 16y ago
So, as I mentioned, I tried going down the road of munging the ports tree and had it kinda working locally, but this was a stopping point for me: "You have to be a bit careful with updating mechanisms if you customise your ports like this: portsnap for one is quite unfriendly to "foreign" files within the ports tree, but csup works fine." All of our FreeBSD users told us they were using portsnap, and it's one of the things I'd talked to Colin about, I think (he wrote it). If we can't safely munge the ports tree in a way that's compatible with the tools our customers use, then we can't munge the ports tree. So, that option is right out. Again, if we break the usual process the user uses to update his system, it's not useful. If we're using multiple tools, multiple processes, and making the user do extra work or doing things that confuse them, we might as well just do it with our own package manager (we already have to maintain that repository for Solaris, Mac OS X, and other crappy systems with no package management to speak of). The other options seemed to require more user involvement (i.e. switching between the standard and our repo manually), or adding an additional wrapper for the ports tools, thus requiring users to use a different command than usual. Further, if the repos are separate, and you switch between them, you no longer have any dependency resolution (even though I complain about it, at least there are some capabilities there...but only if all of ports is available). I guess we could run a complete mirror of the ports tree with our stuff munged in and all indexes and stuff regenerated, but that's just...too awful to think about. I also suspect our users would be unhappy with that solution. In short, as far as I can tell, your link to the discussion on the FreeBSD mailing list confirms my own experience, which is: it doesn't work. But, I could be misunderstanding something in that response. I'm quite clueless about a huge array of FreeBSD and ports questions. The answer could be staring me in the face, but I don't see it...just a rehash of all the problems I ran into when I went down this path a couple years ago. Basically, we either break portsnap (or, more accurately, we'll get broken by portsnap) by munging into the existing ports tree, or we lose all dependency resolution by running a second separate repository. Compare this to yum or apt-get, where a third party repository package is able to resolve from any other repository, including the standard ones, doesn't require any extra wrapper commands or knowledge on the part of the user, and setting it up on the users machine requires editing one configuration file and importing the relevant GPG keys. "Have you considered throwing the question to your FreeBSD users to see if they can offer any help or suggestions to make things easier for themselves?" Of course. The persistent ProFTPd problem I ran into finally got resolved by one of our users after a few weeks, actually. But, the third party repos problem was beyond all their kin, unfortunately. And, unless I'm mistaken, it's unresolved still, though I appreciate your, and the FreeBSD lists, attempts to help figure it out. "Hire someone who uses (and likes) FreeBSD if you're having that much trouble finding what took me about three minutes to find." I did. They didn't have any answers on the questions that remain. As I mentioned, I also hashed out some of this with Colin Percival (who, as far as I can tell, is at least six times smarter than I am, and knows a whole helluva lot more about FreeBSD and ports and portsnap than any one human ought). While we're referencing, I couldn't dig up the bug about pkg_add return code issue, but I did find several people talking about pkgs simply not working: "When a pkg just works, it’s a beautiful and fast thing. The problem is when they don’t. That’s when ports shine." "Back when I first started running FreeBSD, I completely destroyed my system installing the simplest stuff. Eventually, nothing would install. I do mean nothing. I had no idea what was going wrong, I just knew that I was frustrated. So I stopped using FreeBSD for a long time." http://oscartheduck.wordpress.com/2007/07/25/on-ports-vs-pkg_add-a-digression-into-keeping-your-freebsd-programs-up-to-date-with-the-latest-patches-and-software/ http://oscartheduck.wordpress.com/2007/07/25/on-ports-vs-pkg... "Why choose one over the other? Well with packages sometimes the binary doesn't work. Either you happen to have a newer or older file than what the creator compiled it with or you changed something with your system and it doesn't match anymore with the author's compiled environment. However packages are fast to install. No need to break out gcc and compile source code." http://returnbooleantrue.blogspot.com/2009/05/fixing-pkgadd-r-on-freebsd-things-no.html http://returnbooleantrue.blogspot.com/2009/05/fixing-pkgadd-... This was exactly my experience with binary packages. Some worked. Some didn't, and worse, the ones that didn't work provided no warning during installation that they would be inoperable once installed. Several of the packages we require had to be installed from ports because the binaries were non-functional. Making the install time for Virtualmin on FreeBSD a few hours (on a very small CPU-limited VPS, it can actually take days), vs. 10-20 minutes on other equivalently specced systems running CentOS or Debian (the small VPS case can take a couple of hours with these systems). My concern about the return code bug is that it was persistent through multiple major revisions of FreeBSD. The first report of it was about 5.x, I seem to recall. It had existed and been discussed, but never corrected, over the span of a couple of years. And, while the bugs you mention about RPM/yum sound annoying, they never caused me any trouble during my process of building our installation script. We certainly manage to tickle a lot of bugs, because we're doing so much heavy-lifting with the package managers on all these systems; and I've found loads of bugs on all of our supported systems (and I simply killed support for SUSE because their developers simply refused to fix package manager bugs that were showstoppers for us; closing them with unrelated "DUPE" statuses, and denying they were problems, at all; while Red Hat always fixed the bugs I reported, even if it took them a while). The difference is in how much it slows down our deployment, how much additional work it puts on me and our users, how reliably it performs its job as documented, etc. FreeBSD was, so far, our most complex target OS to support because of package manager issues. Technically, Solaris probably ought to hold that honor but we punted there, and took the easy way out, and said "screw the native package manager, it's shit anyway". I tried valiantly to use the FreeBSD package management tools as much as possible, which I paid for in a lot of time and frustration. Another annoying fact is that most discussions about ports and problems therein result in redirection rather than admitting there are problems. Like the very common, "Just use ports" in response to any "binary packages do not work" kind of query; it seems to me that binaries ought to actually work consistently, or at least warn you during installation if they aren't going to work. This is what package management is for: to make sure the stuff you install works. If it doesn't, then the package manager is broken. I just ran into so many instances of packages not working with ports and binary packages, that my level of frustration is probably the highest it's ever been since starting Virtualmin (this company was designed from the get-go to be low frustration, as a response to my prior very stressful hardware company; so when it gets stressful, I get mad!). "> And, we have a nice UI in the product itself that can query the native package manager to display a list of available updates and such. Getting that information out of the ports tree is trivial." But, since we're not in the ports tree, and can't be in the ports tree, it doesn't solve our problem. We could, and probably will, add support for ports to our updates UI (which could then merge the data of available updates with our wbm packages we already know about and display on FreeBSD), and so once installed, we could get close to an equally good experience for FreeBSD users, at least in the UI. The command line is still a mess, since updating our packages would be separate. Anyway, I honestly do appreciate your going to the trouble to help figure it out, even if I'm pretty sure we're actually no closer to a solution that works.

SwellJoe 16y ago

So, as I mentioned, I tried going down the road of munging the ports tree and had it kinda working locally, but this was a stopping point for me: "You have to be a bit careful with updating mechanisms if you customise your ports like this: portsnap for one is quite unfriendly to "foreign" files within the ports tree, but csup works fine." All of our FreeBSD users told us they were using portsnap, and it's one of the things I'd talked to Colin about, I think (he wrote it). If we can't safely munge the ports tree in a way that's compatible with the tools our customers use, then we can't munge the ports tree. So, that option is right out. Again, if we break the usual process the user uses to update his system, it's not useful. If we're using multiple tools, multiple processes, and making the user do extra work or doing things that confuse them, we might as well just do it with our own package manager (we already have to maintain that repository for Solaris, Mac OS X, and other crappy systems with no package management to speak of). The other options seemed to require more user involvement (i.e. switching between the standard and our repo manually), or adding an additional wrapper for the ports tools, thus requiring users to use a different command than usual. Further, if the repos are separate, and you switch between them, you no longer have any dependency resolution (even though I complain about it, at least there are some capabilities there...but only if all of ports is available). I guess we could run a complete mirror of the ports tree with our stuff munged in and all indexes and stuff regenerated, but that's just...too awful to think about. I also suspect our users would be unhappy with that solution. In short, as far as I can tell, your link to the discussion on the FreeBSD mailing list confirms my own experience, which is: it doesn't work. But, I could be misunderstanding something in that response. I'm quite clueless about a huge array of FreeBSD and ports questions. The answer could be staring me in the face, but I don't see it...just a rehash of all the problems I ran into when I went down this path a couple years ago. Basically, we either break portsnap (or, more accurately, we'll get broken by portsnap) by munging into the existing ports tree, or we lose all dependency resolution by running a second separate repository. Compare this to yum or apt-get, where a third party repository package is able to resolve from any other repository, including the standard ones, doesn't require any extra wrapper commands or knowledge on the part of the user, and setting it up on the users machine requires editing one configuration file and importing the relevant GPG keys. "Have you considered throwing the question to your FreeBSD users to see if they can offer any help or suggestions to make things easier for themselves?" Of course. The persistent ProFTPd problem I ran into finally got resolved by one of our users after a few weeks, actually. But, the third party repos problem was beyond all their kin, unfortunately. And, unless I'm mistaken, it's unresolved still, though I appreciate your, and the FreeBSD lists, attempts to help figure it out. "Hire someone who uses (and likes) FreeBSD if you're having that much trouble finding what took me about three minutes to find." I did. They didn't have any answers on the questions that remain. As I mentioned, I also hashed out some of this with Colin Percival (who, as far as I can tell, is at least six times smarter than I am, and knows a whole helluva lot more about FreeBSD and ports and portsnap than any one human ought). While we're referencing, I couldn't dig up the bug about pkg_add return code issue, but I did find several people talking about pkgs simply not working: "When a pkg just works, it’s a beautiful and fast thing. The problem is when they don’t. That’s when ports shine." "Back when I first started running FreeBSD, I completely destroyed my system installing the simplest stuff. Eventually, nothing would install. I do mean nothing. I had no idea what was going wrong, I just knew that I was frustrated. So I stopped using FreeBSD for a long time." http://oscartheduck.wordpress.com/2007/07/25/on-ports-vs-pkg_add-a-digression-into-keeping-your-freebsd-programs-up-to-date-with-the-latest-patches-and-software/ http://oscartheduck.wordpress.com/2007/07/25/on-ports-vs-pkg... "Why choose one over the other? Well with packages sometimes the binary doesn't work. Either you happen to have a newer or older file than what the creator compiled it with or you changed something with your system and it doesn't match anymore with the author's compiled environment. However packages are fast to install. No need to break out gcc and compile source code." http://returnbooleantrue.blogspot.com/2009/05/fixing-pkgadd-r-on-freebsd-things-no.html http://returnbooleantrue.blogspot.com/2009/05/fixing-pkgadd-... This was exactly my experience with binary packages. Some worked. Some didn't, and worse, the ones that didn't work provided no warning during installation that they would be inoperable once installed. Several of the packages we require had to be installed from ports because the binaries were non-functional. Making the install time for Virtualmin on FreeBSD a few hours (on a very small CPU-limited VPS, it can actually take days), vs. 10-20 minutes on other equivalently specced systems running CentOS or Debian (the small VPS case can take a couple of hours with these systems). My concern about the return code bug is that it was persistent through multiple major revisions of FreeBSD. The first report of it was about 5.x, I seem to recall. It had existed and been discussed, but never corrected, over the span of a couple of years. And, while the bugs you mention about RPM/yum sound annoying, they never caused me any trouble during my process of building our installation script. We certainly manage to tickle a lot of bugs, because we're doing so much heavy-lifting with the package managers on all these systems; and I've found loads of bugs on all of our supported systems (and I simply killed support for SUSE because their developers simply refused to fix package manager bugs that were showstoppers for us; closing them with unrelated "DUPE" statuses, and denying they were problems, at all; while Red Hat always fixed the bugs I reported, even if it took them a while). The difference is in how much it slows down our deployment, how much additional work it puts on me and our users, how reliably it performs its job as documented, etc. FreeBSD was, so far, our most complex target OS to support because of package manager issues. Technically, Solaris probably ought to hold that honor but we punted there, and took the easy way out, and said "screw the native package manager, it's shit anyway". I tried valiantly to use the FreeBSD package management tools as much as possible, which I paid for in a lot of time and frustration. Another annoying fact is that most discussions about ports and problems therein result in redirection rather than admitting there are problems. Like the very common, "Just use ports" in response to any "binary packages do not work" kind of query; it seems to me that binaries ought to actually work consistently, or at least warn you during installation if they aren't going to work. This is what package management is for: to make sure the stuff you install works. If it doesn't, then the package manager is broken. I just ran into so many instances of packages not working with ports and binary packages, that my level of frustration is probably the highest it's ever been since starting Virtualmin (this company was designed from the get-go to be low frustration, as a response to my prior very stressful hardware company; so when it gets stressful, I get mad!). "> And, we have a nice UI in the product itself that can query the native package manager to display a list of available updates and such. Getting that information out of the ports tree is trivial." But, since we're not in the ports tree, and can't be in the ports tree, it doesn't solve our problem. We could, and probably will, add support for ports to our updates UI (which could then merge the data of available updates with our wbm packages we already know about and display on FreeBSD), and so once installed, we could get close to an equally good experience for FreeBSD users, at least in the UI. The command line is still a mess, since updating our packages would be separate. Anyway, I honestly do appreciate your going to the trouble to help figure it out, even if I'm pretty sure we're actually no closer to a solution that works.