5 ms·

Qlot: Common Lisp Library Manager

100 points by ducktective 2y ago

koito17 2y ago
Seems like an attempt to bring "lockfiles" or related concepts to Quicklisp. This is a desperately needed quality-of-life improvement for "modern" Common Lisp development. However, I wish people in the Common Lisp community weren't weary of using ASDF directly. It already supports versioning and many features that do not exist in Quicklisp -- despite the fact Quicklisp wraps ASDF -- such as a package that depends on a version of a library newer (or older) than what exists in the current dist. It's kind of impressive that Common Lisp is the only language I use where I can't expect code written today to work next month due to Quicklisp lacking versioning outside of the date attached to dists.
- g11479 2y ago
  faré the guy who briefly took over asdf development before quitting common lisp entirely to do scheme, or whatever he's doing right now, spent significant amount of time turning asdf from a 10kb file into 3000kb file in order to solve problems that only he himself cared about, or possibly only google cared about because faré worked for google at the time (building portability library that ignored and force-superseded existing solutions, and trying to eliminate corners cases of SAT solving). there was a period where he was harrassing (this is a subjective interpretation of the events, but this is my side) implementors to incorporate his recent asdf changes, that were breaking left and right to the point that multiple common lisp maintainers of multiple common lisps were considering pulling asdf from their repos entirely. things have stabilized, because faré quit and there's no longer volatility. we're at a point where there's just no concensus about relying on asdf for anything, and while some people go all in on 30% of available functionality (as opposed to customary 5%), equally many people, the majority, use it as a glorified (load)(load)(load). this is an unfortunate side effect of activist developers in small communities: they can have outsized long term effect on how things are done.
  - kazinator 2y ago
    In the Meta-CVS project, I carried a copy of asdf.lisp in which the package was renamed to zxcv, to isolate from the ASDF churn and be able to run my version of ASDF possibly with my fixes even in a Lisp that has ASDF preloaded. ASDF is the only build system I know which broke on me when I tried to fool it with a link farm (lndir command) into building in a separate directory. It resolved the paths using the truename function which eliminates symbolic links, and then calculated the .fas file names from the canonicalized paths, thereby depositing compiled files into the original directory rather than in the symbolically linked tree. ASDF also depends on symlinks for external organization. .asf files are often symlinks to the real location, and without these symlinks, things don't work.
    - g11479 2y ago
      i have a patched version of asdf 1.369 which was maintained by gary king, with contributions from luminaries like nikodemus, chrodes, weitz, pvaneynd and kevin rosenburg. i've patched it to support the recent extensions that people seem to like (extra parameters to defsystem, like author). you can drop it into your implementation and quicklisp and it just works. asdf 2, asdf 3 happened since, they solve some kind of problems, i guess whatever google needed to do with its fleet, or maybe it's "easier to maintain", but it's not the kind of problems that surface in like 60% of quicklisp packages.
      - galdor 2y ago
        Could you distribute it? I'm not a huge fan of ASDF complexity and have had many headaches trying to make some custom components work. I would welcome any simple alternative, even though I will probably end up writing my own.
- redwoolf 2y ago
  My thoughts as soon as I read the README: what’s wrong with asdf?
  - mike_ivanov 2y ago
    UX
  - blast 2y ago
    ASDF is by far the hardest part of the CL ecosystem to work with. I find it unusably complicated. It's the autotools of CL. Thank god for quicklisp.
    - aidenn0 2y ago
      That's interesting because I find it to be one of the most ergonomic build tools I've ever worked with. It sits somewhere between make and autotools in terms of what it does and I find it far easier to use than either.
      - blast 2y ago
        You've probably learned it more thoroughly than I have, or maybe we have different tastes. I find it just impossible to get into, it all rubs me the wrong way. It is so overcomplicated, and unlispy in the way that FORMAT and LOOP are unlispy, only much worse. I actually like FORMAT and LOOP but they're foreign implants amid the lispiness and that's how ASDF feels as well.
  - g11479 2y ago
    it's fukamachi-ware, guy has a bunch of own solutions in the web development space, i suspect driven by his specific needs. they are, fwiw, not as idiosyncratic as some of the artisanal common lisp one-man solutions can be, but they are distinctly and recognizably their own thing
    - nvy 2y ago
      I use his web stack in some of my projects. His stuff is pretty great, it's just that he basically doesn't write any documentation at all, and the small scraps of documentation he does write assume that you're intimately familiar with the details of his packages. So you need to figure a lot out on your own by digging through source code, which is very irritating when you're trying to be productive.
- aidenn0 2y ago
  The only time I've had CL code break due to lack of versioning was when I was depending on stuff written by the author of Qlot, so I either avoid or locally pin stuff written by him. Most of my dependencies are really quite stable. [edit] I thought of a place where things broke that aren't related to Fukamachi; sometimes foreign libraries have changed to the point where they won't grovel any more, but I don't think that's something that Qlot aims to solve.
- fiddlerwoaroof 2y ago
  > It's kind of impressive that Common Lisp is the only language I use where I can't expect code written today to work next month due to Quicklisp lacking versioning outside of the date attached to dists. I think the lack of version pinning is a positive force in stability: reducing the pain of breaking changes increases the likelihood that library authors will break things.
  - Jach 2y ago
    This seems to be the widely reported experience from people who develop in Common Lisp daily or very frequently, and even in my haphazard development I find things pretty stable. And it's not like version pinning is all that helpful in a living project anyway, because you often want to update things, and I notice my JavaScript-using friends constantly suffering from breakages every time they try. With Lisp, I setup my libraries in jenkins to pull the latest SBCL and quicklisp distro once a month and run tests, it's pretty rare for things to break. Once or twice something with an SBCL release itself, once with a bug involving the interplay of two of Fukamachi's libraries. The SBCL one was noticed by others and fixed pretty quickly, for the library one I was able to use git-bisect to easily find the breaking commit and apply a local fix while I waited for the issue I filed to get addressed. This was during Fukamachi's life-threatening illness period, even, but an alternative maintainer did address the bug on their side. The only annoyance these days is Xach is suffering from who knows what preventing the quicklisp distro from updating since October, and there hasn't been communication with people offering to help, so it's forcing everyone to pull projects in locally (and qlot does help with that) for updates or use an alternative.
- jlarocco 2y ago
  Honestly, I just don't think it bothers people much. When I need a specific version of a library, I clone it to my local projects directory and checkout the version I need. Problem solved. Quicklisp is just downloading a snapshot of the upstream repo anyway. For me, if I could change anything about QuickLisp, I'd add something like Python's `virutalenv`. I suspect I can hack something up by messing ASDF's source registries, but so far I haven't wanted it bad enough to implement it.
mike_ivanov 2y ago
I bet my money on https://github.com/ocicl/ocicl https://github.com/ocicl/ocicl Edit: as much as love fukamachi - but still
- dang 2y ago
  Related: Ocicl – An ASDF system distribution and management tool for Common Lisp - https://news.ycombinator.com/item?id=35973000 https://news.ycombinator.com/item?id=35973000 - May 2023 (32 comments)
- VitoVan 2y ago
  I also bet my money on ocicl, because of its systems.csv for locking versions of dependencies.
  - sauercrowd 2y ago
    qlot has lockfile support with qlfile.lock
gibsonf1 2y ago
This is great news. The updates to SBCL broke some libraries (that is, sbcl got more thorough in finding bugs that some libraries had) and I've had to load patches in for them because of the quicklisp limitation in a patch file. This will make life much simpler as do all of Fukamachi's libraries in general. (His internet suite is fantastic)
behnamoh 2y ago
Those who still use Common Lisp or some variant of Lisp (other than Clojure), what's your reasoning behind choosing this language over the more mainstream options? I understand that Lisp has some powerful features, but almost everyone that I've talked to holds reservations for those. For example, Lisp macros are amazing. BUT... this is not good for communicating code in big teams. Or live (image) programs are an interesting concept, but git versioning them is much more difficult than traditional source code versioning. Or hot code reloading, which sounds like a powerful concept except that companies would rather log and document their codebase. It also makes it challenging to keep track of the state. If you force teams not to use macros, then a great reason behind s-exps goes away. And in general, if you take these features away from Lisp, I'm just not sure what use case Lisp can serve.
- fiddlerwoaroof 2y ago
  Lisp macros don’t actually make code harder to read and think about and, unlike the way widely used tools like Flink and Spark do code generation, it’s easy to see what the generated code looks like. In practices, images are just a deployment tool in CL and most people don’t actually do “version control” by saving and restoring images or using hot code reloading. However, these features enable a development experience that I find much more pleasant than any other language I’ve tried.
- db48x 2y ago
  The “downsides” you list are actually benefits that non–lisp users try to downplay. Especially macros. Macros give an individual the ability to write things in Lisp that would take a team in any other language. There’s nothing that says a team cannot gain the same multiplier.
  - deleted 2y ago
    [deleted]
  - Barrin92 2y ago
    The problem is that teams don't consist of the same people for eternity and when new people come in and you have to figure out the three DSLs that an old lisp wizard has cooked up you run into problems. With Lisp one person has the expressiveness of an entire team, but that also means it might take an entire team to figure out what one person did in the first place. The success of Go stems from the fact that Go code in one place has a 99% chance of looking exactly like it does everywhere else. The power of lisp outsources complexity to readers, the simplicity of a constrained language imposes it on the writers, and because code is read more often than it's written, in a big project consistency beats expressiveness.
nothrabannosir 2y ago
Funny, I did quite a deep dive on this same issue about two years back, and came to exactly the opposite conclusion: keep ASDF, and lean on Nix to do pinning. I now maintain my own Common Lisp package repository, testing daily against nightly SBCL etc, and I've learned that a lot of CL stability is just because test suites are never run. Nix goes a long way to shield me from the haphazard breakages that other commenters mentioned, but of course it comes with a huge downside: you need to use Nix :) (just kidding) Code at https://github.com/hraban/cl-nix-lite https://github.com/hraban/cl-nix-lite if someone is interested. Obviously since this was written by Fukamachi you know it will be good. Much respect to the man.
- zelcon 2y ago
  Really unfortunate that Guix never took off. Guile is so much nicer than Nix slop.
  - kentrado 2y ago
    The biggest issue for me is how slow guix is when doing guix pull. Also, the amount of space taken is unacceptable. Several times I had my HD filled with guix garbage.
  - nothrabannosir 2y ago
    At the risk of derailing the conversation (although Guix is a lisp so maybe not): I agree 100% but also maybe Nix's pragmatism is why it's more popular? "Pragmatism" being a programming language euphemism for "untyped hacky mess".
  - ducktective 2y ago
    >Guix never took off Why is this in past tense? Guix development is active. It's just Nix is more popular and it's understandable since it's the original idea and it's older.
    - pxc 2y ago
      Guix even has a very active, high-quality blog where maintainers detail major technical accomplishments, long-term goals, etc.: https://guix.gnu.org/en/blog/ https://guix.gnu.org/en/blog/ From here it seems like they're growing and advancing well. I wish I could find ready historical data on the numbers of packages and services from, say, 4 years ago vs. today, though. I could have sworn Repology used to show year over year stuff but I can't find it now.
arktos_ 2y ago
Love me some Lisposting. Keep it up fellas

100 points by ducktective 2y ago

Seems like an attempt to bring "lockfiles" or related concepts to Quicklisp. This is a desperately needed quality-of-life improvement for "modern" Common Lisp development. However, I wish people in the Common Lisp community weren't weary of using ASDF directly. It already supports versioning and many features that do not exist in Quicklisp -- despite the fact Quicklisp wraps ASDF -- such as a package that depends on a version of a library newer (or older) than what exists in the current dist. It's kind of impressive that Common Lisp is the only language I use where I can't expect code written today to work next month due to Quicklisp lacking versioning outside of the date attached to dists.

g11479 2y ago
faré the guy who briefly took over asdf development before quitting common lisp entirely to do scheme, or whatever he's doing right now, spent significant amount of time turning asdf from a 10kb file into 3000kb file in order to solve problems that only he himself cared about, or possibly only google cared about because faré worked for google at the time (building portability library that ignored and force-superseded existing solutions, and trying to eliminate corners cases of SAT solving). there was a period where he was harrassing (this is a subjective interpretation of the events, but this is my side) implementors to incorporate his recent asdf changes, that were breaking left and right to the point that multiple common lisp maintainers of multiple common lisps were considering pulling asdf from their repos entirely. things have stabilized, because faré quit and there's no longer volatility. we're at a point where there's just no concensus about relying on asdf for anything, and while some people go all in on 30% of available functionality (as opposed to customary 5%), equally many people, the majority, use it as a glorified (load)(load)(load). this is an unfortunate side effect of activist developers in small communities: they can have outsized long term effect on how things are done.
- kazinator 2y ago
  In the Meta-CVS project, I carried a copy of asdf.lisp in which the package was renamed to zxcv, to isolate from the ASDF churn and be able to run my version of ASDF possibly with my fixes even in a Lisp that has ASDF preloaded. ASDF is the only build system I know which broke on me when I tried to fool it with a link farm (lndir command) into building in a separate directory. It resolved the paths using the truename function which eliminates symbolic links, and then calculated the .fas file names from the canonicalized paths, thereby depositing compiled files into the original directory rather than in the symbolically linked tree. ASDF also depends on symlinks for external organization. .asf files are often symlinks to the real location, and without these symlinks, things don't work.
  - g11479 2y ago
    i have a patched version of asdf 1.369 which was maintained by gary king, with contributions from luminaries like nikodemus, chrodes, weitz, pvaneynd and kevin rosenburg. i've patched it to support the recent extensions that people seem to like (extra parameters to defsystem, like author). you can drop it into your implementation and quicklisp and it just works. asdf 2, asdf 3 happened since, they solve some kind of problems, i guess whatever google needed to do with its fleet, or maybe it's "easier to maintain", but it's not the kind of problems that surface in like 60% of quicklisp packages.
    - galdor 2y ago
      Could you distribute it? I'm not a huge fan of ASDF complexity and have had many headaches trying to make some custom components work. I would welcome any simple alternative, even though I will probably end up writing my own.
redwoolf 2y ago
My thoughts as soon as I read the README: what’s wrong with asdf?
- mike_ivanov 2y ago
  UX
- blast 2y ago
  ASDF is by far the hardest part of the CL ecosystem to work with. I find it unusably complicated. It's the autotools of CL. Thank god for quicklisp.
  - aidenn0 2y ago
    That's interesting because I find it to be one of the most ergonomic build tools I've ever worked with. It sits somewhere between make and autotools in terms of what it does and I find it far easier to use than either.
    - blast 2y ago
      You've probably learned it more thoroughly than I have, or maybe we have different tastes. I find it just impossible to get into, it all rubs me the wrong way. It is so overcomplicated, and unlispy in the way that FORMAT and LOOP are unlispy, only much worse. I actually like FORMAT and LOOP but they're foreign implants amid the lispiness and that's how ASDF feels as well.
- g11479 2y ago
  it's fukamachi-ware, guy has a bunch of own solutions in the web development space, i suspect driven by his specific needs. they are, fwiw, not as idiosyncratic as some of the artisanal common lisp one-man solutions can be, but they are distinctly and recognizably their own thing
  - nvy 2y ago
    I use his web stack in some of my projects. His stuff is pretty great, it's just that he basically doesn't write any documentation at all, and the small scraps of documentation he does write assume that you're intimately familiar with the details of his packages. So you need to figure a lot out on your own by digging through source code, which is very irritating when you're trying to be productive.
aidenn0 2y ago
The only time I've had CL code break due to lack of versioning was when I was depending on stuff written by the author of Qlot, so I either avoid or locally pin stuff written by him. Most of my dependencies are really quite stable. [edit] I thought of a place where things broke that aren't related to Fukamachi; sometimes foreign libraries have changed to the point where they won't grovel any more, but I don't think that's something that Qlot aims to solve.
fiddlerwoaroof 2y ago
> It's kind of impressive that Common Lisp is the only language I use where I can't expect code written today to work next month due to Quicklisp lacking versioning outside of the date attached to dists. I think the lack of version pinning is a positive force in stability: reducing the pain of breaking changes increases the likelihood that library authors will break things.
- Jach 2y ago
  This seems to be the widely reported experience from people who develop in Common Lisp daily or very frequently, and even in my haphazard development I find things pretty stable. And it's not like version pinning is all that helpful in a living project anyway, because you often want to update things, and I notice my JavaScript-using friends constantly suffering from breakages every time they try. With Lisp, I setup my libraries in jenkins to pull the latest SBCL and quicklisp distro once a month and run tests, it's pretty rare for things to break. Once or twice something with an SBCL release itself, once with a bug involving the interplay of two of Fukamachi's libraries. The SBCL one was noticed by others and fixed pretty quickly, for the library one I was able to use git-bisect to easily find the breaking commit and apply a local fix while I waited for the issue I filed to get addressed. This was during Fukamachi's life-threatening illness period, even, but an alternative maintainer did address the bug on their side. The only annoyance these days is Xach is suffering from who knows what preventing the quicklisp distro from updating since October, and there hasn't been communication with people offering to help, so it's forcing everyone to pull projects in locally (and qlot does help with that) for updates or use an alternative.
jlarocco 2y ago
Honestly, I just don't think it bothers people much. When I need a specific version of a library, I clone it to my local projects directory and checkout the version I need. Problem solved. Quicklisp is just downloading a snapshot of the upstream repo anyway. For me, if I could change anything about QuickLisp, I'd add something like Python's `virutalenv`. I suspect I can hack something up by messing ASDF's source registries, but so far I haven't wanted it bad enough to implement it.

mike_ivanov 2y ago

I bet my money on https://github.com/ocicl/ocicl https://github.com/ocicl/ocicl Edit: as much as love fukamachi - but still

dang 2y ago
Related: Ocicl – An ASDF system distribution and management tool for Common Lisp - https://news.ycombinator.com/item?id=35973000 https://news.ycombinator.com/item?id=35973000 - May 2023 (32 comments)
VitoVan 2y ago
I also bet my money on ocicl, because of its systems.csv for locking versions of dependencies.
- sauercrowd 2y ago
  qlot has lockfile support with qlfile.lock

gibsonf1 2y ago

This is great news. The updates to SBCL broke some libraries (that is, sbcl got more thorough in finding bugs that some libraries had) and I've had to load patches in for them because of the quicklisp limitation in a patch file. This will make life much simpler as do all of Fukamachi's libraries in general. (His internet suite is fantastic)

behnamoh 2y ago

Those who still use Common Lisp or some variant of Lisp (other than Clojure), what's your reasoning behind choosing this language over the more mainstream options? I understand that Lisp has some powerful features, but almost everyone that I've talked to holds reservations for those. For example, Lisp macros are amazing. BUT... this is not good for communicating code in big teams. Or live (image) programs are an interesting concept, but git versioning them is much more difficult than traditional source code versioning. Or hot code reloading, which sounds like a powerful concept except that companies would rather log and document their codebase. It also makes it challenging to keep track of the state. If you force teams not to use macros, then a great reason behind s-exps goes away. And in general, if you take these features away from Lisp, I'm just not sure what use case Lisp can serve.

fiddlerwoaroof 2y ago
Lisp macros don’t actually make code harder to read and think about and, unlike the way widely used tools like Flink and Spark do code generation, it’s easy to see what the generated code looks like. In practices, images are just a deployment tool in CL and most people don’t actually do “version control” by saving and restoring images or using hot code reloading. However, these features enable a development experience that I find much more pleasant than any other language I’ve tried.
db48x 2y ago
The “downsides” you list are actually benefits that non–lisp users try to downplay. Especially macros. Macros give an individual the ability to write things in Lisp that would take a team in any other language. There’s nothing that says a team cannot gain the same multiplier.
- deleted 2y ago
  [deleted]
- Barrin92 2y ago
  The problem is that teams don't consist of the same people for eternity and when new people come in and you have to figure out the three DSLs that an old lisp wizard has cooked up you run into problems. With Lisp one person has the expressiveness of an entire team, but that also means it might take an entire team to figure out what one person did in the first place. The success of Go stems from the fact that Go code in one place has a 99% chance of looking exactly like it does everywhere else. The power of lisp outsources complexity to readers, the simplicity of a constrained language imposes it on the writers, and because code is read more often than it's written, in a big project consistency beats expressiveness.

nothrabannosir 2y ago

Funny, I did quite a deep dive on this same issue about two years back, and came to exactly the opposite conclusion: keep ASDF, and lean on Nix to do pinning. I now maintain my own Common Lisp package repository, testing daily against nightly SBCL etc, and I've learned that a lot of CL stability is just because test suites are never run. Nix goes a long way to shield me from the haphazard breakages that other commenters mentioned, but of course it comes with a huge downside: you need to use Nix :) (just kidding) Code at https://github.com/hraban/cl-nix-lite https://github.com/hraban/cl-nix-lite if someone is interested. Obviously since this was written by Fukamachi you know it will be good. Much respect to the man.

zelcon 2y ago
Really unfortunate that Guix never took off. Guile is so much nicer than Nix slop.
- kentrado 2y ago
  The biggest issue for me is how slow guix is when doing guix pull. Also, the amount of space taken is unacceptable. Several times I had my HD filled with guix garbage.
- nothrabannosir 2y ago
  At the risk of derailing the conversation (although Guix is a lisp so maybe not): I agree 100% but also maybe Nix's pragmatism is why it's more popular? "Pragmatism" being a programming language euphemism for "untyped hacky mess".
- ducktective 2y ago
  >Guix never took off Why is this in past tense? Guix development is active. It's just Nix is more popular and it's understandable since it's the original idea and it's older.
  - pxc 2y ago
    Guix even has a very active, high-quality blog where maintainers detail major technical accomplishments, long-term goals, etc.: https://guix.gnu.org/en/blog/ https://guix.gnu.org/en/blog/ From here it seems like they're growing and advancing well. I wish I could find ready historical data on the numbers of packages and services from, say, 4 years ago vs. today, though. I could have sworn Repology used to show year over year stuff but I can't find it now.

arktos_ 2y ago

Love me some Lisposting. Keep it up fellas