6 ms·

Kina Knowledge, using Common Lisp extensively in their document processing stack

192 points by p_l 5y ago

Abhinav2000 5y ago
This part is really interesting: > It is fast - our spatial classifier takes only milliseconds to come to a conclusion about a page (there is additional time prior to this step due to the OpenCV processing - but not too much) and identify it and doesn’t require expensive hardware. Most of our instances run on ARM-64, which at least at AWS, is 30% or so cheaper than x86-64. The s-expression structures align to document structures nicely and allow a nice representation that doesn’t lose fidelity to the original layouts and hierarchies. The article also mentioned they have 3 programmers and 100k lines of code, that sounds impressive.
- neilv 5y ago
  > The article also mentioned they have 3 programmers One of the problems we had with promoting commercial use of Scheme and then Racket was that -- although some companies were using it to great success -- there weren't any job postings for it. It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). And the knowledge of that success wouldn't be well-known. (Because they liked to focus on the work, or because the larger team of business etc. people they were in was also small, or, in at least one case, the business person thought "we use Lisp" would kill business deals even though the code wasn't customer-visible.) So there would be no success stories, no job postings mentioning it as something people should learn, etc. Which, I suppose was good for open communities self-limiting themselves to people who were genuine enthusiasts not motivated by money, and with no need to posture as influencers or do SEO, but... not so great for bringing in large developer base, getting lots of startups using it, etc.
  - pphysch 5y ago
    > It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). Be careful with what you conclude from this observation. Is it that A) Lispers are generally 10x Programmers, or that B) Lisps are generally poor languages for team environments?
    - math-dev 5y ago
      I can’t imagine the choice of language being strongly correlated with programmer skill (it would be elitist to think so, and everybody feels best in the language they spend the most time in). I think C) Lisp is well suited to certain applications, from what I can see, web development or niche areas where exploratory programming is required. The downside of Lisp is lack of good GUI (CAPI is the best they can offer, but this is not as good as other language implementations for various reasons) and not having the backing of Apple, Microsoft, Google or Facebook, and thus lacking in APIs. But given its dynamic development (its a real joy), its very well suited to exploratory programming. Web Development is an undiscovered gem for Lisp. It doesn’t face the issue of GUI / lack of APIs, since you can work directly in HTML / JS / CSS for the front end, and its a serious solution that covers servers, databases and all aspects of the stack very well. Its simply amazing (this is coming from a web developer), but I shouldn’t advertise this too much - its a secret weapon!
      - reikonomusha 5y ago
        One of the main developers of a Lisp compiler is employed by Google to work on exactly that.
        vindarel 5y ago
        exactly… a new GUI lib, is this what you mean? (there is also a group working on Qt5 bindings, they say to have a working prototype and they're ironing things out since some time)
      - p_l 5y ago
        The choice of language is anecdotally strongly correlated with programmer skill, but the best evidence I've seen was not about "Programmers using language X are better than language Y", but about availability of programmers. Counterintuitive, it was the "elitist" lower numbers involved in some languages (not just Lisp) that were found to be advantageous by some companies in hiring. Essentially, it selected out people with lower experience and those not willing to learn a new language. An example I was given was that choosing Common Lisp and combining it with globally remote hiring greatly increased the quality of incoming applicants - and that had they went with the popular option (Python in their case) they'd have to deal with deluge of fresh bootcamp grads of which many had unwarranted high opinion of themselves. Both Kina Knowledge and ITA Software (and from what I heard but can't cite now, other companies as well), teaching people Common Lisp on the job apparently tends to work pretty well, and willingness to master a new language correlates much better with good skills. It does have the disadvantage of not being able to throw a new hire directly tickets to solve, but the timeframes I was given were essentially "it takes around a month for new hire to go from zero to productive lisp programmer".
  - csande17 5y ago
    > It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). One thing that surprised me when I entered the professional world was just how much this is seen as a downside. At almost every level, companies will choose technologies and techniques that allow them to hire more headcount, even if that results in a worse end product. Many startups value the appearance of having a large actively-hiring engineering team, and many BigCo middle managers want lots of direct reports. They don't usually care how much work gets done per employee, and sometimes they don't even care about the total amount of work that gets done across the organization. It's all about getting warm bodies into seats, either to impress investors or to gain status within the organization.
    - hcarvalhoalves 5y ago
      Hiring thousands of developers is a success metric now.
      - Jtsummers 5y ago
        Management is still one of the fastest ways to a high paying job. Managers get rewarded (implicitly or explicitly) for being in charge of more people, either with better pay in the same position or by appearing to be more competitive when going for promotions. This creates a perverse incentive where someone can make decisions which promote a higher headcount at the cost of actual capability or efficiency. It's often not even deliberate. No one sits down and says, "This language or tool kit requires a higher headcount, so I'll select it for my project". But they aren't motivated to find a more efficient approach that results in them being put in charge of a smaller team (because this would cost them and not reward them). EDIT: Some grammar things.
      - cle 5y ago
        It's also a way to hedge risks. If you've got a single programmer working on something, they're a single point of failure and they also have huge leverage (which can be good or bad depending on the person and circumstances). Not that middle management bloat isn't a thing...it definitely is. But I don't think it's as black-and-white as it's made out here.
  - agumonkey 5y ago
    > It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). I felt this so very often. There are economical paradoxes and sociological issues at play. People want to be impressed and a big corp makes it look like hard while 3 lispers does not have the glow and even at lower prices you may fail to sell good work. A lot of this world work this way, if everybody worked smart and efficiently there would be a lot less jobs.
shaunxcode 5y ago
Really interested to play with dlisp if/when it is open sourced!
- gjvc 5y ago
  Yes! This sounds really promising.
- dang 5y ago
  Have they said whether it's a complete CL implementation or just a subset? I have the impression that it gets asymptotically more difficult as you approach the finish line.
  - p_l 5y ago
    Looks like it's CL-inspired so kinda like subset.
  - aidenn0 5y ago
    Assuming you aren't just talking about how any project becomes harder to fix bugs in once you've fixed the easy bugs, it only gets more difficult as your approach the finish line if you don't plan on making a CL implementation from the start. It's really easy to make something that kind-of looks like CL by use of clever smoke and mirrors, and many "Lisp in X" projects do so. See e.g. parenscript's LOOP implementation. Other notes: - Implementing CLOS is a massive undertaking, but there are two reasonably good portable implementations of it (Closette and PCL). - The numeric tower is sufficiently loosely specified that you can typically get away with using any existing bignum library. - LOOP and FORMAT are annoying to implement just because they are big DSLs. I'm also not aware of any portable implementations of them (though SICL might have one). - Most of the rest of the effort will be spent on implementing things that interact with the system. Streams/files/pathnames are all annoying
    - p_l 5y ago
      LOOP can be cheated a bit with, as it originated in separate semi-portable package AFAIK, just like for CLOS you can use PCL for heavy lifting. Specifically there was MIT LOOP package that wasn't exactly ANSI compliant but got you significant portion of the way there. Also, if you're willing, it's possible to just lift a bunch of code from SBCL/CMUCL due to permissive license. MIT LOOP https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/lisp/code/iter/loop/mit/0.html https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/... few other versions including symbolics update to MIT which is supposedly closer to ANSI: https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/lisp/code/iter/loop/0.html https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/...
pphysch 5y ago
> Lisp allows us to scale dramatically and manage a large code base. Wow, really? How big is your company? > Right now, in our core company we have three people, two here in Virginia and one in Mexico City.
- p_l 5y ago
  Sounds like scaling developer power all right. Reminds me of quotes from Google SRE materials about scaling "SRE per managed server count".
- thorweur234 5y ago
  You know Google's flight scheduling software runs on Borg right ? After they acquired ITA, they have few openings in between, and are yet, apparently, one of the biggest consumers of quotas.
  - the_optimist 5y ago
    What's a quota?
    - dataangel 5y ago
      I think he means they one of the biggest consumers of cluster resources. Each group within google probably has a quota for how many resources they are allowed to use. It's unclear here if their consumption is so huge because they are successful or inefficient though...
      - math-dev 5y ago
        Does anybody know what they actually do? I see them mentioned all the time but I don’t get it. Is it just flight information or do they do pricing analysis and other stuff
        p_l 5y ago
        QPX (ITA's, and now Google's) system is responsible for calculating and searching airfares, i.e. the complete "solution" for "I want to get from A to B with those constraints". It's what responds when you ask google flight for a route, essentially. This is quite computationally complex due to many dimensions involved in airfare calculations. This system is essentially implemented on top of SBCL with some small bits of C++ (responsible, IIRC, for handling memory-mapped airfare data files).
deleted 5y ago
[deleted]
zeIMOr5vRXhVpO 5y ago
> I liked the idea of distributing binary applications as well, which we needed to do in some instances, and building a binary runtime of the software was a great draw, too. fwiw this is possible to with clojure using graalvm. Only mentioning this because of the clojure comparison in the earlier paragraph. While I do lament the JVM I haven't found a straightforward way to build statically compiled binaries with sbcl. I would love to be proven wrong though! :D
- mark_l_watson 5y ago
  Give SBCL another try for building compiled applications. If you compile SBCL from source, enable compression of standalone compiled apps. I own a license for LispWorks Pro, which does a very good job building applications, but SBCL is also good for delivering apps.
- brabel 5y ago
  Doesn't `(save-lisp-and-die :executable t)` [1] create static binaries? I had a look at one of my SBCL-made binaries on MacOS, it shows this: ▶ otool -L lisp-enc lisp-enc: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11) EDIT: GraalVM takes several minutes to generate binaries, and the binary may not behave the same as the JVM-based runtime, so I wouldn't recommend using that. [1] http://www.sbcl.org/manual/#Function-sb_002dext-save_002dlisp_002dand_002ddie http://www.sbcl.org/manual/#Function-sb_002dext-save_002dlis...
  - svetlyak40wt 5y ago
    Yeah, you've got a dynamically linked binary. There is a work in progress on SBCL to make it possible to create a really static binaries: https://www.timmons.dev/posts/static-executables-with-sbcl-v2.html https://www.timmons.dev/posts/static-executables-with-sbcl-v... I tried this fork and it worked for me!
    - p_l 5y ago
      Well, the example was on OSX, and it's generally not a good idea to do static linking there ;) (same on many other systems, you need minimally certain amount of dynamically linked stuff). Blaze/Bazel rules for Lisp support building fully static SBCL binaries (except possibly for grabbing dynamic stuff that is forced by glibc). ECL also supports static linking.

192 points by p_l 5y ago

Abhinav2000 5y ago

This part is really interesting: > It is fast - our spatial classifier takes only milliseconds to come to a conclusion about a page (there is additional time prior to this step due to the OpenCV processing - but not too much) and identify it and doesn’t require expensive hardware. Most of our instances run on ARM-64, which at least at AWS, is 30% or so cheaper than x86-64. The s-expression structures align to document structures nicely and allow a nice representation that doesn’t lose fidelity to the original layouts and hierarchies. The article also mentioned they have 3 programmers and 100k lines of code, that sounds impressive.

neilv 5y ago
> The article also mentioned they have 3 programmers One of the problems we had with promoting commercial use of Scheme and then Racket was that -- although some companies were using it to great success -- there weren't any job postings for it. It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). And the knowledge of that success wouldn't be well-known. (Because they liked to focus on the work, or because the larger team of business etc. people they were in was also small, or, in at least one case, the business person thought "we use Lisp" would kill business deals even though the code wasn't customer-visible.) So there would be no success stories, no job postings mentioning it as something people should learn, etc. Which, I suppose was good for open communities self-limiting themselves to people who were genuine enthusiasts not motivated by money, and with no need to posture as influencers or do SEO, but... not so great for bringing in large developer base, getting lots of startups using it, etc.
- pphysch 5y ago
  > It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). Be careful with what you conclude from this observation. Is it that A) Lispers are generally 10x Programmers, or that B) Lisps are generally poor languages for team environments?
  - math-dev 5y ago
    I can’t imagine the choice of language being strongly correlated with programmer skill (it would be elitist to think so, and everybody feels best in the language they spend the most time in). I think C) Lisp is well suited to certain applications, from what I can see, web development or niche areas where exploratory programming is required. The downside of Lisp is lack of good GUI (CAPI is the best they can offer, but this is not as good as other language implementations for various reasons) and not having the backing of Apple, Microsoft, Google or Facebook, and thus lacking in APIs. But given its dynamic development (its a real joy), its very well suited to exploratory programming. Web Development is an undiscovered gem for Lisp. It doesn’t face the issue of GUI / lack of APIs, since you can work directly in HTML / JS / CSS for the front end, and its a serious solution that covers servers, databases and all aspects of the stack very well. Its simply amazing (this is coming from a web developer), but I shouldn’t advertise this too much - its a secret weapon!
    - reikonomusha 5y ago
      One of the main developers of a Lisp compiler is employed by Google to work on exactly that.
      - vindarel 5y ago
        exactly… a new GUI lib, is this what you mean? (there is also a group working on Qt5 bindings, they say to have a working prototype and they're ironing things out since some time)
    - p_l 5y ago
      The choice of language is anecdotally strongly correlated with programmer skill, but the best evidence I've seen was not about "Programmers using language X are better than language Y", but about availability of programmers. Counterintuitive, it was the "elitist" lower numbers involved in some languages (not just Lisp) that were found to be advantageous by some companies in hiring. Essentially, it selected out people with lower experience and those not willing to learn a new language. An example I was given was that choosing Common Lisp and combining it with globally remote hiring greatly increased the quality of incoming applicants - and that had they went with the popular option (Python in their case) they'd have to deal with deluge of fresh bootcamp grads of which many had unwarranted high opinion of themselves. Both Kina Knowledge and ITA Software (and from what I heard but can't cite now, other companies as well), teaching people Common Lisp on the job apparently tends to work pretty well, and willingness to master a new language correlates much better with good skills. It does have the disadvantage of not being able to throw a new hire directly tickets to solve, but the timeframes I was given were essentially "it takes around a month for new hire to go from zero to productive lisp programmer".
- csande17 5y ago
  > It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). One thing that surprised me when I entered the professional world was just how much this is seen as a downside. At almost every level, companies will choose technologies and techniques that allow them to hire more headcount, even if that results in a worse end product. Many startups value the appearance of having a large actively-hiring engineering team, and many BigCo middle managers want lots of direct reports. They don't usually care how much work gets done per employee, and sometimes they don't even care about the total amount of work that gets done across the organization. It's all about getting warm bodies into seats, either to impress investors or to gain status within the organization.
  - hcarvalhoalves 5y ago
    Hiring thousands of developers is a success metric now.
    - Jtsummers 5y ago
      Management is still one of the fastest ways to a high paying job. Managers get rewarded (implicitly or explicitly) for being in charge of more people, either with better pay in the same position or by appearing to be more competitive when going for promotions. This creates a perverse incentive where someone can make decisions which promote a higher headcount at the cost of actual capability or efficiency. It's often not even deliberate. No one sits down and says, "This language or tool kit requires a higher headcount, so I'll select it for my project". But they aren't motivated to find a more efficient approach that results in them being put in charge of a smaller team (because this would cost them and not reward them). EDIT: Some grammar things.
    - cle 5y ago
      It's also a way to hedge risks. If you've got a single programmer working on something, they're a single point of failure and they also have huge leverage (which can be good or bad depending on the person and circumstances). Not that middle management bloat isn't a thing...it definitely is. But I don't think it's as black-and-white as it's made out here.
- agumonkey 5y ago
  > It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). I felt this so very often. There are economical paradoxes and sociological issues at play. People want to be impressed and a big corp makes it look like hard while 3 lispers does not have the glow and even at lower prices you may fail to sell good work. A lot of this world work this way, if everybody worked smart and efficiently there would be a lot less jobs.

shaunxcode 5y ago

Really interested to play with dlisp if/when it is open sourced!

gjvc 5y ago
Yes! This sounds really promising.
dang 5y ago
Have they said whether it's a complete CL implementation or just a subset? I have the impression that it gets asymptotically more difficult as you approach the finish line.
- p_l 5y ago
  Looks like it's CL-inspired so kinda like subset.
- aidenn0 5y ago
  Assuming you aren't just talking about how any project becomes harder to fix bugs in once you've fixed the easy bugs, it only gets more difficult as your approach the finish line if you don't plan on making a CL implementation from the start. It's really easy to make something that kind-of looks like CL by use of clever smoke and mirrors, and many "Lisp in X" projects do so. See e.g. parenscript's LOOP implementation. Other notes: - Implementing CLOS is a massive undertaking, but there are two reasonably good portable implementations of it (Closette and PCL). - The numeric tower is sufficiently loosely specified that you can typically get away with using any existing bignum library. - LOOP and FORMAT are annoying to implement just because they are big DSLs. I'm also not aware of any portable implementations of them (though SICL might have one). - Most of the rest of the effort will be spent on implementing things that interact with the system. Streams/files/pathnames are all annoying
  - p_l 5y ago
    LOOP can be cheated a bit with, as it originated in separate semi-portable package AFAIK, just like for CLOS you can use PCL for heavy lifting. Specifically there was MIT LOOP package that wasn't exactly ANSI compliant but got you significant portion of the way there. Also, if you're willing, it's possible to just lift a bunch of code from SBCL/CMUCL due to permissive license. MIT LOOP https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/lisp/code/iter/loop/mit/0.html https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/... few other versions including symbolics update to MIT which is supposedly closer to ANSI: https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/lisp/code/iter/loop/0.html https://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/...

pphysch 5y ago

> Lisp allows us to scale dramatically and manage a large code base. Wow, really? How big is your company? > Right now, in our core company we have three people, two here in Virginia and one in Mexico City.

p_l 5y ago
Sounds like scaling developer power all right. Reminds me of quotes from Google SRE materials about scaling "SRE per managed server count".
thorweur234 5y ago
You know Google's flight scheduling software runs on Borg right ? After they acquired ITA, they have few openings in between, and are yet, apparently, one of the biggest consumers of quotas.
- the_optimist 5y ago
  What's a quota?
  - dataangel 5y ago
    I think he means they one of the biggest consumers of cluster resources. Each group within google probably has a quota for how many resources they are allowed to use. It's unclear here if their consumption is so huge because they are successful or inefficient though...
    - math-dev 5y ago
      Does anybody know what they actually do? I see them mentioned all the time but I don’t get it. Is it just flight information or do they do pricing analysis and other stuff
      - p_l 5y ago
        QPX (ITA's, and now Google's) system is responsible for calculating and searching airfares, i.e. the complete "solution" for "I want to get from A to B with those constraints". It's what responds when you ask google flight for a route, essentially. This is quite computationally complex due to many dimensions involved in airfare calculations. This system is essentially implemented on top of SBCL with some small bits of C++ (responsible, IIRC, for handling memory-mapped airfare data files).

deleted 5y ago

[deleted]

zeIMOr5vRXhVpO 5y ago

> I liked the idea of distributing binary applications as well, which we needed to do in some instances, and building a binary runtime of the software was a great draw, too. fwiw this is possible to with clojure using graalvm. Only mentioning this because of the clojure comparison in the earlier paragraph. While I do lament the JVM I haven't found a straightforward way to build statically compiled binaries with sbcl. I would love to be proven wrong though! :D

mark_l_watson 5y ago
Give SBCL another try for building compiled applications. If you compile SBCL from source, enable compression of standalone compiled apps. I own a license for LispWorks Pro, which does a very good job building applications, but SBCL is also good for delivering apps.
brabel 5y ago
Doesn't `(save-lisp-and-die :executable t)` [1] create static binaries? I had a look at one of my SBCL-made binaries on MacOS, it shows this: ▶ otool -L lisp-enc lisp-enc: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11) EDIT: GraalVM takes several minutes to generate binaries, and the binary may not behave the same as the JVM-based runtime, so I wouldn't recommend using that. [1] http://www.sbcl.org/manual/#Function-sb_002dext-save_002dlisp_002dand_002ddie http://www.sbcl.org/manual/#Function-sb_002dext-save_002dlis...
- svetlyak40wt 5y ago
  Yeah, you've got a dynamically linked binary. There is a work in progress on SBCL to make it possible to create a really static binaries: https://www.timmons.dev/posts/static-executables-with-sbcl-v2.html https://www.timmons.dev/posts/static-executables-with-sbcl-v... I tried this fork and it worked for me!
  - p_l 5y ago
    Well, the example was on OSX, and it's generally not a good idea to do static linking there ;) (same on many other systems, you need minimally certain amount of dynamically linked stuff). Blaze/Bazel rules for Lisp support building fully static SBCL binaries (except possibly for grabbing dynamic stuff that is forced by glibc). ECL also supports static linking.