6 ms·

This part is really interesting: > It is fast - our spatial classifier takes only milliseconds to come to a conclusion about a page (there is additional time p

by Abhinav2000 5y ago

This part is really interesting: > It is fast - our spatial classifier takes only milliseconds to come to a conclusion about a page (there is additional time prior to this step due to the OpenCV processing - but not too much) and identify it and doesn’t require expensive hardware. Most of our instances run on ARM-64, which at least at AWS, is 30% or so cheaper than x86-64. The s-expression structures align to document structures nicely and allow a nice representation that doesn’t lose fidelity to the original layouts and hierarchies. The article also mentioned they have 3 programmers and 100k lines of code, that sounds impressive.

neilv 5y ago
> The article also mentioned they have 3 programmers One of the problems we had with promoting commercial use of Scheme and then Racket was that -- although some companies were using it to great success -- there weren't any job postings for it. It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). And the knowledge of that success wouldn't be well-known. (Because they liked to focus on the work, or because the larger team of business etc. people they were in was also small, or, in at least one case, the business person thought "we use Lisp" would kill business deals even though the code wasn't customer-visible.) So there would be no success stories, no job postings mentioning it as something people should learn, etc. Which, I suppose was good for open communities self-limiting themselves to people who were genuine enthusiasts not motivated by money, and with no need to posture as influencers or do SEO, but... not so great for bringing in large developer base, getting lots of startups using it, etc.
- pphysch 5y ago
  > It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). Be careful with what you conclude from this observation. Is it that A) Lispers are generally 10x Programmers, or that B) Lisps are generally poor languages for team environments?
  - math-dev 5y ago
    I can’t imagine the choice of language being strongly correlated with programmer skill (it would be elitist to think so, and everybody feels best in the language they spend the most time in). I think C) Lisp is well suited to certain applications, from what I can see, web development or niche areas where exploratory programming is required. The downside of Lisp is lack of good GUI (CAPI is the best they can offer, but this is not as good as other language implementations for various reasons) and not having the backing of Apple, Microsoft, Google or Facebook, and thus lacking in APIs. But given its dynamic development (its a real joy), its very well suited to exploratory programming. Web Development is an undiscovered gem for Lisp. It doesn’t face the issue of GUI / lack of APIs, since you can work directly in HTML / JS / CSS for the front end, and its a serious solution that covers servers, databases and all aspects of the stack very well. Its simply amazing (this is coming from a web developer), but I shouldn’t advertise this too much - its a secret weapon!
    - reikonomusha 5y ago
      One of the main developers of a Lisp compiler is employed by Google to work on exactly that.
      - vindarel 5y ago
        exactly… a new GUI lib, is this what you mean? (there is also a group working on Qt5 bindings, they say to have a working prototype and they're ironing things out since some time)
    - p_l 5y ago
      The choice of language is anecdotally strongly correlated with programmer skill, but the best evidence I've seen was not about "Programmers using language X are better than language Y", but about availability of programmers. Counterintuitive, it was the "elitist" lower numbers involved in some languages (not just Lisp) that were found to be advantageous by some companies in hiring. Essentially, it selected out people with lower experience and those not willing to learn a new language. An example I was given was that choosing Common Lisp and combining it with globally remote hiring greatly increased the quality of incoming applicants - and that had they went with the popular option (Python in their case) they'd have to deal with deluge of fresh bootcamp grads of which many had unwarranted high opinion of themselves. Both Kina Knowledge and ITA Software (and from what I heard but can't cite now, other companies as well), teaching people Common Lisp on the job apparently tends to work pretty well, and willingness to master a new language correlates much better with good skills. It does have the disadvantage of not being able to throw a new hire directly tickets to solve, but the timeframes I was given were essentially "it takes around a month for new hire to go from zero to productive lisp programmer".
- csande17 5y ago
  > It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). One thing that surprised me when I entered the professional world was just how much this is seen as a downside. At almost every level, companies will choose technologies and techniques that allow them to hire more headcount, even if that results in a worse end product. Many startups value the appearance of having a large actively-hiring engineering team, and many BigCo middle managers want lots of direct reports. They don't usually care how much work gets done per employee, and sometimes they don't even care about the total amount of work that gets done across the organization. It's all about getting warm bodies into seats, either to impress investors or to gain status within the organization.
  - hcarvalhoalves 5y ago
    Hiring thousands of developers is a success metric now.
    - Jtsummers 5y ago
      Management is still one of the fastest ways to a high paying job. Managers get rewarded (implicitly or explicitly) for being in charge of more people, either with better pay in the same position or by appearing to be more competitive when going for promotions. This creates a perverse incentive where someone can make decisions which promote a higher headcount at the cost of actual capability or efficiency. It's often not even deliberate. No one sits down and says, "This language or tool kit requires a higher headcount, so I'll select it for my project". But they aren't motivated to find a more efficient approach that results in them being put in charge of a smaller team (because this would cost them and not reward them). EDIT: Some grammar things.
    - cle 5y ago
      It's also a way to hedge risks. If you've got a single programmer working on something, they're a single point of failure and they also have huge leverage (which can be good or bad depending on the person and circumstances). Not that middle management bloat isn't a thing...it definitely is. But I don't think it's as black-and-white as it's made out here.
- agumonkey 5y ago
  > It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). I felt this so very often. There are economical paradoxes and sociological issues at play. People want to be impressed and a big corp makes it look like hard while 3 lispers does not have the glow and even at lower prices you may fail to sell good work. A lot of this world work this way, if everybody worked smart and efficiently there would be a lot less jobs.

neilv 5y ago

> The article also mentioned they have 3 programmers One of the problems we had with promoting commercial use of Scheme and then Racket was that -- although some companies were using it to great success -- there weren't any job postings for it. It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). And the knowledge of that success wouldn't be well-known. (Because they liked to focus on the work, or because the larger team of business etc. people they were in was also small, or, in at least one case, the business person thought "we use Lisp" would kill business deals even though the code wasn't customer-visible.) So there would be no success stories, no job postings mentioning it as something people should learn, etc. Which, I suppose was good for open communities self-limiting themselves to people who were genuine enthusiasts not motivated by money, and with no need to posture as influencers or do SEO, but... not so great for bringing in large developer base, getting lots of startups using it, etc.

pphysch 5y ago
> It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). Be careful with what you conclude from this observation. Is it that A) Lispers are generally 10x Programmers, or that B) Lisps are generally poor languages for team environments?
- math-dev 5y ago
  I can’t imagine the choice of language being strongly correlated with programmer skill (it would be elitist to think so, and everybody feels best in the language they spend the most time in). I think C) Lisp is well suited to certain applications, from what I can see, web development or niche areas where exploratory programming is required. The downside of Lisp is lack of good GUI (CAPI is the best they can offer, but this is not as good as other language implementations for various reasons) and not having the backing of Apple, Microsoft, Google or Facebook, and thus lacking in APIs. But given its dynamic development (its a real joy), its very well suited to exploratory programming. Web Development is an undiscovered gem for Lisp. It doesn’t face the issue of GUI / lack of APIs, since you can work directly in HTML / JS / CSS for the front end, and its a serious solution that covers servers, databases and all aspects of the stack very well. Its simply amazing (this is coming from a web developer), but I shouldn’t advertise this too much - its a secret weapon!
  - reikonomusha 5y ago
    One of the main developers of a Lisp compiler is employed by Google to work on exactly that.
    - vindarel 5y ago
      exactly… a new GUI lib, is this what you mean? (there is also a group working on Qt5 bindings, they say to have a working prototype and they're ironing things out since some time)
  - p_l 5y ago
    The choice of language is anecdotally strongly correlated with programmer skill, but the best evidence I've seen was not about "Programmers using language X are better than language Y", but about availability of programmers. Counterintuitive, it was the "elitist" lower numbers involved in some languages (not just Lisp) that were found to be advantageous by some companies in hiring. Essentially, it selected out people with lower experience and those not willing to learn a new language. An example I was given was that choosing Common Lisp and combining it with globally remote hiring greatly increased the quality of incoming applicants - and that had they went with the popular option (Python in their case) they'd have to deal with deluge of fresh bootcamp grads of which many had unwarranted high opinion of themselves. Both Kina Knowledge and ITA Software (and from what I heard but can't cite now, other companies as well), teaching people Common Lisp on the job apparently tends to work pretty well, and willingness to master a new language correlates much better with good skills. It does have the disadvantage of not being able to throw a new hire directly tickets to solve, but the timeframes I was given were essentially "it takes around a month for new hire to go from zero to productive lisp programmer".
csande17 5y ago
> It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). One thing that surprised me when I entered the professional world was just how much this is seen as a downside. At almost every level, companies will choose technologies and techniques that allow them to hire more headcount, even if that results in a worse end product. Many startups value the appearance of having a large actively-hiring engineering team, and many BigCo middle managers want lots of direct reports. They don't usually care how much work gets done per employee, and sometimes they don't even care about the total amount of work that gets done across the organization. It's all about getting warm bodies into seats, either to impress investors or to gain status within the organization.
- hcarvalhoalves 5y ago
  Hiring thousands of developers is a success metric now.
  - Jtsummers 5y ago
    Management is still one of the fastest ways to a high paying job. Managers get rewarded (implicitly or explicitly) for being in charge of more people, either with better pay in the same position or by appearing to be more competitive when going for promotions. This creates a perverse incentive where someone can make decisions which promote a higher headcount at the cost of actual capability or efficiency. It's often not even deliberate. No one sits down and says, "This language or tool kit requires a higher headcount, so I'll select it for my project". But they aren't motivated to find a more efficient approach that results in them being put in charge of a smaller team (because this would cost them and not reward them). EDIT: Some grammar things.
  - cle 5y ago
    It's also a way to hedge risks. If you've got a single programmer working on something, they're a single point of failure and they also have huge leverage (which can be good or bad depending on the person and circumstances). Not that middle management bloat isn't a thing...it definitely is. But I don't think it's as black-and-white as it's made out here.
agumonkey 5y ago
> It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams). I felt this so very often. There are economical paradoxes and sociological issues at play. People want to be impressed and a big corp makes it look like hard while 3 lispers does not have the glow and even at lower prices you may fail to sell good work. A lot of this world work this way, if everybody worked smart and efficiently there would be a lot less jobs.