7 ms·

How do you use lisp style macros when you have 500 developers working on a single product? If you have 500 developers on a java project, you tell the new perso

How do you use lisp style macros when you have 500 developers working on a single product? If you have 500 developers on a java project, you tell the new person which version of java you’re using, you mention whatever libraries or frameworks you lean on heavily and you give them a walk through the subsystems, their key components and essential abstractions and then you spend time showing how the build system is setup and how to create an environment for the system. Then they start knocking out features. There’s a bit more than I’ve said but overall It’s pretty straightforward, any java-for-hire person knows the score and everyone just cracks on with it. How do you do this when the language itself is programmable? Clojure probably avoids my concern by two facts - it embraces immutability and it only has macros, it doesnt have reader macros. Common Lisp has neither of those - it might be common to write in an immutable style but in Common Lisp you can (setf ) til your heart’s content. You can also define reader macros - i recently followed an example to implement json parsing in the reader. I’m still equal parts impressed, amazed and horrified. So. You can probably tell I’ve fallen for lisp in a big way, but how do you scale up to a big dev team? Much as i love lisp in my short journey so far, it seems like a super power that could only be leveraged by up to a “two pizza team”.

medo-bear 5y ago
> how do you scale up to a big dev team? can you explain why this is such a big concern? as far as i can tell, you can set development guidelines and employ people who will stick to the rules. it's not like you can sneak in macros and present them as functions
- CraigJPerry 5y ago
  What happens when you scale java development to a team of that size? Short version: every optimisation you can dream of gets implemented. Clever use of Off heap datastructures aren’t going to be enough to impress. Forget thinking your custom class loader with cooperating javaAgent that manipulates the bytes of classes to dynamically add or remove instrumentation is going to impress. These things are just taken for granted. Even a bot that identifies dead code after traversing the AST, then re-generates valid source code from the (now pruned) AST, runs the test suite then checks in the change all by itself is seen as meh after a while. No matter how out-there an idea is, the play area is constrained by some immutable platform rules (the java memory model). So no matter how funky things get, any other java dev will still be able to reason about any code in the system. What happens when you remove those platform constraints? What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke?
  - chii 5y ago
    > What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke? you end up with what amounts to bank python: https://calpaterson.com/bank-python.html https://calpaterson.com/bank-python.html
    - CraigJPerry 5y ago
      This is the best answer so far. Bank Python ain’t so bad. I’m going to stop worrying so much about lisp macro abuse. And I’m going to finally start reading let over lambda.
    - medo-bear 5y ago
      the difference being of course that python the language is not designed for this, whereas lisp is
  - medo-bear 5y ago
    > What happens when you remove those platform constraints? What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke? i think lisp is very nicely structured so that you only allow a certain subset to be used according to seniority. if you have good development paractices at hand, then even if you have a very bespoke program it will be well documented and anything ecentric is explained lisp is indeed very powerful, but so is c++ for example. the difference is that the syntax of lisp allows you to be neater where you want power and more conservative where you don't, far more easily
haolez 5y ago
Rails is basically a DSL for web applications and it scales to big teams. It might work this way for Lisp as well.
jonahbenton 5y ago
Highly recommend checking out materials from Nubank, the Brazilian banking startup that now owns Cognitect (and therefore Clojure). They have several hundred engineers on the back end working almost entirely in Clojure. Believed to be the largest Clojure plant, though many other large orgs now have sizable Clojure staffing. I saw some of the Nubank folks speak in person in NYC before the pandemic (and before the acquisition of Cognitect) and it was hands down some of the most impressive work I've encountered. Little in the way of macros, just functions and data. What they have done to scale a Datomic infrastructure is astounding. They have quite a lot of materials online. Worth some time. Now that the Cognitect team is essentially embedded and more resources are going into core, I imagine at scale tooling will see some enhancements in the coming years.
Rovanion 5y ago
I think the lack of reader macros really help here. Macros in Clojure are much more "functions that run at compile time" rather than "look at this fun DSL I threw together". What I think it comes down to though is adherence to convention. A preference hierarchy has grown together with Clojure and it goes: 1. Can it be expressed using data structures, do that. For example a map/dictionary lookup. 2. If that fails, express your more complex idea using a function. 3. If you absolutely must, write a macro. You usually end up with very few macros except for those in the standard library, and those you end up with are more commonly of the convenience kind. One example would be the standard web routing library of the early 10's called Compojure that relied heavily on macros where my current favourite Reitit uses just data.
- filoeleven 5y ago
  The last time “macros in Clojure” came up, I searched through some of the libraries I’ve run across to see how often they’re used and, as you suggest, it ain’t much. Re-frame has just one, and it looks like it’s used for testing. Clara-rules has eight that are used in the library, which implements a forward-chaining Rete rule engine. Core.logic has a handful, and that’s essentially “prolog inside Clojure.” So clearly those libraries get a lot of mileage out of data and functions, with just a dash of macro. I have yet to reach for them myself, and I can’t imagine a situation where I’d need a macro instead of a function. I think they’re probably great for library developers who are smarter than me, and I’m happy to leave them to it.
  - openfuture 5y ago
    I think they come in handy when you've got some functionality that has to happen in a context that you can't easily move between function calls... Then you benefit from the ability to template into the description of that context some actions to perform _before_ you enter into the "black box". That's pretty abstract but honestly its not really such an earthshattering feature. I'd argue it's pretty much necessary for any "final" programming language to be infinitely metaprogrammable (i.e. a lisp-3) but we don't have a mature language like that yet. Probably it will arrive from laskell/hisp kinda synthesis.
invalidOrTaken 5y ago
>How do you use lisp style macros when you have 500 developers working on a single product? In general, you don't, or not at the same rate as functions. In big orgs, macros have three worthy uses: - cleanly separated libraries for the org, providing a common underpinning - smaller projects with in the org - making your language nicer That last one shouldn't be underestimated. I can count on one hand the # of times I've written macros for work, but I use them all the time, as in calling macros others have written. You can very much build a big company on lisp (NuBank the most available-to-mind example). But it's true that at larger team sizes there are extra coordination obstacles that must be surmounted before you get to macro heaven.
- chii 5y ago
  i think the crux is that it's hard to find 500 people who are all at such high level of capability that could make use of macros in lisp - esp. if they all need to follow a similar style of programming (iirc, lisp macros tend to follow idiosyncrasies of the person writing it). It's almost as though you're asking for 500 Michael Angelo to help paint the sistine chapel in order to do it faster.
  - openfuture 5y ago
    Bah. This meme vastly overstate the difficulty... I'm sure there is some nonsense reason why everyone understands templates just fine and the web would literally not exist without them because html is so tedious but "macros" are this mythical complicated impossible to reason about thing??! Seriously there is a limit to how much you can trust all the secondhand garbage "knowledge" you get from reading opinions of people who claim to have tried something. Setting the record straight: - common lisp is fast - common lisp is stable - common lisp is useful! (and interactive! even impatients can get their instant gratification) - common lisp has non-trivial libraries for various useful things - common lisp is simple and has excellent educational resources - common lisp is crufty but the community has such a high average education that actually the resources have aged well and hides at least as much insight and wisdom gems as the type theoretical FP world that we now put on the pedistal we used to keep lisp on. - common lisp is huge. No need to learn it all just hack on what you need and magically you'll be able to make software that works. If macros are such a problem then restrict their use to seniors or "tool devs" (people who go around improving dev ergonomics by extending the language is synonymous to any other DSTool)... This way you have maximum flexibility and by restricting the words you allow you can have this flexibility while you need it while also programmatically restricting your monkeys to a short enough rope that it'll be hard to hang anyone. You can stay in some Java comparible subset (Java because lispers hate on Java like vimers do emacs). Yes I Markov chained a bunch of sentences in this comment... I am too lazy to fully engage with the "someone is wrong on the internet"-thing, anecdotally it has been my feeling that it is no longer a good way to learn something to be "wrong on the internet" nobody is cleaning up anymore... These days everyone has a job; extract from the commons and go rave while Rome burns.
hayley-patton 5y ago
Macros are linguistic abstractions; you describe them as part of your briefing on "key components and essential abstractions". In practice, most macros tend to look the same, e.g. with-<foo> macros only differ in how to acquire and release some <foo>.
closeparen 5y ago
I work in an organization with thousands of engineers; the largest projects have perhaps 10 contributors. Coordination with huge numbers of developers is done through gRPC; doesn’t require compromising on service internals.
- CraigJPerry 5y ago
  That’s a way to reduce the output of your developers - keep them feeling busy but don’t get features to customers any faster. If someone proposed some new dogma to live by: 1. Don’t use method parameters - write some IDL instead, add some steps to your build process to auto gen some bindings from the IDL. Introducing a dependency (with its own security patch lifecycle) in the process. 2. Don’t call a method in a library - instead setup mutual authentication and transport layer security, open a port (probably add load balancing and failover routing), set a reminder to maintain ssl certs before they expire, add distributed logging collectors, add endpoint instrumentation, add service cataloguing and lookup, slow each call down by serialising then deserialising the request then repeat that serdes overhead for the response. … You’d rightly point out their suggestions add complexity and slow things down, what was so bad about calling a library method in the process’s address space in the first place.
  - closeparen 5y ago
    With thousands of developers, we can afford platform teams that make all of those things painlessly on-by-deafault on our standard service frameworks.
didibus 5y ago
Most enterprise software nowadays is broken up in services or micro-services anyways, and monoliths spanning more than 6 to 15 developers are becoming a lot more rare. If you've got a monolith with 500 developers touching the code base you're in trouble even if you used Haskell. Otherwise, macros are actually not super hard to figure out, Lisp macros are easy to write and that means also easy to read and understand, and you've got an interactive development environment so you can quickly explore and try things out to learn how a macro behaves. Like you said, Clojure also doesn't have reader macros, so the syntax is standard across the board, so macro usage is limited in scope to particular call sites. Finally, it's not like you're going to have a myriad of custom macros, and generally when you have them they make things simpler and easier, not harder. So you quickly can ramp up on the common macros of a code base and become familiar with them and start to benefit from their usage as well. They'll also likely have a doc string that documents them, so it's not like you have nothing to go with to understand what they do.
- chii 5y ago
  > Lisp macros are easy to write and that means also easy to read and understand just because something is easy to write doesnt mean it's easy to read and understand - case in point : look at perl.
  - didibus 5y ago
    Fair, in this case my feelings are they are also easy to read and understand, compared to alternatives. Similar behavior implemented with text parsing, or annotation, or pre-processor, or inside the compiler, the code to implement them is a lot less tractable. Once you know how to write macros, it becomes fairly easy to also look at the implementation of a macro and figure out what it does. Macros tend to be very self-contained and often relatively short in implementation.
ByteJockey 5y ago
> How do you use lisp style macros when you have 500 developers working on a single product? May I ask what you are doing that requires 500 devs? I've never heard of a project with that many people on it succeeding.

medo-bear 5y ago

> how do you scale up to a big dev team? can you explain why this is such a big concern? as far as i can tell, you can set development guidelines and employ people who will stick to the rules. it's not like you can sneak in macros and present them as functions

CraigJPerry 5y ago
What happens when you scale java development to a team of that size? Short version: every optimisation you can dream of gets implemented. Clever use of Off heap datastructures aren’t going to be enough to impress. Forget thinking your custom class loader with cooperating javaAgent that manipulates the bytes of classes to dynamically add or remove instrumentation is going to impress. These things are just taken for granted. Even a bot that identifies dead code after traversing the AST, then re-generates valid source code from the (now pruned) AST, runs the test suite then checks in the change all by itself is seen as meh after a while. No matter how out-there an idea is, the play area is constrained by some immutable platform rules (the java memory model). So no matter how funky things get, any other java dev will still be able to reason about any code in the system. What happens when you remove those platform constraints? What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke?
- chii 5y ago
  > What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke? you end up with what amounts to bank python: https://calpaterson.com/bank-python.html https://calpaterson.com/bank-python.html
  - CraigJPerry 5y ago
    This is the best answer so far. Bank Python ain’t so bad. I’m going to stop worrying so much about lisp macro abuse. And I’m going to finally start reading let over lambda.
  - medo-bear 5y ago
    the difference being of course that python the language is not designed for this, whereas lisp is
- medo-bear 5y ago
  > What happens when you remove those platform constraints? What happens when you say, we started off with lisp but we’ve ended up with something quite bespoke? i think lisp is very nicely structured so that you only allow a certain subset to be used according to seniority. if you have good development paractices at hand, then even if you have a very bespoke program it will be well documented and anything ecentric is explained lisp is indeed very powerful, but so is c++ for example. the difference is that the syntax of lisp allows you to be neater where you want power and more conservative where you don't, far more easily

haolez 5y ago

Rails is basically a DSL for web applications and it scales to big teams. It might work this way for Lisp as well.

jonahbenton 5y ago

Highly recommend checking out materials from Nubank, the Brazilian banking startup that now owns Cognitect (and therefore Clojure). They have several hundred engineers on the back end working almost entirely in Clojure. Believed to be the largest Clojure plant, though many other large orgs now have sizable Clojure staffing. I saw some of the Nubank folks speak in person in NYC before the pandemic (and before the acquisition of Cognitect) and it was hands down some of the most impressive work I've encountered. Little in the way of macros, just functions and data. What they have done to scale a Datomic infrastructure is astounding. They have quite a lot of materials online. Worth some time. Now that the Cognitect team is essentially embedded and more resources are going into core, I imagine at scale tooling will see some enhancements in the coming years.

Rovanion 5y ago

I think the lack of reader macros really help here. Macros in Clojure are much more "functions that run at compile time" rather than "look at this fun DSL I threw together". What I think it comes down to though is adherence to convention. A preference hierarchy has grown together with Clojure and it goes: 1. Can it be expressed using data structures, do that. For example a map/dictionary lookup. 2. If that fails, express your more complex idea using a function. 3. If you absolutely must, write a macro. You usually end up with very few macros except for those in the standard library, and those you end up with are more commonly of the convenience kind. One example would be the standard web routing library of the early 10's called Compojure that relied heavily on macros where my current favourite Reitit uses just data.

filoeleven 5y ago
The last time “macros in Clojure” came up, I searched through some of the libraries I’ve run across to see how often they’re used and, as you suggest, it ain’t much. Re-frame has just one, and it looks like it’s used for testing. Clara-rules has eight that are used in the library, which implements a forward-chaining Rete rule engine. Core.logic has a handful, and that’s essentially “prolog inside Clojure.” So clearly those libraries get a lot of mileage out of data and functions, with just a dash of macro. I have yet to reach for them myself, and I can’t imagine a situation where I’d need a macro instead of a function. I think they’re probably great for library developers who are smarter than me, and I’m happy to leave them to it.
- openfuture 5y ago
  I think they come in handy when you've got some functionality that has to happen in a context that you can't easily move between function calls... Then you benefit from the ability to template into the description of that context some actions to perform _before_ you enter into the "black box". That's pretty abstract but honestly its not really such an earthshattering feature. I'd argue it's pretty much necessary for any "final" programming language to be infinitely metaprogrammable (i.e. a lisp-3) but we don't have a mature language like that yet. Probably it will arrive from laskell/hisp kinda synthesis.

invalidOrTaken 5y ago

>How do you use lisp style macros when you have 500 developers working on a single product? In general, you don't, or not at the same rate as functions. In big orgs, macros have three worthy uses: - cleanly separated libraries for the org, providing a common underpinning - smaller projects with in the org - making your language nicer That last one shouldn't be underestimated. I can count on one hand the # of times I've written macros for work, but I use them all the time, as in calling macros others have written. You can very much build a big company on lisp (NuBank the most available-to-mind example). But it's true that at larger team sizes there are extra coordination obstacles that must be surmounted before you get to macro heaven.

chii 5y ago
i think the crux is that it's hard to find 500 people who are all at such high level of capability that could make use of macros in lisp - esp. if they all need to follow a similar style of programming (iirc, lisp macros tend to follow idiosyncrasies of the person writing it). It's almost as though you're asking for 500 Michael Angelo to help paint the sistine chapel in order to do it faster.
- openfuture 5y ago
  Bah. This meme vastly overstate the difficulty... I'm sure there is some nonsense reason why everyone understands templates just fine and the web would literally not exist without them because html is so tedious but "macros" are this mythical complicated impossible to reason about thing??! Seriously there is a limit to how much you can trust all the secondhand garbage "knowledge" you get from reading opinions of people who claim to have tried something. Setting the record straight: - common lisp is fast - common lisp is stable - common lisp is useful! (and interactive! even impatients can get their instant gratification) - common lisp has non-trivial libraries for various useful things - common lisp is simple and has excellent educational resources - common lisp is crufty but the community has such a high average education that actually the resources have aged well and hides at least as much insight and wisdom gems as the type theoretical FP world that we now put on the pedistal we used to keep lisp on. - common lisp is huge. No need to learn it all just hack on what you need and magically you'll be able to make software that works. If macros are such a problem then restrict their use to seniors or "tool devs" (people who go around improving dev ergonomics by extending the language is synonymous to any other DSTool)... This way you have maximum flexibility and by restricting the words you allow you can have this flexibility while you need it while also programmatically restricting your monkeys to a short enough rope that it'll be hard to hang anyone. You can stay in some Java comparible subset (Java because lispers hate on Java like vimers do emacs). Yes I Markov chained a bunch of sentences in this comment... I am too lazy to fully engage with the "someone is wrong on the internet"-thing, anecdotally it has been my feeling that it is no longer a good way to learn something to be "wrong on the internet" nobody is cleaning up anymore... These days everyone has a job; extract from the commons and go rave while Rome burns.

hayley-patton 5y ago

Macros are linguistic abstractions; you describe them as part of your briefing on "key components and essential abstractions". In practice, most macros tend to look the same, e.g. with-<foo> macros only differ in how to acquire and release some <foo>.

closeparen 5y ago

I work in an organization with thousands of engineers; the largest projects have perhaps 10 contributors. Coordination with huge numbers of developers is done through gRPC; doesn’t require compromising on service internals.

CraigJPerry 5y ago
That’s a way to reduce the output of your developers - keep them feeling busy but don’t get features to customers any faster. If someone proposed some new dogma to live by: 1. Don’t use method parameters - write some IDL instead, add some steps to your build process to auto gen some bindings from the IDL. Introducing a dependency (with its own security patch lifecycle) in the process. 2. Don’t call a method in a library - instead setup mutual authentication and transport layer security, open a port (probably add load balancing and failover routing), set a reminder to maintain ssl certs before they expire, add distributed logging collectors, add endpoint instrumentation, add service cataloguing and lookup, slow each call down by serialising then deserialising the request then repeat that serdes overhead for the response. … You’d rightly point out their suggestions add complexity and slow things down, what was so bad about calling a library method in the process’s address space in the first place.
- closeparen 5y ago
  With thousands of developers, we can afford platform teams that make all of those things painlessly on-by-deafault on our standard service frameworks.

didibus 5y ago

Most enterprise software nowadays is broken up in services or micro-services anyways, and monoliths spanning more than 6 to 15 developers are becoming a lot more rare. If you've got a monolith with 500 developers touching the code base you're in trouble even if you used Haskell. Otherwise, macros are actually not super hard to figure out, Lisp macros are easy to write and that means also easy to read and understand, and you've got an interactive development environment so you can quickly explore and try things out to learn how a macro behaves. Like you said, Clojure also doesn't have reader macros, so the syntax is standard across the board, so macro usage is limited in scope to particular call sites. Finally, it's not like you're going to have a myriad of custom macros, and generally when you have them they make things simpler and easier, not harder. So you quickly can ramp up on the common macros of a code base and become familiar with them and start to benefit from their usage as well. They'll also likely have a doc string that documents them, so it's not like you have nothing to go with to understand what they do.

chii 5y ago
> Lisp macros are easy to write and that means also easy to read and understand just because something is easy to write doesnt mean it's easy to read and understand - case in point : look at perl.
- didibus 5y ago
  Fair, in this case my feelings are they are also easy to read and understand, compared to alternatives. Similar behavior implemented with text parsing, or annotation, or pre-processor, or inside the compiler, the code to implement them is a lot less tractable. Once you know how to write macros, it becomes fairly easy to also look at the implementation of a macro and figure out what it does. Macros tend to be very self-contained and often relatively short in implementation.

ByteJockey 5y ago

> How do you use lisp style macros when you have 500 developers working on a single product? May I ask what you are doing that requires 500 devs? I've never heard of a project with that many people on it succeeding.