8 ms·

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineeri

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm bell in your engineering lizard brain goes off), and then it's probably better to commit and learn the ins and outs of that specific problem domain. I think especially the recommendation to commit to it is something that's missing from the article. It's "easy" to point a people and saying "look, you did the thing you said you wouldn't do", but far harder to suggest a course of action. I personally hit the barrier of "things you shouldn't build" in the past, and I've always been happiest (and professional outcomes have been the best) when I decided to break through the barrier and prepare for what lies on the other side, instead of trying to dance around it for ages.

jimmaswell 5y ago
Implementing a small domain-specific language is fairly easy. Why avoid it?
- Karunamon 5y ago
  Because the complexity of the problem tends to scale to your willingness to address said complexity. In other words, it likely won't stay small for long if it gets any users, and now you're the maintainer for a tool used by others.
  - mattgreenrocks 5y ago
    This applies to all software in general. You'll get eaten alive by your success if you don't manage expectations and tightly scope things. At least with a DSL, you can often define a small core language and de-sugar down to that.
  - cbsmith 5y ago
    That principle is not specific to a DSL, so that hardly seems like a reason to avoid a DSL.
    - Karunamon 5y ago
      I'd say it's more of a risk with languages and libraries (things that get deeply embedded into other people's stuff), but I get what you're saying.
- DoctorDabadedoo 5y ago
  Because if it succedds, it needs to be supported long term. If you're doing that inhouse, chances are slim of original designers staying 10-20 years at the same company. Then what you have in your hands is a legacy monolith neglected over its life which might double as core of your stack and it's awful to move away from down the line. Been there, don't recommend it.
  - jimmaswell 5y ago
    In my experience it's fine if you keep its scope limited. The scripting language for an MMO I worked on for example is simple and effective.
    - Scramblejams 5y ago
      Mind sharing which one?
      - jimmaswell 5y ago
        Legends of Equestria
    - TillE 5y ago
      Writing your own scripting language is really not something game developers should be doing for the past decade or so. There are so many good off-the-shelf options ranging from Lua to C#, plus a litany of obscure (but good!) ones. For performance-critical code, even a tiny DSL is not a solution I like unless you're transpiling to C++ or something.
      - jimmaswell 5y ago
        Hooking LUA or similar can introduce a lot of unwanted complexity, be harder for an untrained implementation team member to understand, and be less of an exact fit to the purpose. There are other cases where it makes sense though.
  - mcguire 5y ago
ot 5y ago
I would add encryption/authentication protocols to the list.
- oshiar53-0 5y ago
  Make sure you don't roll your own crypto on your way, though. Unless when it somehow becomes the absolute necessity...or it's for your dissertation.
Cthulhu_ 5y ago
I'm hit by this at the moment. I need to build something so that users can revert to a previous version of a record, but it's constrained to sqlite. I'm (re) building versioned document stores or temporal tables in various POCs now and I'm scared.
mattgreenrocks 5y ago
Yes. Another way to put it is: engineering should be focused on what's needed more than what the Right People On The Internet Say. Additionally, the dirty secret of all those things is they make orders of magnitude more sense to dig into and learn vs the innumerable arcane errors require Googling to fix (obscure Webpack errors, CSS hackery, etc). Your learning accumulates over time.
mamcx 5y ago
You wanna get crazy? When you hit: - a ERP You start to build all the others! (This is where I'm now!)
- hinkley 5y ago
  Look at the bright side. At least Hell is always warm, and plenty of interesting people to talk to.
- Mister_Snuggles 5y ago
  I work on a commercial ERP and this is 100% accurate. It features: * A proprietary language * A compiler for said proprietary language * A way to define tables and views and turn these definitions into real tables and views that live in an SQL database. There is also a custom SQL dialect that gets translated to real SQL on the fly. I'm counting this as having its own database even though it offloads the actual database work to a real database. * One of the products in this ERP suite includes a CMS built on all of the above.
alexjplant 5y ago
IMHO "programming language" would refer specifically to the grammar, type system, semantics, keywords, etc. and "compiler" would refer to the implementation that takes something written in that representation and outputs some type of executable. A pedantic distinction but probably a useful one, especially given how some languages have multiple implementations of their standard. The Programming Languages course I took in college had us implement a toy subset of Ruby targeting the Lua VM. Easily the most interesting and useful class that I had the pleasure of experiencing; it completely changed how I think about languages. I think everybody should have exposure to the basic principles of compilers like syntax trees, flow analysis, calling conventions, and so forth (if not necessarily the more complicated stuff like operational semantics).
- agumonkey 5y ago
  does it affect how you code other kinds of apps too ? knowing how to think about a program interpretation space gives quite a broader view IMO
  - mattgreenrocks 5y ago
    I'd argue the idea of an "interpreter" is one of the most foundational concepts of CS. It sounds really basic, but the idea of being handed a description, and doing/creating something based on that is everywhere. It is really quite beautiful.
    - Zababa 5y ago
      It comes up often too. A few month ago I wanted to make a part of the code I was working on more declarative, so I had it take a javascript object that described a layout, and make actions depending on that object. That's a form of interpreter in a way.
    - chubot 5y ago
      IIRC it's even in the "Design Patterns" book -- the Interpreter pattern. It really does come up a lot.
melony 5y ago
Any tutorials on building the last one? (Beyond basic Rails style CRUD) How do you clone SAP in a weekend?
- hobofan 5y ago
  Heavily depends on what specific incarnation of SAP you mean? If you want to have a SAP replacement for your specific use case, Rails CRUD is the only technological component you need. The much more important part to the recipe is knowledge of the domain model. That's also where SAP's moat lies. Not with it's technological underpinning, but with the all the different industries and their processes which they've transferred into lines of code (and the ability to extend them with further LOC with the help of a "SAP consultant").
derefr 5y ago
> and then it's probably better to commit and learn the ins and outs of that specific problem domain. Even more, it's probably better to start off with a full understanding of these systems, so you can recognize when you're Greenspunning one of these abstractions, and just switch to using a robust off-the-shelf implementation of the same abstraction. (E.g. switch from writing a hand-rolled parser, to using a real parser-combinator library, or writing a formal grammar for a PEG compiler.) Emphasis on "off-the-shelf": the most important thing to get from experience with these abstractions, is the understanding that you it will be less work to use the abstraction that has more power; and, in fact, that much of the tooling around these abstractions is built to assume that you want the full power of the tool by default, such that you have to add configuration to get less. So the workflow is actually "start off ordering 'one with everything', and then streamline your system over time, removing bits and pieces, as you notice properties of your particular situation that mean that you can do with something less powerful at particular stages."
- deleted 5y ago
  [deleted]
- joatmon-snoo 5y ago
  The problem, of course, is that it's impossible to know everything :) IME the best compromise I've found is to learn enough to recognize the problems, so that when I implement a bastardized compiler, I'm aware that that's what I'm doing and that at some point in the future I'll need to decide how deep the implementation should go or switch to actually using some library.
  - derefr 5y ago
    I somewhat disagree — there really is a finite amount of Computer Science to know. The more you dive deep into different domains, the more you find yourself ending up in the same places, learning the same things with different names. What is a SQL query planner but a compiler? Is an OS scheduler really that different from an HPC workload manager, or your GPU's computer-shader arbitrator? Is RDMA against an NVMe storage pool fundamentally new to you if you've already written hierarchical chunk persistence+caching for a distributed key-value store? Etc. At some point, it's all just Computer Science. And not too much of it, either! An amount that could easily be learned in a (good) 4-year undergraduate program.
eulers_secret 5y ago
> …But sometimes you actually _do_ want to build that… That’s not true for everyone. I’ve been a dev for 20 years and have never done any of those, and I have no desire to at all. That is ok! Programming isn’t some kind of progression to run from “n00b hello world” to “1337 compiler hax0r”, it’s a tool to solve problems with. Many of us will never have to solve these problems or have no interest in these problems. So, if you’re like me, don’t worry because you’ve never made a brainfuck interpreter or a database engine. It doesn’t mean you’re a bad dev. You can have a successful career without building these! And if you need them some time, learn them at that time… you may not ever need to.
- hobofan 5y ago
  I think I might have used the expression "do want" to vaguely here. Same as you, I have very little desire to write any of those systems, and still try to avoid them as much as possible (as those endeavors usually require more time/energy, etc. and come with higher risks). Maybe it's better put as "But sometimes all your product requirements strongly suggest that you have to build X and there isn't really a way to avoid it" (and what's available off-the-shelf doesn't fit)?
  - eulers_secret 5y ago
    Thanks for the clarification (even for some rando on the internet), that makes a lot more sense... I think I'll implement a rule for myself where I won't comment on HN posts after first waking up anymore. Agreed that could totally happen!
    - mst 5y ago
      'Put self into read only mode until after the first pint of coffee' is a policy that has significantly reduced the number of times I look back at a comment I've made somewhere and judge myself for its quality. I mean, even fully caffeinated, I still manage to beclown myself on a semi-regular basis, but the 'semi' part is still a net win.
- LeifCarrotson 5y ago
mnw21cam 5y ago
My first job out of university, I ended up doing one of those by mistake. My second job after university, I ended up doing another three of those, but at least we confronted the thing head on and deliberately, knowing that we were aiming to create them. Trust me, it's better to do it deliberately than by mistake.
hinkley 5y ago
Don't forget the paramount of the Mountain of Shit: - a billing system There's a reason why there are b2b companies that specialize in billing. When people branch out it's often billing and X, not X and billing. And even then sometimes they pull back. ADP bought a bunch of companies outside their wheelhouse and then spun them back out.
BeetleB 5y ago
I had a very different reading of the article, colored by my own bad experience. Sometimes you start off with this cool idea/hack to solve a problem that could have been solved some other way. But your cool hack solves the problem and more! Then your fellow team members all want it to be even more powerful, and your cool hack slowly grows into a monstrosity. In these situations, often your manager did not particularly want you to do this, but he let you do what you want. He does want you to be productive, and this doesn't count for that much. But since the team is now dependent on you, he does expect you to maintain it. You now end up with two jobs: The one your manager hired you for, and maintaining this thing. I would love to write my own DSL to solve a problem at work, and this post read to me as a cautionary tale on pursuing that goal - unless management is aligned with your vision. If they're not, solve it the boring (and perhaps nonideal) way.
cjbgkagh 5y ago
Lol, I’ve built all those things. I think the main thing is having respect for the mountain of engineering that went into existing solutions and not rolling your own unless you really really have to. I think an Embedded DSL with post compilation constraint solver based analyzers could take the place of restricted mini-languages while taking advantage of the existing engineering and tooling available for modern languages.
- prionassembly 5y ago
  So like a restricted subset? That's a really neat idea; you should develop it into a paper or something. This restricted subset concept has the huge advantage that you can slip out of the harness in a pinch. So you can reuse a relatively developed Float-less Python program in Full Python and later walk back removing floats. That's really clever. It essentially obviates the whole configuration DSL tango; it also gets people started where the rubber hits the road -- not with syntax and parsing, but with why the DSL exists and what it's supposed to do and not do.
  - cjbgkagh 5y ago
    You can restrict all sort of things, including of course a restricted subset of the language. Such solutions already have headcount at MS which is about as good as it gets for bringing a new thing like this to masses. When it is released and as it matures we can extend it to do a bunch of other cool things like configuration DSL which would really help clean up Linux. Maybe even build scripts. You could do stuff like this before with LISP, or Scala compiler plugins. But the tooling and accessibility will make a big difference.
    - prionassembly 5y ago
      Is there a name for this type of technique I can google?
      - cjbgkagh 5y ago
        For C# there is Roslyn Analyzers which was a foot in the door for a lot of things. F# has LiveCheck which AFAIK is a lot more powerful. I don't think there is much public info out there; Don Syme demos the use of it to add shape checking for Deep Learning models with full IDE interaction, but any arbitrary constraints can be added and constraint solvers are probably the easiest way to manage and compose those constraints. https://youtu.be/0DNWLorVIh4?t=3410 https://youtu.be/0DNWLorVIh4?t=3410
cmrdporcupine 5y ago
I mean I actually really want to build many of those things (the first 3 anyways), and I have. I just don't generally show the results to people or make them use them. I can't vouch for the quality of what I've built, but taking time to attempt to build those kinds of things has always paid off in terms of overall betterment as a programmer. Then I can go back to a day job of mundanely moving protobufs from one place to another.
chillingeffect 5y ago
- a filesystem
munificent 5y ago
I would also add a build system to that list, a tarpit many engineers have fallen into. > not sure that there is a difference to compiler as stated in the article It's hugely different. If you're building a compiler for an existing language, you inadvertently signed yourself up for implementing and supporting the entire language, which is a pretty large engineering task. But if you design a new language, you signed up for not just implementing the language, but also: - Actually designing it, which is a very difficult task with hard trade-offs all over the place. - Documenting it so that others can actually learn it. Technical writing is another hard task that takes years to master. - Syntax highlighting and editor support so that it's as easy to work with as other languages. You will discover that users in the wild apparently use thousands of different editors, all of which you are expected to support. - Static analysis tooling so that code navigation, automated refactoring, linting, etc. all work. Users today expect a rich IDE experience for every language they use. Also, you'll discover the hard way that the architecture of an interactive static analyzer is very different from your batch mode compiler, so now you're likely writing two front ends. - Thousands of answers on StackOverflow, blog posts, introductory videos, etc. Users learn in many different ways and they expect to find education material in the form they prefer.
- crdrost 5y ago
  Note that this also happens pretty much whenever anybody has any sort of template system that they want to support substitution into. You decide that you want to support substitution and suddenly you are building static analysis and documenting and writing syntax highlighting and all this mess. So for example when you want to create a validation system for something, and then you realize that you want to have generics/type-variables/polymorphism, you want validation containers. Bam, the complexity suddenly shoots up! Or maybe you just want to let someone configure with yaml, and then you find yourself with substitutions and substitutions in substitutions and all that mess. Why? The abstract reason is that the underlying mathematics of template substitution is lambda calculus, which is Turing complete. It's hard to learn lambda calculus because it's very abstract, it takes place in a mathematical ideal realm where nothing happens, but this also means that the same pattern reappears in a whole bunch of different contexts, and it is best if we just regard it as a design pattern and conscientiously adopt it. “Oh, this is a Lisp, I have seen the 20 line lisp-in-lisp evaluator, let's just code that in whatever host language and someone else will have thought of the details.”
pklausler 5y ago
In situations of mutual dependence between software components in a big project, implementing a quick "scaffolding" solution for one of them can get development moving on all components independently. A long time ago, I was working on a new incompatible instruction set architecture at a specialized computer systems vendor, and had written a simulator and assembler for it. The O/S group was ready to start porting our UNIX variant, but there was no C compiler. So I wrote one; ANS C89 is a pretty small language. That got everybody unstuck, and the O/S guys could do their port while the compiler group could move at their own pace (and had a partial O/S on which to run compiler test suites). I guess my point is that there can be times when quick "scaffolding" components are reasonable.
cbsmith 5y ago
These aren't "big scary things", they are "technical debt you didn't need to take on". ...and the alternate course of action is clear: use the tools you already have.

jimmaswell 5y ago

Implementing a small domain-specific language is fairly easy. Why avoid it?

Karunamon 5y ago
Because the complexity of the problem tends to scale to your willingness to address said complexity. In other words, it likely won't stay small for long if it gets any users, and now you're the maintainer for a tool used by others.
- mattgreenrocks 5y ago
  This applies to all software in general. You'll get eaten alive by your success if you don't manage expectations and tightly scope things. At least with a DSL, you can often define a small core language and de-sugar down to that.
- cbsmith 5y ago
  That principle is not specific to a DSL, so that hardly seems like a reason to avoid a DSL.
  - Karunamon 5y ago
    I'd say it's more of a risk with languages and libraries (things that get deeply embedded into other people's stuff), but I get what you're saying.
DoctorDabadedoo 5y ago
Because if it succedds, it needs to be supported long term. If you're doing that inhouse, chances are slim of original designers staying 10-20 years at the same company. Then what you have in your hands is a legacy monolith neglected over its life which might double as core of your stack and it's awful to move away from down the line. Been there, don't recommend it.
- jimmaswell 5y ago
  In my experience it's fine if you keep its scope limited. The scripting language for an MMO I worked on for example is simple and effective.
  - Scramblejams 5y ago
    Mind sharing which one?
    - jimmaswell 5y ago
      Legends of Equestria
  - TillE 5y ago
    Writing your own scripting language is really not something game developers should be doing for the past decade or so. There are so many good off-the-shelf options ranging from Lua to C#, plus a litany of obscure (but good!) ones. For performance-critical code, even a tiny DSL is not a solution I like unless you're transpiling to C++ or something.
    - jimmaswell 5y ago
      Hooking LUA or similar can introduce a lot of unwanted complexity, be harder for an untrained implementation team member to understand, and be less of an exact fit to the purpose. There are other cases where it makes sense though.
- mcguire 5y ago

ot 5y ago

I would add encryption/authentication protocols to the list.

oshiar53-0 5y ago
Make sure you don't roll your own crypto on your way, though. Unless when it somehow becomes the absolute necessity...or it's for your dissertation.

Cthulhu_ 5y ago

I'm hit by this at the moment. I need to build something so that users can revert to a previous version of a record, but it's constrained to sqlite. I'm (re) building versioned document stores or temporal tables in various POCs now and I'm scared.

mattgreenrocks 5y ago

Yes. Another way to put it is: engineering should be focused on what's needed more than what the Right People On The Internet Say. Additionally, the dirty secret of all those things is they make orders of magnitude more sense to dig into and learn vs the innumerable arcane errors require Googling to fix (obscure Webpack errors, CSS hackery, etc). Your learning accumulates over time.

mamcx 5y ago

You wanna get crazy? When you hit: - a ERP You start to build all the others! (This is where I'm now!)

hinkley 5y ago
Look at the bright side. At least Hell is always warm, and plenty of interesting people to talk to.
Mister_Snuggles 5y ago
I work on a commercial ERP and this is 100% accurate. It features: * A proprietary language * A compiler for said proprietary language * A way to define tables and views and turn these definitions into real tables and views that live in an SQL database. There is also a custom SQL dialect that gets translated to real SQL on the fly. I'm counting this as having its own database even though it offloads the actual database work to a real database. * One of the products in this ERP suite includes a CMS built on all of the above.

alexjplant 5y ago

IMHO "programming language" would refer specifically to the grammar, type system, semantics, keywords, etc. and "compiler" would refer to the implementation that takes something written in that representation and outputs some type of executable. A pedantic distinction but probably a useful one, especially given how some languages have multiple implementations of their standard. The Programming Languages course I took in college had us implement a toy subset of Ruby targeting the Lua VM. Easily the most interesting and useful class that I had the pleasure of experiencing; it completely changed how I think about languages. I think everybody should have exposure to the basic principles of compilers like syntax trees, flow analysis, calling conventions, and so forth (if not necessarily the more complicated stuff like operational semantics).

agumonkey 5y ago
does it affect how you code other kinds of apps too ? knowing how to think about a program interpretation space gives quite a broader view IMO
- mattgreenrocks 5y ago
  I'd argue the idea of an "interpreter" is one of the most foundational concepts of CS. It sounds really basic, but the idea of being handed a description, and doing/creating something based on that is everywhere. It is really quite beautiful.
  - Zababa 5y ago
    It comes up often too. A few month ago I wanted to make a part of the code I was working on more declarative, so I had it take a javascript object that described a layout, and make actions depending on that object. That's a form of interpreter in a way.
  - chubot 5y ago
    IIRC it's even in the "Design Patterns" book -- the Interpreter pattern. It really does come up a lot.

melony 5y ago

Any tutorials on building the last one? (Beyond basic Rails style CRUD) How do you clone SAP in a weekend?

hobofan 5y ago
Heavily depends on what specific incarnation of SAP you mean? If you want to have a SAP replacement for your specific use case, Rails CRUD is the only technological component you need. The much more important part to the recipe is knowledge of the domain model. That's also where SAP's moat lies. Not with it's technological underpinning, but with the all the different industries and their processes which they've transferred into lines of code (and the ability to extend them with further LOC with the help of a "SAP consultant").

derefr 5y ago

> and then it's probably better to commit and learn the ins and outs of that specific problem domain. Even more, it's probably better to start off with a full understanding of these systems, so you can recognize when you're Greenspunning one of these abstractions, and just switch to using a robust off-the-shelf implementation of the same abstraction. (E.g. switch from writing a hand-rolled parser, to using a real parser-combinator library, or writing a formal grammar for a PEG compiler.) Emphasis on "off-the-shelf": the most important thing to get from experience with these abstractions, is the understanding that you it will be less work to use the abstraction that has more power; and, in fact, that much of the tooling around these abstractions is built to assume that you want the full power of the tool by default, such that you have to add configuration to get less. So the workflow is actually "start off ordering 'one with everything', and then streamline your system over time, removing bits and pieces, as you notice properties of your particular situation that mean that you can do with something less powerful at particular stages."

deleted 5y ago
[deleted]
joatmon-snoo 5y ago
The problem, of course, is that it's impossible to know everything :) IME the best compromise I've found is to learn enough to recognize the problems, so that when I implement a bastardized compiler, I'm aware that that's what I'm doing and that at some point in the future I'll need to decide how deep the implementation should go or switch to actually using some library.
- derefr 5y ago
  I somewhat disagree — there really is a finite amount of Computer Science to know. The more you dive deep into different domains, the more you find yourself ending up in the same places, learning the same things with different names. What is a SQL query planner but a compiler? Is an OS scheduler really that different from an HPC workload manager, or your GPU's computer-shader arbitrator? Is RDMA against an NVMe storage pool fundamentally new to you if you've already written hierarchical chunk persistence+caching for a distributed key-value store? Etc. At some point, it's all just Computer Science. And not too much of it, either! An amount that could easily be learned in a (good) 4-year undergraduate program.

eulers_secret 5y ago

> …But sometimes you actually _do_ want to build that… That’s not true for everyone. I’ve been a dev for 20 years and have never done any of those, and I have no desire to at all. That is ok! Programming isn’t some kind of progression to run from “n00b hello world” to “1337 compiler hax0r”, it’s a tool to solve problems with. Many of us will never have to solve these problems or have no interest in these problems. So, if you’re like me, don’t worry because you’ve never made a brainfuck interpreter or a database engine. It doesn’t mean you’re a bad dev. You can have a successful career without building these! And if you need them some time, learn them at that time… you may not ever need to.

hobofan 5y ago
I think I might have used the expression "do want" to vaguely here. Same as you, I have very little desire to write any of those systems, and still try to avoid them as much as possible (as those endeavors usually require more time/energy, etc. and come with higher risks). Maybe it's better put as "But sometimes all your product requirements strongly suggest that you have to build X and there isn't really a way to avoid it" (and what's available off-the-shelf doesn't fit)?
- eulers_secret 5y ago
  Thanks for the clarification (even for some rando on the internet), that makes a lot more sense... I think I'll implement a rule for myself where I won't comment on HN posts after first waking up anymore. Agreed that could totally happen!
  - mst 5y ago
    'Put self into read only mode until after the first pint of coffee' is a policy that has significantly reduced the number of times I look back at a comment I've made somewhere and judge myself for its quality. I mean, even fully caffeinated, I still manage to beclown myself on a semi-regular basis, but the 'semi' part is still a net win.
LeifCarrotson 5y ago

mnw21cam 5y ago

My first job out of university, I ended up doing one of those by mistake. My second job after university, I ended up doing another three of those, but at least we confronted the thing head on and deliberately, knowing that we were aiming to create them. Trust me, it's better to do it deliberately than by mistake.

hinkley 5y ago

Don't forget the paramount of the Mountain of Shit: - a billing system There's a reason why there are b2b companies that specialize in billing. When people branch out it's often billing and X, not X and billing. And even then sometimes they pull back. ADP bought a bunch of companies outside their wheelhouse and then spun them back out.

BeetleB 5y ago

I had a very different reading of the article, colored by my own bad experience. Sometimes you start off with this cool idea/hack to solve a problem that could have been solved some other way. But your cool hack solves the problem and more! Then your fellow team members all want it to be even more powerful, and your cool hack slowly grows into a monstrosity. In these situations, often your manager did not particularly want you to do this, but he let you do what you want. He does want you to be productive, and this doesn't count for that much. But since the team is now dependent on you, he does expect you to maintain it. You now end up with two jobs: The one your manager hired you for, and maintaining this thing. I would love to write my own DSL to solve a problem at work, and this post read to me as a cautionary tale on pursuing that goal - unless management is aligned with your vision. If they're not, solve it the boring (and perhaps nonideal) way.

cjbgkagh 5y ago

Lol, I’ve built all those things. I think the main thing is having respect for the mountain of engineering that went into existing solutions and not rolling your own unless you really really have to. I think an Embedded DSL with post compilation constraint solver based analyzers could take the place of restricted mini-languages while taking advantage of the existing engineering and tooling available for modern languages.

prionassembly 5y ago
So like a restricted subset? That's a really neat idea; you should develop it into a paper or something. This restricted subset concept has the huge advantage that you can slip out of the harness in a pinch. So you can reuse a relatively developed Float-less Python program in Full Python and later walk back removing floats. That's really clever. It essentially obviates the whole configuration DSL tango; it also gets people started where the rubber hits the road -- not with syntax and parsing, but with why the DSL exists and what it's supposed to do and not do.
- cjbgkagh 5y ago
  You can restrict all sort of things, including of course a restricted subset of the language. Such solutions already have headcount at MS which is about as good as it gets for bringing a new thing like this to masses. When it is released and as it matures we can extend it to do a bunch of other cool things like configuration DSL which would really help clean up Linux. Maybe even build scripts. You could do stuff like this before with LISP, or Scala compiler plugins. But the tooling and accessibility will make a big difference.
  - prionassembly 5y ago
    Is there a name for this type of technique I can google?
    - cjbgkagh 5y ago
      For C# there is Roslyn Analyzers which was a foot in the door for a lot of things. F# has LiveCheck which AFAIK is a lot more powerful. I don't think there is much public info out there; Don Syme demos the use of it to add shape checking for Deep Learning models with full IDE interaction, but any arbitrary constraints can be added and constraint solvers are probably the easiest way to manage and compose those constraints. https://youtu.be/0DNWLorVIh4?t=3410 https://youtu.be/0DNWLorVIh4?t=3410

cmrdporcupine 5y ago

I mean I actually really want to build many of those things (the first 3 anyways), and I have. I just don't generally show the results to people or make them use them. I can't vouch for the quality of what I've built, but taking time to attempt to build those kinds of things has always paid off in terms of overall betterment as a programmer. Then I can go back to a day job of mundanely moving protobufs from one place to another.

chillingeffect 5y ago

- a filesystem

munificent 5y ago

I would also add a build system to that list, a tarpit many engineers have fallen into. > not sure that there is a difference to compiler as stated in the article It's hugely different. If you're building a compiler for an existing language, you inadvertently signed yourself up for implementing and supporting the entire language, which is a pretty large engineering task. But if you design a new language, you signed up for not just implementing the language, but also: - Actually designing it, which is a very difficult task with hard trade-offs all over the place. - Documenting it so that others can actually learn it. Technical writing is another hard task that takes years to master. - Syntax highlighting and editor support so that it's as easy to work with as other languages. You will discover that users in the wild apparently use thousands of different editors, all of which you are expected to support. - Static analysis tooling so that code navigation, automated refactoring, linting, etc. all work. Users today expect a rich IDE experience for every language they use. Also, you'll discover the hard way that the architecture of an interactive static analyzer is very different from your batch mode compiler, so now you're likely writing two front ends. - Thousands of answers on StackOverflow, blog posts, introductory videos, etc. Users learn in many different ways and they expect to find education material in the form they prefer.

crdrost 5y ago
Note that this also happens pretty much whenever anybody has any sort of template system that they want to support substitution into. You decide that you want to support substitution and suddenly you are building static analysis and documenting and writing syntax highlighting and all this mess. So for example when you want to create a validation system for something, and then you realize that you want to have generics/type-variables/polymorphism, you want validation containers. Bam, the complexity suddenly shoots up! Or maybe you just want to let someone configure with yaml, and then you find yourself with substitutions and substitutions in substitutions and all that mess. Why? The abstract reason is that the underlying mathematics of template substitution is lambda calculus, which is Turing complete. It's hard to learn lambda calculus because it's very abstract, it takes place in a mathematical ideal realm where nothing happens, but this also means that the same pattern reappears in a whole bunch of different contexts, and it is best if we just regard it as a design pattern and conscientiously adopt it. “Oh, this is a Lisp, I have seen the 20 line lisp-in-lisp evaluator, let's just code that in whatever host language and someone else will have thought of the details.”

pklausler 5y ago

In situations of mutual dependence between software components in a big project, implementing a quick "scaffolding" solution for one of them can get development moving on all components independently. A long time ago, I was working on a new incompatible instruction set architecture at a specialized computer systems vendor, and had written a simulator and assembler for it. The O/S group was ready to start porting our UNIX variant, but there was no C compiler. So I wrote one; ANS C89 is a pretty small language. That got everybody unstuck, and the O/S guys could do their port while the compiler group could move at their own pace (and had a partial O/S on which to run compiler test suites). I guess my point is that there can be times when quick "scaffolding" components are reasonable.

cbsmith 5y ago

These aren't "big scary things", they are "technical debt you didn't need to take on". ...and the alternate course of action is clear: use the tools you already have.