37 ms·

Dear sir, you have built a compiler

355 points by rachitnigam 5y ago

sesuximo 5y ago
Oof
olivierduval 5y ago
I'm always surprised by the lack of "easy" parser/compiler toolkit for more or less complete DSL... It looks like there's only - either full blown programming language for IT guys - "natural language" AI toolkits (not really usable yet for just simple automation by users) - graphical language like State Machine or "no code", missing loops and requiring mouse and boxes However, most of the time, user simply need some kind of BASIC (like visual basic used a lot in "shadow IT" by users)... but there's really few libraries providing that kind of functionalites
- mrlemke 5y ago
  Sounds like a gap TCL could fill. The syntax is simple and consistent, a developer could create packages that a non-developer end user can use as commands, and it could be wrapped with GUI code that runs on Windows, Mac, or Linux.
  - orthoxerox 5y ago
    Tcl is good, but non-developers could really benefit from more write-time checks. Something that can tell them "there's no command called moveRight, the nearest match is move-right" before their script is executed.
    - mrlemke 5y ago
      I agree. I would really like the same. Komodo IDE may help with this, but I haven't tried it. I usually end up in vim due to habit.
  - blacksqr 5y ago
    Linked article appears to be describing the exact original motivation for writing Tcl: "My students and I had written several interactive tools for IC design, such as Magic and Crystal. Each tool needed to have a command language (in those days people tended to invoke tools by typing commands; graphical user interfaces weren't yet in widespread use). However, our primary interest was in the tools, not their command languages. Thus we didn't invest much effort in the command languages and the languages ended up being weak and quirky. Furthermore, the language for one tool couldn't be carried over to the next, so each tool ended up with a different bad command language. After a while this became tiresome and embarrassing." https://web.stanford.edu/~ouster/cgi-bin/tclHistory.php https://web.stanford.edu/~ouster/cgi-bin/tclHistory.php
- jasfi 5y ago
  What about ANTLR?
  - olivierduval 5y ago
    Actually, I was thinking about having an "already build" BASIC language... not creating a new one each time. As a programmer, you would include the libray, provide the binding to your program... and done ! In a way, it's a bit what is possible with java.scripting but they provide a Javascript engine but no BASIC for end-user
- jacquesm 5y ago
  There's Forth. Many would argue it's not a full blown programming language, it's more like a toolkit to make DSLs.
  - K0balt 5y ago
    It’s great when you don’t need to have a huge codebase.IME forth gets out of control quickly from a maintainability standpoint as you go big, but it is excellent at small dsl type tasks. The maintainability issues with forth might actually be solvable if the right IDE paradigm could be found.
    - jacquesm 5y ago
      I don't think so. Let me try to explain: regular languages are only viable as long as someone is willing to speak them and learn them. Latin being the obvious exception, but that's mostly because of medical science and biology, two very large branches of the scientific tree. A DSL is a language with only a handful of users, each of which is usually in the normal path of employment (obvious exception: Chuck Moore). As soon as someone in a team that relies on a DSL leaves the loss of knowledge is immediate. Unlike any other language that you may have picked up in some other organization the DSL has instantly lost a sizeable fraction of those versed in it. It doesn't take many losses like to kill such a project. Another problem is that even if the original authors hang around, the chances that they will stay fluent in their own DSL diminishes with increasing intervals between work on a particular project. If the code would be written in a 'normal' language then they would stay fluent. So DSLs are not a good match for how we normally think about software, even though there is a 1:1 correspondence between the words in your typical DSL (say a FORTH word) and a function call. Functions tend to be written in terms of other functions, just like FORTH words are written in terms of other words, and yet, the standard library is never far away whereas with FORTH words it could easily be 30 words down before you hit the actual language. I'd love to see something like a DSL based language but with longer term prospects, for a long time I hoped that Smalltalk would be that language but I have mostly given up on that.
      - MaxBarraclough 5y ago
- yetihehe 5y ago
  > However, most of the time, user simply need some kind of BASIC "most of the time" is THE problem here. Once users approach a barrier (missing language feature) they will complain and push you until you implement that feature. After several such features some developer in basement of your building will say "ok, fine, I'll do it" and now you accidentally have a compiler. Edit: I know because I've recently made a "almost universal data watcher" which can react to combination of measured values changed in different devices and execute actions. Simple solution, but I fear the day when someone needs to execute other recipes in response to data and creates loops.
  - Dylan16807 5y ago
    That's a problem, but before that what's the good answer to "how can I take advantage of existing libraries to make a quality but simple BASIC-like DSL?" And in particular, where's a well-thought-out BASIC-like template I can customize?
- bena 5y ago
  I think there's a tipping point. You start with a configuration or markup language. Then you add and add and add and next thing you know, you've built a very simple scripting language. It's kind of like those people who cobble together a Turing complete implementation of HTML+CSS or Magic: the Gathering game. You can kind of stumble into completeness.
- throwfaangus 5y ago
  Just use flex / bison?
  - Koshkin 5y ago
    Looks like these are grossly underappreciated these days. They are wonderful, easy to use tools.
    - neurotrace 5y ago
      I think "easy to use" might be an overstatement. I spent a fair bit of time trying to get in to the flex/bison workflow and it never seemed to click. I've found it much easier to use PEG parsers, parser combinators, or a hand-written recursive descent parser. Based on their ubiquity and the high reviews, I'm sure they're great tools once the initial hump is passed but that initial hump is quite large.
    - duped 5y ago
      I think it's because they're less easy to use tools than writing your own lever/parser, which takes about a day and is fully introspective and debuggable. And if you need decent error reporting at the input it's much easier to thread that through. Plus almost everything language has some kind of PEG or parser combinator library these days.
  - therealcamino 5y ago
    I've always found ANTLR to be more intuitive and maintainable than the flex/bison combination. And the learning curve is gentler.
    - base698 5y ago
      And the tooling is better. I always found it hard to find working examples with bison.
- brandmeyer 5y ago
  The intermediate solution is a programming language and/or runtime specifically designed as an extension system. I've had positive experiences embedding Lua. GNU Guile was specifically designed for this too, if you think your users can deal with the parens.
- rachitnigam 5y ago
  Racket is often heralded as the "programming language programming language" and comes with tons of features to build full blown languages: https://racket-lang.org/ https://racket-lang.org/ The most complete language/ecosystem that really showcases Racket's capabilities is Rosette IMO: https://github.com/emina/rosette/ https://github.com/emina/rosette/
  - noblethrasher 5y ago
    Also worth noting that language that runs this very website was implemented in Racket.
    - mepian 5y ago
      I think it was implemented in PLT Scheme before it stopped being a Scheme implementation and became Racket.
    - eitland 5y ago
      I thought that was Arc?
      - electroly 5y ago
        It is. Arc itself is written in Racket.
        eitland 5y ago
        Ok, so pretty good proof that racket is a working programming language programming language then :-)
    - danuker 5y ago
      Hacker News runs on a language that runs on Racket? Which language is that?
      - krapp 5y ago
- YeGoblynQueenne 5y ago
  There's Prolog and its Definite Clause Grammars (DCG) formalism. Here's a DCG for a tiny subset of natural English (copied from wikipedia [1]): sentence --> noun_phrase, verb_phrase. noun_phrase --> det, noun. verb_phrase --> verb, noun_phrase. det --> [the]. det --> [a]. noun --> [cat]. noun --> [bat]. verb --> [eats]. The syntax is just like BNF. "-->" is "::=", terms in []'s are terminals and the rest are nonterminals. Nonterminals can have arguments and there's special syntax to include entire Prolog programs in the body of rules. I discovered DCGs at the end of my CS degree when I was working on my graduation project, a Magic: the Gathering expert system written in Prolog. I needed a way to script cards for my rules engine, to translate expressions in the M:tG ability text language to function calls in the engine. I really wanted to write a parser using the standard parser generator tools, but I thought it would be too much work for a graduate project. Then I found out about DCGs. With DCGs I could write grammar rules like this: tap(X,Y) --> (['Tap'];[tap]), target(X,Y), (condition ; []), {permanent_Type(Y)}. tap(X,Y) --> (['Tap'];[tap]), all(X,Y), (condition ; []). And that would not only parse, but also _generate_ ability text like "Tap target Kavu" or "Tap all Goblins target player controls" etc. So as a side effect of adding a bunch of abilities to my engine, I also had a generator for new M:tG cards. [Edit: to be clear, the grammar rules above are already, by themselves, a parser for abilities that start with "tap". The Prolog interpreter executes those grammar rules as a program and instantiates their variables in the process. To integrate them with the engine all I had to do was write engine functions that called the nonterminals in the grammar and read the values of variables like X and Y in tap(X,Y).] Admittedly, my grammar was limited because I didn't have the time to hand-code a grammar for the entirety of M:tG ability text, as it was at the time (I graduated 2011). In any case my rules engine was also limited (I had no planeswalkers and static effects were only partially implemented, if I remember correctly; because the rules for static effects are a bit arcane). But I managed to cover a good subset of the rules and cards, so it was possible to play a basic game with creatures with activated and triggered abilities and with non-creature spells [2]. The important detail is that DCGs are syntactic sugar for ordinary Prolog, so I could code my parser in the same language as my engine and my player. My biggest problem was really to draw an imaginary line between the parser and the other two components, because the easiest thing would be to allow them to know about each other and be infinitely entangled in a big ball of mess. Anyway, yeah, there's something out there that is perfectly suited for creating DSLs, if you have the patience [3]. The tradeoff is that you have to learn Prolog. I suspect this is a cost too high for many who just want an easy-to-use tool. Btw, you can't easily run my degree project unfortunately. Like an idiot I wrote it for a proprietary Prolog system and although I tried to port it to the free and open-source SWI-Prolog, it's been a nightmare of incompatibilities. The ported project is here but it breaks unexpectedly all the time: https://github.com/stassa/Gleemin https://github.com/stassa/Gleemin ________ [1] https://en.wikipedia.org/wiki/Definite_clause_grammar#Example https://en.wikipedia.org/wiki/Definite_clause_grammar#Exampl... [2] I also had an alpha-beta minimax AI player. Not that good though. I wanted to program a player with background knowledge of M:tG but I didn't have the time. [3] Or you can learn them from examples... See Experiment 3 on page 19 of the pdf, here: https://link.springer.com/article/10.1007/s10994-020-05945-w https://link.springer.com/article/10.1007/s10994-020-05945-w
- jerf 5y ago
  The easy options: As others have said, embed something existing. I think this is especially true for programming languages. Even if Lua isn't quite what you want, it's probably going to deliver a lot more value than anything you could bash together on your own. And there are a few other options as well. There are several of these "complete toolkits for DSLs", it's just they do take a bit of work to correctly insert into your code base and then expose enough of what you're doing to make it useful. There isn't and as far as I can see can't be something you just install and it magically exposes everything and it's just awesome and takes no work, because even if you're in something like Python where you can just open an interpreter shell in the current context, thereby technically giving the user the power to do anything they want, you still need to create a safe interface for them, document it, test it, etc. (You may not do a lot of technical work to keep them out of where you haven't tested but you will need to provide them a supported area to get anywhere.) For building your own: For small tasks, a parser is just a specialized sort of deserializer. Yea verily these 30 years ago, there wasn't a lot of generic deserializers available, and the ones that existed like asn.1 were completely unsuitable for human use, so "classic" compiler literature spends a lot of time on building your own parser. Now, if you don't have a specialized need, you should use a generic deserializer. I have quite a few JSON-based syntaxes lying around. (They feed "interpreters", but same principle.) Lisp S-expressions, if suitable for your environment, can be adapted. XML could be useful in some specialized circumstances where your input looks more like a document than conventional code. Etc. You should only build a custom parser if it's going to give you more bang for your buck across the whole usage cycle than a generic parser can give you. The reason why I have several things that input JSON is precisely that the amount of usage across all time was just going to be too low to justify doing anything specialized, because it was always a strictly internal tool. (Hint: Just because your compiler accepts JSON as input doesn't mean that that's what the humans have to directly output or edit. My biggest interpreter took in JSON but I provided abundant function helpers that abstracted that away. It was a thin enough abstraction that it could be penetrated if need be, but it avoided most of the verbosity issues as far as the humans were concerned, and gave them the full power of the programming language to generate loops and such. If you squint, I basically borrowed an existing general-purpose programming language that everyone involved already knew to use as a macro language for my language, basically for free. This means the "core" language that I was implementing didn't have to do anything that the already-existing general-purpose language could do in macros, meaning the layer the humans used offered a lot of power for not much implementation cost on my side.) For AST transforms, this is always where your secret sauce is and there's little help to give, beyond a suggestion that this is arguably the canonical case for a functional programming style, even if you are in an otherwise OO language. For the final output, I would suggest that in modern times, the easy answer is, don't bother. You can probably interpret fast enough to solve your problems. If not, there are some options like LLVM that will help vs. what you used to have to do. But I would definitely consider just interpreting the final AST directly. If you intend to do that your AST transforms can be built with interpretation in mind. The real value of a compiler and what distinguishes it from any ol' normal code that ingests something and emits something is the AST in the middle, and the complexity of that AST (probably through significant recursion) and the transforms you run on it, and the corresponding complex behaviors the input can invoke through the recursive input and complex AST transforms. Anything you can do to brush away the incidental details of a compiler by using existing serializations and if at all possible simply skipping the output step of the compiler will hugely increase the bang-for-the-buck you get. You might say, rather than "writing a compiler" consider "using compiler techniques" but not writing a compiler. But not in a way the article was talking about. For the specific case mentioned there, go get an existing parser for your language, get the full AST tree. You can write your transform in terms of what you do understand and just pass through anything you don't. It's slightly harder, but will be fairly effective. It is possible even today, of course, that you will be backed against a wall and be forced to go through the full classical compiler cycle. A programming language, for instance, just can't afford to use any existing parser, it's too much a vital part of the experience to try to use an existing one. Even a new Lisp will probably want its own spin on S-expressions enough that it can't just pick up an existing one unchanged. And maybe you've got something that you expect will be big enough that it just requires a new syntax, and you can afford all the support that goes with that. But the vast majority of the time, with the right toolset, you can indeed obtain most of the value of compiler-like techniques without the cost of the full classical compiler technology stack, because even if it was hypothetically possible to build a "real compiler" that had its own custom syntax rather than some slightly dodgy JSON and emitted LLVM code that resulted in an executable that runs all its tasks in .0003 seconds, the engineering effort for that is so much greater than the JSON input and interpreter that runs in .003 seconds that in most cases it's not worth it.
- derefr 5y ago
  Most people who want this end up just selecting a regular programming language with a powerful-enough macro system + flexible-enough compile-time parser; and then using said language to define a "more or less complete DSL", by redefining all the primitive syntax features of the language to instead be primitive syntax features of the DSL. For example, the "numerical definition" DSL within Elixir's Nx library (https://github.com/elixir-nx/nx/tree/main/nx#numerical-definitions https://github.com/elixir-nx/nx/tree/main/nx#numerical-defin...). Users using this "full" DSL are still using the host language — they're invoking its compiler/runtime and so forth — but they don't necessarily need to care about that. They can still technically access the power of a full-blown programming language, "tucked in" around the edges of the DSL — but you as the DSL's creator, don't have to document that power as being available. As long as users of the DSL can do everything they need to do without any need for "breaking out" of the DSL's little world, they don't need to be aware that they could e.g. make fully-namespaced calls to regular stdlib functions of your runtime.
- lowbloodsugar 5y ago
  JetBrains has a MPS (Meta Programming System) [1] that I believe is for knocking up DSLs. At a lower level, I've used ANTLR [2] a bunch of times, and its StringTemplate even more often. [1] https://www.jetbrains.com/mps/ https://www.jetbrains.com/mps/ [2] https://www.antlr.org https://www.antlr.org
pddpro 5y ago
Oh this brings a lot of memories. I remember writing a DSL for visualization of logs that we collected when we were working in a hyper local geographical search engine. I was fresh out of college and this was when Google maps had yet to penetrate my country. I started out with pylex, yacc and then what was supposed to be a quite simple DSL with a few keywords and a few nodes, very quickly exploded to a very complicated endeavor. Alas, I'll never be able to prove that it was Turing Complete. But that doesn't stop me from believing it.
thriftwy 5y ago
Related: https://steve-yegge.blogspot.com/2007/06/rich-programmer-food.html https://steve-yegge.blogspot.com/2007/06/rich-programmer-foo...
- cormacrelf 5y ago
  Overall decent take on why you should go for a compilers course, but it has a very weird section advocating for non-deterministic type checking. You want probabilistic AI methods to guess that this string is more of an integer? Buddy, knowing what kind of data we have for certain is the only reason we built type checkers in the first place. Take your 90% chance int but 10% chance string and get out of here. Maybe he means “error messages are bad, we should guess what people mean and make better suggestions on type errors” but the case was not argued well. It also stuck with an article-long joke of accusing compiler people of being boring at parties. This is 2007 after all, an unenlightened time.
sunir 5y ago
I think you should almost always build a compiler when presented with a parse then execute problem. It is so much faster to create a small machine that is fed data to specific the program, and a test harness that also drives that machine, then it is to hand code every little case repetitively. Or maybe I like building little compilers.
- tylerscott 5y ago
  I completely agree with the sentiment but also feel like it is just my bias because I enjoy writing compilers.
- deleted 5y ago
  [deleted]
- ianbicking 5y ago
  When you describe it that way it reminds me of SAX [1] – I always hated SAX, but eventually realized it was kind of a tokenizer that left it up to the developer to figure out how to turn that into a compiler, though in this case compiling XML input into some internal data structure or action. [1] https://en.wikipedia.org/wiki/Simple_API_for_XML https://en.wikipedia.org/wiki/Simple_API_for_XML
  - sunir 5y ago
    Weirdly, but sometimes ideas churn around HN, I just replied to another thread about this. See my reply on this thread https://news.ycombinator.com/item?id=29917060 https://news.ycombinator.com/item?id=29917060 At BitFlash, one of the things we had to build was a SAX parser for the SVG DOM. I used the DSL of the W3C spec to compile the SAX parser. One of the more strange things I did was in some contexts (think old school Blackberry) we had a server to pre-parse the SVG so we "knew" it was clean (I'm still sceptical 20 years later this is ever a good strategy, but take it as a given). Because we knew the SVG was clean, there was a faster way to parse the XML than reading the tokens. I used my magic Perl script transformer to compute the lowest entropy decision tree to identify a token with the fewest comparisons, which was surprisingly way more efficient than a trie.
  - 5y ago
3pt14159 5y ago
This is a great article, and if I were writing path-critical code for the situation the author describes I'd use an AST too, but I still munge me some text and most of the time it's good enough. The problem with using AST libraries is that they're hard to understand unless you're used to them. So if you have to choose between a couple hundred lines of python or ruby and a giant AST I think it's smarter to go with the ruby. That way when something like a new emoji comes out, or what have you, the junior dev can handle the minor bug fix. Your AST is still going to choke on it. No way around it, the world has changed. That said, there is an art to writing text processors and keeping things sane. The most important bit I've learned is to avoid transformations during extractions even at the cost of performance. At least until a fully exhaustive test suite is up. It makes it easier to test and it's a clearer separation of responsibilities. I've written a lot of these parsers. One for a custom format for the Nasdaq on a project I was the lead on! I shudder to think of how many bits that thing is processing since it pipes data to and from thousands of data brokers and hedge funds.
bitwize 5y ago
"Select the pistol, and then, select your compiler." https://www.penny-arcade.com/comic/2008/05/26/the-unhorse https://www.penny-arcade.com/comic/2008/05/26/the-unhorse
TillE 5y ago
YAGNI is a good principle here. Whenever I've found myself thinking about reaching for a parser library, I was over-complicating or over-generalizing the problem. Write the code you need to solve the problem you actually have.
- cmyr 5y ago
  I agree completely. Anytime I’m looking at a parser library I just shake my head and close that browser tab. I’m invariably going to want a hand-rolled recursive descent parser two weeks later, so let’s just get on it.
  - orthoxerox 5y ago
    So much this. I've used a parser generator successfully exactly once, and even then I simply didn't care about proper error reporting. I wouldn't mind something reusable that generated red-green trees for me a la Roslyn. I can bang out a recursive descent parser in a day, but not one that can be run after every keystroke.
  - naasking 5y ago
    There's a middle ground with parser combinators.
- quotemstr 5y ago
  The article makes a good argument against YAGNI, at least YAGNI applied in its naive form. The point is that if you're going to make an embedded language, just do it the right way from the start instead of trying to cobble it together with YAGNI-inspired half-assed implementations that break in weird cases.
- TheCoelacanth 5y ago
  YAGNI works well as long as you are willing to re-write your code in the rare cases when you do "need it". It's fine to start without a real parser, and you often will end up not ever needing a real one, but if the complexity of your ad hoc solution starts getting anywhere close to the complexity of a real parser, you are better off re-writing it as a real parser because parsing is an extremely well-studied area of CS while your ad-hoc "not parsing" is not.
bugmen0t 5y ago
Ah, again? Happens to me all the time.
DarylZero 5y ago
Seems like the idea of exposing a programming language to the end-user should have better support so that it's more routine to create a restricted subset of something existing for users to put arbitrary code in. Something with both standardized textual interface and standardized structured editor interface would be great. I guess this was always the dream for LISP but it never got there.
ModernMech 5y ago
This happens to so many people, it’s a tale as old as programming languages. If you ever find yourself saying “wouldn’t it be neat…” or “if I could just only…”, stop yourself immediately — you may be about to accidentally spend the next 3-5 years writing a compiler. So many compilers started out looking to implement small feature X, and then ended up spending ungodly amounts of time writing an entire compiler instead. The scariest thing about compilers is by writing one, it induces a kind of amnesia where you can’t remember why you started writing it in the first place! Soon enough the point of the compiler is to write the compiler, and feature X goes unimplemented. Eventually it will be possible once the compiler is done. But the compiler is never done… never… done…
hobofan 5y ago
I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm bell in your engineering lizard brain goes off), and then it's probably better to commit and learn the ins and outs of that specific problem domain. I think especially the recommendation to commit to it is something that's missing from the article. It's "easy" to point a people and saying "look, you did the thing you said you wouldn't do", but far harder to suggest a course of action. I personally hit the barrier of "things you shouldn't build" in the past, and I've always been happiest (and professional outcomes have been the best) when I decided to break through the barrier and prepare for what lies on the other side, instead of trying to dance around it for ages.
- jimmaswell 5y ago
  Implementing a small domain-specific language is fairly easy. Why avoid it?
  - Karunamon 5y ago
    Because the complexity of the problem tends to scale to your willingness to address said complexity. In other words, it likely won't stay small for long if it gets any users, and now you're the maintainer for a tool used by others.
    - mattgreenrocks 5y ago
      This applies to all software in general. You'll get eaten alive by your success if you don't manage expectations and tightly scope things. At least with a DSL, you can often define a small core language and de-sugar down to that.
    - cbsmith 5y ago
      That principle is not specific to a DSL, so that hardly seems like a reason to avoid a DSL.
vinceguidry 5y ago
You need a compiler when you need language, that is, when you need to interface directly between brain and computer. Everything else can be less messily-specified. A good middle ground is the DSL, domain-specific language. You use the syntax and semantics of the language being used to code in, and change the nouns and verbs. All the nouns and verbs are given meaning as values or functions. Execute in a sandbox so that your DSL can't find the rest of the language. Ruby makes this all trivial. Of course, if you're trapped in a static language with no way to execute significant logic at runtime without a ton of heavy lifting, then yeah, you'll end up maintaining an interpreter / compiler if you're not careful. Really do yourself a favor and see if you can't introduce Ruby, you can adapt it to your needs. You can inherit from BasicObject, BasicObjects can't see anything you don't want them to see.
meinte37 5y ago
The article is “addressed to those who did not want to build a compiler”: what's the point of this article then? It seems to implore to not build compilers, but those who did not want to in the first place, probably actually did not build compilers. The article does point out numerous challenges with building compilers, and, by extension, with software language engineering, which concerns itself not just with building compilers but with the design of languages, and implementation of non-compiler tools around it. It merely points out the existence, and likely occurrence, of those challenges, and it can be surmised that the author is frustrated by frequent experience in his own daily life. A bespoke software language -or DSL, if you will- is not always the best solution. It really depends on what you carve out as the domain, how fast that domain changes, what costs and risks are associated with those changes, and with which people you have to implement those changes. No amount of tooling is going to help you out if the domain you chose to recognize as such isn't changing fast enough, isn't costly or risky enough to change (quickly), or the domain stakeholders are anyway not helping out. But in case you do want to build a DSL, have a look at a language workbench like JetBrains' MPS, or this book I happen to be writing: https://www.manning.com/books/domain-specific-languages-made-easy https://www.manning.com/books/domain-specific-languages-made...
- a_shovel 5y ago
  It's not a warning not to build a compiler, it's more of a story about how feature bloat might cause one to inadvertantly build a compiler without realizing that's what they were doing. The last line reveals it by telling us that each of those "features" was actually a standard compiler component: > Done at last, you say to yourself, without having to build a compiler. > A parser, an intermediate representation, transformation passes, and a code generator. Dear Sir, you have built a compiler.
  - rachitnigam 5y ago
    [OP] Yup, that was the intent. I've seen a lot of academic and industrial hand wringing about not wanting to build a compiler because it's an overkill for the solution only to have the people come back 6 months later having built a compiler. I'm not saying everyone should build a compiler for everything–but when you should build a compiler, you really should bite the bullet and build a compiler.
lupire 5y ago
https://en.m.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.m.wikipedia.org/wiki/Greenspun%27s_tenth_rule "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." (Same applies to popular languages after the original quote.)
- shadowgovt 5y ago
  What often goes unsaid is the corollary: "Any Common Lisp implementation people use contains an ad-hoc, informally-specified, bug-ridden optimizing compiler written in C or Fortran."
  - jhgb 5y ago
    That makes no sense. Which Common Lisp implementations contain a compiler written in C or Fortran?
    - shadowgovt 5y ago
      I should have been clearer: written partially in C or Fortran. SBCL, for example, has C bindings in the /runtime subdirectory.
      - jhgb 5y ago
        > SBCL, for example, has C bindings in the /runtime subdirectory. So nothing to do with SBCL's compiler (which is somewhat confusingly named 'Python' and written entirely in Lisp), then.
        shadowgovt 5y ago
        Well, handwave handwave. Nothing to do with the compiler except that I'd assume if you rip out the /runtime directory you'll get a compiler that can't output working code. The point is that these domains have a tendency to eventually overlap each other.
0xbadcafebee 5y ago
Ways you can avoid building a compiler: 1. Lower the scope of the solution. If your solution is designed to address 100% of the users' problems, it's probably too grand of a solution. Start with 80% of the problems. Identify all the problems, rank them by priority and difficulty, and leave the most difficult and least priority problems out. 2. Lower your expectations. Imagine your great idea. Now imagine how you will implement it. Now imagine it is 100x more difficult than you imagine. Woof, that's hard! Strip down the implementation to the essentials needed to solve the immediate problems. 3. Give the user the minimum possible functionality to address their needs. 4. Go find an existing solution that provides these requirements. If you did it right, you will probably find a solution that you really don't like but already exists and solves the problems you need to solve right now.
- mcguire 5y ago
  And in six months, the users will be hacking around the parts you left missing, probably by emailing Excel spreadsheets around. If you did it right, you will be off to your next place of employment before the wheels completely come off.
choeger 5y ago
The language pattern occurs quite frequently in many domains. Unfortunately, I have so far only seen half-assed interpreters. At best, people did a shallow embedding (interpreter) of their DSL. No one ever did a true compiler. I think the reason for this can be found in the ideal encoding of the input language: In its most concise formulation, a language is a recursive algebraic datatypes. Handling of that input requires a recursive algorithm that deals with all corner cases. Engineers I worked with tend to not see their input language as an ADT, though. They focus on some particular use cases and if there is a specification it is often too large and yet incomplete. But on top of that comes the hesitation to implement a complete algorithm. Nearly every time, some "corner case" is ignored because "nobody uses it" and we end up with an implementation that is factually incorrect and incomplete.
- earleybird 5y ago
  You don't even have to squint to see that BNF description is a sum of products data type. With that, functions on those types write themselves and you're a type-check away from consistency.
hardwaregeek 5y ago
On the flip side, if you start out with "ah shoot I have to write a compiler", that can be paralyzing unless you happen to know how to write a compiler. Sometimes it's best to just write code, do it the wrong way, and then learn the compiler stuff on the fly.
eatonphil 5y ago
It's a hilarious post but I'm scared and curious to ask what this is in response to?
- ____________g 5y ago
  Yes, I agree. It feels like some context is missing here.
  - rachitnigam 5y ago
    [OP] Ah, I've just seen a bunch of people showcase "amazing new tools" that are just bad compilers in disguise. State of the art Database query optimizers are good examples of this.
mikewarot 5y ago
I wrote an inspection tracking system in Turbo Pascal/MS-DOS with some Norand hand-held computers running PL/N back in the late 1980s. As we modified the system to handle new classes of inspections, I ended up having a little configuration file that sat in the folder, that looked just like Pascal. I encrypted that file by XORing it with a random string in a little command line program for the purpose. It also decrypted the file so you could work on it, of course. Those were fun days, except for the lack of GIT, and the resultant stack of floppy disks I kept the source backed up on.
jasoncabot 5y ago
This reminds me of a great post about the typical software evolution going from hard coded values, through configuration and rules engines to a DSL, only to be back where you started. I have seen it happen on multiple projects, it’s hard to spot it at the time as all the changes seem sensible but hindsight is a wonderful thing http://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html http://mikehadlow.blogspot.com/2012/05/configuration-complex...
- wpietri 5y ago
  This line really struck me: “They will have to change at some point, and you don’t want to recompile and redeploy your application just to change the VAT tax rate.” Continuous deployment has changed this for me. I'm still not going to leave tunable constants in the middle of random if statements. But nowadays instead of trying to avoid recompiling and redeploying, I'd rather invest that time into making that very easy.
  - icambron 5y ago
    Yeah, CD is a big game changer here. Managing config separately from code is a big pain (as that piece points out in colorful detail), and as the countervailing painfulness of a deploy drops toward zero, it becomes better and better to just hardcode some consts. "Business rules" just become "code you write". Easier to change, test, and reason about. At my last job we moved to CD and it was fun to see everyone (haltingly, sometimes unwillingly) change their approach to this. There's still a place for configs and even DSLs (multiple deployments, on-prem software, biz logic written by non-programmers, etc) but CD eliminates the huge class of "this is too expensive to deploy changes to" cases.
  - mst 5y ago
    Pulling things out into a configuration file (or environment variable) is valuable when either (a) the value needs to be set per-environment and you have a variable number of environments or (b) the value needs to be changeable by somebody other than the dev team. Also it's worth considering for constants whether what you actually want is a 'resource file' (which could easily just be a file containing all your constants written in your programming language, no need to use a different syntax if that doesn't gain you anything) - learning and internalising (and also learning how to explain to other developers) that a resource file is -not- the same as a configuration file has been a significant quality of life improvement for me.
sohamsankaran 5y ago
From painful experience this is true, especially the parts Rachit wrote to personally call me out.
- rachitnigam 5y ago
  i have never personally called out anyone anywhere ever
agumonkey 5y ago
Who said 'every problem is a parsing problem' ?
user249 5y ago
A lot of guys have a wife and don't know it yet, but she does and is waiting for the ring
Communitivity 5y ago
I feel like this easily could have been addressed to a much younger version of myself on a particular project more than a decade ago (~12 years). My story happened in the early days of SPARQL, when SPARQUl (SPARQL Update Language) was very new, a draft, and definitely not merged into SPARQL yet. Oh, and dinosaurs roamed the earth. That last is from my daughter. My project involved semi-autonomous software agents that each maintained a model of one aspect of a thing being monitored by different types of sensors. Agents were divided into nerves, and brains. The model maintained by a nerve was of a limited aspect, containing explicit, semantic representations of sensor reported values. The models maintained by brain agents were implicit, derived from one or both of two stages of constrained semantic reasoning (closed world model, and some other constraints). Some nerve agents did the first stage of semantic reasoning, more simplistic using description logic programming rules encoded into the model. The results were stored into the model so that they, and parts of the more complex aggregate models derived from them up the chain, could be removed on the next pass. Brain agents did a second pass of semantic reasoning. This pass is where the 'You have implemented a compiler' comes in. That pass used rules encoded in human readable form. We looked at CWM for our rules, but that didn't have enough expressivity and didn't mesh well with some other tools we had. We encoded the rules a format we created, called Sparql++. It had SPARQL, SPARQL Update Language, for loops, while loops, variables, and macros. Our code took the rules and put them into an intermediate form, then translated that intermediate form to EulerSharp rules and ran them, then injected the results into our model. The intermediate form and EulerSharp rules were saved so parts that hadn't changed didn't generate new EulerSharp rules. It was ungainly. It worked well. And it was effectively a compiler.
- bmh100 5y ago
  The nerve and brain concept sounds fascinating, but I don't have enough background in sensors to quite grasp what you mean. Could you please provide a specific example?
  - Communitivity 5y ago
    One example is using it for network monitoring. Imagine an SNMP agent as a sensor, managed by a nerve, which gathers information via SNMP Gets and other techniques to build a semantic model from one or more SNMP Management Information Blocks (MIBs). This then is reasoned over, aggregated up the chain, further reasoned over, and fed into rules that trigger model changes that get propagated to the nerves, who translate those changes into SNMP management actions.
einpoklum 5y ago
I actually find myself needing something which is close to the front-end of a compiler: I get a file in a C-like language (say it's C for the sake of discussion and to make life easy), and I want to figure out the names of the top-level functions defined in this file. I am willing to assume that there are no "Gotcha" macros used, which would redefine keywords or types, or otherwise mess up the syntax. The caveat is that I don't want to include any files - even though this file has some include directives; and not including them would mean some types are not defined etc. What would I do in such a case? Should I take the "not a compiler" approach and start matching regex'es?
- all2 5y ago
  If the syntax for function definitions is relatively fixed, I'd say use regular expressions. Others have mentioned recursive descent parsers, and you'd be implementing the "base case" portion of one of those. Not including "includes" is easy. Just don't go looking for them. --- Edit: the fun part will be trying to implement doc-strings. :D You wind up with a RDP that has a grammar like fn_def := documented | un_documented documented := doc_string def un_documented := def doc_string := <your docstring format regex here> def := <your function defition regex here> Note, for the last two, it will be easier to break those up into pieces. If I were writing a RDP for C# method declarations, I'd have something like def := ACCESS_SCOPE MUTABILITY RETURN_TYPE FN_NAME L_PAREN PARAMS R_PAREN ACCESS_SCOPE := "private" <-- these would grammar "terminals" or the literal strings we are looking for | "public" | <etc> MUTABILITY := static | <others?> RETURN_TYPE := "bool" <-- note that this should actually be much more complex | "int" because you can (almost) arbitrarily specify types | <etc> based on C#'s internal types FN_NAME := <some regular expression for allowed function names in C#> L_PAREN := "(" R_PAREN := ")" Something like that. You'll need to test as you implement, though. Each of the above grammar definitions should correspond 1 to 1 with a function that you implement. If you need/want help with implementation, my email is in my profile. I'd be more than happy to walk you through this (I've done it before :D specifically this use-case, too, where I was trying to auto-document a language that doesn't have any development/documentation tools).
patrec 5y ago
Alternative headline: ivy league PhD candidate attempts some gate-keeping for his future field of employment.
m-hilgendorf 5y ago
At my company (JITX) we've fully committed to compiler architectures in our stack, which makes sense since we're developing an embedded DSL for circuit boards. It's remarkable how many problems are easier when you just accept it's some kind of compiler problem that needs parsing into a tree you can walk with a pass to spit out the required data. For example in audio networks, you can write a buffer allocation and latency compensation solver as a compiler pass over an AST that represents the network topology, collected by walking the network graph objects. It's way easier to write and test than using the same objects as the network itself. I will say one of the downsides (if not tackled early) is incremental and streaming data through the architecture. It's a lot easier to write a batch parser than interactive one - which can hurt if you need partial compilation in the future.
- mst 5y ago
  Something I've found useful when perpetrating custom languages is to, as early as possible, bring up a REPL that handles multiline expressions sensibly. This (a) requires me to build stream/incremental parsing (b) gives me a tangible feature out of doing so that makes my life better (c) means from then on even if the production uses of the language are all batch shaped to begin with I'm regularly dogfooding the non-batch code paths. (because yeah, it -so- does hurt trying to retrofit that later on, so 'finding a way to trick myself into -wanting- to build it in early' was rather a win for me ;)
  - m-hilgendorf 5y ago
    Lucky for us, the language we embed the DSL(1) within has a REPL! (1) http://lbstanza.org/ http://lbstanza.org/
pshirshov 5y ago
What's the problem with building, aaaaah, a COMPILER? I've baked a couple, feel good so far.
evacchi 5y ago
shameless plug: I have done a part 1 and part 2 presentations on this very topic and Your Program as a Transpiler: Improving Application Performance by Applying Compiler Design https://www.youtube.com/watch?v=TWfigR9wGsA https://www.youtube.com/watch?v=TWfigR9wGsA Your Program as a Transpiler: Applying Compiler Design to Everyday Programming https://www.youtube.com/watch?v=BUrY6On1SxM https://www.youtube.com/watch?v=BUrY6On1SxM the second in particular is about reasoning in terms of compiler phases when you have to process something that apparently may not immediately look like a programming language.
gk1256 5y ago
Well, a lot of guys doing algebra without realizing it.
samatman 5y ago
Dear Sir, I have built, not just a small handful of compilers, but a compiler for the parsers for those compilers. Signed, - One who did in fact want to write compilers

355 points by rachitnigam 5y ago

Oof

I'm always surprised by the lack of "easy" parser/compiler toolkit for more or less complete DSL... It looks like there's only - either full blown programming language for IT guys - "natural language" AI toolkits (not really usable yet for just simple automation by users) - graphical language like State Machine or "no code", missing loops and requiring mouse and boxes However, most of the time, user simply need some kind of BASIC (like visual basic used a lot in "shadow IT" by users)... but there's really few libraries providing that kind of functionalites

mrlemke 5y ago
Sounds like a gap TCL could fill. The syntax is simple and consistent, a developer could create packages that a non-developer end user can use as commands, and it could be wrapped with GUI code that runs on Windows, Mac, or Linux.
- orthoxerox 5y ago
  Tcl is good, but non-developers could really benefit from more write-time checks. Something that can tell them "there's no command called moveRight, the nearest match is move-right" before their script is executed.
  - mrlemke 5y ago
    I agree. I would really like the same. Komodo IDE may help with this, but I haven't tried it. I usually end up in vim due to habit.
- blacksqr 5y ago
  Linked article appears to be describing the exact original motivation for writing Tcl: "My students and I had written several interactive tools for IC design, such as Magic and Crystal. Each tool needed to have a command language (in those days people tended to invoke tools by typing commands; graphical user interfaces weren't yet in widespread use). However, our primary interest was in the tools, not their command languages. Thus we didn't invest much effort in the command languages and the languages ended up being weak and quirky. Furthermore, the language for one tool couldn't be carried over to the next, so each tool ended up with a different bad command language. After a while this became tiresome and embarrassing." https://web.stanford.edu/~ouster/cgi-bin/tclHistory.php https://web.stanford.edu/~ouster/cgi-bin/tclHistory.php
jasfi 5y ago
What about ANTLR?
- olivierduval 5y ago
  Actually, I was thinking about having an "already build" BASIC language... not creating a new one each time. As a programmer, you would include the libray, provide the binding to your program... and done ! In a way, it's a bit what is possible with java.scripting but they provide a Javascript engine but no BASIC for end-user
jacquesm 5y ago
There's Forth. Many would argue it's not a full blown programming language, it's more like a toolkit to make DSLs.
- K0balt 5y ago
  It’s great when you don’t need to have a huge codebase.IME forth gets out of control quickly from a maintainability standpoint as you go big, but it is excellent at small dsl type tasks. The maintainability issues with forth might actually be solvable if the right IDE paradigm could be found.
  - jacquesm 5y ago
    I don't think so. Let me try to explain: regular languages are only viable as long as someone is willing to speak them and learn them. Latin being the obvious exception, but that's mostly because of medical science and biology, two very large branches of the scientific tree. A DSL is a language with only a handful of users, each of which is usually in the normal path of employment (obvious exception: Chuck Moore). As soon as someone in a team that relies on a DSL leaves the loss of knowledge is immediate. Unlike any other language that you may have picked up in some other organization the DSL has instantly lost a sizeable fraction of those versed in it. It doesn't take many losses like to kill such a project. Another problem is that even if the original authors hang around, the chances that they will stay fluent in their own DSL diminishes with increasing intervals between work on a particular project. If the code would be written in a 'normal' language then they would stay fluent. So DSLs are not a good match for how we normally think about software, even though there is a 1:1 correspondence between the words in your typical DSL (say a FORTH word) and a function call. Functions tend to be written in terms of other functions, just like FORTH words are written in terms of other words, and yet, the standard library is never far away whereas with FORTH words it could easily be 30 words down before you hit the actual language. I'd love to see something like a DSL based language but with longer term prospects, for a long time I hoped that Smalltalk would be that language but I have mostly given up on that.
    - MaxBarraclough 5y ago
yetihehe 5y ago
> However, most of the time, user simply need some kind of BASIC "most of the time" is THE problem here. Once users approach a barrier (missing language feature) they will complain and push you until you implement that feature. After several such features some developer in basement of your building will say "ok, fine, I'll do it" and now you accidentally have a compiler. Edit: I know because I've recently made a "almost universal data watcher" which can react to combination of measured values changed in different devices and execute actions. Simple solution, but I fear the day when someone needs to execute other recipes in response to data and creates loops.
- Dylan16807 5y ago
  That's a problem, but before that what's the good answer to "how can I take advantage of existing libraries to make a quality but simple BASIC-like DSL?" And in particular, where's a well-thought-out BASIC-like template I can customize?
bena 5y ago
I think there's a tipping point. You start with a configuration or markup language. Then you add and add and add and next thing you know, you've built a very simple scripting language. It's kind of like those people who cobble together a Turing complete implementation of HTML+CSS or Magic: the Gathering game. You can kind of stumble into completeness.
throwfaangus 5y ago
Just use flex / bison?
- Koshkin 5y ago
  Looks like these are grossly underappreciated these days. They are wonderful, easy to use tools.
  - neurotrace 5y ago
    I think "easy to use" might be an overstatement. I spent a fair bit of time trying to get in to the flex/bison workflow and it never seemed to click. I've found it much easier to use PEG parsers, parser combinators, or a hand-written recursive descent parser. Based on their ubiquity and the high reviews, I'm sure they're great tools once the initial hump is passed but that initial hump is quite large.
  - duped 5y ago
    I think it's because they're less easy to use tools than writing your own lever/parser, which takes about a day and is fully introspective and debuggable. And if you need decent error reporting at the input it's much easier to thread that through. Plus almost everything language has some kind of PEG or parser combinator library these days.
- therealcamino 5y ago
  I've always found ANTLR to be more intuitive and maintainable than the flex/bison combination. And the learning curve is gentler.
  - base698 5y ago
    And the tooling is better. I always found it hard to find working examples with bison.
brandmeyer 5y ago
The intermediate solution is a programming language and/or runtime specifically designed as an extension system. I've had positive experiences embedding Lua. GNU Guile was specifically designed for this too, if you think your users can deal with the parens.
rachitnigam 5y ago
Racket is often heralded as the "programming language programming language" and comes with tons of features to build full blown languages: https://racket-lang.org/ https://racket-lang.org/ The most complete language/ecosystem that really showcases Racket's capabilities is Rosette IMO: https://github.com/emina/rosette/ https://github.com/emina/rosette/
- noblethrasher 5y ago
  Also worth noting that language that runs this very website was implemented in Racket.
  - mepian 5y ago
    I think it was implemented in PLT Scheme before it stopped being a Scheme implementation and became Racket.
  - eitland 5y ago
    I thought that was Arc?
    - electroly 5y ago
      It is. Arc itself is written in Racket.
      - eitland 5y ago
        Ok, so pretty good proof that racket is a working programming language programming language then :-)
  - danuker 5y ago
    Hacker News runs on a language that runs on Racket? Which language is that?
    - krapp 5y ago
YeGoblynQueenne 5y ago
There's Prolog and its Definite Clause Grammars (DCG) formalism. Here's a DCG for a tiny subset of natural English (copied from wikipedia [1]): sentence --> noun_phrase, verb_phrase. noun_phrase --> det, noun. verb_phrase --> verb, noun_phrase. det --> [the]. det --> [a]. noun --> [cat]. noun --> [bat]. verb --> [eats]. The syntax is just like BNF. "-->" is "::=", terms in []'s are terminals and the rest are nonterminals. Nonterminals can have arguments and there's special syntax to include entire Prolog programs in the body of rules. I discovered DCGs at the end of my CS degree when I was working on my graduation project, a Magic: the Gathering expert system written in Prolog. I needed a way to script cards for my rules engine, to translate expressions in the M:tG ability text language to function calls in the engine. I really wanted to write a parser using the standard parser generator tools, but I thought it would be too much work for a graduate project. Then I found out about DCGs. With DCGs I could write grammar rules like this: tap(X,Y) --> (['Tap'];[tap]), target(X,Y), (condition ; []), {permanent_Type(Y)}. tap(X,Y) --> (['Tap'];[tap]), all(X,Y), (condition ; []). And that would not only parse, but also _generate_ ability text like "Tap target Kavu" or "Tap all Goblins target player controls" etc. So as a side effect of adding a bunch of abilities to my engine, I also had a generator for new M:tG cards. [Edit: to be clear, the grammar rules above are already, by themselves, a parser for abilities that start with "tap". The Prolog interpreter executes those grammar rules as a program and instantiates their variables in the process. To integrate them with the engine all I had to do was write engine functions that called the nonterminals in the grammar and read the values of variables like X and Y in tap(X,Y).] Admittedly, my grammar was limited because I didn't have the time to hand-code a grammar for the entirety of M:tG ability text, as it was at the time (I graduated 2011). In any case my rules engine was also limited (I had no planeswalkers and static effects were only partially implemented, if I remember correctly; because the rules for static effects are a bit arcane). But I managed to cover a good subset of the rules and cards, so it was possible to play a basic game with creatures with activated and triggered abilities and with non-creature spells [2]. The important detail is that DCGs are syntactic sugar for ordinary Prolog, so I could code my parser in the same language as my engine and my player. My biggest problem was really to draw an imaginary line between the parser and the other two components, because the easiest thing would be to allow them to know about each other and be infinitely entangled in a big ball of mess. Anyway, yeah, there's something out there that is perfectly suited for creating DSLs, if you have the patience [3]. The tradeoff is that you have to learn Prolog. I suspect this is a cost too high for many who just want an easy-to-use tool. Btw, you can't easily run my degree project unfortunately. Like an idiot I wrote it for a proprietary Prolog system and although I tried to port it to the free and open-source SWI-Prolog, it's been a nightmare of incompatibilities. The ported project is here but it breaks unexpectedly all the time: https://github.com/stassa/Gleemin https://github.com/stassa/Gleemin ________ [1] https://en.wikipedia.org/wiki/Definite_clause_grammar#Example https://en.wikipedia.org/wiki/Definite_clause_grammar#Exampl... [2] I also had an alpha-beta minimax AI player. Not that good though. I wanted to program a player with background knowledge of M:tG but I didn't have the time. [3] Or you can learn them from examples... See Experiment 3 on page 19 of the pdf, here: https://link.springer.com/article/10.1007/s10994-020-05945-w https://link.springer.com/article/10.1007/s10994-020-05945-w
jerf 5y ago
The easy options: As others have said, embed something existing. I think this is especially true for programming languages. Even if Lua isn't quite what you want, it's probably going to deliver a lot more value than anything you could bash together on your own. And there are a few other options as well. There are several of these "complete toolkits for DSLs", it's just they do take a bit of work to correctly insert into your code base and then expose enough of what you're doing to make it useful. There isn't and as far as I can see can't be something you just install and it magically exposes everything and it's just awesome and takes no work, because even if you're in something like Python where you can just open an interpreter shell in the current context, thereby technically giving the user the power to do anything they want, you still need to create a safe interface for them, document it, test it, etc. (You may not do a lot of technical work to keep them out of where you haven't tested but you will need to provide them a supported area to get anywhere.) For building your own: For small tasks, a parser is just a specialized sort of deserializer. Yea verily these 30 years ago, there wasn't a lot of generic deserializers available, and the ones that existed like asn.1 were completely unsuitable for human use, so "classic" compiler literature spends a lot of time on building your own parser. Now, if you don't have a specialized need, you should use a generic deserializer. I have quite a few JSON-based syntaxes lying around. (They feed "interpreters", but same principle.) Lisp S-expressions, if suitable for your environment, can be adapted. XML could be useful in some specialized circumstances where your input looks more like a document than conventional code. Etc. You should only build a custom parser if it's going to give you more bang for your buck across the whole usage cycle than a generic parser can give you. The reason why I have several things that input JSON is precisely that the amount of usage across all time was just going to be too low to justify doing anything specialized, because it was always a strictly internal tool. (Hint: Just because your compiler accepts JSON as input doesn't mean that that's what the humans have to directly output or edit. My biggest interpreter took in JSON but I provided abundant function helpers that abstracted that away. It was a thin enough abstraction that it could be penetrated if need be, but it avoided most of the verbosity issues as far as the humans were concerned, and gave them the full power of the programming language to generate loops and such. If you squint, I basically borrowed an existing general-purpose programming language that everyone involved already knew to use as a macro language for my language, basically for free. This means the "core" language that I was implementing didn't have to do anything that the already-existing general-purpose language could do in macros, meaning the layer the humans used offered a lot of power for not much implementation cost on my side.) For AST transforms, this is always where your secret sauce is and there's little help to give, beyond a suggestion that this is arguably the canonical case for a functional programming style, even if you are in an otherwise OO language. For the final output, I would suggest that in modern times, the easy answer is, don't bother. You can probably interpret fast enough to solve your problems. If not, there are some options like LLVM that will help vs. what you used to have to do. But I would definitely consider just interpreting the final AST directly. If you intend to do that your AST transforms can be built with interpretation in mind. The real value of a compiler and what distinguishes it from any ol' normal code that ingests something and emits something is the AST in the middle, and the complexity of that AST (probably through significant recursion) and the transforms you run on it, and the corresponding complex behaviors the input can invoke through the recursive input and complex AST transforms. Anything you can do to brush away the incidental details of a compiler by using existing serializations and if at all possible simply skipping the output step of the compiler will hugely increase the bang-for-the-buck you get. You might say, rather than "writing a compiler" consider "using compiler techniques" but not writing a compiler. But not in a way the article was talking about. For the specific case mentioned there, go get an existing parser for your language, get the full AST tree. You can write your transform in terms of what you do understand and just pass through anything you don't. It's slightly harder, but will be fairly effective. It is possible even today, of course, that you will be backed against a wall and be forced to go through the full classical compiler cycle. A programming language, for instance, just can't afford to use any existing parser, it's too much a vital part of the experience to try to use an existing one. Even a new Lisp will probably want its own spin on S-expressions enough that it can't just pick up an existing one unchanged. And maybe you've got something that you expect will be big enough that it just requires a new syntax, and you can afford all the support that goes with that. But the vast majority of the time, with the right toolset, you can indeed obtain most of the value of compiler-like techniques without the cost of the full classical compiler technology stack, because even if it was hypothetically possible to build a "real compiler" that had its own custom syntax rather than some slightly dodgy JSON and emitted LLVM code that resulted in an executable that runs all its tasks in .0003 seconds, the engineering effort for that is so much greater than the JSON input and interpreter that runs in .003 seconds that in most cases it's not worth it.
derefr 5y ago
Most people who want this end up just selecting a regular programming language with a powerful-enough macro system + flexible-enough compile-time parser; and then using said language to define a "more or less complete DSL", by redefining all the primitive syntax features of the language to instead be primitive syntax features of the DSL. For example, the "numerical definition" DSL within Elixir's Nx library (https://github.com/elixir-nx/nx/tree/main/nx#numerical-definitions https://github.com/elixir-nx/nx/tree/main/nx#numerical-defin...). Users using this "full" DSL are still using the host language — they're invoking its compiler/runtime and so forth — but they don't necessarily need to care about that. They can still technically access the power of a full-blown programming language, "tucked in" around the edges of the DSL — but you as the DSL's creator, don't have to document that power as being available. As long as users of the DSL can do everything they need to do without any need for "breaking out" of the DSL's little world, they don't need to be aware that they could e.g. make fully-namespaced calls to regular stdlib functions of your runtime.
lowbloodsugar 5y ago
JetBrains has a MPS (Meta Programming System) [1] that I believe is for knocking up DSLs. At a lower level, I've used ANTLR [2] a bunch of times, and its StringTemplate even more often. [1] https://www.jetbrains.com/mps/ https://www.jetbrains.com/mps/ [2] https://www.antlr.org https://www.antlr.org

pddpro 5y ago

Oh this brings a lot of memories. I remember writing a DSL for visualization of logs that we collected when we were working in a hyper local geographical search engine. I was fresh out of college and this was when Google maps had yet to penetrate my country. I started out with pylex, yacc and then what was supposed to be a quite simple DSL with a few keywords and a few nodes, very quickly exploded to a very complicated endeavor. Alas, I'll never be able to prove that it was Turing Complete. But that doesn't stop me from believing it.

thriftwy 5y ago

Related: https://steve-yegge.blogspot.com/2007/06/rich-programmer-food.html https://steve-yegge.blogspot.com/2007/06/rich-programmer-foo...

cormacrelf 5y ago
Overall decent take on why you should go for a compilers course, but it has a very weird section advocating for non-deterministic type checking. You want probabilistic AI methods to guess that this string is more of an integer? Buddy, knowing what kind of data we have for certain is the only reason we built type checkers in the first place. Take your 90% chance int but 10% chance string and get out of here. Maybe he means “error messages are bad, we should guess what people mean and make better suggestions on type errors” but the case was not argued well. It also stuck with an article-long joke of accusing compiler people of being boring at parties. This is 2007 after all, an unenlightened time.

sunir 5y ago

I think you should almost always build a compiler when presented with a parse then execute problem. It is so much faster to create a small machine that is fed data to specific the program, and a test harness that also drives that machine, then it is to hand code every little case repetitively. Or maybe I like building little compilers.

tylerscott 5y ago
I completely agree with the sentiment but also feel like it is just my bias because I enjoy writing compilers.
deleted 5y ago
[deleted]
ianbicking 5y ago
When you describe it that way it reminds me of SAX [1] – I always hated SAX, but eventually realized it was kind of a tokenizer that left it up to the developer to figure out how to turn that into a compiler, though in this case compiling XML input into some internal data structure or action. [1] https://en.wikipedia.org/wiki/Simple_API_for_XML https://en.wikipedia.org/wiki/Simple_API_for_XML
- sunir 5y ago
  Weirdly, but sometimes ideas churn around HN, I just replied to another thread about this. See my reply on this thread https://news.ycombinator.com/item?id=29917060 https://news.ycombinator.com/item?id=29917060 At BitFlash, one of the things we had to build was a SAX parser for the SVG DOM. I used the DSL of the W3C spec to compile the SAX parser. One of the more strange things I did was in some contexts (think old school Blackberry) we had a server to pre-parse the SVG so we "knew" it was clean (I'm still sceptical 20 years later this is ever a good strategy, but take it as a given). Because we knew the SVG was clean, there was a faster way to parse the XML than reading the tokens. I used my magic Perl script transformer to compute the lowest entropy decision tree to identify a token with the fewest comparisons, which was surprisingly way more efficient than a trie.
- 5y ago

3pt14159 5y ago

This is a great article, and if I were writing path-critical code for the situation the author describes I'd use an AST too, but I still munge me some text and most of the time it's good enough. The problem with using AST libraries is that they're hard to understand unless you're used to them. So if you have to choose between a couple hundred lines of python or ruby and a giant AST I think it's smarter to go with the ruby. That way when something like a new emoji comes out, or what have you, the junior dev can handle the minor bug fix. Your AST is still going to choke on it. No way around it, the world has changed. That said, there is an art to writing text processors and keeping things sane. The most important bit I've learned is to avoid transformations during extractions even at the cost of performance. At least until a fully exhaustive test suite is up. It makes it easier to test and it's a clearer separation of responsibilities. I've written a lot of these parsers. One for a custom format for the Nasdaq on a project I was the lead on! I shudder to think of how many bits that thing is processing since it pipes data to and from thousands of data brokers and hedge funds.

bitwize 5y ago

"Select the pistol, and then, select your compiler." https://www.penny-arcade.com/comic/2008/05/26/the-unhorse https://www.penny-arcade.com/comic/2008/05/26/the-unhorse

TillE 5y ago

YAGNI is a good principle here. Whenever I've found myself thinking about reaching for a parser library, I was over-complicating or over-generalizing the problem. Write the code you need to solve the problem you actually have.

cmyr 5y ago
I agree completely. Anytime I’m looking at a parser library I just shake my head and close that browser tab. I’m invariably going to want a hand-rolled recursive descent parser two weeks later, so let’s just get on it.
- orthoxerox 5y ago
  So much this. I've used a parser generator successfully exactly once, and even then I simply didn't care about proper error reporting. I wouldn't mind something reusable that generated red-green trees for me a la Roslyn. I can bang out a recursive descent parser in a day, but not one that can be run after every keystroke.
- naasking 5y ago
  There's a middle ground with parser combinators.
quotemstr 5y ago
The article makes a good argument against YAGNI, at least YAGNI applied in its naive form. The point is that if you're going to make an embedded language, just do it the right way from the start instead of trying to cobble it together with YAGNI-inspired half-assed implementations that break in weird cases.
TheCoelacanth 5y ago
YAGNI works well as long as you are willing to re-write your code in the rare cases when you do "need it". It's fine to start without a real parser, and you often will end up not ever needing a real one, but if the complexity of your ad hoc solution starts getting anywhere close to the complexity of a real parser, you are better off re-writing it as a real parser because parsing is an extremely well-studied area of CS while your ad-hoc "not parsing" is not.

bugmen0t 5y ago

Ah, again? Happens to me all the time.

DarylZero 5y ago

Seems like the idea of exposing a programming language to the end-user should have better support so that it's more routine to create a restricted subset of something existing for users to put arbitrary code in. Something with both standardized textual interface and standardized structured editor interface would be great. I guess this was always the dream for LISP but it never got there.

ModernMech 5y ago

This happens to so many people, it’s a tale as old as programming languages. If you ever find yourself saying “wouldn’t it be neat…” or “if I could just only…”, stop yourself immediately — you may be about to accidentally spend the next 3-5 years writing a compiler. So many compilers started out looking to implement small feature X, and then ended up spending ungodly amounts of time writing an entire compiler instead. The scariest thing about compilers is by writing one, it induces a kind of amnesia where you can’t remember why you started writing it in the first place! Soon enough the point of the compiler is to write the compiler, and feature X goes unimplemented. Eventually it will be possible once the compiler is done. But the compiler is never done… never… done…

hobofan 5y ago

I feel like you can apply the same sentiment to many of the "big scary things" in programming. Things that you don't want to build (as far as "common engineering wisdom" is to be believed): - a compiler - a programming language (not sure that there is a difference to compiler as stated in the article) - a database (query engine) - a CMS - a ERP But sometimes you actually _do_ want to build that (even if every alarm bell in your engineering lizard brain goes off), and then it's probably better to commit and learn the ins and outs of that specific problem domain. I think especially the recommendation to commit to it is something that's missing from the article. It's "easy" to point a people and saying "look, you did the thing you said you wouldn't do", but far harder to suggest a course of action. I personally hit the barrier of "things you shouldn't build" in the past, and I've always been happiest (and professional outcomes have been the best) when I decided to break through the barrier and prepare for what lies on the other side, instead of trying to dance around it for ages.

jimmaswell 5y ago
Implementing a small domain-specific language is fairly easy. Why avoid it?
- Karunamon 5y ago
  Because the complexity of the problem tends to scale to your willingness to address said complexity. In other words, it likely won't stay small for long if it gets any users, and now you're the maintainer for a tool used by others.
  - mattgreenrocks 5y ago
    This applies to all software in general. You'll get eaten alive by your success if you don't manage expectations and tightly scope things. At least with a DSL, you can often define a small core language and de-sugar down to that.
  - cbsmith 5y ago
    That principle is not specific to a DSL, so that hardly seems like a reason to avoid a DSL.

vinceguidry 5y ago

You need a compiler when you need language, that is, when you need to interface directly between brain and computer. Everything else can be less messily-specified. A good middle ground is the DSL, domain-specific language. You use the syntax and semantics of the language being used to code in, and change the nouns and verbs. All the nouns and verbs are given meaning as values or functions. Execute in a sandbox so that your DSL can't find the rest of the language. Ruby makes this all trivial. Of course, if you're trapped in a static language with no way to execute significant logic at runtime without a ton of heavy lifting, then yeah, you'll end up maintaining an interpreter / compiler if you're not careful. Really do yourself a favor and see if you can't introduce Ruby, you can adapt it to your needs. You can inherit from BasicObject, BasicObjects can't see anything you don't want them to see.

meinte37 5y ago

The article is “addressed to those who did not want to build a compiler”: what's the point of this article then? It seems to implore to not build compilers, but those who did not want to in the first place, probably actually did not build compilers. The article does point out numerous challenges with building compilers, and, by extension, with software language engineering, which concerns itself not just with building compilers but with the design of languages, and implementation of non-compiler tools around it. It merely points out the existence, and likely occurrence, of those challenges, and it can be surmised that the author is frustrated by frequent experience in his own daily life. A bespoke software language -or DSL, if you will- is not always the best solution. It really depends on what you carve out as the domain, how fast that domain changes, what costs and risks are associated with those changes, and with which people you have to implement those changes. No amount of tooling is going to help you out if the domain you chose to recognize as such isn't changing fast enough, isn't costly or risky enough to change (quickly), or the domain stakeholders are anyway not helping out. But in case you do want to build a DSL, have a look at a language workbench like JetBrains' MPS, or this book I happen to be writing: https://www.manning.com/books/domain-specific-languages-made-easy https://www.manning.com/books/domain-specific-languages-made...

a_shovel 5y ago
It's not a warning not to build a compiler, it's more of a story about how feature bloat might cause one to inadvertantly build a compiler without realizing that's what they were doing. The last line reveals it by telling us that each of those "features" was actually a standard compiler component: > Done at last, you say to yourself, without having to build a compiler. > A parser, an intermediate representation, transformation passes, and a code generator. Dear Sir, you have built a compiler.
- rachitnigam 5y ago
  [OP] Yup, that was the intent. I've seen a lot of academic and industrial hand wringing about not wanting to build a compiler because it's an overkill for the solution only to have the people come back 6 months later having built a compiler. I'm not saying everyone should build a compiler for everything–but when you should build a compiler, you really should bite the bullet and build a compiler.

lupire 5y ago

https://en.m.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.m.wikipedia.org/wiki/Greenspun%27s_tenth_rule "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." (Same applies to popular languages after the original quote.)

shadowgovt 5y ago
What often goes unsaid is the corollary: "Any Common Lisp implementation people use contains an ad-hoc, informally-specified, bug-ridden optimizing compiler written in C or Fortran."
- jhgb 5y ago
  That makes no sense. Which Common Lisp implementations contain a compiler written in C or Fortran?
  - shadowgovt 5y ago
    I should have been clearer: written partially in C or Fortran. SBCL, for example, has C bindings in the /runtime subdirectory.
    - jhgb 5y ago
      > SBCL, for example, has C bindings in the /runtime subdirectory. So nothing to do with SBCL's compiler (which is somewhat confusingly named 'Python' and written entirely in Lisp), then.
      - shadowgovt 5y ago
        Well, handwave handwave. Nothing to do with the compiler except that I'd assume if you rip out the /runtime directory you'll get a compiler that can't output working code. The point is that these domains have a tendency to eventually overlap each other.

0xbadcafebee 5y ago

Ways you can avoid building a compiler: 1. Lower the scope of the solution. If your solution is designed to address 100% of the users' problems, it's probably too grand of a solution. Start with 80% of the problems. Identify all the problems, rank them by priority and difficulty, and leave the most difficult and least priority problems out. 2. Lower your expectations. Imagine your great idea. Now imagine how you will implement it. Now imagine it is 100x more difficult than you imagine. Woof, that's hard! Strip down the implementation to the essentials needed to solve the immediate problems. 3. Give the user the minimum possible functionality to address their needs. 4. Go find an existing solution that provides these requirements. If you did it right, you will probably find a solution that you really don't like but already exists and solves the problems you need to solve right now.

mcguire 5y ago
And in six months, the users will be hacking around the parts you left missing, probably by emailing Excel spreadsheets around. If you did it right, you will be off to your next place of employment before the wheels completely come off.

choeger 5y ago

The language pattern occurs quite frequently in many domains. Unfortunately, I have so far only seen half-assed interpreters. At best, people did a shallow embedding (interpreter) of their DSL. No one ever did a true compiler. I think the reason for this can be found in the ideal encoding of the input language: In its most concise formulation, a language is a recursive algebraic datatypes. Handling of that input requires a recursive algorithm that deals with all corner cases. Engineers I worked with tend to not see their input language as an ADT, though. They focus on some particular use cases and if there is a specification it is often too large and yet incomplete. But on top of that comes the hesitation to implement a complete algorithm. Nearly every time, some "corner case" is ignored because "nobody uses it" and we end up with an implementation that is factually incorrect and incomplete.

earleybird 5y ago
You don't even have to squint to see that BNF description is a sum of products data type. With that, functions on those types write themselves and you're a type-check away from consistency.

hardwaregeek 5y ago

On the flip side, if you start out with "ah shoot I have to write a compiler", that can be paralyzing unless you happen to know how to write a compiler. Sometimes it's best to just write code, do it the wrong way, and then learn the compiler stuff on the fly.

eatonphil 5y ago

It's a hilarious post but I'm scared and curious to ask what this is in response to?

____________g 5y ago
Yes, I agree. It feels like some context is missing here.
- rachitnigam 5y ago
  [OP] Ah, I've just seen a bunch of people showcase "amazing new tools" that are just bad compilers in disguise. State of the art Database query optimizers are good examples of this.

mikewarot 5y ago

I wrote an inspection tracking system in Turbo Pascal/MS-DOS with some Norand hand-held computers running PL/N back in the late 1980s. As we modified the system to handle new classes of inspections, I ended up having a little configuration file that sat in the folder, that looked just like Pascal. I encrypted that file by XORing it with a random string in a little command line program for the purpose. It also decrypted the file so you could work on it, of course. Those were fun days, except for the lack of GIT, and the resultant stack of floppy disks I kept the source backed up on.

jasoncabot 5y ago

This reminds me of a great post about the typical software evolution going from hard coded values, through configuration and rules engines to a DSL, only to be back where you started. I have seen it happen on multiple projects, it’s hard to spot it at the time as all the changes seem sensible but hindsight is a wonderful thing http://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html http://mikehadlow.blogspot.com/2012/05/configuration-complex...

wpietri 5y ago
This line really struck me: “They will have to change at some point, and you don’t want to recompile and redeploy your application just to change the VAT tax rate.” Continuous deployment has changed this for me. I'm still not going to leave tunable constants in the middle of random if statements. But nowadays instead of trying to avoid recompiling and redeploying, I'd rather invest that time into making that very easy.
- icambron 5y ago
  Yeah, CD is a big game changer here. Managing config separately from code is a big pain (as that piece points out in colorful detail), and as the countervailing painfulness of a deploy drops toward zero, it becomes better and better to just hardcode some consts. "Business rules" just become "code you write". Easier to change, test, and reason about. At my last job we moved to CD and it was fun to see everyone (haltingly, sometimes unwillingly) change their approach to this. There's still a place for configs and even DSLs (multiple deployments, on-prem software, biz logic written by non-programmers, etc) but CD eliminates the huge class of "this is too expensive to deploy changes to" cases.
- mst 5y ago
  Pulling things out into a configuration file (or environment variable) is valuable when either (a) the value needs to be set per-environment and you have a variable number of environments or (b) the value needs to be changeable by somebody other than the dev team. Also it's worth considering for constants whether what you actually want is a 'resource file' (which could easily just be a file containing all your constants written in your programming language, no need to use a different syntax if that doesn't gain you anything) - learning and internalising (and also learning how to explain to other developers) that a resource file is -not- the same as a configuration file has been a significant quality of life improvement for me.

sohamsankaran 5y ago

From painful experience this is true, especially the parts Rachit wrote to personally call me out.

rachitnigam 5y ago
i have never personally called out anyone anywhere ever

agumonkey 5y ago

Who said 'every problem is a parsing problem' ?

user249 5y ago

A lot of guys have a wife and don't know it yet, but she does and is waiting for the ring

Communitivity 5y ago

I feel like this easily could have been addressed to a much younger version of myself on a particular project more than a decade ago (~12 years). My story happened in the early days of SPARQL, when SPARQUl (SPARQL Update Language) was very new, a draft, and definitely not merged into SPARQL yet. Oh, and dinosaurs roamed the earth. That last is from my daughter. My project involved semi-autonomous software agents that each maintained a model of one aspect of a thing being monitored by different types of sensors. Agents were divided into nerves, and brains. The model maintained by a nerve was of a limited aspect, containing explicit, semantic representations of sensor reported values. The models maintained by brain agents were implicit, derived from one or both of two stages of constrained semantic reasoning (closed world model, and some other constraints). Some nerve agents did the first stage of semantic reasoning, more simplistic using description logic programming rules encoded into the model. The results were stored into the model so that they, and parts of the more complex aggregate models derived from them up the chain, could be removed on the next pass. Brain agents did a second pass of semantic reasoning. This pass is where the 'You have implemented a compiler' comes in. That pass used rules encoded in human readable form. We looked at CWM for our rules, but that didn't have enough expressivity and didn't mesh well with some other tools we had. We encoded the rules a format we created, called Sparql++. It had SPARQL, SPARQL Update Language, for loops, while loops, variables, and macros. Our code took the rules and put them into an intermediate form, then translated that intermediate form to EulerSharp rules and ran them, then injected the results into our model. The intermediate form and EulerSharp rules were saved so parts that hadn't changed didn't generate new EulerSharp rules. It was ungainly. It worked well. And it was effectively a compiler.

bmh100 5y ago
The nerve and brain concept sounds fascinating, but I don't have enough background in sensors to quite grasp what you mean. Could you please provide a specific example?
- Communitivity 5y ago
  One example is using it for network monitoring. Imagine an SNMP agent as a sensor, managed by a nerve, which gathers information via SNMP Gets and other techniques to build a semantic model from one or more SNMP Management Information Blocks (MIBs). This then is reasoned over, aggregated up the chain, further reasoned over, and fed into rules that trigger model changes that get propagated to the nerves, who translate those changes into SNMP management actions.

einpoklum 5y ago

I actually find myself needing something which is close to the front-end of a compiler: I get a file in a C-like language (say it's C for the sake of discussion and to make life easy), and I want to figure out the names of the top-level functions defined in this file. I am willing to assume that there are no "Gotcha" macros used, which would redefine keywords or types, or otherwise mess up the syntax. The caveat is that I don't want to include any files - even though this file has some include directives; and not including them would mean some types are not defined etc. What would I do in such a case? Should I take the "not a compiler" approach and start matching regex'es?

all2 5y ago
If the syntax for function definitions is relatively fixed, I'd say use regular expressions. Others have mentioned recursive descent parsers, and you'd be implementing the "base case" portion of one of those. Not including "includes" is easy. Just don't go looking for them. --- Edit: the fun part will be trying to implement doc-strings. :D You wind up with a RDP that has a grammar like fn_def := documented | un_documented documented := doc_string def un_documented := def doc_string := <your docstring format regex here> def := <your function defition regex here> Note, for the last two, it will be easier to break those up into pieces. If I were writing a RDP for C# method declarations, I'd have something like def := ACCESS_SCOPE MUTABILITY RETURN_TYPE FN_NAME L_PAREN PARAMS R_PAREN ACCESS_SCOPE := "private" <-- these would grammar "terminals" or the literal strings we are looking for | "public" | <etc> MUTABILITY := static | <others?> RETURN_TYPE := "bool" <-- note that this should actually be much more complex | "int" because you can (almost) arbitrarily specify types | <etc> based on C#'s internal types FN_NAME := <some regular expression for allowed function names in C#> L_PAREN := "(" R_PAREN := ")" Something like that. You'll need to test as you implement, though. Each of the above grammar definitions should correspond 1 to 1 with a function that you implement. If you need/want help with implementation, my email is in my profile. I'd be more than happy to walk you through this (I've done it before :D specifically this use-case, too, where I was trying to auto-document a language that doesn't have any development/documentation tools).

patrec 5y ago

Alternative headline: ivy league PhD candidate attempts some gate-keeping for his future field of employment.

m-hilgendorf 5y ago

At my company (JITX) we've fully committed to compiler architectures in our stack, which makes sense since we're developing an embedded DSL for circuit boards. It's remarkable how many problems are easier when you just accept it's some kind of compiler problem that needs parsing into a tree you can walk with a pass to spit out the required data. For example in audio networks, you can write a buffer allocation and latency compensation solver as a compiler pass over an AST that represents the network topology, collected by walking the network graph objects. It's way easier to write and test than using the same objects as the network itself. I will say one of the downsides (if not tackled early) is incremental and streaming data through the architecture. It's a lot easier to write a batch parser than interactive one - which can hurt if you need partial compilation in the future.

mst 5y ago
Something I've found useful when perpetrating custom languages is to, as early as possible, bring up a REPL that handles multiline expressions sensibly. This (a) requires me to build stream/incremental parsing (b) gives me a tangible feature out of doing so that makes my life better (c) means from then on even if the production uses of the language are all batch shaped to begin with I'm regularly dogfooding the non-batch code paths. (because yeah, it -so- does hurt trying to retrofit that later on, so 'finding a way to trick myself into -wanting- to build it in early' was rather a win for me ;)
- m-hilgendorf 5y ago
  Lucky for us, the language we embed the DSL(1) within has a REPL! (1) http://lbstanza.org/ http://lbstanza.org/

pshirshov 5y ago

What's the problem with building, aaaaah, a COMPILER? I've baked a couple, feel good so far.

evacchi 5y ago

shameless plug: I have done a part 1 and part 2 presentations on this very topic and Your Program as a Transpiler: Improving Application Performance by Applying Compiler Design https://www.youtube.com/watch?v=TWfigR9wGsA https://www.youtube.com/watch?v=TWfigR9wGsA Your Program as a Transpiler: Applying Compiler Design to Everyday Programming https://www.youtube.com/watch?v=BUrY6On1SxM https://www.youtube.com/watch?v=BUrY6On1SxM the second in particular is about reasoning in terms of compiler phases when you have to process something that apparently may not immediately look like a programming language.

gk1256 5y ago

Well, a lot of guys doing algebra without realizing it.

samatman 5y ago

Dear Sir, I have built, not just a small handful of compilers, but a compiler for the parsers for those compilers. Signed, - One who did in fact want to write compilers