15 ms·
Dear sir, you have built a compiler (2022)
- pvg 2y agoThread at the time https://news.ycombinator.com/item?id=29891428 https://news.ycombinator.com/item?id=29891428
- iamthepieman 2y agoI get the solution for this and I know what all the terms mean. But I don't understand the problem. Whether it's facetious or hyperbole or whatever, I just don't get who or what circumstances this is addressing. This is written like a Jeopardy answer. I just don't know what the question is. Can anyone enlighten me?
- throwaheyy 2y agoIt’s a description of https://en.m.wikipedia.org/wiki/Inner-platform_effect https://en.m.wikipedia.org/wiki/Inner-platform_effect.
- iamthepieman 2y agoThat makes total sense. I have been tempted to do this in the past. Fortunately time and resources constraints have kept it to costly sane and maintainable, performant configurations until I learned that I would never create the system I wanted and that it was probably better that I didn't anyways. I guess I've been lucky and didn't even know it.
- anyonecancode 2y agoIs it just me, or would "single page applications" fit easily in the "examples" section there? Sounds kind of trollish when I write it out, but honestly, it fits pretty well, right? We threw away all the things the browser gives you for free and re-implement the back button, history, etc in JavaScript. (and it's somewhat fractal, as within the frameworks we use to do this you'll frequently see people re-implementing things the framework already does).
- ameliaquining 2y agoIt's true that some of the Web platform's downsides, rooted in its split identity between being a document library and being an operating system, are kind of similar to this antipattern, if you squint a bit, although they tend not to be as bad because the outer platform is much more robustly engineered than the average enterprise app. The key difference is that, in the Web platform's case, there's not actually a better alternative on offer. Even with these awkwardnesses, it's still a better app delivery platform than desktop or mobile OSes, because it's dramatically less fragmented, has a more convenient "installation" story (https://xkcd.com/1367/ https://xkcd.com/1367/), and has a better security model (at least compared to desktop OSes). So people need to write rich web apps with arbitrary behavior in it, which requires it to be arbitrarily customizable. Contrast an enterprise app, where the lesson of the "inner-platform effect" idea is that code changes to the outer platform aren't as costly as you think, compared to unmaintainable configuration that interacts in complex ways with the platform primitives. So it's best to allow only customization simple enough to not pose maintainability challenges, and eat the cost of an outer-platform code change whenever you need anything more complicated. But Web developers don't have the option of getting browsers to add new code every time they want to add a complex new feature to their app, so browsers need to support a rich enough set of primitives that those features are already possible. The other way to resolve the tension would be to get rid of the document-library features and instead double down on being an operating system, perhaps based on WebAssembly and <canvas> instead of HTML+CSS+JavaScript, like Flutter for Web uses. But of course people are using the document library, and in some cases it's the easiest way to do something, even at the cost of a little bit of redundancy at intermediate levels of customization. What SPA critics typically want, of course, is for most sites to be satisfied with less feature-richness so as to fit more easily into the document-library model. But the platform has to support everyone's use cases, not just those of people who like HN's minimalist style. (I can't find it now, but there was a great comment on HN awhile ago that said something like: "A lot of HN users basically wish the internet was like how it was in the 90s, except with broadband. But in this respect, we're unusual; most users like features and slick UIs.")
- theendisney 2y ago> Web developers don't have the option of getting browsers to add new code every time they want to add a complex new feature to their app We lack a mechanism for picking sane new features. Browsers add new stuff all the time. Most of it is horrible. [Say] Adding a js assembly has to be the most stuborn way of tollerating new langages. you may do it, as long as the new lang is js!? You can have butter as long as it is yogurt. I dont like pyton. I wouldn't be upset by <script type="pyton"> add some of the dom tools to it and people will have a ton of fun. Might even be useful.
- m463 2y agohttps://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law --> gmail
- MathMonkeyMan 2y agoWhen writing programs that take other programs as inputs, and/or produce other programs as outputs, it's tempting to treat the program as only slightly more structured than its textual representation. The problem is that unless your use case is very limited and is guaranteed to stay that way, supporting more and more language constructs will quickly turn your code into a mess. Compiler design as we learn it (lex/parse, syntax tree, semantic checks, transforms, lowering to codegen) is _the_ solution to the problem of dealing with computer programs as inputs and outputs. Trying to do something less is like solving a dynamic programming problem without knowing dynamic programming: it will only work for a restricted set of inputs.
- franciscop 2y agoI've come very close to be the "sir" mentioned in the article (for hobby stuff only though), luckily always I did catch myself and was able to stop before it was too late. I decided at some point that I do not want to build a compiler/transpiler, and if I do some day, I want for it to be a conscious decision and not an accident like in the article. It starts innocently, e.g. doing some template files and replacing some simple values, then you start to have to do more replacements and more "smart" parsing and then at some point it's too late, as the article suggests. TBF, I did put together a transpiler from PHP to JS, but I didn't build it, just found the different pieces that luckily fit together and hacked around it enough that it could run in the browser.
- DHaldane 2y agoIt's ok to build a compiler sometimes -- it's just very important to make that choice intentionally
- mattgreenrocks 2y agoIndeed, the safer thing is to actually build a few toy compilers on the side so you can get a sense for what they are good for, and what level of effort is required to build and maintain it. Keeping them locked up in the "scary CS" closet only ends up stunting your growth.
- SAI_Peregrinus 2y agoI like to write toy compilers or interpreters as an exercise when learning a new language. Usually for a Forth or Lisp or one of Turing Tarpit languages. It requires some of the most common bits of programming: I/O, lexing, parsing (both of source and of arguments to the compiler), file handling, and some common algorithms & data structures (can't have an AST without a tree).
- lmm 2y agoI'd say the opposite. Building a compiler incrementally, driven by clear needs at each step, as described in the article, tends to work out better than trying to build a compiler from day 1.
- pizza 2y agoIs there a meta-library someone somewhere has already written for when I just want to write 20% of a compiler, and it more or less takes care for me 80% of the common compiler-building-related things I’m likely to need to do?
- coryrc 2y agoYou should read the book Lisp In Small Pieces. It's basically a book about doing that multiple times with every-increasing percentages.
- layer8 2y agoBut can it send email?
- akshayshah 2y agoI enjoyed the article, but the unintentional Easter egg at the end left me in stitches: the link to “If Architects had to work like Programmers” just 404s, which feels spot on.
- fragmede 2y agoIt's not a subtle joke, the link was capture on archive.is: https://archive.is/r4l6C https://archive.is/r4l6C
- IncRnd 2y agoThe text to "If Architects had to work like Programmers" has been around for 20-30 years. You can find the text at many websites. Here is one from 1997: https://www.inf.ed.ac.uk/teaching/courses/seoc2/1997_1998/arch.txt https://www.inf.ed.ac.uk/teaching/courses/seoc2/1997_1998/ar... It's a fun read.
- gfody 2y agomost of this holds up really well except for > Do not worry at this time about acquiring the resources to build the house itself. Your first priority is to develop detailed plans and specifications. Once I approve these plans, I would expect the house to be under roof within 48 hours. ..nowadays would be more like: > The MVP should be move-in-ready ASAP at which point we shall move into the house and live there while you complete the remaining requirements.
- jounker 2y agoWhat’s really funny is that this doesn’t sound far off from the reality of what architects (the house kind) actually have to deal with.
- deleted 2y ago[deleted]
- fragmede 2y agoSo at what point does Kubernetes become justified?
- fragmede 2y agovindicated! https://news.ycombinator.com/item?id=42226005 https://news.ycombinator.com/item?id=42226005
- yuliyp 2y agoWhat at all does that have to do with this post about not-compilers?
- PittleyDunkin 2y agoProbably has to do with becoming mired in duplicated abstractions. I don't think this is the same thing, though, kubernetes composes what it is intended to replace with a different set of goals.
- fragmede 2y agoyou start with the notion that Kubernetes is too complicated. so you build your own deploy and hosting system, add features to it as they come up, and soon you've reinvented Kubernetes poorly
- samus 2y agoMany complaints are about the choice of YAML as a config file format. It is being used for nearly everything, and in many cases fine details of its syntax matter, especially around multiline strings and text blocks[0]. In theory, one can always retreat to JSON, but that's barely better. Then it goes on: you need additional tools like Helm or Kustomize to generate your YAML files. I am not aware whether anybody has ever tried to generate Kubernetes config files from Dhall[1] input files. [0]: https://yaml.org/spec/1.2.2/#23-scalars https://yaml.org/spec/1.2.2/#23-scalars [1]: https://dhall-lang.org/ https://dhall-lang.org/
- chamomeal 2y ago
- teaearlgraycold 2y agoI know of someone that did this for a bespoke form definition language to drive onboarding. Tens of thousands of lines, months of delays, and a bus factor of 1 later it was all eventually ripped out and replaced with plain old page templates. When your 10 question onboarding flow has a back-end class named “PredicateEvaluator” something is wrong.
- dgfitz 2y agoMan, the yocto framework could do for a read over of this.
- Joel_Mckay 2y ago"He who must not be named" has a curse upon that name. Binary optimization through stripped static-embedding and packing is a level of evil even I find upsetting. lol... =3
- kazinator 2y agoRight? It generates scripts which are then executed to do builds. Debugging broken Yocto builds can be a nightmare. You end up stepping into the build directory environment and manually running the generated code. You try some fixes in it, and then guess on which fragment in what bitbake file to backport that fix into.
- swyx 2y agoI wrote a similar recently: Oops! you built a database https://news.ycombinator.com/item?id=34941650 https://news.ycombinator.com/item?id=34941650 direct link https://dx.tips/oops-database https://dx.tips/oops-database
- jll29 2y agoThere is also the related joke [I think by Jamie W. Zawinski of Netscape Navigator fame] that any sufficiently complex program eventually will be enabled to send email, and its sister joke that any sufficiently complex program eventually will have an embedded LISP interpreter (Emacs is an example for both phenomena). Hardware people tend to say "Everything is a machine". Compiler people tend to say "Everything is a compiler". Database people tend to say "Everything is a database/relational database management system". Operating system people tend to say "Everything is an operating system". There are many broadly-applicable paradigms in modern computer science. Linguistic abstraction, i.e. the definition of a Domain Specific Language (DSL), is a very powerful technique that is often the right choice (but don't apply it to read an *.ini file! That is called overkill/distraction). Abelson and Sussman's "Structure and Interpretation of Computer Programs" book (SICP) is the gold standard book to teach you the various forms of abstraction.
- Pedro_Ribeiro 2y agoHaving recently built 90% of a compiler by mistake, I felt like this post was written specifically about me. Hilarious writing, congrats to the author.
- crdrost 2y agoThere is also the opposite, Ed Kmett is on record as saying that he had a million ideas for his personal programming language, and all of the cool stuff it was going to do, but then he ran into Haskell and said “Let’s be real, whatever amazing language I make isn't gonna be half as put together as this and I have this one already in front of me...”
- habitue 2y agoI took the article as being aimed at domain specific languages. Kmett's lens library is just such a DSL
- m463 2y agomaybe you can fiddle a bit and create a yacc/bison or lex/flex
- burnt-resistor 2y agoDon't duplicate what already exists worse except as a brief experiment or for learning. For real use, it will just cost money and time to fix and support for no reason.
- m463 2y agoI was actually joking. If you accidentally create a compiler, it's not too much more yak (yacc) shaving to create a compiler compiler...
- quantadev 2y agoIn software development it's pretty important to know when to build "on top" of something else, and when to start from scratch. Lots of developers will find it much more interesting, challenging, rewarding and just plain fun to develop something from scratch, even when there are better things that already exist. They'll cleverly manipulate and convince the boss, against the better discretion of their elder developers, that they can do it, and if they're one of the better developers, the boss won't want to risk losing them so they'll agree to the escapade. Then said escapade turns into a shambles, as predicted by the elder devs, and the developer who created the mess simply quits and moves to some other job, in search of more fun and greener pastures. Any developer with decades of experience has probably seen this same pattern multiple times.
- wheybags 2y agoThis is a sentiment that I've seen expressed in comment sections many times. I've been programming professionally now for 10 years, and it just doesn't resonate with my experience. Problems with build systems for external dependencies, package managers, and underfeatured / overcomplicated / buggy third party dependencies have been by far the worse issue in my career, compared to problems with homebrewed systems. I'm not saying you're wrong, I don't doubt that many people have the opposite experience. It just makes me feel a bit alien when I read comments like this.
- marcosdumay 2y agoI've been there, on both sides, with homebrew ideas pushed from up and down, some that worked nicely, and some that were complete disasters... And I agree with you. The problems with third party dependencies are way worse than any in-house complete disaster. But that happens almost certainly because everybody is severely biased into adding dependencies. Make people biased into NIH again, and the homebrew systems will become the largest problems again.
- lukan 2y ago"because everybody is severely biased into adding dependencies" When I make a request to chatGPT to show me a example of something with javascript and node - it always brings me a solution with a external libary needed. So I have to add "without external dependencies" - then it presents me a nice and clean solution without all the garbage I don't need. So apparently adding yet another libary seems normal for many people, otherwise this behavior would not replicate in an LLM.
- neilv 2y agoI had this kind of risk in mind when I wrote a server-side "HTML template" feature for Racket. The template language intentionally only handles static chunks of HTML, escaping of values, and a little safety guards. Everything else (including the usual template language behavior like iterating over a collection/stream, such as from a database query result) is done with arbitrary normal Racket language, which the template feature's implementation doesn't have to know about nor handle specially. https://www.neilvandyke.org/racket/html-template/ https://www.neilvandyke.org/racket/html-template/ More recently (for employability reasons, or under-resourced startup pragmatics), doing Python with Flask, JavaScript with SvelteKit, and Swift with SwiftUI, I still miss the clean simplicity and available power that I had with Scheme/Racket.
- tn1 2y agoMany older .NET applications saved programmers from this by providing "C# scripts". The framework includes the compiler and then it's trivial to use the compiled artifact. You can still do it by including the Roslyn libraries. I don't see it as much anymore, or it's some half-baked Python or Lua interface.
- celdon25 2y agoThe new Roslyn incremental generator API is pretty good these days but not well documented yet. I’ve been using it with json-schema to save a lot of boilerplate and provide a more intuitive declarative framework in a large side project.
- CharlieDigital 2y agoI have an older variant on this hacked together as a weekend project. https://github.com/CharlieDigital/js2c https://github.com/CharlieDigital/js2c Takes an input snippet of JSON and builds up classes as serialization targets.
- taeric 2y agoI am not clear on why reaching for an existing compiler's AST would ever be top of list? Don't get me wrong. I think many language design points should be used more. But starting from scratch makes a ton of sense. Skip the parsing stage and build up supported AST style constructs of your own. Done simply, this is basically the command pattern. Keep execution separate from declaration and you should be fine? Sure, you may want a parser for a dedicated serialization language some day. Hard to think you need start there? But starting with the full AST of an existing language feels like a terrible idea. In any world.
- pjungwir 2y agoI've seen this a lot when someone wants to add "workflow automation" or "scripting" to their app. The most success I'd had is embedding either Lua or Javascript (preferably Lua) with objects/functions from the business domain available to the user's script. This is what games do too. I think it's a great way to dodge most of the work. For free you can support flow control, arbitrary boolean expressions, math, etc.
- brunospars 2y agoi yearn for an alternate reality where every unix command/service had the same syntax and a lua interpreter.
- geocar 2y agoYou should take a look at arcan, it's almost exactly that: http://arcan-fe.com http://arcan-fe.com - https://arcan-fe.com/2022/10/15/whipping-up-a-new-shell-lashcat9/ https://arcan-fe.com/2022/10/15/whipping-up-a-new-shell-lash... - https://arcan-fe.com/2024/09/16/a-spreadsheet-and-a-debugger-walks-into-a-shell/ https://arcan-fe.com/2024/09/16/a-spreadsheet-and-a-debugger... I am not using it as a daily driver, because, emacs, but I keep an eye on it because, well, emacs.
- m463 2y agoI'm sorry you get our reality where it is nested quotes, parentheses, dollar signs and backslashes all the way down.
- brunospars 2y agoI lol'ed .
- alganet 2y agohttps://github.com/oasislinux/oasis https://github.com/oasislinux/oasis It's not perfect, but it's clean and moves in this lua direction.
- tonyedgecombe 2y ago
- brunospars 2y agoevery config parser is a compiler. if platforms (e.g. programming languages) made run-time plugins easier, we wouldn't even have config files. Imagine a config file with type checking and control flow. You have it-- it's your programming language. you just need to load the code at runtime, like erlang.
- samus 2y agoYou might want to check out Dhall[0], which is a configuration file format that allows a safe-ish subset of functional programming language idioms. Needless to say that it requires a full compiler :) [0]: https://dhall-lang.org/ https://dhall-lang.org/
- vishnugupta 2y agoThere’s an insider joke at Uber that if you start out building configuration manager you’ll end up with a full blown version control system.
- PittleyDunkin 2y agoI don't think building compilers is that bad, tbh. It's very difficult to do this without realizing it. I've written a dozen different programs that might be considered compilers; some very simple, others very complex and whose life continued once I left the organization. Writing a functional compiler that provides the needs of the organization where existing tooling doesn't takes discipline and focus on what you actually want to accomplish. I don't know what "defining a struct inside a loop" might mean and this strikes me as, very obviously, having no clue what you actually want to build. Perhaps the issue is not building a compiler but rather the lack of focus to begin with.
- praptak 2y agoThe conclusion is similar to the Greenspun quote "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."
- samatman 2y agoI've discovered that deploying this quote as a compliment to someone's ac hoc, informally-specified, bug-ridden, slow, half-of-Common-Lisp code, makes them increasingly nervous until you get to the punchline.
- burnt-resistor 2y ago<old-guy-high-school-glory-days-and-nobody-today> Reminds me of the pain of intentionally building a compiler for Java 2 (subset) to MIPS compiler by writing out each AST node class by hand. And, I did it twice, once in C++03 with bison and flex and again in Java 2 with CUP and JFlex... each was developed to build and run as a host portably across Solaris (sparc), Linux (x86), HP-UX (68k), SGI (MIPS), and Windows (x86) with compiled with targets run on the SPIM emulator. It did have dead code, dead string, and dead variable elimination, but that was as far my optimization passes went. I recall the only build tool I used for each was the portable subset of make without GNU extensions. Speaking of reinventing the wheel, in 1998, I built a flexible almost framework for a "portable" generic installer using Java 2, JWT (native GUI controls), and JNI on Windows to create a program group and desktop shortcut icon. The hilarious part was shipping a full JRE on a CD. It took forever to load but the additional time seemed impressive for expensive, niche software in a way similar to the now fake "loading..." delayed progress bar. </old-guy-high-school-glory-days-and-nobody-today>
- jesse__ 2y agoAre there actually fake loading bars these days? I thought software was mostly just slow and shitty ..
- allthetime 2y agoIf you have a user flow that requires interaction, hits an API that returns instantly, and then changes something fundamentally, it is better UX to make the user wait for a couple seconds with an animation and then some success state indication. If it all happens at once it can be a bit jarring or confusing.
- burnt-resistor 2y agohttps://news.ycombinator.com/item?id=16289380 https://news.ycombinator.com/item?id=16289380
- ronsor 2y agoUsers like to build up anticipation.
- torginus 2y agoI do not understand this rant. If you have the vagues pretensions of being an actual software engineer, and your file format isn't brain-dead simple, the way to parse it tokenize->grammar based parser->ast binding phase. ASTs are simple recursive data structures, if you handle them correctly, it doesn't matter if they contain 50 or 5000 nodes or how they nest, as long as the code is correct. SSA is a nice ish format for representing program code, but it's not the only choice and may or may not be appropriate for your domain. For example, if your language describes data instead of control flow, imo SSA is a bad choice. I have done this and if you take care to do things right, you won't need to bother with these hacky corner cases.
- ris 2y agoTerraform
- bsenftner 2y agoBack in the earlier days of AI, not that early, but the late 80's I was the lead developer for an AI research program being jointly conducted by 3 business professors from MIT, Harvard, and Boston University. We were working on "frame based knowledge representation" - frame of reference based node links between nodes containing something: a number, a word, a sentence, or a "function that combines linked nodes into a new frame of reference". Long story short, we thought we were making a new type of N-dimensional spreadsheet, but after 3 semesters of work one of the advisors at MIT told us we need to meet his colleague, and that guy informed us we had a working compiler for a hybrid of Lisp and C.
- benrutter 2y agoThis was a fun read! It has a link at the bottom to "if architects had to work like software engineers" which sounds fun, but the link no longer works, and searching doesn't bring anything up. Anyone here know where I can find it?
- mysticmode 2y agoSearching on it goes to 1995 written post. I have not read this, just sharing: https://paws.kettering.edu/~jhuggins/humor/design.html https://paws.kettering.edu/~jhuggins/humor/design.html
- tda 2y agoSo what do you use to know if you need to build it yourself or if there is already something out there? Niot being able to find a tool for the problem does not mean it doesn't exist, just that you haven't found it. Especially when you lack the familiarity with the problem to know the correct keywords. I find ChatGPT to be of great help to explore the area, find relevant keywords or the name of the research domain. Sometimes you really need to know exactly what you are looking for before you can find the link to that one super helpful github library that solves you problem. The of course the next step is figuring out if you want to take on the dependency or not... I have wasted hours searching for an (analytical) inverse kinematics library for robotic arms. There are tons of slow non analytical libraries out there, and some horrible ones like ikfast that is a effectively a code generator that spits out c that can be compiled with python bindings. I eventually did find https://github.com/Jmeyer1292/opw_kinematics https://github.com/Jmeyer1292/opw_kinematics, which someone ported rust (for which it was easy to create python bindings).
- einpoklum 2y agoA point to note here, is that even if you're working on a software system that already _is_ a compiler, you might still find you're building a small, different compiler somewhere else within that project. https://imgflip.com/i/9be66w https://imgflip.com/i/9be66w
- casey2 2y agoIt's presented like a magic fact of life, in reality people do what they are taught and are quite impotent without knowledge, most universities have some sort of compiler course probably using the dragon book, or a derivative copy and these students proceed to go out into the real world and write more or less the same implementation they saw in school with the same mistakes. Compilers are interesting, but there is literally no proof that they are optimal for any of their popular applications. Which is what I think you are trying to imply by this narrative you have constructed of people constantly reinventing compilers. This is just the same propagandist argument lispweenies make to claim that their language is special.
- habitue 2y agoSo, we should build more compilers. The only limiting factor, I think, is that it's damn hard to design a good DSL that wraps your domain well and is neither too flexible (increasing boilerplate) nor too rigid (increasing workarounds and escape hatch usage). But generically, a compiler is the exact kind of thing you want when you're doing "Take this data structure and transform it into this other data structure". In a traditional compiler, we usually deserialize the first data structure from a string (parsing), call that data structure a CST, validate the data structure (syntax & type checking), do the transform, then serialize the output. This kind of validate and transform pattern is all over programming though. And it's pretty easy to test with things like property tests. So yeah, we should build little compilers more as abstraction boundaries in our code.
- w10-1 2y agoCIS-10: encoding is the essence of the data+algorithms dyad WORK-n: write an interpreter WORK-n+1: add indirection ...
- stmw 2y agoI have seen this happen countless of times at companies large and small. The article is brilliant in humorously highlighting the denial (usually) or the lack of knowledge (sometimes) that leads engineers down this path again and again.
- ok123456 2y agoGoes along with, "Dear sir, you have built a lisp"---usually from first principles, ad hoc, and with glaring defects.
- ndesaulniers 2y agoI'm reminded of the (broken) C parser in the Linux kernel used for modversions. God forbid you just use libclang.