8 ms·
I am afraid to inform you that you have built a compiler (2022)
- deleted 3y ago[deleted]
- dang 3y agoRelated: Dear sir, you have built a compiler - https://news.ycombinator.com/item?id=29891428 https://news.ycombinator.com/item?id=29891428 - Jan 2022 (175 comments)
- ashton314 3y agoIt's funny how projects grow like this organically. Perhaps burned by experience, one time I implemented a mini-language for specifying some business logic that I knew—just knew—that our client would change his mind on a dozen times and not understand the half of the ramifications of his requests and would only arrive at the solution he really wanted by trial-and-error. Was the little language I made as complex as a "real" compiler? Goodness no. Was I happy to have a flexible language-based solution? True to my predictions, I did get many, many logic change request and handled them with ease. Yes, I was very happy after that.
- alanfranz 3y agoNow the question is: did you make your client pay for each request _just as if you didn’t build such dsl_? In my experience, most developers-as-consultants aren’t good at that. So, when we’re (rightfully) late, we probably reduce our margins. When we do a great job, we don’t make the client pay for his own mistakes.
- jimvdv 3y agoYou are right most developers are not very good at that. I think it’s important to stress that this goes both ways. You should not cut your margins if you take longer, but I also think it is a little unethical to bill more hours than one spends no? At least if you have a clear agreement to bill by the hour.
- awkward 3y agoThe way to ethically handle this is to have a clear and contractual minimum number of billed hours in a day. That will get you paid well for built in efficiencies like OP mentioned, and keep you paid fairly on more substantial change requests.
- thfuran 3y agoIt's outright fraud. It's clearly unethical. But your billing shouldn't be strictly hourly if you're building tooling to speed up your work.
- deleted 3y ago[deleted]
- derefr 3y agoWhat you’re saying is that it’s inherently unethical (to yourself!) to accept hourly billing for work the whole point of which is exponential automation.
- tshaddox 3y ago> did you make your client pay for each request _just as if you didn’t build such dsl_? I'm not exactly sure what you're asking. Should you be charging clients for time that you didn't have to spend because you made some good decisions upfront? That strikes me as odd. If you agreed upfront on the cost for a certain end result, then sure, charge that full agreed-upon amount even if it took you less time than expected. But it seems odd to say "this change only took me an hour, but it would have taken 10 hours if I had made worse choices in designing this system, therefore I'm going to bill you 10 hours." After all, there's always a hypothetical situation where any given change could have taken arbitrarily more time if you had made worse decisions in the past.
- mjr00 3y agoDepends on the project, but if you're handling these client requests anyway, this is not the best solution IMO, at least not for cloud/SaaS. The best solution is to have a good deployment process in place that makes it easy for you to make business logic changes using your language of choice, rather than building and maintaining a secondary language that is understood only by you. No need to create a DSL or compiler or worry about maintenance when you're just using the same tooling as everything else you use. Seen way too many times in my career developers who build the fun, interesting solution (DSL/compiler/parser) instead of the boring, but far more practical solution of fixing their deployment processes.
- yesbutum 3y ago[dead]
- Karellen 3y agoWhen the project I was working on had this problem, the request to build a DSL was based on the idea that each client ought to be able to make their own changes. The client shouldn't have to know a "full" programming language, but a "sufficiently simple" DSL isn't that much more than a config file format, right? They can learn how to make small changes themselves if the language is easy enough, can't they? I tried to convince management that this wouldn't work. If nothing else, because our DSL was invented in-house, there was no-where else the client could find answers on how to make the changes they wanted. There would be no hits on Google, no answers on StackOverflow, no random client employee who happened to have some relevant tech knowledge. If they had any questions at all, they'd need to come to us, and we'd end up writing the changes for them anyway. And, as you point out, we already had a perfectly good programming language - the one everything else (including the DSL) was written in.
- toyg 3y agoI cut that particular gordian node, once, by simply embedding a JS engine, and explaining to them (and documenting) the bare minimum of the language that they needed to achieve their objectives. After a week of getting to grip with it, they loved it so much that they kept it into production for years, happily tweaking scripts as they needed. I should have charged them more than I did.
- Const-me 3y agoI did it with VBScript, Python, LUA, and in .NET both with and without C# compiler frontend. All these methods worked fine for my use cases.
- intelVISA 3y ago> I should have charged them more than I did. That's the problem isn't it? Unless it's SaaS you don't want to solve a problem too well...
- mikepurvis 3y agoWhat were the requirements that led to implementing your own language vs embedding lua?
- duxup 3y agoI’ve passed on using many a package that I thought was too complex for for my needs only to find myself effectively writing my own version/ learning why that other package is the way it is. Suddenly the API looks oddly familiar…. OMG…
- mcny 3y agoYes, everything looks simple and straightforward from a distance. As you get closer, you are faced with dozens of not hundreds of decisions that you need to make. If you're good, you won't be stuck with analysis paralysis.
- wrs 3y agoOther needed topics in this series: “You have built a database”, “You have built an orchestrator”, “You have built an RPC layer”, “You have built a build system”…
- dang 3y agoAnd Greenspun's 10th rule I suppose.
- daniel-s 3y agoFor those that needed to Google like me: "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."
- hinkley 3y agoIn the age of horizontal scalability, what you've built is likely more like a nightmare version of Erlang, rather than Common Lisp.
- namaria 3y agoIt's the Cassandra curse of computer scientists. Business people think they don't have time or resources for correctness, and end up forcing incorrect systems to be held by hand. Which, despite the high cost of this labor, is a decent trade-off because thanks to the internet business models have very low marginal cost per costumer. I think that the fact that Netflix has a very high marginal cost per costumer and actually hires very highly qualified programmers and pioneers systems designs illustrates my point.
- Detrytus 3y agoYou can see that with so called "Big Data" tools. Those which originated as databases (Mongo), ended up adding "Map-reduce" feature. Those which started as map-reduce tools evolved to support SQL (Hadoop->Spark). Those which started as SQL engines (Spark) added support for streaming, while those who started as steaming platforms (Kafka) added SQL support (KSQLDB). Traditional DB engines evolve to allow document data (Postgres with JSON column type). One more decade until one-tool-to-rule-them-all emerges :)
- alexchantavy 3y agoI work on a Python project where I need to take class definitions and generate database query statements because all ORMs that currently exist don’t work for my needs. I'm currently doing this with string templates that I've defined by hand. Is there a smarter way? I've looked into some compiler-like tools (can't remember the specific ones, sorry), and from what I can tell their code generation phase looks very similar to mine in that they use string templates.
- 89vision 3y agohttps://github.com/zio/zio-quill https://github.com/zio/zio-quill This library does exactly what you prescribe. Pretty sure under the hood it's using macros with string templates
- alexchantavy 3y agoVery cool, thank you. Anything in Python?
- 89vision 3y agoI tried searching for something similar in python but didn't find much
- noelwelsh 3y agoScala macros and quasiquote templates do have some notable differences to pure strings. The two main ones are: - the value that's constructed has to be valid code - macro "hygiene" is maintained
- aidos 3y agoWhy don’t ORMs work for you? Have you looked at sqlalchemy?
- alexchantavy 3y agoMy project uses the Neo4j graph database and the ORMs available here aren't great: they don't handle batched writes so they are super slow, or they do weird hacks like requiring that you run a webserver to let them work, or they don't use managed transaction functions so that write operations aren't automatically retried for you.
- eternityforest 3y ago... how does this even happen? What bizarre use case doesn't allow for just using an off the shelf scripting language? I've only done anything like this once and not regretted it, and it's purely visual scripting, if this then that style. Anything that can't be handled by an event that triggers a list of actions, then stops when one returns False, I will hardcore a hack just for that feature. Most actions and triggers are responses to specific use cases.
- renox 3y agoWhat's strange is that if you had started with a Lisp, it would have been much simpler! And yet few people think about using a Lisp for their DSL.
- mbork_pl 3y agoWithout even reading TFA, it is obvious that choosing a Lisp for a DSL is probably at least a good enough choice.
- JoshuaDavid 3y agoI did this once as a junior developer, for a feature flagging system that needed to support arbitrary custom rules. I don't recommend it.
- matt_kantor 3y agoWhy not?
- RandyRanderson 3y agoCreating a configuration file? I am afraid to inform you that you have started writing a compiler. What's the only way to avoid this? Your software not being successful.
- dessimus 3y agoDammit! Every time I try putting together an Ikea bookshelf, this happens!
- guideamigo_com 3y agoAt a billion-dollar fintech, some engineer thought gRPC is too complicated. So, he built someything from scratch. A year later, there was a team of 5 engineers maintaining a half-baked implementation of gRPC. Good for him. Bad for the company.
- amoss 3y agoDon't hate the player, hate the game.
- deleted 3y ago[deleted]
- paulddraper 3y agoWho makes the game?
- hsaliak 3y agoDid that "some engineer" get promoted though? That's the real question.
- chubot 3y agoHaha ... honestly it is surprising how much work an RPC system is. Seems like it should't be -- you're marshaling arguments and return values But multiply that by a few languages, and now you have to paper over error code and exceptions, signed and unsigned ints, unicode and bytes Not to mention network errors, retries, throttling, etc. It tends to grow without bound, mostly because the RPC system tries to do too much. You can't really abstract the network -- that's the #1 lesson When you try to abstract the network, now you OWN a bunch of problems that you can't solve
- erulabs 3y agoTo some extent my entire career has been searching for and destroying said half baked implementations. This saying can be adapted to infra: “half baked, bug ridden kubernetes”, “half baked, bug ridden proxySQL”, “half baked, bug ridden redis”, the list goes on and on. In some ways I feel like my impact has been quite boring, in other ways quite vital. But it’s never made me friends with the kind of developers who look sideways at the idea that other peoples life’s work might be better than their 5 year old weekend project.
- random3 3y agoGiven how easy it is to build a Turing machine, I'd argue that building something you don't envision is the 99% of the time rule and it's already captured by https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." In fact I'd argue that it's much harder to not accidentally build something that you don't envision.
- m463 3y agoWhen we incorporate natural languages into the mix, "In the style of Charles Bukowski, write a hacker news comment that explains ..." maybe our previous works can be externally ad-hoc incorporated into new works by others...
- kallistisoft 3y agoI've experienced the 10th rule first hand... Many years ago I was writing a bunch of modular audio applications that were all interconnected using JACK. So of course I needed some convenient way of storing/restoring the graphs that connected the various components that were always in flux -- each audio experiment had it's own graph, but there were often re-occurring sub-graphs. So, I started to build a small cli utility that could save the current graph and restore it later... of course because of the common sub-graphs it needed to be able to refer to other labeled graphs so i could easily reference and re-use them... and of course to (save myself time) it also needed to have arithmetic composability so that I could add/subtract existing graphs to create new ones... and of course it would be helpful if I could also annotate the graphs so that they could launch the associated programs... So in the end I had a generic DSL that had recursive graph parsing, reification (labels), graph composability, arithmetic composability, and could associate arbitrary labels with executable code. I eventually looked up from my adderal fueled hack-a-thon and realized I had created a half-baked LISP and decided it was time for to go to bed! :)
- intelVISA 3y agoI wonder how many LISPs can be owed to the Turing-complete influence of Adderall
- makach 3y agoHow embarrassing. I've done this. Programming is easy, making things easy is difficult.
- vishnugupta 3y agoThere was a joke at Uber about beginning with a configuration management system and ending up with a version control system. And somewhere else about accidentally building a real time chat service (or was it email? Don’t recall).
- david_allison 3y ago> And somewhere else about accidentally building a real time chat service (or was it email? Don’t recall). Maybe Zawinski's Law?[0] > Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. [0] https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law https://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski's_Law
- NegativeK 3y agoAnd then there's Mallett's Law (not in Wikipedia), which is a bit more modern: Every program attempts to expand until it can render HTML. This usually results in it being able to read email.
- Detrytus 3y agoTotally offtopic but I can't help but wonder why this guy has a site with .pl TLD. He seems to be based in the US, not Poland. Does he think "pl" stands for "programming languages"? :)
- hermitdev 3y ago> Does he think "pl" stands for "programming languages"? More likely pl -> perl, I think.
- tmtvl 3y agoWell, vanity URLs have been a thing for a while now. E.g. lobste.rs.
- paulddraper 3y agoFWIW I see lots of non-Montenegrin people with .me addresses.
- justincredible 3y agoTIL .pl is for Poland. The misuse of .io is probably more common.
- rozap 3y agoA funny thing is that this can go N layers deep. At a previous job we had a DSL which was very well specified, pretty straight forward, and had been written by an exceptionally brilliant engineer (who was even still at the company!). We had two backend services that would operate on that AST in a query context and in a streaming transformation context, and they shared the parser, compiler, and typechecker. The problem was that the engineering team was so fragmented and lacking in stable technical leadership that a large part of the engineering team had no idea about how these things worked, so on every project they were constantly going "Ooops, I created a feature which has its own intermediate representation with slightly different semantics so reuse is impossible". It was super hard to deal with and stuff was constantly being invented and then thrown out because it couldn't be extended. The only thing worse than creating a compiler is being unaware that one already exists and creating a new one on top of the existing one.
- bstpierre 3y agoI accidentally built a kind-of compiler last year. It started as a few sed commands to merge TeX+code -> TeX for a book project. I ran these sed commands from a makefile. Life was easy. But then there were complications, and I needed to make slightly more sophisticated substitutions. So the sed commands moved into an awk script, run by the makefile. This was better than maintaining a handful of little commands that were growing on a weekly basis. Life was good. The transformations I needed kept growing a bunch of little variations, and the awk script became hard to maintain, so I rewrote it in go, with proper parsing and output. (And even unit tests, after the 2nd time I broke some output.) Designing it as almost-a-proper-compiler was 10x better than maintaining an ad hoc script. Life was great, even with the overhead of maintaining a separate processing tool.
- mikhailfranco 3y agoKnuth wanted to write a book, so he spent years writing a typesetting system. Both the book and the system were heroically good.
- amelius 3y agoA compiler typically transforms programs that run in O(N) time into programs that run in O(N) time, for whatever suitable definition of N, so it's not really something to be super thrilled about.
- jcarrano 3y agoThe key is to know, acknowledge and accept where you are going, and to go boldly and deliberately- or not go at all. Say "this problem looks like making a small program so I'm making a mini-language". However, if you find yourself saying ""this looks like a database..." then stop there and please do not build a db.
- tombert 3y agoIt is sort of incredible how often I've ended accidentally re-inventing interpreters or compilers without that being an explicit goal. You start by just adding some kind of configuration for rules, maybe in JSON. Then you start wenting to make more complex rules so you allow some kids of recursive system in your JSON that can nest rules and combine them. Then you find yourself copypasting rules a lot and so you implement some kind of naming convention so you can reuse rules. Then you realize how disgusting your JSON is getting so you dust off a parsing library and make a basic DSL that compiles into that JSON, and then it dawns on you.
- chii 3y agoa formal language with a grammar and rules, is something quite fundamental. Today nobody is surprised that maths shows up in engineering, or [really anywhere](https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...)! I would imagine that this is just an extension of the above idea about maths. To describe something in such a way that it operates formally, you eventually end up with stuff like LISP, or another equivalent language.
- WalterBright 3y agoIt just crept up on me.
- deleted 3y ago[deleted]
- samsquire 3y agoI am especially interested by the characterisation of handling interactions between different AST nodes. I think interactions between features are very hard to think about. I think constructed languages have the opportunity to think about potential interactions that would be useful and aim to support those ones. But there's lots of permutations to features. Just look at async functions in Rust and coloured functions. It's such as pain. It also reminds me and brings up thoughts about "the expression problem" [0] How do you think which combinations of features would be useful upfront? For example: there's interactions between memory management, garbage collection, async, multithreading, coroutines, closures, the stack, FFI. It's all very complicated. [0]: https://en.wikipedia.org/wiki/Expression_problem https://en.wikipedia.org/wiki/Expression_problem