6 ms·

I've written multiple parsers/interpreters in both JS and Python -- Python being my favorite language. From that experience, I've come around to the fact that

by chubot 12y ago

I've written multiple parsers/interpreters in both JS and Python -- Python being my favorite language. From that experience, I've come around to the fact that they're both the wrong language for writing languages -- lexers, parsers, interpreter loops, compilers. I have a good analogy to explain this. Take this OCaml program. let sum_file filename = let file = In_channel.create filename in let numbers = List.map ~f:Int.of_string (In_channel.input_lines file) in let sum = List.fold ~init:0 ~f:(+) numbers in In_channel.close file; sum ;; This is roughly equivalent to: def sum_file(filename): with open(filename) as f: return sum([int(x) for x in f]) Now think of the degree to which OCaml is awkward here. That's exactly how awkward Python and JS are for writing languages :) OCaml is based around ML-style typed records, which are exactly what you need for manipulating languages. I am not a type safety guy, but when you write languages, there are tons of nested conditionals. In Python in JS or C, you end up with bugs in those corners. It is quite easy to crash any non-trivial parser, like the CPython parser, or Clang's parsers, etc. The type safety that ML offers helps a lot with this. The code for writing parsers and interpreters is just SHORTER in OCaml than in Python. I used to think Python was the most concise language. But no, it depends on the problem domain. Try it and you'll see. Someone saying the same thing here: http://flint.cs.yale.edu/cs421/case-for-ml.html http://flint.cs.yale.edu/cs421/case-for-ml.html

kirse 12y ago
I had just stumbled on this C# 1.1 parser written in F# yesterday. I'm not sure if a syntax-tree and parser in 350 lines is decent, but I can at least see what you're talking about in regards to ML/OCaml being well-suited for the task. http://www.fssnip.net/lf http://www.fssnip.net/lf
munificent 12y ago
I agree with you in principle. After all, "ML" got its name "metalanguage" from the fact that it was explicitly designed to be a domain-specific language for implementing programming languages. It's hard to be a DSL at its job. At the same time, that has to be balanced with the ecosystem value of implementing your language in a language lots of others knows. It's a worthwhile investment to implement your language with 4x the code in Python/Java/JS/etc. if you get more than 4x contributions because of it.
- wtetzner 12y ago
  > It's a worthwhile investment to implement your language with 4x the code in Python/Java/JS/etc. if you get more than 4x contributions because of it. Maybe, but I'd say it depends on the quality of the contributions.
- chubot 12y ago
  That's definitely a consideration, depending on your goals. For many language projects, I don't think you need more than one person for the front end. For libraries and code gen, you definitely need contributions. IMO it's actually advantageous to have a compact description of a lexer, parser, and AST that one person edits or very few people edit. That's your language design. At one point I thought: "If ML is so great, then why don't you see it more often in the real world?" And then the revelation was that Python essentially uses ML to describes its AST: http://svn.python.org/projects/python/tags/r32b1/Parser/Python.asdl http://svn.python.org/projects/python/tags/r32b1/Parser/Pyth... This is Zephyr ASDL, an ML-inspired DSL for describing ASTs. A lot of ML people may recognize Andrew Appel from the authors list: https://scholar.google.com/scholar?cluster=11682730813888505200&hl=en&as_sdt=0,5&sciodt=0,5 https://scholar.google.com/scholar?cluster=11682730813888505... So my favorite language's AST is described with ML!!! And has been for 10+ years. (Python has a Grammar file as well in a different syntax). I'm actually interested in a hybrid architecture: an interpreter where OCaml generates the byte code, and then C++ executes it. I think you can produce extremely compact and flexible interpreters with this architecture, and there are several other advantages to dividing it this way. The downside is perhaps a more complex build process, but the OCaml toolchain is quite nice actually, and I have compiled it from scratch. It's head and shoulders above Haskell in that regard. OCaml can produce .o files and link with C/C++, so you still get one executable. I looked at your Wren language which I like a lot. I did wish the front end was more high level and not in C, but that's just me :) There are a lot of people coming around to OCaml. Facebook is using it for Flow and pfff language manipulation: https://github.com/facebook/pfff https://github.com/facebook/pfff (I suspect Google's code analysis tools would be cut down in size by 5x if written like this in OCaml.) And Hack, the statically typed PHP, is written in OCaml. And there are several more minor languages like HaXe and another one that are OCaml. So if you're looking for contributions from language experts, there's definitely a lot that are familiar with OCaml.
mightybyte 12y ago
I 100% agree that JS and Python are the wrong language for writing languages. Haskell, however gets you the best of both worlds. Your sum_file function in Haskell would look like this: sumFile :: FilePath -> IO Int sumFile file = sum . map read . lines <$> readFile file I don't think many people would dispute that Haskell is at least as good as OCaml for writing languages. And strangely enough, the most advanced Perl 6 implementation is even written in Haskell!
- chubot 12y ago
  Right, I was referring to all ML-based languages -- so SML, OCaml, Haskell, F#, and possibly even Rust. Do you know how this would look in Haskell? # Return K most common lines in a file def top_k(f, k): counts = collections.defaultdict(int) for line in f: counts[line] += 1 return sorted(counts.items(), key=lambda x: x[1], reverse=True))[:k] I was actually looking for the OCaml example which does this. I think it was in "Real World OCaml", and I remember it being horribly ugly compared to Python. I couldn't find it though.
  - mightybyte 12y ago
    I would do it something like this: top xs = map fst $ sortBy (compare `on` Down . snd) $ M.toList $ foldr (\x -> M.insertWith (+) x 1) mempty xs This has the type `top :: Ord a => [a] -> [a]`. You might object that I didn't include your k parameter. Because Haskell is a lazy language, I can get your behavior very simply by doing `take n . top` and it will lazily only calculate the first n values. This gives you more abstractive power and lets you write more composable code.
    - chubot 12y ago
      Yeah, so this sort of proves my point :) I don't begrudge anyone if they want to use Haskell or OCaml for everything. But personally I am interested in using ML-like languages for language engineering and not general purpose programming (maybe Rust could change that, but I'm not convinced). I program in 5+ different languages regularly so I've dealt with all the friction involved, mainly by explicitly designing systems as heterogeneous collections of Unix processes.
l_dopa 12y ago
Your comparison isn't really fair. With similar functions from Core's In_channel you can write something like: let sum_file filename = with_file filename (fold_lines ~init:0 ~f:(fun a l -> a + int_of_string l)) "Using the right tool for the job" when it comes to functional vs. "mainstream" languages is a popular meme, but it doesn't hold up to scrutiny. You can always write your own higher-level code for a given domain with a reasonably expressive language. Adding a useful type system, ADTs, etc. to a language like Python is much, much harder (though that's exactly what Microsoft and Google are trying with Javascript).
- chubot 12y ago
  FWIW I took it straight out of Real World OCaml, assuming that that's idiomatic OCaml. See my other comment -- how does top K lines in OCaml look? I recall that was in the book too, but couldn't find it. I remember it being fantastically ugly.
  - ufo 12y ago
    Idiomatic ocaml would be to use one of those "with" functions whenever possible. Keep in mind that RWO is an introductory book and that they have to show the basics before moving to the larger abstractions...
  - l_dopa 12y ago
    Here's one way to write down the K lines example: let top_k chan k = let incr_count m l = let n = try Map.find_exn m l with Not_found -> 0 in Map.add m ~key:l ~data:(n + 1) in In_channel.input_lines chan |> List.fold ~init:String.Map.empty ~f:incr_count |> Map.to_alist |> List.sort ~cmp:(fun (_,a) (_,b) -> compare b a) |> List.sub ~pos:0 ~len:k You could probably code-golph it down to a couple of lines but I find the 'pipe' operator leads to very readable code.
sitkack 12y ago
While the result is probably correct, your argument is, odd. :) Hammers in nails with an orange, and this folks, is how Black and Decker makes better drills than Makita. The single reason that OCaml excels at compilers is due to pattern matching. It _removes_ all those explicit conditionals by making them implicit. http://andrej.com/plzoo/ http://andrej.com/plzoo/ Python has excellent affordances for writing compilers, they just have to be used. https://github.com/mrocklin/multipledispatch/ https://github.com/mrocklin/multipledispatch/ https://github.com/Suor/patterns https://github.com/Suor/patterns
- chubot 12y ago
  It's not just pattern matching -- it's also ADTs and the functional paradigm. (e.g. in Python, functools.partial is uglier than OCaml's application, just as working with dictionaries is uglier in OCaml than Python) Do those Python libraries offer type safety? I don't think there is any way they can. If not, you might as well use OCaml. As I said, Python is my favorite language, but there is no reason to write OCaml in Python or even shell scripts in Python. Why not use the real thing? The Python code still won't be shorter than OCaml, even if you use these fancy libraries.

by chubot 12y ago

kirse 12y ago

I had just stumbled on this C# 1.1 parser written in F# yesterday. I'm not sure if a syntax-tree and parser in 350 lines is decent, but I can at least see what you're talking about in regards to ML/OCaml being well-suited for the task. http://www.fssnip.net/lf http://www.fssnip.net/lf

munificent 12y ago

I agree with you in principle. After all, "ML" got its name "metalanguage" from the fact that it was explicitly designed to be a domain-specific language for implementing programming languages. It's hard to be a DSL at its job. At the same time, that has to be balanced with the ecosystem value of implementing your language in a language lots of others knows. It's a worthwhile investment to implement your language with 4x the code in Python/Java/JS/etc. if you get more than 4x contributions because of it.

wtetzner 12y ago
> It's a worthwhile investment to implement your language with 4x the code in Python/Java/JS/etc. if you get more than 4x contributions because of it. Maybe, but I'd say it depends on the quality of the contributions.
chubot 12y ago
That's definitely a consideration, depending on your goals. For many language projects, I don't think you need more than one person for the front end. For libraries and code gen, you definitely need contributions. IMO it's actually advantageous to have a compact description of a lexer, parser, and AST that one person edits or very few people edit. That's your language design. At one point I thought: "If ML is so great, then why don't you see it more often in the real world?" And then the revelation was that Python essentially uses ML to describes its AST: http://svn.python.org/projects/python/tags/r32b1/Parser/Python.asdl http://svn.python.org/projects/python/tags/r32b1/Parser/Pyth... This is Zephyr ASDL, an ML-inspired DSL for describing ASTs. A lot of ML people may recognize Andrew Appel from the authors list: https://scholar.google.com/scholar?cluster=11682730813888505200&hl=en&as_sdt=0,5&sciodt=0,5 https://scholar.google.com/scholar?cluster=11682730813888505... So my favorite language's AST is described with ML!!! And has been for 10+ years. (Python has a Grammar file as well in a different syntax). I'm actually interested in a hybrid architecture: an interpreter where OCaml generates the byte code, and then C++ executes it. I think you can produce extremely compact and flexible interpreters with this architecture, and there are several other advantages to dividing it this way. The downside is perhaps a more complex build process, but the OCaml toolchain is quite nice actually, and I have compiled it from scratch. It's head and shoulders above Haskell in that regard. OCaml can produce .o files and link with C/C++, so you still get one executable. I looked at your Wren language which I like a lot. I did wish the front end was more high level and not in C, but that's just me :) There are a lot of people coming around to OCaml. Facebook is using it for Flow and pfff language manipulation: https://github.com/facebook/pfff https://github.com/facebook/pfff (I suspect Google's code analysis tools would be cut down in size by 5x if written like this in OCaml.) And Hack, the statically typed PHP, is written in OCaml. And there are several more minor languages like HaXe and another one that are OCaml. So if you're looking for contributions from language experts, there's definitely a lot that are familiar with OCaml.

mightybyte 12y ago

I 100% agree that JS and Python are the wrong language for writing languages. Haskell, however gets you the best of both worlds. Your sum_file function in Haskell would look like this: sumFile :: FilePath -> IO Int sumFile file = sum . map read . lines <$> readFile file I don't think many people would dispute that Haskell is at least as good as OCaml for writing languages. And strangely enough, the most advanced Perl 6 implementation is even written in Haskell!

chubot 12y ago
Right, I was referring to all ML-based languages -- so SML, OCaml, Haskell, F#, and possibly even Rust. Do you know how this would look in Haskell? # Return K most common lines in a file def top_k(f, k): counts = collections.defaultdict(int) for line in f: counts[line] += 1 return sorted(counts.items(), key=lambda x: x[1], reverse=True))[:k] I was actually looking for the OCaml example which does this. I think it was in "Real World OCaml", and I remember it being horribly ugly compared to Python. I couldn't find it though.
- mightybyte 12y ago
  I would do it something like this: top xs = map fst $ sortBy (compare `on` Down . snd) $ M.toList $ foldr (\x -> M.insertWith (+) x 1) mempty xs This has the type `top :: Ord a => [a] -> [a]`. You might object that I didn't include your k parameter. Because Haskell is a lazy language, I can get your behavior very simply by doing `take n . top` and it will lazily only calculate the first n values. This gives you more abstractive power and lets you write more composable code.
  - chubot 12y ago
    Yeah, so this sort of proves my point :) I don't begrudge anyone if they want to use Haskell or OCaml for everything. But personally I am interested in using ML-like languages for language engineering and not general purpose programming (maybe Rust could change that, but I'm not convinced). I program in 5+ different languages regularly so I've dealt with all the friction involved, mainly by explicitly designing systems as heterogeneous collections of Unix processes.

l_dopa 12y ago

Your comparison isn't really fair. With similar functions from Core's In_channel you can write something like: let sum_file filename = with_file filename (fold_lines ~init:0 ~f:(fun a l -> a + int_of_string l)) "Using the right tool for the job" when it comes to functional vs. "mainstream" languages is a popular meme, but it doesn't hold up to scrutiny. You can always write your own higher-level code for a given domain with a reasonably expressive language. Adding a useful type system, ADTs, etc. to a language like Python is much, much harder (though that's exactly what Microsoft and Google are trying with Javascript).

chubot 12y ago
FWIW I took it straight out of Real World OCaml, assuming that that's idiomatic OCaml. See my other comment -- how does top K lines in OCaml look? I recall that was in the book too, but couldn't find it. I remember it being fantastically ugly.
- ufo 12y ago
  Idiomatic ocaml would be to use one of those "with" functions whenever possible. Keep in mind that RWO is an introductory book and that they have to show the basics before moving to the larger abstractions...
- l_dopa 12y ago
  Here's one way to write down the K lines example: let top_k chan k = let incr_count m l = let n = try Map.find_exn m l with Not_found -> 0 in Map.add m ~key:l ~data:(n + 1) in In_channel.input_lines chan |> List.fold ~init:String.Map.empty ~f:incr_count |> Map.to_alist |> List.sort ~cmp:(fun (_,a) (_,b) -> compare b a) |> List.sub ~pos:0 ~len:k You could probably code-golph it down to a couple of lines but I find the 'pipe' operator leads to very readable code.

sitkack 12y ago

While the result is probably correct, your argument is, odd. :) Hammers in nails with an orange, and this folks, is how Black and Decker makes better drills than Makita. The single reason that OCaml excels at compilers is due to pattern matching. It _removes_ all those explicit conditionals by making them implicit. http://andrej.com/plzoo/ http://andrej.com/plzoo/ Python has excellent affordances for writing compilers, they just have to be used. https://github.com/mrocklin/multipledispatch/ https://github.com/mrocklin/multipledispatch/ https://github.com/Suor/patterns https://github.com/Suor/patterns

chubot 12y ago
It's not just pattern matching -- it's also ADTs and the functional paradigm. (e.g. in Python, functools.partial is uglier than OCaml's application, just as working with dictionaries is uglier in OCaml than Python) Do those Python libraries offer type safety? I don't think there is any way they can. If not, you might as well use OCaml. As I said, Python is my favorite language, but there is no reason to write OCaml in Python or even shell scripts in Python. Why not use the real thing? The Python code still won't be shorter than OCaml, even if you use these fancy libraries.