3 ms·

Writing a lexer in Go with Lexmachine (2016)

johnsonjo 9y ago
I just took a class at my university which had us implement our own lexer for Java (I believe it was supposed to be simpler than full fledged Java). The class was titled "programming languages", and we learned about alot of the different properties and mechanics of programming languages. This stuff is really interesting to learn, but can become dry at some times. I always wondered about the details and different aspects of what makes a programming language work. Although it interests me I prefer software and web development and see my career going more that route. I'm definitely appreciative of others that have learned so much about this field and have advanced it. I love the variety of different programming languages available. The rest of my comment is kind of a tangent about how I came to appreciate others work, because of the concept of specialization. When I learned about specialization in my economics class and how it can be more beneficial most of the time to let others who are more 'specialized' than you do that work while you work on your specialization I realized two things. First off learning about specialization took a weight off my shoulder about wanting to learn about absolutely everything in detail. I mean of course this can be taken too far where you don't learn applicable areas of study enough or even at all. There is obviously too much in the world to learn and I would say even to much in the field of computer science than for any one person to know it all. It reminds me of this blog post about what you learn with a phd. http://matt.might.net/articles/phd-school-in-pictures/ http://matt.might.net/articles/phd-school-in-pictures/. Basically the author has you imagine the scope of all human knowledge as a circle. (By the way the author has illustrations of you can't follow my explanation.) He then states that an elementary education is like a somewhat small dot in the center of the circle. You have learned a somewhat small amount of all available knowledge. He then says that a high school diploma and associates degree somewhat expands the circle in every direction. Yet again to go at the analogy your expanding your knowledge in every area. Then a bachelors degree expands out a parabola like sector out of the dot towards the edge of the circle getting thinner the farther out it goes. So basically your building more depth of knowledge in one area. He then explains a masters degree takes you to the edge of human knowledge and that a phd lets you make a little blip of contribution to all of human knowledge. Secondly specialization also taught me to appreciate others' work and choice of career. I mean for example if my pipes break I'm sure glad someone put time and effort into becoming a plumber. It really goes for almost any career choice.
- terminalcommand 9y ago
  You're right, specialization is what makes the economy work. But where do we draw the line? Do we learn something just because it's useful to our carrier? Should we abstain from learning things, because we need to specialize? For example having a working knowledge of lexers and regular expressions could enable someone to write efficient scrapers. Or it could supply one with enough intellectual excitement to keep one's computer skills alive. I am in law school, will be graduating next year. From an economic perspective, all my computer knowledge is moot. But I can't let computers go, much of my identity is built upon my ability to work these machines. It's true: > There is obviously too much in the world to learn and I would say even to much in the field of computer science than for any one person to know it all. However that is not a bad thing, computers provide enough excitement to pass the hours, make you feel smart and give you a sense of connection to a whole community of like-minded tinkerers.
amenghra 9y ago
Some languages have built-in support for writing grammars (parsers, lexers). In many cases, it encourages engineers to write re-usable pieces of code instead of one-off and hard to read regular expressions. Too bad that's not the case in Go. Given that, I would pick a parsec library over a yacc/lex setup. Parsecs are much easier to reason about.

I just took a class at my university which had us implement our own lexer for Java (I believe it was supposed to be simpler than full fledged Java). The class was titled "programming languages", and we learned about alot of the different properties and mechanics of programming languages. This stuff is really interesting to learn, but can become dry at some times. I always wondered about the details and different aspects of what makes a programming language work. Although it interests me I prefer software and web development and see my career going more that route. I'm definitely appreciative of others that have learned so much about this field and have advanced it. I love the variety of different programming languages available. The rest of my comment is kind of a tangent about how I came to appreciate others work, because of the concept of specialization. When I learned about specialization in my economics class and how it can be more beneficial most of the time to let others who are more 'specialized' than you do that work while you work on your specialization I realized two things. First off learning about specialization took a weight off my shoulder about wanting to learn about absolutely everything in detail. I mean of course this can be taken too far where you don't learn applicable areas of study enough or even at all. There is obviously too much in the world to learn and I would say even to much in the field of computer science than for any one person to know it all. It reminds me of this blog post about what you learn with a phd. http://matt.might.net/articles/phd-school-in-pictures/ http://matt.might.net/articles/phd-school-in-pictures/. Basically the author has you imagine the scope of all human knowledge as a circle. (By the way the author has illustrations of you can't follow my explanation.) He then states that an elementary education is like a somewhat small dot in the center of the circle. You have learned a somewhat small amount of all available knowledge. He then says that a high school diploma and associates degree somewhat expands the circle in every direction. Yet again to go at the analogy your expanding your knowledge in every area. Then a bachelors degree expands out a parabola like sector out of the dot towards the edge of the circle getting thinner the farther out it goes. So basically your building more depth of knowledge in one area. He then explains a masters degree takes you to the edge of human knowledge and that a phd lets you make a little blip of contribution to all of human knowledge. Secondly specialization also taught me to appreciate others' work and choice of career. I mean for example if my pipes break I'm sure glad someone put time and effort into becoming a plumber. It really goes for almost any career choice.

terminalcommand 9y ago
You're right, specialization is what makes the economy work. But where do we draw the line? Do we learn something just because it's useful to our carrier? Should we abstain from learning things, because we need to specialize? For example having a working knowledge of lexers and regular expressions could enable someone to write efficient scrapers. Or it could supply one with enough intellectual excitement to keep one's computer skills alive. I am in law school, will be graduating next year. From an economic perspective, all my computer knowledge is moot. But I can't let computers go, much of my identity is built upon my ability to work these machines. It's true: > There is obviously too much in the world to learn and I would say even to much in the field of computer science than for any one person to know it all. However that is not a bad thing, computers provide enough excitement to pass the hours, make you feel smart and give you a sense of connection to a whole community of like-minded tinkerers.

amenghra 9y ago

Some languages have built-in support for writing grammars (parsers, lexers). In many cases, it encourages engineers to write re-usable pieces of code instead of one-off and hard to read regular expressions. Too bad that's not the case in Go. Given that, I would pick a parsec library over a yacc/lex setup. Parsecs are much easier to reason about.