15 ms·

GitHub dropped Pygments

184 points by mkonecny 12y ago

Matthias247 12y ago
I don't know about pygments but my experience with writing a custom highlighter for Sublime Text (aka Textmate, Atom and what Github seems to use now) was that it is not really a good and reliable system for highlighting. It is really easy to highlight simple things (keywords, numbers, ...). However when it comes to more complex scenarios (e.g. where the type of a word depends on the previous one) then the singleline regex based mechanism shows it weakness. Due to that many language support plugins will yield wrong results when you start to split things like function declarations over several lines, even though it's perfectly legal in the languages. Some things can be worked around with the start/end regexes, but nesting those multiple levels deep can get quite akward and I don't think that they were thought of for things beyond braces and multiline comments. Therefore I don't know if Githubs move here is a really good choice. However I think their main motivation might be that this file format already has such a big ecosystem due to Textmate, Sublime and Atom and the parser has a high performance so that they went for it.
- jessaustin 12y ago
  I really would have expected a highlighter written in Coffeescript to do a better job highlighting Coffeescript. https://github.com/atom/highlights#using-in-code https://github.com/atom/highlights#using-in-code (Generic identifier looks the same as class identifier looks the same as a property in an object literal looks the same as a string. I guess maybe orange is just the color used left of an equal or colon, but in that case the string color should be different.)
jarcane 12y ago
It is still difficult for me to express my opinion of this move without simply resorting to strings of profanity. Frankly, I suspect Atom has more to do with it than anything.
- walterbell 12y ago
  Why is Atom more strategically important to Github than existing developers? Is there an economic belief that Atom will reach some larger base of untapped developers who don't know how to use other editors? If a vendor cannot be trusted to host your content without regression, why would you trust that vendor to supply your mission-critical editor?
  - stingraycharles 12y ago
    No, but the article highlights that maybe they are using Github's traction to get a better syntax highlighter for Atom. As in, instead of waiting for a large community of Atom users contribute Atom syntax highlighting, they simply push the same engine onto Github and let that community take care of the syntax highlighting they actually care bout. If this were the case, it would be slightly evil, but very clever business-wise. But we don't know, so all this is is a small conspiracy theory.
    - vegedor 12y ago
      If the strategy is that obvious, how would it still be clever? Rather than simply brazen, that is.
- deleted 12y ago
  [deleted]
cespare 12y ago
I assume the new syntax highlighter is way, way faster than pygments. It's written in C++ rather than Python. (Atom uses the same grammar format, but a Node implementation.)
- walterbell 12y ago
  Why not give existing users the choice of migration and timing?
  - scott_karana 12y ago
    Giving users the choice of "slow" in a webapp means that both hardware and software deployment/management/purchasing is greatly complicated.
    - walterbell 12y ago
      Existing users are being served by existing hardware and software. It wasn't "slow" for them before. Many companies provide existing users with advance notice of technology migration, so they can plan and adapt. Why is advance customer notice incompatible with Github ops planning?
      - __david__ 12y ago
        It's not like they broke git, or some actual important interface. They stopped syntax highlighting on non mainstream languages until new lexers can be written. What on earth would you as a customer have done if they had said, "syntax highlighting in the browser for <your-favorite-language> is going to go away in a week"? Are you going to migrate away from github? Or shrug and say, "bummer"? Github did the most efficient thing: break it and let the people who actually care fix it.
        walterbell 12y ago
        > It's not like they broke git, or some actual important interface Does the vendor or the customer decide what's important to the customer?
isomorphic 12y ago
This sort of thing feeds my paranoia about GitHub being a giant single point of failure in the open-source world. I know the argument: Someone, somewhere has a copy of each repo checked out, so we (the nebulous "we") could reconstruct everything from the diaspora of ".git" directories. It just bothers me to think how dependent OSS has become upon GitHub.
- Demiurge 12y ago
  I don't think it's paranoia. I think it factually is a single point of failure, which is a bad thing(tm). And GMail is a single point of failure for my (and my companies) e-mail, and that bothers me too, despite the fact that I'm not doing anything about it :)
  - isomorphic 12y ago
    Google is a giant single point of failure. Others would rush to fill the vacuum, but it'd be a brutal few months (years) while we all relied upon Bing. Is there any GitHub-esque outfit waiting in the wings that provides free OSS hosting?
    - Twirrim 12y ago
      https://bitbucket.org/ https://bitbucket.org/ https://www.codebasehq.com/ https://www.codebasehq.com/ https://www.fogcreek.com/kiln/ https://www.fogcreek.com/kiln/ https://about.gitlab.com/gitlab-com/ https://about.gitlab.com/gitlab-com/ I must admit, I rather like bitbucket, you can get free private repositories with them too.
      - smoe 12y ago
        I can really recommend the open source gitlab for self hosting repos. I have been running it on a ec2 micro instance for 5-10 people over years without a single problem. https://github.com/gitlabhq/gitlabhq https://github.com/gitlabhq/gitlabhq
abhinavg 12y ago
I honestly don't see a problem here. They decided to change a backend library for a non-essential system in their product. Most services don't ask for permission or make announcements when they make changes like this. The approach seemed to be, if things break, people will report it and we’ll fix it. While this may not be the best approach, the number of languages supported is too high for a person to check each one manually. Generally, I imagine they wouldn't expect a change like this to break anything significant. [people] use it as a portfolio. [..] To suddenly doink the appearance of people’s portfolios is unfortunate. It is very unlikely that syntax highlighting errors in GitHub will affect someone's chances of getting a job. Sure, this switch could cause some issues but they don't seem to be severe enough to kick up a fuss over.
- Demiurge 12y ago
  > It is very unlikely that syntax highlighting errors in GitHub will affect someone's chances of getting a job. And if it does, the problem lies not with the syntax highlighting, not even close...
- conradev 12y ago
  One other thing to note is that it's not just syntax highlighting, it's also recognizing the language a given repository is written in. Some of my repositories that use Logos[1] are now incorrectly classified as a combination of Ruby and Scala[2]. [1] http://iphonedevwiki.net/index.php/Logos http://iphonedevwiki.net/index.php/Logos [2] https://github.com/conradev/Tweaks https://github.com/conradev/Tweaks
  - lelandbatey 12y ago
    Yep. And now Github thinks my Makefile is JavaScript: https://github.com/lelandbatey/link_grab https://github.com/lelandbatey/link_grab
    - lbearl 12y ago
      It comes up as an "Executable File" for me. Guess they are making tweaks still.
alayne 12y ago
Racket is a fringe language. Github has about as many Prolog repositories as Racket repositories. If Racket syntax highlighting was causing performance issues that were noticeable to Github, performance must have really sucked. Why should Github let Racket drag down its capacity?
- walterbell 12y ago
  Is that the message an open-source vendor wants to send to developers: your code is welcome, unless it's a new and unproven "fringe" language? How would a "fringe" developer feel about that vendor's brand after their language has become successful? Would they like/trust/recommend the vendor to upcoming developers?
  - alayne 12y ago
    Racket isn't new. It's a scheme which has been in development for around 20 years. Reading their thread, it sure sounds like they want to work with the community to make Racket code viewing a good experience.
hawkice 12y ago
GitHub: a product so close to our hearts that even extremely well meaning changes that really do help in many cases can sting sometimes. I normally wouldn't understand this type of thing (others say they don't see the problem and it's quite clear where they are coming from), but in a way I _do_ see the author's point of view. When you build something people really care about, any change, no matter how minor, has the opportunity to impact someone. That's why we all build things, isn't it?
ivan4th 12y ago
Seems like Common Lisp syntax highlighting suffered, too.
slantedview 12y ago
It's worth noting that even syntax highlighting on common languages like Java is currently messed up on Github. I hope it's fixed eventually, but kinda lame to take something that wasn't broken and break it.
- needusername 12y ago
  Linguist has trouble telling C and C++ apart.
_pmf_ 12y ago
GitHub is not your personal vanity site.
latkin 12y ago
F# highlighting is also now totally gone with this change. Code is highlighted with some random lexer than doesn't even understand // comments. Rather frustrating, given that a lot of F# development is centered around GH, and GH themselves use F# in a couple of places. Browsing the issues list, this isn't just "fringe" languages, either. Perl, PHP, Go, and Clojure all appear to have regressed to some degree.
shurcooL 12y ago
Interesting coincidence, as I've just changed the highlighter in my Go library for offline rendering of GitHub Flavored Markdown [1] 23 hours ago. [1] - https://github.com/shurcooL/go/commit/6aad35a0a60fd67927f446e140367f932f1b46b2 https://github.com/shurcooL/go/commit/6aad35a0a60fd67927f446...
ttwwmm 12y ago
Rendering of reStructuredText disappeared in GitHub Enterprise ~6 months ago... I wonder if the reasoning was the same. More limited resources in the GHE VM environment?
frontsideair 12y ago
I understand the author's exasperation, but the post is surely filled with loaded words and wild speculations. It may get the message across, but this is not the way to start a conversation.
GFK_of_xmaspast 12y ago
"""pygments.rb had an interesting history of trying to use a Python library in Ruby on a high-traffic web site. They had tried various approaches. The final approach — piping to a long-running Python process — seemed to work well. """ That doesn't fill me with a lot of confidence in this thing.

184 points by mkonecny 12y ago

I don't know about pygments but my experience with writing a custom highlighter for Sublime Text (aka Textmate, Atom and what Github seems to use now) was that it is not really a good and reliable system for highlighting. It is really easy to highlight simple things (keywords, numbers, ...). However when it comes to more complex scenarios (e.g. where the type of a word depends on the previous one) then the singleline regex based mechanism shows it weakness. Due to that many language support plugins will yield wrong results when you start to split things like function declarations over several lines, even though it's perfectly legal in the languages. Some things can be worked around with the start/end regexes, but nesting those multiple levels deep can get quite akward and I don't think that they were thought of for things beyond braces and multiline comments. Therefore I don't know if Githubs move here is a really good choice. However I think their main motivation might be that this file format already has such a big ecosystem due to Textmate, Sublime and Atom and the parser has a high performance so that they went for it.

jessaustin 12y ago
I really would have expected a highlighter written in Coffeescript to do a better job highlighting Coffeescript. https://github.com/atom/highlights#using-in-code https://github.com/atom/highlights#using-in-code (Generic identifier looks the same as class identifier looks the same as a property in an object literal looks the same as a string. I guess maybe orange is just the color used left of an equal or colon, but in that case the string color should be different.)

jarcane 12y ago

It is still difficult for me to express my opinion of this move without simply resorting to strings of profanity. Frankly, I suspect Atom has more to do with it than anything.

walterbell 12y ago
Why is Atom more strategically important to Github than existing developers? Is there an economic belief that Atom will reach some larger base of untapped developers who don't know how to use other editors? If a vendor cannot be trusted to host your content without regression, why would you trust that vendor to supply your mission-critical editor?
- stingraycharles 12y ago
  No, but the article highlights that maybe they are using Github's traction to get a better syntax highlighter for Atom. As in, instead of waiting for a large community of Atom users contribute Atom syntax highlighting, they simply push the same engine onto Github and let that community take care of the syntax highlighting they actually care bout. If this were the case, it would be slightly evil, but very clever business-wise. But we don't know, so all this is is a small conspiracy theory.
  - vegedor 12y ago
    If the strategy is that obvious, how would it still be clever? Rather than simply brazen, that is.
deleted 12y ago
[deleted]

cespare 12y ago

I assume the new syntax highlighter is way, way faster than pygments. It's written in C++ rather than Python. (Atom uses the same grammar format, but a Node implementation.)

walterbell 12y ago
Why not give existing users the choice of migration and timing?
- scott_karana 12y ago
  Giving users the choice of "slow" in a webapp means that both hardware and software deployment/management/purchasing is greatly complicated.
  - walterbell 12y ago
    Existing users are being served by existing hardware and software. It wasn't "slow" for them before. Many companies provide existing users with advance notice of technology migration, so they can plan and adapt. Why is advance customer notice incompatible with Github ops planning?
    - __david__ 12y ago
      It's not like they broke git, or some actual important interface. They stopped syntax highlighting on non mainstream languages until new lexers can be written. What on earth would you as a customer have done if they had said, "syntax highlighting in the browser for <your-favorite-language> is going to go away in a week"? Are you going to migrate away from github? Or shrug and say, "bummer"? Github did the most efficient thing: break it and let the people who actually care fix it.
      - walterbell 12y ago
        > It's not like they broke git, or some actual important interface Does the vendor or the customer decide what's important to the customer?

isomorphic 12y ago

This sort of thing feeds my paranoia about GitHub being a giant single point of failure in the open-source world. I know the argument: Someone, somewhere has a copy of each repo checked out, so we (the nebulous "we") could reconstruct everything from the diaspora of ".git" directories. It just bothers me to think how dependent OSS has become upon GitHub.

Demiurge 12y ago
I don't think it's paranoia. I think it factually is a single point of failure, which is a bad thing(tm). And GMail is a single point of failure for my (and my companies) e-mail, and that bothers me too, despite the fact that I'm not doing anything about it :)
- isomorphic 12y ago
  Google is a giant single point of failure. Others would rush to fill the vacuum, but it'd be a brutal few months (years) while we all relied upon Bing. Is there any GitHub-esque outfit waiting in the wings that provides free OSS hosting?
  - Twirrim 12y ago
    https://bitbucket.org/ https://bitbucket.org/ https://www.codebasehq.com/ https://www.codebasehq.com/ https://www.fogcreek.com/kiln/ https://www.fogcreek.com/kiln/ https://about.gitlab.com/gitlab-com/ https://about.gitlab.com/gitlab-com/ I must admit, I rather like bitbucket, you can get free private repositories with them too.
    - smoe 12y ago
      I can really recommend the open source gitlab for self hosting repos. I have been running it on a ec2 micro instance for 5-10 people over years without a single problem. https://github.com/gitlabhq/gitlabhq https://github.com/gitlabhq/gitlabhq

abhinavg 12y ago

I honestly don't see a problem here. They decided to change a backend library for a non-essential system in their product. Most services don't ask for permission or make announcements when they make changes like this. The approach seemed to be, if things break, people will report it and we’ll fix it. While this may not be the best approach, the number of languages supported is too high for a person to check each one manually. Generally, I imagine they wouldn't expect a change like this to break anything significant. [people] use it as a portfolio. [..] To suddenly doink the appearance of people’s portfolios is unfortunate. It is very unlikely that syntax highlighting errors in GitHub will affect someone's chances of getting a job. Sure, this switch could cause some issues but they don't seem to be severe enough to kick up a fuss over.

Demiurge 12y ago
> It is very unlikely that syntax highlighting errors in GitHub will affect someone's chances of getting a job. And if it does, the problem lies not with the syntax highlighting, not even close...
conradev 12y ago
One other thing to note is that it's not just syntax highlighting, it's also recognizing the language a given repository is written in. Some of my repositories that use Logos[1] are now incorrectly classified as a combination of Ruby and Scala[2]. [1] http://iphonedevwiki.net/index.php/Logos http://iphonedevwiki.net/index.php/Logos [2] https://github.com/conradev/Tweaks https://github.com/conradev/Tweaks
- lelandbatey 12y ago
  Yep. And now Github thinks my Makefile is JavaScript: https://github.com/lelandbatey/link_grab https://github.com/lelandbatey/link_grab
  - lbearl 12y ago
    It comes up as an "Executable File" for me. Guess they are making tweaks still.

alayne 12y ago

Racket is a fringe language. Github has about as many Prolog repositories as Racket repositories. If Racket syntax highlighting was causing performance issues that were noticeable to Github, performance must have really sucked. Why should Github let Racket drag down its capacity?

walterbell 12y ago
Is that the message an open-source vendor wants to send to developers: your code is welcome, unless it's a new and unproven "fringe" language? How would a "fringe" developer feel about that vendor's brand after their language has become successful? Would they like/trust/recommend the vendor to upcoming developers?
- alayne 12y ago
  Racket isn't new. It's a scheme which has been in development for around 20 years. Reading their thread, it sure sounds like they want to work with the community to make Racket code viewing a good experience.

hawkice 12y ago

GitHub: a product so close to our hearts that even extremely well meaning changes that really do help in many cases can sting sometimes. I normally wouldn't understand this type of thing (others say they don't see the problem and it's quite clear where they are coming from), but in a way I _do_ see the author's point of view. When you build something people really care about, any change, no matter how minor, has the opportunity to impact someone. That's why we all build things, isn't it?

ivan4th 12y ago

Seems like Common Lisp syntax highlighting suffered, too.

slantedview 12y ago

It's worth noting that even syntax highlighting on common languages like Java is currently messed up on Github. I hope it's fixed eventually, but kinda lame to take something that wasn't broken and break it.

needusername 12y ago
Linguist has trouble telling C and C++ apart.

_pmf_ 12y ago

GitHub is not your personal vanity site.

latkin 12y ago

F# highlighting is also now totally gone with this change. Code is highlighted with some random lexer than doesn't even understand // comments. Rather frustrating, given that a lot of F# development is centered around GH, and GH themselves use F# in a couple of places. Browsing the issues list, this isn't just "fringe" languages, either. Perl, PHP, Go, and Clojure all appear to have regressed to some degree.

shurcooL 12y ago

Interesting coincidence, as I've just changed the highlighter in my Go library for offline rendering of GitHub Flavored Markdown [1] 23 hours ago. [1] - https://github.com/shurcooL/go/commit/6aad35a0a60fd67927f446e140367f932f1b46b2 https://github.com/shurcooL/go/commit/6aad35a0a60fd67927f446...

ttwwmm 12y ago

Rendering of reStructuredText disappeared in GitHub Enterprise ~6 months ago... I wonder if the reasoning was the same. More limited resources in the GHE VM environment?

frontsideair 12y ago

I understand the author's exasperation, but the post is surely filled with loaded words and wild speculations. It may get the message across, but this is not the way to start a conversation.

GFK_of_xmaspast 12y ago

"""pygments.rb had an interesting history of trying to use a Python library in Ruby on a high-traffic web site. They had tried various approaches. The final approach — piping to a long-running Python process — seemed to work well. """ That doesn't fill me with a lot of confidence in this thing.