5 ms·

I hate this sentiment. I speak English and French and have limited amount of time to hack on software I'm giving away for free. My day job consists entirely i

I hate this sentiment. I speak English and French and have limited amount of time to hack on software I'm giving away for free. My day job consists entirely in English. I have neither the experience, nor the inclination for internationalization of software. Don't like, don't use it. Or fork it, and add it yourself, because obviously you have more free time than I have.

rtpg 12y ago
the thing is if you build for unicode support from the start these conversations don't need to be had. The problem is not enough people treat text as a black box from the start (I can understand unwillingness to support bigger things like RTL)
- a3n 12y ago
  There just isn't a lot of pervasive experience in the development community for multi-language unicode devlopment. Also xlrd is fairly old, although I don't know if that tool is part of what limits this to english. In ten years it might be better.
  - sillysaurus3 12y ago
    Joel Spolsky said that ten years ago. The problem is that devs are afraid to learn unicode. They treat it like learning a foreign language. It's not even a fun problem, like learning a new programming language, so nobody makes time for it. The only people who learn it are those who make it a point of pride to implement something correctly and handle corner cases. Unicode isn't even hard: Use UTF-8. Don't try to measure the length of a string unless you're rendering that string and measuring the length in screen units like pixels. If you do those two things, that's 90% of the effort of making Unicode-safe software. I think both views are valid. Those who don't know how to write Unicode-safe software shouldn't feel shamed into learning Unicode before releasing open source work. Those who already know Unicode should feel happy that they're making other people's lives easier.
    - tracker1 12y ago
      But these are file formats that may well not be encoded in UTF-8.. the formats already exist.. it isn't like he's creating a new spreadsheet format here. Some of them may well be encoded to something that works fine against unicode/utf-8, others not so much.
      - sillysaurus3 12y ago
        So you write FooToUTF8() and UTF8ToFoo(), where Foo is whatever the encoding is in the external format. Done. As far as I know, UTF-8 will work 100% of the time, and is almost always the best internal representation for software you write due to how simple and uniform it is. If something is encoded in some other format, you can probably find a conversion function online.
        tracker1 12y ago
        Okay, so why don't you fork the project, and create your simple Foo/UTF8 methods, and confirm that they are the correct Foo/UTF8 methods for each of the document formats supported. I'm not saying that it's really all that hard, but there are multiple document formats, and versions of those formats. The author obviously didn't need unicode support, so didn't test for it. I'm sure test cases, and a pull request would be welcome.
- tracker1 12y ago
  Okay... let's go into this... how are the strings in excel encoded anyway? I'd be willing to bet money that at least some of the formats in question aren't UTF-8, they are likely ASCII encoded against a character set or code page. Then you have to read that codepage, and convert the necessary characters to their Unicode equivalents, and from there do you downcode to utf-8? Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible? Who's going to go through the different document versions to confirm, and adjust for the various encodings for non-ascii characters? It's not as simple as saying "don't choke on unicode".
  - sheetjs 12y ago
    > how are the strings in excel encoded anyway? Length-prefixed byte arrays encoded using various code pages. There are a small number that excel uses: https://github.com/SheetJS/js-codepage/blob/master/excel.csv https://github.com/SheetJS/js-codepage/blob/master/excel.csv (the columns are CP#, mapping, single/double-byte) > Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible? If we can put together an Apache2-licensed module in JS in an afternoon (https://github.com/SheetJS/js-codepage https://github.com/SheetJS/js-codepage) it can be done in python. > Who's going to go through the different document versions to confirm, and adjust for the various encodings for non-ascii characters? Someone already did that: https://github.com/SheetJS/test_files/tree/master/biff5 https://github.com/SheetJS/test_files/tree/master/biff5 has artifacts for every language type
    - gohrt 12y ago
      > If we can put together an Apache2-licensed module in JS in an afternoon (https://github.com/SheetJS/js-codepage https://github.com/SheetJS/js-codepage) it can be done in python. I thought Python 2 was Unicode-unfriendly. So not as easy as JS.
  - thristian 12y ago
    > Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible? It's written in Python, which comes with support for pretty much every major encoding¹ out of the box, so yes. ¹: https://hg.python.org/cpython/file/cb94764bf8be/Lib/encodings https://hg.python.org/cpython/file/cb94764bf8be/Lib/encoding...
izacus 12y ago
It's one thing to internationalize software. That's hard. But not being able to handle UTF-8 in 21st century is downright shameful. The notion of non-ASCII characters in user Excel documents IS NOT something rare even for English speaking nations. There are tons of people with foreign names, addresses and other personal information which is commonly stored in Excel documents. And the funny thing is: it's usually not alot of additional work to support UTF-8 if you START correctly.
- michaelmior 12y ago
  Calling the work of someone who releases free software that they may have well hacked together in their free time downright shameful seems quite offensive and disregards the work they put into the project. Perhaps the OP had no need to use this code for non-English alphabets or is simply in the early stages and hasn't had the opportunity to fully test and implement this. Regardless, I think people should be applauded for releasing their work instead of shamed. And of course, if it's not a lot of work, a pull request would likely be appreciated ;)
  - izacus 12y ago
    Hmm, I do not agree with you that the sole act of opensourcing a software should bring it above criticism. Any developer that takes a bit of pride in his work should at least keep himself to some minimal standards and I think supporting UTF-8 isn't an unreasonable baseline for software released right now.
    - afarrell 12y ago
      criticism != shaming
    - michaelmior 12y ago
      A few snippets from the Show HN guidelines: > A Show HN needn't be complicated or look slick. HN users are comfortable with work that's at an early stage. > Be respectful. Anyone sharing work is making a contribution, however modest. > When something isn't good, you needn't pretend that it is. But don't be gratuitously negative. I'm not saying it people shouldn't be open to criticism, but I think terms such as downright shameful fall under the category of gratuitously negative.
pyre 12y ago
> I have neither the experience, nor the inclination for internationalization of software. Taking a piece of software and making all of the UI language localized is one thing. Making sure that your program doesn't blow up if it encounters UTF-8 is another thing. Nowadays if your program chokes on UTF-8, I think it's safe to just consider it broken. In any case, looks like this is really where the issue may lie: # for non-English characters def getRealLengh(str): length = len(str) for s in str: if ord(s) > 256: length += 1 return length and: for val in shn.row_values(n): try: val = val.replace('\n',' ') except: pass val = isinstance(val, basestring) and val.strip() or str(val).strip() line += val + ' ' * (30 - getRealLengh(val)) vim.current.buffer.append(line) In accounting for the fix-width layout of non-ASCII characters.
- bobbyi_settv 12y ago
  Are UTF-8 encoded Excel documents actually common? Do they even exist? I thought Excel used CP 1252 on English Windows and the corresponding code pages on other language versions?
  - pyre 12y ago
    I'm pretty sure that xlrd decodes it all to unicode() in Python, so that should be a moot point. You would only need to worry about passing it as utf-8 to Vim at that point.
  - deleted 12y ago
    [deleted]
  - vertex-four 12y ago
    How would it save a document containing multiple languages, then?
    - sheetjs 12y ago
      Excel 97-2003 (XLS) actually uses UTF16LE in that case, not UTF8. Excel 2007+ XLSB exclusively uses UTF16LE -- there is no way to force it to use a codepage
boomlinde 12y ago
The grandparent probably wrote the comment in his spare time, too. Does that limit your entitlement to voice criticism or suggestions? Don't like it, don't read it.
hrktb 12y ago
Perhaps you could take into account that emoji and other fancy characters are heavy utf8 characters. UTF support doesn't usually mean "prepare for Swahili", but more "don't choke on the characters"
jzawodn 12y ago
Good for you for saying so! I'm always "impressed" by folks who immediately dump on FREE software that doesn't meet their exact needs.
jzawodn 12y ago
Good for you for saying so! I'm always "impressed" by folks who immediately dump on FREE software that doesn't meet their exact needs.
ayrx 12y ago
It goes both ways. I hate developers that write some code, dump it on GitHub and say "It's open source, you can always fork it." Whatever happened to taking pride in your work and making it work the best it can? Like other people have pointed out, handling unicode properly does not mean internationalization. Handling utf-8 isn't even difficult if you just keep it in mind.
- deadweight3 12y ago
  I do take pride in my work, and every piece of software I write handles every use case I need it for explicitly. I take offense that you would imply otherwise knowing absolutely nothing of me and my craft. This isn't about UTF-8, it's about an illogical premise where some very shortsighted individuals would rather have only fully fleshed, fully baked products in open source. This is highly illogical, and would bury ideas potentially unexposed for others to coach and assist with. You aren't the judge of what others find difficult, but that doesn't mean that those that struggle with a concept cannot add tremendous value in other areas. By the way, it's not utf-8, it's UTF-8. Whatever happened to taking pride in your writing and making it the best it can be? Everyone has their own criteria for quality, and you can't hope to satisfy everyone. Everyone with even a mildly successful project in open source knows this. Scratch your own itch, make it work, accept any request that meets with your vision, and keep a permissive licence so those that don't can fork. Otherwise, the arrogance being asserted, that you can somehow determine if my contribution is worth of existing, is baffling.
  - dkersten 12y ago
    I agree with you. Anything else just reeks of over-entitlement. I mean, if someone spends their free time to make something useful, that's great! If it doesn't quite meet your standards, that's your problem and you can either: fork it or send a pull request (ie fix it yourself), or pay someone (original author or someone else) to do it. I'd rather people who build stuff for themselves release it to the rest of us than keep it to themselves.
  - Matumio 12y ago
    You are just being baited. I agree with what you say, but did you really want to burn your energy in this discussion?

rtpg 12y ago

the thing is if you build for unicode support from the start these conversations don't need to be had. The problem is not enough people treat text as a black box from the start (I can understand unwillingness to support bigger things like RTL)

a3n 12y ago
There just isn't a lot of pervasive experience in the development community for multi-language unicode devlopment. Also xlrd is fairly old, although I don't know if that tool is part of what limits this to english. In ten years it might be better.
- sillysaurus3 12y ago
  Joel Spolsky said that ten years ago. The problem is that devs are afraid to learn unicode. They treat it like learning a foreign language. It's not even a fun problem, like learning a new programming language, so nobody makes time for it. The only people who learn it are those who make it a point of pride to implement something correctly and handle corner cases. Unicode isn't even hard: Use UTF-8. Don't try to measure the length of a string unless you're rendering that string and measuring the length in screen units like pixels. If you do those two things, that's 90% of the effort of making Unicode-safe software. I think both views are valid. Those who don't know how to write Unicode-safe software shouldn't feel shamed into learning Unicode before releasing open source work. Those who already know Unicode should feel happy that they're making other people's lives easier.
  - tracker1 12y ago
    But these are file formats that may well not be encoded in UTF-8.. the formats already exist.. it isn't like he's creating a new spreadsheet format here. Some of them may well be encoded to something that works fine against unicode/utf-8, others not so much.
    - sillysaurus3 12y ago
      So you write FooToUTF8() and UTF8ToFoo(), where Foo is whatever the encoding is in the external format. Done. As far as I know, UTF-8 will work 100% of the time, and is almost always the best internal representation for software you write due to how simple and uniform it is. If something is encoded in some other format, you can probably find a conversion function online.
      - tracker1 12y ago
        Okay, so why don't you fork the project, and create your simple Foo/UTF8 methods, and confirm that they are the correct Foo/UTF8 methods for each of the document formats supported. I'm not saying that it's really all that hard, but there are multiple document formats, and versions of those formats. The author obviously didn't need unicode support, so didn't test for it. I'm sure test cases, and a pull request would be welcome.
tracker1 12y ago
Okay... let's go into this... how are the strings in excel encoded anyway? I'd be willing to bet money that at least some of the formats in question aren't UTF-8, they are likely ASCII encoded against a character set or code page. Then you have to read that codepage, and convert the necessary characters to their Unicode equivalents, and from there do you downcode to utf-8? Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible? Who's going to go through the different document versions to confirm, and adjust for the various encodings for non-ascii characters? It's not as simple as saying "don't choke on unicode".
- sheetjs 12y ago
  > how are the strings in excel encoded anyway? Length-prefixed byte arrays encoded using various code pages. There are a small number that excel uses: https://github.com/SheetJS/js-codepage/blob/master/excel.csv https://github.com/SheetJS/js-codepage/blob/master/excel.csv (the columns are CP#, mapping, single/double-byte) > Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible? If we can put together an Apache2-licensed module in JS in an afternoon (https://github.com/SheetJS/js-codepage https://github.com/SheetJS/js-codepage) it can be done in python. > Who's going to go through the different document versions to confirm, and adjust for the various encodings for non-ascii characters? Someone already did that: https://github.com/SheetJS/test_files/tree/master/biff5 https://github.com/SheetJS/test_files/tree/master/biff5 has artifacts for every language type
  - gohrt 12y ago
    > If we can put together an Apache2-licensed module in JS in an afternoon (https://github.com/SheetJS/js-codepage https://github.com/SheetJS/js-codepage) it can be done in python. I thought Python 2 was Unicode-unfriendly. So not as easy as JS.
- thristian 12y ago
  > Does the language this library is written in support that translation? Are there modules to do that? Is the license for those module(s) necessary compatible? It's written in Python, which comes with support for pretty much every major encoding¹ out of the box, so yes. ¹: https://hg.python.org/cpython/file/cb94764bf8be/Lib/encodings https://hg.python.org/cpython/file/cb94764bf8be/Lib/encoding...

izacus 12y ago

It's one thing to internationalize software. That's hard. But not being able to handle UTF-8 in 21st century is downright shameful. The notion of non-ASCII characters in user Excel documents IS NOT something rare even for English speaking nations. There are tons of people with foreign names, addresses and other personal information which is commonly stored in Excel documents. And the funny thing is: it's usually not alot of additional work to support UTF-8 if you START correctly.

michaelmior 12y ago
Calling the work of someone who releases free software that they may have well hacked together in their free time downright shameful seems quite offensive and disregards the work they put into the project. Perhaps the OP had no need to use this code for non-English alphabets or is simply in the early stages and hasn't had the opportunity to fully test and implement this. Regardless, I think people should be applauded for releasing their work instead of shamed. And of course, if it's not a lot of work, a pull request would likely be appreciated ;)
- izacus 12y ago
  Hmm, I do not agree with you that the sole act of opensourcing a software should bring it above criticism. Any developer that takes a bit of pride in his work should at least keep himself to some minimal standards and I think supporting UTF-8 isn't an unreasonable baseline for software released right now.
  - afarrell 12y ago
    criticism != shaming
  - michaelmior 12y ago
    A few snippets from the Show HN guidelines: > A Show HN needn't be complicated or look slick. HN users are comfortable with work that's at an early stage. > Be respectful. Anyone sharing work is making a contribution, however modest. > When something isn't good, you needn't pretend that it is. But don't be gratuitously negative. I'm not saying it people shouldn't be open to criticism, but I think terms such as downright shameful fall under the category of gratuitously negative.

pyre 12y ago

> I have neither the experience, nor the inclination for internationalization of software. Taking a piece of software and making all of the UI language localized is one thing. Making sure that your program doesn't blow up if it encounters UTF-8 is another thing. Nowadays if your program chokes on UTF-8, I think it's safe to just consider it broken. In any case, looks like this is really where the issue may lie: # for non-English characters def getRealLengh(str): length = len(str) for s in str: if ord(s) > 256: length += 1 return length and: for val in shn.row_values(n): try: val = val.replace('\n',' ') except: pass val = isinstance(val, basestring) and val.strip() or str(val).strip() line += val + ' ' * (30 - getRealLengh(val)) vim.current.buffer.append(line) In accounting for the fix-width layout of non-ASCII characters.

bobbyi_settv 12y ago
Are UTF-8 encoded Excel documents actually common? Do they even exist? I thought Excel used CP 1252 on English Windows and the corresponding code pages on other language versions?
- pyre 12y ago
  I'm pretty sure that xlrd decodes it all to unicode() in Python, so that should be a moot point. You would only need to worry about passing it as utf-8 to Vim at that point.
- deleted 12y ago
  [deleted]
- vertex-four 12y ago
  How would it save a document containing multiple languages, then?
  - sheetjs 12y ago
    Excel 97-2003 (XLS) actually uses UTF16LE in that case, not UTF8. Excel 2007+ XLSB exclusively uses UTF16LE -- there is no way to force it to use a codepage

boomlinde 12y ago

The grandparent probably wrote the comment in his spare time, too. Does that limit your entitlement to voice criticism or suggestions? Don't like it, don't read it.

hrktb 12y ago

Perhaps you could take into account that emoji and other fancy characters are heavy utf8 characters. UTF support doesn't usually mean "prepare for Swahili", but more "don't choke on the characters"

jzawodn 12y ago

Good for you for saying so! I'm always "impressed" by folks who immediately dump on FREE software that doesn't meet their exact needs.

jzawodn 12y ago

Good for you for saying so! I'm always "impressed" by folks who immediately dump on FREE software that doesn't meet their exact needs.

ayrx 12y ago

It goes both ways. I hate developers that write some code, dump it on GitHub and say "It's open source, you can always fork it." Whatever happened to taking pride in your work and making it work the best it can? Like other people have pointed out, handling unicode properly does not mean internationalization. Handling utf-8 isn't even difficult if you just keep it in mind.

deadweight3 12y ago
I do take pride in my work, and every piece of software I write handles every use case I need it for explicitly. I take offense that you would imply otherwise knowing absolutely nothing of me and my craft. This isn't about UTF-8, it's about an illogical premise where some very shortsighted individuals would rather have only fully fleshed, fully baked products in open source. This is highly illogical, and would bury ideas potentially unexposed for others to coach and assist with. You aren't the judge of what others find difficult, but that doesn't mean that those that struggle with a concept cannot add tremendous value in other areas. By the way, it's not utf-8, it's UTF-8. Whatever happened to taking pride in your writing and making it the best it can be? Everyone has their own criteria for quality, and you can't hope to satisfy everyone. Everyone with even a mildly successful project in open source knows this. Scratch your own itch, make it work, accept any request that meets with your vision, and keep a permissive licence so those that don't can fork. Otherwise, the arrogance being asserted, that you can somehow determine if my contribution is worth of existing, is baffling.
- dkersten 12y ago
  I agree with you. Anything else just reeks of over-entitlement. I mean, if someone spends their free time to make something useful, that's great! If it doesn't quite meet your standards, that's your problem and you can either: fork it or send a pull request (ie fix it yourself), or pay someone (original author or someone else) to do it. I'd rather people who build stuff for themselves release it to the rest of us than keep it to themselves.
- Matumio 12y ago
  You are just being baited. I agree with what you say, but did you really want to burn your energy in this discussion?