6 ms·

I haven't seen the video [do they discuss this?], but the biggest problem I have with Lua is its lack of good unicode support. The Lua wiki suggests slnunicode

I haven't seen the video [do they discuss this?], but the biggest problem I have with Lua is its lack of good unicode support. The Lua wiki suggests slnunicode and ICU4Lua, but neither library has been updated for a year. I'd love to use Lua instead of Python for some tasks, given that LuaJIT kicks ass. But poor support for something as important as unicode is a deal breaker. Am I just missing some information in this regard?

jacques_chester 16y ago
> Am I just missing some information in this regard? A major, and non-negotiable, design goal of Lua is that it targets the ANSI C environment to maximise portability. ANSI C does not define Unicode support. Incidentally, those two libraries might be unchanged because they're stable. The rate of change in the main lua implementation is quite measured.
- kylecordes 16y ago
  (I'm the guy in the video.) I haven't had a need for Unicode in Lua, but when I stumbled across some information about it, the essential story was that there is nothing stopping you from writing code to manipulate Unicode data (obviously), but there is also nothing built in to help you. All the built in string stuff is ANSI/ASCII/whatever. However, the language Io (which often gets mentioned in the same breath as Lua, though it's quite different in some ways) has a stronger Unicode story: http://www.iolanguage.com/scm/io/docs/IoGuide.html#Unicode http://www.iolanguage.com/scm/io/docs/IoGuide.html#Unicode
  - metamemetics 16y ago
    Are you affiliated with Lambda Lounge in STL?
    - kylecordes 16y ago
      Yes! I know the guy who started Lambda Lounge (Alex Miller http://tech.puredanger.com/about http://tech.puredanger.com/about ) and heartily supported its founding; I spoke at it every once in a while; and my firm (Oasis Digital http://oasisdigital.com/ http://oasisdigital.com/ ) sponsors Lambda Lounge by paying for travel expenses for out-of-town speakers. Lambda Lounge is the best software dev user group ever.
    - puredanger 16y ago
      I'm Alex Miller, the founder of Lambda Lounge. If you want to drop me a line email me at contact at puredanger.com
- benatkin 16y ago
  The Lua philosophy is far different from the philosophy of other languages. That's why videos like these, that cover the advantages before the tradeoffs, are a good thing. Knowing the positive things about something really effects my perceptions of the tradeoffs. Also, Unicode is an odd thing to consider to be an essential part of a programming language. Usually when I hear it I think the person must think it's a part of accepting those that speak other languages. I would point out the fact that Ruby took much longer to get Unicode support than Python as a counterpoint to that, as Ruby's creator, Matz, is Japanese.
  - jacques_chester 16y ago
    > Also, Unicode is an odd thing to consider to be an essential part of a programming language. It really depends on what the language is used for. Web-facing applications are, in general, going to received unicode input, so for them it matters. But for an embedded programmer, for example, it might be utterly irrelevant. Horses for courses, really. It just so happens that Lua has other properties that are very attractive to web apps too, which the presentation outlined. -- and yes, still alive and commenting :P
  - arkx 16y ago
    As a Finnish software dev I can say that in these days native support for Unicode is a must. We have a couple of special characters in the alphabet (åäö) and if you have to do a lot of manual work to use these, the programming language is pretty much unusable for real world stuff that involves any use of Finnish. The fact that Ruby took so long to get real Unicode support is due to Japanese resisting Unicode in favor of their own encodings, EUC-JP and Shift JIS, just like Finnish used to cling to ISO-8859-1(5) for so long.
    - silentbicycle 16y ago
      Is Unicode something that needs to be in the core of a language, or is it sufficient to leave it to libraries, if the language's design doesn't prevent it? On this computer (OpenBSD/i386), icu has over 1 MB of libraries and a 15 MB data file. The whole Lua distribution fits in one 200k library. Bloating the core language with that seems impractical.
- silentbicycle 16y ago
  More specifically, Unicode is one of those things (like bignum support) where Lua's design for embedding assumes that if it's a priority, your project will already have settled on a library for it, so anything Lua came bundled with would just get in the way. The GMP library, for example, is almost four times larger than the whole Lua runtime. Besides, Lua strings are interned raw byte-arrays (like atoms in Erlang or Lisp atoms), and can have \0s and other arbitrary binary data. While the standard string library is ignorant of Unicode, there's nothing preventing extra libraries from providing Unicode-aware string operations. Lua also left out native regular expression support, because it would be larger than the rest of the standard libraries put together. You can just load LPEG if it matters.
  - indigoviolet 16y ago
    I wouldn't have a problem with external unicode support if there was one "standard" recommended library that had a strong community behind it. My whole point is that that doesn't appear to exist.
    - silentbicycle 16y ago
      Sure. This is a real issue for new Lua users, though in the long run it's not a showstopper. For roughly half the people using Lua, library choices have already been decided by the big C++ (or whatever) project they're adding Lua to, so the question never arises in the first place. That's the primary use case for Lua. It's a really nice standalone language, too, but it was designed for embedding. Many of its strengths (such as the small, orthogonal, clean design) come from that focus, but so do some quirks. If you use Lua as a standalone language, you'll occasionally need to do some digging for libraries, particularly if you're not comfortable with C. The situation has been improving significantly over the last year or two, though. I don't have a specific Unicode library recommendation, but would suggest checking the mailing list archives (http://lua-users.org/lists/lua-l/ http://lua-users.org/lists/lua-l/).
- indigoviolet 16y ago
  slnunicode hasn't changed or been updated, and has no forum posts or bugs since 2006 [http://luaforge.net/projects/sln/ http://luaforge.net/projects/sln/] ICU4Lua hasn't been touched since April 2009, and is at 0.2B. [http://luaforge.net/projects/icu-lua/ http://luaforge.net/projects/icu-lua/] That sort of status just doesn't inspire confidence in a language. I understand you can use it for game programming, or that you can write your own libraries for it, but that sort of thing doesn't encourage adoption.
  - silentbicycle 16y ago
    ICU4Lua is a wrapper for an existing, stable library. Does (for example) the Python wrapper for sqlite get a lot of commits these days? And if not, does that make it dead and abandoned?
    - indigoviolet 16y ago
      Why is it in beta? Why does the mailing list [http://lists.luaforge.net/pipermail/icu-lua-users/ http://lists.luaforge.net/pipermail/icu-lua-users/] have no activity on it?
      - silentbicycle 16y ago
        Unless something is a major standalone project, discussion usually happens on the main Lua mailing list instead. About the beta label - I don't know. gmail was in beta for a long time, too, though.
metamemetics 16y ago
http://luaforge.net/projects/ul-str/ http://luaforge.net/projects/ul-str/ Last updated June 21st 2010. Readme seems to indicate it supports all the important stuff like compare, concatenate, encode, lower<->upper, Regexp, url escaping. Says beta and I haven't tried it yet tho : /
- indigoviolet 16y ago
  Thanks, that's useful to hear. It isn't on the wiki page: http://lua-users.org/wiki/LuaUnicode http://lua-users.org/wiki/LuaUnicode (which hasn't been touched since Sep 2009).
  - metamemetics 16y ago
    http://luaforge.net http://luaforge.net luaforge seems to still be active and doing a pretty good job aggregating libraries. Check out the "luarocks" and "kepler" project pages for ideas. Or just grab some mysql bindings and start from scratch. Let HNers know if you build a lua blog\cms etc... it's been on my todo list but probably won't get to it until December.
SeanLuke 16y ago
> I haven't seen the video [do they discuss this?], but the biggest problem I have with Lua is its lack of good unicode support. Hear hear. I too don't care much for the argument that Lua doesn't have unicode because C doesn't have unicode. For two reasons. First of all, Lua can be straightforwardly embedded into a host of other languages (for example, http://tinyurl.com/39qhxzw http://tinyurl.com/39qhxzw). Second, who in their right mind thought that a language designed for embedding into commercial applications shouldn't have good I18N? (And honestly, how in the world could someone from Brazil develop a programming language that doesn't even support their own spoken language? It's as if Ruby was developed by someone in Japan.) A bit of history. From what I have gathered in its documentation, Lua steals quite unabashedly from NewtonScript. NewtonScript was a proto-style OO language developed in a hurry for the Newton when the Dylan language wasn't going to deliver. Like Lua, it's a language designed originally for an interpreter, which runs embedded in an outer C++ environment and must interoperate with it. But NewtonScript doesn't just use Unicode pervasively: it was the first major language to do so. And C++ interoperability with the language is just fine. So this Lua-needs-to-work-with-C-and-thus-can't-do-unicode thing is nonsense both in fact and precedent. And while we're on the subject of NewtonScript: Lua's let's-almost-do-proto-style-OO-but-require-the-user-to-do-extra-work is really incredibly annoying. Lua should have had proto OO built into the language, like NewtonScript, and not just "available", in a hacked way, through its meta model. Lua wants to be more "general" than NewtonScript, but it just winds up being (IMO) rather less usable.
- silentbicycle 16y ago
  Lua is as old as NewtonScript. Its authors have mentioned Scheme, SNOBOL, awk, bibtex, Icon, and (IIRC) Self as influences, but I don't recall them mentioning NewtonScript. Javascript also has a lot in common with Lua. I think it's because the trade-offs inherent in embedded scripting languages are making them converge on a similar overall language design, rather than plagiarism. If you're putting Lua in a commercial application, the commercial application itself will provide the i18n, and Lua can use it with very little trouble. Lua's strings are raw byte arrays - you can load arbitrary binary data in them. A library that reads UTF-8 strings (say) can work with them just fine. (And while I don't speak Portuguese, there are examples in PiL that use it without problems.) I disagree with you about whether metatables are annoying, but it's a matter of taste. I haven't ever used NewtonScript, but I do use metatables for quite a bit more than just prototype OO - it's easy to turn a table into a proxy + cache to a function, for example. Also, my Lua redis library (http://github.com/silentbicycle/sidereal http://github.com/silentbicycle/sidereal) turns table reads and writes into syntactic sugar for redis db key, list, and set operations.

jacques_chester 16y ago

> Am I just missing some information in this regard? A major, and non-negotiable, design goal of Lua is that it targets the ANSI C environment to maximise portability. ANSI C does not define Unicode support. Incidentally, those two libraries might be unchanged because they're stable. The rate of change in the main lua implementation is quite measured.

kylecordes 16y ago
(I'm the guy in the video.) I haven't had a need for Unicode in Lua, but when I stumbled across some information about it, the essential story was that there is nothing stopping you from writing code to manipulate Unicode data (obviously), but there is also nothing built in to help you. All the built in string stuff is ANSI/ASCII/whatever. However, the language Io (which often gets mentioned in the same breath as Lua, though it's quite different in some ways) has a stronger Unicode story: http://www.iolanguage.com/scm/io/docs/IoGuide.html#Unicode http://www.iolanguage.com/scm/io/docs/IoGuide.html#Unicode
- metamemetics 16y ago
  Are you affiliated with Lambda Lounge in STL?
  - kylecordes 16y ago
    Yes! I know the guy who started Lambda Lounge (Alex Miller http://tech.puredanger.com/about http://tech.puredanger.com/about ) and heartily supported its founding; I spoke at it every once in a while; and my firm (Oasis Digital http://oasisdigital.com/ http://oasisdigital.com/ ) sponsors Lambda Lounge by paying for travel expenses for out-of-town speakers. Lambda Lounge is the best software dev user group ever.
  - puredanger 16y ago
    I'm Alex Miller, the founder of Lambda Lounge. If you want to drop me a line email me at contact at puredanger.com
benatkin 16y ago
The Lua philosophy is far different from the philosophy of other languages. That's why videos like these, that cover the advantages before the tradeoffs, are a good thing. Knowing the positive things about something really effects my perceptions of the tradeoffs. Also, Unicode is an odd thing to consider to be an essential part of a programming language. Usually when I hear it I think the person must think it's a part of accepting those that speak other languages. I would point out the fact that Ruby took much longer to get Unicode support than Python as a counterpoint to that, as Ruby's creator, Matz, is Japanese.
- jacques_chester 16y ago
  > Also, Unicode is an odd thing to consider to be an essential part of a programming language. It really depends on what the language is used for. Web-facing applications are, in general, going to received unicode input, so for them it matters. But for an embedded programmer, for example, it might be utterly irrelevant. Horses for courses, really. It just so happens that Lua has other properties that are very attractive to web apps too, which the presentation outlined. -- and yes, still alive and commenting :P
- arkx 16y ago
  As a Finnish software dev I can say that in these days native support for Unicode is a must. We have a couple of special characters in the alphabet (åäö) and if you have to do a lot of manual work to use these, the programming language is pretty much unusable for real world stuff that involves any use of Finnish. The fact that Ruby took so long to get real Unicode support is due to Japanese resisting Unicode in favor of their own encodings, EUC-JP and Shift JIS, just like Finnish used to cling to ISO-8859-1(5) for so long.
  - silentbicycle 16y ago
    Is Unicode something that needs to be in the core of a language, or is it sufficient to leave it to libraries, if the language's design doesn't prevent it? On this computer (OpenBSD/i386), icu has over 1 MB of libraries and a 15 MB data file. The whole Lua distribution fits in one 200k library. Bloating the core language with that seems impractical.
silentbicycle 16y ago
More specifically, Unicode is one of those things (like bignum support) where Lua's design for embedding assumes that if it's a priority, your project will already have settled on a library for it, so anything Lua came bundled with would just get in the way. The GMP library, for example, is almost four times larger than the whole Lua runtime. Besides, Lua strings are interned raw byte-arrays (like atoms in Erlang or Lisp atoms), and can have \0s and other arbitrary binary data. While the standard string library is ignorant of Unicode, there's nothing preventing extra libraries from providing Unicode-aware string operations. Lua also left out native regular expression support, because it would be larger than the rest of the standard libraries put together. You can just load LPEG if it matters.
- indigoviolet 16y ago
  I wouldn't have a problem with external unicode support if there was one "standard" recommended library that had a strong community behind it. My whole point is that that doesn't appear to exist.
  - silentbicycle 16y ago
    Sure. This is a real issue for new Lua users, though in the long run it's not a showstopper. For roughly half the people using Lua, library choices have already been decided by the big C++ (or whatever) project they're adding Lua to, so the question never arises in the first place. That's the primary use case for Lua. It's a really nice standalone language, too, but it was designed for embedding. Many of its strengths (such as the small, orthogonal, clean design) come from that focus, but so do some quirks. If you use Lua as a standalone language, you'll occasionally need to do some digging for libraries, particularly if you're not comfortable with C. The situation has been improving significantly over the last year or two, though. I don't have a specific Unicode library recommendation, but would suggest checking the mailing list archives (http://lua-users.org/lists/lua-l/ http://lua-users.org/lists/lua-l/).
indigoviolet 16y ago
slnunicode hasn't changed or been updated, and has no forum posts or bugs since 2006 [http://luaforge.net/projects/sln/ http://luaforge.net/projects/sln/] ICU4Lua hasn't been touched since April 2009, and is at 0.2B. [http://luaforge.net/projects/icu-lua/ http://luaforge.net/projects/icu-lua/] That sort of status just doesn't inspire confidence in a language. I understand you can use it for game programming, or that you can write your own libraries for it, but that sort of thing doesn't encourage adoption.
- silentbicycle 16y ago
  ICU4Lua is a wrapper for an existing, stable library. Does (for example) the Python wrapper for sqlite get a lot of commits these days? And if not, does that make it dead and abandoned?
  - indigoviolet 16y ago
    Why is it in beta? Why does the mailing list [http://lists.luaforge.net/pipermail/icu-lua-users/ http://lists.luaforge.net/pipermail/icu-lua-users/] have no activity on it?
    - silentbicycle 16y ago
      Unless something is a major standalone project, discussion usually happens on the main Lua mailing list instead. About the beta label - I don't know. gmail was in beta for a long time, too, though.

metamemetics 16y ago

http://luaforge.net/projects/ul-str/ http://luaforge.net/projects/ul-str/ Last updated June 21st 2010. Readme seems to indicate it supports all the important stuff like compare, concatenate, encode, lower<->upper, Regexp, url escaping. Says beta and I haven't tried it yet tho : /

indigoviolet 16y ago
Thanks, that's useful to hear. It isn't on the wiki page: http://lua-users.org/wiki/LuaUnicode http://lua-users.org/wiki/LuaUnicode (which hasn't been touched since Sep 2009).
- metamemetics 16y ago
  http://luaforge.net http://luaforge.net luaforge seems to still be active and doing a pretty good job aggregating libraries. Check out the "luarocks" and "kepler" project pages for ideas. Or just grab some mysql bindings and start from scratch. Let HNers know if you build a lua blog\cms etc... it's been on my todo list but probably won't get to it until December.

SeanLuke 16y ago

> I haven't seen the video [do they discuss this?], but the biggest problem I have with Lua is its lack of good unicode support. Hear hear. I too don't care much for the argument that Lua doesn't have unicode because C doesn't have unicode. For two reasons. First of all, Lua can be straightforwardly embedded into a host of other languages (for example, http://tinyurl.com/39qhxzw http://tinyurl.com/39qhxzw). Second, who in their right mind thought that a language designed for embedding into commercial applications shouldn't have good I18N? (And honestly, how in the world could someone from Brazil develop a programming language that doesn't even support their own spoken language? It's as if Ruby was developed by someone in Japan.) A bit of history. From what I have gathered in its documentation, Lua steals quite unabashedly from NewtonScript. NewtonScript was a proto-style OO language developed in a hurry for the Newton when the Dylan language wasn't going to deliver. Like Lua, it's a language designed originally for an interpreter, which runs embedded in an outer C++ environment and must interoperate with it. But NewtonScript doesn't just use Unicode pervasively: it was the first major language to do so. And C++ interoperability with the language is just fine. So this Lua-needs-to-work-with-C-and-thus-can't-do-unicode thing is nonsense both in fact and precedent. And while we're on the subject of NewtonScript: Lua's let's-almost-do-proto-style-OO-but-require-the-user-to-do-extra-work is really incredibly annoying. Lua should have had proto OO built into the language, like NewtonScript, and not just "available", in a hacked way, through its meta model. Lua wants to be more "general" than NewtonScript, but it just winds up being (IMO) rather less usable.

silentbicycle 16y ago
Lua is as old as NewtonScript. Its authors have mentioned Scheme, SNOBOL, awk, bibtex, Icon, and (IIRC) Self as influences, but I don't recall them mentioning NewtonScript. Javascript also has a lot in common with Lua. I think it's because the trade-offs inherent in embedded scripting languages are making them converge on a similar overall language design, rather than plagiarism. If you're putting Lua in a commercial application, the commercial application itself will provide the i18n, and Lua can use it with very little trouble. Lua's strings are raw byte arrays - you can load arbitrary binary data in them. A library that reads UTF-8 strings (say) can work with them just fine. (And while I don't speak Portuguese, there are examples in PiL that use it without problems.) I disagree with you about whether metatables are annoying, but it's a matter of taste. I haven't ever used NewtonScript, but I do use metatables for quite a bit more than just prototype OO - it's easy to turn a table into a proxy + cache to a function, for example. Also, my Lua redis library (http://github.com/silentbicycle/sidereal http://github.com/silentbicycle/sidereal) turns table reads and writes into syntactic sugar for redis db key, list, and set operations.