5 ms·

Fast Lua Serialization (2023)

87 points by synergy20 2y ago

CapsAdmin 2y ago
These days, LuaJIT 2.1 also comes with a highly specialized serializer. https://luajit.org/ext_buffer.html#serialize https://luajit.org/ext_buffer.html#serialize
- pheon 2y ago
  luajit buffer serialization is amazing fast and also surprisingly very space efficient. this should really be at the top of the thread.
sitkack 2y ago
Kinda ironic that the article doesn't discuss it, but Lua was originally a data loading and simulation driving language for Fortran. Lua was the serialization format. Lua itself can load millions of records per second from disk.
- Something1234 2y ago
  I'm going to need to hear more about it because this is absolutely wild.
  - fanf2 2y ago
    https://www.lua.org/doc/hopl.pdf https://www.lua.org/doc/hopl.pdf
    - wahern 2y ago
      Good link. But for those who won't bother reading[1], that paper doesn't support the Fortran assertion. Fortran is only mentioned twice in the HOPL conference paper, once regarding syntax familiarity (lack of semicolons), and again in the context of FFI. In neither case is Fortran mentioned in a privileged manner, but rather alongside multiple other languages to highlight non-C, non-C++-specific considerations and motivations. I know there are Fortran shops that make profitable use of Lua as both an embedded extension language and as a glue language. But Fortran users do so in the same way Lua is used with C, C++, and other languages. That merely reflects that the Lua creators had a good understanding of real-world software ecosystems, and successful in how they shaped the design and implementation to minimize impedance mismatches between Lua and other environments, across syntax, semantics, API, and various implementation details that are invariably ignored or overlooked in comparable languages. [1] For those who like reading, see https://www.lua.org/docs.html#papers https://www.lua.org/docs.html#papers
  - bch 2y ago
    I second this (and see the link in response). Cliché, I’m sorry, but I’ll be looking for reasoning in TFA about 1-based versus 0-based indexing. Better for clerks, w/o programming in mind?
- riidom 2y ago
  What would be the source for that first sentence of yours?
- plorkyeran 2y ago
  Lua was not originally a serialization format. It was a data entry format which was written by hand which then generated the files fed into simulations. Load speed was very important, but they weren't generating Lua. It has been used as a serialization format (for example, World of Warcraft saves UI state by generating Lua files), but that came along much later. https://www.lua.org/history.html https://www.lua.org/history.html covers the early history.
orf 2y ago
> It correctly handles the strange array/map duality of lua tables in the most efficient way it can do safely (serializing both variants concurrently and only writing out the correct one at the end) The article doesn’t expand on this - what are they referring to? Are lua tables arrays? Serializing something common like tables 2x seems like a massive overhead.
- copx 2y ago
  A Lua table is data structure which contains both an array and a hash map part i.e. it accepts both indexed and keyed entries. E.g.: myTable[1] = "red" myTable.name = "Color Codes"
  - Dylan16807 2y ago
    For the actual language, there are no "parts". Every table is a map, and the implementation could do several things behind the scenes. The issue is not knowing if the table can be encoded as an array, since that's more compact in many encodings.
- MattJ100 2y ago
  Lua doesn't have a separate "array" type, as arrays are just a special case of key->value where the key is always a positive integer. While this is nice (it simplifies the language yet Lua tables are incredibly versatile), it does indeed make it a little more work to translate structures to languages/serializations that distinguish between the two types, if the serializer needs to guess at runtime (e.g. if you don't have a predefined schema). I can't comment on the efficiency of whatever this implementation is doing, I haven't read the code. It does sound a little expensive. Also it's not always 100% clear which the correct mapping is, unless the developer explicitly picks one. For example, a serializer has no way to determine whether '{}' is intended to be an empty array or an empty hash. Another common gotcha, e.g. when translating to JSON, is that Lua does not restrict the types of keys (while JSON objects only support string keys), so boolean true/false are possible keys, for example, which cannot be translated to JSON directly.
  - 2y ago
deleted 2y ago
[deleted]
turtledragonfly 2y ago
Lua has its own built-in low-level serialization, in the form of string.pack() and string.unpack()[1]. This is akin to Perl's pack()[2]. But note that this is a tool used for laying out bytes in a certain order (like a C struct), not a general "serialization framework" for traversing data structures and whatnot. But it can be the building block for one. [1] https://www.lua.org/manual/5.4/manual.html#6.4.2 https://www.lua.org/manual/5.4/manual.html#6.4.2 [2] https://perldoc.perl.org/functions/pack https://perldoc.perl.org/functions/pack
dottrap 2y ago
As another extremely simple idea, in my personal experience, just using regular Lua table syntax for serialization and then pre-compiling it with luac so it could be loaded quickly via dofile(), produced results just as fast for loading it in as using lua-protobuf. (I don't remember the difference between writing out Lua tables vs. lua-protobuf because my use case needed to read the data more often than generate it, but it must have not been big enough, if any, otherwise I would probably remember it.) I was loading gigabytes of data for large batch processing.
- csears 2y ago
  Would that be safe even with untrusted data? (Assuming you were the one who serialized and compiled the table)
  - Dylan16807 2y ago
    Are you asking if there are special numbers or strings or some other kind of plain data that would glitch out the interpreter when they're read back in? That would be an impressive failure of a programming language. Yes it's safe as long as you're serializing correctly (which isn't very hard).
    - wahern 2y ago
      By definition you can't trust that "untrusted data" has been serialized correctly. Lua has strong sandboxing capabilities (i.e. ability to limit and control the environment visible from a chunk of code), but the Lua authors years ago explicitly disclaimed the ability to sandbox untrusted code. The compiler and runtime are not bug free. They don't have the resources, notwithstanding that compared to any other non-formally verified implementation their track record is pretty decent, even compared to past and current efforts from Sun, Microsoft, Mozilla, and Google. If you want to run untrusted Lua code, Lua sandboxing should be just the first of multiple line of defense, just as modern web browsers rely on various operating system mechanisms to constrain breakouts.
      - 2y ago
mancporkies88 2y ago
[flagged]
itsthecourier 2y ago
I used to use msgpack when it was schemaful. In a time when Avro, protob, thrift and msgpack were competing against each other and minimum message size was the benchmark New versions are schemaless and I simply stayed in the old one. 10 years using it happily Had to tweak the java version for managing strings over 32 bytes though, it was failing from flutter now
- ahlCVA 2y ago
  As a connoisseur of serialization formats, this is intriguing - I wasn't aware there was a schemaful version of msgpack once, and an ad-hoc search doesn't reveal any leads. Can you point me to some information on how that older format worked?
spc476 2y ago
As the author of org.conman.cbor, I'd be interested in seeing the test data. The only bit that's in C is the low-level bit packing of CBOR values. I might have to revisit the use of LPEG to validate the UTF-8 strings (used because Lua 5.1/5.2 do not have UTF-8 support routines). This also supports references, which are nice (they can allow one to encode a Lua table like "x = {} x[1] = x" (a self-referential table)) that might slow things down as well Also included in the package is a striped down version of my CBOR encoder that doesn't support the CBOR tagging that's about 1/3 the size, so I wonder how that compares against lua_cbor.

87 points by synergy20 2y ago

These days, LuaJIT 2.1 also comes with a highly specialized serializer. https://luajit.org/ext_buffer.html#serialize https://luajit.org/ext_buffer.html#serialize

pheon 2y ago
luajit buffer serialization is amazing fast and also surprisingly very space efficient. this should really be at the top of the thread.

sitkack 2y ago

Kinda ironic that the article doesn't discuss it, but Lua was originally a data loading and simulation driving language for Fortran. Lua was the serialization format. Lua itself can load millions of records per second from disk.

Something1234 2y ago
I'm going to need to hear more about it because this is absolutely wild.
- fanf2 2y ago
  https://www.lua.org/doc/hopl.pdf https://www.lua.org/doc/hopl.pdf
  - wahern 2y ago
    Good link. But for those who won't bother reading[1], that paper doesn't support the Fortran assertion. Fortran is only mentioned twice in the HOPL conference paper, once regarding syntax familiarity (lack of semicolons), and again in the context of FFI. In neither case is Fortran mentioned in a privileged manner, but rather alongside multiple other languages to highlight non-C, non-C++-specific considerations and motivations. I know there are Fortran shops that make profitable use of Lua as both an embedded extension language and as a glue language. But Fortran users do so in the same way Lua is used with C, C++, and other languages. That merely reflects that the Lua creators had a good understanding of real-world software ecosystems, and successful in how they shaped the design and implementation to minimize impedance mismatches between Lua and other environments, across syntax, semantics, API, and various implementation details that are invariably ignored or overlooked in comparable languages. [1] For those who like reading, see https://www.lua.org/docs.html#papers https://www.lua.org/docs.html#papers
- bch 2y ago
  I second this (and see the link in response). Cliché, I’m sorry, but I’ll be looking for reasoning in TFA about 1-based versus 0-based indexing. Better for clerks, w/o programming in mind?
riidom 2y ago
What would be the source for that first sentence of yours?
plorkyeran 2y ago
Lua was not originally a serialization format. It was a data entry format which was written by hand which then generated the files fed into simulations. Load speed was very important, but they weren't generating Lua. It has been used as a serialization format (for example, World of Warcraft saves UI state by generating Lua files), but that came along much later. https://www.lua.org/history.html https://www.lua.org/history.html covers the early history.

orf 2y ago

> It correctly handles the strange array/map duality of lua tables in the most efficient way it can do safely (serializing both variants concurrently and only writing out the correct one at the end) The article doesn’t expand on this - what are they referring to? Are lua tables arrays? Serializing something common like tables 2x seems like a massive overhead.

copx 2y ago
A Lua table is data structure which contains both an array and a hash map part i.e. it accepts both indexed and keyed entries. E.g.: myTable[1] = "red" myTable.name = "Color Codes"
- Dylan16807 2y ago
  For the actual language, there are no "parts". Every table is a map, and the implementation could do several things behind the scenes. The issue is not knowing if the table can be encoded as an array, since that's more compact in many encodings.
MattJ100 2y ago
Lua doesn't have a separate "array" type, as arrays are just a special case of key->value where the key is always a positive integer. While this is nice (it simplifies the language yet Lua tables are incredibly versatile), it does indeed make it a little more work to translate structures to languages/serializations that distinguish between the two types, if the serializer needs to guess at runtime (e.g. if you don't have a predefined schema). I can't comment on the efficiency of whatever this implementation is doing, I haven't read the code. It does sound a little expensive. Also it's not always 100% clear which the correct mapping is, unless the developer explicitly picks one. For example, a serializer has no way to determine whether '{}' is intended to be an empty array or an empty hash. Another common gotcha, e.g. when translating to JSON, is that Lua does not restrict the types of keys (while JSON objects only support string keys), so boolean true/false are possible keys, for example, which cannot be translated to JSON directly.
- 2y ago

deleted 2y ago

[deleted]

turtledragonfly 2y ago

Lua has its own built-in low-level serialization, in the form of string.pack() and string.unpack()[1]. This is akin to Perl's pack()[2]. But note that this is a tool used for laying out bytes in a certain order (like a C struct), not a general "serialization framework" for traversing data structures and whatnot. But it can be the building block for one. [1] https://www.lua.org/manual/5.4/manual.html#6.4.2 https://www.lua.org/manual/5.4/manual.html#6.4.2 [2] https://perldoc.perl.org/functions/pack https://perldoc.perl.org/functions/pack

dottrap 2y ago

As another extremely simple idea, in my personal experience, just using regular Lua table syntax for serialization and then pre-compiling it with luac so it could be loaded quickly via dofile(), produced results just as fast for loading it in as using lua-protobuf. (I don't remember the difference between writing out Lua tables vs. lua-protobuf because my use case needed to read the data more often than generate it, but it must have not been big enough, if any, otherwise I would probably remember it.) I was loading gigabytes of data for large batch processing.

csears 2y ago
Would that be safe even with untrusted data? (Assuming you were the one who serialized and compiled the table)
- Dylan16807 2y ago
  Are you asking if there are special numbers or strings or some other kind of plain data that would glitch out the interpreter when they're read back in? That would be an impressive failure of a programming language. Yes it's safe as long as you're serializing correctly (which isn't very hard).
  - wahern 2y ago
    By definition you can't trust that "untrusted data" has been serialized correctly. Lua has strong sandboxing capabilities (i.e. ability to limit and control the environment visible from a chunk of code), but the Lua authors years ago explicitly disclaimed the ability to sandbox untrusted code. The compiler and runtime are not bug free. They don't have the resources, notwithstanding that compared to any other non-formally verified implementation their track record is pretty decent, even compared to past and current efforts from Sun, Microsoft, Mozilla, and Google. If you want to run untrusted Lua code, Lua sandboxing should be just the first of multiple line of defense, just as modern web browsers rely on various operating system mechanisms to constrain breakouts.
    - 2y ago

mancporkies88 2y ago

[flagged]

itsthecourier 2y ago

I used to use msgpack when it was schemaful. In a time when Avro, protob, thrift and msgpack were competing against each other and minimum message size was the benchmark New versions are schemaless and I simply stayed in the old one. 10 years using it happily Had to tweak the java version for managing strings over 32 bytes though, it was failing from flutter now

ahlCVA 2y ago
As a connoisseur of serialization formats, this is intriguing - I wasn't aware there was a schemaful version of msgpack once, and an ad-hoc search doesn't reveal any leads. Can you point me to some information on how that older format worked?

spc476 2y ago

As the author of org.conman.cbor, I'd be interested in seeing the test data. The only bit that's in C is the low-level bit packing of CBOR values. I might have to revisit the use of LPEG to validate the UTF-8 strings (used because Lua 5.1/5.2 do not have UTF-8 support routines). This also supports references, which are nice (they can allow one to encode a Lua table like "x = {} x[1] = x" (a self-referential table)) that might slow things down as well Also included in the package is a striped down version of my CBOR encoder that doesn't support the CBOR tagging that's about 1/3 the size, so I wonder how that compares against lua_cbor.