7 ms·
It does, of course, but serialization is problematic (with the combination of Type inference and dynamicism this is maybe to be expected). Beyond that there is
by Certhas 7y ago
It does, of course, but serialization is problematic (with the combination of Type inference and dynamicism this is maybe to be expected).
Beyond that there is a bewildering array of packages, and not really a clear consensus for which to use:
https://discourse.julialang.org/t/what-is-the-preferred-way-to-save-variables/21918 https://discourse.julialang.org/t/what-is-the-preferred-way-...
JLD2, which for a while seemed like an emerging consensus option has caused massive dataloss for a number of people:
https://github.com/JuliaIO/JLD2.jl/issues/55 https://github.com/JuliaIO/JLD2.jl/issues/55
The issue is also that Julia packages often use their own container types for convenient analysis. For example the solution type of DiffEq has a number of important features like interpolation built in. If I want to reliably save the data I got from the numerical integration, and don't want to rely on brittle/broken serialization packages, I have to extract the underlying arrays though. That would be fine if not for the fact that there is no documented way (that I've found) to initialize the solution objects from this raw data, so now I have lost access to the convenience functions.
Don't get me wrong, this is all growing pains. I am sure this will be sorted out eventually. But it's important to point out how the intersection of features that enable Julia to do things that are unheard of in other languages (e.g. DiffEqFlux, Zygote) also mean there are some things that still need to be figured out for this language (serialization, useful error messages, interfaces).
- ChrisRackauckas 7y agoJLD2 works with the DiffEq solution and as of last week has a regression test so it will be known if call-overloaded types have an issue in the future. Now that there is less code churn, I suspect this to continue to be supported (the break came from the frantic v1.0 updates).
- Certhas 7y agoThe dataloss issues are not related to this specific issue though. They are purely JLD2 and about the use of mmap which is problematic in hpc contexts. So exactly when you have to save data that is expensive to produce. See the linked issue. The only workaround is to use an undocumented alternative interface for working on the files. A pull request to at least inform people about the issue has been lying around for almost two years. I am happy that activity has picked up again, but don't intend to recommend the use of JLD2 for expensive data for the foreseeable future...