3 ms·

This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be e

This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be efficient if type inference had perfect knowledge of the contents of a DataFrame. There is a lot of work being done to develop a new API that has none of these problems and provides very high performance, which demonstrates that Julia's JIT is up to the task so long as you choose an appropriate API. Julia, like any language, has intrinsic limitations, but this is not a good example: it's an example of how good API design for Julia differs from good API design for other systems.

staticassertion 10y ago
> This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be efficient if type inference had perfect knowledge of the contents of a DataFrame. Sorry if I implied otherwise - what I meant is that the current DF structure is hard for the JIT to optimize, not that this is a fundamental limitation of Julia. Just that JIT's are limited. I shouldn't have said Julia is only a small step forward, I think. A JIT gets you a long way. > There is a lot of work being done to develop a new API that has none of these problems and provides very high performance, which demonstrates that Julia's JIT is up to the task so long as you choose an appropriate API. Can you link to information on this? I'd love to see information on the design around this.
- mandelken 10y ago
  >Can you link to information on this? This blog post is a good starting point on the current shortcomings and design solutions of dataframes in Julia: http://julialang.org/blog/2016/10/StructuredQueries http://julialang.org/blog/2016/10/StructuredQueries
  - staticassertion 10y ago
    Thanks a lot.
- elcritch 10y ago
  JIT's can only do so much by themselves, but the addition of generated functions in Julia gives the ability for library developers to modify generated code just before execution. It's almost like having a fully programmable compiler chain with a fraction of the work or effort. Here's to hoping it'll be put to good usage!

staticassertion 10y ago

> This is not quite accurate as stated: the problem is the combination of a data structure that is not amenable to type inference with an API that would only be efficient if type inference had perfect knowledge of the contents of a DataFrame. Sorry if I implied otherwise - what I meant is that the current DF structure is hard for the JIT to optimize, not that this is a fundamental limitation of Julia. Just that JIT's are limited. I shouldn't have said Julia is only a small step forward, I think. A JIT gets you a long way. > There is a lot of work being done to develop a new API that has none of these problems and provides very high performance, which demonstrates that Julia's JIT is up to the task so long as you choose an appropriate API. Can you link to information on this? I'd love to see information on the design around this.

mandelken 10y ago
>Can you link to information on this? This blog post is a good starting point on the current shortcomings and design solutions of dataframes in Julia: http://julialang.org/blog/2016/10/StructuredQueries http://julialang.org/blog/2016/10/StructuredQueries
- staticassertion 10y ago
  Thanks a lot.
elcritch 10y ago
JIT's can only do so much by themselves, but the addition of generated functions in Julia gives the ability for library developers to modify generated code just before execution. It's almost like having a fully programmable compiler chain with a fraction of the work or effort. Here's to hoping it'll be put to good usage!