3 ms·

I've heard about cloud integration as an issue before, but what is "data engineering"?

• affinepplan 3y ago
  working with databases, streaming data around with stuff like kafka or snowflake, integration with orchestrators like prefect or dagster, robust interfaces for spark, being able to read a directory of parquet files without the GC going insane, this kind of thing I know there are some existing packages that nominally do some of these things, but generally they are understaffed and not fully mature. I love Julia as a language, so I hope this will improve over time. I think it's one of those problems that just requires more adoption before it can be fixed, and things like precompile TTFX improvements in 1.9 are a good way to get that!