6 ms·

I'm a data engineer, it's a fairly new role so it's not well defined yet, but most data engineers write data pipelines to ingest data into a data warehouse and

by Hasnep 3y ago

I'm a data engineer, it's a fairly new role so it's not well defined yet, but most data engineers write data pipelines to ingest data into a data warehouse and then transform it for the business to use. I'm not sure why using a static language would make translating data types difficult, but I add as many typehints as possible to my Python so I rarely do anything with dynamic types. I guess they're saying for small tasks where you're working with lots of types, when using a static language most of your code will be type definitions, so a dynamic language will let you focus on writing the transformation code.

waffletower 3y ago
Using a static language to manipulate complex types, particularly those sourced from a different type system (say complex nested Avro, SQL, or even complex JSON) is much more awkward when the types cannot be normalized into the language automatically as can be done with dynamic languages. Static languages require more a priori knowledge of data types, and are very awkward at handling collections with diverse type membership. Data has many forms in reality -- dynamic languages are much more effective at manipulating data on its own terms.
- chowells 3y ago
  You realize every single thing that dynamically-typed languages can do with data types, statically-typed languages can do too? Except when it matters, they can also choose to do things dynamically-typed languages can't. Lots of people assume static typing means creating domain types for the semantics of every single thing, and then complain that those types contain far more information than they need. Well, stop doing that. Create types that actually contain the information you need. Or use the existing ones. If you're deserializing JSON data, it turns out that the deserialization library already has a type for arbitrary JSON. Just use it, if all you're doing is translating that data to another format. Saying "this data is JSON I didn't bother to understand the internal content of" is a perfectly fine level to work at. Stop inventing work for yourself.
  - belmont_sup 3y ago
    Reminds me of https://lexi-lambda.github.io/blog/2020/01/19/no-dynamic-type-systems-are-not-inherently-more-open/ https://lexi-lambda.github.io/blog/2020/01/19/no-dynamic-typ.... Always a fun revisit.
    - throwaway2037 3y ago
      Excellent blog post. I never saw it before. Thank you to share.
  - bazoom42 3y ago
    You can’t do monkeypatching or dynamically modify the inheritance chain of an object in a statically typed language. But yes, you can have a JsonNode type, which is still better than having every type being “object”.
    - throwaway2037 3y ago
      About monkeypatching, perhaps we have difference definitions. From time to time, I need to modify a Java class from a dependency that I do not own/control. I copy the decompiled class into my project with the same package name. I make changes, then run. To me, this is monkeypatching for Java. Do you agree? If not, how is it different? I would like to learn. Honestly, I discovered that Java technique years ago by accident. Another technique: While the JVM is running with a debugger attached, it is possible to inject a new version of a class. IDEs usually make this seamless. It also works when remote debugging. Do you consider this monkeypatching also?
      - bazoom42 3y ago
        Monkeypatching is programatically adding a method to a class at runtime.
    - chowells 3y ago
      > You can’t do monkeypatching or dynamically modify the inheritance chain of an object in a statically typed language. There's no theoretical reason you can't. No languages that I know of provide that combination features, because monkey-patching is a terrible idea for software engineering... But there's no theoretical reason you couldn't make it happen. I think you've conflated static typing with a static language. They're not the same thing and can be analyzed separately.
- brabel 3y ago
  I love using "mixed" dynamic/static typed languages in these scenarios... you can do that data manipulation without types, but benefit from types everywhere else... my two favourite "mixed" languages are Groovy on the JVM, and Dart elsewhere... Dart now has a very good type system, but still supports `dynamic` which makes it as easy as Python to manipulate data.
topkai22 3y ago
A major problem with doing data transformation in statically typed languages is that its easy to introduce issues during serialization and deserialization. If you have an object [{ name:”ex obj” value:”a sample value” extraProperty:”I’m important” }] And the code class myDTO{ string name; string value; } var myObjs= DerserializeFromFile<myDTO>(filepath) SerializeToFile(myObjs, filePath2) filepath2 would end up with without the extraProperty field. You can also write code like function PrintFullname(person) { WriteLine(person.FirstName + “ “ + person.LastName) } And it will just work so long as the object has those properties. In a statically typed language, you’d have to have a version for each object type or be sure to have a thoughtful common interface between them, which is hard. All that bring said, I generally prefer type safe static languages because that type system has saved my bacon on numerous occasions (it’s great at telling me I just changed something I use elsewhere).
- epgui 3y ago
  I think whether your comment is essentially true or not depends on the language and technology.
- somenameforme 3y ago
  "Most" (I mean "all", but meh - I'm sure there's some obscure exception somewhere) parsers will have the ability to swap between a strict DTO interpretation of some data, and the raw underlying data which is generally going to be something like a map of maps that resolves to strings at the leaf nodes. Both have their uses. The same can also be done easily enough by hand as well, if necessary.
- pharmakom 3y ago
  No, it's not. You can write code in a statically typed language that treats the data as strings. The domain modelling is optional, just choose the level of detail that you need: 1. String 2. JSON 3. MyDTO If you do choose 3, then you can avoid serde errors using property based testing
- FrustratedMonky 3y ago
  Even with a Dynamic Type, it becomes real static type at some point when running, so why wait for the crash to happen at runtime?
throwaway2037 3y ago
Thank you to reply. Your definition of data engineer makes sense. From my experience, I would not call it a new role. People were doing similar things 25 years ago when building the first generation of "data warehouses". (Remember that term from the late 1990s!?) I am surprised that you are using Python for data transformation. Isn't it too slow for huge data sets? (If you are using C/C++ libraries like Pandas/NumPy, then ignore this question.) When I have huge amounts of data, I always want to use something like C/C++/Rust/C#/Java do the heavy lifting because it is so much faster than Python.
- Hasnep 3y ago
  Yes, it's definitely a new word for an old concept, same as the term data scientist for data analyst or statistician. I find Python is fast enough for small to medium datasets. I've normally worked with data that needs to be loaded each morning or sometimes hourly, so whether the transformation takes 1 minute or 10 minutes it doesn't matter. The better way is of course to dump the data into a data warehouse as soon as possible and then use SQL for everything, so I only use Python for things that SQL isn't suited for, like making HTTP requests.

by Hasnep 3y ago

waffletower 3y ago

Using a static language to manipulate complex types, particularly those sourced from a different type system (say complex nested Avro, SQL, or even complex JSON) is much more awkward when the types cannot be normalized into the language automatically as can be done with dynamic languages. Static languages require more a priori knowledge of data types, and are very awkward at handling collections with diverse type membership. Data has many forms in reality -- dynamic languages are much more effective at manipulating data on its own terms.

chowells 3y ago
You realize every single thing that dynamically-typed languages can do with data types, statically-typed languages can do too? Except when it matters, they can also choose to do things dynamically-typed languages can't. Lots of people assume static typing means creating domain types for the semantics of every single thing, and then complain that those types contain far more information than they need. Well, stop doing that. Create types that actually contain the information you need. Or use the existing ones. If you're deserializing JSON data, it turns out that the deserialization library already has a type for arbitrary JSON. Just use it, if all you're doing is translating that data to another format. Saying "this data is JSON I didn't bother to understand the internal content of" is a perfectly fine level to work at. Stop inventing work for yourself.
- belmont_sup 3y ago
  Reminds me of https://lexi-lambda.github.io/blog/2020/01/19/no-dynamic-type-systems-are-not-inherently-more-open/ https://lexi-lambda.github.io/blog/2020/01/19/no-dynamic-typ.... Always a fun revisit.
  - throwaway2037 3y ago
    Excellent blog post. I never saw it before. Thank you to share.
- bazoom42 3y ago
  You can’t do monkeypatching or dynamically modify the inheritance chain of an object in a statically typed language. But yes, you can have a JsonNode type, which is still better than having every type being “object”.
  - throwaway2037 3y ago
    About monkeypatching, perhaps we have difference definitions. From time to time, I need to modify a Java class from a dependency that I do not own/control. I copy the decompiled class into my project with the same package name. I make changes, then run. To me, this is monkeypatching for Java. Do you agree? If not, how is it different? I would like to learn. Honestly, I discovered that Java technique years ago by accident. Another technique: While the JVM is running with a debugger attached, it is possible to inject a new version of a class. IDEs usually make this seamless. It also works when remote debugging. Do you consider this monkeypatching also?
    - bazoom42 3y ago
      Monkeypatching is programatically adding a method to a class at runtime.
  - chowells 3y ago
    > You can’t do monkeypatching or dynamically modify the inheritance chain of an object in a statically typed language. There's no theoretical reason you can't. No languages that I know of provide that combination features, because monkey-patching is a terrible idea for software engineering... But there's no theoretical reason you couldn't make it happen. I think you've conflated static typing with a static language. They're not the same thing and can be analyzed separately.
brabel 3y ago
I love using "mixed" dynamic/static typed languages in these scenarios... you can do that data manipulation without types, but benefit from types everywhere else... my two favourite "mixed" languages are Groovy on the JVM, and Dart elsewhere... Dart now has a very good type system, but still supports `dynamic` which makes it as easy as Python to manipulate data.

topkai22 3y ago

A major problem with doing data transformation in statically typed languages is that its easy to introduce issues during serialization and deserialization. If you have an object [{ name:”ex obj” value:”a sample value” extraProperty:”I’m important” }] And the code class myDTO{ string name; string value; } var myObjs= DerserializeFromFile<myDTO>(filepath) SerializeToFile(myObjs, filePath2) filepath2 would end up with without the extraProperty field. You can also write code like function PrintFullname(person) { WriteLine(person.FirstName + “ “ + person.LastName) } And it will just work so long as the object has those properties. In a statically typed language, you’d have to have a version for each object type or be sure to have a thoughtful common interface between them, which is hard. All that bring said, I generally prefer type safe static languages because that type system has saved my bacon on numerous occasions (it’s great at telling me I just changed something I use elsewhere).

epgui 3y ago
I think whether your comment is essentially true or not depends on the language and technology.
somenameforme 3y ago
"Most" (I mean "all", but meh - I'm sure there's some obscure exception somewhere) parsers will have the ability to swap between a strict DTO interpretation of some data, and the raw underlying data which is generally going to be something like a map of maps that resolves to strings at the leaf nodes. Both have their uses. The same can also be done easily enough by hand as well, if necessary.
pharmakom 3y ago
No, it's not. You can write code in a statically typed language that treats the data as strings. The domain modelling is optional, just choose the level of detail that you need: 1. String 2. JSON 3. MyDTO If you do choose 3, then you can avoid serde errors using property based testing
FrustratedMonky 3y ago
Even with a Dynamic Type, it becomes real static type at some point when running, so why wait for the crash to happen at runtime?

throwaway2037 3y ago

Thank you to reply. Your definition of data engineer makes sense. From my experience, I would not call it a new role. People were doing similar things 25 years ago when building the first generation of "data warehouses". (Remember that term from the late 1990s!?) I am surprised that you are using Python for data transformation. Isn't it too slow for huge data sets? (If you are using C/C++ libraries like Pandas/NumPy, then ignore this question.) When I have huge amounts of data, I always want to use something like C/C++/Rust/C#/Java do the heavy lifting because it is so much faster than Python.

Hasnep 3y ago
Yes, it's definitely a new word for an old concept, same as the term data scientist for data analyst or statistician. I find Python is fast enough for small to medium datasets. I've normally worked with data that needs to be loaded each morning or sometimes hourly, so whether the transformation takes 1 minute or 10 minutes it doesn't matter. The better way is of course to dump the data into a data warehouse as soon as possible and then use SQL for everything, so I only use Python for things that SQL isn't suited for, like making HTTP requests.