3 ms·

Yes! As a data scientist, an important point of my job is evangelizing the data consumer's point of view. It's easy for data producers (e.g. the developers who

Yes! As a data scientist, an important point of my job is evangelizing the data consumer's point of view. It's easy for data producers (e.g. the developers who write the logging code that generates the data I use) to make decisions that make their lives easier, at the expense of making my life much harder. Only by talking to each other can we find the balance that mutually optimizes our joint effort level. A little more effort up-front to log data in a format with well-defined schemas, translates into huge savings in processing, consuming, and understanding the data down the road.

vog 10y ago
> evangelizing the data consumer's point of view Note that this is a fundamental principle in software development, not just for data science: Prefer data structures that are simple to interpret over those simple to generate. This principle follows from two basic observations: 1. Data is usually produced by one application, but consumed by many different applications. 2. Writing a serializer is almost always easier than writing a parser. [1] However, it is not always clear how to apply this principle. For example, as a library or web service author, you define the API on your own, rather than having your users define it. It is extra effort to get feedback from your library users, and to design your API accordingly. That's why most APIs are geared towards the data producers rather than consumers. There is even an enterprise design pattern for this: https://martinfowler.com/articles/consumerDrivenContracts.html https://martinfowler.com/articles/consumerDrivenContracts.ht... [1] The former usually involves a template system and a few helper functions. The latter usually involves regexes embedded in ugly code, parser generators with all their shortcomings (and the fact that most programmers don't bother to learn using them), complex DOM or JSON traversals that fail to handle all special cases, or masses of autogenerated classes automatically derived from some XML schema (or JSON schema).

vog 10y ago

> evangelizing the data consumer's point of view Note that this is a fundamental principle in software development, not just for data science: Prefer data structures that are simple to interpret over those simple to generate. This principle follows from two basic observations: 1. Data is usually produced by one application, but consumed by many different applications. 2. Writing a serializer is almost always easier than writing a parser. [1] However, it is not always clear how to apply this principle. For example, as a library or web service author, you define the API on your own, rather than having your users define it. It is extra effort to get feedback from your library users, and to design your API accordingly. That's why most APIs are geared towards the data producers rather than consumers. There is even an enterprise design pattern for this: https://martinfowler.com/articles/consumerDrivenContracts.html https://martinfowler.com/articles/consumerDrivenContracts.ht... [1] The former usually involves a template system and a few helper functions. The latter usually involves regexes embedded in ugly code, parser generators with all their shortcomings (and the fact that most programmers don't bother to learn using them), complex DOM or JSON traversals that fail to handle all special cases, or masses of autogenerated classes automatically derived from some XML schema (or JSON schema).