4 ms·
Agree 99% except this statement: > Embrace duplication - don't unnecessary add loops or abstractions I’ll usually make a function or perhaps tiny class as soo
by bogeholm 3y ago
Agree 99% except this statement:
> Embrace duplication - don't unnecessary add loops or abstractions
I’ll usually make a function or perhaps tiny class as soon as I start reusing bits of code.
Apart from that, agree as stated. At my previous job (Python shop), a lot of the data engineers came from a Java background, and had a tendency to think top-down. Many things were over engineered ‘just because we might need it’:
- Factory classes used only once or twice in entire code base
- Lets make an AbstractReaderInterface because we might want to abstract the file type or location later (while 100% of files are Parquet on S3)
I’ve really enjoyed using dataclasses and Pydantics BaseModels prolifically, and adding type hints (coupled with type checks in CI).
Model the data, write a well structured imperative workflow, set up CI, write unit tests, enforce typing. Add OOP if needed, then close ticket.
- robertlagrant 3y agoAgreed. The general principle is (and it's a hard balance, of course, to avoid just slapdash work): don't do work up front you might not need. You will make your current work take longer than it should, which you will (correctly) be blamed for, and any time you save in the future you won't get the credit for. This isn't a cynical statement; it's just my experience. Use it to your advantage!