2 ms·
My unpopular opinion is that pandas specifically is terrible for data engineering. The type inference when loading data from files can result in fragile jobs. I
by agent281 3y ago
My unpopular opinion is that pandas specifically is terrible for data engineering. The type inference when loading data from files can result in fragile jobs. In my experience, it's also surprisingly slow for a lot of data engineering tasks. IMO, it should just be used for analysis.
Python's parallelism story also holds it back, but that may be improving soon. It's unlikely it will be as nice as Go's though.
- isoprophlex 3y agoIs that unpopular? I rarely meet people who actually enjoy wading through the baroque, underperforming forest of footguns that is the pandas API.
- agent281 3y agoWhen I would interview data eng candidates, a large subset would import pandas before reading the problem statement. When I've complained about it, a lot of people are surprised to find a data engineer who doesn't like it. That said, it might just be the companies I've worked at. ¯ \ _ ( ツ ) _ / ¯
- isoprophlex 3y agoKeep on educating them I'd say ;)