4 ms·
As a data engineer, I sometimes run into problems like this, but I generally find that 99.9% of them can be solved "well enough" in python, or in a database. Mo
by tomlockwood 5y ago
As a data engineer, I sometimes run into problems like this, but I generally find that 99.9% of them can be solved "well enough" in python, or in a database. Moving code from Python to Rust or similar has been required only a handful of times.
This specific task, the next program I'm working on, seems to be a very "algo" based, comp sci type problem. I think most companies want to say that they have these kind of problems, but the reality often is, they need a batch script on a laptop.
So like, yeah, I have done things like this before in my work, but often, solutions in Python are "good enough".
- dvasdekis 5y agoA company I know hired a computer science PhD after putting them through the full programmer interview, including whiteboarding algos, etc. Literally the hardest crap that anyone working there could think up, and then hired the programmer who got the best score on the test, and the process took multiple months to find this magic individual who could score high enough to satisfy everyone. Once hired (at a fantastic salary), she was then put to work copying data from a pile of CDs into a cloud storage account! Not only that, but she was given a laptop with only 1 CD drive to work with. She quit within a week.
- vba616 5y agoI have trouble believing that story. An adult, who is also a super genius just says "I was given..." and quits? That's so passive! You have one CD drive and assume that's it, nothing can be done, that's all the resources in the whole company? Don't talk to anyone?
- sanitycheck 5y agoPerhaps evaluating the situation and leaving immediately shows just how smart they were!
- vba616 5y agoThat would depend on whether the company was doing something that needed to be done, wouldn't it? Which should be clear before a person starts. If something has to be done, then avoiding it makes you "smart" like avoiding taxes makes you "smart".
- tomlockwood 5y agoWonder what company that was haha :)
- vba616 5y ago>I generally find that 99.9% of them can be solved "well enough" in python Sorry, I didn't mean "by translating to a lower level language". In fact, I especially meant by fixing the script in the existing language.
- tomlockwood 5y agoOh yeah, I do that all the time - sometimes the volume of data gets higher and there are bottlenecks and such, you know? Much refactoring to be done.
- vba616 5y agoAs I've seen on HN, a lot of people hate and do not think in SQL, and avoid it by writing something imperative. Replacing a loop with one query is an example of the sort of thing that I've seen make 3 orders of magnitude difference. Another basic pattern is converting a program or script that loads a whole job into memory into something with fixed size buffers, because specific data made the original explode.