15 ms·

> Simply put, it is nicer to build your systems so that, as much as possible, they use a constant amount of memory irrespective of the input size Really good

> Simply put, it is nicer to build your systems so that, as much as possible, they use a constant amount of memory irrespective of the input size Really good advice - this is a hard earned lesson for many folks. I've worked with quite a few data scientists who were brilliant at experimental design but not necessarily experts in the field of comp sci. Their relatively simple python scripts would run nice and fast initially. As time passed and the organization grew, their scripts would start to run slower and slower as the datasets scaled and swapping to disk started occurring, etc. In some cases they would completely lock up shared machines, taking a good chunk of the team offline for a bit. Anyway, Daniel Lemire's blog is a fantastic resource. I highly recommend taking a look through his publications and open source contributions. I was able to save my employer a lot of money by building on time series compression algorithms [1] and vectorized implementations [2][3] that he has provided. [1] Decoding billions of integers per second through vectorization https://onlinelibrary.wiley.com/doi/full/10.1002/spe.2203 https://onlinelibrary.wiley.com/doi/full/10.1002/spe.2203 [2] https://github.com/lemire/FastPFor https://github.com/lemire/FastPFor [3] https://github.com/searchivarius/PyFastPFor https://github.com/searchivarius/PyFastPFor

alpineidyll3 5y ago
The flip side is that sometimes unnecessary optimizations can slow progress towards the actual goal, especially if the data transformations are sophisticated math.
- Ologn 5y ago
  I have run into this problem more than once. Usually I am not creating the initial data, but am importing it. Usually I have ignored optimizations initially and just loaded and processed the data. Then I go back and hack on the subroutines loading the data and splitting it into records. I was the only programmer working on both projects though. Once I did it with a large RDF (XML) that was a list of books. Initially I loaded and processed it in Perl with XML::Simple, but XML::Twig let me deal with things one record at a time. The other time I had a large Java linked list holding objects, where I grabbed the head, processed it, and then shrunk the list (removeFirst()) as I proceeded (I probably could have optimized the loading even more). As a one-person project, punting on the big load optimization worked for me, as I had a working program even before optimizing. If it was a larger project, what time would be best to optimize might have differed.
- jka 5y ago
  In the context of high-performance, revenue-sensitive data transformations, it'd be interesting to see how reusable free/open-source components would fare, especially with regard to licensing. I can see MIT/permissive licenses being less attractive than usual in an environment like that, because the author might perceive reduced chance of reaping rewards (versus aiding competitors) from publishing components. If improvements led to greater-than-zero reward for publishers and consumers alike, though (as enforced by copyleft approaches), then perhaps the results would be different. All conjecture, really.
kragen 5y ago
> Really good advice - this is a hard-earned lesson for many folks. I've worked with quite a few data scientists who were brilliant at experimental design but not necessarily experts in the field of comp sci. Their relatively simple Python scripts would run nice and fast initially. As time passed and the organization grew, The problem of "as the organization grew,..." is the problem you want to have. 90% of organizations instead have the problem "As we failed to get any interesting results or any funding, and our contributors started drifting away to other projects,..." and sometimes the reason they have that problem is precisely because the initial contributors were too concerned about doing things "the right way" and weren't satisfied with "relatively simple Python scripts [that] run nice and fast initially". When your limiting factor is programmer effort, there are two kinds of systems: the kind that runs as slowly as the users and developers can tolerate, and the kind that people don't use because it lacks the functionality they want, functionality that some other system has. Probably a slower one. Of course you should make things go as fast as you can, and performance can be critical not only for ergonomics but also for the ability to iterate rapidly, and sometimes the performance is the functionality, but when you find yourself facing down the shitty performance of "founder code" that locks up shared machines, don't fall into the trap of wishing that the people had done it right in the first place, before "time passed and the organization grew". In the startup graveyard there are nine other groups who tried to do the same thing, but couldn't quite get it right. Some of them probably had damn fast code, which you aren't looking at, because their organization never grew. And I say that as someone who spent a fair bit of the last week looking at assembly language and thinking about compiler optimization tradeoffs.
austhrow743 5y ago
Do experts in comp sci not write simple python scripts for small datasets?

alpineidyll3 5y ago

The flip side is that sometimes unnecessary optimizations can slow progress towards the actual goal, especially if the data transformations are sophisticated math.

Ologn 5y ago
I have run into this problem more than once. Usually I am not creating the initial data, but am importing it. Usually I have ignored optimizations initially and just loaded and processed the data. Then I go back and hack on the subroutines loading the data and splitting it into records. I was the only programmer working on both projects though. Once I did it with a large RDF (XML) that was a list of books. Initially I loaded and processed it in Perl with XML::Simple, but XML::Twig let me deal with things one record at a time. The other time I had a large Java linked list holding objects, where I grabbed the head, processed it, and then shrunk the list (removeFirst()) as I proceeded (I probably could have optimized the loading even more). As a one-person project, punting on the big load optimization worked for me, as I had a working program even before optimizing. If it was a larger project, what time would be best to optimize might have differed.
jka 5y ago
In the context of high-performance, revenue-sensitive data transformations, it'd be interesting to see how reusable free/open-source components would fare, especially with regard to licensing. I can see MIT/permissive licenses being less attractive than usual in an environment like that, because the author might perceive reduced chance of reaping rewards (versus aiding competitors) from publishing components. If improvements led to greater-than-zero reward for publishers and consumers alike, though (as enforced by copyleft approaches), then perhaps the results would be different. All conjecture, really.

kragen 5y ago

> Really good advice - this is a hard-earned lesson for many folks. I've worked with quite a few data scientists who were brilliant at experimental design but not necessarily experts in the field of comp sci. Their relatively simple Python scripts would run nice and fast initially. As time passed and the organization grew, The problem of "as the organization grew,..." is the problem you want to have. 90% of organizations instead have the problem "As we failed to get any interesting results or any funding, and our contributors started drifting away to other projects,..." and sometimes the reason they have that problem is precisely because the initial contributors were too concerned about doing things "the right way" and weren't satisfied with "relatively simple Python scripts [that] run nice and fast initially". When your limiting factor is programmer effort, there are two kinds of systems: the kind that runs as slowly as the users and developers can tolerate, and the kind that people don't use because it lacks the functionality they want, functionality that some other system has. Probably a slower one. Of course you should make things go as fast as you can, and performance can be critical not only for ergonomics but also for the ability to iterate rapidly, and sometimes the performance is the functionality, but when you find yourself facing down the shitty performance of "founder code" that locks up shared machines, don't fall into the trap of wishing that the people had done it right in the first place, before "time passed and the organization grew". In the startup graveyard there are nine other groups who tried to do the same thing, but couldn't quite get it right. Some of them probably had damn fast code, which you aren't looking at, because their organization never grew. And I say that as someone who spent a fair bit of the last week looking at assembly language and thinking about compiler optimization tradeoffs.

austhrow743 5y ago

Do experts in comp sci not write simple python scripts for small datasets?