itamarst

23 ms·

211.
by itamarst 6y ago
I've seen that, yeah. I've already implemented a memory profiler for Python batch jobs ( https://pythonspeed.com/fil ), but starting to think about how to integrate it into specific pipeline frameworks.
212.
by itamarst 6y ago
Hey, been meaning to reach out. There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, nev
213.
CI for performance: Reliable benchmarking in noisy environments(pythonspeed.com)
1 points by itamarst 6y ago|0 comments
214.
Reproducible and upgradable Conda environments with conda-lock(pythonspeed.com)
2 points by itamarst 6y ago|1 comments
215.
by itamarst 6y ago
I can't imagine that has any significant impact on memory usage, at least for normal multi-GB-RAM servers: the docstrings will only be loaded once.
216.
by itamarst 6y ago
Memory usage is an interesting problem, because the failure mode is so much more painful (crashes and complete lock-ups) than high CPU usage. And even if you return memory to the OS, that doesn't actually solve too-high memory usage. S
217.
Performance engineering requires stable benchmarks(buttondown.email)
3 points by itamarst 6y ago|0 comments
218.
Optimizing your code is not the same as parallelizing your code(pythonspeed.com)
1 points by itamarst 6y ago|0 comments
219.
Poetry vs. Docker Caching: Fight(pythonspeed.com)
1 points by itamarst 6y ago|1 comments
220.
by itamarst 6y ago
It's a tradeoff, yes. You can do more work to rebuild a base image, or you can say "technically have slightly newer version of glibc isn't reproducible but in practice I don't expect that to break anything so I'll l
221.
by itamarst 6y ago
1. "Don't update system packages" is bad advice. There are base Docker images with out-of-date packages that need security updates. CentOS for example doesn't update their base image for months on end. 2. Given you do wa
222.
When to switch to Python 3.9(pythonspeed.com)
1 points by itamarst 6y ago|0 comments
223.
A circular buffer using a mmap() magic trick to make the caller's life easier(github.com)
3 points by itamarst 6y ago|0 comments
224.
From YAGNI to YDNIY(codewithoutrules.com)
3 points by itamarst 6y ago|0 comments
225.
The mmap() copy-on-write trick: reducing memory usage of array copies(pythonspeed.com)
1 points by itamarst 6y ago|0 comments
226.
by itamarst 6y ago
Yes. But the Alpine C library is musl, not glibc, and musl can sometimes be incompatible with code compiled against glibc, so I don't like using it.
227.
Shrink your Conda Docker images with conda-pack(pythonspeed.com)
5 points by itamarst 6y ago|6 comments
228.
by itamarst 6y ago
As other people have said, you can negotiate for this. I've done it at multiple jobs. Here's an interview with someone who has been doing it for 15 years: https://codewithoutrules.com/2018/01/08/part
229.
Estimating and modeling memory requirements for data processing(pythonspeed.com)
1 points by itamarst 6y ago|0 comments
230.
by itamarst 6y ago
Debian Stable currently ships Python 3.7. If that's what you want, great. But if you want Python 3.8, the soon-to-be-released Python 3.9, or other versions, you don't have that.
231.
by itamarst 6y ago
There isn't too much of a downside, except that you need to be a little more careful in your CI, otherwise you can end up rebuilding from scratch each time, thus losing the benefit of faster builds. See here for why and how to fix it:
232.
by itamarst 6y ago
Beyond the other two answers, I added this bit because I keep seeing people make this mistake in their StackOverflow questions about why their Dockerfile isn't working.
233.
by itamarst 6y ago
For image size, it doesn't matter. It does matter for build time. Specifically, multi-stage builds let you get better caching and therefore faster rebuilds, since you can cache the pre-installed gcc etc. layer, while still getting the
234.
A deep dive into the official Docker image for Python(pythonspeed.com)
189 points by itamarst 6y ago|61 comments
235.
Find that bug: Using a search engine as a programmer(codewithoutrules.com)
2 points by itamarst 6y ago|0 comments
236.
by itamarst 6y ago
That's not what I meant, but rather "Python is a Conda package too, a dependency." I'll rephrase to make it clearer.
237.
Options for packaging your Python code: Wheels, Conda, Docker, and more(pythonspeed.com)
3 points by itamarst 6y ago|0 comments
238.
Debugging Python server memory leaks with the Fil profiler(pythonspeed.com)
3 points by itamarst 6y ago|0 comments
239.
A tableau of crimes and misfortunes: the ever-useful `docker history`(pythonspeed.com)
1 points by itamarst 6y ago|0 comments
240.
Too many objects: Reducing memory overhead from Python instances(pythonspeed.com)
1 points by itamarst 6y ago|0 comments

More ›