itamarst

22 ms·

181.
Loading SQL data into Pandas without running out of memory(pythonspeed.com)
4 points by itamarst 6y ago|0 comments
182.
Captcha forces people to mark mailbox as parking meter(twitter.com)
14 points by itamarst 6y ago|6 comments
183.
by itamarst 6y ago
Taylor's most famous story (Schmidt and the pig iron) was... mostly made up. https://journals.aom.org/doi/abs/10.5465/1556350 Hawthorne effect... was also made up. > But only five workers took part in
184.
The worst so-called “best practice” for Docker(pythonspeed.com)
3 points by itamarst 6y ago|0 comments
185.
Speeding up Docker builds in CI with BuildKit(pythonspeed.com)
3 points by itamarst 6y ago|0 comments
186.
The security scanner that cried wolf(pythonspeed.com)
2 points by itamarst 6y ago|0 comments
187.
by itamarst 6y ago
I guess when I say correctness, I mean "how do I know it _continues_ to be correct on data we've never seen before". That's where metamorphic testing can be valuable, because it lets you at least find incorrectness on re
188.
by itamarst 6y ago
To be fair, you can get really very far with non-machine-learning automated techniques (I did good-enough algorithms for an in-situ fluorescent gene sequencing image processing pipeline at one job). I suspect any form of good automated proc
189.
by itamarst 6y ago
Some books I enjoyed about the trade networks across Eurasia: * When Asia Was the World * Religions of the Silk Road: Premodern Patterns of Globalization
190.
Transgressive Programming: the magic of breaking abstraction boundaries(pythonspeed.com)
2 points by itamarst 6y ago|0 comments
191.
You Need to Upgrade Pip(pythonspeed.com)
1 points by itamarst 6y ago|1 comments
192.
by itamarst 6y ago
For memory that are actually two different domains, with different problems, different UX requirements, and different solutions: 1. For server workloads, the main issue is usually leaks. Lacking leaks you can just characterize the workload,
193.
by itamarst 6y ago
Note that py-spy seems problematic in containers—it requires ptrace, which means you need a special capability, and that's a security risk so many environments won't even give people the option to enable it. In addition to vmprof,
194.
All Pythons are slow, but some are faster than others(pythonspeed.com)
5 points by itamarst 6y ago|0 comments
195.
by itamarst 6y ago
Building Docker images is a tricky design problem, as with many ops tasks. The problem is due to 3 issues: 1. A Docker image is the intersection of literally 50 years of technology, starting from Unix in the 1970s, none of which are perfect
196.
by itamarst 6y ago
It still works, they're just not doing new releases.
197.
by itamarst 6y ago
People keep misunderstanding this, so just to clarify: pip still works on Python 2! They just aren't doing any new releases for Python 2. $ virtualenv --python=python2.7 /tmp/venv $ /tmp/venv/bin/pip
198.
by itamarst 6y ago
Thank you! If you have any questions/problems/ideas, please reach out via GitHub or email (itamar@pythonspeed.com).
199.
by itamarst 6y ago
Internally Pandas often uses NumPy arrays, especially for numeric data, so might be able to pass things that way in some cases? E.g. `df["column_name"].values` will you get you a NumPy array.
200.
by itamarst 6y ago
Yeah, I was using DFA. Next step is trying alternative approach, but if that alternative doesn't work I'm going to see about wrapping your package for Python. Thanks for all your work on it!
201.
by itamarst 6y ago
There's definitely a conversion cost. For strings, Python apparently caches the UTF-8 encoded string, so if you _repeatedly_ transfer it to Rust I suspect (but haven't checked) that the cost is much lower. In general I suspect it&
202.
by itamarst 6y ago
I've been playing with PyO3 for prototyping, and wrapped some Rust code to see if it's faster than Python. The experience was very much like using Boost Python (whcih these days has alternative with https://github.com&#
203.
Speed up pip downloads in Docker with BuildKit’s new caching(pythonspeed.com)
10 points by itamarst 6y ago|0 comments
204.
by itamarst 6y ago
I do some consulting where I'm paid to fix specific bugs / add specific features to various open source projects, often in Python data science world. Usually it goes something like: 1. Figure out what the project's contributi
205.
Dying, fast and slow: out-of-memory crashes in Python(pythonspeed.com)
1 points by itamarst 6y ago|0 comments
206.
by itamarst 6y ago
> "We blocked the phrase and other variations of it from trending," a Twitter spokesperson said. "We want trends to promote healthy discussions on Twitter. This means that at times, we may prevent certain content from tren
207.
Docker BuildKit: faster builds, new features, and now it's stable(pythonspeed.com)
1 points by itamarst 6y ago|0 comments
208.
Jane Austen violates what everyone learned in their English class(pemberley.com)
2 points by itamarst 6y ago|0 comments
209.
by itamarst 6y ago
It's been used since the time of Jane Austen (by Jane Austen, in fact), it's perfectly normal: https://pemberley.com/janeinfo/austheir.html
210.
by itamarst 6y ago
In many cases the memory usage is linear with input data, so you can come up with a function that predicts memory usage, add some padding, and then you don't need retries. E.g. this example here: https://pythonspeed.com&#x2F

More ›