pletnes

17 ms·

241.
by pletnes 3y ago
Numpy works fine these days, haven’t tested scipy. I’d give it a go. But the C bits inside numpy won’t go faster.
242.
by pletnes 3y ago
Also in the current design you can stack them. This is probably worth something in terms of wrapping of pallets of cans.
243.
by pletnes 3y ago
On a project I’m working on, tests run 50x faster in WSL. General IDE usage is not that different. I run the IDE in windows and the code in WSL though.
244.
by pletnes 3y ago
I always found the mac and linux versions to be way faster than the windows one, for various Jetbrains IDEs. I have the impression that windows + antivirus + jetbrains is generally quite awful in terms of performance.
245.
by pletnes 3y ago
Documentation or other «static site» content can easily be edited like this, too.
246.
by pletnes 3y ago
Local development on enterprise managed win10 can be fairly confusing. Having a cloud vm/container with linux and local admin can be less troublesome, depending on what you’re doing of course.
247.
by pletnes 3y ago
Sometimes I work on software development, and this mindset («the only valuable asset is the code») makes total sense. But if I work on analytics / datascience projects, the analysis including outputs could be time consuming to run, val
248.
by pletnes 3y ago
What happens to the outputs in this case? I found the outputs to be both the most useful parts of notebooks, but also the most troublesome for diffing and versioning.
249.
by pletnes 3y ago
My naive guess is lack of a good package repository. CPAN made Perl great, even if the language was not to everyone’s taste. Node/npm made javascript huge. Pypi picking up binary wheels made python the obvious choice for any data-heavy
250.
by pletnes 3y ago
True. I’ve been wondering why they don’t facilitate non-spark use better. But then you can’t reasonably invoice 3 VMs extra, obviously.
251.
by pletnes 3y ago
I can’t blame them really. Regular pandas-in-notebook can deal with quite a bit of data and is reasonably convenient. Even if the org and datasets are large, it’s rare to have huge single tables. Then you can go full spark when you have to
252.
by pletnes 3y ago
So Databricks is being replaced by open source, self-managed spark-on-k8s? Broadly speaking? Any concrete recommendations on how to do that in a project?
253.
by pletnes 3y ago
But in the notebooks you run pyspark / scala spark / spark sql code?
254.
by pletnes 3y ago
What has replaced Databricks, in your opinion? Doesn’t seem «dead» to me, yet. Serious question.
255.
by pletnes 3y ago
Seconded. I’ve used Lastpass at work a few times and I have zero idea why they still exist - 1password is much better and othet competitors exist, too.
256.
by pletnes 3y ago
So on a raspberry 3A this works over USB-A… but how? Get a female-to-male adapter/cable somehow?
257.
by pletnes 3y ago
You’d think this would be even more robust with a sqlish storage backend than the usual text files used for history.
258.
by pletnes 3y ago
Xonsh has this. It’s super cool after using it for a while.
259.
by pletnes 3y ago
My macbook charges on 9V phone chargers. Slooowly.
260.
by pletnes 4y ago
Except when it’s a compiler bug.
261.
by pletnes 4y ago
Also the 5-10x compression turns «big data» into «laptop size data» very nicely!
262.
by pletnes 4y ago
A lot of the benefits are nicely present on my macbook. Compact size, fast querying with duckdb (columnar format + indexing), fast read/write for pandas, good type support… would recommend for local tabular data!
263.
by pletnes 4y ago
It also backtracks up the directory tree so you don’t have to cd to run «just test».
264.
by pletnes 4y ago
I found that exporting big tables as a bunch of parquet files is faster and uses less memory than duckdb’s internal format.
265.
by pletnes 4y ago
Duckdb is fantastic. Doesn’t need a schema, either.
266.
by pletnes 4y ago
Before conda, you needed a C and a fortran compiler, a BLAS and LAPACK installation, and various build tools, to install scipy. Scipy is one - 1 - dependency used in science and engineering. The pain used to be massive. Now, we’re complaini
267.
by pletnes 4y ago
Aha. Then I guess performance is the only issue. I guess it should run fine on Intel.
268.
by pletnes 4y ago
The mac M1/M2 shares CPU and GPU memory. Not all GPUs have enough memory available. Transferring data to/from the GPU has a perf disadvantage. Hence, few PCs are capable of running LLMs.
269.
by pletnes 4y ago
What about compression - is this part of arrow itself?
270.
by pletnes 4y ago
Or parquet, for compression?

More ›