MrPowers

22 ms·

181.
by MrPowers 6y ago
Yep, agreed. Here's an article for the bimodal nature of starting salaries for lawyers for interested parties: https://www.biglawinvestor.com/bimodal-salary-distribution-c... The upper end of the biglaw salaries was c
182.
by MrPowers 6y ago
Think the programming talent pool is increasingly getting split into run-of-the-mill and superstar developers. The location based pay argument depends on if you're an average or superstar employee. Average employees (e.g. building CRUD
183.
by MrPowers 6y ago
It seemed like Swift looked like a promising language for data science a few years ago. I'm not familiar with Swift. Can anyone provide additional context as to why the language seemed promising for data?
184.
by MrPowers 6y ago
I've rolled out Scala based Spark interfaces to non-programmers in Databricks notebooks, so it's definitely possible, but only if you stick with the basic language features. Here's a more detailed PySpark vs Scala comparison
185.
by MrPowers 6y ago
Yep, there is even a talk on the Scala Infinity War which argues for one subset version of Scala that it needs in order to survive: https://www.youtube.com/watch?v=v8IQ-X2HkGE&ab_channel=Scala... For me, Scala is a mult
186.
by MrPowers 6y ago
Yep, the Spark codebase is a great example of what a big Scala codebase should look like. Nothing crazy, not too many traits, mainly just "regular functions". Imagine another codebase uses a feature like self-types extensively:
187.
by MrPowers 6y ago
In the Spark world, you can use a tiny subset of the Scala features and enjoy huge productivity gains over the other language APIs (Java & Python). Those productivity gains are wiped out as more crazy language features get used. I don&
188.
by MrPowers 6y ago
Perhaps the better way to phrase this is "the day Lightbend deprecates SBT and tells the community to use Mill", but I agree with the sentiment. Scala needs a new "official stack", hopefully Mill + utest/Munit + sca
189.
by MrPowers 6y ago
I am a Scala programmer & think it's a great language. Here are some arguments for why not Scala: * Li's libs (os-lib, upickle, utest) have clean public interfaces, but most Scala ecosystem libs are hard to use, see the JSON
190.
by MrPowers 6y ago
There were some pipenv red flags back in 2020, see the bottom portion of this blog post: https://jacobian.org/2019/nov/11/python-environment-2020/ Just checked the commit history of the repo and looks li
191.
by MrPowers 6y ago
Python workflows have gotten way better in the last few years: * Poetry for dependency management * pyenv for Python version * Black for automated code formatting The .python-version file automatically sets my Python version. I run black .
192.
by MrPowers 6y ago
Need to know what audience to market to or how to relate to Redditors or both?
193.
by MrPowers 6y ago
Reddit is great if you unsubscribe from all the default subreddits and subscribe to programming subreddits (I'm subscribed to apachespark, julia, python, rust, vim, etc.). My Reddit feed now has more programming related articles than H
194.
by MrPowers 6y ago
Any suggestions for static site generators / where to host? I'm thinking about starting a new blog with Hugo / Digital Ocean. I have a popular Spark blog on Wordpress ( https://mungingdata.com/ ) and can relat
195.
Scala Spark vs. Python PySpark: Which Is Better?(mungingdata.com)
3 points by MrPowers 6y ago|0 comments
196.
by MrPowers 6y ago
The new libs make Scala way more enjoyable: * os-lib: allows for nice filesystem operations * upickle: allows for sane JSON * utest / munit: sane testing (rather than Scalatest) * scalafmt: great go-like automatic code formatting As lo
197.
by MrPowers 6y ago
Scala does have Spark's ML capabilities. Spark's ML capabilities might not be the best, but they can be run on huge datasets. I'm guessing Dask has better model support, but isn't as scalable as Spark ML.
198.
by MrPowers 6y ago
CSV is a file format and Arrow is an in memory data format. The CSV vs Parquet comparison makes more sense. Conflating Arrow / Parquet is a pet peeve of Wes: https://news.ycombinator.com/item?id=23970586
199.
by MrPowers 6y ago
Exactly. Some specific examples. Read a Parquet file into a Pandas DataFrame. Then read the Pandas DataFrame into a Spark DataFrame. Spark & Pandas are using the same Arrow memory format, so no serde is needed. See the "Standard
200.
by MrPowers 6y ago
Seems like OP is focused on building a steady traffic source to sell his product. Combination of a good product + marketing focus. I have a popular Spark blog where I sell Spark books: https://leanpub.com/u/MrPowers .
201.
by MrPowers 6y ago
This article hinges on the "productivity is subjective" argument, which isn't true in all cases. There are objective measures for certain types of productivity: library downloads, Stackoverflow points, blog views, GitHub sta
202.
by MrPowers 6y ago
Databricks recently rebuilt Spark in C++ in a project called "Delta Engine" to overcome the JVM limitations you're pointing out. You're right, Rust is a great way to sidestep the dreaded JVM GC pauses.
203.
by MrPowers 6y ago
Yea, I've been robbed semi-violently and quite violently. I stand by my statement. You're not getting hurt if you manage the situation properly. And the benefits for me are huge. So happy I left a safe life and have a more fun
204.
by MrPowers 6y ago
I have friends with 4 kids under 10 and they live up in the Tesoro section of Poblado. Lots of gated house / apartment communities. Colombia is family oriented, so it's easy to find great housing and fit into the culture with a
205.
by MrPowers 6y ago
I came to Medellin with no Spanish and learned. Took longer and was harder than I expected. The locals don't speak English and most of the magical stuff happens in Spanish. My Spanish speaking personality is quite different from Engl
206.
by MrPowers 6y ago
Safety is also overrated ;) The most fun adventures I've been on have been dangerous. Lots of folks would be happier if they took bigger risks, went to more dangerous spots, and took some life tips from economically disadvantaged folk
207.
by MrPowers 6y ago
I've been living abroad for 5 years, mainly in Medellin, but also in Mexico, Brazil, and the Philippines. Living in Colombia is great for your mental health. People don't talk about Bitcoin, IPOs, or political turmoil. Great nei
208.
by MrPowers 6y ago
This is a low quality article. > Unless you’re living under a rock, you’d probably know that the central banks have been releasing stimulus packages to help those suffering from the pandemic Federal governments run stimulus programs, not
209.
by MrPowers 6y ago
Some additional context: * Companies are querying thousands / tens of thousands of Parquet files stored in the cloud via Spark * Parquet lakes can be partitioned which works well for queries that filter on the partitioned key (and slow
210.
by MrPowers 6y ago
I can't relate with the OP because tech lets you write your own ticket. There are so many different types of jobs and opportunities. > I'm sick of wasting time writing worthless tests You can find a codebase to work on that al

More ›