MrPowers

21 ms·

151.
by MrPowers 6y ago
Federal income taxes are highly economically inefficient. They create a huge deadweight loss of taxation (productive employees wasting their time helping corporations comply with tax law). Land and gas are economically efficient to tax. T
152.
by MrPowers 6y ago
Yep, agreed. Go is a great language for AWS Lambda type workflows. Python isn't as great (Python Lambda Layers built on Macs don't always work). AWS Data Wrangler ( https://github.com/awslabs/aws-data-wrangle
153.
by MrPowers 6y ago
Go has a ton of potential in the data science space. A basic DataFrame library would go a long way. Doesn't have to be as full featured as Pandas. Just something that's maintainable and portable. I wrote a blog post a few months
154.
by MrPowers 6y ago
This article doesn't align with my experience of what I've observed. > I don't think that it's in our interest to be passionate, but it is in employers' interest Meh, I'm into some technologies just cause th
155.
by MrPowers 6y ago
Companies should have different hiring processes for prolific open source code contributors and folks that don't do open source (presuming they want to hire both). There are plenty of amazing developers that don't to open source a
156.
by MrPowers 6y ago
I've found pytest to encourage tests with really long method names, examples from the post: test_refresh_failure test_refresh_with_timeout These get even longer like test_refresh_with_timeout_when_username_is_not_found for exa
157.
by MrPowers 6y ago
There are certainly folks that argue the language it too big. Here's a great talk arguing that a lot of language features should be eliminated and Scala should be made a functional programming language: https://www.youtube.
158.
by MrPowers 6y ago
I maintain open source libs in Python, Ruby, and Scala - the Scala libs are a lot more difficult to maintain. Certain languages prioritize backwards compatibility and maintainability over new features / complicated programming stuff.
159.
by MrPowers 6y ago
Upgrading Spark from Scala 2.11 to Scala 2.12 was a huge effort. See here for more discussion: https://contributors.scala-lang.org/t/spark-as-a-scala-gatew... Some projects are easier to upgrade, but for complex proje
160.
by MrPowers 6y ago
Some responses: 1. Yea, but languages that are backwards compatible have libs that are useful for way longer. I can still use Java JAR files that were built with Java 8, and published many years ago. The open source libs get stale so much
161.
by MrPowers 6y ago
Don't have a lot of Java experience, but think it's backwards compatible, even across major versions, whereas Scala isn't even backwards compatible for minor versions.
162.
Scala projects are difficult to maintain(mungingdata.com)
166 points by MrPowers 6y ago|231 comments
163.
by MrPowers 6y ago
Looks like a cool project. It's better to separate benchmarking results for big data technologies and small DataFrame technologies. Spark & Dask can perform computations on terabytes of data (thousands of Parquet files in parallel)
164.
by MrPowers 6y ago
Spark lets users query via regular SQL or DSL (Scala, PySpark). The DSLs are generally better to work with. Complex queries can be broken up into composable functions that can be mixed and matched to perform different analyses. Functions
165.
by MrPowers 6y ago
The author argues against this strategy of audience building: * remixing feel good content of others * trying to find a product fit by audience building * You know you're creating valuable content if you're scared before hitting t
166.
by MrPowers 6y ago
Twitter is the best place to reach some of the top people in tech (in the big data niche at least). It's a great place to market projects, get a following, and find open source collaborators. It can also be a toxic place that puts you
167.
by MrPowers 6y ago
I worked in finance for 5 years, passed all 3 CFA exams, and was ready to start studying for the GMAT to get in a top MBA school. Decided to take a hard right and learn programming instead. I loved studying finance, but didn't like wo
168.
by MrPowers 6y ago
I don't know if Bitcoin will outperform Facebook, but can explain why they're fundamentally different assets. Facebook is a "productive asset". It's a business that sells to customers and generated 29 billion in p
169.
by MrPowers 6y ago
Emerging markets and value stocks are historically cheap on a relative basis, see the What to Do? section at the end of the article.
170.
by MrPowers 6y ago
Jeremy Grantham co-founded the investment firm GMO. They're famous for 7 year forecasts by asset classes. Here's the bearish December 2020 forecast: https://www.gmo.com/americas/research-library/gmo-7-y
171.
by MrPowers 6y ago
Great point, r5.metal instances have 96 CPUs and 768 GB of RAM. Lots of "big data problems" can actually be solved with a single big EC2 instance. Cluster computing should always be avoided when a single node will do.
172.
by MrPowers 6y ago
Your experience aligns with what I've seen. All the notebooks are in one place. Some are for important production jobs, other are for data exploration. It's easy to make a little edit in a notebook and accidentally break producti
173.
by MrPowers 6y ago
Spark offers Scala, Python, Java, and R APIs. Scala & Python are the most viable options (R lacks a lot of features and Java is only good for ppl that love Java). Scala & PySpark are both great options. Lots of devs are terrified
174.
by MrPowers 6y ago
The last point was for teams that only rely on notebooks, sorry if I didn't make that clear. You're right that all those issues can be sidestepped if you build projects in version controlled Git repos, test the code, and deploy JA
175.
by MrPowers 6y ago
My feeling is that Spark is better for huge ETL jobs and Dask is better for certain types of model building (because of easy access to Python libraries), but don't have any benchmarks to back this up. Would love some benchmark results
176.
by MrPowers 6y ago
Some additional background: * AWS has a managed Spark offering called EMR * EMR pricing ( https://aws.amazon.com/emr/pricing/ ) is lower than Databricks pricing ( https://databricks.com/product/a
177.
by MrPowers 6y ago
Snowflake and Databricks are different, sometimes complementary technologies. You can store data in Snowflake & query it with Databricks for example: https://github.com/snowflakedb/spark-snowflake Snowflake predic
178.
by MrPowers 6y ago
Seems like you're equating superstar developers with years of experience and that hasn't been my observation. PyTorch was created by an undergrad. I've worked with senior devs that have let engineering pass them by and are
179.
by MrPowers 6y ago
Dominique Brezinski's latest Spark Summit talk is what cemented my feeling that superstar economics are at play in the current marketplace (in certain fields): https://databricks.com/session_na20/patterns-and-opera
180.
by MrPowers 6y ago
I agree, the free market decides your wage no only on technical prowess, but also your "influencer score" and your personality traits. Technically savvy folks with a low influencer score may get paid less. Technically savvy folks

More ›