jmartin2683

8 ms·

61.
by jmartin2683 4y ago
More specifically, building data processing stuff and apis on top of arrow using polars, primarily
62.
by jmartin2683 4y ago
Email me James dot Martin at zelis dot com
63.
by jmartin2683 4y ago
Senior engineer working on a data processing pipeline
64.
by jmartin2683 4y ago
…by choice?
65.
by jmartin2683 4y ago
I’m literally hiring 4 of them right now. Hit me up if you’re interested.
66.
by jmartin2683 4y ago
Do you know anyone who is actively excited about learning C++ right now?
67.
by jmartin2683 4y ago
People who call meetings should have to pay for the time of everyone present out of an annual budget. This would result in fewer, more useful meetings. The fact that they’re ‘free’ in some minds (usually those whose job largely consists o
68.
by jmartin2683 4y ago
They use neural networks nowadays to model the transfer function that determines how and when a signal will start to overdrive. It’s impressive.
69.
by jmartin2683 4y ago
https://www.foxnews.com/us/controversial-social-media-influe...
70.
by jmartin2683 4y ago
Datafusion has better support for object_store api (you can just register the store and pass it s3 uris). With polars, you have to have the file on the local file system (which I’ve found to be a huge performance win, cacheing them anyway)
71.
by jmartin2683 4y ago
Variable, but a single partitioned file goes anywhere from single-digit kb to maybe a little under a meg. A query can touch just one or dozens (all lazily concatenated into a single frame).
72.
by jmartin2683 4y ago
Also, if you want to help... we're hiring :)
73.
by jmartin2683 4y ago
We've used both DataFusion and Polars for very similar use cases, though recently we've really begun to coalesce around Polars and most further development will use it, barring any complication that would require something else (D
74.
by jmartin2683 4y ago
The big pro is that the only data that ever has to move is that which is actually accessed. It’s extremely cost-effective to produce, store and query statically vs moving it into something else of sufficient scale to handle the entire univ
75.
by jmartin2683 4y ago
We’re talking about hundreds of gigabytes, pushing towards many terabytes of data with extremely sparse access patterns. Loading into anything (including an rdbms) wouldn’t be cost-effective or practical.
76.
by jmartin2683 4y ago
Requirements largely require selecting from and aggregating across some subset of the whole universe of data, which is on the order of hundreds of GB of parquet. We use partitioning across files (and folder structure in s3) to lazily scan
77.
by jmartin2683 4y ago
We use rust for data processing, etl and to serve apis directly from massive parquet collections using Polars. What would you like to know?
78.
by jmartin2683 4y ago
Rust! Polars/Datafusion are taking over, and for good reason.
79.
by jmartin2683 4y ago
…until everyone in the bottom 20% of the job market stops working.
80.
by jmartin2683 4y ago
Cryptocurrency, as a thing, just needs to go away. How many people have to lose their life savings before we admit a bad idea? All so that NK can pay for nukes, people can buy drugs and extort people on the internet. We’re not making the w
81.
by jmartin2683 4y ago
This seems like a collection o terrible advice. These things are all very conditional, and such blanket statements are nonsense. Postgres is great, but it’s not all the things for all the applications. I wouldn’t hire (or work for!) this
82.
by jmartin2683 4y ago
Bootcamps train narrow and deep, where the number one thing that you have to be to be successful in engineering is wide.
83.
by jmartin2683 4y ago
As you get old, time becomes a lot more valuable as it’s the only thing that you spend every day and can never make more of. Money is cheap and doesn’t really do all that much, beyond a point. The drama of switching jobs becomes less and l
84.
by jmartin2683 4y ago
I’ve been using computers for 30 years and have never met a reliable printer.
85.
by jmartin2683 4y ago
They’re trying to get people to quit. I would.
86.
by jmartin2683 4y ago
I can’t begin to understand how hard those problems are, but I can imagine. Kudos to them!
87.
by jmartin2683 4y ago
Memory things, sure. There is the whole concurrency/parallelism thing, though.
88.
by jmartin2683 4y ago
You’re creating lots and lots of bugs that will come back to haunt to at runtime (midnight on a Friday if the universe has its way).
89.
by jmartin2683 4y ago
I really can’t understand where this idea that rust is ‘hard’ comes from. Everything that it formalizes is something that you should already be conscious of . If not, well… you have a different, bigger problem.
90.
by jmartin2683 4y ago
A lot of people can’t fathom actually doing what they love for a living and what that really means, the implications that is has on your decision making process etc. I completely understand why he’d do that, and can imagine lots of cool pr

More ›