dimberman

6 ms·

1.
Improve the model for everyone: OpenAI and the Navier–Stokes problem(twitter.com)
4 points by dimberman 23d ago|1 comments
2.
by dimberman 23d ago
In 2004, Mark Zuckerberg called us "dumb fucks" for giving him our addresses and photos. Now we're giving model providers our math theorems, pharmaceutical research, and startup business plans.
3.
Show HN: We trained a 32B model to beat Opus 4 at credit card optimization(huggingface.co)
6 points by dimberman 5mo ago|0 comments
4.
by dimberman 6y ago
Astronomer.io | (multiple roles) Fully remote-first compnay. We maintain and offer Apache Airflow as a service to customers ranging from early stage start ups to Fortune 500s. We're hiring across the board. Front-end, python/data
5.
by dimberman 6y ago
Hi @theptip, I'm an Airflow PMC and would love to know a bit more about your comparison :). 1. Have you tried Airflow 2.0? We made some pretty big overhauls both in terms of UI and backend. 2. DAG versioning is currently problematic, b
6.
by dimberman 6y ago
I'd say it's simpler than that. Airflow is a backend project built by backend engineers. Most UI people don't use Airflow or know what it is. @ryanhamilton is the first front-end dev to become a committer on the project and t
7.
by dimberman 6y ago
We're hiring multiple UI experts to rebuild the Airflow UI from scratch in react, so this is just the beginning :).
8.
by dimberman 6y ago
Hi didip! could you make an airflow issue related to that or start a thread in the dev list? That could be interesting! (though you might want to wait until after the holidays as we're all a bit wiped :) )
9.
by dimberman 6y ago
On the OSS side we have a helm chart that is heavily based on the one we use at Astronomer. That should hopefully get you started (or you can reach out to astro and someone will help out with a demo if you want help on that)
10.
by dimberman 6y ago
I'm sure they will eventually, but Astronomer and (I believe) Cloud Composer have day 1 support.
11.
by dimberman 6y ago
Airflow 2.0 :)
12.
by dimberman 6y ago
WAIT. Sorry I misspoke. The only thing that is no longer allowed is using a bitshift operator between a DAG and a task. task_1 >> task_2 is totally fine my_dag >> task_1 Is no longer allowed. Most of your DAG shoul
13.
by dimberman 6y ago
I'm sure they will, I'm not sure on their timeline though.
14.
by dimberman 6y ago
So there are a few options for that if you're interested! 1. If you're using the KubernetesExecutor, you can point to custom images for individual tasks, this will primarily work if you're storing DAGs in git or a volume (or
15.
by dimberman 6y ago
Unfortunately that is one feature that we had to take out, but you should check out the TaskFlow API, it's a very worthwhile trade-off! Edit: Sorry I misspoke here The only thing that is no longer allowed is using a bitshift operator b
16.
by dimberman 6y ago
Thank you for the feedback! I'm gonna pass that on to some AWS experts in the community. One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released separately from Airflow itself. So you won't
17.
by dimberman 6y ago
A sensor would also work here. Especially with the new SmartSensor feature, sensors are basically free so you can set them up for event-based DAG executions.
18.
by dimberman 6y ago
We didn't separate task instances from timestamps YET purely because there was already so much to release that we didn't want to add more potential for bugs/upgrade difficulties. I believe this is on the docket for 2.1 or 3.0
19.
by dimberman 6y ago
I would say at this point Airflow is leaning pretty heavily on being a data tool. I wouldn't recommend it for something like CI/CD for example. Do you have a use-case in mind?
20.
by dimberman 6y ago
Thank you Holden! :D I seriously love that you're an Airflow user since your spark talks first got me into OSS.
21.
by dimberman 6y ago
Yeah we've been benchmarking for a while, it is VISIBLY noticeable. You're gonna love it :)
22.
by dimberman 6y ago
If you're switching to python operators you should check out the TaskFlow API. You can basically build python operators with just python functions and decorators.
23.
by dimberman 6y ago
Hi y'all! Airflow PMC here! Feel free to AMA about Airflow's new features/the roadmap going forward!
24.
by dimberman 6y ago
Hi y'all! Airflow PMC here! Feel free to AMA about Airflow's new features/the roadmap going forward!
25.
Airflow 2.0 has been released(twitter.com)
11 points by dimberman 6y ago|2 comments
26.
by dimberman 6y ago
I would recommend you check out Airflow 2.0. It's a pretty major rebuild in a whole lot of ways (new UI, new DAG API, up to TEN TIMES faster task execution, multiple schedulers at once). I've actually had friends prepared to pick