8 ms·

Grid: AI platform from the makers of PyTorch Lightning

107 points by shcheklein 6y ago

ishcheklein 6y ago
More on this is here https://techcrunch.com/2020/10/08/grid-ai-raises-18-6m-series-a-to-help-ai-researchers-and-engineers-bring-their-models-to-production/ https://techcrunch.com/2020/10/08/grid-ai-raises-18-6m-serie... What do you think folks?
minimaxir 6y ago
So this is the endgame of pytorch-lightning, which was always a mystery to me. (if you haven't used it, it's strongly recommended if you use PyTorch: https://github.com/PyTorchLightning/pytorch-lightning https://github.com/PyTorchLightning/pytorch-lightning ) IMO, open source is at its best when it's supported by a SaaS as it provides a strong incentive to keep the project up-to-date, and the devs of PL have been very proactive.
- wfalcon 6y ago
  yes! our goal is to completely remove any engineering from the AI research -> production lifecycle. Not just a marginal improvement on that experience but a 10x completely different approach.
  - sillysaurusx 6y ago
    Dead end. Train researchers to be engineers. Not the other way around. I wish you luck though. We’ll see in ten years whether programmers are as effective as researchers, or whether researchers are as effective as programmers. In 60 some years of computing, no one has achieved the latter, despite many attempts. Also, I was surprised that this webpage is basically a waitlist and nothing else. No discussion of technique, no docs, no substance. Just a “you like pytorch? Pytorch rules!” type hype. I do like pytorch, but I also like knowing one or two substantive points about what the proposal here is. If you want to train a model from your laptop, it’s a matter of applying to TFRC and kicking off a TPU. The whole ecosystem is in need of massive overhaul. I like the ambition. But I dislike trying to pretend we aren’t programmers. ML is programming, and pretending otherwise will always cause massive, avoidable delays.
    - tasubotadas 6y ago
      I would pick engineers over researchers any day as you can teach an engineer how to do research but the opposite is rarely a case.
      - p1esk 6y ago
        It is much harder to do research then it is to do engineering. I do both at work where we have about 15 ML engineers and only 2-3 of them are capable of doing something novel. Maybe you mean something different by “research” but it is very hard to teach someone how to come up with an idea or a novel method publishable in a top conference. Teaching engineering on the other hand usually goes like this: “Here’s how this works. Here are your options, tradeoffs, pitfalls.” After you’ve done it a few times you mastered it. I forgot who said it: “In engineering, if you don’t know what you’re doing you shouldn’t be doing it. In science, if you know what you’re doing you shouldn’t be doing it.“
        tasubotadas 6y ago
        >Research is "creative and systematic work undertaken to increase the stock of knowledge" I meant this. Academia-related BS is just a one way to do that.
        marcinzm 6y ago
        Very little of the work done in machine learning is truly novel versus simple extensions of what has been done before. There's probably the same amount of novel work in machine learning and engineering except the ML crowd publishes it while the engineering version stays in closed source code bases. There's also a distinction between doing novel machine learning work and publishing successfully on novel machine learning work.
        p1esk 6y ago
        I agree with you. However very few "ML engineers" do any novel work, let alone "truly novel" work. Typically they just take the existing code and apply it to their specific problems with some tweaking. Sometimes they would try to reproduce papers when no code is available. Very few engineers are capable of the NeurIPS quality research - it does not matter if it get published or stays in a private repo. That's much harder to do and to teach than what 90% of typical ML engineers do. That was my point. There's also a ton of bad research being published in second rate conferences, but I don't consider that "research". You won't get "simple extensions of what has been done before" published in a top level conference, the acceptance rates have been extremely low recently.
  - wfalcon 6y ago
    Lightning is built for researchers by researchers... we've already taken a much different approach. 100% agree with you that going the other way is likely not the best approach. Lightning + Grid elevates and turns non experts closer to researchers... ie: focus on building the products and doing science and not the engineering. That's what lightning excels at today. That's the experience Grid will 10x.
  - orbifold 6y ago
    As a scientist: The amount I struggled with getting distributed training on a HPC cluster to work vs. how easy it was with Lightning was eye opening. Almost no code change and finally I can run across 20 nodes with 4 V-100 each :). Plus the automatic SLURM checkpoints and restarts <3.
high_derivative 6y ago
I am extremely pessimistic for ML ops startup like this. At the end of the day, cloud service providers have too much of an incentive to provide these tools for free as a cloud value add. The other thing is that stitching together other open source tools like this is simply not enough value. Who will be incentivised to buy? Saying this as FAANG ML org person where I see the push to open source ops tooling like this.
- DevX101 6y ago
  Sounds like a great reason to acquire a startup like this for $XX million. Might be a great outcome if the team is lean with minimal investment.
  - hobofan 6y ago
    Well that ship seems to have sailed with a ~$19mil investment and a 20 person (+$4mil/year burn in NYC?) team.
- minimaxir 6y ago
  On paper that's the case. Google certainly has made a push toward scalable AI training and deployment. However, it is not fun to use in practice, speaking from experience. Startups beating an incumbent with substantially better UX is always a good story. Improving productivity is an easy winner for potential customers.
- wfalcon 6y ago
  Well... we're not really an ML ops startup haha. I am ALSO pessimistic about ML Ops startups. But calling Grid an ML Ops startup is like calling Lightning Keras... maybe at a quick blink it looks like that, but that's where the similarities end. For what it's worth, a lot of what we're building comes from my experience at FAIR.
  - andrewmutz 6y ago
    If it is not ML Ops, how would you describe it?
seibelj 6y ago
The name is unfortunately close to “The Grid”, an AI website builder that had a lot of buzz then scammed a lot of people out of money then disappeared https://medium.com/@seibelj/the-grid-over-promise-under-deliver-and-the-lies-told-by-ai-startups-40aa98415d8e https://medium.com/@seibelj/the-grid-over-promise-under-deli...
- wfalcon 6y ago
  The grid is how you harness and distribute power and electricity.... like that coming from lightning :) Second, electricity was a great new technology (ie: AI), but you needed the power grid to make it usable - that's grid AI.
  - immigrantsheep 6y ago
    Also related to TRON
sillysaurusx 6y ago
I dislike that pytorch advertises TPU support. Pytorch doesn’t support TPUs. Pytorch supports a gimped version of TPUs that have no access to the TPU CPU, a massive 300GB memory store that handles infeed. No infeed means you have to feed the TPUs manually, on demand, like a gpu. And TPUs are not GPUs. When you try to do that, you’re talking at least a 40x slowdown, no exaggeration. The TPU CPU is the heart of the TPU’s power and advantage over GPUs, and neither pytorch nor Jax support it at all yet. No MLPerf benchmark will ever use pytorch in its current form on TPUs. Luckily, that form is changing. There are interesting plans. But they are still just plans. It’s better to go the other direction, I think. I ported pytorch to tensorflow: https://twitter.com/theshawwn/status/1311925180126511104?s=21 https://twitter.com/theshawwn/status/1311925180126511104?s=2... Pytorch is mostly just an api. And that api is mostly python. When people say they “like pytorch”, they’re expressing a preference for how to organize ML code, not for the set of operations available to you when you use pytorch.
- wfalcon 6y ago
  We currently have google engineers training on TPU pods with PyTorch Lightning. TPU support is VERY real... but yes, sometimes it breaks but PyTorch and Google are working very hard to bridge that gap. But we have dedicated partners at Google on the TPU team working to get Lightning working seamlessly on pods. Check out the discussions here: https://github.com/PyTorchLightning/pytorch-lightning/issues/3058 https://github.com/PyTorchLightning/pytorch-lightning/issues...
  - sillysaurusx 6y ago
    No, you do not support the TPU infeed, and this is a crucial distinction. Saying that you do support this has caused endless confusion and much surprise. It’s almost not an exaggeration to say that you’re lying (sorry for phrasing this so bluntly, but I’ve seriously spent dozens of hours trying to break this misconception due to hype like this). TPU support is real. Pytorch does in fact run on TPUs. But you don’t support TPU CPU memory, the staging area that you’re supposed to fill with training data. That staging area is why a TPU v3-512 pod can train an imagenet resnet classifier in 3 minutes at around 1M examples per second. You will not get anywhere near that performance with pytorch on TPUs. In fact, you’re expected to create a separate VM for every 8 TPU cores. The VMs are in charge of feeding the cores. That’s insane; I’ve driven TPU pods from a single n1-standard-2 using tensorflow. Repeat after me: if you are required to create more than one VM, you do not (yet!) support TPU pods. I wish I could triple underline this and put it in bold. People need to understand the limitations of this technique. Creating 256 VMs to feed a v3-2048 is not sustainable.
deleted 6y ago
[deleted]
visarga 6y ago
How do you handle the security of training data? If the data is super sensitive how do you deal with it? I know the same could be said about Azure and AWS, but the big name cloud providers stake their prestige on having tight security, while a startup has much less to lose.
- mahaniok 6y ago
  On the opposite, startup has everything to lose. No other business lines
  - visarga 6y ago
    Can't answer to 'Proven' on account of comment depth, but I am not concern trolling. I work as a ML engineer and many of our projects and datasets are sensitive. The company has signed contracts of confidentiality for this data and we are really concerned about where it sits and who gets access to it. Another major concern is GDPR - can you reliably track and delete data related to a specific person? You got to think about this right from the start to keep the necessary metadata in the tagging pipeline.
  - marcinzm 6y ago
    A large company may lose more than the value of the impacted business line while a startup is capped at the value of the business line (or rather the cash on hand which is even less). So the maximum loss is much higher in absolute terms for a large company and so they're able to invest more absolute money into security. Plus, philosophically, startups already have such massive risk that another X% due to a potential security breach isn't even worth bothering with versus spending the money on lowering risk in other areas.
  - Proven 6y ago
    Don't think so. In this case security isn't a feature so there's relatively less incentives to put resources on it. If you get lucky and get acquired before you get hacked, you can even dramatically underinvest in security and be just fine. Maybe the GP post is concern trolling, but claiming that security will be rock solid just because they are a startup is also unreasonable.
kbash9 6y ago
Seems like Pytorch lightening is the only first-class citizen in your offering. Is that true? Or are there value-added features for TensorFlow and other non-DL libraries such as scikit-learn? Also, is there support for distributed training for large datasets that don't fit into single instance memory? or just distributed grid-search/hyper-parameter optimization?
- wfalcon 6y ago
  No, we support other frameworks too! Just that if you use lightning you'll have zero friction. Well as with the others... you might run into issues inherent in the other framework's hard to work with designs.
neilc 6y ago
Congratulations to the Grid team on the fundraise and the announcement! Exciting stuff. It seems like there is an emerging consensus that (a) DL development requires access to massive compute, but (b) if you’re only using off-the-shelf PyTorch or TensorFlow, moving your model from your personal development environment to a cluster or cloud setting is too difficult — it is easy to spend most of your time managing infrastructure rather than developing models. At Determined AI, we’ve spent the last few years building an open source DL training platform that tries to make that process a lot simpler (https://github.com/determined-ai/determined https://github.com/determined-ai/determined), but I think it's fair to say that this is still very much an open space and an important problem. Curious to take a look at Grid AI and see how it compares to other tools in the space -- some other alternatives include Kubeflow, Polyaxon, and Spell AI.
- wfalcon 6y ago
  Just to be 100% clear. We want people to build on Lightning and we want companies to deliver value and products for their users. We place no limitations on how Lightning is use or what products people will build. Our in-process patent portfolio is not to limit the use of lightning in any way but for defensibility purposes only. If you have any other inquiries please email legal@pytorchlightning.ai for more details.
- wfalcon 6y ago
  maybe determined should come up with its own API instead of copying Lightning's :) Not a nice move for the opensource spirit. Also, pretty sure it's a violation of our patent and 100% copyright infringement.
  - neilc 6y ago
    Hi William -- we have absolutely not copied any of Lightning's APIs. In fact, our PyTorch API makes some significantly different design choices than Lightning does -- e.g., we require users to step optimizers and run the backward pass explicitly, which is a bit lower-level but allows for more flexibility when using the API. For instance, here is an example of a GAN using our PyTorch API: https://github.com/determined-ai/determined/blob/master/examples/gan/gan_mnist_pytorch/model_def.py https://github.com/determined-ai/determined/blob/master/exam... This is a port of this PyTorch Lightning example: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/domain_templates/generative_adversarial_net.py https://github.com/PyTorchLightning/pytorch-lightning/blob/m... Despite the former being a port of the latter, there are significant differences between the two APIs. More broadly, we welcome competition in this space and think there's a lot that we can all learn from one another.
bkkaggle 6y ago
i used pytorch lightning back in may when i was working on pretraining gpt2 on TPUs (https://bkkaggle.github.io/blog/nlp-research-part-2/ https://bkkaggle.github.io/blog/nlp-research-part-2/). it was really impressive how stable it was especially given how a lot of features were still being added at a very fast pace. also, this was probably the first (and maybe still is?) high-level pytorch library that let you train on tpus without a lot of refactoring and bugs which was a really nice thing to be able to do given how the pytorch-xla api was still unstable at that point. <3
xiaodai 6y ago
Sadly Julia's Flux.jl have fallen behind to the point that I am switching to using pytorch. It's just faster.
- tbenst 6y ago
  Every 3-6 months I try a new project in Flux and it’s definitely catching up quickly! The neural ODE integration is super cool for example. There are also other applications where you might need to calculate higher order gradients or even do forward-mode autodiff, and Flux is first rate there. But I too have had problems porting state of art models in Flux, due to speed and memory issues. The developers are doing an amazing job but only have a fraction of the resources of pytorch or tensorflow. But that will never change unless people adopt and contribute.

107 points by shcheklein 6y ago

More on this is here https://techcrunch.com/2020/10/08/grid-ai-raises-18-6m-series-a-to-help-ai-researchers-and-engineers-bring-their-models-to-production/ https://techcrunch.com/2020/10/08/grid-ai-raises-18-6m-serie... What do you think folks?

minimaxir 6y ago

So this is the endgame of pytorch-lightning, which was always a mystery to me. (if you haven't used it, it's strongly recommended if you use PyTorch: https://github.com/PyTorchLightning/pytorch-lightning https://github.com/PyTorchLightning/pytorch-lightning ) IMO, open source is at its best when it's supported by a SaaS as it provides a strong incentive to keep the project up-to-date, and the devs of PL have been very proactive.

wfalcon 6y ago
yes! our goal is to completely remove any engineering from the AI research -> production lifecycle. Not just a marginal improvement on that experience but a 10x completely different approach.
- sillysaurusx 6y ago
  Dead end. Train researchers to be engineers. Not the other way around. I wish you luck though. We’ll see in ten years whether programmers are as effective as researchers, or whether researchers are as effective as programmers. In 60 some years of computing, no one has achieved the latter, despite many attempts. Also, I was surprised that this webpage is basically a waitlist and nothing else. No discussion of technique, no docs, no substance. Just a “you like pytorch? Pytorch rules!” type hype. I do like pytorch, but I also like knowing one or two substantive points about what the proposal here is. If you want to train a model from your laptop, it’s a matter of applying to TFRC and kicking off a TPU. The whole ecosystem is in need of massive overhaul. I like the ambition. But I dislike trying to pretend we aren’t programmers. ML is programming, and pretending otherwise will always cause massive, avoidable delays.
  - tasubotadas 6y ago
    I would pick engineers over researchers any day as you can teach an engineer how to do research but the opposite is rarely a case.
    - p1esk 6y ago
      It is much harder to do research then it is to do engineering. I do both at work where we have about 15 ML engineers and only 2-3 of them are capable of doing something novel. Maybe you mean something different by “research” but it is very hard to teach someone how to come up with an idea or a novel method publishable in a top conference. Teaching engineering on the other hand usually goes like this: “Here’s how this works. Here are your options, tradeoffs, pitfalls.” After you’ve done it a few times you mastered it. I forgot who said it: “In engineering, if you don’t know what you’re doing you shouldn’t be doing it. In science, if you know what you’re doing you shouldn’t be doing it.“
      - tasubotadas 6y ago
        >Research is "creative and systematic work undertaken to increase the stock of knowledge" I meant this. Academia-related BS is just a one way to do that.
      - marcinzm 6y ago
        Very little of the work done in machine learning is truly novel versus simple extensions of what has been done before. There's probably the same amount of novel work in machine learning and engineering except the ML crowd publishes it while the engineering version stays in closed source code bases. There's also a distinction between doing novel machine learning work and publishing successfully on novel machine learning work.
        p1esk 6y ago
        I agree with you. However very few "ML engineers" do any novel work, let alone "truly novel" work. Typically they just take the existing code and apply it to their specific problems with some tweaking. Sometimes they would try to reproduce papers when no code is available. Very few engineers are capable of the NeurIPS quality research - it does not matter if it get published or stays in a private repo. That's much harder to do and to teach than what 90% of typical ML engineers do. That was my point. There's also a ton of bad research being published in second rate conferences, but I don't consider that "research". You won't get "simple extensions of what has been done before" published in a top level conference, the acceptance rates have been extremely low recently.
- wfalcon 6y ago
  Lightning is built for researchers by researchers... we've already taken a much different approach. 100% agree with you that going the other way is likely not the best approach. Lightning + Grid elevates and turns non experts closer to researchers... ie: focus on building the products and doing science and not the engineering. That's what lightning excels at today. That's the experience Grid will 10x.
- orbifold 6y ago
  As a scientist: The amount I struggled with getting distributed training on a HPC cluster to work vs. how easy it was with Lightning was eye opening. Almost no code change and finally I can run across 20 nodes with 4 V-100 each :). Plus the automatic SLURM checkpoints and restarts <3.

high_derivative 6y ago

I am extremely pessimistic for ML ops startup like this. At the end of the day, cloud service providers have too much of an incentive to provide these tools for free as a cloud value add. The other thing is that stitching together other open source tools like this is simply not enough value. Who will be incentivised to buy? Saying this as FAANG ML org person where I see the push to open source ops tooling like this.

DevX101 6y ago
Sounds like a great reason to acquire a startup like this for $XX million. Might be a great outcome if the team is lean with minimal investment.
- hobofan 6y ago
  Well that ship seems to have sailed with a ~$19mil investment and a 20 person (+$4mil/year burn in NYC?) team.
minimaxir 6y ago
On paper that's the case. Google certainly has made a push toward scalable AI training and deployment. However, it is not fun to use in practice, speaking from experience. Startups beating an incumbent with substantially better UX is always a good story. Improving productivity is an easy winner for potential customers.
wfalcon 6y ago
Well... we're not really an ML ops startup haha. I am ALSO pessimistic about ML Ops startups. But calling Grid an ML Ops startup is like calling Lightning Keras... maybe at a quick blink it looks like that, but that's where the similarities end. For what it's worth, a lot of what we're building comes from my experience at FAIR.
- andrewmutz 6y ago
  If it is not ML Ops, how would you describe it?

seibelj 6y ago

The name is unfortunately close to “The Grid”, an AI website builder that had a lot of buzz then scammed a lot of people out of money then disappeared https://medium.com/@seibelj/the-grid-over-promise-under-deliver-and-the-lies-told-by-ai-startups-40aa98415d8e https://medium.com/@seibelj/the-grid-over-promise-under-deli...

wfalcon 6y ago
The grid is how you harness and distribute power and electricity.... like that coming from lightning :) Second, electricity was a great new technology (ie: AI), but you needed the power grid to make it usable - that's grid AI.
- immigrantsheep 6y ago
  Also related to TRON

sillysaurusx 6y ago

I dislike that pytorch advertises TPU support. Pytorch doesn’t support TPUs. Pytorch supports a gimped version of TPUs that have no access to the TPU CPU, a massive 300GB memory store that handles infeed. No infeed means you have to feed the TPUs manually, on demand, like a gpu. And TPUs are not GPUs. When you try to do that, you’re talking at least a 40x slowdown, no exaggeration. The TPU CPU is the heart of the TPU’s power and advantage over GPUs, and neither pytorch nor Jax support it at all yet. No MLPerf benchmark will ever use pytorch in its current form on TPUs. Luckily, that form is changing. There are interesting plans. But they are still just plans. It’s better to go the other direction, I think. I ported pytorch to tensorflow: https://twitter.com/theshawwn/status/1311925180126511104?s=21 https://twitter.com/theshawwn/status/1311925180126511104?s=2... Pytorch is mostly just an api. And that api is mostly python. When people say they “like pytorch”, they’re expressing a preference for how to organize ML code, not for the set of operations available to you when you use pytorch.

wfalcon 6y ago
We currently have google engineers training on TPU pods with PyTorch Lightning. TPU support is VERY real... but yes, sometimes it breaks but PyTorch and Google are working very hard to bridge that gap. But we have dedicated partners at Google on the TPU team working to get Lightning working seamlessly on pods. Check out the discussions here: https://github.com/PyTorchLightning/pytorch-lightning/issues/3058 https://github.com/PyTorchLightning/pytorch-lightning/issues...
- sillysaurusx 6y ago
  No, you do not support the TPU infeed, and this is a crucial distinction. Saying that you do support this has caused endless confusion and much surprise. It’s almost not an exaggeration to say that you’re lying (sorry for phrasing this so bluntly, but I’ve seriously spent dozens of hours trying to break this misconception due to hype like this). TPU support is real. Pytorch does in fact run on TPUs. But you don’t support TPU CPU memory, the staging area that you’re supposed to fill with training data. That staging area is why a TPU v3-512 pod can train an imagenet resnet classifier in 3 minutes at around 1M examples per second. You will not get anywhere near that performance with pytorch on TPUs. In fact, you’re expected to create a separate VM for every 8 TPU cores. The VMs are in charge of feeding the cores. That’s insane; I’ve driven TPU pods from a single n1-standard-2 using tensorflow. Repeat after me: if you are required to create more than one VM, you do not (yet!) support TPU pods. I wish I could triple underline this and put it in bold. People need to understand the limitations of this technique. Creating 256 VMs to feed a v3-2048 is not sustainable.

deleted 6y ago

[deleted]

visarga 6y ago

How do you handle the security of training data? If the data is super sensitive how do you deal with it? I know the same could be said about Azure and AWS, but the big name cloud providers stake their prestige on having tight security, while a startup has much less to lose.

mahaniok 6y ago
On the opposite, startup has everything to lose. No other business lines
- visarga 6y ago
  Can't answer to 'Proven' on account of comment depth, but I am not concern trolling. I work as a ML engineer and many of our projects and datasets are sensitive. The company has signed contracts of confidentiality for this data and we are really concerned about where it sits and who gets access to it. Another major concern is GDPR - can you reliably track and delete data related to a specific person? You got to think about this right from the start to keep the necessary metadata in the tagging pipeline.
- marcinzm 6y ago
  A large company may lose more than the value of the impacted business line while a startup is capped at the value of the business line (or rather the cash on hand which is even less). So the maximum loss is much higher in absolute terms for a large company and so they're able to invest more absolute money into security. Plus, philosophically, startups already have such massive risk that another X% due to a potential security breach isn't even worth bothering with versus spending the money on lowering risk in other areas.
- Proven 6y ago
  Don't think so. In this case security isn't a feature so there's relatively less incentives to put resources on it. If you get lucky and get acquired before you get hacked, you can even dramatically underinvest in security and be just fine. Maybe the GP post is concern trolling, but claiming that security will be rock solid just because they are a startup is also unreasonable.

kbash9 6y ago

Seems like Pytorch lightening is the only first-class citizen in your offering. Is that true? Or are there value-added features for TensorFlow and other non-DL libraries such as scikit-learn? Also, is there support for distributed training for large datasets that don't fit into single instance memory? or just distributed grid-search/hyper-parameter optimization?

wfalcon 6y ago
No, we support other frameworks too! Just that if you use lightning you'll have zero friction. Well as with the others... you might run into issues inherent in the other framework's hard to work with designs.

neilc 6y ago

Congratulations to the Grid team on the fundraise and the announcement! Exciting stuff. It seems like there is an emerging consensus that (a) DL development requires access to massive compute, but (b) if you’re only using off-the-shelf PyTorch or TensorFlow, moving your model from your personal development environment to a cluster or cloud setting is too difficult — it is easy to spend most of your time managing infrastructure rather than developing models. At Determined AI, we’ve spent the last few years building an open source DL training platform that tries to make that process a lot simpler (https://github.com/determined-ai/determined https://github.com/determined-ai/determined), but I think it's fair to say that this is still very much an open space and an important problem. Curious to take a look at Grid AI and see how it compares to other tools in the space -- some other alternatives include Kubeflow, Polyaxon, and Spell AI.

wfalcon 6y ago
Just to be 100% clear. We want people to build on Lightning and we want companies to deliver value and products for their users. We place no limitations on how Lightning is use or what products people will build. Our in-process patent portfolio is not to limit the use of lightning in any way but for defensibility purposes only. If you have any other inquiries please email legal@pytorchlightning.ai for more details.
wfalcon 6y ago
maybe determined should come up with its own API instead of copying Lightning's :) Not a nice move for the opensource spirit. Also, pretty sure it's a violation of our patent and 100% copyright infringement.
- neilc 6y ago
  Hi William -- we have absolutely not copied any of Lightning's APIs. In fact, our PyTorch API makes some significantly different design choices than Lightning does -- e.g., we require users to step optimizers and run the backward pass explicitly, which is a bit lower-level but allows for more flexibility when using the API. For instance, here is an example of a GAN using our PyTorch API: https://github.com/determined-ai/determined/blob/master/examples/gan/gan_mnist_pytorch/model_def.py https://github.com/determined-ai/determined/blob/master/exam... This is a port of this PyTorch Lightning example: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/domain_templates/generative_adversarial_net.py https://github.com/PyTorchLightning/pytorch-lightning/blob/m... Despite the former being a port of the latter, there are significant differences between the two APIs. More broadly, we welcome competition in this space and think there's a lot that we can all learn from one another.

bkkaggle 6y ago

i used pytorch lightning back in may when i was working on pretraining gpt2 on TPUs (https://bkkaggle.github.io/blog/nlp-research-part-2/ https://bkkaggle.github.io/blog/nlp-research-part-2/). it was really impressive how stable it was especially given how a lot of features were still being added at a very fast pace. also, this was probably the first (and maybe still is?) high-level pytorch library that let you train on tpus without a lot of refactoring and bugs which was a really nice thing to be able to do given how the pytorch-xla api was still unstable at that point. <3

xiaodai 6y ago

Sadly Julia's Flux.jl have fallen behind to the point that I am switching to using pytorch. It's just faster.

tbenst 6y ago
Every 3-6 months I try a new project in Flux and it’s definitely catching up quickly! The neural ODE integration is super cool for example. There are also other applications where you might need to calculate higher order gradients or even do forward-mode autodiff, and Flux is first rate there. But I too have had problems porting state of art models in Flux, due to speed and memory issues. The developers are doing an amazing job but only have a fraction of the resources of pytorch or tensorflow. But that will never change unless people adopt and contribute.