Ask HN: Best practices for training NN across multiple computer/gpu’s

Hi all, My team is working on training a NN with a data set where each inputted record is really large. We’re able to just barely fit 1 record on the GPU and each epoch takes quite a while. So one thing we’re looking at is training on multiple computer/gpu’s in parallel then combining the results. I know that googling yields lots of suggestions. Can anyone recommend best practices or good resources for us to explore? Thanks in advance.