4 ms·

I setup an automated script to setup a AWS g2 instance, train my neural net using tensorflow, copy my model to my personal computer, and spin down. It costs lik

I setup an automated script to setup a AWS g2 instance, train my neural net using tensorflow, copy my model to my personal computer, and spin down. It costs like $5-$10 to train/test most neural network models. My most expensive model cost like $100 and required a ton of time and resources. It took like 4 days or something. You really dont need $2k workstation.. Of course, for personal use I do have a gtx 1080 because I like to game and play tensorflow/caffe

danieltillett 10y ago
Would you mind sharing this script? I am thinking of investing in the 2K workstation, but this sounds like a much more efficient way to go.
- lettergram 10y ago
  I tried to get it released before, but was shut down by the "open source office". So I can't give you the exact script. However, h2o has a script that launches a cluster, that's very similar: https://github.com/ledell/h2oEnsemble-benchmarks/blob/master/ec2/h2o-cluster-launch-instances.py https://github.com/ledell/h2oEnsemble-benchmarks/blob/master... You can also do stuff like following in python to execute some code: os.system("scp script_demo.py " + sys.argv[1] + ":/home/ubuntu/") os.system("ssh " + sys.argv[1] + " sudo chmod 777 /home/ubuntu/script_demo.py") os.system("ssh " + sys.argv[1] + " './script_demo.py'") os.system("scp " + sys.argv[1] + ":/home/ubuntu/output/* ./output/") Then it will execute, after it executes - shut it down.
  - viraptor 10y ago
    Even if it may not matter in this case specifically, this is a terrible example. "sudo chmod 777" in public code is basically "I don't know what I'm doing, but go on, do the same thing yourself" :( You don't need sudo, because it's your file. You need just "chmod 755", not "777". And you don't need chmod in the first place - just run "python script_demo.py".
    - oh_sigh 10y ago
      Yes. On the other hand, there isn't a strong overlap with machine learning experts and security experts(I'm sure there will be in coming years though, as security experts start using NNs to detect anomalous/dangerous behavior).
      - viraptor 10y ago
        I know what you mean, but I disagree with "security experts". You don't need to be a security expert, just know the basics of how privileges work on your system. And if people don't, we just need to keep calling it out, because otherwise code like this ends up in production one day, where it actually matters. Simpler bash equivalent for the record: scp script_demo.py "$1" ssh "$1" python script_demo.py scp "$1":output/* ./output/
        anantzoid 10y ago
        Exactly. If you're a ML person, you must be working with Linux in most cases. And in such case, you need to be familiar with the types of permissions, cause that's relevant even during the installation of various libraries, doing ssh etc.
    - danieltillett 10y ago
      These things come about because people learn the simple way to solve all permission problems is just do everything as su (or root) and chmod everything to 777. The problem is it does solve all problems (well sweep them under the rug) - I guess when the only tool you have is a hammer everything looks like a nail.
    - empath75 10y ago
      If you're launching an AWS instance that doesn't have any ports open and is going to terminate in a few hours, who cares?
      - viraptor 10y ago
        As I said: "Even if it may not matter in this case". But what about the next time? (poster obviously doesn't understand the code) What about people who read this comment and follow the advice because it works? (happens all the time with SO) What about when your script becomes the standard deployment method for the company? (happens everywhere) What about people who don't know about file privileges either and blindly copy what they see here? There's lots of things we do that you could say "who cares?" about. Single letter variables. Comments. Const-correctness. Unnecessary N^2 algorithms. Usually it turns out that either you or someone you work with cares a few months/years afterwards. So just learn to do it correctly the first time. Especially if the correct way takes less time than the "magic fix".
  - ClassyJacket 10y ago
    >but was shut down by the "open source office" Can you expand on that? What happened?
    - lettergram 10y ago
      Basically, I couldn't get authorization to publish internal code. In this case, it was a combination of who was going to maintain it and ensuring nothing internal was leaked.
mrfusion 10y ago
What kind of models do y'all make?
anantzoid 10y ago
Spot instances cost way cheaper. The only downside is you need to create an AMI everytime before termination. But, also, AWS g2 has NVIDIA Grid K50 with 4GB memory, so it's not very good with performance.
- alien_ 10y ago
  The AMI creation is only needed if you store data on the machines, which you shouldn't do anyway, not even with on-demand ones. You should always try to keep the instances stateless and store any data outside the instances, such as on S3 or EFS.

danieltillett 10y ago

Would you mind sharing this script? I am thinking of investing in the 2K workstation, but this sounds like a much more efficient way to go.

lettergram 10y ago
I tried to get it released before, but was shut down by the "open source office". So I can't give you the exact script. However, h2o has a script that launches a cluster, that's very similar: https://github.com/ledell/h2oEnsemble-benchmarks/blob/master/ec2/h2o-cluster-launch-instances.py https://github.com/ledell/h2oEnsemble-benchmarks/blob/master... You can also do stuff like following in python to execute some code: os.system("scp script_demo.py " + sys.argv[1] + ":/home/ubuntu/") os.system("ssh " + sys.argv[1] + " sudo chmod 777 /home/ubuntu/script_demo.py") os.system("ssh " + sys.argv[1] + " './script_demo.py'") os.system("scp " + sys.argv[1] + ":/home/ubuntu/output/* ./output/") Then it will execute, after it executes - shut it down.
- viraptor 10y ago
  Even if it may not matter in this case specifically, this is a terrible example. "sudo chmod 777" in public code is basically "I don't know what I'm doing, but go on, do the same thing yourself" :( You don't need sudo, because it's your file. You need just "chmod 755", not "777". And you don't need chmod in the first place - just run "python script_demo.py".
  - oh_sigh 10y ago
    Yes. On the other hand, there isn't a strong overlap with machine learning experts and security experts(I'm sure there will be in coming years though, as security experts start using NNs to detect anomalous/dangerous behavior).
    - viraptor 10y ago
      I know what you mean, but I disagree with "security experts". You don't need to be a security expert, just know the basics of how privileges work on your system. And if people don't, we just need to keep calling it out, because otherwise code like this ends up in production one day, where it actually matters. Simpler bash equivalent for the record: scp script_demo.py "$1" ssh "$1" python script_demo.py scp "$1":output/* ./output/
      - anantzoid 10y ago
        Exactly. If you're a ML person, you must be working with Linux in most cases. And in such case, you need to be familiar with the types of permissions, cause that's relevant even during the installation of various libraries, doing ssh etc.
  - danieltillett 10y ago
    These things come about because people learn the simple way to solve all permission problems is just do everything as su (or root) and chmod everything to 777. The problem is it does solve all problems (well sweep them under the rug) - I guess when the only tool you have is a hammer everything looks like a nail.
  - empath75 10y ago
    If you're launching an AWS instance that doesn't have any ports open and is going to terminate in a few hours, who cares?
    - viraptor 10y ago
      As I said: "Even if it may not matter in this case". But what about the next time? (poster obviously doesn't understand the code) What about people who read this comment and follow the advice because it works? (happens all the time with SO) What about when your script becomes the standard deployment method for the company? (happens everywhere) What about people who don't know about file privileges either and blindly copy what they see here? There's lots of things we do that you could say "who cares?" about. Single letter variables. Comments. Const-correctness. Unnecessary N^2 algorithms. Usually it turns out that either you or someone you work with cares a few months/years afterwards. So just learn to do it correctly the first time. Especially if the correct way takes less time than the "magic fix".
- ClassyJacket 10y ago
  >but was shut down by the "open source office" Can you expand on that? What happened?
  - lettergram 10y ago
    Basically, I couldn't get authorization to publish internal code. In this case, it was a combination of who was going to maintain it and ensuring nothing internal was leaked.

mrfusion 10y ago

What kind of models do y'all make?

anantzoid 10y ago

Spot instances cost way cheaper. The only downside is you need to create an AMI everytime before termination. But, also, AWS g2 has NVIDIA Grid K50 with 4GB memory, so it's not very good with performance.

alien_ 10y ago
The AMI creation is only needed if you store data on the machines, which you shouldn't do anyway, not even with on-demand ones. You should always try to keep the instances stateless and store any data outside the instances, such as on S3 or EFS.