For that kind of data or larger, I would avoid R and Python and move to writing my own algorithms or try out something for more heavy duty analysis such as Maho

For that kind of data or larger, I would avoid R and Python and move to writing my own algorithms or try out something for more heavy duty analysis such as Mahout or Spark. R and Python are still one box and memory constrained.