3 ms·

Is there any documentation regarding how this allocates / deallocates GPU Memory. While developing Deep Video Analytics [1] one of the biggest issue I have face

Is there any documentation regarding how this allocates / deallocates GPU Memory. While developing Deep Video Analytics [1] one of the biggest issue I have faced is TensorFlow refusing to deallocate GPU memory [2] (freeing up for other processes to use) unless the process is killed/exits. This in combination with allow-growth allocation strategy used by TF makes it unpredictable to run mixed loads. E.g. I would prefer to run a Detector, Indexer along with some computation in a GPU data frame etc. at same time on a single GPU. [1] https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/ https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/ [2] https://github.com/tensorflow/tensorflow/issues/1578 https://github.com/tensorflow/tensorflow/issues/1578

jacquesm 9y ago
Funny I was reading about that yesterday, here is a snippet that might help you if you do not need all memory in one process: options = tf.GPUOptions(per_process_gpu_memory_fraction=0.5) session = tf.Session(config=tf.ConfigProto(gpu_options=options)) That way you limit how much the GPU can use up front and then other applications can use it instead. Of course if you need all of the memory during one phase of the computation, want to free it and then re-allocate it at a later stage you are probably out of luck. But if your pipeline would normally allow for all tasks to run with limited memory you might get away with this strategy at the expense of some speed. Good luck!
- aub3bhat 9y ago
  I use per_process_gpu_memory_fraction extensively [1], but when you are writing code that runs on different GPUs (Titan vs 1080 vs 1070 vs K80) due to change in the total memory available you run risk of under allocating or over allocating GPU memory. [1] https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/blob/master/dvalib/indexer.py#L123 https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/blob/maste...
  - jacquesm 9y ago
    Ah yes, I get your problem now, you have to deal with changing environments, not just your own setup to solve a particular problem. I see. That makes it a lot more complicated. Is there a way to fold the pipeline into a single process? Not pretty from a modularity point of view but it just might get the job done.
    - aub3bhat 9y ago
      I am using Celery workers which run a single tensorflow model in solo concurrency mode. The current workaround is to keep track of per process gpu fraction and then incorporate information about Total GPU memory available when assigning workers/celery process to individual GPUs.
      - jacquesm 9y ago
        Right, so essentially you are doing per process memory management because the layers lower down aren't smart enough to handle this transparently. I'm fairly sure that the nvidia libraries do support releasing memory and from what I see on my own machine memory usage of the GPU does go up and down when running longer jobs. So maybe it isn't so much a structural problem as a problem in one particular module that you exercise a lot?

jacquesm 9y ago

Funny I was reading about that yesterday, here is a snippet that might help you if you do not need all memory in one process: options = tf.GPUOptions(per_process_gpu_memory_fraction=0.5) session = tf.Session(config=tf.ConfigProto(gpu_options=options)) That way you limit how much the GPU can use up front and then other applications can use it instead. Of course if you need all of the memory during one phase of the computation, want to free it and then re-allocate it at a later stage you are probably out of luck. But if your pipeline would normally allow for all tasks to run with limited memory you might get away with this strategy at the expense of some speed. Good luck!

aub3bhat 9y ago
I use per_process_gpu_memory_fraction extensively [1], but when you are writing code that runs on different GPUs (Titan vs 1080 vs 1070 vs K80) due to change in the total memory available you run risk of under allocating or over allocating GPU memory. [1] https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/blob/master/dvalib/indexer.py#L123 https://github.com/AKSHAYUBHAT/DeepVideoAnalytics/blob/maste...
- jacquesm 9y ago
  Ah yes, I get your problem now, you have to deal with changing environments, not just your own setup to solve a particular problem. I see. That makes it a lot more complicated. Is there a way to fold the pipeline into a single process? Not pretty from a modularity point of view but it just might get the job done.
  - aub3bhat 9y ago
    I am using Celery workers which run a single tensorflow model in solo concurrency mode. The current workaround is to keep track of per process gpu fraction and then incorporate information about Total GPU memory available when assigning workers/celery process to individual GPUs.
    - jacquesm 9y ago
      Right, so essentially you are doing per process memory management because the layers lower down aren't smart enough to handle this transparently. I'm fairly sure that the nvidia libraries do support releasing memory and from what I see on my own machine memory usage of the GPU does go up and down when running longer jobs. So maybe it isn't so much a structural problem as a problem in one particular module that you exercise a lot?