# Loading extra memory in GPU 0 using DDP

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-extra-memory-in-gpu-0-using-ddp%2F43697%3E%5C
**Category:** Intermediate\
**Created:** [June 18, 2023, 2:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-extra-memory-in-gpu-0-using-ddp%2F43697 "2023-06-18T14:33:37Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![AvivSham](https://avatars.discourse-cdn.com/v4/letter/a/59ef9b/32.png) [@AvivSham](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAvivSham)%5C
**Post date:** [June 18, 2023, 2:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-extra-memory-in-gpu-0-using-ddp%2F43697%2F1 "2023-06-18T14:33:37Z")

</div>

Hi All,  
I’m trying to fine-tune Whisper model on a custom dataset using a Multi-GPU machine. Specifically, my machine has 4 v100 GPUs. When running with single GPU i.e. setting the following env variable `CUDA_VISIBLE_DEVICES=0` with batch size of 16 the model trains as expected. However, when training with all 4 GPUs and running with 16 batch size per GPU I get OOM error. Even when reducing the batch size to 8 per GPU the OOM error pops up. Even when running with batch size equal to 4 per GPU, GPU 0 is loaded with ~14GB while the other 3 card use only 8GB. When running with batch size per GPU = 4 it’s even slower compare with running on a single GPU with batch size=16. I’m not sure what I’m doing wrong here, but I assume that if I was able to run batch size=16 on a single GPU I should be able to do the same across 4 cards i.e. batch size of 4\*16 in total.

I will appreciate your help a lot.  
Thanks.