CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 17.75 MiB is free. Process 28377 has 990.00 MiB memory in use. Process 28378 has 990.00 MiB memory in use. Process 29944 has 492.00 MiB memory in use. Process 29947 has 492.00 MiB memory in use. Process 29946 has 492.00 MiB memory in use. Process 30011 has 492.00 MiB memory in use. Process 30015 has 492.00 MiB memory in use. Process 29963 has 492.00 MiB memory in use. Process 29945 has 492.00 MiB memory in use. Process 29994 has 492.00 MiB memory in use. Process 30038 has 492.00 MiB memory in use. Process 30013 has 492.00 MiB memory in use. Including non-PyTorch memory, this process has 2.62 GiB memory in use. Process 31249 has 2.40 GiB memory in use. Process 31250 has 6.38 GiB memory in use. Process 31251 has 3.75 GiB memory in use. Of the allocated memory 2.16 GiB is allocated by PyTorch, and 160.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF