CUDA out of memory. Tried to allocate 128.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 52.38 MiB is free. Process 6040 has 492.00 MiB memory in use. Process 6039 has 492.00 MiB memory in use. Process 6042 has 492.00 MiB memory in use. Process 6044 has 492.00 MiB memory in use. Process 6046 has 492.00 MiB memory in use. Process 6038 has 492.00 MiB memory in use. Process 6043 has 492.00 MiB memory in use. Process 6037 has 492.00 MiB memory in use. Process 6045 has 492.00 MiB memory in use. Process 6920 has 3.49 GiB memory in use. Process 6921 has 2.36 GiB memory in use. Process 7503 has 492.00 MiB memory in use. Process 2422 has 990.00 MiB memory in use. Process 2421 has 990.00 MiB memory in use. Process 4224 has 3.66 GiB memory in use. Including non-PyTorch memory, this process has 1.80 GiB memory in use. Process 13376 has 3.79 GiB memory in use. Of the allocated memory 1.33 GiB is allocated by PyTorch, and 162.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF