CUDA out of memory. Tried to allocate 128.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 23.06 MiB is free. Process 4129 has 492.00 MiB memory in use. Process 4116 has 492.00 MiB memory in use. Process 4098 has 492.00 MiB memory in use. Process 4128 has 492.00 MiB memory in use. Process 4131 has 492.00 MiB memory in use. Process 4096 has 2.11 GiB memory in use. Process 4114 has 1.84 GiB memory in use. Process 17512 has 2.75 GiB memory in use. Including non-PyTorch memory, this process has 3.20 GiB memory in use. Process 13458 has 3.48 GiB memory in use. Process 31843 has 2.72 GiB memory in use. Process 4739 has 492.00 MiB memory in use. Process 17133 has 988.00 MiB memory in use. Process 21776 has 990.00 MiB memory in use. Process 4540 has 990.00 MiB memory in use. Of the allocated memory 2.62 GiB is allocated by PyTorch, and 282.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF