CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 4.50 MiB is free. Process 26899 has 492.00 MiB memory in use. Process 2698 has 492.00 MiB memory in use. Process 27622 has 988.00 MiB memory in use. Process 27623 has 990.00 MiB memory in use. Process 27630 has 988.00 MiB memory in use. Process 27631 has 988.00 MiB memory in use. Process 29104 has 492.00 MiB memory in use. Process 29079 has 492.00 MiB memory in use. Process 29137 has 492.00 MiB memory in use. Process 29140 has 492.00 MiB memory in use. Process 29138 has 492.00 MiB memory in use. Process 29136 has 492.00 MiB memory in use. Process 29103 has 492.00 MiB memory in use. Process 29086 has 492.00 MiB memory in use. Process 29087 has 492.00 MiB memory in use. Process 29120 has 492.00 MiB memory in use. Including non-PyTorch memory, this process has 2.93 GiB memory in use. Process 6476 has 4.56 GiB memory in use. Process 5150 has 2.90 GiB memory in use. Process 5123 has 1.86 GiB memory in use. Of the allocated memory 2.39 GiB is allocated by PyTorch, and 238.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF