CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 21.69 MiB is free. Process 19746 has 492.00 MiB memory in use. Process 19768 has 492.00 MiB memory in use. Process 21956 has 492.00 MiB memory in use. Process 21957 has 492.00 MiB memory in use. Process 21959 has 492.00 MiB memory in use. Process 22302 has 1.67 GiB memory in use. Including non-PyTorch memory, this process has 3.02 GiB memory in use. Process 22310 has 1.55 GiB memory in use. Process 22309 has 2.75 GiB memory in use. Process 22306 has 2.09 GiB memory in use. Process 22305 has 2.93 GiB memory in use. Process 17023 has 492.00 MiB memory in use. Process 7316 has 5.00 GiB memory in use. Of the allocated memory 2.58 GiB is allocated by PyTorch, and 134.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF