CUDA out of memory. Tried to allocate 128.00 MiB. GPU 0 has a total capacty of 21.99 GiB of which 50.44 MiB is free. Process 8324 has 492.00 MiB memory in use. Process 8343 has 492.00 MiB memory in use. Process 8325 has 492.00 MiB memory in use. Process 8362 has 492.00 MiB memory in use. Process 8379 has 492.00 MiB memory in use. Process 8344 has 492.00 MiB memory in use. Process 8326 has 492.00 MiB memory in use. Process 8342 has 492.00 MiB memory in use. Process 8345 has 492.00 MiB memory in use. Process 8363 has 492.00 MiB memory in use. Process 9280 has 1.88 GiB memory in use. Process 9279 has 1.63 GiB memory in use. Process 9282 has 2.90 GiB memory in use. Including non-PyTorch memory, this process has 2.65 GiB memory in use. Process 8659 has 492.00 MiB memory in use. Process 8663 has 492.00 MiB memory in use. Process 14490 has 7.03 GiB memory in use. Of the allocated memory 2.18 GiB is allocated by PyTorch, and 166.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF