Why My DIPY Array Jobs Were Taking Forever on HPC
I was running MP-PCA denoising on diffusion MRI data using DIPY on an HPC cluster. The jobs started normally, but the runtimes were wildly inconsistent. Some scans finished in a couple of hours, others took 16 hours, and many more hit the 24-hour walltime limit without finishing or reporting any errors. It took me longer than I’d like to admit to figure out why. At first, I thought some scans were simply more computationally demanding, or that certain jobs were running on slower cluster nodes. After exploring a few possibilities, I finally traced the problem to thread oversubscription in the underlying numerical libraries. ...