One of the Things I Enjoy About My Work

Our lab processes MRI data using the HCP Minimal Preprocessing Pipelines through QuNex , a software suite that provides a standardized interface for running neuroimaging pipelines. QuNex manages much of the workflow, including how data is organized, how different processing steps are called, and where inputs and outputs are expected to be.

A former lab member set up the whole system years ago, along with additional scripts and configurations for running everything on our cluster. By the time I started using it, the workflow was well established: update the session IDs, submit the jobs, and wait for the results. Everything else happened behind layers of scripts I had never really needed to look at.

That changed recently when we wanted to add two preprocessing steps to our diffusion MRI data: MP-PCA denoising and Gibbs ringing correction. Both are commonly used to improve diffusion data quality and are recommended in the ENIGMA-DTI Preprocessing Guidelines .

Anyone who has worked with MRI data knows that changing preprocessing steps can be quite a hassle. These pipelines involve many sequential operations, and the output of one step often becomes the input to the next. Changing something early in the process can mean rerunning a substantial portion of the pipeline, sometimes almost from scratch.

And of course, this was one of those cases. We needed to go back to the raw diffusion data, apply the two additional steps, and then run the HCP diffusion preprocessing again.

Working within the constraints

The complication was that these weren’t steps I could simply enable by changing a command-line argument in QuNex. They had to be run separately, outside of QuNex, and I needed to figure out how to feed the resulting data back into the existing pipeline.

And there were a few other things to consider.

Our lab already had hundreds of sessions processed through the original pipeline, and those outputs were being used in ongoing analyses. I couldn’t risk renaming or reorganizing the existing directories, so I needed to leave everything untouched and keep the new results completely separate.

QuNex also expects the HCP diffusion inputs to follow a specific directory structure and naming convention. I couldn’t simply point it to my newly preprocessed files and expect everything to work. Modifying QuNex itself or rewriting the existing scripts wasn’t an option either.

One possibility was to copy the existing data into a separate directory and work from there. But with hundreds of MRI sessions, duplicating all that data wasn’t particularly appealing. MRI datasets take up enough space already.

So the challenge was to find a way to sneak the additional preprocessing steps into the existing workflow without disrupting anything that was already there.

Following the Trail

To figure out what was possible, I started digging into the existing scripts, following how QuNex called the underlying HCP pipeline and where different inputs and outputs were coming from.

Untangling the web of interconnected scripts was quite an exercise. One script would lead to another, sometimes several others, and I had to trace the data through each stage, figure out what was happening behind the scenes, and track down the various input files, parameters, outputs, and logs along the way.

I wasn’t trying to rewrite the pipeline. The goal was to understand it well enough to introduce the modified data while keeping everything else working as before.

Finding a workaround

Eventually, I figured out that I could make the pipeline use my newly preprocessed diffusion data, as long as the files appeared in the locations and under the names it expected.

The catch was that the pipeline also needed a whole collection of supporting files from earlier preprocessing steps. I could copy everything into a new directory, but that would mean duplicating a lot of unchanged data.

Instead, I created a separate directory that mimicked the structure of the original one. I put the newly preprocessed diffusion files where the pipeline expected them, and used symbolic links for the remaining files.

Symbolic links are essentially pointers to files stored elsewhere. They let you reference existing files from another location without actually copying them.

So from the pipeline’s perspective, the new directory looked just like a complete set of inputs. In reality, most of it was simply pointing back to the original files, while the modified diffusion data was stored separately. A bit of filesystem trickery, but exactly what I needed.

I also dug through the QuNex documentation and found a command-line argument that allowed me to redirect the outputs to a separate location, without having to modify the underlying scripts.

After putting everything together and testing it, the pipeline ran! It picked up the newly preprocessed data, wrote the results to the right place, and left the original files and outputs untouched. No unnecessary duplication, and no changes to the existing workflow.

More Than Just Getting It Done

It took me a few days to figure all of this out, from understanding the existing pipeline to coming up with a workaround and testing everything on the server.

Could an AI agent have done it faster? Possibly! AI has been a huge productivity boost for me, and I’ve gotten pretty used to letting AI agents take care of entire tasks for my personal projects. But this was a little different.

The pipeline was part of a shared environment used by the entire lab, and I couldn’t risk disrupting anyone else’s work. Even with detailed documentation and context, an agent could still overlook an important dependency or make an incorrect assumption. Of course, I could have asked an agent to document everything it did, but reading through that documentation wouldn’t have given me quite the same level of understanding or confidence in the changes as working through them myself.

Besides, I genuinely enjoyed the detective work! There’s something really satisfying about digging into a problem that doesn’t have an obvious solution, gradually piecing the puzzle together, and finally getting everything to work after all that effort.