In the world of computing being able to execute multiple commands at the same time is a big deal. Even early operating system were able to execute multiple commands at once. Now days multiple execute is corner stone to everything from virtualization hardware to operating systems to games and audio / video processing. In fact, I beat you have more than one browser tab open right now, and probably running more than one program as well. Multi-core processors increase parallel command execution even further!
Why, then, is it not easier to run multiple commands in parallel in user space? Maybe I want to download 3 files at the same time? Maybe I want to perf. compare three scripts at the same time. Or any number of uses of parallel execution. I am not talking about running a program in the background with
command 1 & nor series execution with
&&. True same-time process execution
One of the best Linux tools to run commands in parallel is, maybe not surprisingly, called parallel. Go figure, a tool named after what it does :S. It is available via most Linux distributions package manager. The –help is helpful and the
man parellel is in depth. Some of it can seem daunting but remember, no one knows everything. Just learn what you can
Before trying out the new fancy tool lets get a base line. I created a script downloads a 5MB file three times, in a loop of three. So 9 downloads of 5MB is about 45MB (./lib/test_0.sh). Here is the result of three runs.
Iteration 1: 1m53s Iteration 2: 1m59s Iteration 3: 2m8s Average: ~ 2m
Now, we run the same
wget command against the same three targets, but this time via parallel (test_1.sh). Once again lets do it three times to get an average.
Iteration 1: 1m8s Iteration 2: 1m22s Iteration 3: 1m30s Average: 1m20s
Is this a synthetic test limited by my bandwidth? Sure, but even in this small example it saved approx. 30% per script execution. Imagine larger, longer running processes; or even more processes. With
parallel each iteration only takes as long as the longest response.
parallel as an async tool for Linux CLI. Where you were once limited to running commands in series you can now execute in parallel. Want to move an entire directory of files as fast as the disk can handle? Iterate the file list and
mv via parallel. Want to download 5 files at the same time?
parallel it. Want to run all your clean up scripts at the same time?
This article was just an introduction to the
parallel tool. In the next article in this series I will show how to use the
parallel tool in a automated pipeline to decrease test execution time. Stay tuned for that one.