The Situation
In the world of computing being able to execute multiple commands at the same time is a big deal. Even early operating system were able to execute multiple commands at once. Now days multiple execute is corner stone to everything from virtualization hardware to operating systems to games and audio / video processing. In fact, I beat you have more than one browser tab open right now, and probably running more than one program as well. Multi-core processors increase parallel command execution even further!
Why, then, is it not easier to run multiple commands in parallel in user space? Maybe I want to download 3 files at the same time? Maybe I want to perf. compare three scripts at the same time. Or any number of uses of parallel execution. I am not talking about running a program in the background with command 1 & nor series execution with &&. True same-time process execution
The Solution
One of the best Linux tools to run commands in parallel is, maybe not surprisingly, called parallel. Go figure, a tool named after what it does :S. It is available via most Linux distributions package manager. The –help is helpful and the man parellel is in depth. Some of it can seem daunting but remember, no one knows everything. Just learn what you can
An Example
Before trying out the new fancy tool lets get a base line. I created a script downloads a 5MB file three times, in a loop of three. So 9 downloads of 5MB is about 45MB (./lib/test_0.sh). Here is the result of three runs.
Iteration 1: 1m53s
Iteration 2: 1m59s
Iteration 3: 2m8s
Average: ~ 2mNow, we run the same wget command against the same three targets, but this time via parallel (test_1.sh). Once again lets do it three times to get an average.
Iteration 1: 1m8s
Iteration 2: 1m22s
Iteration 3: 1m30s
Average: 1m20sIs this a synthetic test limited by my bandwidth? Sure, but even in this small example it saved approx. 30% per script execution. Imagine larger, longer running processes; or even more processes. With parallel each iteration only takes as long as the longest response.
In Conclusion
Think of parallel as an async tool for Linux CLI. Where you were once limited to running commands in series you can now execute in parallel. Want to move an entire directory of files as fast as the disk can handle? Iterate the file list and mv via parallel. Want to download 5 files at the same time? parallel it. Want to run all your clean up scripts at the same time? parallel them.
Soon(tm)
This article was just an introduction to the parallel tool. In the next article in this series I will show how to use the parallel tool in a automated pipeline to decrease test execution time. Stay tuned for that one.
