Multiprocessing
Multiprocessing refers to the capability of two or more processors to concurrently execute multiple independent program executables called processes. Since processes share nothing but a file system, multiprocess programs are restricted to problems that can be split into entirely independent tasks. Inter-process communication is done by sending (time absorbing) messages between nodes using libraries like OpenMPI.
Note! zombie processes which have completed execution, but are not labeled as terminated.
Since in multiprocess programs each process runs independently, if one crashes, the other will execute their work. This may lead to
Examples
Naive ManyMatMul
Suppose we have to calculate ~100,000 matrix multiplications. We have 200,000 files containing each and matrix, and we output 100,000 files containing . In the naive approach, we have 1000 processes that each complete 100 matrix multiplications, such that:
- Process 0 does tasks 0 - 99.
- Process 1 does tasks 100 - 199.
- Process 999 does tasks 999,900 - 999,999.
However, load balancing for this approach does not work well if each A or B matrix isn’t of the same size. We therefore need a way to dynamically allocate work.