Distributed Systems
Parallel AI
Parallel AI refers to the application of parallel and distributed computing techniques to artificial intelligence workloads, especially the training and serving of large machine-learning models. The approach encompasses data, model, pipeline, and expert parallelism, along with optimizer and memory sharding, to scale computation across multi-GPU, multi-node, and heterogeneous systems.
Parallel computing
Parallel computing is the design and use of computer systems that perform multiple computations simultaneously by dividing a problem into parts executed at the same time. It underpins high‑performance computing, large‑scale data processing, and modern AI/ML workloads, and spans architectures from multicore CPUs and GPUs to distributed clusters and supercomputers.