DualPipe: An innovative bidirectional pipeline parallelism algorithm

DualPipe DualPipe is an innovative bidirectional pipeline parallelism algorithm introduced in the DeepSeek-V3 Technical Report. It achieves full overlap of forward and backward computation-communication phases, also reducing pipeline bubbles. For detailed information on computation-communication overlap, please refer to the profile data. Schedules Example DualPipe scheduling for 8 PP ranks and 20 micro-batches in two directions. The micro-batches in the reverse direction are symmetric to those in the forward direction, so we omit their batch ID for illustration simplicity. Two cells enclosed by a shared black border have mutually overlapped computation and communication Pipeline Bubbles and Memory Usage Comparison Method Bubble Parameter Activation 1F1B (PP-1)(𝐹+𝐵) 1× PP ZB1P (PP-1)(𝐹+𝐵-2𝑊) 1× PP DualPipe (PP/2-1)(𝐹&𝐵+𝐵-3𝑊) 2× PP+1 𝐹 denotes the execution time of a forward chunk, 𝐵 denotes the execution time of a full backward chunk, 𝑊 denotes the execution time of a "backward fo ... Read full article.