先進的計算基盤システムシンポジウム DMA Tofu 6 MPI RDMA 6 3 (1 ) RDMA (2 ) 3 MPI MPI 3 MPI 127us, 47GB/s 9,216 MPI Bcast 106GB/s 31 MPI 2 MPI 2 Tofu Eager : 6 7 2

Size: px

Start display at page:

Download "先進的計算基盤システムシンポジウム DMA Tofu 6 MPI RDMA 6 3 (1 ) RDMA (2 ) 3 MPI MPI 3 MPI 127us, 47GB/s 9,216 MPI Bcast 106GB/s 31 MPI 2 MPI 2 Tofu Eager : 6 7 2"

きみつぐつねざき
5 years ago
Views:

1 先進的計算基盤システムシンポジウム MPI 82,944 MPI Tofu 6 RDMA MPI MPI 127us, MPI 47GB/s 9,216 MPI Bcast 106GB/s The Design of MPI Communication Facility for K computer Shinji Sumimoto, Takahiro Kawashima, Naoyuki Shida, Takayuki Okamoto, Kenichi Miura, Atsuya Uno, Motoyoshi Kurokawa, Fumiyoshi Shouji and Mitsuo Yokokawa This paper describes the design of high performance MPI communication which enables high performance communication with minimized memory usage on the 82,944 node K computer The Tofu interconnect of K computer uses six dimension torus/mesh direct topology for realizing higher performance and availability on hundreds thousand node system However, in such a ultra scale system, communication performance degradation by increasing hops and network congestion and much memory consumption with increasing number of nodes are still problems to solve To solve the problems, MPI communication facility of the K computer uses RDMA based communication and buffer allocation policy that limits buffer based communication to neighbor communication and the other uses less-memory usage communication method As a result of these designs, MPI communication facility of the K computer realizes 127us MPI communication latency and 47GB/s MPI communication bandwidth with less memory usage, and a collective communication of MPI Bcast achieves 106GB/s on 9,216 node K computer TOP500 1) Road Runner 2) 10PFLOPS Linpack HPC TOP500 (HPCI) 3) 105PFLOPS 1 HPC MPI, FUJITSU, RIKEN 82,944 6 Tofu 4) 6) 6 MPI Open MPI 7) Open MPI send-receive 237

2 先進的計算基盤システムシンポジウム DMA Tofu 6 MPI RDMA 6 3 (1 ) RDMA (2 ) 3 MPI MPI 3 MPI 127us, 47GB/s 9,216 MPI Bcast 106GB/s 31 MPI 2 MPI 2 Tofu Eager : Tofu Tofu Tofu Tofu 6 Rendezvous : Tofu 6 ( : X,Y,Z,A,B,C) RDMA 32x32x32x2x3x2(393,216 ) 24x18x17x2x3x2(88,128 ) 24x18x16x2x3x2(82,944 (Zero-Copy ) ) Y,A,C A,B,C MPICH 8) Open MPI Eager Rendezvous 12 4 Tofu (TNI) 32 MPI 31 MPI STAG( ) Eager Rendezvous RDMA STAG 10 RDMA RDMA Descriptor Piggy Back (1 ) 66 ( 33 ) RDMA Descriptor (2 ) Direct Descriptor Piggy Back RDMA Direct Descriptor (3 ) MPI Open MPI 238

先進的計算基盤システムシンポジウム 10 256GB 16GB (MTU) MPI InfiniBand Re- ceive Queue 33 MPI 31,32 MPI RDMA Direct: Rendezvous 31,32 RDMA Tofu Send/Recieve Pipeline: Tofu InfiniBand RDMA Eager Eager MPI Eager MPI

3 先進的計算基盤システムシンポジウム GB 16GB (MTU) MPI InfiniBand Re- ceive Queue 33 MPI 31,32 MPI RDMA Direct: Rendezvous 31,32 RDMA Tofu Send/Recieve Pipeline: Tofu InfiniBand RDMA Eager Eager MPI Eager MPI Eager Queue 1 Open MPI Open MPI 1 Open 4 MPI MPI COLL PML,BML,BTL MPI API SPARC Open RDMA MPI 3 MPI 42 MPI 41 Open MPI 33 MPI Open MPI MPI InfiniBand 3 Eager RDMA: 3 RDMA Eager MPI MPI( ) Tofu Eager Send/Receive: Eager 3 InfiniBand Send/Receive (MPI) Eager 128KBx2 239

先進的計算基盤システムシンポジウム 43 3 41 InfiniBand RDMA 3 : (1,2,3 Receive Queue ) 6 Eager RDMA, Eager Send/Receive, Direct RDMA 3 Tofu RDMA 2 Eager Send/Receive : Send/Receive Pipeline RDMA Pipeline (Read,Write)

4 先進的計算基盤システムシンポジウム InfiniBand RDMA 3 : (1,2,3 Receive Queue ) 6 Eager RDMA, Eager Send/Receive, Direct RDMA 3 Tofu RDMA 2 Eager Send/Receive : Send/Receive Pipeline RDMA Pipeline (Read,Write) (Barrier, Reduce, RDMA Pipeline Bcast, Allreduce) TNI Tofu 091us 5) 44 MPI MPI MPI Open MPI Tofu Open PML, BML, BTL 3 MPI MPI PML, BML, BTL 33 Open MPI Tofu 1 Open MPI 100ns 2 Eager 2 MPI Eager Eager Open MPI (Tofu Rendezvous LLP) (Tuned coll, TBI coll) Eager RDMA PML, BML, BTL RDMA TNI Pipeline Open MPI 240

5 先進的計算基盤システムシンポジウム Rendezvous MPI Open MPI Tofu RDMA 5 tuned COLL, tbi COLL 43 RDMA TNI Tofu STAG Open MPI PML RDMA MPI Bcast MPI Allreduce 9) MPI Bcast MPI Allreduce RDMA Descriptor Trinaryx3 Piggy Back RDMA RDMA 64bit 8 Descriptor Direct Descriptor 1 RDMA MPI Rendezvous MPI Tofu LLP Open MPI (1 ) Init PML (2 ) (3 ) Eager RDMA : Eager RDMA (4 ) MPI RDMA (5 ) (128 ) MPI 0-16 Piggy Back RDMA Direct Descriptor DMA Eager RDMA ( 44 ) Eager RDMA 32KB(128B x 256) Eager Send : 3 RDMA Tofu 3 Eager RDMA Direct RDMA 1 2 Eager Send Eager 1MB Send RDMA Direct 2KB 1 18KB 40 60KB Direct RDMA : Direct RDMA 241

6 先進的計算基盤システムシンポジウム TNI MPI Eager Send 9,216 (3 48x6x32) Eager RDMA SPARC64 VIIIfx(8 Core,2GHz) + ICC 16GB Tofu 6D Torus/mesh(5GB/s(x2)x10link) OS Linux SPARC RDMA Data Polling 128 RDMA 128 RDMA Eager RDMA MPI IMB(Intel MPI Benchmarks) 8Core 1 1 (1Core) MPI Data Polling,Hard Queue Polling MPI (16 ) 5 4 ( ) 5 Eager Send Direct RDMA 18KB 62 6 MPI MPI MPI 47GB/s 7 TNI 2 7 Tofu TNI 3 TNI 4 4,4 CPU ICC 091us 32 16GB/s Piggy Back RDMA MPI RDMA MPI MPI MPI Piggy Back RDMA 8 9,216 MPI Bcast 127us 96 Open MPI Chain 242

7 先進的計算基盤システムシンポジウム 6 9 MPI Allreduce Recursive Doublling Trinaryx3 71GB/s Open MPI Ring 14GB/s MPI (1,024) 3 Simple Spread AlltoAll MPI 7 TNI 10 8 MPI Bcast Trinaryx3 9, GB( 106sGB/s Open MPI MAX=1024), 129GB( ) 014GB( Chain 08GB/s ) 008GB(MPI Init/Finalize) 1325 MPI Init/Finalize 006GB 9 9,216 MPI Allreduce 9) Open MPI Ring 10 9,216 MPI Init/Finalize Simple Spread AlltoAll 243

8 先進的計算基盤システムシンポジウム 129GB 16GB 80% GB 10% 7 10,000 CRAY XT-5 IBM Blue Gene CRAY XT-5 CRAY XT-5 MPI MPICH2 CRAY MPI MPI Open MPI Tofu 1 1 CRAY XT-6 (NIC) RDMA NIC XT-5 1) Super Computer TOP500: MPI 2) Los Alamos Lab Roadrunner: IBM Blue Gene/L,P,Q IBM Blue Gene/ L,P,Q MPI MPICH 3) : 10) Tofu 1 4) Yuichiro Ajima, Shinji Sumimoto, and Toshiyuki Shimizu Tofu: A 6d mesh/torus interconnect Tofu for exascale computers In IEEE Computer, pp, 36 40, Nov ) Yuichiro Ajima, Yuzo Takagi, Tomohiro Inoue, Shinya Hiramoto, and Toshiyuki Shimizu The tofu interconnect In Hot Interconnects, pp 87 Tofu 94, ) ( Blue Gene Q 80 7, pp , ) OpenMPI: 8) MPICH2: MPICH Open projects/mpich2/ MPI 9),,,,,,,,, MPI Allreduce 2011-ARC-197, Nov ) George Almási, Philip Heidelberger, Charles J Archer, Xavier Martorell, C Chris Erway, 82,944 José E Moreira, B Steinmacher-Burow, and MPI Yili Zheng Optimization of mpi collective communication on bluegene/l systems In Proceed- Tofu ings of the 19th annual international conference on Supercomputing, ICS 05, pp , New 6 York, NY, USA, 2005 ACM RDMA MPI 127us, 47GB/s 9216 MPI Bcast 106GB/s Open MPI 244

[4] ACP (Advanced Communication Primitives) [1] ACP ACP [2] ACP Tofu UDP [3] HPC InfiniBand InfiniBand ACP 2 ACP, 3 InfiniBand ACP 4 5 ACP 2. ACP ACP

[4] ACP (Advanced Communication Primitives) [1] ACP ACP [2] ACP Tofu UDP [3] HPC InfiniBand InfiniBand ACP 2 ACP, 3 InfiniBand ACP 4 5 ACP 2. ACP ACP InfiniBand ACP 1,5,a) 1,5,b) 2,5 1,5 4,5 3,5 2,5 ACE (Advanced Communication for Exa) ACP (Advanced Communication Primitives) HPC InfiniBand ACP InfiniBand ACP ACP InfiniBand Open MPI 20% InfiniBand Implementation