IPSJ SIG Technical Report Vol.2012-ARC-202 No.13 Vol.2012-HPC-137 No /12/13 Tightly Coupled Accelerators 1,a) 1,b) 1,c) 1,d) GPU HA-PACS

Size: px

Start display at page:

Download "IPSJ SIG Technical Report Vol.2012-ARC-202 No.13 Vol.2012-HPC-137 No /12/13 Tightly Coupled Accelerators 1,a) 1,b) 1,c) 1,d) GPU HA-PACS"

そよよしくに
5 years ago
Views:

1 Tightly Coupled Accelerators 1,a) 1,b) 1,c) 1,d) HA-PACS HA-PACS TCA (Tightly Coupled Accelerators) TCA PEACH2 1. (Graphics Processing Unit) HPC GP(General Purpose ) TOP500 [1] CPU PCI Express (PCIe) CPU HA-PACS TCA (Tightly Coupled Accelerators) [2] 1 a) hanawa@ccs.tsukuba.ac.jp b) kodama@cs.tsukuba.ac.jp c) taisuke@cs.tsukuba.ac.jp d) msato@cs.tsukuba.ac.jp 1.1 HA-PACS TCA HPC () HA-PACS :TCA (Tightly Coupled Accelerators) HA-PACS TCA HA- PACS/TCA TCA PCIe TCA 8 16 TCA InfiniBand TCA InfiniBand TCA NVIDIA Kepler Tesla Tesla K20 K20 M c 2012 Information Processing Society of Japan 1

2 [3] K20 Direct Support for RDMA[4] PCIe CPU 2. HA-PACS/TCA PEACH2 2.1 PEACH2 PEACH2 HA-PACS/TCA TCA [5] PEACH2 TCA PCI Express PEARL PCIe PEARL(PCI Express Adaptive and Reliable Link) [6] PCIe PC [7] Ethernet, InfiniBand I/O PCIe Gen 2 2.5GHz, 5GHz Gen 3 8GHz ( 2Gbps, 4Gbps, 8Gbps *1 ) ( x4 ) PCIe CPU CPU Root Complex (RC) EndPoint (EP) PCIe PEARL PEARL PCIe PEACH (PCI Express Adaptive Communication Hub) [8] PCIe RC EP CPU RC PCIe PEACH PCIe PCIe [9] RC EP *1 Gen3 128b130b 7.88Gbps PEARL 3 A A B B (1) A PCI Express A (2) A B CPU (3) B PCI Express B PEARL PCI Express A A B B PEACH2 HA-PACS/TCA FPGA PEACH2 PCIe DMA FPGA PCIe Gen2 x8 4 IP Altera Stratix IV GX[10] FPGA TCA PEACH2 PCIe Gen 2 x8 4 1 CPU PCIe RC EP FPGA partial reconfiguration RC EP 2.2 PEACH2 HA-PACS/TCA 1 PEACH2 PCIe CPU RC 4, IB HCA, PEACH2 EP PCIe PEACH CPU PCIe PCIe PCIe Kepler Tesla K20 CUDA 5.0 Direct Support for RDMA [11] PCIe PCIe c 2012 Information Processing Society of Japan 2

3 G2 x8 G2 x8 G2 x8 G3 x16 PEA CH2 1 G2 x8 0 CPU QPI G3 PCIe x16 1 G3 x16 2 CPU G3 x16 3 HA-PACS/TCA G3 x8 IB HCA To PEACH2 (Root Complex) NIOS (CPU) CPU & side (Endpoint) DMAC Memory Routing function To PEACH2 (Root Complex / Endpoint) To PEACH2 (Endpoint) PEACH2 NVIDIA NDA Direct Support for RDMA PEACH QPI *2 PEACH2 0, 1 1 ( 1 ) PEACH2 HA-PACS PEACH2 ( 2 ) 0, 1 0, 1 (3) Xeon E5 CPU 2 PCIe PCIe 2.3 PEACH2 PEACH2 2 4 PCIe Gen2 x8 N(orth), E(ast), W(est), S(outh) N EP, E EP, W RC PEACH2 S RC EP PEACH2 S DMA chaining DMA DMA *2 2 PEACH2 FPGA DDR3 SDRAM PEACH2 PEACH2 FPGA Altera NIOS Gigabit Ethernet, RS-232C, 2.4 DMA DMA chaining DMA PCIe Altera PCIe IP chaining DMA [12] IP PCIe 255 DMA 2.5 PCIe PEACH2 PCIe 3 PEACH2 PCIe 64bit PEACH2 ( 512Gbyte) *3 *3 512Gbyte BIOS c 2012 Information Processing Society of Japan 3

PCIe 1 PEACH2 Node 3 Node 2 Node 1 Node 0 0 CPU 5 PEACH2 ( ) 0 3 3 4 1 2 PEACH2 PEACH2 Node3 Node2 Node1 Node0 N 0 N 1 N 2 4 *4 PCIe 4 4 PEACH2 N PEACH2 PEACH2 2.

7cm 312mm) Gen2 x8 PCIe 3 (E,W,S) E, W x8, S x16 8 Altera FPGA Stratix IV GX DDR3 SO-DIMM 1 PCIe FPGA Gigabit Ethernet JTAG HA-PACS/TCA PEACH2 PCIe Gen2 IP

4 PCIe 1 PEACH2 Node 3 Node 2 Node 1 Node 0 0 CPU 5 PEACH2 ( ) PEACH2 PEACH2 Node3 Node2 Node1 Node0 N 0 N 1 N 2 4 *4 PCIe 4 4 PEACH2 N PEACH2 PEACH2 2.6 PEACH2 5 PEACH2 *4 BIOS N 3 PCIe [13] (106.7cm 312mm) Gen2 x8 PCIe 3 (E,W,S) E, W x8, S x16 8 Altera FPGA Stratix IV GX DDR3 SO-DIMM 1 PCIe FPGA Gigabit Ethernet JTAG HA-PACS/TCA PEACH2 PCIe Gen2 IP 250MHz 2.7 TCA TCA NVIDIA CUDA [14] CUDA 4.0 PCI UVA (Unified Virtual Addressing) *5 PCI CUDA cudamemcpypeer() UVA cudamemcpy() TCA chaining DMA API 3. PEACH2 FPGA 1 HA-PACS *5 Direct 1 Direct Peer-To-Peer Transfers and Memory Access c 2012 Information Processing Society of Japan 4

5 1 CPU Xeon E5 2.6GHz 2 OS 128GB NVIDIA K20 Intel S2600IP, W2600CR SuperMicro X9DRG-QF Linux, CentOS6.3 kernel {9,14}.1.el6.x86 64 NVIDIA-Linux-x {51,64} CUDA 5.0 PEACH2 Direct Support for RDMA 3.1 DMA PEACH2 FPGA DMA (1) FPGA PCIe ( 2 ) Chaining DMA chaining DMA DMA FPGA CPU (TSC) PEACH2-CPU PEACH2 DMA streaming PEACH2 chaining DMA DMA write PEACH2 DMA read PEACH2 DMA write, DMA read DMA 4Kbyte 3.3 Gbyte/sec DMA write PEACH2 DMA CPU PCIe PCIe Gen2 x8 4Gbyte/sec PEACH2-256byte 16byte 2byte, LCRC 4byte, 1byte, 1byte 256 4Gbyte/sec =3.66Gbyte/sec 93% PEACH2 PCIe IP Chaining DMA DMA write DMA read CPU write 2Kbyte write chaining DMA DMA 8 4Kbyte % 2 7 8Kbyte PEACH2-(K20) FPGA chaining DMA CUDA driver API ( 1 ) cumemalloc() (2) () PCIe (3) FPGA chaining DMA ( 4 ) cumemhostalloc() cumemcpydtoh() FPGA (3) PEACH2-CPU DMA PCIe (BAR) CPU Kbyte DMA CPU DMA CPU DMA 830MB/s CPU PCIe QPI c 2012 Information Processing Society of Japan 5

6 3500 Mbytes/sec CPU(write) CPU(read) K20(write) K20(read) Mbytes/sec CPU(write) CPU(read) K20(write) K20(read) Bytes Bytes 6 PEACH2-CPU, DMA (255 ) 7 PEACH2-CPU, DMA (1 ) CPU(write) CPU(read) Mbytes/sec CPU(write) CPU(read) K20(write) K20(read) Mbytes/sec RemoteCPU (write) Counts Bytes 8 PEACH2-CPU, DMA (4096byte ) 9 PEACH2-CPU, CPU DMA (255 ) DMA 100MB/s 3.2 PEACH2 PEACH2 DMA PIO( CPU store ) (1) 1PEACH2 2 (A, B ) A, B PCIe ( 2 ) PEACH2 (3) A PCIe B 4byte store (4) A B (5) B (6) (2) 1 PC 782ns DMA PEACH2 CPU DMA PEACH2 4Kbyte DMA DMA DMA PEACH2 PEACH2 DMA c 2012 Information Processing Society of Japan 6

7 DMA DMA 4. PCIe Non transparent bridge (NTB) [15] NTB PCIe EP 2 Endpoint 2 RC NTB PCIe PCIe BIOS EP PEACH2 PCIe -PEACH2 APEnet+[16], [17] FPGA 3D Torus Fermi Direct APEnet+ QSFP+ PCIe 2.2 NVIDIA CUDA 5 CUDA Computing Capability 3.5 Direct Support for RDMA RDMA [4] InfiniBand [18] PEACH2 Direct Support for RDMA InfiniBand PCIe TCA MPI 5. HA-PACS/TCA PEACH2 DMA PEACH HA-PACS TCA API TCA HA- PACS/TCA 800TFlops 1PFlops HA-PACS TCA Joel Scherpelz NVIDIA NVIDIA JAPAN JST-CREST [1] Dongarra, J., Meuer, H., Stromaier, E. and Simon, H.: TOP500 List, [2] Tightly Coupled Accelerators ( ) Vol ARC-201, No. 26, pp. 1 8 (2012). [3] NVIDIA Corp.: NVIDIA Tesla Kepler Computing Accelerators. KSeriesOverviewLR.pdf. [4] NVIDIA Corp.: NVIDIA Direct. [5] HA-PACS () Vol HPC-130, No. 21, pp. 1 7 (2011). [6] Hanawa, T., Boku, T., Miura, S., Okamoto, T., Sato, M. and Arimoto, K.: Low-Power and High-Performance Communication Mechanism for Dependable Embedded Systems, Proceedings of 2008 International Workshop on Innovative Architecture for Future Generation Processors and Systems, pp (2008). [7] PCI-SIG: PCI Express Base Specification, Rev. 3.0 (2010). [8] Otani, S., Kondo, H., Nonomura, I., Uemura, M., Hayakawa, Y., Oshita, T., Kaneko, S., Asahina, K., Arimoto, K., Miura, S., Hanawa, T., Boku, T. and Sato, M.: An 80Gbps Dependable Communication SoC with PCI Express I/F and 8 CPUs, 2011 IEEE International Solid-State Circuits Conference, pp (2011). [9] PCI-SIG: PCI Express External Cabling Specification, Rev. 1.0 (2007). c 2012 Information Processing Society of Japan 7

8 [10] Altera Corp.: Stratix IV Device Handbook. [11] NVIDIA Corp.: Developing A Linux Kernel Module Using RDMA For Direct /rc/docs/Direct RDMA.pdf. [12] Altera Corp.: IP Compiler for PCI Express user guide. pci express.pdf. [13] PCI-SIG: PCI Express Card Electromechanical (CEM) Specification, Rev. 2.0 (2007). [14] NVIDIA Corp.: NVIDIA CUDA: Compute Unified Device Architecture. [15] Gudmundson, J.: Enabling Multi-Host System Designs with PCI Express Technology, (2004). [16] Ammendola, R. et al.: APEnet+: high bandwidth 3D torus direct network for petaflops scale commodity clusters, Journal of Physics, Conference Series, Vol. 331, Part 5, No. 5 (2011). [17] Rosetti, D. et al.: Leveraging NVIDIA Direct on APEnet+ 3D Torus Cluster Interconnect (2012). GTC2012/PresentationPDF/S0282-GTC Torus-Cluster.pdf. [18] Mellanox Technologies: Mellanox OFED Direct, products dyn&product family=116&menu section=34. c 2012 Information Processing Society of Japan 8

07-二村幸孝・出口大輔.indd

07-二村幸孝・出口大輔.indd GPU Graphics Processing Units HPC High Performance Computing GPU GPGPU General-Purpose computation on GPU CPU GPU GPU *1 Intel Quad-Core Xeon E5472 3.0 GHz 2 6 MB L2 cache 1600 MHz FSB 80 GFlops 1 nvidia