PC RWC 01/10/31 2 1
SCore 1,024 PC SCore III PC 01/10/31 3 SCore SCore Aug. 1995 Feb. 1996 Oct. 1996 1997-1998 Oct. 1999 Oct. 2000 April. 2001 01/10/31 4 2
SCore University of Bonn, Germany University of Heidelberg, Germany University of Tuebingen, Germany Oxford University, England Warwick University, England 01/10/31 5 Host PC RWC SCore III NEC Express Servers Dual Pentium III 933 MHz 512 Mbytes of Main Memory # of Hosts 512 Hosts (1,024 Processors) Networks Myrinet-2000 2 Ethernet Links Linpack Result 618.3 Gflops This is the world fastest PC cluster at August of 2001 01/10/31 6 3
Myrinet-2000 2000 2 Gbps full duplex NIC Lanai DMA Engines HOST NIC Outgoing/Incoming Message 16 port switch warm hall routing 01/10/31 7 SCore Version 4 System Software High Performance Communication Libs PMv2 15.0 usec Round Trip time 233 MB/s Bandwidth MPICH-SCore MPI Library 24.4 usec Round Trip time 228 MB/s Bandwidth PM/Ethernet Network Trunking Utilizing more than one NIC Global Operating System SCore-D Single/Multi User Environment Gang scheduling Checkpoint and restart Parallel Programming Language MPC++ Multi-Thread Template Library Shared Memory Programming Support Omni OpenMP on SCASH 10 times faster than Fast Ethernet + TCP/IP Three times as fast as Gigabit Ethernet + TCP/IP OMNI/SCASH SCASH PM/Shmem PM/Shmem driver Applications MPC++ MPICH-SCore SCore-D Global Operating System PM/Myrinet PM/Myrinet driver PMv2 Myrinet NIC PM firmware PM/Ethernet PM/Ethernet driver PVM-SCore Ethernet driver PM/UDP Socket UDP/IP Ethernet NIC PBS Linux User Level Kernel Level NIC Level 01/10/31 8 4
PM PM vs. GM DMA GM DMA 01/10/31 9 MPI Point to Point MPI Communication Bandwidth 2.50E+08 PM/Myrinet GM 1.0E+08 PM/Ethernet PM/Ethernet (2Way) 2.00E+08 LAM/MPI Bandwidth (Byte/sec) 1.50E+08 1.00E+08 Bandwidth (Byte/sec) 1.0E+07 1.0E+06 5.00E+07 0.00E+00 1.E+00 1.E+01 1.E+02 1.E+03 1.E+04 1.E+05 1.E+06 1.E+07 Message Size (Byte) 1.0E+05 1.0E+00 1.0E+01 1.0E+02 1.0E+03 1.0E+04 1.0E+05 1.0E+06 1.0E+07 Message Size (Byte) 01/10/31 10 5
Application Benchmark 512x256x256 01/10/31 11 Application Benchmark IS (Class C) 700 600 500 PM/Myrinet GM PM/Ethernet PM/Ethernet (2Way) TCP/IP(LAM) Total Mops 400 300 200 100 0 0 50 100 150 200 250 300 Number of Procs 01/10/31 12 6
Application Benchmark FT (Class C) FFT 12000 10000 8000 PM/Myrinet GM PM/Ethernet PM/Ethernet (2Way) TCP/IP(LAM) Total Mops 6000 4000 2000 0 0 50 100 150 200 250 300 Number of Procs 01/10/31 13 Application Benchmark Total Mops 40000 35000 30000 25000 20000 15000 10000 5000 LU (Class C) PM/Myrinet GM PM/Ethernet PM/Ethernet (2Way) TCP/IP(LAM) SSOR(Symmetric Successive Over-Relaxation) CFD 0 0 50 100 150 200 250 300 Number of Procs 01/10/31 14 7
Application Benchmark MG (Class C) 16000 14000 12000 PM/Myrinet GM PM/Ethernet PM/Ethernet (2Way) TCP/IP(LAM) Total Mops 10000 8000 6000 4000 2000 0 0 50 100 150 200 250 300 Number of Procs 01/10/31 15 PC 01/10/31 16 8
SCore III : 2 2 PC 2 Ethernet Myrinet Myrinet Clos128 Gigabit Ethernet 01/10/31 17 Myrinet-2000 2000 E128 Switches are connected by eight port switches. #6 #4 #2 #0 #1 E128 E128 nodes nodes nodes nodes nodes nodes #3 #5 #7 01/10/31 18 9
Ethernet eth1 Summit7i eth0 Summit7i #6 #4 #6 #2 #4 #0 #2 #0 #3 #1 #3 #7 #1 #5 #5 #7 01/10/31 19 01/10/31 20 10
01/10/31 21 01/10/31 22 11
01/10/31 23 16 Ethernet 4 2 Ethernet PC 5 01/10/31 24 12
(1/2) (800 Mbytes) RPM anaconda installation tool Score MBR The First Partition contains the installation image Other Partitions are empty SCore II NEC Express Servers (2U type) with Myrinet 5 SCSI disks 2 6 1 minutes and half for one disk copy 01/10/31 25 (2/2) 1st stage 2nd stage IP Kickstart anaconda 01/10/31 26 13
Each rack, Each modules, and All racks Score rcstest all-to-all Stressing Myrinet network in terms of network packets memory and Lanai Processor of Myrinet NIC Stressing processors and memory in Some initial hardware failures appear at at the the stress test!!!! 01/10/31 27 Connection between Myrinet Card and PCI bus slot Performance degradation We have not found the reason Connection between Myrinet Line Card and back-plane No communication CRC errors Connection between Myrinet Card and Cable No communication 01/10/31 28 14
PC PC Ethernet Ethernet Linux Ethernet NIC Ethernet Switch 01/10/31 29 8 01/10/31 30 15
PC PC 01/10/31 31 NIC 01/10/31 32 16
PC vs vs 01/10/31 33 PC PC PC Linux SCore 01/10/31 34 17
PC 19, 2, 7 SCore Omni OpenMP www.pccluster.org PC 01/10/31 35 Real World Computing Partnership is over, but SCore Development is continued 01/10/31 36 18