001.dvi

Transcription

1 THE SCIENCE AND ENGINEERING DOSHISHA UNIVERSITY, VOL.XX, NO.Y NOVEMBER 2003 Construction of Tera Flops PC Cluster System and evaluation of performance using Benchmark Tomoyuki HIROYASU * Mitsunori MIKI * and Hiroshi ARAKUTA ** (Received October 6, 2004) Complicated and diverse of objective problems with the development of technology, demand of high performance computer is increasing. Instead of vector superocomputer, in order to meet the needs of these demand, PC cluster systems have gotten a lot of attention in recent years. PC cluster systems consist of many PCs connected by network and are used for parallel or distributed computation. In scientific and engineering fields, HPC cluster systems are attractive to the computationally intensive tasks. We setting up of HPC cluster system called Supernova, toward a performance of 1 Tera Flops. Supernova is composed 256-node and running Linux Operating System. We evaluated this cluster system using High-Performance LINPACK Benchmark and Himeno Benchmark. In this paper, we present the result of performance and the combination of parameters using these Benchmarks in Supernova. Through experimenting we obtained knowledge to perform parameter tuning for better performance in these Benchmarks and knowledge about construction of PC Cluster System. Key words PC Cluster, Himeno Benchmark, LINPACK Benchmark, Linux PC,,, 1. 1 PC 1, 2) 500 TOP500 Supercomputer Sites 3) TOP500 * Department of Knowledge Engineering and Computer Sciences, Doshisha University, Kyoto Telephone: , Fax: , [email protected], [email protected] ** Graduate Student, Department of Knowledge Engineering and Computer Sciences, Doshisha University, Kyoto Telephone: , Fax: , [email protected]

2 TOP500 1TFlops Supernova Cluster System Supernova Benchmark High-Performance LINPACK Benchmark Supernova 2. PC PC 1 CPUOS PC HPC(High Performance Computing) HPC HPC Beowulf Beowulf 1990 NASA Beowulf 4, 5, 6) Beowulf Linux FreeBSD OS MPI PVM Beowulf SCore 7, 8) SCore Beowulf RWCP Real-World Computing Partnership Beowulf SCore SCore Linux PC HA(High Availability) HA HA 3. Supernova Cluster System 1TFlops Fig. 1 Supernova Cluster System Supernova Supernova AMD 64bit CPU Opteron 512CPU Table 1 Fig. 1. Supernova Cluster System. Table 1. Supernova. 256 CPU AMD Opteron 1.8GHz 512 Memory 2GB 256 total 512GB OS Turbolinux for AMD64 mpich TCP/IP Gigabit Ethernet

3 Opteron 1 2 FPU 1 1 Supernova Rpeak (1) TFlops Rpeak =#CPU ClockFrequency #FPU (1) 3.1 Opteron Supernova AMD Opteron Opteron Opteron HyperTransport HyperTransport 9) I/O 3 HyperTransport 19.2GB/s 3.2 Supernova Force10 Networks E1200 E Tbps Supernova 256 E Supernova Supernova 3.1 Opteron HyperTransport 100Mbps 1Gbps PC Myricom Myrinet InfiniBand 10) Supernova 4. Benchmark 4.1 Benchmark PC Benchmark 11) LINPACK 12) Nas ParallelBenchmark 13) Benchmark LIN- PACK 4.2 Benchmark Benchmark Poisson Jacobi

4 4.2.1 Benchmark Benchmark Benchmark Jacobi Table 2 Table 2. Benchmark. Array Size #Array Elements XS S M L XL x y z CPU CPU Poisson Poisson (2) 2 u x u y u = f(x, y, z) (2) z2 (2) (3) f i,j,k = u i+1,j,k 2u i,j,k + u i 1,j,k x 2 + u i,j+1,k 2u i,j,k + u i,j 1,k y 2 + u i,j,k+1 2u i,j,k + u i,j,k 1 z 2 (3) x = y = z (3) (4) u i,j,k = 1 6 [u i+1,j,k + u i 1,j,k + u i,j+1,k + u i,j 1,k + u i,j,k+1 + u i,j,k 1 ( x) 2 f i,j,k ] (4) u i,j,k Gauss-Seidel Jacobi Benchmark Jacobi u m+1 i,j,k (m) (5) u m+1 i,j,k = 1 6 [um i+1,j,k + um i 1,j,k + um i,j+1,k + um i,j 1,k + u m i,j,k+1 + um i,j,k 1 ( x)2 f i,j,k ] Benchmark Jacobi (5) 4.3 LINPACK Benchmark LINPACK Tenessee J.Dongarra LU Fortran LINPACK BLAS Basic Linear Algebra Subprograms LINPACK 3 LINPACK Benchmark N=100 N=100 LU DGEFA DGESL SGEFA SGESL 2 LU x Toward Peak Performance N=1000 Highly Parallel Computing TOP500 HPL HPL

5 LINPACK (6) = 2 3 N 3 + O(N 2 ) (6) A LU x LU LU A n n b n Ax = b (7) A LU A L U (8) A = LU (8) n n L L ij L ij (9) L ij =0(i>j) (9) 4.4 High-Performance LINPACK Benchmark HPL High-Performance LINPACK Benchmark LINPACK BLAS ATLAS Automatically Tuned Linear Algebra Software goto-library HPL HPL HPL Fig. 2 2 LU Fig. 3 Panel Factorization Panel Broadcast LU L U (13) n n U U ij U ij (10) N P0 P2 P U ij =0(i<j) (10) P1 P3 P A LU (7) (11) N Global Array Pn : Process Number Local Array Ax = LUx = b (11) y = Ux Ly = b (12) (12) Fig. 2. Factorization U. Factorization U y =(y 1 y 2 y n ) (12) y n y n 1 y 1 O(n 2 ) y (13) x O(n 2 ) Ux = y (13) L update Broadcast Fig. 3. L update Broadcast.

6 4.4.2 HPL 16 N P Q Broadcast N P Q Panel Broadcast Look-ahead Update long U mix L1 U alignment Panel Factorization Panel Factorization Performance[MFlops] Performance[MFlops] 8.0E+02 GNU PGI 6.0E E E E+00 (1,1,1) Division of Grid (a) 1CPU (b) 2CPU 3.0E+03 GNU PGI 2.0E E E+00 (1,1,4) (1,2,2) (1,4,1) (2,1,2) (2,2,1) (4,1,1) Division of Grid Performance[MFlops] Performance[MFlops] 1.5E+03 GNU PGI 1.0E E E+00 (1,1,2) (1,2,1) (2,1,1) Division of Grid (c) 4CPU (d) 8CPU Performance[MFlops] 6.0E+03 GNU PGI 4.0E E E+00 (1,1,8) (1,2,4) (1,4,2) (1,8,1) (2,1,4) (2,2,2) (2,4,1) (4,1,2) (4,2,1) (8,1,1) Division of Grid 1.0E+04 GNU PGI 8.0E E E E E+00 (1,1,16) (1,2,8) (1,4,4) (1,8,2) (1,16,1) (2,1,8) (2,2,4) (2,4,2) (2,8,1) (4,1,4) (4,2,2) (4,4,1) (8,1,2) (8,2,1) (16,1,1) Division of Grid (e) 16CPU Fig. 4.. Factorization Factorization 5. Benchmark Benchmark Benchmark Fig. 4 Fig. 5 M CPU 1CPU 2CPU 4CPU 8CPU 16CPU GNU Fortran Compiler 3.2 PGI Fortran Compiler 5.0 Fig. 4 PGI GNU Fig. 5 CPU PGI GNU Difference of Performance[MFlops] 1.0E+04 GNU PGI 8.0E E E E E #CPU Fig. 5. CPU. PGI CPU (x y z) Fig. 4 1CPU (1 1 1) 2CPU (2 1 1) 4CPU (2 1 2) 8CPU (2 2 2) 16CPU (4 2 2) CPU

7 PGI PGI -O1 -fthread-jumps -defer-pop -O2 -O1 -O3 -O2 -O2 -finline-functions -fast -Mvect=assoc -Mvect=cachesize:* -Mcache align -Mnontemporal Prefetch 128 M CPU (4 2 2) Table 3 -fast -Mvect=assoc -O MFlops Supernova RIKEN BMT 2003 PC 1 Table 3.. Compile Option Performance [MFlops] None fast -Mvect=assoc -O fast -Mvect=assoc -O Mcache align -Mnontemporal Mvect=cachesize: O Mvect=assoc,cachesize: HPL LINPACK HPL N N HPL HPL N N N 6.2 HPL 6.3 P Q P Q P Q P Q P Q 6.4 Panel Broadcast Panel Broadcast Increasing- 1ring Increasing-2ring Bandwidth-reducing 3 Panel Factorization modified 3 6 normal modified

8 normal Update Panel Factorization modified Update Panel Factorization 7. HPL 7.1 HPL ) ATLAS CPU N:10000 BCAST:1ring gcc3.2formit-frame-pointer -O3 -funroll-loops atlas Fig n 28n (n=2) (n=3) (n=4) (n=5) (a) 1CPU n 28n Fig (n=2) (n=3) (n=4) (n=5) (b) 2CPU Fig Fig (a) 1CPU Fig (b) 2CPU Fig Panel Broadcast Panel Broadcast Fig. 8 BCAST Table 4 gcc3.2-fomit-framepointer -O3 -funroll-loops atlas Table 4. BCAST. 64cpu 128cpu 256cpu 512cpu N (P Q) (8 8) (8 16) (16 16) (16 32) rg 1rM 2rg 2rM Lng LnM Topology rg 1rM 2rg 2rM Lng LnM Topology (a) 64CPU (b) 128CPU rg 1rM 2rg 2rM Lng LnM Topology (c) 256CPU (d) 512CPU Fig. 8. Panel Broadcast. Fig. 8 BCAST CPU normal modified modified normal Fig. 8 normal modified Fig. 8 CPU Supernova BCAST Long bandwidth reducing modified modified Long Supernova rg 1rM 2rg 2rM Lng LnM Topology

9 7.3 HPL AT- LAS goto-library Fig. 9 Table 5 gcc3.2-fomit-frame-pointer -O3 -funroll-loops Table 5.. 4cpu 8cpu 16cpu 32cpu 64cpu N (P Q) (2 2) (2 4) (4 4) (4 8) (8 8) library atlas libgoto opteron-r0.7.so BCAST Increasing-1ring Performance[GFlops] atlas libgoto_opteron-r0.7.so #CPU #CPU (a) (b) Fig. 9.. Fig. 9(a) atlas goto-library Fig. 9(b) atlas goto-library Fig. 9(a) goto-library Fig. 9(b) CPU goto-library 7.4 (P Q) P Q CPU (P Q) (16 32) Supernova Difference of Performance[GFlops] Fig. 10 Table 6 gcc3.2-fomit-frame-pointer -O3 -funroll-loops Table 6.. N (1 512) (2 256) (4 128) (P Q) (8 64) (4 128) library libgoto opteron-r0.7.so BCAST Increasing-1ring Performance[GFlops] (1,512) (2,256) (4,128) (8,64) (16,32) Process Grid Fig Fig. 10 (P Q) (16 32) (1 512) (2 256) HPL HPL HPL 100% (1 512) (2 256) 7.5 N 6.1 N HPL Supernova 80% N N HPL N

10 Table 7.. 1cpu 2cpu 4cpu 8cpu 16cpu 32cpu 64cpu 128cpu 256cpu 512 cpu N (P Q) (1 1) (1 2) (2 2) (2 4) (4 4) (4 8) (8 8) (8 16) (16 16) (16 32) BCAST Increasing-1ring library atlas Fig (P Q) (16 32) BCAST LnM gcc3.2-fomit-frame-pointer -O3 -funroll-loops Fig. 11. HPL. Fig. 11 N Supernova HPL Table 8 Table 8.. N (P Q) (16 32) BCAST Long bandwidth reducing modified library libgoto opteron-r0.7.so Supernova E1200 E1200 NETGEAR 15) Gigabit Switch GS524T 1CPU 512CPU Table 7 Fig. 12 gcc3.2 -fomit-frame-pointer -O3 -funroll-loops atlas Performance[GFlops] Difference of Performance[GFlops] Fig. 12. GS524T E #CPU (a) #CPU (b).

11 Fig. 12(a) E1200 Fig. 12(b) CPU E1200 GS524T CPU E1200 Fig. 12(a) 128CPU CPU 128CPU PC GS524T PC 9. Benchmark LINPACK HPL Benchmark Benchmark PGI HPL ATLAS HPL gotolibrary PC Benchmark Supernova 16CPU 11271MFlops RIKEN BMT 2003 PC Supernova 1 HPL Supernova 1.169TFlops 63.4% TOP PC 1 Supernova 1) Rajkumar Buyya. High Performance Cluster Computing: Architecture and Systems, Vol. 1. Prentice Hall, ) Rajkumar Buyya. High Performance Cluster Computing: Programming and Applications, Vol. 2. Prentice Hall, ) TOP500 Supercomputer Sites. top500.org/. 4) T. Sterling, D. Savarese, D. J. Beeker, J. E. Dorband, U. A. Renawake, and C. V. Packer. Beowulf: A parallel workstation for scientific computation. In Proceedings of the 24th International Conference on Parallel Processing, pp , ) Donald J. Becker, Thomas Sterling, Daniel Savarese, John E. Dorband, Udaya A. Ra nawak, and Charles V. Packer. BEOWULF: A PAR- ALLEL WORKSTATION FOR SCIENTIFIC COMPUTATION. In Proceedings of International Conference on Parallel Processing, ) T. L. Sterling, J. Salmon, D. J. Beeker, Savarese, and D. F. Savarese. How to build a beowulf: A guide to the implementation and application of pc clusters. MIT Press, ) PC Cluster Consortium. or.jp/. 8) H. Tezuka, A Hori, Y. Ishikawa, and M. Sato. Pm: An operating system coordinated high

12 performance communication library. In Highperformance Computing and Networking97, pp , ) HyperTransport Consortium. hypertransport.org/. 10) InfiniBand Trade Association Home Page. http: // 11) Himeno Benchmark xp Home Page. HimenoBMT/index.html. 12) The linpack benchmark. org/benchmark/top500/lists/linpack.html. 13) The NAS Parallel Benchmarks Home Page. 14) HPL Algorithm. benchmark/hpl/algorithm.html. 15) NETGEAR Home Page. com/.