GSIC TSUBAME Grid Cluster(TGC) 2

Size: px

Start display at page:

Download "GSIC TSUBAME Grid Cluster(TGC) 2"

りさこみしま
5 years ago
Views:

1 TSUBAME Grid Cluster Tokyo-tech Supercomputer and UBiquitously Accessible Mass-storage Environment) (GSIC)

2 GSIC TSUBAME Grid Cluster(TGC) 2

3 (GSIC) IT TSUBAME SuperTITANET IT IT 3

4 4

5 TSUBAME Tokyo-tech Supercomputer and UBiquitously Accessible Mass-storage Environment TSUBAME ( ) 5

TSUBAME (1) Sun SPARC WS Sun SuperSparc ~50MHz, 1+1+2+2+4+4 =14 Processors

6 TSUBAME (1) Sun SPARC WS Sun SuperSparc ~50MHz, =14 Processors Switched 10Base-T + Myrinet Solaris + MPI + Ninf backend 0.7GFLOPS ( TSUBAME 1/10 ) 6

7 TSUBAME (2) AMD Athlon Athlon Palomino 1.3Ghz, 78 processors (Rpeak 208GFlops) 1 Trunked 100Base-T Ethernet June 2001 The Top500 : 77.4GigaFlops AMD 439 7

6 TeraFlops 47 716GFlops Athlon Helix 2 PC AMD 1,2 : AMD

8 TSUBAME (3) Presto III June 2002 AMD, Tyan, Appro CPU Dual 512 proc AthlonMP Rpeak 1.6 TeraFlops GFlops Athlon Helix 2 PC AMD 1,2 : AMD Press Release Nov GigaFlops PC 5 5.8TeraFlops (Linux Networx) PrestoIII Self-made 8

9 TOP500 Nov. 12, 2002 Nov GigaFlops PrestoIII Self-made 3 9

10 (1) Titech Grid(2002/3-2006/3) Titech Grid E-Science ( 13 ) 800 PC (Super TITANET) E-Science 2 10

11 (2) Titech Grid(2002/3-2006/3) 1300 processors, 2.5 TFlops, 15 and 4 24-processor Satellite 12 30km Super TITANET (1-4Gbps) NEC Express 5800 Series Blade Servers Grid-wide Single System Image via Grid middleware: Globus, Ninf- G, Condor, NWS, GSIC Main Clusters (256 processers) x 2 systems in just 5 cabinets Super SINET (10 Gbps MOE National Backbone Network) to other Grids 11

12 (3) Titech Grid(2002/3-2006/3) (10 nodes) CPU:PentiumIII 1400MHz x2, Mem:512MB,OS:Linux2.4.19, Network: 100 Base-T MPI :mpich C ( ) 94.1Mbps 330us (5 nodes) C ( ) 1.2km 30km 94.0Mbps 770us 90.3Mbps 50us 12

13 ~90% 200~300 users, 24/713 usage

14 TSUBAME (1) 64bit x86 CPU (Dual Core Opteron) INT/FP : Best applications & software availability: OS (Linux/Solaris/Windows), languages/compilers/programming tools, libraries, Grid tools, all ISV Applications 14

15 TSUBAME (2) Hybrid Architecture: General-Purpose Scalar + SIMD Vector Acceleration w/ ClearSpeed CSX Teraflops 90 KW PB All Disk Storage Architecture (no tapes), 1.1Petabyte Overall HW data loss: once / 1000 years High bandwidth NAS I/O: ~60GBytes/s Livermore Benchmark Unified Storage and Cluster interconnect: low cost, high bandwidth, unified storage view from all nodes w/o special I/O nodes or SW 15

Sun Fire X4500 24Gbps 10Gbps InfiniBand 2,304

16 TSUBAME Sun Fire X nodes, 5,240CPU10,480Cores 50TFlops(peak) Total 85TFlops (peak) ClearSpeed Advance Accelerator Board 360 boards, 35TFlops(peak) Super Titanet Sun Fire X Gbps 10Gbps InfiniBand 2,304 ports NEC istorage S1800AT 42 nodes, 1PB disk storage 0.1 PB RAID6 16

17 Galaxy 4 Enterprise-Class Data Center Compute Engine SunFire X4600 Type-A(639nodes) Full performance Opteron processor modules (8 -total) Dual-Core Available CPU Module Type-B(16nodes) Redundant, hot-plug fans 6 PCI-Express slots - 4 x8-lane slots - 2 x4-lane slots 24 deep 2 PCI-X slots (133MHz/64-bit) : 21.4 TB : 50.4 TFlops

18 ClearSpeed Advance Accelerator Board

19 TSUBAME Computing Node 19

20 Voltaire ISR 9288 Infiniband Swithc Router 10Gbps 30Gbps HPC 1

21 Backend view 21

SunFire X4500 Thumper High perforamance Server with a lot of local data (

socket, dual core Opteron Processor Subsystem with 8x DIMMS (on risers) (42

0 Peta Bytes 1 + 1 Hot-Swap Power Supplies 4RU 10x Hot-Swap 80mm Fans 2x

22 SunFire X4500 Thumper High perforamance Server with a lot of local data ( :: 42 ) 48x GB SATA Hard Drives (12-24TB Raw/6-12TB RAID 5) 2 socket, dual core Opteron Processor Subsystem with 8x DIMMS (on risers) (42 (42 Total) : 1.0 Peta Bytes Hot-Swap Power Supplies 4RU 10x Hot-Swap 80mm Fans 2x Half-Height PCI Cards SATA Backplane with 48x HDD Connectors Hot-Swap I/O Subsystem (under Processor Subsystem): 1x 10/100 (mgmt.) 2x USB 2.0 4x GbE

23 istorage S1800AT 23

24 24

25 TSUBAME Titech Grid Cluster 2 A TSUBAME 3 (600m 2 ), 350m 2 ( ) 76 ( ) , ( ) 1.2 TSUBAME 2 B TSUBAME & Storage TSUBAME 1 25

26 Titech TSUBAME ~80+ racks 350m2 floor area 1.2 MW (peak) 26

27 Node Rear Local Infiniband Switch (288 ports) Currently 2GB/s / node Easily scalable to 8GB/s / node Cooling Towers (~20 units) ~500 TB out of 1.1PB 27

TSUBAME 2006 3 2 28 : NEC SX-5, SGI Origin2000, HP GS320 3 1 ~3 7: 3 8 ~ 3 31 : TSUBAME 4 3

28 TSUBAME : NEC SX-5, SGI Origin2000, HP GS ~3 7: 3 8 ~ 3 31 : TSUBAME 4 3 ~5 : ( ) TB 23 2.( ) The Top500 6 : ( ) 299, (4748 CPUs), SSO, WebDAV, Portal SSO, 10 : 28

29 Lustre by Cluster File System Inc. 10,000 Cluster File System Inc. Open Source Software CFS Inc. ($5,000/ ) PNNL: HPCS2; LLNL: Thunder, MCR/PVC, ALC, Lilac; NCSA: Tungsten 32

30 Client, Lustre by Cluster File System Inc. MDS, OST Meta Data Server (MDS), Object Storage Targets (OST) 33

31 Lustre config. information, network connection details, & security management LDAP server directory operations, meta data, & concurrency MDS Client Client Client Client Client Client Client Client Client Client recovery, file status, file creation file I/O & file locking OST OST OST OST OST OST OST OST OST OST OST OST OST OST OST OST OST OST OST OST OST 34

32 Lustre vs NFS NFS NFS NFS Single Point of Failure I/O Luster NFS 35

33 Luster 100MB/s 36

34 availability OS/ ( /4~6 ) ( ) SE/ / ( ) ( ) 37

TSUBAME Grid Cluster http://www.itc.u-tokyo.ac.

35 TSUBAME Grid Cluster

36 39

37 ( 2-3 ) Top

38 >40TFlops 1PB ( ) 41

39 XX UU YY ZZ 42

40 43

41 44

42 45

43 HPC 2004 HP HPC HPC HPC => HPC 46

44 NEC AMD:Fab36 47

0.1 istore CPU (Dual Core Opteron 880/885 2.4/2.6GHz, 10480 ) Sun Galaxy 4600 8-Way/16-Core 655 50.

45 0.1 istore CPU (Dual Core Opteron 880/ /2.6GHz, ) Sun Galaxy Way/16-Core TFlops GigaFlops SIMD CSX TeraFlops) Infiniband 10Gbps 288 Voltaire ISR9288 Lustre NAREGI 48

46 49

TSUBAME 27th Top500 7 2006 6 30 1 2 3 4 5 6 7 8 9 10 T/V N NB P Q Time Gflops ----------------------------------------------------------------- WR10R2R4 1334160 240 36

47 TSUBAME 27th Top T/V N NB P Q Time Gflops WR10R2R e TeraFlops Opteron Only, 648 nodes ~14 Terabytes memory 76.56% Efficiency 50

TSUBAME as No. 1 in Asia-Pacific Top500 June, 2006 #147: 3.76TFlops Petroleum Company (G), Saudi Arabia #228: 3.39TFlops, Institute of High Performance Computing (IHPC), Singapore #35: 10.

48 TSUBAME as No. 1 in Asia-Pacific Top500 June, 2006 #147: 3.76TFlops Petroleum Company (G), Saudi Arabia #228: 3.39TFlops, Institute of High Performance Computing (IHPC), Singapore #35: 10.3TFlops, China Meteorological Administration #141: 3.83TFlops Semiconductor Company (L), India #200: 3.52TFlops, Semiconductor Company (L), Malaysia #22: 15.7TFlops, Korea Meteorological Administration #346: 2.77TFlops, Gaming Company, Taiwan #7: 38.18TFlops TSUBAME, Japan #47: 8.97TFlops, Australian Partnership for Advanced Computing #147: 3.76TFlops, WETA Digital, New Zealand 51

49 TSUBAME Manufacturer Computer Procs RMax RPeak Eff Nmax Matrix Mem (GB) Est. Time (sec) Est time (hours) Sys Mem Mem Eff IBM LLNLBG/L 1E % % IBM IBM BG/L % % IBM SGI Dell Cray Inc. NEC IBM LLNLASCI Purple % % NASA/SGI Columbia % % Sandia Thunderbird % % Sandia Redstorm % % Earth- Simulator % % Marenostru % % m PPC IBM Astron BG/L % % ORNL Cray Cray Inc % % XT3, Sun/NEC Titech Galaxy % % Top500 Linpack 52

50 TSUBAME / Machine Peak CPU Peak Watts/ Watts MFLOPS Cores GFLOPS CPU /Watt TSUBAME(Opteron) ,000 50, TSUBAME(w/CS) 11, ,000 85, Earth Simulator ,000,000 40, ASCI Purple ,000,000 77, Orion (DS-96) Low power Cluster BG/L (rack) , BG/P? (rack) , Next Gen (2010) ,

51 ($!! (!!! &!! (!"'%)& (#(!)$ (&#'" MPI Basic BW / Latency Single lane usage (preliminary) Node-node latency min=4.74, max=5.78 microsec Max BW=1103~ MB Max ½ = 16384~32768 bytes '!! "!! $!! *+,*(-./0(70<=>?@>;A Very uniform, high performance! $!"' $%& #$ " :;4.! MPI BW *+,*$-./0) *+,*(-./01 *+,*(-./0(/ *+,*%-./0(( *+,*%-./0( *+,*"-./0# *+,*#-./0% *+,*(-./0(/ *+,*(-./0(B *+,*(-./0# *+,*(-./0% *+,*(-./0) *+,*(-./01 *+,*(-./0(( *+,*$-./0( *+,*$-./0# *+,*$-./0% *+,*$-./0) *+,*$-./01 *+,*$-./0(( *+,*#-./0( *+,*#-./0# *+,*#-./0% *+,*#-./0) *+,*#-./01 *+,*#-./0(( *+,*"-./0( *+,*"-./0# *+,*"-./0% *+,*"-./0) *+,*"-./01 *+,*"-./0(( *+,*%-./0( *+,*%-./0# *+,*%-./0% *+,*%-./0) *+,*%-./01 *+,*%-./0(( *+,*&-./0( %!!! "%!! "!!! #%!! #!!! 1.23 $%!! $!!! (%!! (!!! %!!! (!"'%)& (#(!)$ (&#'" $!"' $%& #$ *+,*(-./0(5/:92;38 MPI Latency "! 54 *+,*&-./0# *+,*%-./0) *+,*"-./0(( *+,*"-./0# *+,*#-./0) *+,*$-./0(( *+,*$-./0# *+,*(-./0) *+,*(-./0(/

52 Scaling Gaussian Test397!%))!())!!"#$%&"!))) 01234/356 *)) ")) %)) #!"$%"' &##$! &(!$'%" %')$!!* %!*$+" %%&$%&' &))$&"( #&+$&&% ()) )!" '( %* "%!(* (&" '*% &!(!)(%,-./ 55

53 ClearSpeed 1. ISV Matlab, Mathematica, Amber, 2. BLAS/DGEMM, IMSL, FFTW (Fortran/C bindings) 3. SIMD C SIMD 56

54 Mathematica Performance Data without CS Mathematica Notebook Data for system with NO CSX600 Advance card Maximum ~ 3 GFLOPS DGEMM 57

55 Mathematica Performance Data with CS Mathematica Notebook Data for system with CSX600 Advance card Maximum ~ 40 GFLOPS DGEMM 58

56 100TFlops 59

57 (1) 100 CIP C/P 60

58 (2) (+ ) 100TB 300TB 61

59 WebMO (Gaussian,NWChem,GAMESS, MOPAC, Molpro ) Web ( NEDO SSO ) 1. (3) IT

60 Tokyo Tech Portal 63

61 General Purpose DataCenter Hosting All University AAA and system 50TB of Thumper, 9 Galaxy 1 nodes from TSUBAME (7 ) GB ( ) ( ) (Open CourseWare) (VM ) 64

62 65 NII 7 CSI NI I NAREGI H17.2

63 66

09中西

09中西 PC NEC Linux (1) (2) (1) (2) 1 Linux Linux 2002.11.22) LLNL Linux Intel Xeon 2300 ASCIWhite1/7 / HPC (IDC) 2002 800 2005 2004 HPC 80%Linux) Linux ASCI Purple (ASCI 100TFlops Blue Gene/L 1PFlops (2005)