I/O 2

Size: px

Start display at page:

Download "I/O 2"

きよはるひろもり
5 years ago
Views:

1 !!

2 I/O 2

3 X as a Service 3

4 Software as a Service (SaaS) Platform as a Service (PaaS) Infrastructure as a Service (IaaS) Web Google Apps, Salesforce.com Google App Engine, Windows Azure Amazon EC2 4

5 Pay-as-you-go Web IaaS Cloud bursting IT 5

6 6

7 HPC AIST Super Cluster AIST Green Cloud AIST Super Cloud

8 AIST 10 F-32 P-32 M-64 79

9 AIST

10 20, ,000 ACM/IEEE Supercomputing 2005 Best Paper Award Full Electron Calculation Beyond 20,000 Atoms: Ground Electronic State of Photosynthetic Proteins

! NEB+Hybrid QM/MD! GridRPC + MPI! (1193 CPUs)! 58 5000 QM QM QM QM QM MPI AIST NCSA 64!4 CPUs for QM SDSC 64!2 CPUs for QM Purdue 64!4 CPUs for QM USC 64!

11 ! NEB+Hybrid QM/MD! GridRPC + MPI! (1193 CPUs)! QM QM QM QM QM MPI AIST NCSA 64!4 CPUs for QM SDSC 64!2 CPUs for QM Purdue 64!4 CPUs for QM USC 64!1 CPUs for QM RPC RPC USC QM QM SDSC QM QM MPI NCSA Purdue QM QM QM QM MPI AIST F32 41!1 CPUs for MD AIST F32 64!3 CPUs for QM AIST P32 64!4 CPUs for QM system energy QM QM RPC MD MD MD NEB MD MPI RPC scheduler QM QM MPI Number of Execution end-1 end-2 reaction 0 Elapsed time

12 Top500 P

AIST Green Cloud Server (AGC)! 2010 3!! Dell PowerEdge M1000e!8!! Dell PowerEdge M610!16!8 =128!

13 AIST Green Cloud Server (AGC)! !! Dell PowerEdge M1000e!8!! Dell PowerEdge M610!16!8 =128! Intel Xeon E5540! GHz, 8MB )! 48GB! 300GB HDD!2!! Mellanox M3601Q FI Infiniband I/F 4X QDR 32port!! Gigabit Ethernet

14 AIST Super Cloud Server (ASC)! !! Dell PowerEdge M1000e!12!! Dell PowerEdge M610!16!12 =192! Intel Xeon E5620! 2 2.4GHz, 12MB )! 24GB! 600GB HDD!2!! 10GB Ethernet!! Gigabit Ethernet!

15 1/10 AIST ASC AIST AGC AIST ASC (M ) (M /5 ) (TFlops) (KW) (KW) (Gflops/KW) (Gflops/KW) /Tflops)

16 ASC Pay as you go 200 BMM BMM KVM 150 KVM OpenNebula KVM OS: Scientific Linux 6.0 OS: CentOS 5.6 CentOS 4.9 opensuse 11.4 BMM CentOS 5.6 opensuse

17 AIST Super Cluster AIST Green Cloud AIST Super Cloud 1~2 T2K IT 17

18 18

19 産総研クラウドのイメージ仮想クラスタ計算の種類規模に応じた資源の要求利用者ごとに仮想的な専用クラスタ計算機を提供物理クラスタ e.g., ASC 従来のセンタ運用とは異なる基盤ソフトウェアが必要 19

20 VMWare ESXi Xen KVM Hyper-V OS Eucalyptus OpenStack CloudStack OpenNebula Nimbus Wakame Rocks Condor VMWare vsphere! OSS 20

21 IaaS VM VM Instance Instance Instance Instance 21

22 Rocks: UCSD CentOS (RHEL) Rolls Eucalyptus UCSB Amazon EC2/S3 OpenStack: NASA Rackspace NASA Eucalyptus OpenNebula OpenNebula OpenStack 22

23 Rocks Eucalyptus OpenStack OpenNebula OS RHEL5 1 VMM Xen Xen KVM Xen KVM VMWare Hyper-V VM VLAN /home libvirt VMM VM 23

24 OpenNebula (Complutense University of Madrid) Apache License 2.0 C12G Labs (FP7) 24

25 OpenNebula global network CLI or Sunstone GUI VM host 1 VM host 2 frontend ONED Sche duler VMM /srv/cloud SSH VMM /srv/cloud SSH /srv/cloud -- one `-- images VM local network VM NFS scp 25

26 OpenNebula CUI Host ID VM ID Sunstone Web GUI 26

27 Contextualization VM VM VM root SSH /etc/hosts /etc/resolv. conf NFS VM oned from OpenNebula Documentation 27

28 Contextualization VM (test.one) context.sh % onevm create test.one 28

29 MAC_PREFIX:IP OpenNebula VM IP MAC MAC IP VM MAC IP NIC % onevnet show 1 (snip) LEASES INFORMATION LEASE=[IP= , MAC=02:00:c0:a8:39:d1, USED=0, VID=-1] 02:00 IP IP 4 octet 16 29

30 OS OS VLAN OpenNebula 3.0 PCI I/O 30

31 VM OS VM SDSC P2V H/W ASC 31

32 VMM QEMU Xen libvirt Linux libvirt OpenStack QEMU -cpu host I/O 32

33 I/O 33

34 ! IO! 34

35 " DB HPC 35

36 IO IO emulation PCI passthrough SR-IOV VM1 Guest OS Guest driver VM2 VM1 Guest OS Physical driver VM2 VM1 Guest OS Physical driver VM2 VMM vswitch VMM VMM Physical driver NIC NIC NIC Switch (VEB) IO emulation PCI passthrough SR-IOV VM sharing Performance 36

37 AIST Green Cloud AGC 1 16 HPC Compute node Dell PowerEdge M610 CPU Intel quad-core Xeon E5540/2.53GHz x2 Chipset Intel 5520 Memory 48 GB DDR3 InfiniBand Mellanox ConnectX (MT26428) Blade switch Host machine environment OS Debian Linux kernel amd64 KVM Compiler gcc/gfortran MPI Open MPI VM environment InfiniBand Mellanox M3601Q (QDR 16 ports) VCPU 8 Memory 45 GB 1 1 VM 37

38 MPI Point-to-Point (higher is better) Bandwidth [MB/sec] PCI KVM Bare Metal Bare Metal KVM k 10k 100k 1M 10M 100M 1G Message size [byte] Bare Metal: 38

39 NPB BT-MZ: 300 (higher is better) 100 Performance [Gop/s total] Degradation of PE: KVM: 2%, EC2: 14% Bare Metal KVM Amazon EC2 Bare Metal (PE) KVM (PE) Amazon EC2 (PE) Parallel efficiency [%] Number of nodes 0 39

Bloss: Bloss: 120 100 MPI OpenMP Parallel Efficiency [%] 80 60 40 Degradation of PE:

40 Bloss: Bloss: MPI OpenMP Parallel Efficiency [%] Degradation of PE: KVM: 8%, EC2: 22% Number of nodes Bare Metal KVM Amazon EC2 Ideal 40

41 HPC "InfiniBand PCI HPC ", SACSIS2011, pp , KVM Xen PVM NAB-MZ Bloss "HPC ", ACS37. NUMA KVM Xen HVM Takano, et al., "Toward a practical "HPC Cloud": Performance tuning of a virtualized InfiniBand cluster", CUTE2011, December VMM HPC Challenge benchmark 41

42 PCI PCI NG PCI Bonding PCI NIC NIC IO NIC active-standby bonding S SR-IOV NIC VF IO PV 1 NIC 42

43 GesutOS eth0 (virtio) bond0 eth1 (igbvf) tap0 Host OS tap0 Host OS br0 br0 eth0 (igb) eth0 (igb) SR-IOV NIC SR-IOV NIC 43

44 GesutOS eth0 (virtio) bond0 eth1 (igbvf) (qemu) device_del vf0 tap0 Host OS tap0 Host OS br0 br0 eth0 (igb) eth0 (igb) SR-IOV NIC SR-IOV NIC 44

45 GesutOS eth0 (virtio) bond0 (qemu) migrate -d tcp:x.x.x.x:y GesutOS $ qemu -incoming tcp:0:y... tap0 Host OS tap0 Host OS br0 br0 eth0 (igb) eth0 (igb) SR-IOV NIC SR-IOV NIC 45

46 (qemu) device_add pci-assign, host=05:10.0,id=vf0 GesutOS eth0 (virtio) bond0 eth1 (igbvf) tap0 Host OS tap0 Host OS br0 br0 eth0 (igb) eth0 (igb) SR-IOV NIC SR-IOV NIC 46

47 MPI Guest OS eth0 (virtio) bond0 rank 1 eth1 (igbvf) # tap0 br0 eth0 (igb) tap br0 eth0 (igb) rank 0 SR-IOV NIC SR-IOV NIC NIC /24 47

48 48

49 GridARS!!"#$ " # " $%&#'# # " %(#)&%(*+%(*,%(-#'# #! $ $%&"$ " # " &%( #! $ '"#$ " # ".(,*/#'# # ".(,*&#'# # +%(.(&*& &%(.(&*&.01234#; =. GRC &%(,%(.(,*/ GRC.(,*/ GRC.(,*/.01234#5 789:.(&*&.01234#< &%( +%(.(&*& GRS.(&*/,%( &%(.(&*&.01234#6 49

Taiwan ASTI Philippines HCMUT HUT IOIT-Hanoi IOIT-HCM Vietnam CeNAT-ITCR Costa Rica UValle Colombia MU

50 PRAGMA Grid/Clouds UZH Switzerland LZU China CNIC China JLU China KISTI KMU Korea AIST OsakaU UTsukuba Japan SDSC USA IndianaU USA UoHyd India NECTEC KU Thailand MIMOS USM Malaysia HKU HongKong ASGC NCHC Taiwan ASTI Philippines HCMUT HUT IOIT-Hanoi IOIT-HCM Vietnam CeNAT-ITCR Costa Rica UValle Colombia MU Australia BESTGrid New Zealand UChile Chile 26 institutions in 17 countries/regions, 23 compute sites, 10VM sites 50

51 51 %(#)$*+,#-+,$.$/+01+2$$ >?,# >?,# >?,# >?,# >?,# #'#/$34#%5$!+678$9:0$ EF4$3/G=0<5$!+678$CD"$ (4$34#%5$!+678$9:0$?8<7<$3H<-<05$!+678$9:0$ >?&# >?&# >?&# >?&# >?&# >?&# >?,# >?,# >?,# >?,# >?,# IJ%!"$I2=1$J=B:$ #K8,:L$3H<-<05$ %(#)$MA=67MA<7:$.$/+01+2$ %(#)$I:+N2=1$.$OB+88$ 4/#'$%A,+1+67$.$/+01+2$ ;/*/$JL+RS$ T$D"$1:-B+K$#62=-,$ D"$(L<N:$ 6+-=:1$S2+L$$ NJ<2L$ D"$(L<N:$ 6+-=:1$S2+L$$ NJ<2L$ D"$(L<N:$ 6+-=:1$S2+L$$ NJ<2L$ D"$(L<N:$ 6+-=:1$S2+L$$ NJ<2L$ D"$(L<N:$ 6+-=:1$S2+L$$ NJ<2L$ Condor Master D"$(L<N:$ 6+-=:1$S2+L$$ NJ<2L$ >?&# >?,# T$I2=1$J<2L$/B=:0,$ T$I2=1$J<2L$#:2Q:2$ slave slave slave slave slave slave UA,$<BB$,+N:,G:2$ #,+2:$D"$=L<N:8$=0$IS<2L$8K8,:L8$!A0$QLV1:-B+K$862=-,8$<,$U!%I"%$#=,:8$ /+-K$D"$=L<N:8$+0$':L<01$S2+L$NJ<2L$ "+1=SKW8,<2,$D"$=08,<06:8$<,$U!%I"%$8=,:8$

52 2011 OpenNebula OSS OpenStack PCI I/O 52

template.dvi

template.dvi XXVI W I D E P R O J E C T 26 26 1 WIDE 2010 1 WIDE WIDE Cloud 2010 2 3 4 5 6 NAT64 7 2 2010 7 2 WIDE 2.1 WIDE WIDE WIDE WIDE 50 2.2 WIDE 13:00 14:45 IaaS 15:00 16:45 WIDE StarBED 17:00 19:00 2.3 193