paper+.dvi

Size: px

Start display at page:

Download "paper+.dvi"

みちしげひろもり
4 years ago
Views:

1 Vol. 40 No. 5 May 1999 MPI y y y MPI MPI/MBCF MPI/MBCF write eager 2 write eager FIFO 2 MPI/MBCF round-trip time peak bandwidth NAS Parallel Benchmarks Implementation and Evaluation of a High Performance MPI Library with the Memory-Based Communication Facilities Kenji Morimoto, y Takashi Matsumoto y and Kei Hiraki y This paper describes an ecient implementation of the Message Passing Interface (MPI) library based on the shared memory model. Our implementation, called MPI/MBCF, combines two protocols to utilize shared memory communication facilities; the write protocol and the read protocol. In the write protocol, the remote write is used for communication with no buering. In the eager protocol, the Memory-Based FIFO is used for buering by the library. These two protocols are switched autonomously according to the precedence of send and receive functions. The performance of our library was evaluated on a cluster of workstations. We measured the round-trip time and the peak bandwidth, and executed the NAS Parallel Benchmarks. The results show that it is ecient to construct a message passing library with the MBCF which is based on the shared memory model. 1. send, receive read, write 2 latency bandwidth MMU y Department of Information Science, Faculty of Science, University oftokyo load, store 1

2 2 May 1999 MPI Message Passing Interface Ver ),9) MBCF: Memory-Based Communication Facilities 7) MPI/MBCF SSS{ CORE MPI 2 MBCF 3 MPI/MBCF MPI/MBCF 5 NAS Parallel Benchmarks MBCF 2.1 MBCF MBCF Strategic Memory System 6) MBCF (1) read, write (2) (3) MBCF MPP TLB (4) read, write swap, FIFO write, fetch and add (5) MBCF 2.2 MBCF MBCF MPI MPI/MBCF FIFO ID ID MBCF ack FIFO FIFO FIFO FIFO

3 Vol. 40 No. 5 MPI 3 FIFO FIFO 1 FIFO FIFO ack FIFO FIFO 2.3 MBCF 100BASE-TX Hub MBCF Axil 320 model Sun SPARCstation20 85 MHz SuperSPARC CPU 2 1 Sun Microsystems Fast Ethernet SBus Adapter 2.0 SMC TigerStack TX Hub Bay Networks BayStack 350T Hub OS SSS{ CORE Ver. 1.1a 2 one-way latency peak bandwidth One-way latency ack ack 1 MBCF WRITE FIFO MBCF FIFO one-way latency Hub Peak bandwidth 2 Hub Hub MBCF WRITE MBCF FIFO peak bandwidth one-way latency 24.5 s, peak bandwidth MB/s MB/s 100BASE-TX bandwidth 12.5 MB/s 5) MPI Ver MPI_Isend() MPI_Irecv() MPI/MBCF MPI MPI (1) (2) (3) ANY 2 2 4

4 4 May Table 1 100BASE-TX MBCF one-way latency s One-way latency of MBCF with 100BASE-TX (in microseconds) data-size (bytes) MBCF WRITE MBCF FIFO BASE-TX MBCF peak bandwidth Mbytes/s Table 2 Peak bandwidth of MBCF with 100BASE-TX (in Mbytes/s) 2 data-size (bytes) MBCF WRITE, MBCF FIFO, MBCF WRITE, MBCF FIFO, MPI MPI_Isend() MPI_Irecv() MBCF MPI MPI/MBCF MBCF MPI HW buf network HW buf 1 Fig. 1 Communication with no buering FIFO FIFO FIFO FIFO FIFO FIFO FIFO MPI

5 Vol. 40 No. 5 MPI 5 HW network HW FIFO HW network HW FIFO buf buf buf buf Fig. 2 2 Communication with single buering 3 2 Fig. 3 Communication with double buering MPI/MBCF FIFO FIFO FIFO 1 FIFO FIFO MPI FIFO FIFO FIFO 2 FIFO FIFO FIFO FIFO FIFO FIFO FIFO MPI MPI_Isend() MPI/MBCF (1) (2) (2) FIFO MPI_Irecv() (1) FIFO

6 6 May 1999 (2) (2) (2) FIFO FIFO FIFO (2) FIFO (3) (3) (2) FIFO ANY MPI/MBCF 1 1 MBCF WRITE FIFO MBCF FIFO MPI_Isend() MPI_Irecv() MPI/MBCF FIFO FIFO MPI MPICH 4) eager FIFO write FIFO eager MPI_Wait() FIFO FIFO MPI MPI/MBCF 2.3 round-trip time peak bandwidth MBCF OS SSS{CORE Ver. 1.1a OS SunOS MPI MPICH Ver ) SunOS MPICH Argonne National Laboratory Mississippi State University MPI TCP write SR NSR 4.2 Round-trip time

7 Vol. 40 No. 5 MPI 7 Sender Receiver message MPI Isend() (by MBCF FIFO) enqueue -hhhhhhhhhh- MBCF FIFO referred by MPI Wait() (((((((((( acknowledgment queues (by the MBCF system) (for messages) MPI Irecv() 4 eager Fig. 4 Execution sequence where send precedes receive (eager protocol) 6B BBN dequeue copied to buer Sender Receiver request for sending (by MBCF FIFO) MPI Irecv() enqueue (((((((((( MBCF FIFO queues (for requests) message (by MBCF WRITE) MPI Isend() 6 -hhhhhhhhhh- - directly written to buer dequeue 5 write Fig. 5 Execution sequence where receive precedes send (write protocol) Sender Receiver MPI Isend() enqueue enqueue -hhhhhhhhhh- (((((((((( ( MBCF FIFO MBCF FIFO queues ((((((((( queues (for requests) (for messages) next MPI Isend() - 6B dequeue BBN discarded MPI Irecv() MPI Irecv() 6 Fig. 6 Execution sequence where send conicts with receive 6B BBN dequeue copied to buer Hub round-trip time 2 MPI MPI_Irecv() (1) MPI_Send() (2) MPI_Wait() (1) MPI_Wait() (2) MPI_Send() 3 round-trip time SSS{CORE MPI/MBCF 0.5 s 1 SunOS MPICH 10 s 1024 MPI/MBCF TCP MPICH SR NSR SR write FIFO

8 8 May BASE-TX MPI round-trip time s Table 3 Round-trip time of MPI with 100BASE-TX (in microseconds) message size (bytes) SR NSR MPICH SR SR 0byte round-trip time 71 s MBCF one-way latency 24.5 s 4byte MPI 4.3 Peak bandwidth Hub Hub peak bandwidth 2 MPI 2 peak bandwidth 4 4byte 1Mbyte peak bandwidth 4Kbyte 7 SRH, NSRH Hub SR, NSR peak bandwidth SRF, NSRF Hub SR, NSR peak bandwidth bandwidth MPI/MBCF peak bandwidth MPICH/TCP bandwidth MPI/MBCF NSR peak bandwidth SR SR write SR NSR bandwidth SR peak bandwidth MB/s MB/s 100BASE-TX MB/s MBCF MB/s, MB/s 5. NAS Parallel Benchmarks 5.1 NAS Parallel Benchmarks NAS Parallel Benchmarks NPB NASA Ames Research Center NPB 1.0 1) MPI NPB 2.x 2) 5 3 CFD EP MG 3 CG FT 3 IS CFD LU Symmetric SOR LU SP BT NPB 2.x IS C + MPI IS Fortran90 + MPI 8 class S class W class A

9 Vol. 40 No. 5 MPI BASE-TX MPI peak bandwidth Mbytes/s Table 4 Peak bandwidth of MPI with 100BASE-TX (in Mbytes/s) message size (bytes) SRH NSRH SRF NSRF MPICH Bandwidth (Mbytes/s) MPI/MBCF (SRH) MPI/MBCF (NSRH) MPI/MBCF (SRF) MPI/MBCF (NSRF) MPICH Message size (bytes) x BASE-TX MPI Peak bandwidth Fig. 7 Peak bandwidth of MPI with 100BASE-TX class B class C SSS{CORE MPI/MBCF NPB Rev. 2.3 Hub SunOS

10 10 May 1999 SunOS MPICH Ver. 1.1 gcc g FT g77 class W class A SSS{CORE SunOS MPI/MBCF SR NSR MPI/MBCF 8 SP, BT 9 MPI MBCF 6 EP 2 26 EP 7 MG byte Kbyte 1 1 MPI/MBCF MPICH MPI_ANY_SOURCE SR SR 8 CG Kbyte Kbyte 1 1 MG MG MG SR 9 IS Mbyte MPI/MBCF SR 10 LU byte 1 1 MPI/MBCF, MPICH LU MG MPI_ANY_SOURCE SR 11 SP Kbyte 1 1 SR 12 BT Kbyte Kbyte 1 1 SR SR, NSR 5.4 MPI/MBCF MPICH MPI/MBCF MPICH MG MPICH MPI/MBCF MPICH

11 Vol. 40 No. 5 MPI 11 5 NPB Table 5 Characteristics of NPB programs EP MG CG IS LU SP BT (Mbytes/s) ( /s) (%) NPB EP Table 6 Execution time of NPB EP (in seconds) SR [speed-up] [1.00] [2.00] [4.00] [8.00] NSR [speed-up] [1.00] [2.00] [4.00] [8.00] MPICH [speed-up] [1.00] [2.07] [3.91] [7.73] 7 NPB MG Table 7 Execution time of NPB MG (in seconds) SR [speed-up] [1.00] [1.65] [2.66] 7.44 [5.02] NSR [speed-up] [1.00] [1.65] [2.66] 8.01 [4.66] MPICH [speed-up] [1.00] [1.24] [1.85] [2.83] 8 NPB CG Table 8 Execution time of NPB CG (in seconds) SR [speed-up] [1.00] [1.83] [3.30] [6.15] NSR [speed-up] [1.00] [1.79] [3.22] [5.85] MPICH [speed-up] [1.00] [1.72] [2.47] [4.71] MPI/MBCF NSR FIFO eager MPI MPICH NSR MPICH MPI SunOS TCP SSS{CORE FIFO MPI/MBCF SR NSR CG, IS, LU, SP SR SR SR NSR eager write NSR SR 6. MPI MPI AP1000, AP1000+, AP3000 put, get MPIAP 10),11) Cray T3D Shared Memory Access library CRI/EPCC MPI 3) MPP MPI MPICH get 1 MPI-EMX 12) EM-X write MPI_Irecv() EM-X AP3000 T3D MPP

12 12 May NPB IS Table 9 Execution time of NPB IS (in seconds) SR [speed-up] [1.00] 6.35 [1.60] 4.51 [2.25] 2.90 [3.50] NSR [speed-up] [1.00] 6.35 [1.60] 4.69 [2.17] 3.72 [2.73] MPICH [speed-up] [1.00] 7.09 [1.45] 5.61 [1.83] 4.81 [2.13] 10 NPB LU Table 10 Execution time of NPB LU (in seconds) SR [speed-up] [1.00] [1.92] [3.57] [6.28] NSR [speed-up] [1.00] [1.91] [3.52] [6.10] MPICH [speed-up] [1.00] [1.77] [3.37] [5.84] 11 NPB SP Table 11 Execution time of NPB SP (in seconds) SR [speed-up] [1.00] [3.63] [8.30] NSR [speed-up] [1.00] [3.62] [7.74] MPICH [speed-up] [1.00] [2.93] [6.01] 12 NPB BT Table 12 Execution time of NPB BT (in seconds) SR [speed-up] [1.00] [3.98] [9.20] NSR [speed-up] [1.00] [3.98] [9.13] MPICH [speed-up] [1.00] [2.93] [6.53] MPI 7. MPI MPI/MBCF 1 FIFO MPI 100BASE-TX round-trip time 71 s, peak bandwidth MB/s MB/s NAS Parallel Benchmarks SunOS MPICH/TCP MPI/MBCF IPA 1) Bailey, D., Barszcz, E., Barton, J., Browning, D., Carter, R., Dagum, L., Fatoohi, R., Fineberg, S., Frederickson, P., Lasinski, T., Schreiber, R., Simon, H., Venkatakrishnan, V. and Weeratunga, S.: THE NAS PARALLEL BENCHMARKS, Technical Report RNR , NASA Ames Research Center (1994).

13 Vol. 40 No. 5 MPI ) Bailey, D., Harris, T., Saphir, W., Wijngaart, R., Woo, A. and Yarrow, M.: The NAS Parallel Benchmarks 2.0, Technical Report NAS , NASA Ames Research Center (1995). 3) Cameron, K., Clarke, L. and Smith, G.: CRI/EPCC MPI for CRAY T3D (1995). http: // 4) Gropp, W., Lusk, E., Doss, N. and Skjellum, A.: A High-Performance, Portable Implementation of the MPI Message-Passing Interface Standard, Parallel Computing, Vol. 22, No. 6, pp. 789{828 (1996). 5) Matsumoto, T. and Hiraki, K.: MBCF: A Protected and Virtualized High-Speed User- Level Memory-Based Communication Facility, Proc. of Int. Conf. Supercomputing, pp. 259{ 266 (1998). 6), :, 93-ARC-101, Vol. 93, No. 71, pp. 113{120 (1993). 7), : SSS{CORE, 53 1, pp. 37{38 (1996). 8) Message Passing Interface Forum: MPI: A Message-Passing Interface Standard (1995). 9) Message Passing Interface Forum: MPI-2: Extensions to the Message-Passing Interface (1997). 10) Sitsky, D. and Hayashi, K.: Implementing MPI for the Fujitsu AP1000/AP1000+ using Polling, Interrupts and Remote Copying, JSPP '96, pp. 177{184 (1996). 11) Sitsky, D. and Mackerras, P.: System Developments on the Fujitsu AP3000, Proc. of 7th Parallel Computing Workshop (1997). 12),,, : MPI, JSPP '98, pp. 199{206 (1998). ( ) ( ) ACM IBM T J Watson 1990

1 M32R Single-Chip Multiprocessor [2] [3] [4] [5] Linux/M32R UP(Uni-processor) SMP(Symmetric Multi-processor) MMU CPU nommu Linux/M32R Linux/M32R 2. M

1 M32R Single-Chip Multiprocessor [2] [3] [4] [5] Linux/M32R UP(Uni-processor) SMP(Symmetric Multi-processor) MMU CPU nommu Linux/M32R Linux/M32R 2. M M32R Linux SMP a) Implementation of Linux SMP kernel for M32R multiprocessor Hayato FUJIWARA a), Hitoshi YAMAMOTO, Hirokazu TAKATA, Kei SAKAMOTO, Mamoru SAKUGAWA, and Hiroyuki KONDO CPU OS 32 RISC M32R