macchan-m-thesis

Size: px

Start display at page:

Download "macchan-m-thesis"

たみえすずがみね
5 years ago
Views:

1 2008 ( 20 ) IP Rodney Van Meter

2 2008 ( 20 ) IP CPU I/O IP I/O IP IP MIPS CPU Ethernet IP 200 CPU IP : CPU 4. FPGA 5. TLB

3 Abstract of Master s Thesis - Academic Year 2008 The Design and Implementation of An IP Bus Bridge Summary What a computer can do is limited by its hardware. The hardware configuration of a computer is statically determined by the physical attachment of devices, such as CPUs, memories and expansion peripherals. Device connections are done using a bus, but buses are limited in distance, number of devices and concurrent transactions, and reconfigurability. Even specialized I/O networks only partially eliminate these problems. However, the hardware resources required by applications vary. Improved flexibility in dynamic system configuration is necessary. This thesis proposes the most flexible and scalable alternative possible: using an IP-level network as the interconnect for both memory and I/O devices. An IP network can improve flexibility, cost and scalability. A bus bridge that provides protocol translation between the computer bus and Internet Protocol has been designed, implemented, and evaluated. The IP Bus Bridge connects the memory and peripherals of a MIPS-based system over Ethernet, with or without IP. The performance of the system with and without cache is evaluated for latencies up to 200 msec, demonstrating that it is possible to connect memory via IP. Key words: 1. BusBridge 2. Bus 3. CPU 4. FPGA 5. TLB Takeshi Matsuya Keio University, Graduate School of Media and Governance

4 PC LAN PC Memory Interconnects NUMA (Non-Uniform Memory Access) (Distributed Memory) Chipset-level Connections vpro Peripheral buses & networks PCI (Peripheral Component Interconnect) PCI Express BUS InfiniBand RDMA (Remote Direct Memory Access) BUS over Ethernet Microcode-level connections Virtualization IP-level connections iscsi & USB/IP IP Use Cases i

5 TLB TLB TLB TLB TLB LAN CPU CPU ii

6 A...63 iii

7 PC LAN Memory PC AMT PCI Express PCI Express Peripheral over IP CPU CPU IP-NUMA ON DEMAND TLB TLB FIFO LAN way set associative) TLB TLB...38 iv

8 CACHE LAN CPU CPU I/O LAN (IPv4) (IPv4)...56 v

9 1 PC LAN Memory PCI PCI Express FPGA TLB vi

10 1 LAN 1.1 [1] CPU CPU CPU CPU [2] 1 CPU CPU 1 1

11 1.1.1 PC LAN [4] LAN 1 PC LAN [3] PC LAN LAN PC 1 PC LAN Memory!" #$ %&' () *+(),-.7AB%&./0.! :; 09<=>?-.//@! AB1C.7@ 8D :;.889<=>?-.//E! AB1CD7. FG456 FF9:; E889<=>?- D@@@! AB1HIC.7@ FG :;.7@FJ<=>?- D@@D! AB1CKLMN?--C.7@OI0P D7EJ:; DJ<=>?- D@@Q! AB1CKLMN?--CD7@OI.FP EJ:; 0J<=>?- D7Z3[./08! 1KKK0@D78CCCC.@R32KE.@9<M-.7DE9<=>?-.//E! 1KKK0@D78SCC.@@R32KHTI.@@9<M-.D7E9<=>?-.///! 1KKK0@D78U<C.@@@R32KHT.J<M-.DE9<=>?- D@@G! 1KKK0@D78UVC.@JR32KHBIG.@J<M-.7DEJ<=>?- D@@I!.@@JR32K.@@J<M-.D7EJ<=>?- 879K9XY\.//8! KWXCWY39COLHDHDHDP FG456 FF9:; DFG9<=>?-.//Q! 2WYH2WY39OAB.88P FG :;.7@FFJ<=>?- D@@.! WWYH2WY39OABHD.@@P FG456 DFF9:; D7.88J<=>?- D@@G! WWYDH2WY39OABDHFG@@P FG456 G@@9:; F7GJ<=>?- D@@F! WWY8H2WY39OAB8H.GG@@P FG456 /@@9:;.G7GJ<=>?- 2 PC LAN Memory 2

12 1.1.2 HD CPU CPU 1.2 CPU IP [23][24][25][26]

13 (ISO OSI )

14 2.2 PC PC PC PC 4 CPU CPU FSB(Front Side Bus) CPU Intel GMCH Graphics and Memory Controller Hub I/O Intel ICH(I/O Controller HUB) 4 PC 2.3 Memory Interconnects CPU NUMA (Non-Uniform Memory Access) NUMA[5][12][13] CPU CPU 5

15 CPU Memory CPU NUMA SMP CPU CPU (Distributed Memory) CPU PC MPI[20] (Message Passing Interface) PVM[21] (Parallel Virtual Machine) CPU MPI PVM CPU 2.4 Chipset-level Connections vpro vpro Intel PC Intel VT (Virtualization Technology) PC Intel AMT (Active Managemtn Technology) vpro CPU LAN 5 vpro PC PC Web 1. Fan CPU (BIOS OS) 6

16 4. OS PC CPU 5 AMT 2.5 Peripheral buses & networks PCI (Peripheral Component Interconnect) PCI SIG (Special Interest Group) 1992 PCI1.0[8][14][17] PCI Express[15] PC 7

17 2 2 PCI 33MHz/66MHz 32/64bit 32bit 133MB/s 533MB/s PCI Express BUS PCI Express BUS 2002 PCI-SIG PCI Intel NGIO (Next Generation I/O) PCI 3GIO (Third Generation I/O) PCI-Express NGIO HP IBM Future I/O InfiniBand 3 3 PCI Express GHz ( 1 ) 32/64bit 32bit 250MB/s( 1) 4GB/s( 8) 8b/10b PCI-Express 6 5 8

18 TLP (Transaction Layer Packet) TLP 1) 2) I/O 3) 4) 5) ACK/NACK 8b/10b 6 PCI Express PCI Express 7 32bit 12 64bit 16 9

7 PCI Express 2.5.3 InfiniBand InfiniBand I/O InfiniBand 2.5Gbps 8B/10B 250MB/s 4X 10Gbps 12X 30Gbps 4X HPCC (High Performance Computing Clusters) PC CPU 2.5.4 RDMA (Remote Direct Memory Access) CPU TCP/IP RDMA[32] CPU PC 2.

19 7 PCI Express InfiniBand InfiniBand I/O InfiniBand 2.5Gbps 8B/10B 250MB/s 4X 10Gbps 12X 30Gbps 4X HPCC (High Performance Computing Clusters) PC CPU RDMA (Remote Direct Memory Access) CPU TCP/IP RDMA[32] CPU PC BUS over Ethernet BUS over Ethernet Ethernet PCI-Express ExpEther[18] PCI-Express Ethernet PCI-Express 10

20 PCI-Express I/O I/O 8 Node-1 Node-2 Node-1 1 Node-2 2 $20000 Node-1 CPU $ Microcode-level connections CPU Virtualization Virtualization I/O PC 11

21 1. 2. CPU OS 2.7 IP-level connections IP [28][29][30][31] iscsi & USB/IP iscsi USB/IP IP SCSI iscsi[19][22] (Internet Small Computer System Interface) USB USB over IP 9 SCSI USB API

22 9 Peripheral over IP ) CPU 2) I/O 3) 4) IP 5) InfiniBand IP IP HPCC 13

23 4!"#$%&'( )*+,-./0,- 123,-.* : ;<=>?@A.BC<?D>BB<DCE F G F G F H AI+;J F G F G F K ALM123 FN: G G G F O )PQRE<CST<U<TA)>BB<DCQ>BE F F F G G V AU*?> G WNH G F G X *<?QRP<?YTAZ[E<E G F F G G \ A*).]A*).A^_R?<EE G F F G G ` A.BaQBQbYBc F F F G FNK d Aef;J G G G F g :h A^_R^CP<? G FNO F WNV G :: ;QD?>D>c<ST<U<TAD>BB<DCQ>BE g g g g g :H AiQ?C[YTQjYCQ>B G F F F F :K.*ST<U<TAD>BBB<DCQ>BE G FNO G F G :O AQk)k./+kbA>U<?A.* G F G F W Ethernet Frame 14

24 3 IP IP 3.1 CPU DMA CPU CPU CPU 10 CPU LAN CPU LAN CPU CPU 10 Node-1 CPU Node-2 2A CPU 10 CPU 15

25 CPU LAN LAN 11 Node-1 CPU Node-2 2A LAN Node-1 CPU 2A Node-2 CPU Use Cases 16

26 CPU Node-1 Node-2 NORA (No Remote Memory Access) Message Passing 17

27 13 CPU NORA NUMA 14 CPU NUMA NUMA IP-NUMA 14 IP-NUMA ON DEMAND 18

28 15 16 I/O 16 19

29 LAN 4. 20

30 5. CPU PC CPU TLB[3][6][10] 17 21

CPU 18 1. CPU 2. CPU TLB 3. TLB TLB 4. 5.

31 CPU CPU 2. CPU TLB 3. TLB TLB CPU 18 TLB CPU CPU 2. CPU TLB 3. TLB CPU 22

32 4. LAN LAN TLB ID 5. LAN 6. CPU 19 TLB CPU Memory CPU 23

33 CLOCK ADDR DATA RD WR BUSREQ BUSACK BUSREQ ~READY (WAIT) 24

34 3.5.2 CPU Instruction CPU CPU Data CPU 5 5 CPU I CPU D CLOCK 1 ADDR 16~32 DATA 8 32 RD 1 WR 1 BUSREQ 1 BUSACK 1 ~READY (WAIT)

Node1 Node2 Node1 $0000000 $100000 Node2 CPU 4.6.1 21 3.

35 Node1 Node2 Node1 $ $ Node2 CPU Ethernet Frame IP 100Base-T Ethernet IPv4 UDP IPv6 UDP PC PCI-Express Ethernet Frame Ethernet Frame IPv4 UDP 2 PC PCI Express IPv4 26

36 2.3 GB Ethernet 10GB Ethernet IP PCI Express 1 2.5Gbps (250MB/s) Ethernet Frame IPv4 UDP IPv6 UDP CPU LAN LAN FIFO LAN FIFO 27

37 LAN FIFO FIFO $101 9bit 3. IP UDP FIFO 4. CRC32 $ $102 FIFO 9B/8B 8 $100 9 Ethernet RJ FIFO 28

38 LAN LAN [7] FIFO FIFO LAN TLB 29

39 TLB TLB TLB ~READY(WAIT) 30

40 CPU WAY 1 Index WAY Index 1 WAY $0000 $1000 $1000 $ WAY $0000 $1000 $2000 $3000 WAY 2 WAY Index Tag LRU (Least Recently Used) WAY 1 CPU 4 (32bit) CPU Index 6bit 64 Way 31

41 TLB (Translation Lookaside Buffer) TLB TLB TLB I/O 32

42 I/O I/O 28 TLB 28 2-way set associative) TLB 29 TLB TLB Tag Tag2 1 4G 1 2 I/O TLB Type 33

43 TLB Host ID 29 TLB 3.9 LAN TLB 34

44 CPU

45 4.2.1 Verilog HDL[11][16] Veirlog FPGA FPGA 6 6 FPGA FPGA Xilinx Spartan 3AN Starter Kit FPGA Spartan 3AN 75 RJ-45 (100Base-TX) LAN PHY SMSC MHz VGA 15 8 LED, LCD, Audio PS/ Verilog HDL FPGA Xilinx ISE 10.1 Verilog HDL VeritakWin MIPS[9] MIPS GNU as

46 Clock Width ADDR,DATA 1 CPU 12.5MHz 32bit Read 2 CPU 12.5MHz 32bit Read,Write 3 50MHz 32bit Read,Write 4 25MHz 32bit Read,Write, Ready, CPU CPU LAN TLB TLB TLB TLB 37

47 4.6.1 TLB ID 9 TLB TLB 2 8 TLB TLB CPU 9 TLB!" #$ %&'( )* +,-./ :;;0<=59= >,-./0L54M3I:07889:;; N OE'/PQRS-1456T U,-./0VMW; :;; >? XABCDE Y,-./0-145=31I U RSL:Z19W[+S-7\[?S]V^[>SPH_/]H `,-./0]54M:0HI5a6: + +Sbc&defgh i,-./0j:580ki6w + +Slmnmompq TLB TLB Verilog HDL 32 wire [`TLB_ENTRY-1:0] TLBI_Hit; assign TLBI_Hit= TLBI_Logical_Address_Start[1]<=INST_ADDR&INST_ADDR<=TLBI_Logical_Address_End[1]? 1 : 0 TLBI_Logical_Address_Start[1]<=INST_ADDR&INST_ADDR<=TLBI_Logical_Address_End[2]? 2 : 0 TLBI_Logical_Address_Start[1]<=INST_ADDR&INST_ADDR<=TLBI_Logical_Address_End[3]? 3 : 0 TLBI_Logical_Address_Start[2]<=INST_ADDR&INST_ADDR<=TLBI_Logical_Address_End[4]? 4 : 0 32 TLB 38

4.6.3 TLB CPU 33 CPU INST_ADDR CPU INST_ADDR TLBI_Hit TLB CurI_Remote 0 33 CPU 34 0000F 00010 00010 TLB

48 4.6.3 TLB CPU 33 CPU INST_ADDR CPU INST_ADDR TLBI_Hit TLB CurI_Remote 0 33 CPU F TLB INST_ADDR TLBI_Hit 1 2 INST_ADDR CurI_Remote 1 CPU INST_WAIT (~READY) WAIT LAN 100Base-T CPU WAIT 39

49 34 40

50 bit CACHEI_Tag `define CACHE_ENTRY 64 ;Cache Line Size 64 Bytes (512bit) assign CacheI_Tag = inst_addr[ 31:12 ]; assign CacheI_Index = inst_addr[ 11:6 ]; assign CacheI_Offset = inst_addr[ 5:0 ]; reg CACHEI_LRU [ 0:`CACHE_ENTRY-1 ]; 0:Way0 1:Way1 reg [7:0] CACHEI_Machine_Address [ 0:`CACHE_ENTRY-1 ]; reg [19:0] CACHEI_Tag? [ 0:`CACHE_ENTRY-1 ]; reg [511:0] CACHEI_Data? [ 0:`CACHE_ENTRY-1 ]; reg CACHEI_Invalid? [ 0:`CACHE_ENTRY-1 ]; 0:Valid 1:Invalid reg CACHEI_Modified? [ 0:`CACHE_ENTRY-1 ]; 0: 1:Dirty( ) reg CACHEI_Owner? [ 0:`CACHE_ENTRY-1 ]; 0: 1:Multi ( ) 35 CACHE 41

51 4.8 LAN LAN 24 2 FIFO FIFO 36 LAN 100BASE-T RAM FIFO if ( E_RX_DV ) begin // Receive Data is valid if ( rx_status == 0 ) begin // rx_status <= E_RXD[3:0] == 4'b1101? 1 : 0; fifo_r_wr <= E_RXD[3:0] == 4'b1101? 1 : 0; fifo_r_din[8:0] <= 9'b ; rx_nibble <= 0; end else if ( ~E_RXD[4] ) begin // if ( rx_nibble == 0 ) begin fifo_r_wr <= 0; fifo_r_din[3:0] <= E_RXD[3:0]; rx_nibble <= 1; // 4bit end else begin fifo_r_wr <= 1; // 4bit fifo_r_din[8:4] <= {1'b0,E_RXD[3:0]}; rx_nibble <= 0; end end else begin // RX_ER rx_status <= 0; fifo_r_wr <= 1; 36 LAN Read Memory Write Memory Completion 10 42

!" #$%& #'& ( )*+,-.*/012-)*34*56-7869-:+:9* (;< = )*+,-.*/012-)*34*56-7869046-:+:9* ((< >?186*-.

52 !" #$%& #'& ( )*+,-.*/012-)*34* :+:9* (;< = )*+,-.*/012-)*34* :+:9* ((< >?186*-.*/012-)*34* :+:9* (@< A?186*-.*/012-)*34* :+:9* (B< C D+:9*-E8F*-F-85-GFH+I8,-J)*5*1H*,K =@< L D0/MI*680F-7869-N+6+ )*3O@;< P D0/MI*680F N+6+ )*3OD;< 4.9 CPU 32bit RISC CPU[6] CPU CPU 32bit ) 2) 3) ALU 4) 5) 37 CPU 43

53 IF RR IF IF RR EX IF,RR,EX,MA,RW CPU CPU TLB LAN 32 RISC CPU 44

54 CPU I/O 8 32 CPU CPU Z80 45

40 8 8 11 11 FPGA CPU I/O ROM RAM/VRAM LAN Xilinx

56 FPGA CPU I/O ROM RAM/VRAM LAN Xilinx Spartan 3E Starter Kit Xilinx Spartan 3E (50 ) FPGA Z-80(4MHz) I/O FPGA Keyboard Video Controller(80 25 ) DMA Controller Beep BASIC ROM(32Kbytes) 16Kbytes ( ) FPGA 100Base-T 47

57 TLB 1. I/O I/O $00~$ I/O 2. $F300~$FEB

58 3. CPU $F300~$F8DB 2 Node-1 Node-2 CPU 2 CPU !"#$%&'()*+,-."#$%&/$#'(,- 0"/$#12(345678(9:;< =>?@#AB C C C 1. I/O I/O CPU CPU IP 49

59 !"#$%&'( )*+,-./0,- 123,-.*45 @ 50

60 5.3 CPU dummynet[33] 16ms 200ms 14 HUB HUB dummynet (16ms RTT) ( ) dummynet (200ms RTT) ( ) IPv4 UDP IPv6 UDP Ethernet 44 2 Node 51

44 32 15 15 FPGA CPU I/O ROM RAM LAN Xilinx Spartan 3E

61 FPGA CPU I/O ROM RAM LAN Xilinx Spartan 3E Starter Kit Xilinx Spartan 3E (50 ) FPGA 32bit RISC (12.5MHz) I/O FPGA LED SWITCH 8Kbytes ( ) 16Kbytes ( ) FPGA 100Base-T Node-1 CPU Tektronix TDS MHz (1GS/s) 52

62 CPU CPU s 12.49MHz CPU TP HUB Node HUB HUB 1 Ethernet TP Ethernet 7.9 s IPv s 2 HUB 26.4 s 34.8 s 12490KHz TP Ethernet 126.6KHz

63 17 SI"TFUJVL WXJYP3Z[ "\]^_`3Z[!"#$ %&'()*+,-,./ 012'34* %&'()*+,-,./ 012'34* %&'()*+,-,./ 012'34* 5 678/9:/7 ;<= 5>?@?AA 55<A =A@=5A >?<B C;@DDA > EFGB 5><B DA@?HA 5C<> ;H@;?A >D<D CB@;>A C EFG? 5H<??B@5AA 5?<C?A@=;A CB<D >D@;BA B IJKL MNOO-:/7PQR5?O) MNOO-:/7PQR>AAO) IJKL!"#$ %&'()*+,-,./ 012'34* %&'()*+,-,./ 012'34* %&'()*+,-,./ 012'34* 5 678/9:/7 > EFGB 5?@AA>?><B= >AA@A5C B<== C EFG? B IJKL A<ADAH 5>@B=A@AAA ms Ethernet TP 95ms I(JK LM0NORS LM0NOPQ &'()*+,-. (/01234 &'()*+,-. (/ :;97 <= ><? 5@A5 5@AB > C*DE 5EF GEF 5@AG 5@A< G C*D? 5FF E>E 5@AE 5@5> E ',H.!"#$% ',H. A@<B 54

64 45 LAN (IPv4) ms 16ms 200ms 200ms 17 5Hz CPU 12.5MHz 46 A !"#$% &'()$% &'*++$% ;<=> (,-./ (+*+ (9:+ * ;<=>?:+ 55

65 46 (IPv4) CPU 200ms CPU 1 64 A $C IP 8 IP CPU I/O 2 TLB CPU 56

66 200ms 57

67 ) CPU 2) I/O 3) 4) IP 5) IP CPU I/O IP 8 CPU I/O IP TLB 2 32bit MIPS 200ms 58

68 6.2 UDP 6.3 IP CPU I/O IP OS Linux OS CPU 59

69 Rodney Van Meter Rodney Van Meter ITC WIDE Segway 60

70 [1] Gordon E. Moore,"Cramming more components onto integrated circuits",electronics, Vol.38,April [2] John von Neumann,"First Draft of Report on the EDVAC", June [3], 4, (2008) [4],, (1991) [5], 1996 [6],, CQ (2006) [7], Ethernet, CQ (2006) [8], PCI, CQ (2005) [9] Gerry Kane, mips RISC Architecture R2000/R3000 [10], 486, (1994) [11], HDL VLSI 1999 [12] Curt Schimmel, Symmetric Multiprocessing and Caching for Kernel Programmers Addison-Wesley Publishers [13] Interface, CQ 2007 [14], PCI I/O, (1999) [15], PCI Express, (2007) [16], Verilog-HDL, (2002) [17], FPGA PCI, LSI 94-21, pp , [18], ExpEther, Dec [19] J. Satran and K. Meth and C. sapuntzakis and M. Chadalapaka and E. Zeidner,"Internet Small Computer Systems Interface", April 2004, RFC3720. [20] Fran Berman and Geoffrey C. Fox and Anthony J. G. Hey, Grid Computing, John Wiley & Sons Inc,

71 [21], Linux,,2007. [22] Tom Clark, IP SAN,,2003. [23],, 100BASE-TX,,, pp (November 1997). [24],,,, Vol.98, No.233( ) pp [25],,,, Vol , No.1( ) pp [26],,,,Vol.100, No.20( ) pp [27] Martin W. Sachs and Avraham Leff and Denise Sevigny, LAN and I/O Conver-gence:A Survey of the Issues, IEEE Computer, pp.24-32,dec [28] Rodney Van Meter, Greg Finn,Steve Hotz, Bruce Parham, Netstation Project, [29] Garth A. Gibson and Rodney Van Meter, Network Attached Storage Architecture, Communications of the ACM Vol.43,pp.37-45,Nov [30] Rodney Van Meter, A Brief Survey of Current Work on Network Attached Peri-pherals, ACM Operating Systems Review,pp.63-70,Jam [31] Rodney Van Meter and Greg Finn and Steve Hotz, VISA: Netstation's Virtual In-ternet SCSI Adapter, [32] S. Bailey and T. Talpey, The Architecture of Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) on Internet Protocols, December 2005, RFC 4296 [33] Luigi Rizzo, dummynet, 62

72 A 63

73 1.align 4 2.set noreorder 3.set noat 4.text 5.globl _start 6 _start: A0000 addiu $10,$0, B0000 addiu $11,$0, addiu $1,$0, c addiu $2,$0, addiu $3,$0,0 14 loop1: addiu $4,$0, nop c nop 18 loop2: B0001 addiu $11,$11, addiu $4,$4, nop c nop nop B5020 add $10,$10,$ nop c nop FFF7 bne $4,$2,loop nop nop c nop nop addiu $3,$3, nop c nop nop FFEB bne $3,$1,loop nop c nop nop nop BC addiu $5,$0,lvar c nop nop nop nop c ACAA0000 sw $10,0($5) nop nop nop c nop 51 00a nop 52 00a nop 53 loop3: 54 00a A j loop ac nop 56 00b nop 57 00b nop 58 00b nop bc lvar:.long 0x

Shonan Institute of Technology MEMOIRS OF SHONAN INSTITUTE OF TECHNOLOGY Vol. 41, No. 1, 2007 Ships1 * ** Development of a Small-Mid Range Paral

MEMOIRS OF SHONAN INSTITUTE OF TECHNOLOGY Vol. 41, No. 1, 2007 Ships1 * ** ** ** Development of a Small-Mid Range Parallel Computer Ships1 Makoto OYA*, Hiroto MATSUBARA**, Kazuyoshi SAKURAI** and Yu KATO**