netmap Web syslog NetFlow I/O netmap netmap OS fluentd 10GbE 1,500 50,000pps 100% netmap NetFlow i

Size: px

Start display at page:

Download "netmap Web syslog NetFlow I/O netmap netmap OS fluentd 10GbE 1,500 50,000pps 100% netmap NetFlow i"

さみがうん
5 years ago
Views:

1 netmap

2 netmap Web syslog NetFlow I/O netmap netmap OS fluentd 10GbE 1,500 50,000pps 100% netmap NetFlow i

3 Abstract Of Bachelor s Thesis Academic Year 2018 Proposal and Implementation of High-Throughput Log Forwarder Using Netmap Summary Server administrators have the opportunity to manage log information of systems and networks such as syslog and NetFlow in order to safely operate diversifying Web services. Since the log information of systems and networks that must be managed is also becoming enlarged as the amount of traffic that flies over the network increases, in order to handle them efficiently, in the middle of log output and collection machines, There is a mechanism called a log forwarder that performs forwarding. However, in conventional log forwarders, performance problems in dealing with the increasing traffic volume in the future are assumed. Therefore, in this research, throughput is improved by using netmap, a high-speed packet I/O framework which improved existing packet processing method, and it is also possible to handle traffic expected to increase further in the future We propose a log forwarder. Using netmap reduces the cost of memory copying performed by the OS for each packet, reduces processing time per packet, and realizes response to a large amount of traffic from multiple servers. As a result, we succeeded in handling a lot of traffic from log output server built as the assumed environment, and can not handle forwarding applications such as fluentd, which was used conventionally, with 1,500 byte long packets at 10 GbE even for inputs exceeding 50,000 pps, packet processing of 100% could be realized. This research has significance in showing a new solution to the collecting method for enormous log data that continues to increase beside the network that will further develop in the future. Keywords netmap, Network Traffic, NetFlow, Log Forwarder Bachelor of Arts in Environment and Information Studies Keio University Kazuaki Masuda ii

4 NetFlow NDE (v5) NDE (v5) NDE (v9) sflow IPFIX netmap OS netmap Intel DPDK fluentd UDP Director iii

5 iv

6 1.1 [1][2] NetFlow (version 5) [5] NDE (version 5) NetFlow v NetFlow v NDE (version 9) [8] sflow NIC NIC netmap netmap netmap fluentd UDP Director gforwarder v

7 2.1 NetFlow v NetFlow v sk buff netmap iperf gforwarder rule dic rule box Ethernet frame [22] fluentd gforwarder UDP Director gforwarder vi

8 1 1.1 Raspberry Pi NetFlow NetFlow IP Cisco NAT NetFlow 1

9 NetFlow 1.1 config 1.1: [1][2] 1.2 2

10 NAT NetFlow 2017 Interop Tokyo ShowNet 161GB [3] GbE 40GbE NIC PC OS API I/O OS 1.2 netmap I/O OS gforwarder

11 1.2: 4

12 2 NetFlow I/O netmap I/O 2.1 NetFlow NetFlow Cisco 1996 IP Flow Exporter(Exporter) Flow Collector(Collector) NetFlow Cisco Huawei NetStream[4] 2.1: NetFlow (version 5) [5] 5

13 NetFlow Cisco NetFlow NetFlow version 5 7 [6] IP IP L3 ToS 7 NetFlow IP ToS (DSCP) AS AS NetFlow NDE(NetFlow Data Export) NDE UDP 2.2: NDE (version 5) NDE (v5) NetFlow version [7] 6

14 2.3: NetFlow v5 2.1: NetFlow v5 version NetFlow (=5) count ( 30) sysuptime unix seconds unix nanoseconds flow sequence number engine type engine ID ID sampling interval 2bits: 14bts: 7

15 2.1.2 NDE (v5) NetFlow version [7] 2.4: NetFlow v5 version 9 Flexible NetFlow version 5 Exporter Collector 8

16 2.2: NetFlow v5 src IP addr IPv4 dst IP addr IPv4 next hop ip addr IPv4 input interface index SNMP interface idx output interface index SNMP interface idx packets bytes first sysuptime( ) last sysuptime( ) src port TCP/UDP dst port TCP/UDP flags TCP IP protocol IP (6=TCP, 17=UDP) TOS IP TOS src AS AS dst AS AS src netmask length IPv4 dst netmask length IPv4 Collector NDE (v9) NetFlow version : NDE (version 9) [8] NetFlow version 9 FlowSet FlowSet Template FlowSet Options Template FlowSet Data FlowSet NetFlow version 9 Template 9

17 2.2 sflow sflow InMon NetFlow [9] NetFlow 1 NetFlow IPX Appletalk XNS IP L3 L2 2.6: sflow Collector NetFlow IP

18 2.3 IPFIX IPFIX Cisco NetFlow v9 IETF [10] Template Withdraw Message IPsec TLS 2.4 netmap netmap Pisa Luigi Risso I/O [11] FreeBSD Linux Windows OS OS [12] OS NIC RX( ) TX( ) NIC (head) (tail) sk buff sk buff linux linux/include/linux/skbuff.h [13] : sk buff head data tail end (1) OS

19 ( 1 ) DMA RX head sk buff ( 2 ) head ( 3 ) NIC CPU NIC sk buff ( 4 ) sk buff tail CPU sk buff head tail 2.7: NIC (2) 2.8 TX CPU ( 1 ) OS sk buff TX ( 2 ) tail 12

20 NIC DMA ( 3 ) NIC head ( 4 ) sk buff sk buff 2.8: NIC sk buff 1GbE 10GbE 84byte 1GbE 10GbE NIC 1GbE NIC 1.49Mpps 10GbE NIC 14.88Mpps CPU netmap netmap 13

21 netmap 2.9 netmap NIC NIC netmap netmap netmapr netmap NIC netmap NIC netmap NIC netmap netmap netmap netmap NIC NIC OS NIC head tail netmap sk buff netmap NIC head tail netmap 2.9: netmap (1) 2.10 NIC ( 1 ) RX netmap ( 2 ) netmap netmap netmap NIC head ( 3 ) netmap tail ( 4 ) 14

22 head tail netmap netmap ( 5 ) head ( 6 ) NIC tail tail netmap NIC netmap netmap tail NIC tail 2.10: netmap (2) 2.11 netmap TX head tail netmap NIC netmap netmap ( 1 ) head ( 2 ) NIC tail NIC ( 3 ) ( 4 ) netmap tail ( 5 ) 15

NIC DPDK NIC 2.4.1 UIO CPU netmap API 2.

23 2.11: netmap OS Socket API 1 I/O 2.5 Intel DPDK Intel DPDK Linux OS netmap NIC DPDK NIC UIO CPU netmap API 2.6 fluentd fluentd[14] Treasure Data Linux UNIX OS 16

24 fluentd INPUT 2.12: fluentd Nagios[15] DB fluentd OUTPUT 2.7 UDP Director UDP Director[16] Lancope NetFlow sflow OS IP syslog[17] SNMP[18] syslog Kiwi Syslog Server[19] SNMP Net-SNMP[20] Exporter Cisco Collector Cisco Stealthwatch Flow Collector ShowNet [3] 17

25 2.13: UDP Director 18

26 3 netmap 3.1 netmap Socket API Socket API iperf : OS NIC FreeBSD 11.0-RELEASE-p9 Intel 82599ES 10-Gigabit Fedora release 25 Intel 82599ES 10-Gigabit : netmap [bytes] [s] [pps] 1,000,000, ,051, iperf

27 3.3: iperf [bytes] [s] [pps] 1,000,000, , bytes Socket API Kpps netmap 8.05Mpps 8 netmap Socket API netmap Socket API netmap OS gforwarder [21] 20

28 4 4.1 gforwarder OS BSD UDP/IP IP Collector( ) IP NetFlow 1. Exporter 2. Collector 3. IP 4. 2 Collector Exporter netmap

4.1: netmap 4.3 4.1 4.3.1 gforwarder OS 2011 netmap FreeBSD 4.

29 4.1: netmap gforwarder OS 2011 netmap FreeBSD 4.1: gforwarder OS FreeBSD 11.0-RELEASE-p9 (GENERIC) NIC Intel 82599ES 10-Gigabit SFP/SFP+ C netmap v

30 3 (1) 4.1 gforwarder Collector gforwarder 4.2: Collector A A A B rule dic rule dic rule box rule dic rule box rule dics rule num 23

31 4.3: rule dic struct in addr srcaddr int dstport struct in addr dstaddr char priority 4.4: rule box struct rule dic[ ] rule dics rule dic int rule num (2) poll poll I/O (3) 2 IP UDP (1) IP UDP IP rule dic dstaddr TX 2.4 netmap zero-copy poll 24

32 5 gforwarder 5.1 gforwarder 1.2 Socket API Exporter 10GbE NIC netmap NetFlow NetFlow NetFlow 3.1 Collector netmap I/O pkt-gen Exporter Collector : Exporter Collector 25

33 : src IP dst Port dst IP netmap Exporter (1) Collector pkt-gen 5.1 Exporter netmap NetFlow Collector Ethernet 174bytes bytes 10GbE 5.2: Ethernet frame [22] Interframe gap 12 bytes MAC preamble 8 bytes MAC header 14 bytes payload size 174 bytes Ethernet CRC 4 bytes ( )bps (212bytes 8) = 5, 896, 226pps 5.90Mpps (2) pkt-gen 26

34 MTU 1,500bytes 1,000pps 5.2 Etherframe 1,538 bytes 10Gbps ( )bps (1, 538bytes 8) = 812, 744pps fluentd fluentd tagomoris fluentd fluent-plugin-datacounter[23] : gforwarder gforwarder Mpps 5.90Mpps 92.20% 27

35 5.2.2 gforwarder fluentd : fluentd (pps) (pps) (%) 1,000 1, ,000 5, ,000 9, ,000 28, ,000 27, ,000 25, ,744 30, : gforwarder (pps) (pps) (%) 1,000 1, ,000 5, ,000 10, ,000 50, , , , , , , : 28

36 fluentd 50,000pps 812,744pps 3.71% gforwarder 100% gforwarder 5.3?? UDP Director [16] : UDP Director gforwarder gforwarder UDP Director 2010 INPUT 5,336,287 pps 37,500 pps OUTPUT 5,401,010 pps 75,000 pps UDP Director INPUT OUTPUT netmap gforwarder Socket API 100% Socket API Socket API netmap gforwarder Collector 29

37 6 6.1 NetFlow CGN netmap I/O OS gforwarder 10GbE NIC netmap fluentd 96% fluentd 3.7% 812,744pps 100% Cisco UDP Director INPUT 142 OUTPUT Socket API netmap netmap 30

38 Socket API netmap INPUT OUTPUT Collector INPUT ARP Ethernet gforwarder DNS ARP ICMP RARP

39 Rodney D. Van Meter III SFC ( ) ShowNet NetFlow Arch 32

40 [1] ntop. last visited on [2] Cisco stealthwatch enterprise. security/stealthwatch/index.html. last visited on [3] Interop Tokyo ShowNet NOC Team. Shownet2017 report. www. slideshare.net/interoptokyo-shownet/shownet2017-report last visited on [4] Netstream tecnology white paper. enenterprise/download/hw_ last visited on [5] Netflow - wikipedia. last visited on [6] Cisco ios netflow data seet. https : / / www. cisco. com / c / en / us / products / collateral / ios-nx-os-software / ios-netflow / product _ data _ sheet0900aecd80173f71.html. last visited on [7] Flowscan architecture. arch.xml. last visited on [8] Netflow version 9. https : / / www. ibm. com / support / knowledgecenter / ja / SSCVHB _ / collector / cnpi _ netflow _ v9. html. last visited on [9] sflow. last visited on [10] IPFIX overview /collector/cnpi_collector_IPFIX_overview.html. last visited on [11] Luigi Rizzo. netmap: A novel framework for fast packet i/o. In 2012 USENIX Annual Technical Conference (USENIX ATC 12), pp , Boston, MA, USENIX Association. 33

41 [12] Giuseppe Lettieri. An introduction to netmap - events. sigcomm.org/sigcomm/2017/files/tutorial-netmap/01-intro.pdf. last visited on [13] IBM - linux. / developerworks/jp/linux/library/l-linux-networking-stack/index.html. last visited on [14] fluentd. last visited on [15] Nagios. last visited on [16] UDP director. https : / / www. lancope. com / sites / default / files / udp-director-data-sheet-emark.pdf. last visited on [17] The syslog protocol - rfc last visited on [18] Simple network management protocol - rfc doc/rfc1157/. last visited on [19] Kiwi syslog server. kiwisyslogserver.html. last visited on [20] Net-snmp. last visited on [21] Phi-phi/gforwarder - github. last visited on [22] The calculations: 10gbits/s wirespeed - netoptimizer. blogspot.jp/2014/05/the-calculations-10gbits-wirespeed.html. last visited on [23] tagomoris/fluent-plugin-datacounter - github. fluent-plugin-datacounter. last visited on

untitled

untitled 25: Part ( ) Chief Technology Officer mshindo@fivefront.com SNMP MRTG HP/OV RMON INTERNET WEEK 2006/12/08 Copyright 2006 Fivefront Corporation, All Rights Reserved. 2 1 SNMP IfInUcastPkts, IfOutUcastPkts