Intrusion Detection Method using Online Learning by Kouki Takahata BA Thesis at Future University Hakodate, 2017 Advisor: Ayahiko N

Size: px

Start display at page:

Download "Intrusion Detection Method using Online Learning by Kouki Takahata BA Thesis at Future University Hakodate, 2017 Advisor: Ayahiko N"

のぶみつすみだ
4 years ago
Views:

1 Intrusion Detection Method using Online Learning by Advisor: Ayahiko Niimi Department of Media Architecture Future University Hakodate January 31, 2017

2 Abstract Cyber-attacks such as spam mail or DDoS attack are suffered in recent years. To detect those attacks, many researches have been conducted on detecting intrusions by monitoring packets passing network equipment. In this study, I propose the method that detecting cyber-atacks with characteristics with low memory and naive processing using online machine learning. I used feature values by each TCP session and disscussed effectiveness of online learning by comparing the accuracy between SCW (Soft Confidence-Weighted) as online machine learning algorithm and SVM (Support Vector Machine) as offline algorithm. In the experiment, I used CCC (Cyber Clean Center) DATAset provided MWS(Malware WorkShop) as attacked honeypot s log data. The experimental results shows the accuracy of SVM using RBF kernel resulted in approximately 90% and the accuracy of SCW resulted in approximately 80%. We conclude that although SCW is expected to decrease low memory and low processing speed, it could not be kept the sufficient accuracy. Keywords: Data Mining, Security, Online Learning, Network, SCW : DDoS.. SCW(Soft Confidence-Weighted). TCP SVM SCW MWS() CCC DATAset RBF SVM 9 SCW 8 SCW :, SCW

3 Snort Botnet Detection by Monitoring Group Activities in DNS Traffic SmartShifter C&C CW (Confidence-Weighted learning) SCW (Soft Confidence-Weighted learning) CCC DATAset i

4 ii

5 ZeuS FTP 2011 P2P ZeuS GameOverZeuS [1] Web DDoS [2] [3]. (IDS:Intrusion Detection System) [3] IDS IDS DNS HTTP [4][5] [6] 1

6 1.2 SCW SoftConfidence-Weighted

7 2 (1) (2) (3) Snort Snort[7] TCP 80 GET alert tcp any any -> any 80 ( content :!" GET ";) 2.2 3

8 2.2.1 Botnet Detection by Monitoring Group Activities in DNS Traffic DNS DNS SmartShifter SmartShifter [8] SmartShifter EM SmartShifter KDDCup99 82% IDS C&C TCP SVM HTTP IRC P2P 3 SVM AdaBoost [9] 36 AdaBoost AdaBoost 4

9 SVM EM 2.1: [7] [4][5] [8][6][9] 5

10 Algorithm 1 t Algorithm 1 1: w (1) = 0 2: for t = 1, 2,... do 3: if y (t) w (t)t < E then 4: w (t+1) = w (t) + y (t) αax (t) 5: else 6: w (t+1) = w (t) 7: end if 8: end for [10] 2 6

11 1 Algorithm 2 η Algorithm 2 1: w (1) = 0 2: for t = 1, 2,... do 3: if y (t) w (t)t 0 then 4: w (t+1) = w (t) + y (t) ηx (t) 5: else 6: w (t+1) = w (t) 7: end if 8: end for CW (Confidence-Weighted learning) CW[11] CW (µ (t+1), Σ (t+1) ) = arg min D KL (N(µ, Σ) N(µ (t), Σ (t) )) µ,σ subject to Pw N(µ,Σ)(y (t) w T x (t) 0) η (3.1) µ, Σ D KL µ (t+1) = µ (t) + α (t) y (t) Σ (t) x (t) Σ (t+1) = Σ (t) β (t) Σ (t) x T x (t) Σ (t) (3.2) α β α = max 0, 1 v t ζ ( m tψ + m 2 ϕ 4 t 4 + v tϕ 2 ζ) β = α t ϕ ut + v t α t ϕ (3.3) 7

12 CW SCW (Soft Confidence-Weighted learning) CW SCW[12] CW Pw N(µ,Σ)(y (t) w T x (t) 0) η (3.4) y (y) µ T x (t) ϕ x (t)t Σx (t) (3.5) l ϕ (µ, Σ, x (t), y (t) ) = max(0, ϕ x (t)t Σx (t) y (y) µ T x (t) ) (µ (t+1), Σ (t+1) ) = arg min D KL (N(µ, Σ) N(µ (t), Σ (t) )) µ,σ + Cl ϕ (µ, Σ, x (t), y (t) ) (3.6) CW µ (t+1) = µ (t) + α (t) y (t) Σ (t) x (t) Σ (t+1) = Σ (t) β (t) Σ (t) x T x (t) Σ (t) (3.7) CW α, β 1 α = min C, max 0, v t ζ ( m tψ + m 2 t β = ϕ v tϕ 2 ζ) α t ϕ ut + v t α t ϕ (3.8) Algorithm CW SCW 3.2 CW SCW Σ x 8

13 Algorithm 3 SCW 1: Inputs: parameters C > 0, η > 0 2: Initialize: µ 0 = (0,..., 0) T, Σ 0 = I 3: for t = 1,...T do 4: x t R d 5: ŷ t = sign(µ t 1 x t ) 6: y t 7: l ϕ (N(µ t 1, Σ t 1 ); (x t, y t )) 8: if l ϕ (N(µ t 1, Σ t 1 ); (x t, y t )) > 0 then 9: µ t+1 = µ t + α t y t Σ t x t 10: Σ t+1 = Σ t β t Σ t x T t x t Σ t 11: α t β t (SCW-I,SCW-II) 12: end if 13: end for CW SCW 3.1: CW SCW SCW CW 9

14 :

15 4.2: 4.3: pcap csv csv csv SCW 4.3 SCW SCW SCW CPU SVM SVM Libsvm n O(n 3 ) 11

16 [13] 4.4: 12

17 5 5.1 SVM SCW [13] SVM NaiveBayes SVM SVM SCW CCC DATAset[14] Wire- Shark 2 10Fold : Python Numpy Scipy scikit-learn pandas Wireshark Numpy Python Python Scipy Numpy SCW scikit-learn Python SVM 13

18 pandas Python Wireshark IP CCC DATAset MWS anti-malware engineering WorkShop [14] CCC DATAset MWS CCC DATAset MWS CCC Cyber Clean Center 3 OS tcpdump Windows XP : CCC DATAset IP IP TCP, UDP 14

19 5.2.2 Wireshark [15] BitTorrent MSN Messenger P2P 5.3 pcap Line Skype : FTP FTP 23 Yandex Mail SMTP over SSL 465 SSH Openshift.com SSH 22 Web Google HTTP 80, 443 Yahoo Twitter 2ch Wireshark HTTP 80 Skype (TLS) 443 Line (TLS) 443 Radiko RTMP 1935 Youtube, HTTP 80, IP IP CCC DATAset pcap tshark pandas TCP tshark Wireshark CUI ID ID tshark ID CSV CSV pandas 1 CSV

20 5.1: 5.4: [6] TCP SVM TCP 1 TCP : sec byte byte 16

21 5.4 (accuracy) 10Fold SCW η,c η = {1.0, 10.0}, C = {1.0, 10.0, 100.0, } 5.6 SVM RBF SVM : SCW η C : SVM γ C RBF RBF RBF RBF RBF RBF RBF RBF RBF SVM SVM C = {1.0, 10.0, 100.0, } RBF SVM C = {10.0, 100.0, } γ = { , , 0.001} 17

22 SCW SVM SVM 18

23 6 6.1 SCW Python SVM scikit-learn C 6.2 SVM SVM 9 SVM 5 SCW 7 8 SVM SVM SVM SCW SCW SVM SVM SVM SVM SVM SVM SCW SVM SVM SCW 19

24 7 TCP SCW(Soft Confidence-Weight Learning) SVM SCW CCC DATAset2011 SVM 8 SCW 7 SCW SCW 20

25 MWS( ) CCC DATAset 21

26 [1] N. Etaher, G. R. S. Weir, and M. Alazab. From ZeuS to Zitmo: Trends in Banking Malware. In 2015 IEEE Trustcom/BigDataSE/ISPA, Vol. 1, pp , August [2] M. Feily, A. Shahrestani, and S. Ramadass. A Survey of Botnet and Botnet Detection. In 2009 Third International Conference on Emerging Security Information, Systems and Technologies, pp , June [3] A. Milenkoski, M. Vieira, S. Kounev, A. Avritzer, and B. D. Payne. Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices. ACM Computing Surveys (CSUR), Vol. 48, No. 1, pp. 1 41, September [4] H. Choi, H. Lee, H. Lee, and H. Kim. Botnet Detection by Monitoring Group Activities in DNS Traffic. In 7th IEEE International Conference on Computer and Information Technology (CIT 2007), pp , October [5] Guofei Gu, Roberto Perdisci, Junjie Zhang, and Wenke Lee. BotMiner: Clustering Analysis of Network Traffic for Protocol- and Structure-independent Botnet Detection. In Proceedings of the 17th Conference on Security Symposium, SS 08, pp , Berkeley, CA, USA, USENIX Association. [6]. C&C., Vol. 56, No. 9, pp , September [7] Snort - Network Intrusion Detection & Prevention System. [8]. :5. 3., Vol. 46, No. 1, January [9] , Vol. 2012, No. 3, pp , October [10]..,, April [11] Mark Dredze, Koby Crammer, and Fernando Pereira. Confidence-weighted Linear Classification. In Proceedings of the 25th International Conference on Machine Learning, ICML 08, pp , New York, NY, USA,

27 [12] Jialei Wang, Peilin Zhao, and Steven CH Hoi. Exact soft confidence-weighted learning. arxiv preprint arxiv: , [13] Abdiansah Abdiansah and Retantyo Wardoyo. Time Complexity Analysis of Support Vector Machines (SVM) in LibSVM. International Journal of Computer Applications, Vol. 128, No. 3, pp , October [14]. MWS 2011 Datasets. 2011, Vol. 2011, No. 3, pp. 1 5, October [15].. Master s thesis,,

29 CCC DATAset SCW SVM

29 jjencode JavaScript

29 jjencode JavaScript Kochi University of Technology Aca Title jjencode で難読化された JavaScript の検知 Author(s) 中村, 弘亮 Citation Date of 2018-03 issue URL http://hdl.handle.net/10173/1975 Rights Text version author Kochi, JAPAN http://kutarr.lib.kochi-tech.ac.jp/dspa