Vol. 0 No Intrusion Detection System IDS IDS HTTP LAN 3 Machine Learning Based IDS with Automatic Training Data Generation Akira Yamada, Yutak

Size: px

Start display at page:

Download "Vol. 0 No Intrusion Detection System IDS IDS HTTP LAN 3 Machine Learning Based IDS with Automatic Training Data Generation Akira Yamada, Yutak"

ときなうばら
5 years ago
Views:

1 Vol. 0 No Intrusion Detection System IDS IDS HTTP LAN 3 Machine Learning Based IDS with Automatic Training Data Generation Akira Yamada, Yutaka Miyake, Keisuke Takemori and Toshiaki Tanaka Although many intrusion detection systems based on learning algorithms have been proposed to detect unknown attacks or variants of known attacks, most systems require sophisticated training data for supervised learning. Because it is not easy to prepare the training data, the anomaly detection systems are not widely used in the practical environment. On the other hand, misuse detection systems that use signatures to detect attacks are deployed widely. However, they are not able to detect unknown attacks or variants of known attacks. So we have proposed a new anomaly detection system, which detects the variants of known attacks without preparing the training data. In this system, we use outputs of signature-based conventional IDS to generate the training data for anomaly detection. This system identifies novel features of attacks, and generates generalized signatures from the output of IDS to detect the variant attacks. We conducted experiments on the prototype system with three types of traffic data, 1999 DARPA IDS Evaluation Data, attacks by vulnerability scanner and actual traffic. The results show that our scheme can detect the variants of attacks efficiently, which cannot be detected by the conventional IDS. 1. (IDS: Intrusion Detection System) IDS KDDI KDDI R&D Laboratories Inc. IDS 3),6),16),17),20) IDS IDS IDS 1),10) 8),9),18),21) 0

2 Vol. 0 No. 0 1 DARPA IDS evaluation data 1998, ),13) Web 11) DARPA DARPA 15) IDS 19) 7) p IDS HTTP LAN 3 2 IDS 3 IDS IDS 2.1,2.2 IDS IDS IDS IDS CodeRed II ISAPI.ida attempt..[.fl..r..!..e...@.g.dq."5...[...p...hag fp get /default.ida?xxxxxxxxxxxxxxxxxxx XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX alert tcp any any -> any 80 (msg:"web-iis ISAPI.ida attempt"; flow:to_server, established; uricontent:".ida?"; nocase;... 1 CodeRed II Fig. 1 Packet Payload and Signature of CodeRed II Warm Fig. 2 2 IDS A Flow of Machine Learning Based IDS. IDS Snort 3) 1 HTTP URI ida? 2.2 IDS IDS 2 ( 2) IDS ( 1 ) ( 2 ) ( 3 )

3 (tcpdump ) ( ) ) ( ) ( ) a5d fb3d 0006 a34f db e1ab 10ac ac c000 a34f db e134 10ac e da 000e 005a 0000 a5d fb3d 0006 ffff ffff ffff c a IP_len TOS len c p n n n p n... odds(x_is_hostile)= P(positive x) P(negative x) ( ) ( ) ( ) odds(x_is_hostile)=? Model ( ) Positive/ Negative 3 Mahoney Fig. 3 An Example of Componets that Mahoney et al. s System Employs. ( 1 ) ( 2 ) 3 Mahoney 14) Mahoney tcpdump 2.3 FP (False Positive) FN (False Negative) 2 IDS FP ISAPI.ida attempt ida? URI FP IDS IDS IDS IDS FP IDS IDS IDS IDS IDS IDS DARPA IDS evaluation data 1998, ),13) Web 11) DARPA 15) IDS 1-1: IDS FP FN 1-2: IDS 1-3: 3. IDS 3.1 IDS IDS IDS 2 IDS , 1999 DARPA IDS Evaluation Data 12),13) 3.2 DARPA 13) DARPA 54 54

4 Vol. 0 No. 0 3 (a) Conventional Machine Learning 4 Fig. 4 Proposed System. IDS 2-1: 2-2: 4. 2 IDS , tcpdump tcpdump (b) Machine Learning with Automatic Learning Data Generation Fig. 5 5 Automatic Learning Data Generation. IP ID ( 1 ) ID vi(i = 0, 1...) ID IP IP, ID vi = {T ime vi scrip vi dstip vi srcp ort vi dstp ort vi } ( 2 ) ID aj (j = 0, 1...) ID IP IP ID aj = {T ime aj scrip aj dstip aj srcp ort aj dstp ort aj} ( 3 ) ID vi ID aj ID vi ID aj ± T ID ai ID vj 4.3 2)

5 Table 1 1 Parameters for Learning Algorithm. 2 0 Gini (impurity function) Gini Gini Gini S, C j S, C j A Gini gini(s) gini(s) = 1 P (S, C j) 2 P (S, C j ) = Cj S j Gini ginisplit(s, A) S j S j ginisplit(s, A) = j S j S gini(s) gain(s, A) = gini(s) ginisplit(s, A) GET /welcome.htm HTTP/1.0\r\n Connection: Keep-Alive\r\n User-Agent: Mozilla/4.08 [en] (WinNT; I)\r\n Host: Accept: image/gif, image/x-xbitmap, */*\r\n Accept-Encoding: gzip\r\n Accept-Language: en\r\n Accept-Charset: iso ,*,utf-8\r\n \r\n Data (15 bytes) ID (a) An Example of HTTP Request 08/12/04-20:24: Request-line Connection User-Agent Host Accept Accept-Encoding Accept-Language (b) An Example of Feature vector for HTTP Request 6 HTTP Fig. 6 HTTP Request. HTTP IDS Snort 3) Gini GUI Gtk HTTP 2 Table 2 System Components of Prototype. IDS Snort GUI HTTP Gini Gtk 5.2 HTTP IP TCP HTTP HTTP 5) 6(a) 6(b) ( 1 ) ID IP ( 2 ) Request-Line [bytes] ( 3 ) general-header, request-header,entity-header field-name field-value [bytes] fieldname 0 field-value field-name ( 4 ) message-body ASCII [bytes] ( 5 ) HTTP [bytes] 5.3 GUI 7

6 Vol. 0 No. 0 5 crashiis 8 mscan 39 4 Nessus HTTP Table 4 HTTP Request Generated by Nessus. 7 Fig. 7 Graphical User Interface WEB-MISC 696 WEB-CGI 444 GUI WEB-IIS 297 WEB-PHP 30 WEB-FRONTPAGE 16 WEB-COLDFUSION DARPA Nessus 4) DARPA Nessus 2-2 LAN 3 (i) (ii) (iii) 1999 DARPA IDS evaluation data Nessus 4) LAN 6.2 DARPA IDS Evaluation Data DARPA 4 5 HTTP DARPA DARPA HTTP Table 3 HTTP Request included in DARPA Data. apache back 167 phf 4 ps HTTP 7 apache2 back crashiis mscan ntinfoscan phf ps Nessus Nessus 4) Snort Snort LAN LAN 6.4 Snort HTTP HTTP Snort Nessus HTTP LAN LAN

7 LAN HTTP. Table 5 HTTP Request Captured at a Corporate LAN Gateway CodeRed WEB-IIS 19 Nimda WEB-IIS 32 WEB-CGI 81 WEB-MISC 17 WEB-PHP HTTP 149 Snort LAN 5 Nimda 32 CodeRed II 19 Snort DARPA 7 3 Nessus DARPA IDS Evaluation Data apache2 back ntinfoscan phf phf ntinfoscan 4 back, crashiis, mscan ps DARPA HTTP Host,Connect, User-Agent, Referer Accept 7 Nimda CodeRed II Table 7 Splitting Rules for Nimda and CodeRed II. CodeRed <Connnection 0.011, II 0.073<Content-type 33.0, <Request Content-Type t Nimda 0.011<Connnection Connection HTTP field-name 7.3 Nessus 2727 HTTP DARPA Nessus HTTP Nessus 701 Snort Nessus 7.4 LAN Nimda CodeRed II 7 8 Nimda field-name Content-Type Content-type CodeRed II Connection Connnection Nimda CodeRed 7.5 IDS

8 Vol. 0 No. 0 7 Table 6 6 Evaluation Summary. TP TN FP FN TP TN FP FN (i)darpa (apache2) (apache2) (back) (crashiis) (mscan) (ntinfoscan) (phf) (ps) (ii) Nessus (iii) LAN Gateway TP: True Positive, TN: True Negative, FP: False Positive, FN: False Negative 8 Nimda CodeRed II Snort Table 8 Snort Signatures for Nimda and CodeRed II. CodeRed II Nimda alert tcp any any -> any 80 (msg: WEB-IIS ISAPI.ida attempt ; flow:to server,established; uricontent:.ida? ; nocase; reference:arachnids, 552; classtype:web-application-attack; reference:bugtraq, 1065; reference:cve,can ; sid:1243; rev:8;) alert tcp any any -> any 80 (msg: WEB-IIS cmd.exe access ; flow:to server,established; content: cmd.exe ; nocase; classtype:webapplication-attack; sid:1002; rev:5;) 9 Nimda CodeRed II Table 9 A New Signature for Nimda and CodeRed II Variant. CodeRed II Nimda alert tcp any any -> any 80 (msg: CodeRed Variants ; flow:established;content: Contenttype ;) alert tcp any any -> any 80 (msg: Nimda Variants ; flow : established; content : Connnection ; ) 7 9 CodeRed Nimda FP FP 8. IDS IDS DARPA 7 3 Nessus Snort ) Barbara, D., Wu, N. and Jajodia, S.: Detecting Novel Network Intrusions Using Bayes Estimators, Proceedings of the 1st SIAM International Conference on Data Mining (SDM-01) (2001). 2) Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J.: Classification and Regression Trees, CRC Pr I Llc (1984). 3) Caswell, B. and Roesch, M.: Snort, The Open Source Network Intrusion Detection System, 4) Deraison, R.: Nessus, 5) Fielding, R., Irvine, U., Gettys, J., Mogul,

9 J., Compaq, Frystyk, H., Masinter, L., Xerox, Leach, P., Microsoft and Berners-Lee, T.: Hypertext Transfer Protocol HTTP/1.1, rfc ) Internet Security Systems, Inc.: realsecure, 7) Kreibich, C. and Crowcroft, J.: Honeycomb - Creating Intrusion Detection Signatures Using Honeypots, Proceedings of the 2nd Workshop on Hot Topics in Networks (HotNets-II) (2003). 8) Kruegel, C., Toth, T. and Kirda, E.: Service Specific Anomaly Detection for Network Intrusion Detection, Proceedings of the 2002 ACM symposium on Applied computing table of contents (2004). 9) Kruegel, C. and Vigna, G.: Anomaly Detection of Web-based Attacks, Proceedings of the 10th ACM conference on Computer and communication security (CCS 2003) (2003). 10) Lee, W. and Stolfo, S.J.: A framework for constructing features and models for intrusion detection systems, ACM Transactions on Information and System Security (TISSEC), Vol. 3, No. 4, pp (2000). 11) Lincoln Laboratory, Massachusetts Institute of Technology: LINCOLN LABORATORY, 12) Lippmann, R., Cunningham, R. K., Fried, D.J., Graf, I., Kendall, K.R., Webster, S.E. and Zissman, M. A.: Results of the DARPA 1998 Offline Intrusion Detection Evaluation, Recent Advances in Intrusion Detection (1999). 13) Lippmann, R., Haines, J. W., Fried, D. J., Korba, J. and Das, K.: The 1999 DARPA off-line intrusion detection evaluation. Computer Networks, Computer Networks: The International Journal of Computer and Telecommunications Networking, Vol. 34, No. 4, pp (2000). 14) Mahoney, M. V. and Chan, P. K.: Learning nonstationary models of normal network traffic for detecting novel attacks, Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2002). 15) Mahoney, M. V. and Chan, P. K.: An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection, Recent Advances in Intrusion Detection, 6th International Symposium, RAID 2003, Pittsburgh, PA, USA, September 8-10, 2003, Proceedings (2003). 16) NFR Security, Inc.: Network flight recorder, 17) Paxson, V.: Bro: a system for detecting network intruders in real-time, Computer Networks: The International Journal of Computer and Telecommunications Networking, Vol. 31, No , pp (1999). 18) Porras, P. A. and Neumann, P. G.: EMER- ALD: Event Monitoring Enabling Responses to Anomalous Live Disturbances, Proceedings of the 20th National Information Systems Security Conference (1997). 19) Provos, N.: Honeyd - A Virtual Honeypot Daemon, Proceedings of 10th DFN-CERT Workshop (2003). 20) Roesch, M.: Snort lightweight intrusion detection for networks, Proceedings of the 13th Conference on Systems Administration (LISA- 99) (1999). 21) Staniford, S., Hoagland, J. A. and McAlerney, J. M.: Practical Automated Detection of Stealthy Portscans, Journal of Computer Security, Vol. 10, No. 1/2, pp (2002). A.1 DARPA LINCOLIN Web 11) Snort ( 10) DARPA IP 7 DARPA 11 8 DARPA apache2

10 Vol. 0 No. 0 9 Attack 10 DARPA Snort Table 10 Snort Signatures for DARPA Data. Signature apache2 alert tcp any any -> any 80 (msg: apache2 ; flow:established; content: f f 31 2e 31 0d 0a d e 74 3a f d 0a d e 74 3a f d 0a ; ) apache2 alert tcp any any -> any 80 (msg: apache2 ; flow:established; content: f f 62 0a a a a a c a a 0d 0a d e 74 3a f d 0a d e 74 3a f d 0a ; ) back alert tcp any any -> any 80 (msg: back ; flow:established; content: GET //// ; ) back alert tcp any any -> any 80 (msg: back ; flow:established; content: GET /cgi-bin//// ; ) crashiis alert tcp any any -> any 80 (msg: crashiis ; flow:established; content:../.. ; ) mscan alert tcp any -> any 80 (msg: mscan ; flow:established; content: GET /cgi-bin/phf ; ) mscan alert tcp any -> any 80 (msg: mscan ; flow:established; content: GET /cgi-bin/test-cgi ; ) mscan alert tcp any -> any 80 (msg: mscan ; flow:established; content: GET /cgi-bin/handler ;) ntinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: HEAD / HTTP/1.0 ;) ntinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: GET /*.idc HTTP/1.0 ;) ntinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: GET /cgi-bin/ HTTP/1.0 ;) ntinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: GET /scripts/ HTTP/1.0 ;) ntinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: GET /cgi-bin/perl.exe?-v HTTP/1.0 ;) ntfinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: GET /scripts/perl.exe?-v HTTP/1.0 ;) ntfinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: GET /scripts /tools /newdsn.exe HTTP/1.0 ;) ntfinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: GET / vti bin/fpcount.exe? Page=default.htm ; content: Image=3 ; content: Digits=15 HTTP/1.0 ;) ntfinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: GET /scripts/*%0a.pl HTTP/1.0 ;) ntfinfoscan alert tcp any -> (msg: ntinfo ; flow:established; content: GET /samples/search/queryhit.htm HTTP/1.0 ;) phf alert tcp any any -> any 80 (msg: phf ; flow:established; content: phf? ; ) ps alert tcp any -> (msg: ps ; flow:established; content: tester.tar ; ) Attack apache2 back crashiis mscan ntinfoscan, phf 11 DARPA Table 11 Splitting Rules for DARPA Data. Splitting Rules <total<= , - <method<= <total<= , <method<= <total<= , - <data<=1.804, - <User-Agent<=23.030, - <Host<=7.013, - <method<= <total<= , 1.804<data<= <total<= , - <data<=1.804, - <User-Agent<=23.030, - <Host<=7.013, <method<= ps - <total<= , - <data<=1.804, <User- Agent<=47.000, 0.013<Connection<=10.000, - < method < = normal - <total<= , - <data<=1.804, - <User-Agent<=23.030, 7.013<Host<= normal - <total<= , - <data<=1.804, <User- Agent<=47.000, - <Connection<=0.013 normal - <total<= , - <data<=1.804, <User- Agent<=47.000, 0.013<Connection<=10.000, < method < =

11 class=back normal=0.141 ps=0.142 crashiis=0.142 phf=0.142 ntinfo=0.142 back=0.15 mscan=0.142 <data> -inf<data<= class=back normal=0.164 ps=0.165 crashiis=0.165 phf=0.165 ntinfo=0.165 back=0.175 <total> 1.827<data<= class=mscan mscan=1 <total> -inf<total<= class=ps normal=0.198 ps=0.201 crashiis=0.201 phf=0.201 ntinfo=0.2 <Host> <total<= class=back normal=0.006 back= inf<total<= class=mscan mscan= <total<= normal normal=1 -inf<host<= class=crashiis normal=0.003 crashiis=0.332 phf=0.332 ntinfo=0.332 <Referer> 8.005<Host<= class=ps normal=0.494 ps=0.506 <Connection> -inf<referer<= class=crashiis crashiis=0.333 phf=0.333 ntinfo=0.333 <User-Agent> 0.291<Referer<= inf<connection<= <Connection<= class=normal normal=1 14. class=normal normal=1 15. class=ps normal=0.015 ps=0.985 <Frame> -inf<user-agent<= class=crashiis crashiis=0.333 phf=0.333 ntinfo=0.333 <method> <User-Agent<= class=normal normal=1 -inf<frame<= class=ps ps= <Frame<= class=ps normal=0.028 ps=0.972 <User-Agent> -inf<method<= class=crashiis crashiis= <method<= class=phf phf=0.5 ntinfo=0.5 <Frame> -inf<user-agent<= class=normal normal= <User-Agent<= ps normal=0.001 ps= inf<frame<= clss=phf phf=0.787 ntinfo= <Frame<= class=ntinfo ntinfo=1 8 Fig. 8 Decision Tree.

& Vol.5 No (Oct. 2015) TV 1,2,a) , Augmented TV TV AR Augmented Reality 3DCG TV Estimation of TV Screen Position and Ro

& Vol.5 No (Oct. 2015) TV 1,2,a) , Augmented TV TV AR Augmented Reality 3DCG TV Estimation of TV Screen Position and Ro TV 1,2,a) 1 2 2015 1 26, 2015 5 21 Augmented TV TV AR Augmented Reality 3DCG TV Estimation of TV Screen Position and Rotation Using Mobile Device Hiroyuki Kawakita 1,2,a) Toshio Nakagawa 1 Makoto Sato