9 8 7 (x-1.0)*(x-1.0) *(x-1.0) (a) f(a) (b) f(a) Figure 1: f(a) a =1.0 (1) a 1.0 f(1.0)

Transcription

1 1 ( ) CPU ( ) 2 1. a f(a) =(a 1.0) 2 (1) a ( ) 1(a) f(a) a (1) a f(a) a =2(a 1.0) (2) 2 0 a f(a) a =2(a 1.0) = 0 (3) 1

2 9 8 7 (x-1.0)*(x-1.0) *(x-1.0) (a) f(a) (b) f(a) Figure 1: f(a) a =1.0 (1) a 1.0 f(1.0) = ( ) 2 =0.0 f(a) 0 a = ( ) ( ) ( ) 1 a (k+1) = a (k) α f(a) a a=a (k) (4) a (k) k a f(a) a a=a (k) a = a (k) a α 1 α a f(a) a f(a) a =2(a 1.0) (5) a (k+1) = a (k) 2α(a (k) 1.0) (6) 1(b) f(a) a f(a) a 1.0 a a /* * Program to find the optimum value * which minimizes the function f(a) = (a - 1.0)^2 2

3 * using Steepest Decent Method */ #include <stdio.h> #include <stdlib.h> #include <math.h> double f(double a) { return((a-1.0)*(a-1.0)); double df(double a) { return(2.0*(a-1.0)); main() { double a; int i; double alpha = 0.1; /* Learning Rate */ /* set the initial value of a by random number within [-50.0:50.0] */ a = * (drand48() - 0.5); printf("value of a at Step 0 is %f, ", a); printf("value of f(a) is %f\n", f(a)); for (i = 1; i < 100; i++) { /* update theta by steepest decent method */ a = a - alpha * df(a); printf("value of a at Step %d is %f, ", i, a); printf("value of f(a) is %f\n", f(a)); f df a 100 alpha 0.1 Value of a at Step 0 is , Value of f(a) is Value of a at Step 1 is , Value of f(a) is Value of a at Step 2 is , Value of f(a) is Value of a at Step 3 is , Value of f(a) is Value of a at Step 4 is , Value of f(a) is Value of a at Step 5 is , Value of f(a) is Value of a at Step 6 is , Value of f(a) is Value of a at Step 7 is , Value of f(a) is Value of a at Step 8 is , Value of f(a) is Value of a at Step 9 is , Value of f(a) is Value of a at Step 10 is , Value of f(a) is Value of a at Step 11 is , Value of f(a) is Value of a at Step 12 is , Value of f(a) is Value of a at Step 13 is , Value of f(a) is Value of a at Step 14 is , Value of f(a) is Value of a at Step is , Value of f(a) is Value of a at Step 16 is , Value of f(a) is Value of a at Step 17 is , Value of f(a) is

4 Value of a at Step 18 is , Value of f(a) is Value of a at Step 19 is , Value of f(a) is Value of a at Step 20 is , Value of f(a) is Value of a at Step 21 is , Value of f(a) is Value of a at Step 22 is , Value of f(a) is Value of a at Step 23 is , Value of f(a) is Value of a at Step 24 is , Value of f(a) is Value of a at Step 25 is , Value of f(a) is Value of a at Step 26 is , Value of f(a) is Value of a at Step 27 is , Value of f(a) is Value of a at Step 28 is , Value of f(a) is Value of a at Step 29 is , Value of f(a) is Value of a at Step 30 is , Value of f(a) is Value of a at Step 31 is , Value of f(a) is Value of a at Step 32 is , Value of f(a) is Value of a at Step 33 is , Value of f(a) is Value of a at Step 34 is , Value of f(a) is Value of a at Step 35 is , Value of f(a) is Value of a at Step 36 is , Value of f(a) is Value of a at Step 37 is , Value of f(a) is Value of a at Step 38 is , Value of f(a) is Value of a at Step 39 is , Value of f(a) is Value of a at Step 40 is , Value of f(a) is Value of a at Step 41 is , Value of f(a) is Value of a at Step 42 is , Value of f(a) is Value of a at Step 43 is , Value of f(a) is Value of a at Step 44 is , Value of f(a) is Value of a at Step 45 is , Value of f(a) is Value of a at Step 46 is , Value of f(a) is Value of a at Step 47 is , Value of f(a) is Value of a at Step 48 is , Value of f(a) is Value of a at Step 49 is , Value of f(a) is Value of a at Step 50 is , Value of f(a) is Value of a at Step 51 is , Value of f(a) is Value of a at Step 52 is , Value of f(a) is Value of a at Step 53 is , Value of f(a) is Value of a at Step 54 is , Value of f(a) is Value of a at Step 55 is , Value of f(a) is Value of a at Step 56 is , Value of f(a) is Value of a at Step 57 is , Value of f(a) is Value of a at Step 58 is , Value of f(a) is Value of a at Step 59 is , Value of f(a) is Value of a at Step 60 is , Value of f(a) is Value of a at Step 61 is , Value of f(a) is Value of a at Step 62 is , Value of f(a) is Value of a at Step 63 is , Value of f(a) is Value of a at Step 64 is , Value of f(a) is Value of a at Step 65 is , Value of f(a) is Value of a at Step 66 is , Value of f(a) is Value of a at Step 67 is , Value of f(a) is Value of a at Step 68 is , Value of f(a) is Value of a at Step 69 is , Value of f(a) is Value of a at Step 70 is , Value of f(a) is Value of a at Step 71 is , Value of f(a) is Value of a at Step 72 is , Value of f(a) is

5 Value of a at Step 73 is , Value of f(a) is Value of a at Step 74 is , Value of f(a) is Value of a at Step 75 is , Value of f(a) is Value of a at Step 76 is , Value of f(a) is Value of a at Step 77 is , Value of f(a) is Value of a at Step 78 is , Value of f(a) is Value of a at Step 79 is , Value of f(a) is Value of a at Step 80 is , Value of f(a) is Value of a at Step 81 is , Value of f(a) is Value of a at Step 82 is , Value of f(a) is Value of a at Step 83 is , Value of f(a) is Value of a at Step 84 is , Value of f(a) is Value of a at Step 85 is , Value of f(a) is Value of a at Step 86 is , Value of f(a) is Value of a at Step 87 is , Value of f(a) is Value of a at Step 88 is , Value of f(a) is Value of a at Step 89 is , Value of f(a) is Value of a at Step 90 is , Value of f(a) is Value of a at Step 91 is , Value of f(a) is Value of a at Step 92 is , Value of f(a) is Value of a at Step 93 is , Value of f(a) is Value of a at Step 94 is , Value of f(a) is Value of a at Step 95 is , Value of f(a) is Value of a at Step 96 is , Value of f(a) is Value of a at Step 97 is , Value of f(a) is Value of a at Step 98 is , Value of f(a) is Value of a at Step 99 is , Value of f(a) is a 1.0 f(a) a f(a) =(a 1.0) 2 (a +1.0) 2 (7) f(a) a f(a) a =4.0a(a 1.0)(a +1.0) (8) 2(a) (b) 3 2 ( ) ( ) 5

6 (x-1.0)*(x-1.0)*(x+1.0)*(x+1.0) * x * (x-1.0)*(x+1.0) (a) f(a) (b) f(a) Figure 2: f(a) Table 1: (t) (x1) (x2) (x3) (m) (kg) (cm) (kg) (t) (x1) (x2) (x3) (x1) (x2) (x3) (t) y(x1,x2,x3) = a 0 + a 1 x1+a 2 x2+a 3 x3 (9 ) (t) (x1) (x2) (x3) {< t l,x1 l,x2 l,x3 l > l =1,..., l y(x1 l,x2 l,x3 l )=a 0 + a 1 x1 l + a 2 x2 l + a 3 x3 l (10) a 0,a 1,a 2 a <x1 l,x2 l,x3 l > t l y l (t l y l ) 2 2 ε 2 (a 0,a 1,a 2,a 3 ) ε 2 (a 0,a 1,a 2,a 3 ) = 1 ε 2 l = 1 6 (t l y l ) 2

7 = 1 {t l (a 0 + a 1 x1 l + a 2 x2 l + a 3 x3 l ) 2 (11) ε 2 (a 0,a 1,a 2,a 3 ) 2 (11) a 0 ε 2 a 0 = 2{( 1 t l ) a 0 ( 1 1) a 1 ( 1 x1 l ) a 2 ( 1 x2 l ) a 3 ( 1 x3 l ) = 2{ t a 0 a 1 x1 a 2 x2 a 3 x3 (12) 0 ε2 a 0 =0 a 0 = t a 1 x1 a 2 x2 a 3 x3 (13) t, x1, x2 x3 t, x1, x2 x3 t = 1 x1 = 1 x2 = 1 x3 = 1 t l (10) x1 l x2 l x3 l (14) y(x1 l,x2 l,x3 l )= t + a 1 (x1 l x1) + a 2 (x2 l x2) + a 3 (x3 l x3) () a 1, a 2 a 3 ε 2 (a 1,a 2,a 3 ) = 1 = 1 (t l y l ) 2 {t l t a 1 (x1 l x1) a 2 (x2 l x2) a 3 (x3 l x3) 2 (16) a 1 ε 2 = 1 {t l t a 1 (x1 l x1) a 2 (x2 l x2) a 3 (x3 l x3){x1 l x1 a 1 = {σ t1 a 1 σ 11 a 2 σ 21 a 3 σ 31 (17) a 2 a 3 ε 2 a 2 = {σ t2 a 1 σ 12 a 2 σ 22 a 3 σ 32 (18) ε 2 a 3 = {σ t3 a 1 σ 13 a 2 σ 23 a 3 σ 33 (19) 7

8 σ 11 = 1 σ 12 = 1 σ 13 = 1 σ 21 = 1 σ 22 = 1 σ 23 = 1 σ 31 = 1 σ 32 = 1 σ 33 = 1 σ t1 = 1 σ t2 = 1 σ t3 = 1 (x1 l x1)(x1 l x1), (x1 l x1)(x2 l x2), (x1 l x1)(x3 l x3), (x2 l x2)(x1 l x1), (x2 l x2)(x2 l x2), (x2 l x2)(x3 l x3), (x3 l x3)(x1 l x1), (x3 l x3)(x2 l x2), (x3 l x3)(x3 l x3), (t l t)(x1 l x1), (t l t)(x2 l x2), (t l t)(x3 l x3) (20) σ 12 = σ 21, σ 13 = σ 31, σ 23 = σ 32 (21) 0 0 a 1 σ 11 + a 2 σ 12 + a 3 σ 13 = σ t1 a 1 σ 21 + a 2 σ 22 + a 3 σ 23 = σ t2 a 1 σ 31 + a 2 σ 32 + a 3 σ 33 = σ t3 (22) Σa = σ (23) Σ a, σ σ 11 σ 12 σ 13 a 1 σ t1 Σ= σ 21 σ 22 σ 23, a = a 2, σ = σ t2 σ 31 σ 32 σ 33 a 3 σ t3 (24) 8

9 Σ Σ Σ 1 Σ 1 a =Σ 1 σ (25) Σ 3 3 Σ 1 = 1 σ 22 σ 33 σ23 2 σ 12 σ 33 + σ 13 σ 23 ( σ 12 σ 23 + σ 13 σ 22 ) σ 12 σ 33 + σ 13 σ 23 σ 11 σ 33 σ13 2 (σ 11 σ 23 σ 12 σ 13 ) (26) Σ ( σ 12 σ 23 + σ 13 σ 22 ) (σ 11 σ 23 σ 12 σ 13 ) σ 11 σ 22 σ12 2 Σ Σ Σ = σ 11 σ 22 σ 33 σ 11 σ23 2 σ12σ 2 33 σ13 2 σ 22 +2σ 12 σ 13 σ 23 Σ 0 a 0 = a 1 = a 2 = a 3 = (27) (x1 = 30) (x2 = 165) (x3 = 55) y = x x x55 = (28) Σ ( ) (11) a 0 ε 2 a 0 = 2 1 {ε l ε l a 0 = 2 1 ε l = 2 1 a 1, a 2 a 3 ε 2 a 1 = 2 1 ε 2 a 2 = 2 1 ε 2 a 3 = 2 1 {ε l ε l a 1 = 2 1 {ε l ε l a 2 = 2 1 {ε l ε l a 3 = 2 1 ε l x1 l = 2 1 ε l x2 l = 2 1 ε l x3 l = 2 1 (t l y(x1 l,x2 l,x3 l )) (29) (t l y(x1 l,x2 l,x3 l ))x1 l (t l y(x1 l,x2 l,x3 l ))x2 l (t l y(x1 l,x2 l,x3 l ))x3 l (30) 9

10 a (k+1) 0 = a (k) 0 α ε2 a a0=a = a (k) (k) 0 +2α a (k+1) 1 = a (k) 1 α ε2 a a1=a = a (k) (k) 1 +2α a (k+1) 2 = a (k) 2 α ε2 a a2=a = a (k) (k) 2 +2α a (k+1) 3 = a (k) 3 α ε2 a a3=a = a (k) (k) 3 +2α (t l y(x1 l,x2 l,x3 l )) (t l y(x1 l,x2 l,x3 l ))x1 l (t l y(x1 l,x2 l,x3 l ))x2 l (t l y(x1 l,x2 l,x3 l ))x3 l (31) (x1) (x2) (x3) (t) (x1,x2,x3) 100 x1 = x1 100, x2 = x2 100, x3 = x3 100 (32) #include <stdio.h> #define NSAMPLE #define XDIM 3 main() { FILE *fp; double t[nsample]; double x[nsample][xdim]; double a[xdim+1]; int i, j, l; double y, err, mse; double derivatives[xdim+1]; double alpha = 0.2; /* Learning Rate */ /* Open Data File */ if ((fp = fopen("ball.dat","r")) == NULL) { fprintf(stderr,"file Open Fail\n"); exit(1); /* Read Data */ /* Teacher Signal (Ball) */ fscanf(fp,"%lf", &(t[l])); /* Input input vectors */ for (j = 0; j < XDIM; j++) { fscanf(fp,"%lf",&(x[l][j])); 10

11 /* Close Data File */ fclose(fp); /* Print the data */ printf("%3d : %8.2f ", l, t[l]); for (j = 0; j < XDIM; j++) { printf("%8.2f ", x[l][j]); printf("\n"); /* scaling the data */ /* t[l] = t[l] / tmean;*/ for (j = 0; j < XDIM; j++) { x[l][j] = x[l][j] / 100.0; /* Initialize the parameters by random number */ for (j = 0; j < XDIM+1; j++) { a[j] = (drand48() - 0.5); /* Open output file */ fp = fopen("mse.out","w"); /* Learning the parameters */ for (i = 1; i < 20000; i++) { /* Learning Loop */ /* Compute derivatives */ /* Initialize derivatives */ for (j = 0; j < XDIM+1; j++) { derivatives[j] = 0.0; /* update derivatives */ /* prediction */ y = a[0]; for (j = 1; j < XDIM+1; j++) { y += a[j] * x[l][j-1]; /* error */ err = t[l] - y; /* printf("err[%d] = %f\n", l, err);*/ /* update derivatives */ derivatives[0] += err; for (j = 1; j < XDIM+1; j++) { derivatives[j] += err * x[l][j-1]; 11

12 for (j = 0; j < XDIM+1; j++) { derivatives[j] = -2.0 * derivatives[j] / (double)nsample; /* update parameters */ for (j = 0; j < XDIM+1; j++) { a[j] = a[j] - alpha * derivatives[j]; /* Compute Mean Squared Error */ mse = 0.0; /* prediction */ y = a[0]; for (j = 1; j < XDIM+1; j++) { y += a[j] * x[l][j-1]; /* error */ err = t[l] - y; mse += err * err; mse = mse / (double)nsample; printf("%d : Mean Squared Error is %f\n", i, mse); fprintf(fp, "%f\n", mse); fclose(fp); /* Print Estmated Parameters */ for (j = 0; j < XDIM+1; j++) { printf("a[%d]=%f, ",j, a[j]); printf("\n"); /* Prediction and Errors */ /* prediction */ y = a[0]; for (j = 1; j < XDIM+1; j++) { y += a[j] * x[l][j-1]; /* error */ err = t[l] - y; printf("%3d : t = %f, y = %f (err = %f)\n", l, t[l], y, err); 12

13 a[0]= , a[1]= , a[2]= , a[3]= , 0 : t = , y = (err = ) 1 : t = , y = (err = ) 2 : t = , y = (err = ) 3 : t = , y = (err = ) 4 : t = , y = (err = ) 5 : t = , y = (err = ) 6 : t = , y = (err = ) 7 : t = , y = (err = ) 8 : t = , y = (err = ) 9 : t = , y = (err = ) 10 : t = , y = (err = ) 11 : t = , y = (err = ) 12 : t = , y = (err = ) 13 : t = , y = (err = ) 14 : t = , y = (err = ) Threshold Linear Logistic (a) (b) (c) Figure 3: 1943 McCulloch Pitts M (±1) <x 1,x 2,...,x M > y 13

14 ( ) M y = U( a i x i + a 0 ) (33) U(η) i=1 U(η) = { 1, if η>0 1, if η 0 (34) 3(a) McCulloch Pitts 1949 Hebb ( ) Hebb x 1 x 2 x 3 a 1 a 2 a 3 f z x 4 a 4 Figure 4: 1957 Rosenblatt 4 ( ) Rosenblatt 5 ADALINE 1960 Widrow Hoff ADALINE(Adaptive Linear Neuron) M y = a i x i + a 0 (35) i=1 (a 0,a 1,...,a M ) McCulloch Pitts Rosenblatt 3(b) ADALINE (x1) (x2) (x3) (x4) 14

15 50 Fisher 1936 (t =1) (t =0) (x1) (x2) (x3) (x4) ADALINE y(x1,x2,x3,x4) = a 0 + a 1 x1+a 2 x2+a 3 x3+a 4 x4 (36) ADALINE (a 0,a 1,a 2,a 3,a 4 ) a (k+1) 0 = a (k) 0 +2α (t l y l ) 100 a (k+1) 1 = a (k) 1 +2α (t l y l )x1 l 100 a (k+1) 2 = a (k) 2 +2α (t l y l )x2 l 100 a (k+1) 3 = a (k) 3 +2α (t l y l )x3 l 100 a (k+1) 4 = a (k) 4 +2α (t l y l )x4 l (37) 100 t l y l l ADALINE x1 l x2 l x3 l x4 l l ADALINE ADALINE 1 0 #include <stdio.h> #include <stdlib.h> #define frand() rand()/((double)rand_max) #define NSAMPLE 100 #define XDIM 4 main() { FILE *fp; double t[nsample]; double x[nsample][xdim]; double a[xdim+1]; int i, j, l; double y, err, mse; double derivatives[xdim+1]; double alpha = 0.1; /* Learning Rate */ /* Open Data File */ if ((fp = fopen("niris.dat","r")) == NULL) { fprintf(stderr,"file Open Fail\n"); exit(1);

16 /* Read Data */ /* Input input vectors */ for (j = 0; j < XDIM; j++) { fscanf(fp,"%lf",&(x[l][j])); /* Set teacher signal */ if (l < 50) t[l] = 1.0; else t[l] = 0.0; /* Close Data File */ fclose(fp); /* Print the data */ printf("%3d : %8.2f ", l, t[l]); for (j = 0; j < XDIM; j++) { printf("%8.2f ", x[l][j]); printf("\n"); /* Initialize the parameters by random number */ for (j = 0; j < XDIM+1; j++) { a[j] = (frand() - 0.5); /* Open output file */ fp = fopen("mse.out","w"); /* Learning the parameters */ for (i = 1; i < 1000; i++) { /* Learning Loop */ /* Compute derivatives */ /* Initialize derivatives */ for (j = 0; j < XDIM+1; j++) { derivatives[j] = 0.0; /* update derivatives */ /* prediction */ y = a[0]; for (j = 1; j < XDIM+1; j++) { y += a[j] * x[l][j-1]; /* error */ err = t[l] - y; /* printf("err[%d] = %f\n", l, err);*/ /* update derivatives */ derivatives[0] += err; for (j = 1; j < XDIM+1; j++) { derivatives[j] += err * x[l][j-1]; 16

17 for (j = 0; j < XDIM+1; j++) { derivatives[j] = -2.0 * derivatives[j] / (double)nsample; /* update parameters */ for (j = 0; j < XDIM+1; j++) { a[j] = a[j] - alpha * derivatives[j]; /* Compute Mean Squared Error */ mse = 0.0; /* prediction */ y = a[0]; for (j = 1; j < XDIM+1; j++) { y += a[j] * x[l][j-1]; /* error */ err = t[l] - y; mse += err * err; mse = mse / (double)nsample; printf("%d : Mean Squared Error is %f\n", i, mse); fprintf(fp, "%f\n", mse); fclose(fp); /* Print Estmated Parameters */ printf("\nestimated Parameters\n"); for (j = 0; j < XDIM+1; j++) { printf("a[%d]=%f, ",j, a[j]); printf("\n\n"); /* Prediction and Errors */ /* prediction */ y = a[0]; for (j = 1; j < XDIM+1; j++) { y += a[j] * x[l][j-1]; /* error */ err = t[l] - y; if ((1.0 - y)*(1.0 - y) <= (0.0 - y)*(0.0 - y)) { if (l < 50) { printf("%3d [Class1 : correct] : t = %f, y = %f (err = %f)\n", l, t[l], y, err); 17

18 else { printf("%3d [Class1 : not correct] : t = %f, y = %f (err = %f)\n", l, t[l], y, err); else { if (l >= 50) { printf("%3d [Class2 : correct] : t = %f, y = %f (err = %f)\n", l, t[l], y, err); else { printf("%3d [Class2 : not correct] : t = %f, y = %f (err = %f)\n", l, t[l], y, err); 2 niris.dat Estimated Parameters a[0]= , a[1]= , a[2]=0.1394, a[3]= , a[4]= , 0 [Class1 : correct] : t = , y = (err = ) 1 [Class1 : correct] : t = , y = (err = ) 2 [Class1 : correct] : t = , y = (err = ) 3 [Class1 : correct] : t = , y = (err = ) 4 [Class1 : correct] : t = , y = (err = ) 5 [Class1 : correct] : t = , y = (err = ) 6 [Class1 : correct] : t = , y = (err = ) 7 [Class1 : correct] : t = , y = (err = ) 8 [Class1 : correct] : t = , y = (err = ) 9 [Class1 : correct] : t = , y = (err = ) 10 [Class1 : correct] : t = , y = (err = ) 11 [Class1 : correct] : t = , y = (err = ) 12 [Class1 : correct] : t = , y = (err = ) 13 [Class1 : correct] : t = , y = (err = ) 14 [Class1 : correct] : t = , y = (err = ) [Class1 : correct] : t = , y = (err = ) 16 [Class1 : correct] : t = , y = (err = ) 17 [Class1 : correct] : t = , y = (err = ) 18 [Class1 : correct] : t = , y = (err = ) 19 [Class1 : correct] : t = , y = (err = ) 20 [Class2 : not correct] : t = , y = (err = ) 21 [Class1 : correct] : t = , y = (err = ) 22 [Class1 : correct] : t = , y = (err = ) 23 [Class1 : correct] : t = , y = (err = ) 24 [Class1 : correct] : t = , y = (err = ) 25 [Class1 : correct] : t = , y = (err = ) 26 [Class1 : correct] : t = , y = (err = ) 27 [Class1 : correct] : t = , y = (err = ) 28 [Class1 : correct] : t = , y = (err = ) 29 [Class1 : correct] : t = , y = (err = ) 30 [Class1 : correct] : t = , y = (err = ) 18

19 31 [Class1 : correct] : t = , y = (err = ) 32 [Class1 : correct] : t = , y = (err = ) 33 [Class2 : not correct] : t = , y = (err = ) 34 [Class1 : correct] : t = , y = (err = ) 35 [Class1 : correct] : t = , y = (err = ) 36 [Class1 : correct] : t = , y = (err = ) 37 [Class1 : correct] : t = , y = (err = ) 38 [Class1 : correct] : t = , y = (err = ) 39 [Class1 : correct] : t = , y = (err = ) 40 [Class1 : correct] : t = , y = (err = ) 41 [Class1 : correct] : t = , y = (err = ) 42 [Class1 : correct] : t = , y = (err = ) 43 [Class1 : correct] : t = , y = (err = ) 44 [Class1 : correct] : t = , y = (err = ) 45 [Class1 : correct] : t = , y = (err = ) 46 [Class1 : correct] : t = , y = (err = ) 47 [Class1 : correct] : t = , y = (err = ) 48 [Class1 : correct] : t = , y = (err = ) 49 [Class1 : correct] : t = , y = (err = ) 50 [Class2 : correct] : t = , y = (err = ) 51 [Class2 : correct] : t = , y = (err = ) 52 [Class2 : correct] : t = , y = (err = ) 53 [Class2 : correct] : t = , y = (err = ) 54 [Class2 : correct] : t = , y = (err = ) 55 [Class2 : correct] : t = , y = (err = ) 56 [Class2 : correct] : t = , y = (err = ) 57 [Class2 : correct] : t = , y = (err = ) 58 [Class2 : correct] : t = , y = (err = ) 59 [Class2 : correct] : t = , y = (err = ) 60 [Class2 : correct] : t = , y = (err = ) 61 [Class2 : correct] : t = , y = (err = ) 62 [Class2 : correct] : t = , y = (err = ) 63 [Class2 : correct] : t = , y = (err = ) 64 [Class2 : correct] : t = , y = (err = ) 65 [Class2 : correct] : t = , y = (err = ) 66 [Class2 : correct] : t = , y = (err = ) 67 [Class2 : correct] : t = , y = (err = ) 68 [Class2 : correct] : t = , y = (err = ) 69 [Class2 : correct] : t = , y = (err = ) 70 [Class2 : correct] : t = , y = (err = ) 71 [Class2 : correct] : t = , y = (err = ) 72 [Class2 : correct] : t = , y = (err = ) 73 [Class2 : correct] : t = , y = (err = ) 74 [Class2 : correct] : t = , y = (err = ) 75 [Class2 : correct] : t = , y = (err = ) 76 [Class2 : correct] : t = , y = (err = ) 77 [Class2 : correct] : t = , y = (err = ) 78 [Class2 : correct] : t = , y = (err = ) 79 [Class2 : correct] : t = , y = (err = ) 80 [Class2 : correct] : t = , y = (err = ) 81 [Class2 : correct] : t = , y = (err = ) 82 [Class2 : correct] : t = , y = (err = ) 83 [Class1 : not correct] : t = , y = (err = ) 84 [Class2 : correct] : t = , y = (err = ) 85 [Class2 : correct] : t = , y = (err = ) 19

20 86 [Class2 : correct] : t = , y = (err = ) 87 [Class2 : correct] : t = , y = (err = ) 88 [Class2 : correct] : t = , y = (err = ) 89 [Class2 : correct] : t = , y = (err = ) 90 [Class2 : correct] : t = , y = (err = ) 91 [Class2 : correct] : t = , y = (err = ) 92 [Class2 : correct] : t = , y = (err = ) 93 [Class2 : correct] : t = , y = (err = ) 94 [Class2 : correct] : t = , y = (err = ) 95 [Class2 : correct] : t = , y = (err = ) 96 [Class2 : correct] : t = , y = (err = ) 97 [Class2 : correct] : t = , y = (err = ) 98 [Class2 : correct] : t = , y = (err = ) 99 [Class2 : correct] : t = , y = (err = ) ADALINE S(η) = exp(η) 1 + exp(η) (38) 3(c) M y = S( a i x i + a 0 ) (39) i=1 ADALINE 6.1 (39) ( ) 0 y 1 y 100 ( ) 100 L = (y l ) tl (1 y l ) 1 tl (40) log(l) = 100 {t l log y l +(1 t l ) log(1 y l ) 20

21 = = 100 {t l log{ exp(η l) 1 + exp(η l ) +(1 t 1 l) log{ 1 + exp(η l ) 100 {t l η l log{1 + exp(η l ) (41) a 0 log(l) 100 = {t l exp(η 100 l) a exp(η l ) = {t l y l (42) a 1 a 2 a 3 a 4 log(l) a 1 = log(l) a 2 = log(l) a 3 = log(l) a 4 = 100 {t l x1 l exp(η l) 1 + exp(η l ) x1 100 l = {(t l y l )x1 l 100 {t l x2 l exp(η l) 1 + exp(η l ) x2 100 l = {(t l y l )x2 l 100 {t l x3 l exp(η l) 1 + exp(η l ) x3 100 l = {(t l y l )x3 l 100 {t l x4 l exp(η l) 1 + exp(η l ) x4 100 l = {(t l y l )x4 l (43) 100 a (k+1) 0 = a (k) 0 + α (t l y l ) 100 a (k+1) 1 = a (k) 1 + α (t l y l )x1 l 100 a (k+1) 2 = a (k) 2 + α (t l y l )x2 l 100 a (k+1) 3 = a (k) 3 + α (t l y l )x3 l 100 a (k+1) 4 = a (k) 4 + α (t l y l )x4 l (44) ADALINE ADALINE #include <stdio.h> #include <stdlib.h> 21

22 #include <math.h> #define frand() rand()/((double)rand_max) #define NSAMPLE 100 #define XDIM 4 double logit(double eta) { return(exp(eta)/(1.0+exp(eta))); main() { FILE *fp; double t[nsample]; double x[nsample][xdim]; double a[xdim+1]; int i, j, l; double eta; double y, err, likelihood; double derivatives[xdim+1]; double alpha = 0.1; /* Learning Rate */ /* Open Data File */ if ((fp = fopen("niris.dat","r")) == NULL) { fprintf(stderr,"file Open Fail\n"); exit(1); /* Read Data */ /* Input input vectors */ for (j = 0; j < XDIM; j++) { fscanf(fp,"%lf",&(x[l][j])); /* Set teacher signal */ if (l < 50) t[l] = 1.0; else t[l] = 0.0; /* Close Data File */ fclose(fp); /* Print the data */ printf("%3d : %8.2f ", l, t[l]); for (j = 0; j < XDIM; j++) { printf("%8.2f ", x[l][j]); printf("\n"); /* Initialize the parameters by random number */ for (j = 0; j < XDIM+1; j++) { a[j] = (frand() - 0.5); 22

23 /* Open output file */ fp = fopen("likelihood.out","w"); /* Learning the parameters */ for (i = 1; i < 100; i++) { /* Learning Loop */ /* Compute derivatives */ /* Initialize derivatives */ for (j = 0; j < XDIM+1; j++) { derivatives[j] = 0.0; /* update derivatives */ /* prediction */ eta = a[0]; for (j = 1; j < XDIM+1; j++) { eta += a[j] * x[l][j-1]; y = logit(eta); /* error */ err = t[l] - y; /* update derivatives */ derivatives[0] += err; for (j = 1; j < XDIM+1; j++) { derivatives[j] += err * x[l][j-1]; /* update parameters */ for (j = 0; j < XDIM+1; j++) { a[j] = a[j] + alpha * derivatives[j]; /* Compute Log Likelihood */ likelihood = 0.0; /* prediction */ eta = a[0]; for (j = 1; j < XDIM+1; j++) { eta += a[j] * x[l][j-1]; y = logit(eta); likelihood += t[l] * log(y) + (1.0 - t[l]) * log(1.0 - y); printf("%d : Log Likeihood is %f\n", i, likelihood); fprintf(fp, "%f\n", likelihood); 23

24 fclose(fp); /* Print Estmated Parameters */ printf("\nestimated Parameters\n"); for (j = 0; j < XDIM+1; j++) { printf("a[%d]=%f, ",j, a[j]); printf("\n\n"); /* Prediction and Log Likelihood */ /* prediction */ eta = a[0]; for (j = 1; j < XDIM+1; j++) { eta += a[j] * x[l][j-1]; y = logit(eta); if ( y > 0.5) { if (l < 50) { printf("%3d [Class1 : correct] : t = %f, y = %f\n", l, t[l], y); else { printf("%3d [Class1 : not correct] : t = %f, y = %f\n", l, t[l], y); else { if (l >= 50) { printf("%3d [Class2 : correct] : t = %f, y = %f\n", l, t[l], y); else { printf("%3d [Class2 : not correct] : t = %f, y = %f\n", l, t[l], y); Estimated Parameters a[0]= , a[1]= , a[2]= , a[3]= , a[4]= , 0 [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y =

25 14 [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class2 : not correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class2 : not correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class1 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y =

26 69 [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class1 : not correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = [Class2 : correct] : t = , y = A B x y z Figure 5:

27 I x =(x 1,x 2,...,x I ) T K z =(z 1,...,z K ) T ζ j = I a ij x i + a 0j i=1 y j = S(ζ j ) J z k = b jk y j + b 0k (45) j=1 y j j a ij i j b jk j k N {x p, t p p =1,...,N ε 2 = 1 N N t p z p 2 = 1 N p=1 N ε 2 (p) (46) p=1 ε 2 ε 2 a ij = 1 N ε 2 b jk = 1 N N p=1 N p=1 ε 2 a ij = 1 N ε 2 (p) b jk = 1 N N 2γ pj ν pj x pi p=1 N 2δ pk y pj (47) p=1 ν pj = y pj (1 y pj ) K γ pj = δ pk b jk k=1 δ pk = t pk z pk (48) a 0j b 0k x p0 =1 y p0 =1 a ij a ij α ε2 a ij b jk b jk α ε2 b jk (49) 27

28 α δ b jk γ Quick Prop 28