2002 avidemux MPEG-4 : 2003 2 5 : : G99P045-1
MPEG-4 MPEG-4 PC MPEG-4 MPI XviD MPEG-4 MPEG avidemux MPI MPEG-4 PE
1 1 1.1........................... 1 1.2................................ 1 2 2 2.1 MPEG-4............................ 2 2.2 MPEG-4............ 4 2.3.............. 4 2.4............ 5 3 6 3.1........................... 6 3.1.1 PE............................ 6 3.1.2 Myrinet............................ 6 3.2........................... 9 3.2.1 OS............................... 9 3.2.2 SCore 5.2........................... 9 4 avidemux 12 4.1........................... 12 4.2.............................. 12 4.3.................................. 14 4.4......................... 14 4.4.1............................. 14 4.4.2......................... 17 4.4.3... 18 4.4.4............................ 21 4.4.5..................... 22 4.4.6................ 24 5 avidemux 29 i
6 33 6.1............................ 33 6.2.............................. 33 6.2.1........ 33 6.2.2 2-Pass................... 34 6.2.3............... 34 36 ii
2.1...................... 3 2.2 MPEG-4 VOP.................. 5 3.1 Folon-III/Folon-IV............... 8 3.2 SCore................. 10 4.1.................. 15 4.2.................. 16 5.1 avidemux...................... 32 iii
2.1 MPEG......................... 2 3.1 FOLON-III.................... 7 3.2 FOLON-IV.................... 7 4.1 MPI.................. 13 5.1 DivX 640x360 35899frames to DivX CQ4.............. 30 5.2 Avi 320x240 900frames to DivX CQ4................ 31 iv
1 1.1 MPEG-4 MPEG-4 PC MPEG-4 PC Cluster Folon3 Folon4 MPEG-4 avidemux MPI(Message Passing Interface) RWCP(: Real World Computing Partnership) SCore 1.2 2 MPEG-4 avidemux 3 PC Cluster folon3 folon4 4 avidemux 5 avidemux 6 1
2 avidemux MPEG-4 MPE-4 2.1 MPEG-4 MPEG Moving Picture Expert Group ISO( ) MPEG MPEG 4 2.1 MPEG-4 MPEG-4 / MPEG-4 MPEG-1/2 MPEG-4 2.1: MPEG MPEG-1 384kbps 4Mbps VideoCD MPEG-2 1Mbps 100Mbps DVD MPEG-4 10Mbps MPEG-7 - EPG 2
I B B P I I P : 2.1: (AC/DC 8x8 ) ( ) ( ) MPEG-4 VO(Video Object) VO VOP(Video Object Plane) VOP 4 I-VOP( VOP) P-VOP( VOP) B-VOP( VOP) S-VOP( VOP) VOP I-VOP VOP P-VOP I P-VOP VOP B-BOP I P-VOP I P-VOP VOP S-VOP VOP 3
2.2 MPEG-4 API ( ) MPEG-4 XviD API 1. API 2. 2.3 2.3 PE PE 1 1PE ( ) 4
2.2: MPEG-4 VOP 2.4 MPEG-4 GOV(Group Of Video Object Plane) I,B,P,S-VOP MPEG-4 GOV 2-Pass ( 1-Pass 2-Pass ) I-VOP I-VOP MPEG-4 5
3 PC FOLON-III FOLON-IV 3.1 3.1.1 PE FOLON-III 9 CPU PC Myrinet2000 100BASE-TX Ethernet Pentium-III 1GHz 1152MB FOLON-IV 17 CPU PC FOLON-III Myrinet2000 100BASE- TX Ethernet Myrinet2000 FOLON-III serial FOLON-IV fiber CPU Pentium-III-S 1.26GHz 4096MB 16 2048MB FOLON-III 3.1.2 Myrinet Myrinet Myricom Gigabit LAN Ethernet Myrinet 1. Ethernet 6
3.1: FOLON-III CPU L2 chache main memory PE0 roquefort Pentium-III 1GHz 256kB 1152MB PE1 roquefort01 Pentium-III 1GHz 256kB 1152MB PE2 roquefort02 Pentium-III 1GHz 256kB 1152MB PE3 roquefort03 Pentium-III 1GHz 256kB 1152MB PE4 roquefort04 Pentium-III 1GHz 256kB 1152MB PE5 roquefort05 Pentium-III 1GHz 256kB 1152MB PE6 roquefort06 Pentium-III 1GHz 256kB 1152MB PE7 roquefort07 Pentium-III 1GHz 256kB 1152MB PE8 roquefort08 Pentium-III 1GHz 512kB 1152MB 3.2: FOLON-IV CPU L2 chache main memory comte Pentium-III 1.26GHz 512kB 4096MB PE0, PE1 comte00 Pentium-III 1.26GHz 512kB 2048MB PE2, PE3 comte01 Pentium-III 1.26GHz 512kB 2048MB PE4, PE5 comte02 Pentium-III 1.26GHz 512kB 2048MB PE6, PE7 comte03 Pentium-III 1.26GHz 512kB 2048MB PE8, PE9 comte04 Pentium-III 1.26GHz 512kB 2048MB PE10, PE11 comte05 Pentium-III 1.26GHz 512kB 2048MB PE12, PE13 comte06 Pentium-III 1.26GHz 512kB 2048MB PE14, PE15 comte07 Pentium-III 1.26GHz 512kB 2048MB PE16, PE17 comte08 Pentium-III 1.26GHz 512kB 2048MB PE18, PE19 comte09 Pentium-III 1.26GHz 512kB 2048MB PE20, PE21 comte10 Pentium-III 1.26GHz 512kB 2048MB PE22, PE23 comte11 Pentium-III 1.26GHz 512kB 2048MB PE24, PE25 comte12 Pentium-III 1.26GHz 512kB 2048MB PE26, PE27 comte13 Pentium-III 1.26GHz 512kB 2048MB PE28, PE29 comte14 Pentium-III 1.26GHz 512kB 2048MB PE30, PE31 comte15 Pentium-III 1.26GHz 512kB 2048MB 7
backborn network NIS Server, NFS Server, Router comte15 Ethernet Hub Myrinet Switch comte00 comte folon-iv Ethernet Hub roquefort07 Myrinet Switch roquefort folon-iii 3.1: Folon-III/Folon-IV 8
Myrinet 16 Myrinet Myrinet 2. NIC Myrinet NIC LANai LANai NIC NIC NIC NIC NIC NIC 4 3. Gigabit Ethernet 1Gbps Myrinet-2000 2Gbps 4Gbps 3.2 3.2.1 OS FOLON Redhat Linux7.3 SCore5.2 SCore 3.2.2 SCore 5.2 Score (RCWP) PC PC 3.2.2 SCore SCore 9
3.2: SCore PMv2 PMv2 PMv2 Myrinet Ethernet UDP Shmem SCore-D SCore-D Linux PM SCore-D SCore-D SCore-D MPICH-SCore MPICH-SCore MPI MPICH PMv2 MPI MPICH-SCore Version 1.0 Version 2.0 MPICH- 1.2.0 MPICH-1.2.1 MPICH-SCore Version 2.0 10
MPICH-SCore Version 1.0 Version 1.0 PVM-SCore PVM SCore PVM3.4 PVM PVM-SCore SCore PVM SCACH SCACH PMv2 PBS MRJ ( : ) NASA PBS(Protable Batch System) SCore MPC++ C++ C++ / Omni OpenMP OpenMP API RWCP Omni OpenMP SCASH OpenMP TACO C++ TACO 11
4 avidemux 4.1 PE PE GUI PE PE avidemux pre30 4.2 MPI MPI MPI C++ mpic++ mpic++ [ options ] filename... mpic++ MPICH-SCore MPI PATH 4.2 configure make./configure CC=mpicc CXX=mpic++ 12
mpicc mpic++ mpif77 mpif90 C MPI C++ MPI Fortran77 MPI Fortran90 4.1: MPI /usr/bin/ld: BFD 2.11.93.0.2 20020207 assertion fail elf-strtab.c:262 make SCore checkpointing SCore checkpointing static avidemux static configure./configure CC="mpicc -nostatic" CXX="mpic++ -nostatic" source= oss_out.cpp object= oss_out.o libtool=no \ depfile=.deps/oss_out.po tmpdepfile=.deps/oss_out.tpo \ depmode=gcc3 /bin/sh../../admin/depcomp \ mpic++ -DHAVE_CONFIG_H -I. -I. -I../.. -I/usr/include/kde/ar ( ) -fno-check-new -c -o oss_out.o test -f oss_out.cpp echo./ oss_out.cpp mv: cannot stat.deps/oss_out.tpo : ar cru libadm_audiodevice.a ADM_deviceArts.o ADM_audiodevice.o ADM_deviceoss.o oss_out.o ar: oss_out.o: gcc 2.96 dependency style gcc3 configure Makefile avidemux/adm audiodevice/ MPI configure avidemux/adm audiodevice/makefile 13
CC = gcc CPP = gcc -E CXX = g++ CXXXPP = g++ -E 4.3 avidemux GTK+ Gtk-WARNING **: cannot open display: localhost:10.0 ssh X Forwarding ssh ( :4.3) DISPLAY scout scout ( :4.3) 4.4 4.4.1 MPI C++ #include <mpi++.h> C #include <mpi.h> include MPI PE PE main.cpp myid( ),nprocs( PE ) 14
ssh login user s console cluster host DISPLAY=localhost:10.0 cluster host DISPLAY=localhost:10.0 "scout -g pcc" environment variables copied cluster host DISPLAY=localhost:10.0 X Forwarding user s console cluster host "scrun avidemux" DISPLAY=localhost:10.0 cluster host avidemux DISPLAY=localhost:10.0 cluster host avidemux DISPLAY=localhost:10.0 4.1: 15
ssh login user s console cluster host DISPLAY=user s console:10.0 "scout -g pcc" cluster host DISPLAY=user s console:10.0 environment variables cop cluster host DISPLAY=user s console:10.0 user s console cluster host "scrun avidemux" DISPLAY=user s console:10.0 cluster host avidemux DISPLAY=user s console:10.0 cluster host avidemux DISPLAY=use s console:10.0 4.2: 16
int myid,nprocs; int main(int argc, char *argv[]) { extern int myid; // extern int nprocs; // PE 4.4.2 MPI MPI Init MPI PE main MPI Init MPI Comm rank MPI Comm size int main(int argc, char *argv[]) { pthread_t tid; FCT_VOID *thread; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_COMM_WORLD,&myid); MPI_Comm_size(MPI_COMM_WORLD,&nprocs); dll GTK+ gtk init PE avidemux GTK+ atexit 17
atexit(onexit); gtk_set_locale(); gtk_init(&argc, &argv); gdk_rgb_init(); 4.4.3 PE gtk main PE PE children main PE gtk widget show PE0 window1 if(myid==0) { gtk_widget_show(window1); gtk_main(); } else { children_main(); } return 0; } PE GUI avidemux/gtk gui.cpp void HandleAction(Action action) (avidemux/gui action.hxx enum ) switch PE action MPI Isend PE GUI 18
void HandleAction(Action action) { uint32_t nf=0; mybcast_pe0(null, 0, MPI_CHAR, (int)action, MPI_COMM_WORLD); switch(action) { case ACT_LoadWork: GUI_FileSelRead("Select workbench to load ", A_loadWorkbench); updateloaded(); return; break; case ACT_VideoConfigure: printf("\n **\n"); videocodecconfigure(); return; break; PE PE switch gtk main children main HandleAction event switcher void children_main(void) { int eventtag; printf("node %d stand by ok.\n",myid); while(1){ eventtag = MPI_ANY_TAG; mybcast_children(null, 0, MPI_CHAR, &eventtag, MPI_COMM_WORLD); event_switcher(eventtag); } } 19
void event_switcher(int eventtag) { uint32_t nf=0; switch(eventtag) { case 444444: exit(0); case ACT_LoadWork: recvfile(eventtag); updateloaded(); return; case ACT_VideoConfigure: videocodecconfigure(); return; event switcher HandleAction case 444444 GUI FileSelRead recvfile mybcast pe0 mybcast children MPI MPI Isend, MPI Irecv PE0 void mybcast_pe0(void *buff, int count, MPI_Datatype datatype, int tag, MPI_Comm MPI_Request mpirequest; MPI_Status mpistatus; int dest = 1; while(1) { if(dest > nprocs -1) break; MPI_Isend(buff, count, datatype, dest, tag, comm, &mpirequest); MPI_Wait(&mpirequest, &mpistatus); dest = dest * 2; } } 20
void mybcast_children(void *buff, int buffsize, MPI_Datatype datatype, int *retu MPI_Request mpirequest; MPI_Status mpistatus; int tag, count, distance, dest; tag = *returntag; MPI_Irecv(buff, buffsize, datatype, MPI_ANY_SOURCE, tag, comm, &mpirequest); MPI_Wait(&mpirequest, &mpistatus); if(tag== MPI_ANY_TAG) { *returntag=mpistatus.mpi_tag; } distance = myid-mpistatus.mpi_source; MPI_Get_count(&mpistatus, datatype, &count); while(1) { distance = distance * 2; dest = myid + distance; if(dest > nprocs -1) break; MPI_Isend(buff, count, datatype, dest, *returntag, comm, &mpirequest); MPI_Wait(&mpirequest, &mpistatus); } } 4.4.4 atexit atexit onexit PE PE 444444 21
void onexit( void ) { MPI_Request mpirequest; MPI_Status mpistatus; if(myid==0) mybcast_pe0(null,0,mpi_char,444444,mpi_comm_world); filtercleanup(); printf("node %d exiting...\n",myid); MPI_Finalize(); } 444444 PE exit(0) onexit 4.4.5 PE PE PE PE PE PE PE PE PE avidemux case ACT_LoadWork: GUI_FileSelRead("Select workbench to load ", A_loadWorkbench); GUI FileSelRead GUI FileSelRead A loadworkbench GUI FileSelRead avidemux/adm tookkit/filesel.cpp start thread PE 22
void start_thread(void *ptr) { ( ) fd=fopen(selected_file,"rb"); if(!fd) { GUI_Alert("Cannot open this file!"); /* PE */ filesize = -1; mybcast_pe0(&filesize, 1, MPI_INT, 0, MPI_COMM_WORLD); return; } else { fseek(fd, 0, SEEK_END); filesize = ftell(fd); fseek(fd, 0, SEEK_SET); buff = (char *)malloc(filesize); fread(buff, filesize, 1, fd); // mybcast_pe0(&filesize, 1, MPI_INT, 0, MPI_COMM_WORLD); // mybcast_pe0(buff, filesize, MPI_CHAR, 1, MPI_COMM_WORLD); free(buff); } fclose(fd); _callback(selected_file); PE GUI FileSelRead recvfile recvfile PE switch GUI FileSelRead 23
void recvfile(int action) { ( ) mpitag = 0; // mybcast_children(&buffsize,1,mpi_int,&mpitag,mpi_comm_world); // if(buffsize < 0) return; buff = (char *)malloc(buffsize); mpitag = 1; // mybcast_children(buff,buffsize,mpi_char,&mpitag,mpi_comm_world); ( ) fd = fopen(tmpfilename, "wb"); if(!fd) { printf("node: %d can t make the tmpfile\n", myid); return; } if(!(fwrite(buff, buffsize, 1, fd))) { return; } free(buff); fclose(fd); switch(action) { case ACT_LoadWork: A_loadWorkbench(tmpfilename); break; ( ) } return; } 4.4.6 Save AVI PE PE 24
avidemux/gui savenew.cpp A SaveAudioNVideo PE 1PE = ( - ) / PE PE = + (1PE x ) PE = PE + 1PE - 1 PE PE PE PE PE PE PE PE if(myid==0)filegather_recvfiles(); else filegather_sendfile(); PE filegather sendfile 1. PE ( PE ) 2. PE 25
void filegather_recvfiles(void) { ( ) for(i = 1; i<nprocs; i++) { // MPI_Recv(&buffsize,1,MPI_INT,MPI_ANY_SOURCE,0,MPI_COMM_WORLD,&mpistatus); source = mpistatus.mpi_source; ( ) buff = (char *)malloc(buffsize); // MPI_Recv(buff,buffsize,MPI_CHAR,source,1,MPI_COMM_WORLD,&mpistatus); ofstream out(tmpsaveavifilename, ios::out ios::binary); out.write(buff,buffsize); out.close(); free(buff); } } PE filegather recvfiles 1. PE ( PE ) 2. PE 3. PE 1 26
void filegather_sendfile(void) { ( ) ifstream in(tmpsaveavifilename, ios::in ios::binary); in.seekg(0,ios::end); filesize = in.tellg(); in.seekg(0,ios::beg); buff = (char *)malloc(filesize); in.read(buff,filesize); MPI_Send(&filesize,1,MPI_INT,0,0,MPI_COMM_WORLD); // MPI_Send(buff,filesize,MPI_CHAR,0,1,MPI_COMM_WORLD); // free(buff); in.close(); } MPI Gather PE PE avidemux A openavi(avi ) A appendavi( AVI ) A SaveAudioNVideo( audioprocessmode videoprocessmode 0 AVI ) 27
A_openAvi_pe0(tmpsaveavifilename00); for(i=1;i<nprocs;i++) { tmpsaveavifilename[filename_length] = 0 + (int)(i / 10); tmpsaveavifilename[filename_length+1] = 0 +(i % 10); tmpsaveavifilename[filename_length+2] = \0 ; A_appendAvi_pe0(tmpsaveavifilename); } tmpsaveavifilename[filename_length] = \0 ; framestart = tmpframestart; frameend = tmpframeend; A_SaveAudioNVideo_pe0(realname); 28
5 avidemux avidemux 640x360, 35899 DivX 320x240, 900 AVI DivX CQ( ) 4 3 folon4 SMP 1 1PE 5.1 PE PE 1 / PE 5.1 sec total bcasting merging gathering PE0 enctime max PE enctime min PE enctime pe0 PE0 enctime avg PE waittime max PE PE0 PE DivX PE PE avidemux Avi PE 29
PE total bcasting merging gathering enctime max enctime min 1 1500 0 0 0 1500 1500 2 766 1.13 3.46 54.4 757 708 3 529 2.22 7.46 50.4 520 470 4 404 2.22 3.51 43.0 400 348 5 336 3.33 7.51 41.3 327 271 6 280 3.33 3.39 36.9 276 224 7 242 3.33 3.47 31.5 238 192 8 219 3.33 6.09 25.5 212 163 9 195 4.65 9.93 13.1 184 145 10 172 4.44 3.61 9.96 168 133 11 161 4.66 8.87 3.95 151 120 12 150 4.44 3.53 12.0 146 109 13 148 4.44 8.65 13.0 139 99.5 14 136 4.44 3.54 16.0 132 94.1 15 126 4.66 3.43 14.8 122 88.3 16 122 4.44 8.45 11.8 113 80.6 PE enctime pe0 enctime avg waittime max 1 1500 1500 0 2 708 732 1.27 3 471 487 1.87 4 358 368 10.4 5 287 293 16.8 6 240 243 15.8 7 207 210 15.2 8 187 183 24.3 9 171 164 27.2 10 159 148 25.9 11 148 134 27.7 12 134 123 25.4 13 127 113 27.2 14 116 106 22.3 15 108 98.0 19.7 16 101 92.0 20.9 5.1: DivX 640x360 35899frames to DivX CQ4 30
PE total bcasting merging gathering enctime max enctime min 1 18 0 0 0 18 18 2 9.11 1.20 0.234 0.0716 8.83 8.81 3 6.48 2.40 0.514 0.158 5.94 5.81 4 4.90 2.61 0.252 0.103 4.62 4.53 5 3.97 3.60 0.205 0.211 3.75 3.55 6 3.52 3.81 0.301 0.269 3.20 2.95 7 3.17 3.81 0.224 0.374 2.94 2.49 8 2.53 3.60 0.249 0.0534 2.26 2.13 9 2.36 5.00 0.215 0.151 2.13 1.88 10 2.12 4.79 0.257 0.0558 1.81 1.70 11 2.64 4.82 0.226 0.728 2.40 1.58 12 2.22 5.01 0.224 0.480 1.98 1.46 13 1.82 4.79 0.338 0.0582 1.43 1.31 14 1.72 4.79 0.280 0.146 1.43 1.21 15 2.06 4.79 0.310 0.472 1.75 1.22 16 2.62 4.79 0.276 1.19 2.29 1.05 PE enctime pe0 enctime avg waittime max 1 18 18 0 2 8.81 8.82 0.0292 3 5.81 5.87 0.0203 4 4.54 4.57 0.0225 5 3.55 3.64 0.0125 6 2.95 3.10 0.0105 7 2.58 2.65 0.0862 8 2.23 2.20 0.100 9 1.99 1.98 0.113 10 1.81 1.75 0.118 11 1.68 1.74 0.0966 12 1.51 1.69 0.0518 13 1.43 1.35 0.121 14 1.29 1.29 0.0791 15 1.28 1.45 0.0676 16 1.16 1.53 0.133 5.2: Avi 320x240 900frames to DivX CQ4 31
5.1: avidemux 32
6 6.1 PE PE MPEG-4 6.2 6.2.1 PE PE MPEG-1/2/4 GOP(GOV) 1/ PE I/O I/O MPEG AVI I/O mmap HDD 33
6.2.2 2-Pass 2-Pass PE 1-Pass 1-Pass PE 1-Pass 6.2.3 PE GUI GUI PE 34
Folon avidemux mean 35
[1] P., :MPI,,2001. [2],,,,, : Linux,, 2002. [3] PC Cluster Consortium, 2001, http://www.pccluster.org/ [4] : GIMP,, 1999. [5] : MPI GIMP core,, 2002. [6] avidemux, 2002, http://fixounet.free.fr/avidemux/ [7] MPEG Home Page, 2003, http://mpeg.telecomitalialab.com/ [8] ISO/IEC JTC1/SC29/WG11 N2564 Overview of the MPEG-4 Standard, 1998. [9] PIONEER R&D, 2002, http://www.pioneer.co.jp/crdl/tech/index.html 36