大規模共有メモリーシステムでのGAMESSの利点

Size: px

Start display at page:

Download "大規模共有メモリーシステムでのGAMESSの利点"

やすもりなぐも
5 years ago
Views:

1 Technical white paper GAMESS GAMESS Gordon Group *1 Gaussian Gaussian1 Xeon E GAMESS 1 RHF ROHF UHF GVB MCSCF SCF Energy CDFpEP CDFpEP CDFpEP CD-pEP CDFpEP SCF Gradient CDFpEP CDFpEP CDFpEP CD-pEP CDFpEP SCF Hessian CD-p-- CD-p CD-p-- -D-p-- MP2 energy CDFpEP CDFpEP CD-pEP CD-pEP MP2 gradient CDFpEP -D-pEP CD-pEP CI energy CDFp-- CD-p CD-p-- CD-p-- CI gradient CD CC energy CDFpE- CDF-E EOMCC excitations CD E- CD E Semi-empirical: DFT energy CDFpEP CD-pEP CD-pEP DFT gradient CDFpEP CD-pEP CD-pEP TD-DFT energy CDFpEP CD-p-- TD-DFT gradient CDFpEP Mopac energy y Y y y Mopac gradient y Y y n 1. p parallel Xeon E7

2 GAMESS GAMESS 1 4 Replicated data Distributed data 2 $SYSTEM MWORD, MEMDDI Node#0 p=0 Node#1 p=2 X X Replicated data Replicated data Node#0 p=0 Replicated data Node#1 p=1 Replicated data Node#0 p=1 X Node#1 p=3 X Distributed data Distributed data Distributed data Distributed data 1. GAMESS ( 1 core Memory = MWORD + MEMDDI / ( ) MWORD, MEMDDI word 8x10^6 1 2 compute process CP data server ( DS) 16 8 p=2 p=0 p=1 p=2 p=0 DS MB GAMESS CP DS DS 3 1 (runtype=optimize) (runtype=hessian) FCM $VEC, $HES (runtype=vscf) cc-pvdz MP2/CCSD/ CCSD(T) 3 $CONTRL SCFTYP=RHF RUNTYP=VSCF CCTYP=CCSD(T) ISPHER=1 $END $SYSTEM TIMLIM= MWORDS=100 $END $BASIS GBASIS=CCD $END $VSCF NGRID=16 PETYP=DIRECT $END $GUESS GUESS=HUCKEL $END $DATA H2O CCSD(T)/cc-pVDZ Anharmonic Frequency CNV 2 O H $END Exp. MP2 CCSD CCSD(T) 1 ( ) ( ) ( ) (SATA DISK ) (RAM DISK: /dev/shm

MP2 IO HDD RAM-DISK RAM-DISK 2 IO 2 C540 (fullerene) C540 PC Gamess test2 *2 MOPAC PM3 SCFTYPE RHF PM3 1 PM3 GAMESS http://spec.

$scf soscf=.t. $end $DATA Case Elapsed Time(sec) R-PM3 ENERGY SCF iter. 1 (default) 175 2349.3718701794 15 2 (optimize) 178 2349.3718701796 16 3

3 MP2 IO HDD RAM-DISK RAM-DISK 2 IO 2 C540 (fullerene) C540 PC Gamess test2 *2 MOPAC PM3 SCFTYPE RHF PM3 1 PM3 GAMESS *3 3 -xhost CPU SSE AVX -ipo Nehelam TURBO $CONTRL SCFTYP=RHF RUNTYP=gradient nprint=-5 ISPHER=1 $END $SYSTEM TIMLIM=3000 MWORDS=100 $END $BASIS GBASIS=PM3 $END $GUESS GUESS=HUCKEL $END $scf soscf=.t. $end $DATA Case Elapsed Time(sec) R-PM3 ENERGY SCF iter. 1 (default) (optimize) (optimize + turbo) PM3 Default: -i8 -O2 Optimize: -i8 -xhost -ipo -O3 -no-prec-div -unroll2 -static -scalar-rep- 3 C20H16N4 (porphyrin) 4 $CONTRL RUNTYP=GRADIENT INTTYP=HONDO ICUT=10 CITYP=CIS NPRINT=-5 $END $SYSTEM TIMLIM=6000 MWORDS=100 $END $GUESS GUESS=HUCKEL $END $SCF DIRSCF=.T. $END $CIS NSTATE=1 CHFSLV=DIIS $END $DATA GAMESS MPI DDI DDI GAMESS VERSION = 1 MAY 2012 (R2) Intel-MPI Mathlib Intel MKL 13.0 Fortran Intel fortran 13.0 [composer_xe_ ]

4 PC GAMESS test6 7 DDI 1 16 CP DS Cores Elapsed Time(sec) FINAL RHF ENERGY SCF iter (1) RHF SCF DENSITY MATRIX < 2.00E ELapsed time[s] Elapsed time scale cores 2. (2) MPI GAMESS SCF MAXIT=30 30 UNCONVERGED MPI ERROR Infinibad-FDR Gigabit mpiexec Intel-MPI I_MPI_DEVICE= shm or rdssm or ssm Infiniband rdssm Gagabit ssm 5 SCF Cores SCF iterations infiniband SCF iterations Gigabit Elapsed Time Infiniband(rdssm)/Gigabit(ssm)

5 GAMESS GAMESS SL230Gen8 DL980G7 Processor Intel E Ghz Intel E Ghz Sockets, Core 2socket, 16core 8socket, 80core Memory 8GB DIMM x16 32GB DIMM x128 OS RedHat 6.3 RedHat 6.4 Interconnect InfiniBand FDR - 6. SMP CP MEMDDI MEMDDI distributed data MEMDDI DS CPU CP 2 CPU Intel-MPI I_MPI_WAIT_MODE (csh ) setenv I_MPI_WAIT_MODE enable SystemV ipcs -l sysctl -w kernel.shmmax= xxx kernel.shmmax = 68,719,476,736 (64GB) kernel.shmall = 4,294,967,296 (4GB) kernel.shmmni = 4,096 (4GB) kernel.shm_rmid_forced = 0 vm.hugetlb_shm_group = 0 DS DL980G7 Cores CP Elapsed Time RHF ENERGY ITERATIONS DS Cores CP Elapsed Time RHF ENERGY ITERATIONS DS DS % DS off %

6 40 35 jobs per day cores 3. DL980 DS 2 GAMESS with DS on SL230 GAMESS with DS on DL980 GAMESS without DS on DL980 MP2 CCSD Hatree-Fock HF MP2 CCSD SCF 2 MP2 2 CCSD 5-6 *4 CCSD MP CCSD, CCSD(T) Gradient 4 CCSD, CCSD(T) GAMESS nodereplicated distributed storage shmmax node-replicated process replicated distributed storage(memddi) DDI TCP-IP 9 EXETYP=CHECK CCSD CC MAXCC=1 30 MAXCC=30 HF 2000 CCSD AO MO AO MO MO CPU CP-0 CP-1 CP-2 CP-79 MWORDS t-ai MWORDS t-ai MWORDS t-ai MWORDS t-ai Gamess process node-replicated t-ij,ab fully distributed storage of the [VV 00], [VV 00], [V0 V0], [V0 00],[00 00] integrals The area of this entire big box is MEMDDI SysV shared memory segment More SysV shared memory segment DS-80 DS-81 DS-82 DS-159 Gamess process 4. CCSD,CCSD(T)

Replicated memory(mb) Distributed memory(mb) HF 36 0.3 MP2 89 4171 CCSD 240 332,152 CCSD(T) 1.2 75,752 9. EXETYP=CHECK 1000 jobs per day 100 10 1 0.1 HF MP2 DFT (b3lyp) CCSD CCSD (T) 0.

7 Replicated memory(mb) Distributed memory(mb) HF MP CCSD ,152 CCSD(T) , EXETYP=CHECK 1000 jobs per day HF MP2 DFT (b3lyp) CCSD CCSD (T) cores 5. CCSD MAXCC=30 1. Data Server 2. CPU Compute Process MP2 2 CCSD 4 CCSD 4. CCSD GPGPU libcchem *1) *2) *3) *4) Xeon E7 HP ProLiant DL980 G7 Xeon E GHz GB DIMM 128 OS Red Hat 6.4

HP High Performance Computing（HPC）

HP High Performance Computing（HPC） ACCELERATE HP High Performance Computing HPC HPC HPC HPC HPC 1000 HPHPC HPC HP HPC HPC HPC HP HPCHP HP HPC 1 HPC HP 2 HPC HPC HP ITIDC HP HPC 1HPC HPC No.1 HPC TOP500 2010 11 HP 159 32% HP HPCHP 2010 Q1-Q4