1,a) 1,b) 1,c) 1,d) OpenFOAM OpenFOAM A Bottleneck and Cooperation with the Post Processes in Numerical Calculation of Transient Phenomena Taizo Kobayashi 1,a) Yoshiyuki Morie 1,b) Toshiya Takami 1,c) Mutsumi Aoyagi 1,d) Abstract: Research and development of exa-scale computing is advanced energetically. On the other hand, new bottlenecks appear in the post processing of large scale numerical simulations. A numerical simulation of transient phenomena takes more than 90% of time for the post processing. In this paper, the details of the bottlenecks are discussed. Then as a solution for the bottlenecks, cooperation between simulation and the post processing is proposed. An implementation of it for OpenFOAM is also reported. Keywords: Coupled simulation, Cooperated calculation, Transient phenomena, Numrical simulation, Post processing, OpenFOAM 1. CPU [1] 1 Research Institute for Information Technology, Kyushu University a) tkoba@cc.kyushu-u.ac.jp b) morie.yoshiyuki.404@m.kyushu-u.ac.jp c) takami@cc.kyushu-u.ac.jp d) aoyagi@cc.kyushu-u.ac.jp FrontFlow[2], [3] OpenFOAM[4] OpenFOAM 1000 c 2013 Information Processing Society of Japan 1
1 Yes No 1 HPC 1 (JHPCN) [5] JHPCN 2012 1 35 10 12-NA09 1 JHPCN 12-DA03 12-MD02 12-MD03 12-MD04 12-MD05 12-MD06 12-NA05 12-NA10 12-NA11 GPGPU 12-NA16 12-MD06 1 1 HPC JHPCN 2 2. [6] [7] c 2013 Information Processing Society of Japan 2
(a) (b) 2 (a) (b) 3 (a) (b) 2 Points Cells Faces 2D 158,762 78,492 314,856 3D 4,048,431 3,924,600 11,896,692 3 Version Fujitsu PRIMERGY CX400 Xeon E5-2680 2.70GHz OpenFOAM 2.2.2 Intel Compiler XE, Intel 64 12.1.5 MPI IMPI 4.0.3.008 InfiniBand 2.1 CFD OpenFOAM LES rhopisofoam 2 2 3 3 Lighthill Howe 2.2 Open- FOAM MPI Super Computer PC Cluster for Post Processing Local PC 1TB 1TB / 300GB Local Lab. Resources Yes No Strage I/O 4 OpenFOAM PC I/O (refeq:1) = / (1) 17 864 5 10 5 200 36,720,000 gzip 200 kb 5 TB ( 1 ) : 10 ( a ) : blockmesh ( b ) : decomposepar ( 2 ) : rhopisofoam, 20 ( 3 ) c 2013 Information Processing Society of Japan 3
Super Computer PC Cluster Local PC 0 1 2 3 4 5 6 7 8 9 10 11 day 6 Super Computer PC Cluster for Post Processing Local PC 1TB 1TB 1GB 300GB Local Lab. Resources 30MB/sec Yes No Strage I/O Network Stream OpenFOAM PC I/O 5 OpenFOAM 20 3 MPI tar ftp I/O 0 1 Super Computer PC Cluster Local PC 2 Next Calc Cycle. ( a ) tar : 2 ( b ) ftp tar : 2 ( c ) tar : 2 ( d ) reconstructpar, 2 ( e ) foamtovtk, 2 ( f ) vorticity, Lighthill, Howe, sample, 2 4 5 TB 300GB 100MB 5 20 2 3 2 9 10 2 3 day 7 OpenFOAM 20 10 6 10 10 2 3 I/O 3. c 2013 Information Processing Society of Japan 4
I/O I/O I/O [8][9] 4 6 I/O tar ftp 7 3.1 OpenFOAM OpenFOAM OpenFOAM C++ ( 1 ) ( 2 ) ( 3 ) ρ + (ρv) = 0 (2) t OpenFOAM solve ( ); fvm::ddt(rho) + fvm::div(rhoflux, rho) fvsolution fvschemes fvsolution solvers rho solver preconditioner tolerance PBiCG; DILU; 1e-08; reltol 0.2; fvschemes ddtschemes default divschemes default Euler; none; div(rhoflux,rho) Gauss upwind; OpenFOMA-2.2.2 scotch[10] MPI MPI send, Isend, receive, Ireceive, allreduce OpneFOAM OpenCAE [11] OpenFOAM HPC C++ 3.2 OpenFOAM 6 LES rhopisofoam MPI reconstructpar OpenFOAM C++ OpenFOAM 8 I/O c 2013 Information Processing Society of Japan 5
8 OpenFOAM INstream ONstream I/O stream I/O IFstream, OFstream I/O 8 Nstream Nstream IOstream OpenFOAM C++ ACE[12] 4. JHPCN OpenFOAM C++ OpenFOAM HPC OpenFOAM Fstream Nstream OpenFOAM OpenFOAM OpenFOAM [1] Vol.53 No.8 [2] GUO, Yang, Chisachi KATO, and Yoshinobu YAMADE. Basic Features of the Fluid Dynamics Simulation Software FrontFlow/Blue. 58.1 (2006): 11-15. [3] FrontFlow: http://www.ciss.iis.u-tokyo.ac.jp/ rss21/theme/multi/fluid/fluid softwareinfo.html [4] OpenFOAM: Open Field Operation And Manipulation: The Open Source CFD Toolbox, http://www.openfoam.com [5] JHPCN :, http://jhpcnkyoten.itc.u-tokyo.ac.jp/sympo/ [6] T. Iwasaki, T. Kobayashi, K. Takahashi, T. Takami, A. Nishida, M. Aoyagi: Numerical study on the function of tone holes of a recorder like instrument from the viewpoint of the aerodynamic sound theory, ICA2013/POMA, 19, 035024, 2013 [7] M. Miyamoto, Y. Ito, T. Iwasaki, T. Akamura, K. Takahashi, T. Takami, T. Kobayashi, A. Nishida, M. Aoyagi: Numerical study on acoustic oscillations of 2d and 3d flue organ pipe like instruments with compressible les, Acta Acustica united with Acustica, 99, 154171, 2013 [8] c 2013 Information Processing Society of Japan 6
. 2011-HPC-130(20),(SWoPP2011) pp.1-7, 2011-07-20 [9]. 2011-HPC-132(4), (Hokke19) pp.1-6, 2011-11-21 [10] Scoch: Software package and libraries for sequential mesh and hypergraph partitioning, http://www.labri.fr/perso/pelegrin/scotch/ [11] OpenCAE : CAE http://www.opencae.jp [12] ACE: The ADAPTIVE Communication Environment http://www.dre.vanderbilt.edu/ schmidt/ace.html c 2013 Information Processing Society of Japan 7