I/O VM CPU VMM CPU VMM SoC VMM CPU I/O I/O I/O Performance Evaluation for Hardware Translation of I/O Address Map in Embedded Virtualization Tomoaki Ukezono and Koichi Araki Recently, exploiting virtualization to design of embedded system is getting a lot more attention lately. Embedded virtualizations can achieve minimal modification for software and higher software reusability when the software is ported to new platforms. From the benefits, improvement in productivity and bugs that is caused by software modification can be attained. Furthermore, robust and secure system can also be established preventing that CPUs give privileged mode to migrated VM. On the other hand, VMM which is used to virtualization decreases effectiveness of execution in CPUs, since the VMM has overhead for emulation of instructions. For this reason, introducing virtualization to embedded systems that are attached weight to real time execution is prevented. Our objective is to establish an embedded virtualization system that can be benefited from VMMs and can be achieved more effective CPU execution implementing VMMs in hardware as a function of SoC. In this paper, in particular, we focus on the translation of I/O address map in embedded virtualization and propose a hardware I/O address map translator. In addition, we evaluate performance improvement by the hardware I/O address map translator. 1. 1.1 1967 IBM System360 40 CP40 1) Japan Advanced Institute of Science and Technology (JAIST) Center for Highly Dependable Embedded Systems Technology, JAIST IBM System360 OS OS(VM:Virtual Machine) OS VM 131
OS 2 1 Xen 2) KVM 3) (type1 ) 2 Virtual BOX 4) VMWare Workstation 5) (type2 ) OS VMM(Virtual Machine Monitor) 1.2 VMM VMM VM I/O VMM VMM 6)7) fault () VMM failure VM VMM VM failure VM CPU VMM CPU VM VMM 2000 VMM VMM L4 8) VMM PikeOS 9) OKL4 10) CODEZERO 12) NOVA 11) PikeOS VMM A350 A400M OKL4 VMM Android 10 CODEZERO ARM VMM NOVA VMM VMM OS VMM 13) WindRiver Hypervisor 14) RTS Hypervisor 15) VMWare MVP 16) VMM VMM VMM System-on-Chip(SoC) 132
SoC IC LSI LSI SoC VMM VMM VM CPU SoC RTL IP LSI SoC IP VMM VM VMM VMM VMM I/O I/O I/O I/O VMM I/O VMM I/O VM I/O I/O TLB VMM I/O 割込 / 例外 (1) 1 Process (2) VMへ遷移 VM 特権命令 (3) (5) 特権命令エミュレーション Hardware (4) VM VMM. VMM 2. I/O 2.1 VMM OS VMM 1 1 Intel VT-x 17) CPU VMM VM VM VM VMM ( 1 ) VMM ( 2 ) VMM VM ( 3 ) VMM ( 4 ) VMM VM ( 5 ) VM VM VMM (VMM ) OS OS VMM (1) (2) Intel VT-x VMM 133
トラップベースレジスタ TBR VM 選択と VM トラップベクトルの呼び出し 主記憶 VMM のトラップベクトル VMM の機能プログラム VM のトラップベクトル 特権命令実行 例外発生 例外原因 PC レジスタを使い例外原因命令をロード 例外原因命令のオペーコードから命令種を解析 VM の機能プログラム 例外原因アドレスレジスタの値から I/O アドレスを生成 2 VM VMM. 生成したアドレスへのデータ転送を例外原因命令で実行 2 VMM VMM VMM VM VM VMM VMM VM VMM VM 2.2 I/O VMM I/O I/O VMM TLB I/O 1 I/O I/O VM VMM I/O VM I/O VMM VM I/O I/O 3 3 2 (3) (4) VMM VM I/O MMU VMM I/O MMU Fault 3 例外原因 PC の次のアドレスへ復帰 I/O. I/O CPU Precise Exception PC MMU MMU I/O 2 PC I/O I/O I/O I/O I/O 3. I/O I/O MMU I/O 134
実行中 VM 番号 参照 I/O アドレス 4 VM 番号変換前アドレス I/O. 4 変換後アドレス オフセット 変換後 I/O アドレス TLB VM ID VM ID VMM VM ID VM VM1 VM2 I/O I/O VM I/O VMM SoC SoC VM I/O TLB SRAM SoC VM I/O I/O SoC VM SRAM I/O SRAM I/O 18 4 256 4 274 274 4 274 2 4. 4.1 Digilent Atlys TM Spartan- 6 FPGA 18) FPGA OpenRISC 1000 19) OR1200 20) ORPSoC 21) Spartan-6 FPGA DDR2 SDRAM 128MB 1Gbit ether phy HDMI AC-97 audio SPI Flash 16MB USB UART USB HID Host LED I/O I/O I/O VMM I/O OR1200 OpenCores Verilog HDL RTL FPGA ORPSoC I/O Verilog HDL RTL ORPSoC RTL Spartan-6 1 VM OS Linux OpenCores GNU 135
1 2 Xilinx xst14.4 Xilinx map14.4 Xilinx par14.4 Xilinx ChipScope Pro 14.4 OS:Linux FPGA Board : Digilent Atlys FPGA:Xilinx xc6slx45csg324c SoftCPU:OR 1200 SoC:ORPSoC NETGEAR Fast Ethernet Switch FS108 VMM 6767 6750 LUT 12625 12564 RAM16BWER 41 41 RAM8BWER 3 3 OS:CentOS6.4 CPU: Phenom 9500 2.2GHz Memory:4GB DSP481A1 4 4 Mather Board:ASUS M3A NIC: オンボード ギガビットイーサ 3 HW 100Mbps 100Mbps VMM (MHz) 42.18 41.051 HW nuttcp: サーバー nuttcp: クライアント 5 I/O. Linux 3.1 OpenRISC OR1200 I/O Linux I/O ORPSoC I/O VMM OpenRISC 2 I/O Linux I/O 3 5 nuttcp 22) nuttcp TCP UDP nuttcp 1 TCP 100 (Mbps) nuttcp TCP Read/Write 4.2 1 2 VMM I/O OR1200 HW 3 I/O 0.2% LUT 0.4% VMM HW 2 FPGA I/O 1 3 1.129MHz VMM HW 32 2 (32 ) VMM 136
通信速度向上率 1.16 4 VMM 1.14 1.12 1.1 (%) 9,801,763 121,946,838 8.03 1.08 1.06 1.04 1.02 1 アドレス変換部実行 (VMM) nuttcp 実行 linux 実行 (sleep 等 ) アドレス変換部の命令実行数 トータル命令実行数 t 7. 6. 4.3 I/O 6 I/O 4.1 nuttcp VMM 1 HW 1.1 I/O 10% Read/Write VMM I/O I/O VMM I/O 16KB 14% 32KB 6% 10.03% OR1200 VMM 1 nuttcp 1K 4 nuttcp 5 7 4 (t:time) 3 nuttcp linux 7 4 VMM I/O nuttcp exit nuttcp Linux sleep OS sleep nuttcp 8.03% 10.36% OR1200 1 1 sleep 5. VMM SoC 137
VMM I/O I/O I/O 14% 10% 3 VM SoC I/O SRAM VMM I/O VM VMM FPGA 1) B. Bitner and S. Greenlee, z/vm - A Brief review of Its 40 Year History, http://www.vm.ibm.com/vm40hist.pdf, IBM Corporation, 2012. 2) B. Clark, T. Deshane, E. Dow, S. Evanchik, M. Finlayson, J. Herne and J. N. Matthews, Xen and the Art of Repeated Research, Proc. of the annual conference on USENIX Annual Techinical Conference, pp.135-144, 2004. 3) A. Kivity, Y. Kamay and D. Laor, KVM: The Linux Virtual Machine Monitor, Proc. of the Linux Symposium, pp.225-230, 2007. 4) H. Davis, M. B. Skov, M. Stougaard and F. Vetere, Virtual box: supporting mediated family intimacy through virtual and physical play, Proc. of the 19th Australasian Conference on Computer-Human Interaction: Entertaining User Interfaces, pp. 151-159, 2007. 5) Getting Started with VMware Workstation, in http://www.vmware.com/pdf/desktop/ws90- getting-started.pdf, VMWare, Inc, 2012. 6) C. Clark, K. Fraser, S. Hand, J.G. Hansen,E. Jul, C. Limpach, I. Pratt, and A. Warfield, Live migration of virtual machines, Proc. of 2nd ACM/USENIX Symposium on Networked Systems Design and Implementation, 2005. 7) M. Nelson, B.-H. Lim, and G.Hutchins, Fast transparent migration for virtual machines, Proc. of the annual conference on USENIX Annual Techinical Conference, pp.391-394, 2005. 8) J. Liedtke, On -kernel construction, Proc. of 15th ACM Symposium on Operating System Principles, pp.237-250, 1995. 9) R. Kaiser and S. Wagner, Evolution of the PikeOS microkernel, Proc. of first International Workshop on Microkernels for Embedded Systems, 2007. 10) G. Heiser, B. Leslie, The OKL4 Microvisor: Convergence Point of Microkernels and Hypervisors, Proc. of the first ACM Asia-Pacific Workshop on Systems, pp.19-24, 2010. 11) U. Steinberg, B. Kauer, NOVA: a microhypervisorbased secure virtualization architecture, Proc of the 5th European conference on Computer systems, pp.209-222, 2010. 12) CODEZERO R Embedded Hypervisor,B LABS, in http://b-labs.com/products/. 13), AXE, Inc, in in http://www.axe-inc.co.jp/hotaru/index.html 14) Wind River Hypervisor, Wind River Systems, Inc, in http://www.windriver.com/products /product-notes/pn Hypervisor 0611.pdf 15) RTS Real-Time Embedded Hypervisor, Real-Time Systems GmbH, in http://www.realtime-systems.com/real-time hypervisor/index.php 16) VMware and Trango, VMware, Inc, in in http://www.vmware.com/company/acquisitions /trango/ 17) Intel R Virtualization Technology and Intel R Active management Technology in Retail Infrastructure, Intel White Paper, 138
http://research.cs.wisc. edu/areas/os/readinggroup/os/papers /vanderpool ia32.pdf, 2006. 18) Atlys TM Board Reference Manual, Digilent, Inc, http://www.digilentinc.com/data/products /ATLYS/Atlys rm.pdf 19) OpenRISC Architecture Manual, OPEN- CORES.ORG, http://www.da.isy.liu.se/courses /tsea44/openrisc/openrisc arch3.pdf, 2003. 20) OR1200 OpenRISC Processor, OPEN- CORES.ORG, in http://opencores.org/ or1k/or1200 OpenRISC Processor 21) ORPSoc, OPENCORES.ORG, in http://opencores.org/or1k/orpsoc 22) nuttcp, nuttcp development team, in http://nuttcp.org/nuttcp/welcome%20page.html 139