ThumbStick: 1,a) 2,b) ThumbStick ThumbStick ThumbStick: Two-dimensional Thumb Motion Recognition System for One-handed Smartwatch Input Aoyama Shuhei 1,a) Shizuki Buntarou 2,b) Abstract: In this paper, we show the system ThumbStick, which realizes a two-dimensional input on a smartwatch, with only the thumb of the hand wearing the smartwatch. The system estimates the thumb position using the change of the wrist s contour. Thus, the user can operate the smartwatch with a joystick-like two-dimensional input using the thumb. 1. 1 Department of Computer Science, Graduate School of Systems and Information Engineering, University of Tsukuba 2 Faculty of Engineering, Information and Systems, University of Tsukuba a) aoyama@iplab.cs.tsukuba.ac.jp b) shizuki@cs.tsukuba.ac.jp 623
1 ThumbStick. Fig. 1 Operation image of a smartwatch with ThumbStick. ThumbStick Thumb- Stick 1 2. 2.1 Gong WristWhirl [1] Sun Float [2] Kerber [3] [4] Huang DigitSpace [5] Huang 2.2 / / Zhang Tomo [6] Rekimoto [7]Dementyev [8] Fukui [9] Ortega-Avila [10] Gong [1] McIntosh SensIR [11] McIntosh 3. 3.1 3.2 2A 2B 624
2 A B C Fig. 2 Wrist-worn sensor device. A) Overview, B) the side of sensors array, C) the inside of the device. 4 Fig. 4 Experimental application P29 P0 P1 A B S13 S14 S15 S12 S11 P30 S10 S9 3 Fig. 3 The thumb position to be identified. S8 S7 S6 S5 S4 S3 S2 S1 PVA Filament 1.75 mm natural, Shenzhen Esun Industrial 3D FLASHFORGE Dreamer, Apple Tree 1 mm 2C (CD74HC4067, Texas Instruments) 16 TPR-105F, GENIXTEK 7.68 mm BLE nrf51822 Rev.3 SoC, Nordic Semiconductor mbed mbed TY51822r3, Switch Science) 2 mm 4. 1 24 3 P0 P29 P30 P0 P29 P30 5 AB Fig. 5 Experiment environment: A) Posture, B) The position of each sensor in the sensor device. 4.1 Processing ver 2.2.1 4 3 4.2 5A 5B BLE USB PC 625
Table 1 1 Confusion matrix of classification result of thumb positions. P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 P18 P19 P20 P21 P22 P23 P24 P25 P26 P27 P28 P29 P30 P0 20 60 10 10 P1 50 20 20 10 P2 20 30 40 10 P3 50 10 30 10 P4 10 20 10 30 20 10 P5 10 10 30 20 30 P6 10 20 20 20 20 10 P7 10 10 20 20 10 20 10 P8 10 10 10 20 30 20 P9 30 10 20 10 30 P10 30 20 30 20 P11 30 30 40 P12 10 10 30 50 P13 10 50 40 P14 40 60 P15 10 50 20 10 10 P16 30 50 10 10 P17 20 30 10 20 10 10 P18 10 20 30 20 10 10 P19 40 20 10 20 10 P20 10 10 30 40 10 P21 20 40 30 10 P22 40 20 40 P23 10 10 20 20 40 P24 10 20 20 40 10 P25 10 30 20 40 P26 20 70 10 P27 60 40 P28 10 10 30 20 30 P29 20 10 60 10 P30 10 90 4.3 5A 12 P29 P0 1 2.0 P30 1 P0 10 4.4 Weka Machine Learning Toolkit[12] 10 fold 10 10 30 16 10 10 1 16 LIBVSVM[13] 1 16.1% 24 3 76.8% 48 2 5 91.3% 90% 5. ThumbStick ThumbStick ThumbStick 2 626
ThumbStick A B C 5.1 SONY Xperia Z5 E6653; Android 6.0.1 SONY SmartWatch 3 SWR50; Android Wear 1.5.0 BLE 5.2 ThumbStick LIBSVM[13] 6A 6B 0 1 X Y P 29 X = {P rob i cos(rad i )} r ( r < X < r) i=p 0 P 29 Y = {P rob i sin(rad i )} r ( r < Y < r) i=p 0 P rob i i Rad i i r 6B 5.3 6C 6 ThumbStick A B C) Fig. 6 Application screenshot: A) Smartphone application - before training, B) smartphone application - after training, C) smartwatch application 6. ThumbStick Float[2] 7. 627
[1] Gong, J., Yang, X.-D. and Irani, P.: WristWhirl: Onehanded Continuous Smartwatch Input Using Wrist Gestures, in Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST 16, pp. 861 872, New York, NY, USA (2016), ACM. [2] Sun, K., Wang, Y., Yu, C., Yan, Y., Wen, H. and Shi, Y.: Float: One-Handed and Touch-Free Target Selection on Smartwatches, in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI 17, pp. 692 704, New York, NY, USA (2017), ACM. [3] Kerber, F., Lessel, P. and Krüger, A.: Same-side Hand Interactions with Arm-placed Devices Using EMG, in Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA 15, pp. 1367 1372, New York, NY, USA (2015), ACM. [4],,, Vol. 40, No. 2, pp. 389 398 (1999). [5] Huang, D.-Y., Chan, L., Yang, S., Wang, F., Liang, R.- H., Yang, D.-N., Hung, Y.-P. and Chen, B.-Y.: DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions, in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 16, pp. 1526 1537, New York, NY, USA (2016), ACM. [6] Zhang, Y. and Harrison, C.: Tomo: Wearable, Low- Cost Electrical Impedance Tomography for Hand Gesture Recognition, in Proceedings of the 28th Annual ACM Symposium on User Interface SoftwareTechnology, UIST 15, pp. 167 173, New York, NY, USA (2015), ACM. [7] Rekimoto, J.: GestureWrist and GesturePad: Unobtrusive Wearable Interaction Devices, in Proceedings of the 5th IEEE International Symposium on Wearable Computers, ISWC 01, pp. 21 27, Washington, DC, USA (2001), IEEE Computer Society. [8] Dementyev, A. and Paradiso, J. A.: WristFlex: Lowpower Gesture Input with Wrist-worn Pressure Sensors, in Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST 14, pp. 161 166, New York, NY, USA (2014), ACM. [9] Fukui, R., Watanabe, M., Shimosaka, M. and Sato, T.: Hand Shape Classification in Various Pronation Angles using a Wearable Wrist Contour Sensor, Advanced Robotics, Vol. 29, No. 1, pp. 3 11 (2015). [10] Ortega-Avila, S., Rakova, B., Sadi, S. and Mistry, P.: Non-invasive Optical Detection of Hand Gestures, in Proceedings of the 6th Augmented Human International Conference, AH 15, pp. 179 180, New York, NY, USA (2015), ACM. [11] McIntosh, J., Marzo, A. and Fraser, M.: SensIR: Detecting Hand Gestures with a Wearable Bracelet Using Infrared Transmission and Reflection, in Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST 17, pp. 593 597, ACM (2017). [12] Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P. and Witten, I. H.: The WEKA Data Mining Software: An Update, Vol. 11, pp. 10 18, New York, NY, USA (2009), ACM. [13] Chang, C.-C. and Lin, C.-J.: LIBSVM: A Library for Support Vector Machines, Vol. 2, pp. 27:1 27:27, New York, NY, USA (2011), ACM. 628