2007 3606U104-2 2008 2 4
Control of Data Flows Based on Drag-and-Drop between Appliances Using a Camera Phone Satoru MITSUI Thesis submitted in partial fullfillment of the requirements for the degree of Master in Information and Computer Science Student ID 3606U104-2 Submission Date February 4, 2008 Supervisor Professor Tatsuo Nakajima Department of Computer Science Graduate School of Science and Engineering Waseda University
iii
iv Abstract In this thesis, I propose an interaction model to be able to control the connection between information appliances in the next-generation home network environment by using a camera phone. More information appliances which have the remote accessibility there are, more connections between information appliances are expected. This system adopts the extended direct manipulation technique based on drag-and-drop model in graphic user interface and allows users realize data flows and those directions between information appliances. By using this system, users are not confused about appliances manipulations even in the environment which has many multifunctional information appliances. The evaluation experiment showed that the proposed interaction model provide the usability and the ease of learning to users.
i Abstract iii iv iv v 1 1 1.1.................................... 1 1.2.................................... 2 1.3.................................... 5 1.4............................... 5 2 6 2.1..................... 6 2.1.1 Pick-and-drop........................... 6 2.1.2 mediablocks............................ 6 2.1.3 InfoStick.............................. 7 2.1.4 Touch-and-connect........................ 7 2.1.5 transtick.............................. 7 2.2........................ 8 2.2.1 DataTiles............................. 8 2.3 GUI........... 8 2.4...................... 9 3 10 3.1........................... 10 3.1.1................... 10
ii 3.1.2........................ 11 3.2......... 12 3.2.1................... 12 3.2.2........... 13 3.2.3.......................... 15 4 19 4.1.................................... 19 4.1.1................... 19 4.1.2..................... 20 4.1.3..................... 21 4.1.4.................. 27 4.2.................................... 28 4.2.1............................. 28 4.2.2.......................... 30 4.2.3 UI............... 32 5 35 5.1.................................. 35 5.2............................. 35 5.3.................................. 36 5.4.................................. 36 5.5.............................. 37 5.5.1................... 37 5.5.2....................... 38 6 40 41 42
iii 1 1 [13]................. 2 1 2 [11]............... 3 3 1.............................. 11 3 2 GUI................... 12 3 3.................... 13 3 4..................... 14 3 5................... 14 3 6.............................. 15 3 7................. 16 3 8........................ 17 3 9.............................. 18 4 1......................... 22 4 2 2........................ 23 4 3...................... 23 4 4................ 23 4 5 Sony Type X.................... 24 4 6 X.................... 24 4 7 X................... 24 4 8......................... 25 4 9..................... 25 4 10........................... 25 4 11........................ 25 4 12 UI...................... 25 4 13......................... 26 4 14................. 26
iv 4 15................. 26 4 16................. 26 4 17................................ 26 4 18................ 27 4 19............................. 27 4 20 Nokia N93................................. 29 4 21 VisualCodes................................ 29 4 22.............................. 31 5 1 1.............................. 36 5 2 2.............................. 36
v 4 1 VisualCode ID UDN...................... 32 5 1 PC......... 37 5 2................... 38
1 1,,. 1.1 e-life [10] PDA PC DVD AV PC [2] 1 1 [13] 1 2 DLNA DLNA PC
1 2 1 1: [13] LAN 1.2 [14] 1. 2. 3. Sony
1 3 1 2: [11]
1 4 2 3 ECHONET AV IEEE1394 DLNA 1 PC IP GUI 2.4 Shneiderman direct manipulation technique [8] Pick-and-drop[5] GUI 1 2 ECHONET
1 5 1.3 1.4 2 3 4 5 6
6 2 2.1 2.1.1 Pick-and-drop Pick-and-drop[5] ID 2.1.2 mediablocks mediablocks[9] MIT Ullmer PC
2 7 ID 2.1.3 InfoStick InfoStick[4] SFC CCD InfoStick ID 3.2.1 InfoStick 2.1.4 Touch-and-connect Touch-and-connect[3] 2.1.5 transtick transtick[12]
2 8 PC 2.2 2.2.1 DataTiles DataTiles[6] Ullmer 2.3 GUI GUI
2 9 2.4 3 Shneiderman Direct manipulation[8] 1. 2. 3. 4.
10 3 GUI 3.1 3.1.1 1.1 IP IP AV 3 3 1 AV DLNA UPnP PC ECHONET UPnP PC http://www.upnp.org/
3 11 3 1: UPnP 3.1.2
3 12 3.2 GUI 3.2.1 1. 2. 3. 4. 3 2 3 2: GUI GUI
3 13 3.2.2 5.5 3 3 3 3: 3 4 3 5
3 14 3 4: 3 5:
3 15 3.2.3 3.1.2 2 GUI 3 6:
3 16 GUI 3 7 コンテンツ連携 ( 例 : ディスプレイの静止画をプリンタで印刷する ) ディスプレイ スナップショットを撮る画像取得プリンタ印刷する デバイス選択 (Slect Device) 機能選択 (Select Function) データ選択 (Select Data) 移動 デバイス選択 (Select Device) 機能選択 (Select Function) 移動元デバイスでの操作 移動先デバイスでの操作 連携 印刷設定の UI 枚数濃さ両面? 印刷範囲 3 7: GUI Web
3 17 1 2 3 8:
3 18 GUI コンフィグレーション連携 ( 例 : 温度計をもとにエアコンの温度を自動調整する ) 温度計 温度を送るデータ取得エアコン温度で自動調整 10 10 デバイス選択 (Slect Device) 機能選択 (Select Function) データ選択 (Select Data) 移動 デバイス選択 (Select Device) 機能選択 (Select Function) 移動元デバイスでの操作 移動先デバイスでの操作 連携 自動接続設定の UI 温度と室温の調整 期間など 3 9: 2
19 4 2.4 DataFlowController 4.1 4.1.1 DataFlowController UPnP 1. 2. 3. 1.
4 20 2. 3. PC 1.1 1.2 PC PC PC GUI AV 4.1.2 2.4 Augmented Reality [1] CCD ID DataFlowController
4 21 4.1.3 3.1.2 2.4 DataFlowController 4 1 4 2 4 3, 4 4 X 4 5 X 4 6 X 4 7
4 22 4 1:
4 23 4 2: 2 4 3: 4 4:
4 24 4 8 4 9 4 10 4 11 UI 4 12 4 5: Sony Type X 4 6: X 4 7: X 4 13 4 14 4 15 4 18 UI 30 4 17
第4章 システムの設計と実装 25 図 4 8: 再生可能な動画のリスト 図 4 9: 移動元デバイスを選択した状態 図 4 10: ディスプレイの機能 図 4 11: 動画再生中のディスプレイ 図 4 12: 動画再生中のコントロール UI
第4章 システムの設計と実装 26 図 4 13: 時計デバイスのアイコン 図 4 14: 時計デバイスの機能 移動元選 択時 図 4 15: 時計デバイスの時刻データダイ 図 4 16: 時計デバイスの機能 移動先選 アログ 択時 図 4 17: 時計の同期
4 27 4 18: X 4.2.3 4.1.4 DataFlowController 4 19 4 19: 1
4 28 4.2 PC Nokia N93 S60 3rd 4 20 N93 LAN UPnP Symbian C++ TCP/IP API Symbian C++ Sony Type X 4 5 PC Java J2SE UPnP SOAP 1. VisualCodes 2. CyberLink UPnP 3. 4. UI 1. 2. 9000 2000 4.2.1
4 29 4 20: Nokia N93 VisualCodes VisualCodes[7] ID Symbian S60 2nd API 3rd ID 4 21: VisualCodes
4 30 CyberLink CyberLink UPnP UPnP CyberLink C++ Java Perl C C Symbian C++ 30,000 XML CyberLink LAN SOAP QuickTime for Java Sony Type X MPEG-2 (MPEG-2 PS) MPEG-2 PC API QuickTime for Java 4.2.2 4 22 CyberLink ID ApplianceInfo 16 description.xml UDN 10 4 1 D&D D&D UI UPnP UI CyberLink: http://www.cybergarage.org/net/index.html 2006 http://www.ipa.go.jp/jinzai/esp/2006youth/gaiyou/1-05.html
4 31 Destination Device Source Device Service Application Service Application UPnP UPnP Manipulation Visual Tag Visual Tag Manipulation Destination Service UPnP Controller Destination Service UI Controller UPnP Source Service UPnP Controller Source Service UI Controller discovery Camera Module Code D&D Manger Device Info Appliances' Info Mobile Phone 4 22:
4 32 VisualCode ID UDN 0x00000000000013a9139749 <UDN>uuid:000000000000-1010-8000-0013a9-139749</UDN> 4 1: VisualCode ID UDN UI DataFlowController UPnP A 3 B 5 6 4.2.3 UI UI getname() UI displaydragsearchmenucontent() displaydragsearch- MenuConfig() displaydragselectmenucontent() displaydragselectmenuconfig display- DropSearchMenuContent() displaydropsearchmenuconfig() executeassociation() UI # i f n d e f SERVICEAPP H # d e f i n e SERVICEAPP H 4.1: ServiceAppUI.h # i n c l u d e <c o e c n t r l. h> # i n c l u d e A p p l i a n c e I n f o. h / / CLASS DECLARATION c l a s s C A p p l i a n c e I n f o ;
4 33 / * * * ServiceAppUI * * / c l a s s MServiceAppUI { p u b l i c : / / v i r t u a l TDesC& getname ( ) { } ; v i r t u a l void setname ( ) { } ; / / v i r t u a l TDesC& getdragfunctionname ( ) { } ; v i r t u a l TDesC& getdropfunctionname ( ) { } ; / / v i r t u a l TBool d i s p l a y D r a g S e a r c h M e n u C o n t e n t ( ) { } ; v i r t u a l TBool displaydragsearchmenuconfig ( ) { } ; / / v i r t u a l TBool d i s p l a y D r a g S e l e c t M e n u C o n t e n t ( ) { } ; v i r t u a l TBool d i s p l a y D r a g S e l e c t M e n u C o n f i g ( ) { } ; / / v i r t u a l TBool d isplaydropsearchmenucontent ( MServiceAppUI * dragserviceappui, TDes& AssocName ) { } ; v i r t u a l TBool displaydropsearchmenuconfig ( MServiceAppUI * dragserviceappui, TDes& AssocName ) { } ; / /
4 34 v i r t u a l TBool e x e c u t e A s s o c i a t i o n ( C A p p l i a n c e I n f o * d r a g D e v i c e I n f o, C A p p l i a n c e I n f o * d r o p D e v i c e I n f o, TDesC& AssocName ) { } ; p r i v a t e : } ; # e n d i f / * SERVICEAPP H * /
35 5 5.1 3 5.2 Nokia N93 4 20 1 Sony Type X 4 5 PC 1 5 1 5 2 6 5 20 1 20
第5章 ユーザスタディ 36 図 5 2: 実験環境その 2 図 5 1: 実験環境その 1 5.3 実験手順 まず最初に 被験者に実験の狙いを説明し 実験内で想定する以下の 2 種類のシ ナリオについて解説する その後 被験者にシステムに関する簡単な説明を行う 本 システムがドラッグ ドロップを元にして家電を操作できることを述べた後に実験 を開始する コンテンツ連携 ハードディスクレコーダーからテレビ番組を選択し ディスプレイで再生する コンフィグレーション連携 正常な時計のデータを送信し 時刻がずれる時計を定期的に同期させる 5.4 評価方法 被験者へのインタビュー項目は 本実験で行ってもらう作業を日常どれくらい行っ ているかという質問と 実験の感想に関する質問の二種類である 日常の作業の頻度について PC 上などで普段ドラッグ ドロップを行うか いつどのような時に行うか 右ドラッグ ドロップは行うか 日常生活の中で家電連携を行うことはあるか
5 37 GUI 5.5 5.5.1 PC 2 5 1 Alt+ 2 4 6/6 2/6 1/6 1/6 5 1: PC
5 38 2 2 2 5 2 2 2/6 2/6 2/6 5 2: 5.5.2 6 5 GUI GUI UI 2 GUI
5 39 2 3 VisualCodes CyberLink 90 ID RFID IC 2
40 6 ID RFID UPnP GUI
41 Mr. Fahim Kawsar Mr. Vili Lehdonvirta Ambient Intelligence
42 [1] R.T. Azuma, et al. A survey of augmented reality. Presence: Teleoperators and Virtual Environments(1054-7460), Vol. 6, No. 4, pp. 355 385, 1997. [2] IT. e-japan -2004. http://www.kantei.go.jp/jp/singi/it2/kettei/ejapan2004/040615honbun.html, 6 2004. [3] Y. Iwasaki, N. Kawaguchi, and Y. Inagaki. Touch-and-connect: a connection request framework for ad-hoc networks and the pervasive computing environment. Pervasive Computing and Communications, 2003.(PerCom 2003). Proceedings of the First IEEE International Conference on, pp. 20 29, 2003. [4] N. Kohtake, J. Rekimoto, and Y. Anzai. InfoStick: An Interaction Device for Inter- Appliance Computing. Handheld and Ubiquitous Computing: First International Symposium, HUC 99, Karlsruhe, Germany, September 1999: Proceedings, 1999. [5] J. Rekimoto. Pick-and-drop: a direct manipulation technique for multiple computer environments. Proceedings of the 10th annual ACM symposium on User interface software and technology, pp. 31 39, 1997. [6] J. Rekimoto, B. Ullmer, and H. Oba. DataTiles: a modular platform for mixed physical and graphical interactions. Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 269 276, 2001. [7] M. Rohs and B. Gfeller. Using camera-equipped mobile phones for interacting with real-world objects. Advances in Pervasive Computing, pp. 265 271, 2004. [8] B. Shneiderman. 1.1 Direct manipulation: astep beyond programming languages. Sparks of Innovation in Human-Computer Interaction, 1993. [9] B. Ullmer, H. Ishii, and D. Glas. mediablocks: physical containers, transports, and controls for online media. Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pp. 379 386, 1998. [10] Life. elife. Technical report,, 4 2003. [11].. Technical report,, 8 2004.
43 [12],. transtick:. The 11th Workshop on Interactive Systems and Software, 2003. [13].. Technical Report 1,, 7 2005. [14],.,, 2004.