- PDF Free Download

Size: px

Start display at page:

Download ""

きみのしんわたぬき
6 years ago
Views:

1 THE WORLD IN 30 MINUTES Tim Bray RubyKaigi 2007 Sun Microsystems

3 /[a-za-z]+/ This is probably a bug.

4 The Problems We Have To Solve Identifying characters Storage Byte character mapping Transfer Good string API

5 Published in 1996; it has 74 major sections, most of which discuss whole families of writing systems. 1996年刊; 世界の書記体系についての書籍

7 Identifying Characters 文字の識別

8 1,114,112 Unicode Code Points 17 Planes each with 64k code points: U+0000 U+10FFFF BMP Han Characters Dead Languages & Math Basic Multilingual Plane Non-BMP Astral Planes A 0000 B 0000 C 0000 D 0000 E 0000 F 0000 Private Use Language Tags ,024 characters defined in Unicode 5.0

9 The Basic Multilingual Plane (BMP) U+0000 U+FFFF A000 B000 C000 D000 E000 F000 Alphabets Asian-language Support Punctuation Han Characters Yi Hangul Private Use Surrogates * (*: Legacy-Compatibility junk)

10 Unicode Character Database 00C8;LATIN CAPITAL LETTER E WITH GRAVE;Lu;0;L; ;;;;N;LATIN È CAPITAL LETTER E GRAVE;;;00E8; Character #200 is LATIN CAPITAL LETTER E WITH GRAVE, a lower-case letter, combining class 0, renders L-to-R, can be composed by U+0045/U+0300, had a different name in Unicode 1, isn t a number, lowercase is U+00E8.

11 $ U+0024 DOLLAR SIGN

12 Ž U+017D LATIN CAPITAL LETTER Z WITH CARON

13 U+00AE REGISTERED SIGN

14 ή U+03AE GREEK SMALL LETTER ETA WITH TONOS

15 Ж U+0416 CYRILLIC CAPITAL LETTER ZHE

16 א U+05D0 HEBREW LETTER ALEF

17 ظ U+0638 ARABIC LETTER ZAH

18 ਗ U+0A17 GURMUKHI LETTER GA

19 ઈ U+0A88 GUJARATI LETTER II

20 ฆ U+0E06 THAI CHARACTER KHO RAKHANG

21 U+0F12 TIBETAN MARK RGYA GRAM SHAD

22 Ꮊ U+13BA CHEROKEE LETTER ME

23 ᐑ U+1411 CANADIAN SYLLABICS WEST-CREE WII

24 ᠠ U+1820 MONGOLIAN LETTER ANG

25 U+2030 PER MILLE SIGN

26 ⅝ U+215D VULGAR FRACTION FIVE EIGHTHS

27 U+21A9 LEFTWARDS ARROW WITH HOOK

28 U+221E INFINITY

29 U+2764 HEAVY BLACK HEART

30 U+3055 HIRAGANA LETTER SA

31 U+30C0 KATAKANA LETTER DA

32 U+4E2D (Han character)

33 U+8A9E (Han character)

34 U+AC7A (Hangul syllabic)

35 ! U+1D12B (Non-BMP) Musical Symbol Double Flat

36 U+2004E (Non-BMP) (Han character)

37 Nice Things About Unicode Huge repertoire Room for growth Private use areas Sane process Unicode character database Ubiquitous standards/tools support

38 Difficulties With Unicode Combining forms Awkward historical compromises Han unification

39 Han Unification (漢字統合) 賛成: en.wikipedia.org/wiki/han_unification 反対: tronweb.super-nova.co.jp/characcodehist.html 中立: Alternatives (代替案) For Japanese scholarly/historical work: 文字鏡, also see Tron, GTCode. Also see Wittern, Embedding Glyph Identifiers in XML Documents.

40 Byte Character Mapping U+4E2D (Han character) How do I encode 0x4E2D in bytes for computer processing

41 Storing Unicode in Bytes バイト列のエンコード方式公式: UTF-8, UTF-16, UTF-32 実用的: ASCII, EBCDIC, Shift-JIS, Big5, GB18030, EUC-JP, EUC-KR, ISCII, KOI8, Microsoft code pages, ISO-8859-*, and others.

42 UTF-* Trade-offs UTF-8: Most compact for Western languages, C-friendly, non-bmp processing is transparent. UTF-16: Most compact for Eastern languages, Java/C#-friendly, C-unfriendly, non-bmp processing is horrible. UTF-32: wchar_t, semi-c-friendly, 4 bytes/char. Note: Video is 100MB/minute... Web search: characters vs. bytes

43 Text Arriving Over the Network ネットワーク経由のテキスト $Ž ήж ﻅא ਗઈฆ Ꮊ ᐑ ᠠ ⅝ さダ中語걺!

44 encoding it s in. An XML document knows what - Larry Wall

47 What Python 3000 Will Do String str unicode bytes str bytes mutable int range(256) encode/decode API bytes(s, "Latin-1") bytes str (. b1.find(b2)) (. not b.upper()) Unicode I/O API (Guido s Slide) ( ) April 19, 2006 (c) 2006 Python Software Foundation 47

48 Core Methods With I18n Issues == =~ [] []= eql gsub gsub! index length lstrip lstrip! match rindex rstrip rstrip! scan size slice slice! strip strip! sub sub! tr tr!

49 Problem: Missing Character Iterator Maybe String#each_char Maybe change String#each Maybe String#chars

50 On Case-folding Lower-case I : i or ı Upper-case i : I or İ Upper-case ß Upper-case é Just Say No! ほんとダメだってば!

51 Regexp and Unicode stag = "<[^/]([^>]*[^/>])>" etag = "</[^>]*>" empty = "<[^>]*/>" Will Ruby 2.0 do these alnum = '\p{l} \p{n} ' + '[\x{4e00}-\x{9fa5}] ' + '\x{3007} [\x{3021}-\x{3029}]' wordchars = '\p{l} \p{n} ' + "[-._:'] " + '\x{2019} [\x{4e00}-\x{9fa5}] \x{3007} ' + '[\x{3021}-\x{3029}]' e.g. won t-go word = "((#{alnum})((#{wordchars})*(#{alnum})))" text = "(#{stag}) (#{etag}) (#{empty}) #{word}" regex = /#{text}/

52 Referring to Characters if in_euro_area append 0x20ac # Euro elsif in_japan append 0xa5 # Yen else append '$' end Common idiom while writing XML.

54 Who s Working on the Problem Matz/Ko1: M17n for Ruby 2 Julik: ActiveSupport::MultiByte (in edge Rails) Nikolai: Character encodings project (rubyforge.org/projects/char-encodings/) JRuby guys: Ruby on a Unicode platform

55 Thank You! this talk:

25 II :30 16:00 (1),. Do not open this problem booklet until the start of the examination is announced. (2) 3.. Answer the following 3 proble

25 II :30 16:00 (1),. Do not open this problem booklet until the start of the examination is announced. (2) 3.. Answer the following 3 proble 25 II 25 2 6 13:30 16:00 (1),. Do not open this problem boolet until the start of the examination is announced. (2) 3.. Answer the following 3 problems. Use the designated answer sheet for each problem.