Tutorial

Internationalization and Localization
in Ruby and Ruby on Rails

http://www.sw.it.aoyama.ac.jp/2012/pub/RubyRails/

IUC 36, Santa Clara, CA, U.S.A., 22 October 2012

Martin J. DÜRST

duerst@it.aoyama.ac.jp

Aoyama Gakuin University

© 2012 Martin J. Dürst, Aoyama Gakuin University

Overview

• Introduction
• Ruby
• Strings and encodings
• String operations
• More about encodings
• Transcoding
• Encoding models
• Ruby on Rails

Introductions

• Programming experience?
• Programming with Ruby/Rails?
• Internationalization/Globalization experience?
• Unicode knowledge?

Goals

• Understanding concepts and model
• Understanding how to use them
• Understanding background
• Have fun with Ruby

Conventions Used

(Ruby) code is green, monospace
puts "Hello Ruby!"

Variable parts are orange
puts "some string"

Encoding are indicated with a _subscript

• "Юに코δ"_UTF-8
• "ユニコード"_SJIS

Results are indicated with "⇒": 1 + 1 ⇒ 2

Frequent Example Юに코δ

• Ю: Cyrillic uppercase YU
• に: Hiragana NI
• 코: Hangul KO
• δ: Greek delta

Ruby

• Created by Yukihiro Matsumoto (Matz)
• Easy for beginners,
  deep for experts
• Object-oriented throughout, but not obtrusive
• (internal) DSLs
• Metaprogramming
• IRB: Interactive Ruby Shell

Basic Ruby

5.times { puts "Hello Ruby!" }

• Everything is an object
• Methods can take blocks ({ } or do end)
• Functions/methods don't need parentheses
• No semicolons at end of line

Ruby Versions

• Ruby 1.8 and before
• Ruby 1.9 and later

Ruby 1.8 and before

• Strings are sequences of bytes
  "Юに코δ".length ⇒ 10
  
• Very little I18N functionality
• Not discussed in this tutorial

Ruby 1.9 and later

• Strings are sequences of characters (codepoints)
  "Юに코δ".length ⇒ 4
  
• Includes core I18N functionality
• Topic of this tutorial

A note about Ruby versions and Unicode versions: The Ruby core team is very conservative in introducing new Unicode versions as bug fixes. New versions therefore only get added on a minor version upgrade (e.g. 1.9.1, 1.9.2, 1.9.3,...).

We still can get 10 as a result with:
"Юに코δ".bytesize ⇒ 10

Ruby Implementations

• MRI (Matz's Ruby Implementation), aka C-Ruby
  available on many platforms (download for Windows)
• JRuby: Ruby on JVM
• Rubinius: Ruby (mostly) in Ruby
• MagLev: Ruby on GemStone/S JIT
• IronRuby: Ruby on Microsoft CLR
• ...

This tutorial is about MRI/C-Ruby, which is the unofficial standard

Ruby Internationalization Goals

• No information loss
• No unnecessary transcoding
• Early detection of encoding problems
• Mostly backwards compatible
• Simple things should stay simple

String Representation

Strings are instances of the String class

Each string is:

1. A sequence of characters
2. Together with a character encoding
   (e.g.: US-ASCII, UTF-8, GB18030, ISO-8859-1, Shift_JIS,...)
   "Юに코δ".encoding.to_s ⇒ "UTF-8"

Fundamentally different from other programming languages.

Java, JavaScript: UTF-16; Python: 8-bit/UTF-16/UTF-32; Perl: 8-bit/UTF-8; C/C++: mb/wc

Objects with Encoding

• String: "Юに코δ"
• Symbol: :Юに코δ
• Regexp: /Юに코δ/
• Identifiers: Юに코δ

Setting an Encoding

(Very simplified, more details later)

• encode (actual transcoding):
  "abcδ".encode 'SJIS' ⇒ "abcδ"_SJIS
• force_encoding (change label, keep bytes):
  "abc\83\C2".force_encoding 'SJIS' ⇒ "abcδ"_SJIS

Terminology: Transcoding

• String method: encode
  (Matz thought that transcode 'smells' of Video converters,...)
• Implementation: transcode.c,
  RUBYSOURCE/enc/trans
• Abstraction as object: Encoding::Converter (very rarely used)
• This tutorial: transcode

Terminology: Other

• character implies code-point
• encoding means character encoding
• String vs. String: former preferred if clear from context

Encoding Clash

"Юに코δ"_UTF-8 + "Юに코δ"_UTF-8
⇒ "Юに코δЮに코δ"_UTF-8

"Юに코δ"_UTF-8 + "Юに코δ"_UTF-16
⇒ Encoding::CompatibilityError

Another example:
"Dürst"_ISO-8859-1 == "Dürst"_ISO-8859-2 ⇒ false

Trying to combine strings with different encodings, as here with concatenation (+), leads to an exception. There are some exceptions (sic!) to this rule that we will look at later. The reasoning for the error here is that transcoding should not happen without the programmer being aware of it.

Trying to compare two character-by-character identical strings in different encodings will produce false, even if these strings are, as in the above example, also byte-for-byte identical. Again, the reason for the result is that encoding mismatches should be detected early. In addition, a simple byte-for-byte comparison could produce false positives.

String Pieces

Strings can be cut up, and iterated over, in four ways:

Lines: "Юに코δ".lines.to_a
⇒ ["Юに코δ"]

Characters: "Юに코δ".chars.to_a
⇒ ["Ю", "に", "코", "δ"]

Codepoints: "Юに코δ".codepoints.to_a
⇒ [1070, 12395, 46020, 948]

Bytes: ..bytes.to_a
⇒ [208, 174, 227, 129, 171, 235, 143, 132, 206, 180]

As the result of chars shows, Ruby treats characters as Strings of length 1. They were integers up to Ruby 1.8. Using the same class for both strings and characters avoids the distinction between characters and strings of length 1. This matches Ruby's "big classes" policy. It also leaves the door open for 'characters' other than single codepoints.

each_line, each_char are older names for lines, chars, codepoints, and bytes. The methods lines, chars, codepoints, and bytes return Enumerators. Here we just use to_a to produce arrays. Enumerators can be used directly for iteration with each, or with separate iterators for mapping with map, selection with select/reject.

The need for several enumerators on a single object, resulting form the change of string/character representation between Ruby 1.8 and 1.9, was one of the main motivators for introducing enumerators into Ruby. This is an interesting example of how internationalization concerns can affect more 'fundamental' language features. The orthogonality resulting from separating what to enumerate over and how to iterate results in very expressive code.

Operations on Strings

Fully internationalized:

• Everything relating indexing
  e.g. reverse, [], ...
• Everything relating to (strict) equality
  e.g. include?, index, ,...
• Everything using regular expressions
  e.g. sub, gsub, scan, split, tr, ...

Not internationalized:

• Casing: capitalize, downcase, ...

Regular Expressions

Oniguruma: Very potent regular expression engine (caution: fork between Ruby Oniguruma and main branch)

Literals: /.../options

Encoding options: n: ASCII, e: EUC-JP, s: SJIS,
u: UTF-8

For UTF-8, implements character classes/scripts

However, not fully UTS#18 (Unicode Regular Expressions) compatible

Regular Expression Details

UTS #18 recommends: \s, \p{space}, \p{Whitespace} → Unicode whitespace

In Ruby: \s → ASCII whitespace; \p{space}, \p{Whitespace} → Unicode whitespace

Examples:

"abc def" =~ /\s/ ⇒ 3(i.e. found)
"abc\u00A0def" =~ /\s/ ⇒ nil
"abc\u00A0def" =~ /\p{space}/ ⇒ 3(i.e. found)
"abc\u00A0def" =~ /\p{Whitespace}/ ⇒ 3(i.e. found)

Keeping \s to mean ASCII whitespace only was done for backwards compatibility. This can be explained as follows: If somebody wrote a script doing some processing where they wanted to match ASCII whitespace characters, they used \s. If Ruby would change \s to suddenly match more characters than before, the meaning of that program would change. Maybe it would change just in the right way. But there's also a good chance that it will change in ways not intended by the programmer. (See also https://bugs.ruby-lang.org/issues/7154.)

Encodings

• Information about Encodings
• Properties of Encodings
• Important Encodings
• System Encodings

Encodings as Objects

String to encoding:
e = Encoding.find "WinDOwS-1252"

Encoding to string: e.to_s ⇒ "Windows-1252"

Methods most often accept both Strings and Encodings to indicate encodings

Constants for Encodings: Encoding::Windows_1252, Encoding::WINDOWS_1252

(We see that encoding names are case-insensitive, and are canonicalized. The internal canonicalization is specific to each encoding, and can start with a lower-case letter. Encodings are also exposed as constants in the Encoding module. There is always a constant with all letters in uppercase. There is also a constant that is the same or close to the internal canonical name, but starts with an uppercase letter. For the constants, hyphens are converted to underscores.)

Information about Encodings

List all encodings:

Encoding.list.join("\n")

Count the encodings supported:

Encoding.list.length

Ruby currently supports close to 100 encodings

Properties of Encodings

• Aliases
• Dummy
• ASCII-compatible

Encoding Aliases

Aliases are only separate names, not separate encodings

To list all encoding names:

Encoding.name_list.join("\n")

To list all (active) aliases:

Encoding.aliases.each { |a, e| puts a + " => " + e }

ASCII-compatible Encodings

Most, but not all encodings are ASCII-compatible:

Encoding::UTF-8.ascii_compatible?
⇒ true

Encoding::UTF-16.ascii_compatible?
⇒ false

ASCII-compatible means ASCII characters stay ASCII bytes:

Encoding::Shift_JIS.ascii_compatible?
⇒ true

Encoding::ISO_2022_JP.ascii_compatible?
⇒ false

Non-ASCII-compatible Encodings

List of non-ASCII-compatible encodings:

Encoding.list.reject(&:ascii_compatible?)

UTF-16BE, UTF-16LE, UTF-16

UTF-32BE, UTF-32LE, UTF-32

UTF-7

ISO-2022-JP, ISO-2022-JP-2, ISO-2022-JP-KDDI

CP50220, CP50221

Almost ASCII-compatible Encodings

• Some encodings have some ASCII characters redefined
  e.g.: \x5C → ¥
• In Ruby, such redefinitions are ignored
  (but a font may still show 'as-if')
• Example: Some transcoding libraries distinguish Shift_JIS and Windows31J in this respect, but not Ruby

Dummy Encodings

List of dummy encodings:

Encoding.list.select(&:dummy?)

UTF-16, UTF-32, UTF-7

ISO-2022-JP, ISO-2022-JP-2, ISO-2022-JP-KDDI

CP50220, CP50221

Dummy encodings are encodings which are treated as opaque byte sequences. But they are labeled, and may be transcoded to some other encodings.

Exceptional Encodings

• Fine distinctions: Shift_JIS vs. Windows31J (aka CP932 aka SJIS)
• For Emoji: UTF8-DoCoMo, UTF8-KDDI,
  UTF-SoftBank
• Stateless equivalents:
  stateless-ISO-2022-JP

Important Encodings

In Ruby, all encodings are equal, but some encodings are more equal than others:

[Adapted from George Orwell's Animal Farm.]

• US-ASCII
• ASCII-8BIT (alias BINARY)
• UTF-8

US-ASCII Encoding

• LCD of most encodings
• Strictly 7-bit
• ASCII-only data
• ASCII-compatible encodings

ASCII-Only Data

ASCII-only data is frequent in programs.

Ruby treats this specially.

This also makes it easy for programmers who don't think about encoding when working only with US-ASCII. Internally, Ruby caches whether a string is ASCII-only or not, to increase performance.

"Data"_SJIS.ascii_only? ⇒ true

"データ"_SJIS.ascii_only? ⇒ false

No Clashes

ASCII-only data does not cause Encoding::CompatibilityError,
even if encoding isn't US-ASCII

"Юに코δ"_UTF-8 + "Data"_SJIS ⇒ "Юに코δData"_UTF-8

"Data"_SJIS + "Юに코δ"_UTF-8 ⇒ "DataЮに코δ"_UTF-8

Works similar for other string operations.

Testing for Clashes

Test with Encoding.compatible? string1 string2

ASCII-8BIT

Alias: BINARY

Use for binary data

Use when high-bit bytes' semantics are unknown (but 7-bit bytes are ASCII)

UTF-8

• Ruby's preferred encoding for Unicode
• Dedicated escape syntax: \u (makes string UTF-8 independent of source encoding)
  "\uABCD\u{A}\u{A FC 3000 12345}"
• Dedicated command-line option: -U
• Strong property support in regular expressions
• Performance optimizations
• Pivot for encoding conversions
  (e.g. Shift_JIS → UTF-8 → KOI-8)

System Encodings

• Source encoding
  Only ASCII-compatible encodings can be source encodings
• Locale encoding
  
• Filename encoding
• External encoding
  
• Internal encoding
  

Source Encoding

• Indicated in comment on first line of source file:
  # ...coding: UTF-8 ...
• More text allowed at ...
• Can be second line if first line is #!/usr/local/ruby
• Limited to ASCII-compatible (no EBCDIC, no UTF-16,...)
• US-ASCII assumed if missing
• UTF-8 "BOM" also defines source encoding

Non-ASCII Identifiers

π = 3.14...

Possible, but not recommended (possible exceptions: basic education, special terminology)

Problems:

• Many scripts don't have case distinctions
  → use C_class_name
• Ruby only treats ASCII uppercase as uppercase
• Encodings are relevant for identifiers, too:
  π_UTF-8 ≠ π_SJIS

Locale Encoding

• Set from environment
• Queried with Encoding.locale_charmap

Filesystem Encoding

• Represents encoding of file names in file system
• May be UTF-16 on Windows
• Otherwise, equal to locale encoding

External encoding

• Used when reading in data
• Default is locale encoding
• Set with Encoding.default_external=(use sparingly!)
• Queried with Encoding.default_external

Internal encoding

• Used when reading in data
• Default is nil (i.e. unknown)
• Set with Encoding.default_internal=
• Queried with Encoding.default_internal(use sparingly!)

Encoding Integrity Check

"abc\xFE"_UTF-8.valid_encoding? ⇒ false
"abc\xC0\x80"_UTF-8.valid_encoding? ⇒ false
"abc\xC2\x80"_UTF-8.valid_encoding? ⇒ true
"abc\xCE\xA2"_UTF-8.valid_encoding? ⇒ true(there is no uppercase final Sigma (yet!?))

Checks for code structure, not unassigned codepoints

How to Avoid Invalid Data

• For performance, Ruby checks encoding validity only when necessary
• Check on input (read)
• Transcode
  Caution: transcoding to same encoding is no-op
• Don't use low-level escapes and access methods

Transcoding Overview

• String transcoding
• Transcoding options
• Failures and fallbacks
• Transcoding on input/output

Transcoding String to String

• "ユニコード"_SJIS.encode('UTF-8)
  ⇒ "ユニコード"_UTF-8
• Destructive equivalent: encode!

  string1 = "ユニコード"_SJIS

  string1.encode! 'UTF-8'

  string1 ⇒ "ユニコード"_UTF-8

Unfortunately, force_encoding is destructive, no non-destructive equivalent.

Explicit to and from Encodings

Example: Force UTF-8 "double-encoding" (not recommended)

"Юに코δ"_UTF-8.force_encoding('iso-8859-1').encode('UTF-8')

Equivalent:

"Юに코δ"_UTF-8..encode('UTF-8', 'iso-8859-1')

Note order of arguments: to ← from

(I always disliked the order of arguments in iconv, but it was unavoidable here, because usually, just the 'to' encoding is needed, so that has to come first.)

Transcoding Failures

• Problem with input: Invalid byte sequence
  "abc\xFE"_UTF-8.encode 'SJIS' ⇒ Encoding::InvalidByteSequenceError
• Problem with output: Undefined conversion
  "Юに코δ"_UTF-8.encode 'SJIS' ⇒ Encoding::UndefinedConversionError

Fallback Options

• undef: What to do with undefined characters (:replace/:ignore(:ignore can lead to security issues!))
• invalid: What to do with invalid bytes (:replace/:ignore(:ignore can lead to security issues!))
• replace: String used as replacement
  (default: U+FFFD or '?')
• fallback: Object (e.g. Hash) used to look up replacement

Additional options for newline conversion and XML escaping

Transcoding on Input and Output

Methods that open files/streams take an encoding: option:

"external": External encoding

"external:internal": External and internal encoding

If 'internal' is undefined (here or otherwise), then external is used for labeling input.

If 'internal' is defined, then conversion takes place.

Caution: Has to be set explicitly for stdin, stdout,...

Operations on IOs

• IO: Class for input and output (parent of File,...)
• external_encoding/internal_encoding
• set_encoding: changes external/internal encoding
• readchar/readbyte/readline: partial reading
• bytes/chars/lines: enumerator for reading

Command-line Options

• -E, --encoding: Set default external or external:internal encoding
• -U: Set default internal encoding to UTF-8

Adding an Encoding or Transcoding

• Encodings are added in RUBYSOURCE/enc
• Transcodings are added in
  RUBYSOURCE/enc/trans
• Start with an example close to what you need, and copy/modify
• Lists of encodings and transcodings are constructed automatically
  (see encdb.h, transdb.h)
• Contact me for advice,
  or to add your work to Ruby

Encoding Models Overview

• Encoding Models for Applications
• Encoding Models for Libraries

Encoding Models for Applications

• Any single encoding
  
• One single encoding
  
• One single internal encoding
• Mixed Encodings
• others?

An encoding model describes which encoding(s) can be used in what part of an application (e.g. externally, internally). It defines the conditions and restrictions with respect to string processing that the application (programmer) has to maintain. When creating a Ruby application, it is important to choose the appropriate encoding model.

Any Single Encoding

• One run of application ⇔ one encoding
• Different runs may use different encodings
• On different systems, different encodings may be used (e.g. different Unix or Windows locales)
• Often assumes ASCII-compatible scripts, and works on ASCII parts only
• Suitable for simple scripts
• No declared source encoding
• Typical example: Replace some HTML/XML tags with others
• Advice: Okay as long as it doesn't hurt

One Single Encoding

• All of the application uses one single encoding
• Could be any encoding, but fixed when code created
• Assumptions about encoding hard-coded
• Preferably use UTF-8
• Example: New stand-alone application with i18n needs
• Advice: The ideal future

For applications that use more character semantics outside the ASCII range, or that keep data for a long time, the ideal solution is to use only a single encoding. This should be UTF-8 because that covers all of Unicode and works best with Ruby.

One Single Internal Encoding

• Outside words has many encodings
• Inside, only a single encoding
• Conversion at boundaries
• Emulation of other (programming) languages' encoding models
• Use UTF-8 internally
• Advice: Realistic choice in many cases

If data (e.g. files) in other encodings also have to be handled, it will in most cases be best to adopt the model of most other programming languages: Use Unicode (i.e. UTF-8 for Ruby) inside, and convert on input/output.

Mixed Encodings

• Many encodings outside and inside
• Full power of Ruby encodings available
• Ruby takes care of bookkeeping
• Transcoding needs to be explicit
• Example: Decoding email messages
• Advice: Don't go there if you don't have to

Encoding Models for Libraries

• Operate on ASCII only, work on all ASCII-compatible encodings
• Operate on arbitrary single encodings
  Example: CSV standard library
• Use single encoding only, library user has to adapt
  E.g.: UTF-8

Important: Don't touch system encodings (except maybe for a framework)

Libraries for Unicode Support

Ruby leaves open some important i18n support:

• Full case conversions
• Unicode normalizations
• Text segmentation,...

These can be done with:

• UnicodeUtils
• ActiveSupport::Multibyte

UnicodeUtils

By Stefan Lang, in pure Ruby

Example:

require "unicode_utils/upcase"

UnicodeUtils.upcase("weiß") => "WEISS"

UnicodeUtils.upcase("i", :tr) => "İ"

ActiveSupport

• Toolbox used in different parts of Rails
• ActiveSupport::Multibyte
  • Strings-as-characters before Ruby 1.9
  • Unicode Normalization
  • Grapheme Boundaries
  • Full Unicode case conversions,...

Ruby on Rails

• Extracted from Basecamp by David Heinemeier-Hansen (DHH)
• Revolutionary Web Framework (Revolutionary when it came out in 2004, now widely copied but still leading.)
• Overall structure: MVC
• Database-backed (ActiveRecord)
• Convention over Configuration

Rails and Encodings

• Uses UTF-8 out of the box
• Can use other encodings (but why would you)
• Make sure database is set up for UTF-8 before you start

Rails I18n Framework - Message Translation

• How to translate texts in the application: t
• What language to use
• Where to store the translations (backend plugins)
• see http://ruby-i18n.org/wiki for more

Text Replacement: Templates

In templates, replace

<h1>Hello Rails!</h1>

with

<h1><%= t 'welcome.rails' %></h1>

welcome.rails is a structured key used for looking up the translated string

Text Replacement: Backends

• Default: YAML files
• gettext
• SQL database
• ...

Possible Future Work

• Integration of advanced Unicode support into Ruby core
• More flexible encoding model
  
  • More checks (ASCII no longer special)
  • More automatic conversions
• Automatic detection of localizable content in Rails templates
• Better locale support in Ruby and Rails
• Your idea/contribution here!

Q & A

Send questions and comments to Martin Dürst, duerst@it.aoyama.ac.jp

A new version of this tutorial is available at:

http://www.sw.it.aoyama.ac.jp/2015/pub/RubyRails