---
title: "Introduction to tidycjk"
author: 'Youzhi Yu<br><span style="font-weight: 400; font-size: 0.85em;">University of Chicago</span>'
bibliography: tidycjk.bib
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 3
vignette: >
  %\VignetteIndexEntry{Introduction to tidycjk}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  message = FALSE,
  warning = FALSE,
  comment = "#>"
)
```

## Overview

Almost every text tool in R assumes that a word is a run of characters between
two spaces. That assumption is invisible until it fails, and it fails
completely on Chinese, Japanese and Korean, because Chinese and Japanese do not
put spaces between words at all. Hand `我今天很開心` to a whitespace tokeniser
and it returns one token: the whole sentence. Split it per character instead
and you get six, which cuts `開心` ("happy") in half and destroys the very
signal you were trying to measure.

The problem is not only segmentation. A CJK character occupies **two** terminal
columns but counts as **one** character, so `nchar()` and the width of a
printed table disagree, and every console table containing CJK text comes out
ragged. Fullwidth digits `１２３` look like digits and do not parse as a
number. A Japanese place name written only in Han characters cannot be told
from Chinese by script alone, and most libraries guess anyway.

**tidycjk** addresses these as separate, honest problems, each built on the
Unicode standard [@uax11; @uax14; @uax29] and on ICU, the Unicode
Consortium's own library, rather than on hand-written rules:

| Task | Function(s) |
|------|-------------|
| Detect | `has_cjk()`, `cjk_script()`, `cjk_detect_language()`, `cjk_ratio()` |
| Summarise a column | `cjk_summary()`, `cjk_char_counts()` |
| Segment | `cjk_segment()`, `cjk_tokens()`, `cjk_segmenters()`, `register_cjk_segmenter()` |
| Sentences, n-grams, cleaning | `cjk_sentences()`, `cjk_ngrams()`, `cjk_strip_punct()` |
| Order | `cjk_sort()`, `cjk_order()` |
| Measure width | `cjk_width()` |
| Lay out | `cjk_pad()`, `cjk_truncate()`, `cjk_wrap()` |
| Normalise | `to_halfwidth()`, `to_fullwidth()` (width), `cjk_normalize()` (Unicode forms) |
| Romanise | `cjk_romanize()` |
| Convert Han | `cjk_simplify()`, `cjk_traditionalize()` |
| Convert kana | `to_hiragana()`, `to_katakana()` |
| Hangul jamo | `cjk_jamo()`, `cjk_compose_jamo()` |
| Reference data | `cjk_blocks()` |

```{r setup}
library(tidycjk)
```

## Two layers, one contract

Most verbs work on a plain character vector, in the manner of **stringr**.
Three of them, `cjk_summary()`, `cjk_char_counts()` and `cjk_tokens()`, take
`(data, column)` instead and return a tibble, so they drop into a **dplyr**
pipeline without an adaptor [@wickham2014tidy]. Those three want a data
frame; they do not accept a bare character vector.

The contract is the same throughout, and it is worth stating once because the
rest of this vignette relies on it. Every vector-layer verb is vectorised:
`NA` in gives `NA` out, zero-length input gives zero-length output, and
nothing silently recycles a ragged pair: a mismatch is an error rather than
a warning and a short answer.

The tidy verbs answer about the column rather than about each row, so they
summarise rather than map. `cjk_char_counts()` and `cjk_tokens()` return no
rows for an empty column; `cjk_summary()` always returns exactly one, with
`NA` where there was nothing to average.

```{r corpus}
posts <- data.frame(
  id = 1:5,
  text = c(
    "我今天很開心",            # Chinese
    "こんにちは、元気ですか",  # Japanese, with kana
    "안녕하세요",              # Korean
    "東京都",                  # Japanese written without kana
    "no CJK here at all"
  )
)

cjk_summary(posts, text)
```

`prop_with_cjk` answers "how many of these rows contain any CJK at all";
`mean_ratio` answers "how CJK are they". They come apart exactly where you
would want them to: a corpus of English with one stray ideograph per row
scores near 1 on the first and near 0 on the second.

```{r counts}
cjk_char_counts(posts, text)
```

## Detection, and the value of refusing to guess

`cjk_script()` reports the dominant script; `cjk_detect_language()` reports the
language.

```{r detect}
cjk_script(posts$text)
cjk_detect_language(posts$text)
```

Rows one, four and five come back `NA`. Row five is uninteresting: it has no
CJK in it at all. Rows one and four are the point. `東京都`
is Tokyo Metropolis, unambiguously Japanese, but it is written entirely in
Han characters, and **nothing in the script distinguishes it from Chinese**.
Japanese written without kana and Chinese are the same character set. Any
library that answers `"chinese"` there is guessing, and it will be wrong on
Japanese input in a way you cannot detect downstream, because the output looks
exactly like a confident correct answer.

`tidycjk` returns `NA`. If your corpus is known to be Chinese, ask for the
guess explicitly, so that the assumption lives in your script where a reader
can see it rather than in a library default where they cannot:

```{r detect-optin}
cjk_detect_language(posts$text, han_only = "chinese")
```

## Segmentation

Where a word ends is a fact about a language, not about Unicode, so it needs a
dictionary. `tidycjk` dispatches on an engine name rather than committing to
one, and two engines ship with the package.

`"icu"` is a real word segmenter, using the dictionary-based break iterators
that ICU carries [@icu] and **stringi** exposes [@gagolewski2022stringi]:

```{r segment-icu}
cjk_segment("我今天很開心", engine = "icu")
```

`我 / 今天 / 很 / 開心`: words, not characters. `"character"` is the
dictionary-free baseline, and the contrast shows what a segmenter is for:

```{r segment-character}
cjk_segment("我今天很開心", engine = "character")
```

In a pipeline, `cjk_tokens()` is the CJK-aware counterpart of **tidytext**'s
`unnest_tokens()` [@silge2016tidytext]:

```{r tokens}
cjk_tokens(posts[1:2, ], text, engine = "icu")
```

`vignette("segmentation", package = "tidycjk")` covers the engines, their
limits, and how to register **gibasa** [@gibasa] or jiebaR.

## Sentences and n-grams

Two preprocessing steps that generic tooling gets wrong. A regular expression
on `[.!?]` finds no sentence boundary at all in Chinese, because the
terminator is `。`:

```{r sentences}
cjk_sentences("我今天很開心。你呢？天氣很好！")
```

Character n-grams are the dictionary-free baseline for Chinese retrieval and
classification. Most words are one or two characters, so bigrams capture the
majority of them with no model at all, and they hold up on the names and
coinages where a segmenter is least reliable:

```{r ngrams}
cjk_ngrams("中文很好")
```

No gram is formed across whitespace, so nothing straddles two words of a
Latin run. Punctuation is a different matter, and `cjk_strip_punct()` is the
step that deals with it. It replaces each mark with a space rather than
deleting it, so a bigram cannot span the full stop between two sentences:

```{r strip-punct}
cjk_strip_punct("他說（今天）。真好")

cjk_ngrams(cjk_strip_punct("好。天"))        # nothing spans the stop
cjk_ngrams(cjk_strip_punct("好。天", ""))    # deleting it invents 好天
```

`gsub("[[:punct:]]", "", x)` is the spelling to avoid: R's default engine
resolves the class through the C library, so it removes nothing under
`LC_ALL=C`, and `perl = TRUE` removes nothing from CJK in any locale.

## Sorting

`sort()` reads `LC_COLLATE`, so the order it gives Han text is a property of
the session rather than of the data. These three surnames come out in code
point order under `C` and `en_US.UTF-8`, in pinyin order under `zh_CN.utf8`,
and in a third order under `ja_JP.utf8`. Each is defensible; none is
reproducible, and that is the trap: the same script sorts a name column
differently on a colleague's machine and says nothing about it.

The fallback order is worth knowing, because it is not arbitrary. Within the
base block the unified ideographs are arranged in KangXi radical-stroke
order, so code point order really is radical order. It is simply not
phonetic, and it stops being radical order across blocks, since every
Extension A character sorts before every base-block one, leaving a rare
character nowhere near its radical-mates.

Naming the collation is what makes the answer the same everywhere:

```{r sort}
x <- c("張", "王", "李")   # Zhang, Wang, Li

x[order(vapply(x, utf8ToInt, integer(1)))]  # code point order, explicitly
cjk_sort(x, locale = "zh")                  # pinyin
cjk_sort(x, locale = "zh-u-co-stroke")      # stroke count
```

## Width, and why your tables are ragged

`nchar()` counts characters. A terminal counts columns, and Unicode Annex #11
assigns CJK characters two of them [@uax11].

```{r width}
labels <- c("中文", "abcd", "日本語")

nchar(labels)
cjk_width(labels)
```

Pad to a width rather than to a character count and the column lines up:

```{r pad}
cat(paste0("|", cjk_pad(labels, 8), "|"), sep = "\n")
```

Truncation and wrapping follow the same rule. Wrapping additionally respects
the Unicode line breaking algorithm [@uax14], so it will not put a break
before a closing bracket or a full stop:

```{r wrap}
cat(cjk_wrap("我今天很開心，因為天氣非常好而且朋友來看我", 12))
```

`vignette("width-and-layout", package = "tidycjk")` goes further.

## Normalisation and transliteration

`to_halfwidth()` narrows fullwidth forms and does **only** that, unlike an
`NFKC` pass, which also rewrites ligatures, Roman numerals and circled
numbers:

```{r halfwidth}
as.numeric(to_halfwidth("１２３"))
to_halfwidth("½ Ⅸ ①")   # NFKC would rewrite all three
```

When the blunt instrument *is* the right one, for matching rather than display,
`cjk_normalize()` is it, and it also answers the case a reader may not expect:
plain `"nfc"` rewrites most of the CJK compatibility ideographs, because they
have singleton canonical mappings.

```{r normalize}
ga <- c("\u304c", "\u304b\u3099")   # composed, then KA + voiced mark
nchar(ga)
nchar(cjk_normalize(ga))
cjk_normalize("ＡＢ　①", form = "nfkc")
```

Romanisation, Han script conversion, kana conversion and Hangul jamo
decomposition are all available on the same contract:

```{r translit}
cjk_romanize("中文")
cjk_simplify("漢字")
to_hiragana("カタカナ")
cjk_jamo("한")
```

Each has limits worth knowing before you trust it. Romanisation reads each
Han character on its own, so a word like `银行` ("bank") gets the wrong
reading; simplified/traditional conversion resolves characters from context
but does not substitute regional vocabulary.
`vignette("transliteration", package = "tidycjk")` covers each.

## What tidycjk is built on

`cjk_blocks()` exports the block table the whole package rests on, so the
definition of "CJK" can be read rather than guessed at:

```{r blocks}
head(cjk_blocks(), 8)
nrow(cjk_blocks())
```

Nothing here makes a network request, and the package needs no compiled code
of its own.

## References