{
  "_id": "6ac60c09f511ae58c09a5087",
  "Package": "tidycjk",
  "Type": "Package",
  "Title": "Tidy Tools for Chinese, Japanese and Korean Text",
  "Version": "0.2.0",
  "Authors@R": "c(person(given = \"Youzhi\",\nfamily = \"Yu\",\nrole = c(\"aut\", \"cre\"),\nemail = \"yuyouzhi666@icloud.com\"))",
  "Description": "A tidy toolkit for text that is written in Chinese,\nJapanese or Korean. Most text tooling in R assumes that words\nare separated by whitespace, which CJK writing does not use, so\nordinary summaries of a text column either treat a sentence as\none undifferentiated blob or split it into isolated characters.\nWord segmentation is a pluggable engine, with dictionary-based\nsegmentation for Chinese and Japanese supplied through 'ICU'\nand further engines registrable by the caller, because where a\nword ends is a fact about a language and not about Unicode.\n'tidycjk' classifies characters by Unicode block, reports which\nscript and which language a text is written in, measures how\nmuch of a text is CJK, and turns those measurements into\ntibbles that slot straight into a 'tidyverse' workflow. It also\nmeasures display width in terminal columns, pads, truncates and\nwraps to a width rather than to a character count, and\nnormalises fullwidth and halfwidth forms surgically --\nincluding composing halfwidth katakana voiced marks into single\ncode points -- while offering the Unicode normalisation forms\nseparately for when a full 'NFKC' fold is what is wanted.\nRomanisation, conversion between simplified and traditional\nHan, conversion between the two kana syllabaries, and\ndecomposition of Hangul syllables into jamo are provided on the\nsame vectorised contract. Sentence segmentation, character\nn-grams, punctuation removal by Unicode category and\nlocale-aware ordering cover the preprocessing that generic\ntooling gets wrong on CJK, where the sentence terminators are\nideographic, a code point sort is not an alphabetical one, and\nthe POSIX punctuation class is resolved by the C library rather\nthan by Unicode. Language detection deliberately returns NA\nrather than guessing when a text is written in Han characters\nonly, because Japanese written without kana cannot be\ndistinguished from Chinese by script alone. Everything is\nderived from the Unicode specification; the package makes no\nnetwork requests and needs no compiled code of its own.",
  "License": "GPL (>= 3)",
  "URL": "https://pursuitofdatascience.github.io/tidycjk/,\nhttps://github.com/PursuitOfDataScience/tidycjk",
  "BugReports": "https://github.com/PursuitOfDataScience/tidycjk/issues",
  "Encoding": "UTF-8",
  "Language": "en-GB",
  "Roxygen": "list(markdown = TRUE)",
  "Config/testthat/edition": "3",
  "VignetteBuilder": "knitr",
  "RoxygenNote": "7.3.2",
  "Config/pak/sysreqs": "libicu-dev",
  "Repository": "https://pursuitofdatascience.r-universe.dev",
  "Date/Publication": "2026-10-05 17:36:49 UTC",
  "RemoteUrl": "https://github.com/pursuitofdatascience/tidycjk",
  "RemoteRef": "HEAD",
  "RemoteSha": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
  "NeedsCompilation": "no",
  "Packaged": {
    "Date": "2026-10-07 08:28:49 UTC",
    "User": "root"
  },
  "Author": "Youzhi Yu [aut, cre]",
  "Maintainer": "Youzhi Yu <yuyouzhi666@icloud.com>",
  "_user": "pursuitofdatascience",
  "_type": "src",
  "_file": "tidycjk_0.2.0.tar.gz",
  "_fileid": "https://r2.ropensci.org/066c954cd712356677078ea0b8991a331108ac1b136da4fce3683c28bceda304",
  "_filesize": 846321,
  "_sha256": "066c954cd712356677078ea0b8991a331108ac1b136da4fce3683c28bceda304",
  "_expires": "2027-01-15T09:08:22.000Z",
  "_created": "2026-10-07T08:28:49.000Z",
  "_published": "2026-10-07T09:08:25.183Z",
  "_buildurl": "https://github.com/r-universe/pursuitofdatascience/actions/runs/37593080696",
  "_status": "success",
  "_host": "GitHub-Actions",
  "_jobs": [
    {
      "job": 112702261795,
      "time": 131,
      "config": "linux-devel-x86_64",
      "r": "4.7.0",
      "check": "OK",
      "artifact": "11470661547"
    },
    {
      "job": 112702262042,
      "time": 106,
      "config": "linux-release-x86_64",
      "r": "4.6.1",
      "check": "OK",
      "artifact": "11470217424"
    },
    {
      "job": 112702262072,
      "time": 98,
      "config": "macos-oldrel-arm64",
      "r": "4.5.3",
      "check": "OK",
      "artifact": "11470373071"
    },
    {
      "job": 112702261923,
      "time": 116,
      "config": "macos-release-arm64",
      "r": "4.6.1",
      "check": "OK",
      "artifact": "11471610906"
    },
    {
      "job": 112699217407,
      "time": 223,
      "config": "source",
      "r": "4.6.1",
      "check": "OK",
      "artifact": "11470650052"
    },
    {
      "job": 112702261803,
      "time": 113,
      "config": "wasm-release",
      "r": "4.6.0",
      "check": "OK",
      "artifact": "11470211827"
    },
    {
      "job": 112702261961,
      "time": 100,
      "config": "windows-devel-x86_64",
      "r": "4.7.0",
      "check": "OK",
      "artifact": "11470006083"
    },
    {
      "job": 112702261875,
      "time": 90,
      "config": "windows-oldrel-x86_64",
      "r": "4.5.3",
      "check": "OK",
      "artifact": "11470535555"
    },
    {
      "job": 112702262036,
      "time": 97,
      "config": "windows-release-x86_64",
      "r": "4.6.1",
      "check": "OK",
      "artifact": "11470710201"
    }
  ],
  "_upstream": "https://github.com/pursuitofdatascience/tidycjk",
  "_commit": {
    "id": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
    "author": "Youzhi Yu <yuyouzhi666@icloud.com>",
    "committer": "Youzhi Yu <yuyouzhi666@icloud.com>",
    "message": "fix: harden 0.2.0 for CRAN: one input contract, Unicode 18, true docs\n\nA release pass over the whole package: every verb run against edge\ncases, every claim in the help pages, vignettes, NEWS and cran-comments\nre-run rather than re-read. Most of what it found was documentation\nasserting something the code does not do; the rest was behaviour.\n\nBehaviour:\n\n* Undecodable text is one error from every verb. Left to stringi, the\n  same invalid bytes came back five ways: our message, U+FFFD from the\n  ICU transforms, the raw bytes from cjk_sentences() and cjk_sort(),\n  nchar()'s own error from cjk_wrap(), and the byte silently dropped\n  by the \"icu\" engine. .cjk_as_text() now checks once, with a call that\n  raises (stri_length()), and names the argument; `pad`, `ellipsis` and\n  `replacement` are checked the same way, and \"bytes\" strings refused.\n* Korean and Chinese text with the katakana middle dot U+30FB came back\n  \"japanese\". The dot is punctuation whose Script_Extensions include Han\n  and Hangul; it and U+FF65 no longer count as kana.\n* The block table is current to Unicode 18.0: CJK Extension J, and the\n  four kana extension blocks ?cjk_blocks claimed were covered and were\n  not, split where the Script property changes. Cross-checked against\n  the 18.0 Blocks.txt and Scripts.txt: names and bounds match, the one\n  mislabel is the documented U+1B123, and the only script letters left\n  out are in the deliberately excluded blocks plus U+02EA/U+02EB, now\n  documented.\n* cjk_jamo() and cjk_compose_jamo() apply the Hangul syllable\n  arithmetic and nothing else. Built on NFD and NFC they also split\n  accented letters and replaced compatibility ideographs, which in a\n  Korean column rewrites Hanja. The round trip is now exact; a test\n  checks composition against NFC on 2,000 random jamo strings and\n  decomposition against NFD on all 11,172 syllables.\n* cjk_wrap() always fills greedily. It took stringi's optimal fit up to\n  10,000 characters, on the documented ground that the ",
    "time": 1791221809
  },
  "_maintainer": {
    "name": "Youzhi Yu",
    "email": "yuyouzhi666@icloud.com",
    "login": "pursuitofdatascience",
    "description": "CUDA out of memory",
    "uuid": 54338793
  },
  "_distro": "resolute",
  "_registered": true,
  "_dependencies": [
    {
      "package": "R",
      "version": ">= 3.5.0",
      "role": "Depends"
    },
    {
      "package": "dplyr",
      "version": ">= 1.1.0",
      "role": "Imports"
    },
    {
      "package": "stringi",
      "role": "Imports"
    },
    {
      "package": "tibble",
      "role": "Imports"
    },
    {
      "package": "utils",
      "role": "Imports"
    },
    {
      "package": "knitr",
      "role": "Suggests"
    },
    {
      "package": "rmarkdown",
      "role": "Suggests"
    },
    {
      "package": "testthat",
      "version": ">= 3.0.0",
      "role": "Suggests"
    }
  ],
  "_owner": "pursuitofdatascience",
  "_selfowned": true,
  "_usedby": 0,
  "_updates": [
    {
      "week": "2026-32",
      "n": 9
    },
    {
      "week": "2026-33",
      "n": 1
    },
    {
      "week": "2026-34",
      "n": 1
    },
    {
      "week": "2026-37",
      "n": 5
    },
    {
      "week": "2026-39",
      "n": 2
    },
    {
      "week": "2026-41",
      "n": 2
    }
  ],
  "_tags": [
    {
      "name": "v0.1.0",
      "date": "2026-08-22"
    },
    {
      "name": "v0.2.0",
      "date": "2026-10-05"
    }
  ],
  "_stars": 0,
  "_contributors": [
    {
      "user": "pursuitofdatascience",
      "count": 13,
      "uuid": 54338793
    },
    {
      "user": "claude",
      "count": 7,
      "uuid": 81847
    }
  ],
  "_userbio": {
    "uuid": 54338793,
    "type": "user",
    "name": "Y. Yu",
    "followers": 35,
    "description": "CUDA out of memory"
  },
  "_downloads": {
    "count": 297,
    "source": "https://cranlogs.r-pkg.org/downloads/total/last-month/tidycjk"
  },
  "_devurl": "https://github.com/pursuitofdatascience/tidycjk",
  "_pkgdown": "https://pursuitofdatascience.github.io/tidycjk/",
  "_searchresults": 10,
  "_rbuild": "4.6.1",
  "_assets": [
    "extra/citation.cff",
    "extra/citation.html",
    "extra/citation.json",
    "extra/citation.txt",
    "extra/contents.json",
    "extra/manual.pdf",
    "extra/NEWS.html",
    "extra/NEWS.txt",
    "extra/readme.html",
    "extra/readme.md",
    "extra/tidycjk.html"
  ],
  "_homeurl": "https://github.com/pursuitofdatascience/tidycjk",
  "_realowner": "pursuitofdatascience",
  "_cranurl": true,
  "_releases": [
    {
      "version": "0.1.0",
      "date": "2026-09-09"
    }
  ],
  "_exports": [
    "cjk_blocks",
    "cjk_char_counts",
    "cjk_compose_jamo",
    "cjk_detect_language",
    "cjk_jamo",
    "cjk_ngrams",
    "cjk_normalize",
    "cjk_order",
    "cjk_pad",
    "cjk_ratio",
    "cjk_romanize",
    "cjk_script",
    "cjk_segment",
    "cjk_segmenters",
    "cjk_sentences",
    "cjk_simplify",
    "cjk_sort",
    "cjk_strip_punct",
    "cjk_summary",
    "cjk_tokens",
    "cjk_traditionalize",
    "cjk_truncate",
    "cjk_width",
    "cjk_wrap",
    "has_cjk",
    "register_cjk_segmenter",
    "to_fullwidth",
    "to_halfwidth",
    "to_hiragana",
    "to_katakana"
  ],
  "_help": [
    {
      "page": "cjk_blocks",
      "title": "The Unicode blocks 'tidycjk' recognises",
      "topics": [
        "cjk_blocks"
      ]
    },
    {
      "page": "cjk_char_counts",
      "title": "Count the CJK characters in a text column",
      "topics": [
        "cjk_char_counts"
      ]
    },
    {
      "page": "cjk_detect_language",
      "title": "Which language is the text written in?",
      "topics": [
        "cjk_detect_language"
      ]
    },
    {
      "page": "cjk_jamo",
      "title": "Decompose and recompose Hangul syllables",
      "topics": [
        "cjk_compose_jamo",
        "cjk_jamo"
      ]
    },
    {
      "page": "cjk_ngrams",
      "title": "Character n-grams",
      "topics": [
        "cjk_ngrams"
      ]
    },
    {
      "page": "cjk_normalize",
      "title": "Apply Unicode normalisation",
      "topics": [
        "cjk_normalize"
      ]
    },
    {
      "page": "cjk_pad",
      "title": "Pad text to a display width",
      "topics": [
        "cjk_pad"
      ]
    },
    {
      "page": "cjk_ratio",
      "title": "What share of the text is CJK?",
      "topics": [
        "cjk_ratio"
      ]
    },
    {
      "page": "cjk_romanize",
      "title": "Romanise CJK text",
      "topics": [
        "cjk_romanize"
      ]
    },
    {
      "page": "cjk_script",
      "title": "Which CJK script dominates the text?",
      "topics": [
        "cjk_script"
      ]
    },
    {
      "page": "cjk_segment",
      "title": "Split CJK text into words",
      "topics": [
        "cjk_segment"
      ]
    },
    {
      "page": "cjk_segmenters",
      "title": "Segmentation engines",
      "topics": [
        "cjk_segmenters",
        "register_cjk_segmenter"
      ]
    },
    {
      "page": "cjk_sentences",
      "title": "Split text into sentences",
      "topics": [
        "cjk_sentences"
      ]
    },
    {
      "page": "cjk_simplify",
      "title": "Convert between simplified and traditional Han",
      "topics": [
        "cjk_simplify",
        "cjk_traditionalize"
      ]
    },
    {
      "page": "cjk_sort",
      "title": "Sort CJK text by pronunciation or stroke",
      "topics": [
        "cjk_order",
        "cjk_sort"
      ]
    },
    {
      "page": "cjk_strip_punct",
      "title": "Remove punctuation from CJK text",
      "topics": [
        "cjk_strip_punct"
      ]
    },
    {
      "page": "cjk_summary",
      "title": "Summarise CJK content in a text column",
      "topics": [
        "cjk_summary"
      ]
    },
    {
      "page": "cjk_tokens",
      "title": "One row per token",
      "topics": [
        "cjk_tokens"
      ]
    },
    {
      "page": "cjk_truncate",
      "title": "Truncate text to a display width",
      "topics": [
        "cjk_truncate"
      ]
    },
    {
      "page": "cjk_width",
      "title": "Display width in terminal columns",
      "topics": [
        "cjk_width"
      ]
    },
    {
      "page": "cjk_wrap",
      "title": "Wrap text to a display width",
      "topics": [
        "cjk_wrap"
      ]
    },
    {
      "page": "has_cjk",
      "title": "Does the text contain CJK characters?",
      "topics": [
        "has_cjk"
      ]
    },
    {
      "page": "to_halfwidth",
      "title": "Normalise fullwidth and halfwidth forms",
      "topics": [
        "to_fullwidth",
        "to_halfwidth"
      ]
    },
    {
      "page": "to_hiragana",
      "title": "Convert between hiragana and katakana",
      "topics": [
        "to_hiragana",
        "to_katakana"
      ]
    }
  ],
  "_pkglogo": "https://github.com/pursuitofdatascience/tidycjk/raw/HEAD/man/figures/logo.png",
  "_readme": "https://github.com/pursuitofdatascience/tidycjk/raw/HEAD/README.md",
  "_rundeps": [
    "cli",
    "dplyr",
    "generics",
    "glue",
    "lifecycle",
    "magrittr",
    "pillar",
    "pkgconfig",
    "R6",
    "rlang",
    "stringi",
    "tibble",
    "tidyselect",
    "utf8",
    "vctrs",
    "withr"
  ],
  "_vignettes": [
    {
      "source": "tidycjk.Rmd",
      "filename": "tidycjk.html",
      "title": "Introduction to tidycjk",
      "author": "Youzhi YuUniversity of Chicago",
      "engine": "knitr::rmarkdown",
      "headings": [
        "Overview",
        "Two layers, one contract",
        "Detection, and the value of refusing to guess",
        "Segmentation",
        "Sentences and n-grams",
        "Sorting",
        "Width, and why your tables are ragged",
        "Normalisation and transliteration",
        "What tidycjk is built on",
        "References"
      ],
      "created": "2026-09-10 10:15:14",
      "modified": "2026-10-05 17:36:49",
      "commits": 4
    },
    {
      "source": "transliteration.Rmd",
      "filename": "transliteration.html",
      "title": "Romanisation, script conversion and kana",
      "author": "Youzhi YuUniversity of Chicago",
      "engine": "knitr::rmarkdown",
      "headings": [
        "Four conversions, four different levels of trust",
        "Hangul jamo: exact",
        "Kana: safe, with two things to know",
        "Simplified and traditional Han: right on characters, blind to vocabulary",
        "Romanisation: depends entirely on the language",
        "Width forms are a separate concern",
        "Normalisation is the fourth axis",
        "References"
      ],
      "created": "2026-09-10 10:15:14",
      "modified": "2026-10-05 17:36:49",
      "commits": 4
    },
    {
      "source": "segmentation.Rmd",
      "filename": "segmentation.html",
      "title": "Segmenting CJK text",
      "author": "Youzhi YuUniversity of Chicago",
      "engine": "knitr::rmarkdown",
      "headings": [
        "The problem",
        "Choosing an engine",
        "\"icu\": dictionary-based, and already installed",
        "\"character\": the honest baseline",
        "Why there is no default",
        "locale does not pick the dictionary",
        "In a pipeline",
        "Registering another engine",
        "gibasa, for Japanese",
        "jiebaR, for Chinese",
        "A caution about argument matching",
        "Comparing engines",
        "References"
      ],
      "created": "2026-09-10 10:15:14",
      "modified": "2026-10-05 17:36:49",
      "commits": 4
    },
    {
      "source": "width-and-layout.Rmd",
      "filename": "width-and-layout.html",
      "title": "Width, padding, truncation and wrapping",
      "author": "Youzhi YuUniversity of Chicago",
      "engine": "knitr::rmarkdown",
      "headings": [
        "One character is not one column",
        "cjk_pad()",
        "cjk_truncate()",
        "cjk_wrap()",
        "Why this is not \"split every N columns\"",
        "One rule that is not universal",
        "Indents",
        "The shared contract",
        "Ambiguous width",
        "References"
      ],
      "created": "2026-09-10 10:15:14",
      "modified": "2026-10-05 17:36:49",
      "commits": 3
    }
  ],
  "_score": 5.079181246047625,
  "_indexed": true,
  "_nocasepkg": "tidycjk",
  "_universes": [
    "pursuitofdatascience"
  ],
  "_binaries": [
    {
      "r": "4.7.0",
      "os": "linux",
      "version": "0.2.0",
      "date": "2026-10-07T08:36:51.000Z",
      "distro": "resolute",
      "commit": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
      "fileid": "https://r2.ropensci.org/41955dfbbac2fb0f9b329669dbd53eee95e0e237d698c71be457e5ec76a3a32f",
      "status": "success",
      "check": "OK",
      "buildurl": "https://github.com/r-universe/pursuitofdatascience/actions/runs/37593080696"
    },
    {
      "r": "4.6.1",
      "os": "linux",
      "version": "0.2.0",
      "date": "2026-10-07T08:36:39.000Z",
      "distro": "resolute",
      "commit": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
      "fileid": "https://r2.ropensci.org/1125b479e5cea82a0f56ac6ed95fadf1c6b69943e454396070c1bce4a76c783c",
      "status": "success",
      "check": "OK",
      "buildurl": "https://github.com/r-universe/pursuitofdatascience/actions/runs/37593080696"
    },
    {
      "r": "4.5.3",
      "os": "mac",
      "version": "0.2.0",
      "date": "2026-10-07T08:46:35.000Z",
      "commit": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
      "fileid": "https://r2.ropensci.org/8dcb881e577aa747506356255daa17fbdbcd19384a7279dd38508540e45cde44",
      "status": "success",
      "check": "OK",
      "buildurl": "https://github.com/r-universe/pursuitofdatascience/actions/runs/37593080696"
    },
    {
      "r": "4.6.1",
      "os": "mac",
      "version": "0.2.0",
      "date": "2026-10-07T08:57:56.000Z",
      "commit": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
      "fileid": "https://r2.ropensci.org/31acea72d91196f9d31f8a882a381d74f328696f88726e6a7f48b6048c2ab12a",
      "status": "success",
      "check": "OK",
      "buildurl": "https://github.com/r-universe/pursuitofdatascience/actions/runs/37593080696"
    },
    {
      "r": "4.6.0",
      "os": "wasm",
      "version": "0.2.0",
      "date": "2026-10-07T08:36:49.000Z",
      "commit": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
      "fileid": "https://r2.ropensci.org/9db8e973df83152b025516d3a1e91b3866449926a29d3e7e5832b1d891d699e1",
      "status": "success",
      "buildurl": "https://github.com/r-universe/pursuitofdatascience/actions/runs/37593080696"
    },
    {
      "r": "4.7.0",
      "os": "win",
      "version": "0.2.0",
      "date": "2026-10-07T08:30:15.000Z",
      "commit": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
      "fileid": "https://r2.ropensci.org/583c44218f54374768d85505c5ec4d7c13e53eb62dd4ae825712653ab80264b2",
      "status": "success",
      "check": "OK",
      "buildurl": "https://github.com/r-universe/pursuitofdatascience/actions/runs/37593080696"
    },
    {
      "r": "4.5.3",
      "os": "win",
      "version": "0.2.0",
      "date": "2026-10-07T08:30:07.000Z",
      "commit": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
      "fileid": "https://r2.ropensci.org/648e062be02a139b88e02e45db036379c03423bbf15fa1d750299d4a834afc9c",
      "status": "success",
      "check": "OK",
      "buildurl": "https://github.com/r-universe/pursuitofdatascience/actions/runs/37593080696"
    },
    {
      "r": "4.6.1",
      "os": "win",
      "version": "0.2.0",
      "date": "2026-10-07T08:30:24.000Z",
      "commit": "6adddca5cb50ac5bf581f6440f301951a8a043a3",
      "fileid": "https://r2.ropensci.org/870c731549bf539b97bc3cdc5fa0c5eb8714960a234e0dc4247ef55fd42351c6",
      "status": "success",
      "check": "OK",
      "buildurl": "https://github.com/r-universe/pursuitofdatascience/actions/runs/37593080696"
    }
  ]
}