No new verbs. emoji_context() returns its answers a great deal faster, and
emoji_collocations(), which answered a non-Latin-script corpus wrongly in a
non-UTF-8 locale, now answers it the same in every locale. A source reading
before submission then found eleven smaller defects, each a wrong,
order-dependent or crashing answer on an input the documentation already
covered: a user-supplied lexicon, a one-row input, a count past integer range,
where = "final" in the incongruity profile, trailing Unicode whitespace under
policy = "strip", and the C-locale order under one dplyr option. They are
listed below, each with a test. The rest of the release is documentation and
the infrastructure that checks it.
The bundled crosswalks track Unicode emoji 16.0, via emoji 16.0.0.
Re-generating them from data-raw/crosswalks.R against that release reproduces
the shipped data exactly, so the catalogue is current.
emoji_context() was still quadratic in the emoji per row, and
emoji_collocations() inherited it. 0.4.0 bounded the tokenising to the
window, which was half the cost. The other half is that substr() on a
multi-byte string rescans from its first byte to reach a character offset,
so cutting one window still cost as much as the text before it; and on a row
whose masked text is all spaces -- which is what a row of nothing but emoji
becomes -- no bounded slice ever yields a token. That is the shape a chat or
reaction corpus is full of.
Each row now gets one index of its code points and token boundaries, and
every window is read off it. 3200 emoji in one row went from 7.1s to about
a tenth of a second, and the verb is flat in the emoji per row: the same
3200 spread over 320 rows costs the same tenth, where the two were more than
twenty times apart before. A realistic mixed row and unit = "char" are
each around three times faster. A row holding a single emoji keeps the old
path, which is still cheaper there, so nothing regresses on short texts.
The output is unchanged, and checked to be: byte-identical on 13977
occurrences across eight window/unit combinations, punctuation,
no-break and ideographic spaces, ZWJ sequences and keycaps included.
emoji_collocations() reported no collocations at all for a
non-Latin-script corpus in a non-UTF-8 locale. It trimmed leading and
trailing punctuation off each context word with [[:alnum:]], which
resolves through the C library: under LC_CTYPE=C nothing outside ASCII is
alphanumeric, so a Hebrew, Arabic, Thai, Devanagari or CJK word was trimmed
away to nothing and then dropped for being empty, and "cafe" spelled with an
acute came back as caf. The same corpus reported its collocations normally
in a UTF-8 session, which is what kept this out of sight: the answer moved
with the session rather than with the data.
The trim now reads Unicode's own tables through PCRE properties, and a token
holding no letter and no digit is not a word. Byte-identical to 0.4.0 on the
bundled corpus in a UTF-8 locale, and identical between en_US.UTF-8 and
LC_ALL=C. One deliberate difference: a superscript digit is now kept,
where [[:alnum:]] cut it.
The same verb counted two spellings of a Cyrillic or Greek word as two
words in a non-UTF-8 locale. Its case fold used an explicit chartr()
table precisely so tolower()'s locale rules could not reach it, but the
table covered only ASCII and Latin -- what the catalogue needs, where this
verb folds the user's text. The table now covers every script in the Basic
Multilingual Plane with a 1:1 lower-case mapping, 1158 pairs in all, derived
from tolower() rather than written by hand and reproducing it exactly in a
UTF-8 session. The fold is also faster than before, because it now tests for
a non-ASCII byte and uses a 26-pair table when there is none.
It stops at the Basic Multilingual Plane on purpose: wchar_t is 16 bits
on Windows, so
chartr() and tolower() work on UTF-16 code units there and mangle a
character outside it. Astral scripts are left to the trailing tolower(),
exactly as before this table existed.
Multi-character mappings stay absent: the German sharp s and the Greek
iota-subscript capitals lower-case to more than one code point, which
chartr() cannot express and tolower() does not do either.
That is three bugs of one shape, after tolower() in the fold and
iswspace() in the whitespace class, so a test now scans the package for
every locale-dependent regex class rather than waiting for a fourth.
A data frame with duplicate column names failed with tibble's message
rather than one of ours. A base data.frame may carry two columns of the
same name; a tibble may not, and the verbs that add columns return a tibble.
The conversion failed with "Column name a must not be duplicated. Use
.name_repair to specify repair", which names an argument no verb in this
package has and an internal, repaired_names(), the user cannot reach, and
never says which verb raised it. It was the one error here that did not tell
the user what to do. 0.4.0 had already fixed the dangerous half of this, where
the duplicated column is the text column and [[ would silently read
whichever came first; what was left was the benign half, which matters only
because the verbs that add columns return a tibble. The check now happens
where the frame becomes a tibble, so one message serves every affected verb,
and it is worded like its sibling: same problem, same remedy. The verbs that
build their own output (emoji_summary(), emoji_frequency(),
emoji_dfm() and the rest) never had the problem and still accept such a
frame.
emoji_incongruity_profile(where = "final") credited glyphs a row was
never scored on. With where = "final" a row is scored on the run of
emoji that ends it, but the profile gave the row's gap to every emoji in the
row and counted all of them in n, its "scored occurrences". A glyph that
opened messages and never closed one could top the table on gaps computed
without it. Only the glyphs a row was scored on are credited now, and
where = "all" is unchanged.
A registered lexicon could be read through the wrong column. The
scoring verbs keyed a registered table on their own by, which defaults to
"emoji", so a lexicon registered with by = "glyph" that also carried a
column named emoji (a label, say) was keyed on the label and scored
nothing, silently, in emoji_score(), emoji_sentiment() and
emoji_emotion(). A registered lexicon is now always read through the
column it was registered with, which is what ?emoji_score already
promised. The same page said score is ignored for a registered lexicon;
it is honoured, as the page's own error advice assumed, and now says so.
Two rows for one emoji, one scored and one NA, gave an answer that
depended on their order. A lexicon may list both spellings of an emoji
when they agree, and the duplicate check ignores NA while comparing, but
the lookup read whichever row came first: the qualified heart scored 0.5 or
nothing depending on which spelling sat on top. Each emoji now takes its
non-NA value (per dimension, for an emotion lexicon), so row order cannot
matter. A real disagreement is still refused.
emoji_emotion() counted an emoji as scored when it had nothing to be
scored on. A custom lexicon row whose every dimension was NA, or whose
only value was infinite and dropped with a warning that the emoji "count as
unscored", still counted towards .emoji_n_scored while every score in the
row stayed NA. It no longer does, which matches what emoji_score() and
emoji_sentiment() report for the same table.
emoji_emotion_label() could label a row from a column it never
computed. It picked the .emoji_<emotion> columns out of its result by
name, which also caught any the input still carried from an earlier call,
so emoji_emotion(df, text) |> emoji_emotion_label(text, lexicon = own)
with own scoring only joy came back "fear" from the bundled profile.
The label is now taken from exactly the dimensions the call scored.
emoji_emotion() named its columns on a one-row input. Each of the
eight .emoji_<emotion> columns came back as a named vector (joy = 0.69)
on a single row and as a plain one on any other number of rows, because R
keeps a dimension name when a one-row matrix drops to a vector. The test
that checks every verb for named columns only ever used four rows; it now
runs on zero, one, two and four.
A count past integer range crashed three verbs.
emoji_ngrams(n = 1e10), emoji_context(window = 1e10) and
emoji_collocations(window = 1e10) passed validation and then failed on R's
"NAs introduced by coercion to integer range", and a window of 1e9,
which stays in range, still overflowed the window arithmetic. A large count
now means "all of it": no n-gram is longer than any row, and the window
covers the whole text.
emoji_sanitize(policy = "strip") trimmed only ASCII whitespace off the
ends. It trimmed with trimws(), whose default set is ASCII, so a row
ending in an emoji and an ideographic space came back with a trailing space
and the ideographic one. The ends are now trimmed with the package's own
whitespace set, the one ?emoji_ratio describes and every other verb
splits and tests text with.
The C-locale order the help pages promise now holds under
options(dplyr.legacy_locale = TRUE). Eleven verbs sort their glyphs or
words through dplyr::arrange(), which follows that global option into the
session's collation; with it set, under an English locale, every one of
them returned its ties in a different order. They now ask for the C locale
explicitly.
emoji_score() did not validate score. A vector reached an internal
%in% and failed as R's "the condition has length > 1", naming neither the
argument nor the verb. It is now refused like every other string argument.
emoji_lexicons() listed a registered lexicon's text columns among its
dimensions, columns emoji_score(score = ) then refuses. Only the
numeric and logical columns are listed now.
emoji_categorize() and emoji_ngrams() were the last two verbs the
pre-release audit listed as pooling grouped data silently. Neither does.
emoji_categorize() answers per row and carries the grouping through
untouched, so a grouping cannot change what it returns; it drops the rows
with no emoji, as its @return says, which is not pooling because dropping
a row cannot merge two groups. emoji_ngrams() returns one row per n-gram
occurrence keyed by .row_number and carries none of the user's columns, so
there is no grouping for it to ignore. Both are pinned by tests now.emoji_position() already states that positions are
logical rather than visual order. A test now holds it there, because it is
the sentence a right-to-left corpus depends on.vignette("reversible-preprocessing"). It runs the
emoji_sanitize() loss ladder rather than describing it, shows the
shortcode round trip surviving skin tones, flags, ZWJ sequences and keycaps,
demonstrates that wrap is part of the reversibility contract (a wrap
without colons restores nothing, silently), and shows how to keep the emoji
signal as feature columns while the text you send to a model holds no emoji.
It also makes the detection limitation concrete: a bare U+2764 carries
text presentation, so it is not detected and survives a policy meant to
remove it.text_score recipe at
eval = FALSE, for tidytext with AFINN and for sentimentr, next to the
word-list scorer it uses to stay runnable.?emoji_sanitize and
?emoji_token_cost all point at the new article.reviews does not have. Both blocks are
self-contained now, and a test evaluates every README chunk in document
order so an unevaluated one can only lean on an evaluated one above it.emoji_seasonality()'s denominator is, and that emoji_context()'s
character windows are code points and can cut a grapheme.emoji_collocations() cannot find a collocation in an unsegmented script,
and said nothing about it. 0.5.0 fixed the character class that dropped
non-Latin context words, which makes a Chinese, Japanese, Thai, Khmer, Lao or
Burmese corpus a supported input. What the verb does not do is segment one:
its tokens are runs of non-whitespace, so a clause in such a script is a
single token. Measured: three rows of Chinese meaning "the weather is good
today", "my mood is good today" and "going to Beijing tomorrow" yield three
tokens, each a whole clause, each with n = 1, so nothing clears the default
min_n and the result is empty. The same three sentences in English yield
good, is and today at n = 2. Both help pages now say so, and say that
the segmentation has to happen before the text reaches the verb, since the
result cannot be passed to a tokeniser to repair it. emoji_context()'s
unit = "char" sidesteps the question and is the better default on such
text.?emoji_sanitize enumerates every way the round
trip can fail through wrap, and said nothing about the input text:
text_to_emoji() cannot tell a :shortcode: token the sanitiser wrote from
one the text always held, so a Slack, Discord or GitHub export where people
type :thumbsup: comes back with an emoji it never contained. The count
moves with it, from 0 to 1 on "nice work :thumbsup:", and it compounds
with real emoji rather than replacing them. The damage is bounded at one
pass, and the colon shapes that are not shortcodes ("meet at 10:30",
"ratio 3:4") were already safe and stay so. Both help pages now state it
from their own side, and four tests pin the inflation and the sentences.?emoji_trend rendered its definition of share as part of a list
item. The Markdown that roxygen renders folds an unindented line directly
after a list item into the item, so the paragraph defining share appeared
as the tail of the emoji_seasonality() bullet. It is its own paragraph
again, and a test scans every roxygen block for the same shape.?tidyEmoji said 262 of the regional-indicator pairs are real flags, and
the figure is 259. The page had counted in the three subdivision flags
(England, Scotland, Wales), which are tag sequences rather than pairs, and
the test pinning it summed the two subgroups without asking which rows are
pairs.?emoji_unicode_releases described the emoji series as starting at 1.0,
while its own table begins at 0.6 and 0.7, the labels Unicode gives the
emoji of Unicode 6.0 and 7.0.?emoji_lexicons gave type as sentiment or emotion, where a registered
lexicon reports "custom", and now says which columns dimensions lists.citation("tidyEmoji") asked for the sentiment lexicon to be cited by
users of three of the nine verbs that read it, and the emotion lexicon by
users of one of three. The header now names every one, and a test derives
the list from the package's own call graph."shortcode" can leave an
emoji behind, when the reason is a glyph too new to name. The introduction
said the four datasets are regenerated "from the current Unicode emoji
list"; only the two crosswalks are, and the scripts live in the source
repository rather than in the installed package.man/ is now verified to be what roxygen2 generates. 0.4.0's help pages
were produced by a stand-in converter because no R was available in the
authoring environment, which left "re-run the real thing and commit the
difference" outstanding for two releases. Running roxygen2 7.3.2 over the
tree reproduces 49 of the 51 .Rd files byte for byte and NAMESPACE
exactly; the only two that differ are the two edited in this release. The
debt is discharged rather than carried again.inst/CITATION hardcoded a version fallback that had already slipped a
release. The file derives the publication year rather than writing it out,
with a comment explaining that a fixed year "goes stale silently the first
time the release slips past New Year". The same reasoning applies to the
version, which was written out as "0.4.0" while the package was 0.5.0. It is
now derived from the installed package, with no version literal left in the
file, and two tests hold it there: one that the cited version matches
packageVersion(), one that the file still renders when meta is
unavailable. citation("tidyEmoji") was already correct in normal use, since
the fallback only fires when meta is missing; this removes the trap rather
than a visible defect.cran-comments.md had gone stale on one of its own figures, claiming 170
entries in inst/WORDLIST where the file holds 166. It is the one file in the
repo whose entire purpose is claims a reviewer can reproduce, so the count is
now derived in test-invariants.R alongside the marked-UTF-8 breakdown and
the skip inventory, and cannot drift again unnoticed.R/, man/ and tests/. One comment
in test-regression-0.2.1.R held the last non-ASCII byte in the suite, in a
cross-reference to a planning document that is not shipped and no longer has
the section it named. The comment now describes the defect each block pins.None of this changes the package, but all of it was promised in an earlier release and kept slipping.
data-raw/benchmark.R makes the timing baseline repeatable instead of a
number measured once in a session, and it grew a second axis that holds the
emoji count constant while varying how densely they are packed -- the axis
the row-count axis cannot see, and the one the fix above is about. Its ratio
now compares the densest corpus against the cheapest, which is the direction
the defect moves it; the previous max / min was dominated by per-row
overhead and scored every verb near its threshold on a healthy build.emoji-translate.R at 97.2%, which is why the job's floor is on
the total rather than per file. The floor is enforced inside the job because
the codecov upload needs a CODECOV_TOKEN secret and is allowed to fail
without one: a job whose only output is an upload nobody accepted reports
success for nothing. Reading the uncovered expressions rather than counting
them found three pieces of dead code, now gone.LC_COLLATE=C and en_US.UTF-8, and a job running
it under LC_ALL=C. Both halves of the locale have cost this package bugs
and every one of them was green on every other flavour. Each job refuses to
run if the locale it asked for did not take effect.knitr
tangles out of the introduction vignette, so a checking machine that merely
lacked a Suggests package failed. It recognises both now.spelling is wired into tests/spelling.R (non-failing, skipped on CRAN)
and into the weekly job, where it does fail. The check reports 0 unknown
words across the help pages, both vignettes, README.md and NEWS.md;
inst/WORDLIST is 167 entries.testthat floor was wrong. DESCRIPTION asked for
>= 3.0.0 while the suite has called expect_no_error() and
expect_no_warning() since 0.4.0, both of which arrived in 3.1.5. It now
declares >= 3.1.5, and the table in test-invariants.R that couples each
declared floor to the argument needing it covers testthat too.This release delivers the first wave of the feature roadmap filed as issue #5: the items that are cheap, research-grounded, and need no new dataset and no new dependency. Most of them are recombinations of machinery the package already had -- the Novak lexicon's annotation counts, the reference table's Unicode version column, the grapheme-aware locator -- read out in a way no R package exposed before.
Miller et al. (2016) found that readers of the same rendering disagree about
whether an emoji is positive, neutral or negative roughly a quarter of the
time. The bundled Emoji Sentiment Ranking keeps the raw
negative/neutral/positive annotation counts behind its collapsed score,
so that disagreement was already inside the package as an empirical
distribution. It is now a number.
emoji_ambiguity() reports per-glyph annotation shares and one of four
disagreement statistics -- Shannon entropy (the default), gini,
neutral_share or ci_width -- with a rank over the whole lexicon.emoji_risk() is the per-row version: .emoji_ambiguity_mean,
.emoji_ambiguity_max and .emoji_n_ambiguous.emoji_flag_ambiguous() is the content-QA shortlist: the emoji in your
corpus most likely to be misread.emoji_sentiment(se = TRUE) adds .emoji_sentiment_se, so a glyph annotated
eight times no longer carries the same authority as one annotated eight
thousand times.emoji_context() returns one row per emoji occurrence with a window of the
surrounding text, in words or characters. All other emoji are blanked out of
the window, and character offsets stay exact.emoji_collocations() aggregates those windows into an emoji-word table
scored by pointwise mutual information, shaped like widyr::pairwise_count()
output. Corpus-derived senses have neither the licence problem nor the
staleness problem of an imported sense inventory.emoji_trend() counts emoji per period ("day", "week", "month",
"quarter", "year") and returns a complete period-by-emoji grid, so a
trend line does not silently skip its zeros.emoji_turnover() reports vocabulary churn between consecutive periods:
jaccard, n_new, n_lost, n_core.emoji_version_profile() breaks a corpus down by the Unicode emoji version
that introduced each glyph, and emoji_adoption_lag() compares first use in
the corpus with the release date. Both come almost free from the version
column the reference table already carries.emoji_seasonality() aggregates by month, weekday or hour, returning every
level of the cycle including the empty ones, with fixed English labels so a
script's output does not change with the machine that runs it.emoji_unicode_releases() is the version-to-release-date lookup behind the
two verbs above. It is a function rather than a bundled dataset: at a few
dozen rows it belongs beside the code that uses it.emoji_incongruity() measures the signed gap between a row's text sentiment
and its emoji sentiment -- the sarcasm feature in NLP, the (in)congruence
variable in marketing research. emoji_congruence() is the same engine under
the marketing framing; emoji_incongruity_profile() reports which glyphs go
against the grain of their host text.text_score from
tidytext, sentimentr, vader or a model. Because those live
on incompatible scales, scale has no default -- you have to say how the
two sides were made comparable.NA, never 0, in every new column.as_emoji_type(), emoji_type() and emoji_faceness() recode the Unicode
group and subgroup into face, gesture, person, nature, food,
place, activity, object, symbol, flag and component. The
emotional (face) versus semantic (object) contrast is the key variable in the
consumer-behaviour literature and is now a one-liner.emoji_sanitize() applies one named policy -- "keep", "strip", "name",
"shortcode" or "placeholder" -- to a text column. The capability mostly
existed; the value is an argument that shows up in a script diff and in a
methods section.emoji_token_cost() reports exact .emoji_bytes, .emoji_codepoints and
.emoji_graphemes plus a clearly-labelled .emoji_token_estimate, or the
real count if you pass your own tokenizer.emoji_provenance() puts every version an emoji result depends on in one
row: tidyEmoji, the emoji package, the Unicode emoji version, the size
of the detectable emoji set, and the lexicons.emoji_unicode_version() reports that Unicode version on its own.inst/CITATION now credits the package and the two lexicon papers users have
to cite anyway.This section is long because the release was audited exhaustively rather than spot-checked, and both kinds of finding are recorded: changes you can observe, and verifications that confirmed existing behaviour was already right. The second kind is kept deliberately -- knowing that a figure was re-derived from the data, or that a formula was checked against an independent implementation, is worth as much to the next maintainer as knowing what moved.
Entries whose first sentence is bold are the ones where something was actually wrong and got fixed -- in the package, in its documentation, or in a test that was passing for the wrong reason. There are one hundred and eighteen of them, and reading just those leads gives the release without the verification detail. Not all one hundred and eighteen changed observable behaviour: several record a test that could not have failed, or a figure the documentation quoted incorrectly, which are worth the same prominence because both meant something was unverified.
The whole of this release's polish was audited against the version it
started from, by installing both side by side and comparing 57 verb calls
over the bundled 2000-row corpus. 54 are byte-identical, and the three
that differ are exactly the three intended behaviour changes, in exactly the
places they should be: emoji_emotion() |> emoji_emotion_label() gains the
eight emotion columns and nothing else (every shared column identical, and
the scores equal emoji_emotion()'s own); and
emoji_incongruity(scale = "rank") and "zscore" change on precisely the
373 scored rows, with NA placement untouched, scale = "none" and
emoji_congruence() byte-identical, and the rank-scale mean gap now
0 (-2.1e-17) where it had been -0.0042. Every guard, message and validation
change in this release leaves valid input alone.
The same comparison over 29 adversarial inputs -- list, matrix and
data-frame columns, duplicated and missing column names, a wide frame, a
non-finite score, dd/mm/yyyy dates, an unknown measure, an
emotion-shaped lexicon, NA search queries -- shows no guard was
loosened: nothing that was rejected before is accepted now. Five inputs
that used to produce a plausible-looking result are refused (the list and
data-frame text columns, a bare list to emoji_tokens(), a whole dd/mm/yyyy
column, an unrecognised measure); two now warn and drop instead of
answering from bad data; eight kept erroring but say something useful
instead of surfacing dplyr's, tibble's or match.arg()'s internals; and
twelve are unchanged in value, message and warning alike.
A duplicated column name silently resolved to the first column. The
bare-name fast path added earlier this release tested
nm %in% names(data), which is true even when the name appears twice, and
[[ then returns the first -- so emoji_summary() and emoji_frequency()
answered from whichever column came first while the verbs that convert
data to a tibble failed with tibble's own message instead.
dplyr::select(), which that path replaced, had rejected the ambiguity for
all of them. A data frame really can carry the name twice --
read.csv(check.names = FALSE) on a sheet with repeated headers does it --
and picking one of two columns unasked is a wrong answer that looks right.
All ten verbs tried now report the ambiguity, naming the argument and the
count, for every column argument.
Checked that this release's type guards have not caught anything a caller
might reasonably hold text in: difftime, complex, AsIs, noquote, a
factor with an NA level and a custom-classed double all still work as a
text column, and difftime is still refused as time or text_score
with the type message rather than a coercion.
Verified emoji_context()'s window content, which no test had checked.
The earlier round proved the bounded-slice optimisation agrees with the
package's own naive version -- but two implementations that cut the window
in the same wrong place would agree just as well. It is now compared
against a reference sharing no code with the package: every glyph span is
masked to spaces, the two sides are split by a character-at-a-time
whitespace splitter, and the nearest window tokens are taken. 5160
windows over window values 0, 1, 2, 4 and 9 match exactly, and the
documented promise holds -- with three emoji in a row, the middle one's
window reaches past its neighbours to real words and never quotes the
neighbours. emoji_collocations()'s word list was checked the same way
against an independent tokenisation (400 comparisons, no difference), along
with each rule its documentation states: lower-cased, leading and trailing
punctuation stripped but internal punctuation and digits kept, and a word
counted once per occurrence however often it repeats in the window. No
defect was found in either; both are now pinned.
Pinned the version cluster's behaviour against a future emoji
release, the one way this package can break without anything here
changing. emoji_version_profile() and emoji_adoption_lag() read the
introducing version out of the installed reference table and join it to a
release-date table kept in tidyEmoji's own source; the day Unicode 18 ships,
that label will not be in the table. R CMD check cannot reveal this,
because it runs against today's emoji. Injecting 18.0, the
data-file spelling E18.0, an absurd 99.9 and a missing version now
confirms all four degrade rather than fail: the glyph keeps its row, its
release_date and lag_days are NA instead of invented, share_tokens
still sums to 1, the other glyphs are untouched, and
emoji_unicode_version() follows the new label. Also pinned the release
table itself -- version unique across both numbering series, version_num
equal to the parsed label (string ordering would put 10.0 before 9.0),
no missing dates, and within each series a later version never released
earlier.
Closed the gaps in the grouping contract. Grouping is what makes
group_by() |> verb() |> summarise() work, and it is upheld by two separate
mechanisms: .emoji_as_tibble() passing a grouped_df through untouched,
and .emoji_regroup() re-deriving the indices for the three verbs that
rewrite the text column. Only 13 row verbs were pinned; the seven omitted
ones -- emoji_emotion(), emoji_emotion_label(), emoji_incongruity(),
emoji_congruence(), emoji_sanitize(), emoji_to_text() and
text_to_emoji(), which is all three of the rewriting verbs -- are now
covered too, so a stray ungroup() cannot change semantics silently. The
mirror case is pinned as well: emoji_summary(),
emoji_extract_unnest() and emoji_flag_ambiguous() must not be grouped,
because they do not return the caller's rows and so have no group column to
carry. And .emoji_regroup() is now tested in the case it was written for,
in its sharpest form -- grouped by the text column where
policy = "strip" maps two distinct texts onto one, collapsing three groups
into two and shifting every index. All four rewrites leave each group's
indices pointing at rows that really carry its key, and are
indistinguishable from regrouping from scratch. No defect was found; the
behaviour was already right in all 22 verbs.
A "bytes"-encoded text column failed with R's message, not the
package's. A character vector whose Encoding() is "bytes" is a bag of
bytes R refuses to read as characters: nchar(type = "chars"), gsub(),
tolower() and substr() each stop on one. Emoji detection is built out of
precisely those, so 22 verbs already failed on such a column -- but they
failed with "bytes encoding is not supported by this function", thrown
from inside R, naming no argument, no column and no remedy. Strings arrive
marked this way from readBin()/rawToChar() and from text decoded with
the wrong encoding upstream. The shared text resolver now catches it, so all
22 give one message that names the argument, the column, how many values
carry the mark, and the fix. Coercing was rejected deliberately: a string is
usually marked "bytes" because it is not valid UTF-8, so enc2utf8()
cannot repair it and would substitute replacement characters. Only the
caller knows what the bytes meant, which is why the message says iconv()
and says why enc2utf8() will not do.
The same mark on a string argument is now refused as well -- sep,
placeholder and query, which reached gsub() and tolower() with R's
message. wrap was the worst of them and the only one that did not error at
all: emoji_to_text(format = "shortcode") pasted the template into the
rewritten text, which marked the output column "bytes" too. The verb
returned a perfectly ordinary-looking tibble whose text column would then
stop the next thing that touched it, arbitrarily far from the cause. That
emoji_to_text() never returns a bytes-encoded column is now a test in its
own right. latin1- and unknown-marked text, and factors, are unaffected --
they are readable, and the guard is only about the encoding R will not read.
The collation test had a field that asserted nothing. The snapshot
guarding "no ordered output depends on LC_COLLATE" compared
emoji_pairs()'s output across locales -- but over a fixture whose every
row held a single emoji, and both emoji_pairs() and
emoji_cooccurrence() pair within a row. The field was character(0),
which compares equal to itself in every locale, so the ordering of those
two verbs had never been tested. collocs was nearly as empty: its fixture
was paste("good", ...), so the word column was three copies of "good"
and no permutation of it could differ either. Both now use fixtures that
produce real output, and the test refuses any field that is empty or
constant -- the two shapes that make a comparison vacuous.
The same test only tried C and en_US.UTF-8, which order the fixture
identically, so it could not have caught a collation bug even in a
populated field. It now also runs under Danish, Czech and Estonian
collation, which disagree with C about where a-ring, ch and o-tilde
sort, and it asserts that the fixture words re-order under at least one
locale it reached -- so a fixture that stops being collation-sensitive
fails instead of passing quietly. Coverage went from 16 outputs to 31,
adding emoji_trend(), emoji_turnover(), emoji_seasonality(),
emoji_cooccurrence(), emoji_tokens(), emoji_extract_unnest(),
emoji_type(), emoji_version_profile() and emoji_adoption_lag(). It
runs faster than before despite that (8.9s to 3.9s): the old shape called
emoji_search("hand") three times per locale and six other verbs twice.
Documented why the head of emoji_ambiguity()'s ranking can show something
that is not an emoji. The docs already explained the tie at rank = 1 by
annotation count, which is right as far as it goes -- but three of those
five rows are box-drawing and dingbat characters, not emoji. The lexicon was
built from 2015 tweets and 233 of its 969 rows are absent from the reference
table, so no corpus this package analyses can yield them. They carry 6% of
the lexicon's annotations, and 86% have fewer than 50, so the
n_annotations filter the page already recommends removes 200 of the 233 as
a side effect. Every figure is pinned twice, from the data and from the
rendered Rd; the coverage figures on [emoji_sentiment_lexicon] (736 in the
reference table, 233 not, 3790 distinct keys, "about 19%") were re-derived
and all hold.
A non-numeric score column registered happily and then scored nothing.
register_emoji_lexicon() resolves the score column at registration
precisely so a bad table fails there rather than later -- but it checked
only that the column existed, not that it held numbers. A score column
of text (which is what a stray "NA" or a decimal comma makes of a whole
column) therefore registered without complaint, and at scoring time reached
mean(), which returns NA with R's own "argument is not numeric or
logical" warning. Meanwhile .emoji_n_scored still counted the emoji as
scored, so the row claimed a score it did not have -- contradicting that
column's documented meaning, where 0 is exactly "had emoji the lexicon
could not score". Both the registration path and the bare-data-frame path
now refuse it and say what to look for. A logical column still works, and a
genuinely NA numeric score is still counted as unscored.
Two lexicon rows for one emoji let the row order pick the score.
Spellings that differ only by a variation selector share one code-point key,
so a table listing both U+2764 and U+2764 U+FE0F holds one emoji twice.
The lookup took whichever came first: swapping two rows of the caller's own
table changed .emoji_score from 0.5 to -0.5, silently -- the same
failure as the duplicated column name fixed earlier this release. Identical
scores are the normal case and still collapse quietly, and an NA beside a
value is not a disagreement; only genuinely conflicting scores are refused,
naming the key. Neither bundled lexicon has a duplicated key at all, which
is now asserted so the guard cannot start reaching them.
by is validated as a single string. It reaches %in%, so
by = c("emoji", "other") surfaced R's own "the condition has length > 1"
from inside register_emoji_lexicon() and emoji_score(), naming neither
the argument nor the verb.
Checked the suite the way a suite should be checked: by breaking the package
and seeing whether it notices. Nine mutations were applied one at a time to
the numeric and ordering core -- weakening emoji_position()'s
short-text guard, making whitespace-only text score NA per token, dropping
emoji_dfm()'s glyph tiebreaker, shifting its tf-idf denominator, moving
the Monday the week buckets to, moving the month a quarter starts on,
ordering emoji_unicode_releases() by version string instead of number,
turning off fixed = TRUE in emoji_search(), and removing the
"bytes"-encoding guard. All nine were caught, by 1 to 92 failing tests
each; none survived. Three were caught by a single test apiece, which is the
thin margin worth knowing about rather than a defect.
Verified emoji_context(unit = "char"), the half of that verb an earlier
round left alone. The word unit was checked against a reference sharing no
code with the package; the character unit has its own bounded-slice
optimisation and its own "enough characters after trimming" fallback, and
neither had been checked against anything but itself. 4135 windows over
window values 0, 1, 3, 7 and 25 match exactly, including the doubling
fallback that whitespace-heavy input triggers. Also pinned that a masked
neighbour can pad a character window but can never appear in it.
Tied together top_n_emojis()'s two naming branches, which had nothing
connecting them. emoji_name comes from emoji_frequency()'s shortcode
when duplicated = FALSE and from a many-to-many join onto
emoji_unicode_crosswalk when TRUE, so a change to either could leave
them disagreeing about what an emoji is called -- and only the expanded
branch was exercised anywhere. Over a corpus holding every catalogued glyph,
the single name is the first of the expanded names for all 4830 of them,
and neither branch leaves a glyph nameless: 189 reference rows carry no
alias, but canonicalisation matches on the codepoint key, so such a glyph
borrows the alias of its other spelling. Also pinned that n cuts on
distinct emoji rather than rows, so a glyph with several aliases cannot eat
another glyph's slot, and that n = 0 gives a typed zero-row tibble in both
branches.
Verified the declared R (>= 4.1.0) floor exactly rather than by
inspection: every base-package function the code calls -- collected by
walking the syntax tree of R/ and tests/, 443 distinct call names --
exists in a real R 4.1.0 installation. The only name that resolves on 4.4
but not 4.1 is %||%, which entered base R in 4.4.0 and which this package
has always defined for itself.
emoji_turnover() could not tell new from lost. Mutation testing
found this one: swapping n_new to compute setdiff(a, b) instead of
setdiff(b, a) -- so that "new emoji" reports the lost count -- passed
the entire suite. Two tests looked like they covered it and neither could.
One asserts n_new == 1 and n_lost == 1; the other checks each against
set arithmetic done by hand, but over a fixture whose first period pair
gains exactly one glyph and loses exactly one. Both numbers are 1, so no
permutation of them is observable. The assertions were right; the fixtures
had no teeth -- the same failure as the empty snapshot field fixed above,
in a subtler form. There is now a deliberately asymmetric case (three
glyphs arrive, none leaves), the reverse of it, an assertion that the
fixture can distinguish the two, and the identity that fixes the
direction: n_new + n_core is the later period's vocabulary and
n_lost + n_core the earlier one's. The implementation was correct
throughout; only the test was blind. Re-running the mutation now fails four
assertions.
emoji_provenance() mis-described its own headline number.
n_emoji was documented as "the size of the detectable emoji set" but
reports nrow(emoji_reference()) -- rows of the reference table, which are
spellings. With emoji 16.0.0 that is 5042 rows carrying only 3790
distinct code-point keys, because an emoji whose presentation can be
selected appears both with and without U+FE0F, and 212 of the 5042 are
not detectable in text as written. This is a function built to be pasted
into a methods section, so "5042 detectable emoji" is a number that would
have gone into papers overstating the vocabulary by 1252. The value is
unchanged -- it is the table's size, which is the right thing for a
provenance row to record -- and the documentation now names the quantity,
points at length(unique(emoji_reference()$key)) for the count of distinct
emoji, and notes that the lexicon strings count rows the same way. Also
established the fact that makes the overstatement harmless rather than a
defect: no emoji is lost to an undetectable spelling, because every one of
the 3790 keys is reachable through at least one spelling that is detected.
emoji_version_profile() could not tell type shares from token shares.
Found by extending round 79's search from one column pair to every pair a
verb could swap: exchanging the expressions behind share_types and
share_tokens passed the whole suite. The only assertion either column
ever had was sum(share_tokens) == 1 -- and both columns sum to 1, so
the swap was invisible. There is now a fixture where the two genuinely
differ (one glyph three times, another once, so share_types is 0.5 where
share_tokens is 0.75), each column checked against its own count, and an
assertion that the fixture separates them. Re-running the swap fails four
assertions. As with n_new/n_lost, the implementation was right and only
the test was blind.
The same sweep exchanged nine other pairs -- .emoji_first/.emoji_last,
.emoji_per_char/.emoji_per_token, .emoji_only's polarity,
.emoji_n_typed/.emoji_n_face, n_types/n_tokens,
release_date/first_seen, n_texts/n_with_emoji, p_neg/p_pos and
emoji_summary()'s two counts -- and every one was caught. The
seasonality pair was the closest call: its own fixture reads 2 for both
columns, and the month where they differ is asserted for neither, but two
unrelated tests fail on the swap anyway. It now has a fixture that
separates them directly rather than relying on that.
Audited every guard in the package by shadowing stop inside its own
namespace, which makes one suite run log every error the tests actually
trigger. 53 of the 54 guards fired; the one that never did was
emoji_sentiment()'s fallback for a lexicon it cannot use, and reaching it
exposed two message defects rather than one.
emoji_sentiment(lexicon = "emotag1200") said the name was invalid. The
bundled emotion lexicon is the only input that reaches that guard, and the
message was "lexicon must be 'novak2015', a registered lexicon, or a data
frame" -- telling a user whose name is a perfectly valid bundled lexicon
that it is not, and pointing nowhere. It now says the lexicon is an emotion
lexicon, that this verb needs one score per emoji, and names both
[emoji_emotion()] for the per-emotion profile and
emoji_score(lexicon = ...) for the mean over its dimensions -- the answer
emoji_score() already gave its own callers. emoji_emotion()'s reciprocal
case now mirrors it instead of only listing what it wants.
A guard offered an option it rejects. .emoji_lexicon_lookup()'s type
message read "lexicon must be a name (string), a data frame, or NULL for
the default" -- and NULL fails that very guard, since is.character(NULL)
is FALSE. No verb accepts or documents lexicon = NULL. The message now
describes what is taken and names what was passed. Tightening it to a
single name also fixed two messages that were R's rather than the
package's: a length-2 name reached %in% inside an if and produced "the
condition has length > 1", character(0) produced "argument is of length
zero", and NA_character_ fell through to a misleading "Unknown lexicon
NA".
Two guards remain that no test can reach, and deliberately so: the final
else in each of emoji_sentiment() and emoji_emotion().
.emoji_lexicon_lookup() returns one of four shapes -- a data frame, or a
record of type sentiment, emotion or custom -- and both verbs now handle all
four, so those branches are unreachable defence against a fifth rather than
untested behaviour.
The deprecated duplicated_unicode was more permissive than its
replacement, and silently so. It converted with
isTRUE(x) || identical(x, "yes"), which collapses everything it does not
recognise to FALSE. So duplicated_unicode = "TRUE" and
duplicated_unicode = 1 -- both entirely plausible for an argument that
once took a string -- returned the opposite of what was asked, without a
word, while NA, "Yes", character(0) and a length-2 vector all passed
unnoticed too. The current duplicated rejects every one of them through
.emoji_check_flag(), so migrating to the deprecated spelling was a
downgrade in safety. It now accepts exactly the four values it ever meant --
TRUE, FALSE, "yes", "no" -- each still giving what the modern
spelling gives, and errors on anything else naming the replacement. Found by
extending the guard audit to warnings: both bare warning() sites fire in
the suite, and probing the deprecation surface instead turned this up.
Fifteen enum arguments answered a typo by naming a variable the caller
never wrote. match.arg() reports its own formal, so
emoji_context(unit = "words") said "'arg' should be one of "word",
"char"" and emoji_sanitize(policy = c("keep", "strip")) said "'arg'
must be of length 1". emoji_turnover() had already been given a
hand-rolled check for precisely this, with a comment saying it exists to
name measure "rather than match.arg()'s own 'arg'" -- and the fix was
never carried to the other fifteen call sites, the same
written-where-noticed-and-never-grepped pattern that left seven aggregators
without a grouped-input guard in 0.3.0. They now share
.emoji_match_arg(): unit, policy, format, weighting, period,
by (twice), measure (five verbs), method, where and scale all name
themselves and list their options.
That conversion changes no accepted value. match.arg()'s behaviours are
preserved deliberately and pinned: a value identical to the whole choice
vector still means "no value supplied" and takes the first (which is why
emoji_context(unit = c("word", "char")) is accepted and
emoji_sanitize(policy = c("keep", "strip")) is not), exact matches win,
and unambiguous prefixes still resolve, so by = "mon" keeps working while
an ambiguous one is refused rather than guessed. The accept/reject matrix
over full values, prefixes, wrong case, unknown values and length-2 vectors
is unchanged across all fifteen.
Converting them meant writing each choice set at the call site, since
match.arg(x) reads it from the formal and a helper cannot. That is the one
risk the change introduces, so a test now derives every choice set from the
function's own formals and requires the error message to list exactly it --
no more, no fewer. (emoji_incongruity()'s scale is deliberately
defaultless, so it has no formal to derive from and is covered by the
message test instead.)
Closed the last gap in the grouping story. .emoji_warn_grouped() exists
because "silently ignoring a grouping turns a per-group question into a
global one", and an earlier round pinned which verbs keep a grouping and
which drop it -- but not whether the droppers say so. Three do not:
emoji_ngrams(), emoji_extract_unnest() and emoji_context() discard the
grouping, and the grouping column with it, without a word. All three are
defensible, and for a reason worth stating: each returns .row_number
instead of the caller's columns, and none of them pools rows, so there is no
per-group answer being quietly globalised -- unlike the thirteen verbs that
do pool and therefore warn. That distinction was recorded nowhere. Of 37
help pages exactly two mentioned grouping, both promising it is kept. All
four reshapers now say which side they are on, and point at .row_number
as the way to join your columns back.
The three-way classification is now a contract rather than an accident:
every data-first verb must either keep the grouping (and the column it
names), or warn and ignore it, or be one of the three named reshapers that
return .row_number and none of the caller's columns. The silent set is
asserted to be exactly those three, so a new verb cannot join it by
accident; both other groups are asserted non-empty so the classification
cannot go vacuous; and a verb that warns is required not to return the
grouping, which would be the confusing middle case of warning about
something it went on to keep.
Verified the statistical core against the raw annotation counts rather than
against itself, since these are numbers that reach papers. All four
ambiguity measures recompute exactly over all 969 rows -- Shannon entropy in
nats with 0 log 0 taken as 0, Gini impurity, the neutral share, and the
standard error from Var(X) = E[X^2] - E[X]^2 for X in {-1, 0, 1} --
and every documented figure on the page holds: the median glyph has 18
annotations, 69.0% have fewer than 50, 11 of the top 20 do, the five rows
tied at rank = 1 carry 3, 3, 3, 9 and 15, and the entropy/count Spearman
correlation is 0.556, which is the "0.56" the page quotes. gini's
documented 2/3 ceiling is now asserted alongside entropy's log(3) one,
which already was.
Pinned the invariant that makes emoji_sentiment(se = TRUE) meaningful: the
score map and the standard-error map cover exactly the same emoji, with no
NA in either. The row mean drops glyphs the score map cannot score and
the standard error drops glyphs the se map cannot, so if those two sets
ever diverged the reported error would be the error of a mean nobody
computed. .emoji_n_scored is now checked to equal the standard error's own
denominator over a many-glyph corpus, and the propagation
(sqrt(sum(se^2)) / n_scored) re-derived independently. Nothing was wrong;
nothing had been checking.
emoji_type()'s .emoji_type is NA for two different reasons -- a row
with no emoji, and a row whose every emoji the recode cannot type -- and the
page documented only the first. The recode maps the ten Unicode groups the
catalogue currently uses and starts from NA, so a glyph in a group added
to Unicode after your emoji package was built has no type. It is the
same conflation [emoji_categorize()] already describes for
.emoji_category, and the fix is the same: say both, and say how to tell
them apart -- [emoji_faceness()]'s .emoji_n_typed is NA when the row had
no emoji and 0 when it had emoji that could not be typed.
Pinned the recode's coverage and its behaviour on the untypable path.
Nothing falls through today: all 5042 catalogue rows type, every type
produced is one of the eleven declared levels, and all eleven are reachable
so none is dead. An unknown group yields NA rather than erroring or
landing silently in a catch-all. Forcing a glyph untyped confirms the rest
of the contract holds around it: the row keeps its other emoji's type
instead of being dropped, .emoji_n still counts the glyph because it is
still an emoji, .emoji_n_typed falls to match, and .emoji_faceness is
the share among typable emoji -- NA rather than a division by zero when
none of them is.
Swept the "NA for two reasons" class across every verb instead of meeting
it one at a time. Over a fixture holding all three causes -- a row with no
emoji, a row whose emoji the lexicon or the recode cannot use, and an NA
text -- the design turns out uniform and sound: every answer column is
NA for all three, and every count column (.emoji_n_scored,
.emoji_n_typed, .emoji_n_ambiguous) is 0 for "had emoji I could not
use" and NA for "had no emoji". That is the contract
[emoji_sentiment()] states, and [emoji_score()], [emoji_emotion()] and
[emoji_risk()] each restate it. Two verbs carrying the same count column
never said it: [emoji_faceness()] and [emoji_incongruity()]. Both do now,
and the whole pattern is pinned rather than left as a coincidence of six
separate implementations.
A documentation-pinning helper reported gaps that did not exist. Rd
prose is line-wrapped, so a sentence written as one line in the roxygen
source arrives from tools::Rd_db() with a newline in the middle -- and a
fixed = TRUE match for that sentence fails. Three of this round's own
assertions failed exactly that way, on pages whose wording was already
correct. The test helper gained rd_flat(), which collapses whitespace
first, and the thirteen existing assertions that matched a multi-word phrase
against unflattened text were converted to it. None of them was failing --
each passed only because its phrase happened not to straddle a break, so any
edit to the surrounding prose could have turned a correct page into a red
test. The pass count is unchanged by the conversion, which is the point.
The suite's dependency on one emoji release was undeclared and
unexplained. About seventy counts derived from the catalogue are pinned
here -- 5042 rows, 3790 distinct keys, 969 lexicon rows, 736/233, 212
undetectable spellings, 1252 carrying U+FE0F, 216, 200 -- deliberately,
because the documentation quotes them and the pinning is what catches
doc-vs-data drift. But nothing recorded which release they came from and
nothing checked it, so a new Unicode version would have surfaced as dozens
of unrelated-looking failures with no statement of the cause, plus four help
pages quietly naming a release the user does not have. There is now a single
constant, a canary test that fails loudly and says exactly which figures
need re-deriving, and an assertion that no help page names a different
release. Simulating the update confirms three tests fail, all three named so
the cause is legible, rather than seventy that are not.
Declared emoji (>= 16.0.0), which had no version floor at all. The other
floors in DESCRIPTION are keyed to arguments the code calls, which is
why emoji was not among them -- tidyEmoji reads its data, not its
functions. The reason it needs one is different and just as real: every
catalogue figure in the documentation describes a single release, and an
older emoji has fewer rows, so those figures would simply be wrong.
The floor, the version the help pages name and the pinned counts are now
governed by one constant and asserted to agree.
cran-comments.md asserted something a reviewer could test and find
false. Its explanation of the one "possibly invalid URL" note said "The
site itself is up: curlGetHeaders(url, verify = FALSE) returns 200".
That was true when written and is not now: the redirect target refuses TCP
connections outright, with or without TLS verification, so the claim would
not reproduce for anyone checking it. The passage now separates what the
package can vouch for from what it cannot. The documented URL is healthy and
that is the part that matters -- https://hdl.handle.net/11356/1048 returns
HTTP/2 302 to https://www.clarin.si/repository/xmlui/handle/11356/1048,
verified again for this release -- and the note describes a third-party
CLARIN.SI host reached only by following that redirect. Both observed
failure modes are recorded, the original certificate-chain one and the
present timeout, precisely because they differ: the point is that the
failure is not the package's and not stable enough to characterise. No claim
is made about the target's availability.
Re-checked every other URL in the package over the network: the two
citations, the EmoTag repository, the pkgdown site and its introduction
article, the issue tracker, the CRAN page and the three badges all return
200, and only doi.org and the CRAN page redirect (both to their expected
canonical targets). The t.co links the URL sweep turns up are rows of
inst/extdata/ata_tweets.csv, not documentation, and are not fetched.
Added the offline half as a test, since the network half cannot run on CRAN:
every URL in the help pages and DESCRIPTION must be well formed -- https,
no whitespace, no stray backtick or quote swept in from the surrounding
markup, no trailing punctuation, and a parseable host and path. That is the
class of defect that turns a working link into a reported one.
A second stale claim in cran-comments.md. It said "Two tests skip: one
gated on readr (>= 2.0.0) ... and one skip_on_cran() maintenance check"
-- written when the suite was much smaller, and describing a suite that no
longer exists: there are now seven skip_on_cran() tests. The passage
reports the re-measured inventory and names all seven, marks the R 4.1.0
figure as history rather than a current measurement (that tree cannot be
rebuilt here, so asserting a fresh number would be inventing one), and
explains why a tarball check skips three more than a source check -- they
read README.Rmd, the R/ tree and the vignette corpus, none of which a
tarball carries.
The rest of the file was re-verified rather than trusted. The marked-UTF-8
breakdown is exact -- emoji_unicode_crosswalk$unicode 5761,
emoji_sentiment_lexicon$emoji 969, emoji_emotion_lexicon$emoji 150,
category_unicode_crosswalk$unicodes 10, summing to the 6890 it quotes --
and no other column in any bundled dataset carries a marked string, as it
claims. Against a live CRAN index: still no reverse dependencies, the
published version is still 0.3.0, and emoji is still 16.0.0, which
confirms the version floor added above is satisfiable and that the
documented catalogue figures describe the release CRAN currently ships. The
package count was refreshed from 24,744 to 24,748.
Those figures are now derived in a test rather than asserted in prose, so
they cannot drift again: the skip count, each component of the UTF-8
breakdown and their total, the claim that no non-glyph column is marked, and
the licence. Writing it caught its own bug first -- counting occurrences of
the string skip_on_cran() in the test sources returns nine, because the
literal appears in the counting code and in comments. It counts lines that
are the call, and the runtime skip count confirms the figure independently.
README.md is knitted from README.Rmd, so every #> line in it is output
the package produced at some past moment. An existing test compares the two
files' prose, which catches an edit made to README.md instead of to its
source -- but not the likelier rot: prose and code both untouched while the
package's output moves underneath them. This release changed several
messages and added columns, so it was worth checking. It had not gone
stale -- a fresh github_document render is byte-identical to the shipped
file -- but nothing was checking, and now something does.
That test compares the #> lines rather than the whole render, deliberately.
The prose is re-wrapped by pandoc, whose line breaking differs between
versions, so diffing whole files would fail on a different pandoc even when
the README is perfectly current. Three cosmetic markers are normalised for
the same reason: testthat forces cli.unicode = FALSE, so pillar decorates
its tibbles in ASCII (x, ~, i) where a maintainer's render uses
Unicode (\u00d7, \u2026, \u2139). Each was found by running the
comparison and reading what differed rather than by guessing. Everything
that can actually go stale -- row and column counts, column names, values --
is compared byte for byte.
Fixed a loop in last round's own new test that iterated zero times. Its
"no non-glyph column carries a marked string" check ran over a hand-written
list of datasets that included emoji_tweets -- which is the deprecated
synonym for [emoji_filter()], a function, whose names() is empty. That
entry therefore asserted nothing, the same vacuity that left
emoji_pairs()'s collation untested for a release. The list is now derived
from utils::data(), so it is exactly the four datasets the package ships
and a fifth would be covered automatically, and the number of character
columns visited is pinned at 13 rather than bounded below -- a new one fires
the test and has to be classified as a glyph column or not.
Measured where the check's time goes, since this release's suite has grown
by roughly 2,100 assertions. R CMD check --as-cran is 158s of CPU: 100s
tests, 12s re-building the vignette, 7s incoming feasibility (its 143s of
wall time is network, not compute). Inside the suite no single test
dominates -- the slowest is 4.5s and 21 tests over a second account for 64%
of the total -- so there is nothing pathological to trim, and the whole
check sits comfortably inside CRAN's per-flavour budget.
Audited the suite for assertions that cannot fail, since the last two rounds
each found a defect in a test rather than in the package. Two sweeps, both
clean. Comparing every block's expect_* count in its source against the
number it actually executes at runtime found one shortfall across all
478 blocks, and that one is a genuine either/or: the tolower() audit
asserts against the R/ sources when they are present and against the
installed namespace when they are not, so exactly one of its two assertions
runs. Scanning for assertions whose two operands are syntactically identical
-- expect_equal(x, x), expect_true(TRUE) -- found none.
The second sweep is cheap and the test sources ship, so it is now a test
rather than a one-off: no assertion in the suite may compare something to
itself, and every test_that block must contain at least one expectation
(testthat warns about an empty block but does not fail, which is easy to
miss in a suite this size). Both were verified by injecting the defects they
look for and confirming each fires.
Added metamorphic tests -- properties that must hold between related
inputs. They catch a class unit tests cannot, because they need no known
answer: a verb can be wrong in a way no fixture reveals and still be caught
by "these two corpora must agree". Four families, all of which the package
already satisfied:
additivity (every glyph's count over a concatenated corpus is its count
in the parts, and emoji_summary()'s two counters add);
scale invariance (doubling every row doubles emoji_frequency(),
emoji_pairs() and n_tokens exactly, while share_types, share_tokens
and emoji_collocations()'s PMI -- all ratios -- do not move at all);
padding invariance (adding emoji-free rows leaves emoji_frequency(),
emoji_pairs(), top_n_emojis() and the version profile identical, and
moves only the n_total those rows belong to); and
row independence.
Row independence is the sharpest of the four, and it is now a contract: for
thirteen row verbs, verb(data)[i, ] must equal verb(data[i, ]). Exactly
one row verb is allowed to break it -- emoji_incongruity(scale = "rank"),
whose whole purpose is to map each score to its percentile within the
corpus -- and the test requires it to differ, so that if rank scaling ever
quietly became scale = "none" the suite would say so. Verified by
injecting a corpus coupling into emoji_ratio() (dividing by the corpus
maximum) and confirming the test fails.
Extended the same four families to the calendar arithmetic and to
emoji_dfm(). Translation invariance is a strong property for a cycle, and
a bucketing bug would show up as a shift leaking into the counts: a whole
number of years cannot move a month cycle, a whole number of weeks cannot
move a weekday cycle, and in a zone without DST whole days of seconds
cannot move an hour cycle. emoji_trend() keeps its counts and shifts every
period by one constant offset; emoji_turnover(), which compares periods to
each other, is unmoved entirely. None of the four time verbs depends on the
order of the rows.
Pinned the DST case as behaviour rather than leaving it implicit. Adding 86400 seconds twice across spring forward is not adding two calendar days, so the displayed hours move by one and the cycle moves with them -- correct, because the verb buckets by the timestamp's own zone as documented. The test asserts the buckets are the displayed hours on both sides of the shift, and that the emoji total is conserved across it. (That area turned out to be well defended already: injecting a UTC-instead-of-local hour read fails six tests.)
emoji_dfm() conserves its cells: aggregating rows into documents with
doc_id redistributes them without creating any, the total equals
emoji_frequency()'s, permuting rows permutes the matrix rows identically,
and duplicating the corpus leaves the first copy's rows untouched -- tf-idf
included, since N and df both double and log(N / df) cancels.
Checked the session cache, which nothing had examined. The package memoises six things -- the reference table, its key set, the sentiment and emotion maps, the ambiguity table and the type map -- plus the user-writable lexicon registry. A memoised accessor has a failure mode no other test could see: computing one value, storing another, and so answering differently the first time than afterwards. All six now assert that the first call equals the second and third with the slot cleared beforehand, and that the call actually populates its slot, so the memo is real rather than decorative. Clearing a derived slot is safe because each repopulates from a source that cannot change within a session.
The registry is the one slot a caller can rewrite, so it gets a coherence
check: registering a second table under a name already used must replace the
first rather than be shadowed by it, emoji_lexicons() must list the name
once and report the new row count, and re-registering with a different
glyph column must still resolve -- which it does, because
register_emoji_lexicon() stores a computed key column. Verified the
whole guard by making emoji_reference() memoise a table one row shorter
than the one it returns; the new test fails.
The submission notes now cite cross-platform evidence instead of asserting
it. cran-comments.md said "0 errors | 0 warnings | 0 notes" and then
explained that the local 1 WARNING and 3 NOTEs are artefacts of this host.
Every flavour of the GitHub Actions matrix now reports Status: OK for the
submitted commit -- macOS and Windows on R 4.6.1, Ubuntu on R-release,
R-devel and R-oldrel-1 -- and all five re-build the vignette and run the
full suite. None of the four local findings reproduces anywhere the host is
not ours, which is the evidence for that reading rather than our word for
it. The note also records what CI does not cover: it passes
--no-manual, so the PDF manual is checked only locally. Each of those
claims was read out of the run logs rather than inferred.
Confirmed the package survives a check without its Suggests, which some CRAN
flavours run. Hiding the Suggests from the library path turned out not to
prove anything here -- the R system library carries readr and stringr
and they cannot be masked -- so the property is established by construction
instead, and it holds: neither the package's own code nor a single example
references a Suggests package. Both are now asserted, reading the
namespace and the help database so the check works in a tarball too.
Every Suggests use in the vignette is properly conditional. Two chunks look
ungated and are not: the setup chunk uses knitr, which is the
VignetteBuilder and therefore always present, and its library(ggplot2)
sits inside if (has_plot_pkgs). The six plotting chunks that use forcats
and stringr all carry eval = has_plot_pkgs. One apparent third case was
a comment explaining why readr::read_csv() is not used -- worth noting
because a naive scan counts that as a dependency.
Verified inst/CITATION, which nothing had checked and which is what a user
pastes into a paper. It renders three entries -- the package and the two
lexicon papers whose data ships here -- and every field agrees with
DESCRIPTION: version, URL, author, and the title, which differs only by
the conventional package-name prefix. The paper details hold against the
live records too: the EmoTag page range 8957-8967 matches the ACL Anthology
exactly, and the PLoS DOI resolves to the right article.
The offline half is now a test, including one assertion aimed at a specific
trap: the file falls back to a hardcoded version string when meta is
unavailable, so the test requires the rendered version to equal
DESCRIPTION's. At the next release that fallback goes stale, and this is
what will say so.
Measured line coverage for the first time: 99.23% across R/, with
fourteen expressions uncovered. Reading them was more useful than the
number. Thirteen are defensive early-returns and the two unreachable lexicon
fallbacks -- if (!nrow(m)) return(s) in the gap slicer, the non-data-frame
guard in the lexicon resolver, the non-character guard in the encoding
check, and so on. Every one guards a condition no caller can produce, and
they are kept: an internal helper that is safe only because of what its
callers happen to do is a helper waiting to be reused wrongly.
One uncovered line was genuinely reachable and simply untested:
emoji_trend()'s early return when the top_n cut leaves no emoji to
follow. top_n = 0 is a legitimate request, and the answer is correct --
a typed zero-row tibble whose five columns have the classes a populated
call gives, so the two bind. That is now asserted, together with
emoji_flag_ambiguous(top_n = 0).
Confirmed the generated files are not stale, which nothing had checked since
this loop began editing roxygen blocks: regenerating man/ and NAMESPACE
from the current sources produces no diff, and RoxygenNote matches the
roxygen2 that would do it.
This release's prose broke the package's spell check, and nothing said
so. inst/WORDLIST and Language: en-GB exist because the maintainer
runs spelling::spell_check_package(); against the commit this loop started
from it reported zero flagged words. Against the current tree it
reported 53 -- every one introduced by the documentation and release
notes written here. None is a misspelling: they are package names
(readr, tidyr, forcats, pandoc), R vocabulary (formals, AST,
tidyselect, memoise), domain terms (keycap, dingbats, faceness,
which is already an exported concept) and hyphenation artefacts (mis from
"mis-described"). Only four reach a help page. All 53 are now in
inst/WORDLIST, which is back to reporting zero.
The word list keeps the ordering it had. spelling::update_wordlist()
re-sorts case-sensitively, which would have rewritten the whole file and
buried 53 real additions in an 86-line diff; restoring the original
case-insensitive order makes the change exactly 53 added lines and nothing
removed.
Corrected two references left stale by moving the vignette corpus to
inst/extdata/ last round -- one of them in NEWS.md, which ships. Re-read
the whole tarball manifest afterwards rather than assuming the move was
self-contained: the root carries only DESCRIPTION, NAMESPACE, NEWS.md
and README.md, and inst/ carries CITATION, WORDLIST, the three
vignette artefacts and the corpus.
Re-examined why the remaining tests skip on CRAN, having just found that one
long-standing skip was hiding a real defect. Four of the eight were not
gated on an unshipped file, so their skip was a choice; timing them showed
cost was not the reason either -- 4.7s between them, out of 123s. Two have a
real reason and keep it: the declared-R-minimum check reads the installed
dependencies' own R floors, and the select()-avoidance check compares two
wall-clock timings. Both interrogate the checking machine rather than the
package, and neither is a fair question to ask a CRAN flavour.
The other two had no reason beyond caution and now run everywhere:
emoji_dfm() at its widest, which builds a 3791-column table and is
precisely the shape most likely to behave differently on a platform that
cannot be tested here, and the audit that no example except
?register_emoji_lexicon registers a lexicon -- a hazard created by
R CMD check running every example in one session, so a CRAN flavour is
where it most needs to hold. Twelve more assertions now run on all thirteen
CRAN flavours, and the tarball check's skips fall from ten to eight.
The vignette's code, as R ships it to users, could not be run. R installs
the tangled introduction.R precisely so a reader can execute the vignette,
and browseVignettes() offers it. It read its corpus as
read.csv("ata_tweets.csv"), a bare relative path that resolves while the
vignette is being built in vignettes/ and nowhere afterwards: the corpus
shipped in the tarball but was never installed, so the script failed on line
28 with "cannot open the connection". The corpus now lives in
inst/extdata/ and both the vignette and the shipped script reach it with
system.file(). The tarball grew by 3.4 KB, because the file moved rather
than being duplicated; the installed package grows by the file's 150 KB,
which is the cost of the script working.
Two tests stop skipping as a result. They were gated on the corpus being
reachable, which inside R CMD check it never was -- tests run from
.Rcheck/tests/, where ../../vignettes does not exist. Now that the
corpus is installed they run everywhere, including on CRAN, and the tarball
check's skip count falls from twelve to ten.
Swept every number of two or more digits on a help page against every
number appearing anywhere in the test suite. Seventeen were unaccounted for;
twelve are code points, citation years or the CLARIN handle. The other five
were real claims about this package's data that nothing checked, and all
five are exact: ?emoji_search's "189 of the catalogue's 5042 rows carry no
GitHub-style alias" and "7 of the 198 rows emoji_search("face") returns",
and ?emoji_density's "115 of the 560 emoji-bearing rows in the corpus
behind the introduction vignette contain a multi-code-point emoji". All five
are now derived in tests rather than asserted in prose -- the last of the
documented figures to be unpinned.
Measured how the verbs scale, which no test could show: every fixture in the
suite is a handful of rows, so an accidentally quadratic verb would be
invisible. All nineteen are linear -- at 1k, 4k and 16k rows the 16-fold
input gives 16-fold time across the board, the slowest being
emoji_ngrams() at 6.7s for 16k rows.
Verified the one cost model the documentation actually states.
?emoji_cooccurrence warns that the result "grows with the square of the
distinct emoji in a document" and quotes 319,600 pairs for 800 and 7.2
million for 3790. Both are exact, 3790 really is the catalogue's
distinct-emoji count rather than an invented ceiling, and the measured
timings are quadratic and land where the page says -- 0.09s, 0.24s, 0.81s
and 3.03s for 100, 200, 400 and 800, so 800 is the promised "few seconds".
The arithmetic and the one-row-per-pair property are now tests; the timing
cannot be, so it is recorded here.
A related detail the measurement turned up, in the model's favour: cost
depends on distinct emoji, not on occurrences. 4000 rows pooled into two
documents is faster than 4000 one-row documents, because a document holding
6000 glyphs drawn from a 10-emoji vocabulary still has only 45 pairs. 800
distinct spellings likewise collapse to 726 emoji and 263,175 pairs --
exactly C(726, 2), which is the model confirming itself.
The deployed pkgdown site was checked against this release rather than assumed current: its home, reference index, introduction article and news pages all serve, the index links all 50 topics (51 help pages less the package doc, which pkgdown excludes by design), and the reference pages carry the wording added in this release.
Verified the pkgdown reference index, since a new exported verb that nobody
adds to _pkgdown.yml breaks pkgdown::build_site() with "topics missing
from index" -- after the release, on someone else's machine.
pkgdown::check_pkgdown() reports no problems: all 51 help topics are
reachable. A hand-rolled version of that check flagged two entries the real
one accepts, because an alias counts as covering its topic (emoji_tweets
lives on emoji_filter's page) and the package-level doc is excluded by
design -- asking the tool rather than reimplementing its rules was the
difference between two false alarms and a clean answer.
That check is deliberately not in the test suite. Adding it raised a
second R CMD check WARNING -- "'::' or ':::' import not declared from
'pkgdown'" -- and the only way to clear it is to declare pkgdown in
Suggests, which would make every CRAN check machine install a heavy
dependency for a test that can never run there: _pkgdown.yml is
build-ignored, so the test would skip in the tarball every time. The site
index is verified here instead, and build_site() catches a regression on
the first run after one.
The cran-comments.md skip-count assertion no longer hardcodes the number
in two places. It counts the skip_on_cran() call sites and derives the
word the file must contain, so adding a skipped test forces the submission
notes to be updated -- which is the point of the coupling -- without also
forcing an edit to the test.
The missing-column message added earlier this release listed every column in
data, which is the caller's data and can be wide. A 500-column frame
produced a 5074-character error and a 40-column survey export 665 --
burying the one name that is wrong behind the hundreds that are not -- and a
frame with no columns at all trailed off as Available: .. Five names are
now listed and the rest counted (and 495 more), so the message is 84-118
characters whatever the data looks like, and a frame with no columns says so.
A test pins that ?register_emoji_lexicon's example is the only one that
registers a lexicon. R CMD check runs every example in one session, so a
registration is visible to every topic sorting after it, and
emoji_lexicons() reports what is registered -- today that verb sorts before
register_emoji_lexicon() and the printed output is unaffected, which is
alphabetical luck rather than design. Running all 46 topics' examples
forward and in reverse also errors nowhere, so nothing else depends on the
order.
?emoji_pairs now says what doc_id costs. The result has a row per pair,
so it grows with the square of the distinct emoji in a document -- a
conversation or a day is cheap, pooling a whole corpus under one id is not.
Measured: 800 distinct emoji in one document is 319,600 pairs and 3.5s, and
the bundled corpus pooled into a single document (187 distinct emoji) is
17,391 pairs in 0.27s. The behaviour is inherent to the question rather than
a defect, but the shape of the cost was not written down.
emoji_incongruity_profile()'s statistics are now derived independently in
the tests -- the last formula in the package that no test computed for
itself. n, mean_incongruity, sd_incongruity and n_flips all match
crediting every glyph in a row with that row's gap, across all three
scale values, along with flip_rate, the documented sort order, the NA
standard deviation for a single occurrence, and min_n selecting rows
without changing a statistic. No defect found.
Two scale limits nothing had reached are now tested: emoji_dfm() at its
widest -- one document holding every distinct emoji, 3791 columns, all
names distinct and non-empty, tf-idf zero everywhere as log(N/df) requires
for a single document -- and a wide multi-document table keeping its
documented column ordering and row sums. Also that a missing column is
reported identically for all four column arguments, naming the argument and
the column and never a same-named variable from the caller's session.
A misspelled column name could be reported as the contents of one of your
variables. tidyselect falls back to an external vector when a bare name
is not a column, and text is a common variable name -- so
emoji_sentiment(df, text), on a data frame whose column had been renamed,
in a session where text was also a character vector, gave
Can't select columns that don't exist. Columns `from the global ...` and ... -- naming the contents of the caller's variable as though they were
column names -- plus a tidyselect deprecation warning advising all_of(),
which is not what the caller meant. That is the same failure mode this
resolver was written to replace, arriving by another route. A bare name is
now checked against names(data) directly: the message names the argument
and the missing column and lists the ones that exist, with no spurious
deprecation warning. Where the column does exist it still wins over a
variable of the same name, and every other tidyselect form -- a string, a
position, all_of(), starts_with() -- still resolves through
dplyr::select(), two-column selections included.
That also makes the commonest call much cheaper. Resolving a bare column name
no longer runs dplyr::select(): 2000 resolutions went from 2.0s to
0.058s, and a realistic loop of 500 per-group emoji_sentiment() calls
from 1.32s to 0.93s. Large-corpus timings are unchanged, being detection
bound. It also retires the note added earlier this release about five verbs
resolving the name twice -- at 0.03ms a resolution there is nothing left to
restructure.
The time verbs' buckets follow the session timezone when the column has
none, and nothing said so. ?emoji_trend promised that a date-time is
bucketed by the day it "displays as in its own timezone" -- but a POSIXct
built by as.POSIXct("2024-01-01 23:30"), which is what most CSV readers
give you, has no tzone of its own, so R displays it in the session's. The
same column reports hour 23 in one session and hour 4 in another; measured
across UTC, America/New_York, Asia/Tokyo and Pacific/Kiritimati, the
hour-of-day buckets came out 0/23, 0/1, 13/14 and 18/19. That is correct
POSIXct behaviour rather than a defect -- a column with no timezone has
none to display in -- but it is the same reproducibility trap as the
collation and ctype dependence this release fixed, and it was undocumented.
?emoji_trend's time now says to tag the column
(as.POSIXct(x, tz = "UTC")) when the result has to be reproducible.
Three tests pin the timezone axis, which nothing had varied: a tagged
POSIXct gives byte-identical days, hours and months across four timezones
spanning a date line and a DST boundary; a Date column is immune across all
five bucket widths; and an untagged POSIXct demonstrably follows the
session, so the documented warning cannot quietly stop being true.
A time column written dd/mm/yyyy was read as dates in the year 1, and
nothing said so. as.Date()'s %Y accepts a one- or two-digit year, so
as.Date("01/02/2024", format = "%Y/%m/%d") is 0001-02-20 -- year 1, month
2, day 20. The character branch tried "%Y/%m/%d" as a fallback, so a whole
column in dd/mm/yyyy or mm/dd/yyyy form -- most CSVs written outside
ISO-land -- parsed without complaint, and every time verb bucketed on years
1, 3, 12 and so on. The four-digit year is now required before parsing, which
sends those values to the paths that already existed for unreadable dates:
mixed with real dates they warn and are dropped, and a column where nothing
reads as a date errors. Every form that worked still works, including
"2024-1-1" and a trailing time.
The error for a wholly unreadable time column now diagnoses what it found
instead of restating the argument's type, quotes the value it choked on, and
points at as.Date(format = ) -- so the dd/mm/yyyy case above arrives with
a route out. ?emoji_trend's time documents the accepted forms, that a
column with nothing readable is an error rather than all-NA, and names
"01/02/2024" as the case to convert first.
A bad lexicon was reported against tbl, an argument the scoring verbs
do not have. emoji_score(), emoji_sentiment() and emoji_emotion()
share their lexicon validation with register_emoji_lexicon(), whose own
formal is called tbl -- so passing a data frame as lexicon produced
No score column found in `tbl`; supply `score` , telling the user to fix
an argument that function does not take. That is the same failure mode the
column resolver's arg parameter was added for, and the helpers now take one
too: all six messages reachable through lexicon = name lexicon, and the
three reachable through tbl = still name tbl. Two of them also read
better for it -- Lexicon has no score column `nope` is now
`lexicon` has no column `nope` to take the score from.
Found by inspecting all 43 stop() messages in the package for whether each
names something a caller can actually type. The lexicon helpers were the only
exception; every other message either names a real argument or fills a %s
with one at run time. A test now enforces it, allowing the argument-with-value
form (period = "hour") that two messages use.
.emoji_text_col() resolved the text column's name twice. The
one-value-per-row check added this release routed the read back through
.emoji_col(), which resolves the name again -- and resolving it runs
dplyr::select(), about a millisecond a call. Invisible next to detection on
a real corpus (20,000 rows is unchanged at 0.26-0.86s a verb), but the whole
cost of a verb called once per group in a loop over a split data frame. The
check now takes the name it already has, so the read is back to one
resolution.
emoji_turnover(measure = ) swallowed a value it did not recognise. It
is the only argument in the package that takes several values at once, and
match.arg(several.ok = TRUE) returns the ones it matches without a word
about the rest -- so measure = c("jaccard", "flip") produced the jaccard
column and no hint that the second name was wrong. That is the
absorbed-argument failure mode this release swept everywhere else. An
unrecognised value is now an error that names measure and the value, and
an empty, NULL, NA or non-character measure gets one message instead of
match.arg()'s internal 'arg' must be of length >= 1. What had to survive
does: abbreviations ("jac") still match, duplicates are still ignored, the
four-measure default is unchanged, and the columns still come back in the
fixed statistic order rather than the order asked for. ?emoji_turnover now
says all four of those things.
A matrix column passed as time, doc_id or text_score failed four
different ways -- including not failing. A matrix holds one element per
cell, so a 2x2 column carries four values for two rows, and nothing
downstream noticed: a character time gave emoji_trend() and
emoji_adoption_lag() invalid 'times' argument, emoji_seasonality()
missing value where TRUE/FALSE needed, and emoji_turnover() a
result, computed over periods that were not in the data. A doc_id made
emoji_dfm() report four documents for a two-row frame, with the id column
filled from rows that do not exist. A text_score passed the type guard
outright -- a matrix is numeric -- and reached tibble as
Assigned data `gap` must be compatible with existing data. The
one-value-per-row check added for text last round now lives in the shared
column reader, so all four arguments get it and all ten verbs give the same
message. emoji_dfm() was the one column read that bypassed that helper;
it no longer does, and a test asserts no other does either.
The type and length checks now report in the informative order. A
data-frame column's length() is its column count, so the length check
fired first and said nothing about the real problem; the type check runs
before it. A POSIXlt column is unaffected -- R's length() method counts
times rather than list components -- so emoji_seasonality(period = "hour")
still accepts one.
A list column passed as text was scored from its own deparsed source
code. Every verb reads the text column through one helper that calls
as.character() -- which is exactly what lets a factor column work, and is
harmless on a numeric, Date or logical one. On a list column it deparses:
a column holding list(c("a", "<U+1F600>")) was read as the source text
c("a", "<U+1F600>"), the emoji inside that was detected, and the row came
back with a real-looking .emoji_sentiment of 0.5718 for data that never
contained an emoji. A data-frame column deparsed the same way. Both are now
refused, with a message that says why; every atomic column still works
exactly as before.
A matrix column errored opaquely in two verbs and silently over-counted in
a third. A matrix holds one element per cell, not per row, so a 2x2
matrix column gave emoji_sentiment() and emoji_tokens() a tibble error
naming a variable from this package's own source
(Assigned data `glyphs` must be compatible with existing data) while
emoji_frequency() happily counted all four cells against two rows. The
shared reader now requires one value per row, so all six verbs tried give the
same message; a single-column matrix has one element per row and still works.
emoji_score() sent an emotion-lexicon user down a dead end. It
averages the eight emotion dimensions into one number for the bundled
"emotag1200", but that averaging is special-cased to the bundled table: a
registered or inline lexicon of exactly the same shape has no score column,
so the call failed with "No score column found in tbl; supply score" --
advice that cannot help, since score names a single column and the user
wants the mean. The message now names both real routes (emoji_emotion()
for the profile, or score = "joy" to score on one dimension) and says the
averaging is specific to the bundled lexicon; ?emoji_score says the same. A
table with neither a score nor an emotion column keeps the original message.
No behaviour changed beyond the message.
emoji_emotion() |> emoji_emotion_label() silently threw away all eight
emotion scores. The label verb computes the scores internally and then
drops every .emoji_<emotion> column by name -- which cannot tell a column
it just added from one that was already in data. So the obvious way to get
the profile and the label kept only the label, and a caller's own column of
one of those names was deleted rather than overwritten. Both contradict a
@return that says the label is added. It now drops only the columns that
call introduced: emoji_emotion_label() on plain data returns exactly what
it always did, and the chain now returns all thirteen columns with the label
and the scores agreeing with the standalone calls. ?emoji_emotion_label
says so.
Found by sweeping the pipelines a user would actually write, which nothing
had checked -- every verb had only ever been tested on its own, against a
package whose headline claim is that the verbs "compose naturally with the
pipe". Fourteen chains are now tests, along with the grouped contract end to
end (group_by() through a row verb into summarise(), grouping surviving a
chain, and a text-rewriting verb re-deriving the groups when the user grouped
by that very column), re-running ten row verbs on their own output, and
feeding four aggregators a row verb's output column.
?emoji_cooccurrence said it "is emoji_pairs() under another name, with
one addition", and it also silently drops an argument. There is no
directed on emoji_cooccurrence(), so a reader taking that sentence
literally gets unused argument (directed = TRUE) with no explanation. The
omission is right -- a co-occurrence matrix is symmetric, so an ordered pair
has no meaning on it, and neither would the diagonal the verb exists to add
-- but only the addition was documented. The description now states both,
with the reason and where to go for directed, and a test pins the two
signatures as differing by exactly directed and diagonal.
Found by a cross-product sweep of every enum and flag argument -- 182
combinations across seventeen verbs, now a test. method x scale x where
for the three mismatch verbs, weighting x doc_id for emoji_dfm(),
directed x sort x diagonal x doc_id for the relational pair, by x
measure x top_n for emoji_trend(), all four measure subsets of
emoji_turnover(), and the rest. Every combination that exists returns a
data frame with nameable columns; the one that did not is the item above.
emoji_tokens() was the one verb that did not validate data, and a bare
list or a character matrix silently succeeded. It called
.emoji_as_tibble() before resolving the text column, and
tibble::as_tibble() converts a list or a matrix quite happily -- so the
is.data.frame(data) guard every other verb enforces never saw the original
object. emoji_tokens(list(text = c("a", "b")), text) returned a result
where the other twenty-five verbs error, and NULL, a number, a function and
an environment reached dplyr's or R's own internals
(cannot coerce class "function" to a data.frame) rather than the package's
message. The column is now resolved first, so all twenty-six verbs give
`data` must be a data frame for all seven wrong shapes. A data frame
that simply lacks the column stays a different error, delegated to
dplyr::select() so the message names the column.
The detection figures this release quotes were wrong, in NEWS.md and in
cran-comments.md both. They read "exact detection of the reference table
goes from 80.1% to 95.8%" and "spellings that lose a joiner from 793 to 2",
which pairs a no-repair baseline with an after-rules-1-and-2 one. Staging the
rules and measuring each: no repair gives 63.2% (3189 of 5042 spellings
read as exactly one emoji equal to the whole spelling) and 1643 orphaned
joiners; rule 1 alone, which is UAX #29's GB11, gives 75.5% and 1025; rule 2
leaves 793; rule 3 takes those to 2. No staging produces 80.1%, and 793 is
where rule 3 starts, not where the three rules together do. Both documents
now give 63.2% to 95.8% and 1643 to 2, with the intermediate figures stated
so each is attributable, and a test derives all six from the rules.
Four help topics had no @seealso at all -- and they were the four other
pages point at: emoji_categorize(), emoji_emotion_label(),
emoji_extract_unnest() and emoji_filter(). A reader arriving at one from
the reference index or a search had no route onward, while the other 44
exports did. Each now points at its pair and its neighbours, and a test
requires every exported topic to carry one.
?emoji_sanitize's reversibility table promised more than wrap allows.
It says policy = "shortcode" restores the original text, calls it "the only
policy that permits it", and quotes a 100% byte-exact figure -- but the
section never mentioned wrap, an argument of the same function that voids
the promise. text_to_emoji() looks for exactly :shortcode:, so measured
across eleven templates: the default ":{x}:" restores the text exactly; a
decorating wrap ("[:{x}:]", ":{x}:!") brings the emoji back but leaves
the decoration; a wrap with no :token: ("{x}", "<{x}>", "@{x}@",
":{x}", "{x}:", "a{x}b") restores nothing at all and does so
silently, leaving the shortcode in the text as an ordinary word; and
"::{x}::" brings the emoji back wrapped in leftover colons, so it is not
even stable under a second pass. All four cases are now in the section, with
the precondition stated on ?emoji_to_text's wrap and in
?text_to_emoji's opening sentence, and pinned by a test.
The grouped-input warning is now asserted for all fourteen cross-row aggregators at once: each warns exactly once, names itself rather than the helper or the verb it delegates to, and does not tell the user to report a bug in tidyEmoji. Previously one verb was spot-checked.
The introduction vignette could not be built without its Suggests
packages, so R CMD build failed outright on a library tree lacking one.
It read its example corpus with readr::read_csv() and drew its charts with
ggplot2, forcats and stringr, all unconditionally --
optional dependencies used as if they were required. Verified by building on
an R 4.6.0 tree with neither 'readr' nor 'forcats': Error: Vignette re-building failed. The corpus is now read with utils::read.csv() wrapped
in tibble::as_tibble() (byte-identical full_text, same encodings, same
900 emoji tokens, identical verb output), and the nine plotting chunks are
gated on a requireNamespace() flag. The vignette builds either way, and
the readr Suggests entry now exists only for one guarded test.
Five documentation-pinning tests were silently inert on every CRAN
flavour. They read ../../man/*.Rd, which exists only when the suite runs
from the source tree; inside R CMD check the tests execute from
<pkg>.Rcheck/tests/testthat/ and skipped with "man/ not available". So
every assertion that a figure or a column name really appears in the rendered
help page -- the mechanism several of this release's fixes rely on -- was
never actually checked where it mattered. They now read the installed help
database via tools::Rd_db(), whose as.character() reconstructs the
markup, so \code{...} matches still work. Two more tests were reaching for
NEWS.md, DESCRIPTION and the vignette source the same way; all three are
installed with the package, so they read the installed copy when the source
tree is absent. Skips inside R CMD check drop from 14 to 6 -- two
deliberate skip_on_cran() maintenance checks, three that genuinely need
R/*.R, and one that needs the vignette's CSV -- and 160 more assertions run
there than before.
cran-comments.md contradicted itself about this host's check result,
reporting both "1 WARNING and 3 NOTEs" and, three paragraphs later, "one
WARNING and two NOTEs" for the same run. The duplicate paragraph is gone. Its
other stale figures are corrected or removed: R 4.6.0 gives Status: 2 NOTEs
rather than Status: OK, the package has 16 URLs rather than the 12 claimed
(so the count is no longer asserted -- only the one URL that is reported),
and the CRAN-index size is dated as a snapshot. The three new dependency
floors and the vignette change above are now described there too.
?emoji_density said "emoji per character" without saying that a
character is a code point, which reads as per-grapheme and is not. A
multi-code-point emoji inflates the denominator by all of its code points, so
the same visible text gives different answers: "hi <emoji>" is four
graphemes either way, but .emoji_per_char is 0.25 for a
single-code-point smiley, 0.200 for a two-code-point flag and 0.100 for a
seven-code-point ZWJ family. 115 of the 560 emoji-bearing rows in the
vignette's corpus contain a multi-code-point emoji, so it is not an edge
case. emoji_ratio() had always stated this basis and emoji_position()
states the opposite one (each emoji counts as one position, because a
proportion of a message has to); emoji_density() sat between them saying
neither. It now states the basis, gives those three figures, and points at
.emoji_per_token, which does not move with an emoji's internal length. This
closes the last open action from the audit that fixed
.emoji_rel_position: "grep for other nchar() uses on user text". All
thirteen are now accounted for -- four feed a documented user-facing figure,
nine are internal offsets.
cran-comments.md also undercounted its own skips. It said "seven
tests skip, all seven skip_on_cran()", and a CRAN run prints ten: the
seven, plus three that stand down because a tarball does not carry the
file they read (README.Rmd, the package sources, and the build-ignored
.github/). A reviewer running the check would have seen the letter
contradicted by its own output. The test that couples the count only ever
counted skip_on_cran() call lines, which is why it never noticed, so it
now also requires the file to name each stand-down message that exists in
the suite.
cran-comments.md said five CI flavours when the matrix defines six.
The workflow gained an r: '4.1' job precisely so the declared floor of
R 4.1.0 is built against, oldrel-1 sitting far above it, and the file a
reviewer reads to learn what was tested went on saying "five" in three
places and listing the other five. It now says six, names the 4.1 job
(R 4.1.3 on that runner) and says why it is there. A test couples the
spelled-out count and every runner name to the workflow itself, so the
two cannot drift again; it reads a build-ignored file, so it runs from the
source tree and stands down inside R CMD check. The stale claim inside
the test that checks the declared minimum is corrected too: it said CI
cannot reach the floor, which stopped being true when that job was added.
Nothing guarded the figure references, and this package has already
shipped a broken one. man/figures/lifecycle-deprecated.svg was missing
while ?top_n_emojis pointed at it, and no check caught it: a
\figure{} sits inside the html branch of an \ifelse, so the PDF
manual takes the other branch and never looks. The figure was restored
earlier in this release and then left unprotected. A test now reads every
figure name out of the installed help database and requires the file to
exist and be non-empty, in man/figures from the source tree or
help/figures from an installed package, which is where R CMD check
reads them. It also pins the two references that exist today, so a
reference vanishing is as visible as a file vanishing, and checks that the
README points at the logo by its repository path, which is what makes it
render on the CRAN package page.
The vignette builds without its optional packages; the script knitr
tangles out of it does not, and nothing said so. A chunk's eval
option is a build-time instruction and does not survive into the extracted
code, so inst/doc/introduction.R -- the file
edit(vignette("introduction", package = "tidyEmoji")) opens -- calls
ggplot() unconditionally, while library(ggplot2) above it stays behind
the requireNamespace() gate. Sourcing it on an installation without
ggplot2, forcats and stringr therefore fails partway
through, which is exactly the situation the gate was written to survive.
The vignette now says which half of the script runs on the declared
dependencies alone and what to install for the rest, and a test holds that
claim to the installed file. Nothing changed in the code: the chunks stay
ungated so the rendered vignette shows the plotting code as a reader wants
to read it, and dropping it from the tangled script would take working
code away from everyone who does have the three packages. Verified
separately that the script runs to completion, with zero warnings, when
they are present.
The other four companion arguments are inert on the branches they do not
belong to, and now say so. emoji_sanitize()'s placeholder and wrap
each serve one policy, and emoji_score()'s by and score only apply
when lexicon is a table rather than a name. All four were quietly ignored
elsewhere, changing nothing but leaving the reader to infer it. Each
@param now says which branch it belongs to and that it is ignored off
it, as ?emoji_to_text already did for wrap under format = "name".
They stay silent deliberately, and the page says why: policy and
lexicon are meant to be variables, so a script sweeping all five policies
with one placeholder = set would be warned at four fifths of its calls.
That is the line between these four and the threshold above, which is a
number the caller chose in order to move a result. emoji_sentiment(se =)
is the one companion that cannot be ignored, needing annotation counts only
the bundled lexicon carries, and it already errored. Tests pin all of it:
silent, and byte-identical to the call without the inert argument.
threshold did nothing under method = "sign_flip", quietly. That
method flags a polarity flip rather than measuring a gap, so there is
nothing for a threshold to cut, and a call giving both had half of it
absorbed without a word. It is the same shape the package refuses
elsewhere: emoji_turnover() errors on a measure value it cannot
honour, and emoji_to_text() on a wrap with no placeholder. Here the
answer is right rather than wrong, so the proportionate signal is a
warning, and it fires only when the caller actually typed the argument,
after match.arg() so an abbreviated method = "sign" is caught too. All
three verbs of the family do it. The page also records that a threshold at
or below 0 flags every scored row, abs(gap) never being negative.
Three verbs take a top_n and only one said how it cuts.
?top_n_emojis records that a tie straddling position n is settled by
the glyph, and that a corpus with fewer emoji than n returns all of them
rather than padding. emoji_trend() and emoji_flag_ambiguous() do the
same thing and said neither, so a reader could not tell whether their cut
was arbitrary. Both now say it, and a test drives all three with four
glyphs tied on every key so that only the glyph order can decide, and
checks top_n above the available count and top_n = 0 on each.
emoji_emotion_label() labelled a row with no dominant emotion
"anger". max.col(ties.method = "first") returns the first column when
nothing wins, and the first of the eight in Plutchik order is anger, so a
custom lexicon scoring an emoji the same across all eight dimensions --
zero across the board, most obviously -- came back asserting anger. The
row now gets NA, which .emoji_n_scored still distinguishes from a row
with nothing to score, since that count stays at its value rather than
becoming NA too. A genuine winner is untouched, and so is an ordinary
two-way tie, which the documented Plutchik order still settles. Only
reachable through lexicon = : none of the bundled lexicon's 150 glyphs
scores flat across the eight.
?emoji_emotion_label said ties are broken in Plutchik order without
saying it happens. It does: 3 of the bundled lexicon's 150 glyphs tie
for their top emotion, and because Plutchik order is alphabetical the
tie-break quietly favours the early names. U+1F3A4 scores anticipation
and joy at 0.39 and is labelled anticipation; U+1F619 scores joy and
trust at 0.83 and is labelled joy. The page now names both, so a reader
knows the rule is load-bearing rather than theoretical, and a test holds
all three ties to it.
emoji_seasonality()'s emoji_per_text had no definition, and the
table hands you both candidate denominators. It is n_emoji / n_texts,
an average over every text in the level rather than over the ones
carrying an emoji, and the two readings are far apart on a mixed corpus:
four emoji spread over three texts, one of which carries all four, is
1.33 one way and 4 the other. The @return listed the column name and
stopped, while printing n_texts and n_with_emoji side by side, so a
reader had no way to tell which had been used. It now gives the formula,
the worked contrast and the one-line recipe for the other reading, and
says the column is NA rather than 0 where a level holds no text.
emoji_trend()'s share was already precise about its own denominator;
a test now also pins the consequence that a top_n cut does not
renormalise it, so a followed glyph keeps the share it had in the full
call.
?tidyEmoji's contract glossed .position once, and it means two
different things. The page lists five structural index names and
describes them together, .position among them as "the character offset of
an occurrence". That is right for emoji_context(), where it is a
code-point offset into the text and
substr(text, .position, .position + nchar(.emoji) - 1) returns the glyph.
It is wrong for emoji_ngrams(), where .position is the index within the
row's emoji sequence, so the first n-gram of a row is 1 whatever the text
looks like. Both verb pages were already right about their own column; the
contract that a reader meets first was not. It now singles .position out
as the one of the five whose unit depends on the verb, and
?emoji_context names the unit and the round trip rather than saying
"character position", which is the vagueness ?emoji_position had already
avoided for .emoji_first and .emoji_last.
emoji_context(unit = "char") can return half a grapheme, and did not
say so. "Character" there means code point, the unit nchar() and
substr() count, so a window can begin or end part-way through a cluster:
where an e carries a combining acute (U+0065 U+0301), window = 1
hands back the bare U+0301, a diacritic with nothing to sit on.
?emoji_density and ?emoji_ratio both spell the code-point basis out
already; this is the verb where it shows up in the output rather than
only in a denominator, and it was the one that did not mention it. The
unit argument now says it, names that case, and points at the two ways
round it (a word window, or a couple of characters more than you need).
Emoji are not affected, being masked out whole before the window is cut.
The README comparison would have failed on the declared R minimum, for a
reason outside the package. It re-renders README.Rmd and compares every
#> line byte for byte, normalising the three ASCII markers testthat
forces. Column padding was not normalised, and it moves without the
package moving: pillar pads to a glyph's display width, and utf8 has
changed its mind about the East-Asian-ambiguous ones. U+263A is width 2
under utf8 1.2.1, which is what R 4.1.0 ships, and width 1 under 1.2.6.
One column shifted by one space and the whole comparison failed. Runs of
spaces are now collapsed before comparing, so every value, column name and
row count still has to match while the padding cannot decide it. Found by
re-running the suite on R 4.1.0, which is the only failure it produced:
14664 pass, 1 skip (the readr (>= 2.0.0) gate, that tree carrying 1.4.0).
Nothing said which verbs merge two spellings of one emoji, and they do
not all agree. Fourteen verbs report a glyph. Six hand back the spelling
they found (emoji_frequency(), top_n_emojis(), emoji_tokens(), the
two extraction verbs and emoji_context()), because an occurrence is the
text you had. Eight collapse both spellings onto the catalogue's
(emoji_pairs(), emoji_cooccurrence(), emoji_ngrams(),
emoji_dfm(), emoji_collocations(), emoji_trend(),
emoji_adoption_lag() and emoji_incongruity_profile()), because an
item, a node, a feature or a series is one thing per emoji; and
emoji_turnover() and emoji_version_profile() count that way in
n_types without reporting a glyph at all. On a corpus holding both
spellings of U+1F642 U+200D U+2195 the split is one line:
emoji_frequency() returns two rows of n = 1 where
emoji_cooccurrence() returns one node of n = 2. ?tidyEmoji gains a
Which spelling comes back section naming both lists and the remedy (the
name is the same for both spellings, and as_emoji_name() supplies it
for a column of your own), and the four verb pages that were silent point
at it. Only eight of the fourteen had said anything, and the sentence
?tidyEmoji already carried -- that the key "resolves identically in
every verb" -- is about the metadata join and invites exactly the wrong
generalisation. A test derives the split from behaviour, so a verb that
changes sides fails rather than quietly contradicting the page.
Two of the four time verbs never said an undated row is dropped.
emoji_trend() and emoji_adoption_lag() both state it;
emoji_seasonality() and emoji_turnover() did not, and seasonality is
where the omission bites, because it returns every level of the cycle
whether the data reaches it or not. The table therefore looks complete
while every count in it is over the dated rows: on a four-row corpus
with one undated row, sum(n_texts) is 3 rather than 4, and an emoji
sitting in that row reaches neither n_emoji nor share. Both pages say
it now, seasonality with the consequence for its own columns and a pointer
to [emoji_summary()] for counting the corpus itself. A test holds all four
pages to the same sentence and checks the arithmetic behind it.
?emoji_lexicons called n the "number of emoji", and it is a row
count. For the two bundled lexicons those coincide, 969 and 150, because
each has one row per code-point key, and the page now says so rather than
leaving it to luck. A registered lexicon need not: two spellings of one
emoji are two rows and score one glyph, and a row whose glyph yields no key
at all is counted here and matched never. It is the distinction
?emoji_provenance already spells out for n_emoji, missing from the one
other place the package reports a lexicon's size.
?register_emoji_lexicon did not say that a repeat registration
replaces. It does, silently and completely, and since there is no public
counterpart that removes a registration, re-registering is the only way to
change a lexicon. The page says that now, and that a key column already
in the caller's table is replaced rather than trusted, the registry
looking a lexicon up by that column.
emoji_token_cost(tokenizer = ) took its answer on trust. The
function is the widest surface the verb has, and three wrong answers from
it were silent. A negative count went straight into
.emoji_token_estimate, which is the nonsense top_n_emojis(n = )
already refuses. An infinite or out-of-integer-range one became NA
through R's own NAs introduced by coercion to integer range, a warning
naming neither this argument nor this verb. And a data frame is a list, so
lengths() counted its columns and the shape check passed whenever the
caller happened to have as many columns as the data has rows. All three
now error, naming the values that came back. NA is still accepted and
passed through, for a tokeniser that cannot answer for a row, and a
fractional count still rounds up.
A user lexicon's Inf was counted as a score. The score column is
refused if it is not numeric, on the stated reasoning that a value mean()
cannot use must not be reported as scored, and a non-finite text_score
warns and is treated as missing in emoji_incongruity() for the same
reason. Inf fell between the two: numeric, so the type guard passed, and
not NA, so .emoji_n_scored counted the emoji and .emoji_score came
back Inf, while NaN and NA in the same column were already counted as
unscored. One column, two answers for the same kind of unusable value. A
non-finite score now warns and is treated as missing wherever a lexicon is
read, so emoji_score(), emoji_sentiment(), emoji_emotion() and a
registered lexicon all agree.
A custom emotion lexicon skipped the type check its sentiment
counterpart makes. A character or factor emotion column reached
as.matrix() and failed later with R's own 'x' must be numeric, naming
neither the argument, the column nor the verb: the failure mode the
score-column guard exists to replace. emoji_emotion() now names the
offending columns and their classes. Logical and integer dimensions are
numbers and stay accepted.
An all-missing emotion dimension reported NaN, not NA.
colMeans(na.rm = TRUE) over a dimension no emoji in the row has is 0/0,
and the NaN reached the output column. Every other verb reports an
unknown value as NA, and emoji_sentiment() returns NA_real_ for
exactly this case. Only reachable through lexicon = , since the bundled
emotion lexicon has no missing cells.
?emoji_incongruity's invariance claim reads as more than it holds.
The page says the two scalings are computed over the rows the comparison is
defined on, so rows with no scorable emoji cannot move the answer, and
subsetting to the scored rows before calling gives the same numbers. Both
are true, and both are easy to read as "the gap is a property of the two
scores". It is not: "rank" and "zscore" are relative, so a gap's size
depends on how many scored rows it was computed over. The sharpest case is
duplicating a scored corpus exactly, which adds no information whatsoever
and still multiplies every rank gap by 2 * (n - 1) / (2 * n - 1) (0.933
at n = 8, 0.995 at n = 100, 0.999 at n = 500) and every z-scored one
by sqrt((2 * n - 1) / (2 * (n - 1))), because sd() divides by n - 1.
The page now says where the invariance stops, and what to reach for when a
number has to travel between corpora. Found by a sweep that runs every verb
on a corpus and on the same corpus with every row repeated: counts must
double, per-row measures and every documented share must not move, and
emoji_collocations()'s PMI must not either, being a ratio of
probabilities. Everything else in that sweep already held, and all of it is
now pinned, the scaling factors included.
Six help pages named a sort key but not enough of them to order the
output. Every verb here sorts deterministically, and had been pinned as
doing so, but a reader following the @return could not predict the
result. ?top_n_emojis is the clearest case: it promises descending n
with ties broken by the glyph, "so the order is deterministic", while
duplicated = TRUE leaves several rows sharing both. The answer, now
written down and pinned over 200 catalogue glyphs, is
emoji_unicode_crosswalk's order for the glyph's codepoint key, with the
first row carrying the same name duplicated = FALSE reports; the same
paragraph records a consequence of joining on the key rather than the
spelling that the page had never mentioned, namely that a glyph collects
the aliases of every spelling of itself, so the unqualified
U+26F9 U+200D U+2640 is listed under woman_bouncing_ball and under the
two aliases its fully-qualified spelling carries. The other five:
emoji_collocations() sorts by measure descending, then the other of the
two, then the glyph and the word, where ties in pmi are the common case
rather than the exception; emoji_trend() by period, then measure, then
the glyph, which is what separates the zeros it fills in;
emoji_incongruity_profile() by flip_rate, then n, then the glyph,
where the first key is 0 for every glyph that never flips;
emoji_adoption_lag() by n then the glyph; and
emoji_version_profile() oldest first, with the emoji whose version the
reference table cannot give pooled into one last row rather than a first
one. A test now sorts each result independently by the documented keys and
requires the two to agree, and requires each to survive a permutation of
the input rows.
R CMD check --as-cran now passes under LC_ALL=C as well. Examples,
the vignette build and the full test suite all run there, and the result is
the usual host artefacts and nothing else. It had never been tried, and the
four test failures the previous entry describes are what it found.
Four tests and six warnings were a latent red on a non-UTF-8 checking
flavour. Nothing had ever run the suite under LC_ALL=C: the earlier
locale sweeps compared verb output, which is locale-proof, and
R CMD check runs in whatever locale the machine has. Doing it gives 4
failures and 6 warnings, and not one of them is the package. R
transliterates a string it cannot represent natively on the way into a
symbol and on the way out of deparse(), so rlang::sym("<glyph>")
becomes the symbol <U+0001F602> and as.character(list(c("a", "<glyph>"))) deparses to the escape; three tests were measuring that
rather than anything here, including the README render, since pillar
prints an ASCII tibble there too. The three are now gated on
l10n_info()$"UTF-8" with the reason given, the two AST scans that
parse() every test file no longer report R's "unable to translate"
warning against whichever test is running, and the three marker characters
the README comparison needs are built with intToUtf8() instead of a
\uXXXX escape, which is what emitted those warnings at parse time before
any test had started. Under LC_ALL=C the suite is now 14074 pass, 0 fail,
0 warn, 3 skip.
?emoji_position never documented the convention its own tests called
documented. A text that collapses to a single position, meaning one emoji
and nothing else at all, cannot tell its start from its end, and
.emoji_rel_position is 0 there. The test asserting it said "documented
as 0" and the help page said so nowhere. It matters for the one filter the
column exists for: .emoji_rel_position > 0.8 finds sentence-final emoji
and skips every emoji-only message, which is the commonest shape in a chat
or reaction corpus. The page now states the convention, points at
emoji_ratio()'s .emoji_only for finding that family, and records that
one character either side settles the ambiguity (a trailing space scores
the emoji 0, a leading one 1). Pinned for a one-code-point glyph and a
ZWJ family, since the convention counts positions rather than code points.
The reproducibility claim behind the bundled data is now verified. Every
dataset's @source names the data-raw/ script that builds it and the
vignette says they "are regenerated from the current Unicode emoji list by
the scripts in data-raw/", but nothing had ever re-run them. All four
reproduce their .rda byte-for-byte against emoji 16.0.0:
emoji_unicode_crosswalk (5761 rows), category_unicode_crosswalk (10),
emoji_sentiment_lexicon (969) and emoji_emotion_lexicon (150). The two
crosswalks derive purely from emoji::emojis, so a skip_on_cran() test now
rebuilds them and requires an exact match -- which will also flag the
datasets as stale the next time emoji ships new glyphs.
data-raw/emoji_emotion_lexicon.R cited next_release.md §4.1 for why the
lexicon is keyed on a selector-stripped code point. That file is a planning
document rewritten each release, and §4.1 is now about skin-tone modifiers,
so the pointer had gone stale. The reason is stated in the script instead.
Two dependency arguments the package relies on had no declared version floor, and one of them would have stopped the vignette building. Found by running the suite on R 4.1.0 -- the version DESCRIPTION declares as the minimum -- and on R 4.6.0.
lifecycle::deprecate_warn(env=, user_env=) does not exist in lifecycle
1.0.0; it arrived in 1.0.3, whose release notes are exactly the
caller-environment work those arguments were added for. Every
grouped-input call in the package goes through .emoji_warn_grouped(),
so on an older lifecycle a dozen verbs would have failed with "unused
argument". Now lifecycle (>= 1.0.3).readr::read_csv(show_col_types=) arrived in readr 2.0.0. R 4.1.0 here
ships readr 1.4.0, where the introduction vignette fails to build and
the corpus regression test errored instead of skipping -- its guard
checked that readr was installed, not that it was new enough. Now
readr (>= 2.0.0) in Suggests, and the test gates on
minimum_version = "2.0.0".
The suite now passes on R 4.1.0, 4.4.1 and 4.6.0.Two tests guard the class. One walks the AST of every R/ file and checks
that each named argument passed to a pkg::fun() call is a formal of that
function, so an "unused argument" cannot hide until someone runs against the
version that lacks it. The other holds a table of the five arguments the
package would break without -- lifecycle's env/user_env, dplyr's
relationship and .locale, tidyr's delim, readr's show_col_types --
and requires DESCRIPTION to declare a floor at least as high as each one
needs.
?emoji_unicode_crosswalk and the vignette both called it "one row per
emoji name", and a join by that column silently duplicates rows. The
mapping is many-to-many in both directions: 5761 rows cover 4698 distinct
names and 4853 distinct glyphs. A glyph repeats for every alias it has (the
documented direction), and a name repeats for every spelling of the emoji it
names -- 973 do, because the qualified and unqualified forms are separate rows
sharing one alias, so A_button_blood_type_ names both U+1F170 U+FE0F and
the bare U+1F170. Both places now give the row unit as a (name, glyph) pair
and say to join on key, which collapses the spellings. A test pins all three
counts and that a repeated name never spans two different emoji.
The introduction vignette said two time verbs need no timestamp; only one
does. emoji_version_profile() runs without a time column;
emoji_adoption_lag() uses the same version lookup but still has to find each
glyph's first appearance, so it requires one -- as the vignette's own next
sentence says two lines later. The claim came from misreading emoji-time.R's
"two of these are almost free", which meant needing no new dataset. Both the
sentence and the comment that caused it are fixed, and a test derives the
count from the verbs' formals.
The vignette no longer teaches a superseded function. It reached for
tidyr::separate_rows(), which tidyr marks [Superseded] and says will
"only receive critical bug fixes", pointing at separate_longer_delim()
instead; since the vignette is the package's primary teaching document, that
is the call a reader copies. Switched, with tidyr (>= 1.3.0) now declared to
match -- the same release vintage as the existing dplyr (>= 1.1.0) floor. A
test scans the vignette, the README and every R/ file for fourteen
superseded tidyverse calls so none creeps back, and the vignette's overview
table is asserted to name every export bar the soft-deprecated
emoji_tweets().
emoji_ambiguity()'s entropy column held negative zero on 166 of its 969
rows. -sum(1 * log(1)) is -0 for a glyph its annotators were unanimous
about. It compares equal to zero and prints as 0, so nothing downstream was
wrong -- but sprintf("%.3f", x) renders it -0.000, and formatting a table
for a paper is the commonest thing anyone does with that column. An entropy
of -0.000 reads as a bug in the package.
?emoji_ambiguity never warned that the head of its ranking is an artefact
of thin evidence. The lexicon's annotation counts run from 1 to 14622 --
median 18, and 69% below 50 -- and entropy, Gini and neutral share are shape
statistics that do not care how many annotations produced the shape. Three
annotators splitting one-one-one score the maximum entropy of log(3), which
is why five of the rows tied at rank = 1 have 3, 3, 3, 9 and 15
annotations, and why 11 of the top 20 have fewer than 50. So
head(emoji_ambiguity()) -- the help page's own first example -- returned
box-drawing characters and arrows, which reads as the function being broken.
The details now say to filter on n_annotations first (as the introduction
vignette already did), and the example demonstrates the filter. They also say
where the bias is not: entropy is positively correlated with the annotation
count overall (Spearman 0.56), because a thinly annotated glyph is usually
unanimous. What three annotators can do that thousands cannot is hit the
exact maximum. "ci_width" is named as the measure that accounts for thin
evidence by construction, being a Wald interval that scales as 1 / sqrt(n)
at a given spread -- its marginal correlation with the count is only -0.19,
so the formula is the claim, not the correlation. Behaviour is unchanged.
?tidyEmoji told readers to supply "the emoji-presentation (qualified)
form" to fix an undetected glyph, and for 12 emoji that advice does not
work. The variation selector does not always go at the end. For 200 of the
212 catalogue spellings detection skips, appending U+FE0F repairs them; the
exceptions are the keycap sequences -- #, * and 0 to 9 followed by the
enclosing keycap mark U+20E3 -- where the selector belongs between the
two. U+0031 U+FE0F U+20E3 is detected; U+0031 U+20E3 U+FE0F is not, so a
reader who appended got the same zero count and no hint why. The page now
gives the rule that repairs all 212 (insert U+FE0F after the first code
point) and names the exception. It also now states the 212 figure alongside
the existing 216, since those count different things -- spellings that are
themselves undetectable, versus spellings that break when the selector is
dropped -- and an unequal pair of numbers about the same subject invites one
being "corrected" into the other.
Detection is now tested in context, not only on bare glyphs. Every one of the 5042 catalogue spellings is checked beside words, punctuation, brackets, a bare letter, an orphan joiner, a stray selector and another emoji on either side: the glyph count never changes, and next to a second emoji the split is exactly the two of them (bar the 212 undetectable spellings, which contribute nothing). Span well-formedness -- ordered, non-overlapping, inside the string, and slicing back to the glyph -- is asserted over five string shapes. No defect found; the invariant every verb rests on simply had no test outside the bare case.
One infinite text_score made every row's mismatch result infinite.
scale = "zscore" takes sd() of the column, which is NaN as soon as any
value is infinite; the degenerate branch then subtracted an infinite mean, so
the whole column came back Inf or NaN. Measured on 100 rows with a single
Inf: 0 of the 60 scorable rows had a finite gap. A non-finite
text_score is now treated as missing -- which is what NaN already was,
since na.rm = TRUE drops it -- and reported, in the style of the
unreadable-date warning. The damage stays on its own row: the same 100 rows
now give 59 finite gaps under all three scales.
?emoji_emotion's @return said the long form carries .emoji_emotion and
.emoji_score "instead of the eight columns", which reads as a promise that
.emoji_n and .emoji_n_scored survive. They do not -- long = TRUE
returns neither. That matters because the same help page opens by telling you
to read .emoji_n_scored alongside .emoji_n before concluding a corpus
carries no emotion, advice a long-form user could not follow on a lexicon
covering 4% of emoji. The @return now says the counts are dropped and where
to get them. Behaviour is unchanged: the long form has had this shape since
0.3.0.
Three claimed equivalences now have the differential tests they never had.
.emoji_window_at() slices a bounded piece of the masked text rather than
the whole side, on the argument that only the nearest window tokens can
matter; .emoji_final_glyphs() walks back over blank gaps rather than
searching for the longest qualifying suffix directly. Both are now compared
against a naive implementation over randomly generated text (thousands of
comparisons each, across window 0 to 13 and both unit values), and the
eight worked examples in ?emoji_incongruity's "ends the text" paragraph are
asserted one by one. No defect found in either -- but neither was tested, and
an optimisation that is only nearly equivalent is the kind that survives a
release.
scale = "rank" and scale = "zscore" in the text-emoji mismatch verbs
were scaled over the wrong population, and reported mismatch that was not
there. Each side was ranked (or standardised) over its own non-missing
rows. text_score is normally present on every row, while the emoji score
is missing wherever a row has no scorable emoji, so the two percentiles
referred to different populations and the emoji-free rows -- which
contribute nothing to the output, their own gap being NA -- shifted the
answer for the rows that do. On 100 rows whose emoji sentiment equals their
text score exactly, so the true gap is 0 throughout, adding 100 emoji-free
rows with higher text scores moved the mean .emoji_incongruity to +0.50
on the rank scale and +0.92 on the z-score scale, and made
.emoji_incongruent flag 54 of the 100 as mismatched. Both scales are
now computed over the rows the comparison is defined on, so subsetting to
the scored rows gives the same numbers as calling on everything.
emoji_congruence() and emoji_incongruity_profile() share the engine and
are fixed with it. scale = "none" was never affected and is unchanged --
which is why this survived: 34 of the 40 scale = arguments in the test
suite were "none", and the six that were not asserted only bounds, NA
placement and the degenerate one-row branch, never a value against a known
truth. That is now tested.
?tidyEmoji's naming contract described two classes of output column and
the package has three. Five dotted names are not .emoji_* --
.row_number, .position, .period, .period_prev and .period_label --
and they are the ones a reader is most likely to meet first, since
emoji_extract_unnest(), emoji_context(), emoji_dfm() and the three time
verbs lead with them. The page now names all three shapes, says the
structural-index list is complete, notes that emoji_dfm()'s column names
are glyphs, and says that the reserved-prefix rule applies to the text column
too if you happen to have named it .emoji_n. A test asserts no verb invents
a fourth dotted name and that the page lists the five.
The same contract said the overwrite was always harmless, and twice it is
not. It illustrates the reserved-prefix rule with emoji_sentiment() then
emoji_position(), where both write .emoji_n and "both mean the same
thing". Two shared dotted names do not. .emoji_n_scored counts the emoji
that verb's lexicon could score, and the lexicons cover different emoji:
U+203C U+FE0F scores 0 under emoji_sentiment(), emoji_score(),
emoji_risk() and emoji_incongruity() and 1 under emoji_emotion(), so
the natural chain of the first and the last leaves a count describing one
lexicon beside a sentiment from the other. .emoji_sentiment averages every
emoji in the row from emoji_sentiment() but only the trailing run from
emoji_incongruity(where = "final"). Both are now named in the contract,
with the remedy, and a test pins the divergence in both directions as well
as the fact that .emoji_n really is the same measurement in all eight
verbs that write it.
emoji_search(NA_character_) failed with R's own "missing value where
TRUE/FALSE needed" instead of the verb's own message. nzchar() defaults
to keepNA = FALSE, which reads NA_character_ as the two-character string
"NA", so the non-empty guard passed and the NA propagated to an internal
if (!any(hit)). The guard now tests is.na() first, as the other
string-argument checks in the package already did.
?text_to_emoji gave the right round-trip figure with the wrong reason.
It said the emojize/demojize round trip returns "identical bytes for the 79%
that were already fully qualified". The 79% is correct (4002 of 5042
entries), but those are not the fully-qualified entries, and stating it that
way invites the number being "corrected" to the fully-qualified share
(3790/5042 = 75%). Which spelling comes back is decided by the shortcode
table, not by the input's selectors, so the two sets differ in both
directions: 212 unqualified entries keep their bytes -- the bare heart
U+2764 among them, because that is the spelling :heart: maps to -- and 92
already-qualified ones do not, such as the man detective
U+1F575 U+FE0F U+200D U+2642, which comes back with a second selector on
the gender sign. The details now say which question is which, add the
guarantee that matters -- the 1040 differ by U+FE0F alone, never by more --
and point at ?emoji_sanitize, which already tabulated both denominators
correctly. Every figure and both worked examples are pinned to the data.
?emoji_search listed 11 of the 17 strings that resolve differently
through as_emoji() than through text_to_emoji(), punctuated as if that
were the whole set. It now names all 17 and says so; camel, satellite
and train were missing from every list in the package. The set is derived
from the data in the tests, so a new emoji release cannot leave it stale.
?emoji_search promised that its shortcode column "recovers every row
exactly" through text_to_emoji(), and for some rows there is nothing to
recover it from. 189 of the catalogue's 5042 rows carry no GitHub-style
alias, so emoji_search() reports shortcode = NA for them -- 7 of the 198
rows emoji_search("face") returns, for one. Neither the details nor the
@return said the column could be NA; the @return had gone out of its
way to say that keyword is the empty string rather than NA, which made
the omission read as a guarantee. Both now say so, and point at the emoji
column or as_emoji_shortcode() -- which does answer for all 189, borrowing
the alias of the glyph's other spelling -- as the way through.
as_emoji_shortcode() was documented as returning the glyph's "first
shortcode", and for 175 of the catalogue's 5042 rows it does not. It
returns one shortcode per emoji, resolved through the codepoint key -- the
first alias of the fully-qualified (RGI) spelling -- which is what makes it
agree with emoji_to_text() and survive a round trip. 344 keys carry a
different first alias on each of their two spellings, so
as_emoji_shortcode("\u2764") is "heart" where the bare U+2764 row's
own first alias is "red_heart". emoji_search() reports the other one, the
matched row's own alias, since a search result is a row -- so the two verbs
can disagree about the same glyph. Both resolve back to the same emoji
through text_to_emoji(), so nothing was broken, but neither help page said
which it gave. Both now do, and the 344/175 split is derived from the data in
the tests. No behaviour changed.
?emoji_score and ?emoji_emotion now state the .emoji_n_scored
convention that ?emoji_sentiment and ?emoji_risk already spelled out:
0 means the row had emoji the lexicon could not score, NA that it had no
emoji to score. All five verbs behaved this way; two of them did not say so.
?emoji_ratio now covers empty text, which is neither of the two cases it
described: .emoji_ratio is NA (no characters to take a share of) while
.emoji_only is FALSE (an empty string is not a row of emoji).
Whitespace-only text, by contrast, has a real ratio of 0.
?emoji_pairs now says what sort = FALSE does. It sorts by item1 then
item2 in the C locale -- a fixed order, not the order the pairs happened to
be counted in -- which the parameter name alone did not suggest.
Grouped data frames no longer break six verbs, and no longer lose their
grouping in the rest. Two independent bugs, found by calling every verb on
a grouped input:
emoji_to_text(), text_to_emoji(), emoji_sanitize(), emoji_context()
and emoji_collocations() resolved their text column with
dplyr::select(), which silently re-adds the grouping columns, so a grouped
data frame made the selection return two names and the call failed with
`text` must select exactly one column -- an error blaming an argument
the user had got right. And the verbs that work a row at a time returned
tibble::as_tibble(data), which strips the grouped_df class, so
group_by(author) |> emoji_sentiment(text) |> summarise(...) quietly
collapsed to one corpus-wide row instead of one row per author. Those verbs
now carry the grouping through, as dplyr::mutate() and dplyr::filter()
do; the three that rewrite the text column in place re-derive the group
indices if the user grouped by that column. See ?tidyEmoji for the
contract.
Seven cross-row aggregators were silently pooling grouped input.
emoji_version_profile(), emoji_trend(), emoji_turnover(),
emoji_seasonality(), emoji_adoption_lag(), emoji_collocations() and
emoji_incongruity_profile() had no grouped-input guard, so a user grouping
by author, platform or date got a corpus-wide answer that looked like a
per-group one. All seven now warn. emoji_cooccurrence() and
emoji_flag_ambiguous() warned under the name of the verb they delegate to
(emoji_pairs(), emoji_frequency()) and now warn under their own. The
guard lives in one helper rather than being copy-pasted into each verb --
that copy-paste is why the seven were missed -- and the helper reports the
warning against the caller's frame, so it no longer appends "Please report
the issue" to a warning about the user's own data.
A missing, ambiguous or misspelled column argument now names the argument
the user actually wrote. Every verb resolved its column with
dplyr::pull(), whose errors are phrased in terms of var -- a formal of
pull() that appears in no tidyEmoji signature. emoji_sentiment(df) said
`var` is absent but must be supplied, emoji_trend(df, text) said
the same about the missing time, a two-column selection said
`!!enquo(var)` must select exactly one column, and a typo was
reported as object 'txet' not found, as if the user's own code had a free
variable in it. All 38 call sites now go through one resolver: the messages
name text, time, text_score or doc_id, and a misspelling is reported
as Column \txet` doesn't exist`.
emoji_extract_nest() accepted a missing text column. It resolved
{{ text }} in the data mask rather than as a column selection, so
emoji_extract_nest(df) returned a bogus empty list-column instead of
erroring, and a misspelled column was not caught either. It now uses the
same resolver as every other verb, and still returns data with its class
and grouping intact.
The package's own help page was stale relative to DESCRIPTION: the
paragraph describing interpretation risk, context, time, incongruity and the
sanitiser was missing from ?tidyEmoji. The build no longer ships the
.claude directory, which R CMD check --as-cran flagged as a hidden file
included in error.
Zero-row input now really does return a typed zero-row tibble. Five
verbs built their output columns with ifelse(), which takes the result's
type from its arguments, so emoji_density(), emoji_ratio(),
emoji_faceness() and emoji_risk() returned logical columns where a
populated call returns double or integer, and emoji_tokens() returned
an unspecified .emoji instead of character. Splitting a corpus, mapping a
verb and binding the pieces back therefore produced a different schema
depending on whether any chunk was empty. The values are unchanged.
A fractional count errors instead of being silently truncated.
top_n_emojis(n = 2.5) returned two rows, emoji_context(window = 2.7) used
a window of two, emoji_ngrams(n = 2.9) built bigrams: in each case the
number the user wrote was not the number that was used -- the same failure
mode as the head(n = -1) this release already caught, in the other
direction. n, top_n, window and min_n now require a whole number;
NULL and Inf keep their "all of them" meanings where they had one.
emoji_ngrams(sep = ) is checked too: it reached paste(collapse = ), which
failed with invalid 'collapse' argument and an internal call stack for NA
or a number, and silently used only the first element of a longer vector.
emoji_extract_nest() now returns a tibble, like every other verb.
?tidyEmoji promises "every verb ... returns a tibble", but this was the one
row verb that did not route its output through the shared helper, so a plain
data.frame in gave a plain data.frame back while the other seventeen
returned a tibble -- which also meant a list-column printing badly instead of
as <list>. A grouped input still stays grouped. The whole output contract is
now asserted across every verb at once rather than verb by verb.
Test line coverage went from 96.6% to 99.4%, and closing the gap found four
documented features that no test exercised: emoji_score() with the
"emotag1200" lexicon (the mean over its eight emotion dimensions),
emoji_sentiment(lexicon = ) given a data frame or a registered lexicon,
emoji_trend(by = "quarter"), and sort = FALSE on the relational verbs.
Also now covered: the lexicon aliases "sentiment" and
"emoji_sentiment_lexicon", a factor time column, the degenerate branches
of the rank and z-score rescalings, emoji_incongruity_profile()'s zero-row
return, and every argument-validation error on the lexicon surface -- each of
which had been checked by hand in an earlier round and never written down.
The nine lines still uncovered are guards whose callers validate first.
commonmark and xml2 join Suggests. The test that checks NEWS.md
parses calls utils::news(), and R's Markdown news reader calls both of them
unguarded -- packages that were present in the development library only
because roxygen2 and testthat pull them in. Declaring them means CI installs
them and the test runs, rather than skipping everywhere but the maintainer's
machine. NEWS.md's version headings are now also checked directly,
without parsing Markdown at all, so that invariant holds even where the
reader is unavailable.
DESCRIPTION declares Language: en-GB. The field was missing, so
spelling::spell_check_package() defaulted to en-US and flagged 55 correct
British spellings -- licence, normalised, analysed, summarise,
behaviour, neighbouring -- as errors, which made the check unusable as a
gate. The prose was already consistently British: across ten British/American
word pairs, the R sources, help pages, vignette, README, NEWS and
cran-comments.md contain zero American spellings. The only American
tokens are the exported names emoji_categorize() and emoji_sanitize(),
which follow R convention and stay.
A new inst/WORDLIST records the 84 remaining terms the dictionary cannot
know -- author surnames, package names, and vocabulary like codepoint,
grapheme, shortcode, keycaps, ZWJ, Plutchik, idf. With the
language declared and the wordlist in place, spell_check_package() now
reports zero, so a future typo is visible instead of buried in 139 lines of
false positives.
The test suite now passes in a non-UTF-8 locale. Ten test files carried 114
literal non-ASCII characters inside string literals -- zero-width joiners,
gender signs, hearts, the no-break and ideographic spaces -- and R parses a
source literal byte-wise under LC_ALL=C, so each became a run of
replacement characters and every fixture built from one silently tested the
wrong string. All 114 are now \u / \U escapes, and the suite gives
identical results under LC_ALL=C and a UTF-8 locale. Two new tests keep it
that way: one asserts R/ is pure ASCII (the invariant 0.4.0 introduced so
the PDF manual builds, checked by hand until now), the other that test string
literals stay escaped. The package's own detection was never affected --
verified by re-running the whole-catalogue sweep under LC_ALL=C, which
gives exactly the same 4830 of 5042 as a UTF-8 locale.
The test suite no longer attaches dplyr. One test file called
library(dplyr), which in a single testthat session leaks into every
alphabetically-later file and masks filter(), lag(), intersect(),
setdiff(), setequal(), union() and testthat::matches() for all of
them -- so nine set-operation call sites were resolving to dplyr's generics
purely because of filename order. They worked, but by accident. The attach is
gone, those calls are base::-qualified, and the suite's behaviour no longer
depends on the order its files happen to sort in.
as_emoji() and text_to_emoji() accept every GitHub alias, not just
the primary one each emoji is listed under, and that is now tested. The
reference table keeps only an emoji's first alias as its shortcode, so 751
of the 4698 aliases -- "grinning_face", "satisfied",
"face_with_tears_of_joy" -- resolve solely through as_emoji()'s fallback
to emoji::emoji_name. Nothing exercised that path.
emoji_incongruity(threshold =) is documented as the gap "at or above which"
.emoji_incongruent is TRUE, and the boundary is now tested. A row sitting
exactly on the threshold is classified as incongruent; nothing had pinned
that, so the comparison could have drifted to a strict one and silently
reclassified every borderline row.
emoji_search()'s three search fields are now tested separately. It is
documented to match against keywords, name and shortcodes, and some
queries match on one field only -- "grinning_face" and "thumbsup" appear
in no name (names use spaces, not underscores) and in no keyword -- so a
regression that dropped the shortcode field would have gone unnoticed. Also
pinned: the search is case-insensitive, a query matching nothing returns a
typed zero-row tibble, and the +1 shortcode's regex metacharacter is safe.
emoji_dfm()'s column order is now guaranteed independent of the order the
rows arrive in. Columns are sorted by descending total count with the glyph
as tiebreak; without that tiebreak, tied columns fell back to the order the
glyphs happened to appear in the data, so the same corpus sorted differently
produced a differently-ordered feature matrix. The behaviour was already
correct -- this release pins it, along with row-order independence for
emoji_frequency(), top_n_emojis(), emoji_version_profile(),
emoji_pairs(), emoji_cooccurrence() and emoji_collocations().
?emoji_emotion_label now documents that ties are broken in Plutchik order,
so the winning emotion is deterministic and does not depend on a row's
position in the data -- it was a code comment only. The help page also points
at emoji_emotion() for the profile the label collapses, since a near-tie is
invisible in a single winning name.
The introduction vignette's "design choices" list now mentions how grouping
composes, which became a design choice in this release and was documented
only on ?tidyEmoji.
?emoji_incongruity now says what "ends the text" means for
where = "final". Only whitespace may follow the last glyph, so
"great \U0001f602" has a final run and "great \U0001f602." does not -- a
trailing full stop, bracket or quote mark disqualifies it, which is easy to
meet unaware in a punctuated corpus. The behaviour is unchanged; it was
simply not stated.
A POSIXct time column was bucketed by its UTC day, not its own.
as.Date() on a date-time converts in UTC whatever the object's tzone
says, so an emoji posted at 23:30 New York time was counted on the next
calendar day -- and for an evening-heavy corpus, systematically so. It also
made the package disagree with itself: emoji_seasonality(period = "hour")
reads format(x, "%H") and had always used the timestamp's own zone, so the
same row could be hour 23 and the following day at once. emoji_trend(),
emoji_turnover(), emoji_adoption_lag() and
emoji_seasonality(period = "month" / "weekday") now all take the calendar
day the timestamp displays as. The session's TZ does not affect any of
them, before or after.
A time column of strings silently shrank the corpus. A value that would
not parse as a date became NA, and the time verbs then dropped the row --
indistinguishable, in the result, from a row whose date was genuinely
missing. emoji_trend(), emoji_turnover(), emoji_seasonality() and
emoji_adoption_lag() now say how many values were unreadable and show the
first one; a real NA still passes without comment, and a column where
nothing parses still errors.
?emoji_sanitize now documents which policies can be undone. The five
policies were presented as five parallel options; they are a ladder of
information loss, and how far down it you stepped only became visible when
you tried to restore the emoji after a model call. The help page gains the
table: "shortcode" is the only rewriting policy that round-trips (losslessly,
including skin tones, flags, ZWJ sequences, keycaps and U+FE0F forms),
"placeholder" keeps where but not which, and "strip" keeps neither.
232 of the catalogue's 2501 zero-width-joiner sequences were detected as
their component emoji rather than as one emoji. The repair that rejoins
sequences the upstream regex does not know required the gap between two
matches to be exactly one joiner. In a sequence whose middle component is a
text-presentation code point, that component is not matched either, so the
gap is ZWJ + component + ZWJ and the rule declined. The damage was a
wrong count, not a missing one -- 🚶 U+200D U+2640 U+200D
U+27A1 U+FE0F, "woman walking facing right", arrived as two emoji,
"person walking" and "right arrow", neither of which the text contains, and
most of the 232 are 2023-2024 additions. The repair now also merges when the
gap holds a joiner and the union of the two spans is itself a catalogued
emoji: exact, so it can only join code points that really do spell one emoji,
and the original rule still covers everything the catalogue has not heard of.
2499 of the 2501 now detect as one glyph; the two that do not are unqualified
spellings whose every component needs U+FE0F, and their qualified forms are
found. This changes .emoji_n, frequency tables and every count derived from
them for corpora containing these sequences.
A second, subtler form of the same defect needed a second rule. Both merge
rules need two matches to work with, and a sequence whose only detectable
component is one of its parts yields a single match -- so there was no pair to
merge and the sequence arrived as that part. U+2764 U+200D U+1F525 ("heart
on fire") with its selectors omitted read as U+1F525 ("fire"), and
U+1F9D4 U+200D U+2642 ("man: beard") as "person with beard". Counting
glyphs cannot see this -- one match is still one glyph -- so the test for it
looks for a joiner left outside every detected span. A lone match beside
such a joiner is now grown outwards while the span stays a catalogued emoji,
bounded by the longest catalogued emoji (10 code points), never crossing a
neighbouring match, and skipped entirely unless the string actually has an
orphaned joiner.
Together the three rules take exact detection of the reference table from 63.2% to 95.8% (3189 of 5042 spellings read as exactly one emoji equal to the whole spelling, against 4830 now) and orphaned joiners from 1643 to 2 -- the two being spellings with no detectable component at all, both of which have a canonical form that is found. Staged, so each figure is attributable: rule 1 alone, which is UAX #29's GB11, gives 75.5% and leaves 1025 orphaned joiners; rule 2 leaves 793; rule 3 takes those to 2. Well-formed text costs about 6% more; text that was broken now costs more and is right.
emoji_dfm() names one column per emoji glyph, so a doc_id column named
with one of those glyphs was overwritten by the count column and the document
identifiers vanished without a word. It is now an error that says which
column to rename.
emoji_categorize() silently dropped rows that do contain emoji. It
filtered on .emoji_category being non-NA, and that column is NA for two
different reasons: the row has no emoji, or the row's emoji are not in the
reference table. The second case is real and grows with every Unicode
release -- detection is grapheme-aware, so a zero-width-joiner sequence newer
than your installed emoji is found as one emoji but cannot be
categorised -- and those rows vanished from the result. 0.2.1 fixed one
instance of this (a U+FE0F-qualified heart went missing) by repairing that
particular join; the conflation behind it survived. The filter is now on
"contains at least one emoji", so such a row is kept with
.emoji_category = NA, and nrow(emoji_categorize()) now always equals
nrow(emoji_filter()).
emoji_risk() treated the same row two ways. A row holding emoji the
ambiguity lexicon cannot score -- anything added to Unicode after 2015 --
correctly got .emoji_n_scored = 0, but .emoji_n_ambiguous = NA, where the
count of ambiguous glyphs found is genuinely zero. The @return had promised
NA only for rows with no emoji at all, so the documentation was right and
the code was wrong. .emoji_ambiguity_mean and .emoji_ambiguity_max stay
NA there, because there is nothing to average.
emoji_key(), the codepoint key every glyph-to-metadata join goes through,
had two "no key here" values: NA for empty or missing input, but "" for a
string made of nothing but variation selectors. Every consumer had to
remember to filter both. It is now NA in all three cases; a lexicon row
whose glyph is a stray U+FE0F is ignored rather than keyed on "".
?emoji_sanitize's reversibility claim is now measured rather than
asserted, and the measurement is stronger than the old wording: for all 3790
emoji in their canonical spelling -- the spelling a keyboard emits -- the
"shortcode" round trip returns the original text byte for byte, 100% of
the time. Feed it one of Unicode's shorter spellings, with the U+FE0F
selectors omitted, and it returns the canonical one instead; across all 4853
catalogued spellings that is 79.5% byte-identical, with every difference
being U+FE0F alone and never more. Tests assert the exact claim rather than
a byte-identity threshold, because that rate falls each time detection
improves -- a spelling that used to fragment now merges and normalises.
The lexicon coverage ceilings are now on the help pages, with the right
denominator. emoji_emotion() can score 150 glyphs -- about 4% of the
3790 distinct emoji tidyEmoji can detect -- and emoji_sentiment() about
19%, with nothing added to Unicode after 2015. A user who does not know
that reads a column of NA as "no emotional content" rather than "not in the
lexicon". Both verbs and both dataset pages now state the figure, name the
{emoji} version it is computed against, and point at .emoji_n_scored as
the per-row answer. (The internal roadmap had quoted 3.0% and 19.2% against
the reference table's 5042 rows; the table stores the qualified and
unqualified forms of an emoji separately, so the denominator is 3790 distinct
codepoint keys.)
?emoji_emotion and ?emoji_emotion_label named fewer columns than they
return: the eight emotion columns and the .emoji_n / .emoji_n_scored
counts were missing, and the completely different shape of long = TRUE
(one row per row per emotion) was not in the @return at all.
README.md is now byte-for-byte what README.Rmd renders to, so it cannot
drift from its source unnoticed.
The lexicon registry accepted two registrations it could never honour.
register_emoji_lexicon("novak2015", ...) succeeded, showed up in
emoji_lexicons() as a second row with the same name, and was then
unreachable, because a bundled name resolves before the registry is
consulted -- so lexicon = "novak2015" still got the bundled table. All six
bundled names are now refused with the list of them. And a lexicon with no
usable score column registered happily and only failed at first use, from
inside emoji_score(), in a message naming tbl -- an argument of a call
that had long since returned; the score column is resolved at registration.
An NA name is rejected too.
The detection contract is now on the help pages. Code points that are
emoji only when they carry U+FE0F -- the bare heart U+2764 being the one
people meet -- are not detected, which was documented on
?emoji_sentiment_lexicon but nowhere a user counting emoji would look.
?tidyEmoji gains a Detection section with the measured size of the
exclusion (1252 of 5042 catalogue emoji carry U+FE0F; 216 of those are
undetectable without it), the reason the default does not change (U+00A9,
U+00AE and U+2122 are in the same set), and the fact that it affects
detection only and never the U+FE0F-stripped join. The .emoji_* prefix is
documented as reserved: a verb overwrites a column of its own output name
without warning, which is what makes the verbs chainable.
emoji_dfm(doc_id = ) no longer orders its rows by the session's
collation. Documents were grouped with factor(), whose levels are sorted
with the locale's collation, so the row order of the result could differ
between machines -- the same class of bug 0.3.0 fixed for glyph ordering in
emoji_pairs() and the dfm's columns. Documents now appear in the order
their id is first seen in the data. emoji_pairs(),
emoji_cooccurrence() and emoji_dfm() share one grouping helper, so they
cannot drift apart again.
emoji_sanitize(policy = "strip") tidies only the whitespace left behind by
a removed glyph, and only on rows that actually contained one.
emoji_sanitize(policy = "strip") could leave an emoji behind. Deleting
a span makes the characters on either side of it adjacent, and on malformed
input those two spell an emoji the original text never contained: a bare
U+2603 beside an orphan U+FE0F becomes the qualified snowman, # beside
a stray U+FE0F U+20E3 becomes a keycap, and two regional indicators
separated by an emoji become a flag. Found by stripping 1500 randomly
assembled strings and re-detecting: 1 came back still carrying an emoji,
from the one policy whose whole promise is that they are gone. strip now
repeats until there is nothing left to remove, which terminates because
every pass shortens the row. The rescan is gated on the row still holding a
non-ASCII code point, so ordinary text pays nothing and a corpus of
accented words pays about 10%. "placeholder" was not affected, because a
non-empty token keeps the neighbours apart, and "name" and "shortcode"
are documented to leave an unnameable glyph in place, which
?emoji_sanitize now says in its own words rather than by reference. The
page is careful about which policies clear the column and on what terms:
"strip" does so whatever it is handed, "placeholder" does unless the
token you supply is itself emoji-forming (one that is an emoji survives,
and a lone U+FE0F binds to whatever the removed glyph stood next to),
and "name" and "shortcode" do not when the glyph has no name. All four
cases are pinned.
Arguments given nonsense now error instead of quietly returning a
different answer. An audit of every argument that reaches a base R function
without validation found one shape of bug repeated across the package: a
value the function cannot honour was absorbed rather than rejected.
emoji_to_text(wrap = ) now requires the {x} placeholder, since a template
without it replaces every emoji with the same literal string;
top_n_emojis(n = ) rejects a negative n, which head() had silently read
as "drop the last row"; and every TRUE/FALSE argument
(emoji_pairs(directed = , sort = ), emoji_cooccurrence(diagonal = ),
emoji_emotion(long = ), emoji_context(keep_text = ),
top_n_emojis(duplicated = ), emoji_sentiment(se = )) is now checked,
because isTRUE() reads every non-TRUE value as FALSE -- so
long = "yes" used to return the wide form without complaint. The deprecated
top_n_emojis(duplicated_unicode = "yes") still works: the check runs after
the lifecycle conversion.
emoji_score() gains the lexicon = "novak2015" default that
emoji_sentiment() always had, so calling it without a lexicon works instead
of raising a missing-argument error.
The reference-manual sources are ASCII throughout, so the PDF manual builds everywhere.
next_release.md, the repo's release ledger, gains a section 13 recording
what wave 1 shipped, the third-audit defect, and the design decisions locked
at implementation.
Three test assertions that were weaker than the behaviour they described are
now exact. Two counted the residual of the ZWJ segmentation work with
<= 2 -- the catalogued spellings that go undetected, and those that leave a
joiner orphaned -- where the true number is exactly two, and one bounded a
row count that is exactly two. An inequality in that position accepts a
regression back up to the bound without reporting it, and a count cannot see
the residual set change while its size stays the same. The two residual
spellings (U+1F441 U+200D U+1F5E8 and U+1F3F3 U+200D U+26A7) are now
named, with the reason they are irreparable asserted alongside them: neither
carries U+FE0F, and each one's fully-qualified sibling is detected as a
single glyph, so the residual is confined to the spelling and never reaches
the emoji. No package code changed.
emoji_search() and emoji_collocations() no longer change answer with the
session's locale. Both fold case before matching, and tolower() honours
LC_CTYPE: under a Turkish or Azerbaijani locale it maps "I" to a dotless
i (U+0131) rather than "i", so any query containing a capital I stopped
matching the catalogue's ASCII text. Measured under tr_TR.utf8,
emoji_search("FIRE") returned 0 rows instead of 27, "SMILING" 0 instead
of 23, "INDIA" 0 instead of 2, and "I" 37 instead of 4640;
emoji_collocations() counted "BIG" and "big" as two different words.
Matching now folds A-Z with chartr() before tolower(), which settles
ASCII deterministically while still folding non-ASCII, so a query for
"VICUÑA" or "O’CLOCK" keeps working. This is not a behaviour change in an
ASCII or Western locale: the new fold agrees with tolower() on all 5042
catalogue names, 10701 keywords and 5761 aliases. emoji_type()'s
group-to-type recode was checked and was never affected -- its subgroups are
already lower-case and none of the literals it matches contains an i.
Case-insensitive search missed accented names outside a UTF-8 locale.
The fold above settled A-Z with chartr() and left the rest to
tolower(), on the reasoning that tolower() "still folds non-ASCII". It
does -- but only where the locale is UTF-8. Under LC_ALL=C tolower()
returns every non-ASCII letter untouched, and 24 catalogue names carry an
uppercase accented letter: the Åland Islands, Curaçao, Côte d'Ivoire,
RĂ©union, SĂŁo TomĂ© & PrĂncipe, St. BarthĂ©lemy, TĂĽrkiye and the seventeen
Japanese button glyphs, whose names are quoted with typographic quotation
marks. So emoji_search("ĂĄland") found the flag in a UTF-8 session and
0 rows in a C one, and no error said why. The fold is now driven from an
explicit table throughout: A-Z plus the 200 cased Latin letters of the
Latin-1 Supplement and Latin Extended-A/-B, folded with chartr(), which is
a plain code-point substitution and so is subject to neither the Turkish
rule nor the locale's character set. tolower() still runs afterwards for
anything the table does not name. Only 1:1 pairs are listed -- an uppercase
code point whose lowercase form is a single, different code point -- which
is what keeps the result byte-identical to tolower() in a UTF-8 locale
rather than a new folding of its own; it was re-verified against
tolower() over all 5042 names, 10701 keywords and 5761 aliases, and the C
locale merely catches up. The invariant test that claimed the fold does not
depend on LC_CTYPE only ever exercised the Turkish half, so it passed
throughout; it is now joined by one that pins the accented answers by hand,
because tolower() cannot produce them in the locale where it matters.
A factor or numeric glyph column in a lexicon failed with R's message.
emoji_key() never coerced, so a by = column that was a factor reached
nzchar() -- whose message is only "requires a character vector" -- and a
numeric one
reached utf8ToInt(). Neither named the argument, the column or the verb,
and the second is the likelier mistake: pointing by = at the wrong column.
as_emoji_name() and emoji_ambiguity() already coerced; the lexicon path
was the one that forgot.
A code point that is not valid UTF-8 produced the literal key "NA".
utf8ToInt() answers NA_integer_, and NA != 0xFE0F is NA, so the
variation-selector filter kept it and sprintf("%X", NA) made the string
"NA" -- a real key that every undecodable value collided on, and that
consumers read as present rather than missing. It now returns NA_character_
like every other unkeyable input.
A data-frame lexicon carrying a column named type was silently
ignored. emoji_sentiment() tested lex$type before is.data.frame(lex),
and the lookup returns a user's data frame unchanged -- so a lexicon with a
type column reading "sentiment" was discarded and the bundled Novak
scores used in its place, with no message. $ partial-matches, so types,
type_of and type_label did it too. emoji_score() and emoji_emotion()
already branched in the safe order.
emoji_emotion() accepted a lexicon that emoji_score() refuses. The
duplicate-key check lived inside the sentiment path only, so a custom
emotion lexicon listing both U+2764 and U+2764 U+FE0F with different
scores gave an answer that changed when the caller reordered their own
table. One table now gets one answer whichever verb reads it.
emoji_emotion_label() returned NA when any one emotion was NA.
max.col() answers NA_integer_ for a row holding a missing value, so a
row that scored on seven emotions and was NA on the eighth came back
unlabelled even though its maximum was unambiguous.
A long ZWJ chain could lose its tail. .emoji_extend_zwj() wrote
for (e in (b + 1L):min(...)) without establishing that the sequence counts
upwards. Rule 1 merges arbitrarily long chains, so a span longer than ten
code points is reachable, and there the loop ran backwards over spans
inside the current match and shrank it. A twelve-code-point kiss sequence
joined to a grinning face came back as ten, with the trailing glyph counted
nowhere.
.emoji_category was ordered by first appearance. So
"Flags|Smileys & Emotion" and "Smileys & Emotion|Flags" were two values
for one combination, and count(.emoji_category) split every multi-category
row across as many groups as the corpus had orderings -- which is what the
introduction vignette does, one chart of which was wrong for this reason.
Sorted into the crosswalk's fixed order now, as emoji_type() already was.
emoji_sanitize(policy = "keep") skipped every content check. It
resolved the column's name, so a typo was caught, but never read it: the
"bytes"-encoding refusal, the list-column refusal and the one-value-per-row
check all belonged to the other policies. A guard one policy skips is a
guard the caller cannot rely on.
emoji_sentiment(se = TRUE) shrank the error when a glyph repeated. The
standard error divided by the occurrence count, so one emoji pasted twice
reported the same uncertainty as two independent glyphs and
strrep(emoji, 100) claimed a tenfold reduction -- for a row carrying
exactly as many annotations as the single-glyph row. Repeating a glyph adds
no annotations. It is now weighted over distinct glyphs, which leaves the
answer for genuinely distinct emoji unchanged to the last digit. The test
that should have caught it asserted the property over a fixture of one glyph
twice, the single case where the property is false.
emoji_seasonality() reported a structural zero as a measured zero. A
cycle level the data never reaches had emoji_per_text = NA and share = 0
in the same row -- two columns disagreeing about whether there was data.
share is NA there now, n_texts remains the discriminator. Every level
is returned so a bar chart has no invisible gaps, and a zero-height bar for
"no data" was exactly the gap that was meant to be visible.
emoji_score() returned its three columns in the reverse order of every
other scorer. It is the generic emoji_sentiment(), emoji_emotion(),
emoji_risk(), emoji_incongruity() and emoji_faceness() sit on, and the
only one putting the measurement before the counts, so selecting by position
gave different columns from the generic and the specific verb. Now
.emoji_n, .emoji_n_scored, .emoji_score, like the rest.
emoji_token_cost() truncated a fractional tokeniser result. A
tokeniser answering 2.6 was recorded as 2. It rounds up, matching the
built-in heuristic.
A doc_id list-column was deparsed instead of read. .emoji_col() had
no atomic guard, so emoji_dfm() and emoji_pairs() grouped documents by
their deparsed R source and two different ids that deparse alike merged into
one. This is the same hazard the text column was given a guard for in this
release; the other three column arguments now share it.
A future emoji release could have silently undone the version fix.
.emoji_fill_by_key() parsed the upstream version with a bare
as.numeric(), while every consumer strips a leading E first -- E17.0
being the spelling Unicode's own data files use. Had a release carried it,
every entry would have been NA, the fill would have done nothing, and the 1252
rows without a version this release exists to repair would have come back,
with nothing failing.
?emoji_ambiguity recommended ci_width as the remedy for thin
evidence, and it is not one. It is a Wald interval, so it is exactly zero
wherever the spread is zero: 166 of the lexicon's 969 rows report
ci_width = 0, on annotation counts running from 1 to 68. Ranked ascending
it puts the thinnest unanimous glyphs first, as the most certain rows in the
table. The page now says so, and still points at n_annotations.
emoji_score(lexicon = "emotag1200") returns an intensity, not a
valence. EmoTag1200's eight dimensions are each on [0, 1] and four are
negatively valenced, so their mean is non-negative and puts a maximally
angry emoji near a maximally joyful one -- while "novak2015" is a signed
valence on [-1, 1]. Both arrive as .emoji_score. The @return now names
both scales and says they must not be pooled.
?emoji_trend claimed a completeness it does not deliver. The grid is
complete over the observed periods, and "observed" means periods holding
at least one emoji: a period whose rows carry no emoji, and any calendar
gap, is absent. The three time verbs answer this differently on purpose --
emoji_turnover() keeps emoji-free periods, emoji_seasonality() returns
every level unconditionally -- and that was documented nowhere, so joining
two of them on .period quietly lost rows. All three policies are now
stated together on ?emoji_trend.
The Detection section only ever said what detection misses. It
quantifies that to four significant figures and said nothing about what it
falsely includes. Two things: an invalid regional-indicator pair such as
U+1F1FD U+1F1FD is one well-formed grapheme cluster and is counted as an
emoji with name = NA, and an orphan skin-tone modifier is counted as an
emoji and named, because the Component group is in the reference table.
Both are defensible as raw detection and misleading as a corpus statistic,
so both are now named, with as_emoji_type() != "component" given as the
filter.
The missing lifecycle badge. ?top_n_emojis expands
lifecycle::badge("deprecated") into a reference to
man/figures/lifecycle-deprecated.svg, which was not in the package: the
HTML help page and the pkgdown reference rendered a broken image. The PDF
manual takes the \ifelse else-branch, which is why the PDF-focused checks
never caught it.
The suite would have gone red on every CRAN flavour the day emoji
17.0.0 ships. Roughly seventy catalogue figures are pinned deliberately,
because the pinning is what catches documentation drifting from data -- but
emoji's version tracks the Unicode emoji version, DESCRIPTION sets no
upper bound, and Unicode Emoji 17.0 is already out. Those assertions are now
gated on the installed release matching the documented one, so the mismatch
is reported once, by one test, where the maintainer can act on it. Nothing
skips while emoji is 16.0.0.
The caches derived from the emoji reference table are now dropped whenever
that table is rebuilt. ref_keys and the type recode are computed from
emoji_reference(), each guarded only on is.null(), so nothing connected
the three: a rebuilt reference left the other two pointing at the previous
catalogue. ref_keys is the one that would have mattered, because the
zero-width-joiner repair tests membership in it, so a stale copy changes
detection rather than only metadata. One function now names the derived
slots, so a cache added later is invalidated by construction.
The bundled sentiment lexicon's two structural invariants are asserted, in
data-raw/ and again over the shipped data so a check runs them:
occurrences equals the three class counts summed, and no two rows
canonicalise to one code-point key. emoji_sentiment() reads occurrences
while emoji_ambiguity() recomputes the sum, and both resolve a duplicate
key first-wins and silently, so either violation would have made two verbs
disagree about the same glyph without raising anything. Both hold today; the
emotion script already asserted its analogues.
data-raw/ no longer keeps its own copy of the code-point key function. Two
verbatim copies had drifted from the real one -- neither had the guard that
keys a selector-only string to NA rather than "" -- so the bundled key
columns were generated by a function that no longer matched the one used to
look them up. The crosswalks still rebuild byte-for-byte.
Tests that register a lexicon now restore the registry when they finish. Eleven registrations were still live when the suite ended, which made results depend on file order, and one test coped by wiping the registry outright -- discarding every registration made before it and so depending on line order too. A source guard now requires every registering test to restore, which is order-independent in a way that inspecting the registry at the end cannot be.
The two bundled lexicons' licences were named in the documentation and their
notices never shipped. inst/LICENSE.note now carries the MIT text and
copyright line for EmoTag1200, the CC BY-SA 4.0 URI for the Emoji Sentiment
Ranking, and the Creative Commons declaration that makes CC BY-SA 4.0
one-way compatible with the GNU GPL version 3.
CI runs weekly as well as on push, and the matrix gained a job on R 4.1.0,
the declared minimum, which no job previously exercised. cran-comments.md
reports its check result per environment rather than as one headline its own
next paragraph contradicted.
"Nothing but whitespace" meant different things in different sessions.
The same defect as the fold above, in a second place. emoji_ratio()'s
.emoji_only is documented as "the text contains emoji and nothing else but
whitespace", and the test for it, the token split behind
.emoji_per_token, emoji_context()'s word split and
emoji_incongruity(where = "final")'s "only whitespace may follow the last
glyph" all asked that question with \s or [[:space:]]. Both resolve
through the C library's iswspace(), which depends on the locale and on
the platform. So an emoji followed by an ideographic space (U+3000) was
emoji-only in a UTF-8 session and not emoji-only under LC_ALL=C -- and
CRAN checks on flavours whose iswspace() need not agree with glibc's.
All six sites now use one explicit character class: Unicode's White_Space
property, written out. That also settles an inconsistency glibc had left
behind, since U+00A0 and U+202F -- the two no-break spaces -- were the
only White_Space characters it rejected, so U+3000 counted and U+00A0
did not; both count now. U+200B remains excluded, because despite the name
Unicode does not give the zero-width space White_Space. The bundled corpus
has one row containing U+00A0 and no value in it changes, so no figure
quoted anywhere moves. The test asserting that emoji_context() and
emoji_density() "tokenise identically" had reimplemented the density side
inline, so it compared the package against a stale copy of itself and failed
on the corrected code; it now reads the count back out of the verb.
emoji_version_profile() and emoji_adoption_lag() now resolve a Unicode
version for every emoji they detect. The upstream emoji::emojis table
records the introducing version on the unqualified member of a variation
pair and leaves it NA on the fully-qualified one, so 1252 of 5042 reference
rows carried no version -- among them everyday glyphs like the red heart
(U+2764 U+FE0F), the smiling face (U+263A U+FE0F) and the skull and
crossbones (U+2620 U+FE0F). Those all fell into
emoji_version_profile()'s version = NA bucket, which is precisely what
that verb exists to report, and emoji_adoption_lag() returned NA for
them. A version describes the emoji, not one spelling of it, so the reference
table now fills version across every glyph sharing a codepoint key -- the
same normalisation the rest of the package already applied to names,
shortcodes, types and lexicon joins. No key carries two different versions,
so this only ever fills gaps; every one of the 1252 is recovered and the
unknown-version row disappears for the full catalogue. Profile counts shift
accordingly (Emoji 0.6 goes from 656 to 719 types), and lag_days is now
computed where it previously could not be. A glyph with a genuinely unknown
version is still reported rather than dropped.
The ordering verbs now document how they settle ties, which several of them
enforced in code without promising. emoji_frequency() already said
"descending n with ties broken by the glyph so the order is deterministic";
top_n_emojis(), emoji_dfm(), emoji_ambiguity() and
emoji_flag_ambiguous() applied the same discipline but said nothing.
top_n_emojis()'s @return now records the sort order, that the glyph order
decides which side of the cut a straddling tie falls on, and that a corpus
with fewer than n distinct emoji returns all of them rather than padding.
emoji_dfm()'s records that emoji columns run in descending corpus total
with ties broken by the glyph, computed in the C locale so the column order
is safe to index by position and does not follow the session's collation.
Both ambiguity verbs now state that rank uses ties.method = "min": tied
glyphs share the lowest rank of their group and the next distinct value skips
ahead, so ranks are deliberately not consecutive. No behaviour changed.
?emoji_sentiment_lexicon stated its coverage inconsistently. It read "969
rows, covering about 19% of the distinct emoji tidyEmoji can detect (3790
distinct codepoint keys)", but 969/3790 is 25.6%. The 19% is the right
figure and the attribution was wrong: only 736 of the 969 rows resolve to a
detectable emoji, and 736/3790 is 19.4% -- which is also exactly how many of
the 3790 detectable emoji emoji_sentiment() scores. The paragraph now says
so, and its arithmetic is checkable against the sentence that follows it
(736 resolving + 233 absent = 969). The ?emoji_emotion_lexicon figure
needed no change: all 150 of its rows resolve, so 150/3790 = 4.0% was
already exact.
emoji_search()'s documentation now says how to get back from its
shortcode column to a glyph. The column is the emoji's first alias, and
passing it to text_to_emoji() recovers every row exactly, while as_emoji()
resolves a bare string by Unicode name first and so returns a different emoji
for the eleven strings that name one emoji and alias another (dog, cat,
cow, pig, tiger, mouse, rabbit, horse, whale, kiss,
sunglasses). That is the documented precedence rather than a defect, but
emoji_search() advertised its result as ready for "piping into other verbs"
without saying which path is safe. Its @return also now records that
keyword is the empty string, not NA, when the query matched the name or a
shortcode rather than a keyword.
emoji_lexicons() no longer returns columns carrying stray element names
after register_emoji_lexicon(). The registry is a named list, so the
lapply()/vapply() that built the dimensions and n columns returned
named results and dplyr::bind_rows() then padded the bundled rows with
"". The tibble printed normally, which is why this went unnoticed, but
extracting a column showed it: emoji_lexicons()$n printed a stray name
header above the values. Both columns are now unnamed, and a test asserts
that no verb returns a column with element names -- checked across the 42
exported functions that return a data frame, with a custom lexicon
registered.
The declared R dependency is raised from R (>= 3.5.0) to R (>= 4.1.0),
which is the oldest R the package can actually be installed on. The old value
was a promise it could not keep: the current dplyr and tidyr both require
R >= 4.1.0, and install.packages() serves only current versions, so a user
on R 3.5 or 4.0 got an opaque dependency-resolution failure instead of a clear
message about their R version. Nothing in the package's own code needed more
than R 3.5 -- the floor comes entirely from the dependency chain -- and no
behaviour changed. A test now checks the declared minimum against the
installed hard dependencies' own floors, since the CI matrix only reaches
oldrel-1 and so cannot detect this class of drift.
The introduction vignette described 68% of emoji-bearing entries as "the overwhelming majority" (and "the vast majority" in a figure's alt text) where the measured share carrying exactly one emoji is 68.2%. Both now say "about two-thirds".
text_to_emoji() no longer claims flatly to be the inverse of
emoji_to_text(). It is the inverse up to the presentation selector:
because both directions resolve through the U+FE0F-stripped code-point key,
an unqualified glyph and its fully-qualified form share one shortcode and the
fully-qualified form is what comes back. Feeding the whole catalogue through
and back preserves the key for all 5042 entries and the exact bytes for 79%
of them; a second pass changes nothing. The details now say so, and say to
compare with emoji_key() rather than string equality. They no longer
attribute that 79% to the entries "that were already fully qualified": the
two sets differ -- 212 unqualified entries keep their bytes and 92 qualified
ones do not -- because which spelling comes back is decided by the shortcode
table, not by the input's selectors.
as_emoji()'s resolution order is now documented. It accepts names and
shortcodes in one argument, and 464 strings belong to both namespaces -- the
exact Unicode name of one emoji and a shortcode alias of another. It has
always tried the exact Unicode name first, so as_emoji("dog") is the emoji
named "dog" (U+1F415) rather than the one aliased :dog: (a dog face,
U+1F436). For 17 of the 464 the namespaces disagree and as_emoji()
differs from text_to_emoji() by design, since a :dog: token is explicitly
a shortcode; both places now say which is which and how to ask for one
namespace specifically. No behaviour changed.
Emoji-dense rows no longer cost quadratic time. Five hot paths reached a
character offset with substr()/substring(), which rescans a multi-byte
string from its first byte on every call, so the work grew with the square of
the emoji in a row: glyph slicing (used by nearly every verb), the splice in
the translation verbs, the residual test in emoji_ratio(), the
trailing-run walk-back behind emoji_incongruity(), and the per-occurrence
window loop in emoji_context(), which handed the whole prefix and suffix to
a windowing function that only ever needed the few tokens next to the glyph.
The four that cut text around glyph spans now share one helper that converts
the string once and slices code points past a threshold, keeping ordinary
rows on the existing path; emoji_context() reads a bounded slice anchored
at the glyph. Measured on a row holding 6400 emoji: emoji_ratio() about
thirteen times faster, emoji_summary() and emoji_sentiment() about ten,
the trailing-run walk-back about seven, emoji_context() about four, and
every stage of the engine now scales linearly. Results are unchanged -- the
fast paths are pinned to the slow ones by tests that compare them directly,
and fall back whenever utf8ToInt() cannot represent the string.
README.md's emoji_ambiguity() line gained the annotation-count caveat
that ?emoji_ambiguity and the vignette already carried.
Test coverage for this release, in one place rather than scattered through
the notes above. The suite now also pins: every one of the 49 exports called
as the first thing in a fresh session (several populate a shared cache
lazily, which a suite running them in order cannot expose); the functional
type recode as a clean partition of all 32 Unicode subgroups in Smileys &
Emotion and People & Body; the pluggable-lexicon API through every
consumer that documents it, registered and inline, sentiment- and
emotion-shaped, full and partial; non-syntactic column names ("my text",
"2024", "if", "x-y") and every tidyselect form that resolves to one
column; 182 combinations of the enum and flag arguments; fourteen verb
pipelines and the grouped contract end to end; the two internal
optimisations that claim exact equivalence, against naive implementations;
and README.md against README.Rmd's prose, so it cannot drift unnoticed.
Three tests that scanned R/*.R now walk the namespace instead, so they run
inside R CMD check rather than skipping: skips there are down from 14 to 5,
262 more assertions than at the start of the pass, and the five that remain
cannot be reached from an installed package.
emoji_emotion() scores each row's emoji across the eight Plutchik emotions
(anger, anticipation, disgust, fear, joy, sadness, surprise, trust), using the
new bundled emoji_emotion_lexicon (EmoTag1200, Shoeb & de Melo 2020, MIT).
Supports a long form (long = TRUE) with one row per (row, emotion).emoji_emotion_label() adds the dominant emotion per row.emoji_lexicons() lists bundled and registered
lexicons, register_emoji_lexicon() adds your own, and emoji_score() is the
generic scorer all the verbs share. emoji_sentiment() gains a lexicon
argument (default "novak2015", unchanged behaviour).emoji_pairs() returns a tidy, graph-ready edge list
(item1, item2, n) of the emoji that co-occur in the same document --
each row is a document, or supply doc_id to pool rows -- with
directed = TRUE to order pairs by first appearance. emoji_cooccurrence()
is the same with an optional diagonal (each emoji's document frequency).
emoji_ngrams() slides a window over each row's emoji in reading order and
returns one row per consecutive n-gram.emoji_position() reports where emoji sit in each text
(first/last character position and mean relative position in [0, 1]),
emoji_density() reports emoji per character and per token, and
emoji_ratio() reports the share of the text's characters that are emoji
plus an .emoji_only flag.emoji_dfm() builds a document-by-emoji feature table (weightings: counts,
binary, tf-idf), keeping every document -- including emoji-free ones -- so the
result binds row-for-row to outcome columns in modelling workflows.emoji_pairs(), emoji_cooccurrence(),
emoji_ngrams(), emoji_dfm()) canonicalise glyphs through the package's
codepoint key, so qualified and unqualified forms of the same emoji (for
example the victory hand with and without U+FE0F) count as one
node/feature. emoji_frequency() intentionally still reports the exact
extracted glyph.emoji_to_text() replaces emoji in a text column with their Unicode names or
shortcodes (demojize -- useful for accessibility and NLP preprocessing), and
text_to_emoji() is the inverse (emojize).as_emoji_name(), as_emoji_shortcode() and as_emoji() for
ad-hoc conversion.emoji_search() finds emoji by keyword, name or shortcode and returns a tidy
tibble of matches.emoji_emotion_lexicon.man/figures/logo.svg is the vector master, logo.png the raster copy).emoji_ratio() recognising an emoji-only row; all of that is fixed. Emoji
separated by anything other than a ZWJ are unaffected.text_to_emoji() no longer misses a :shortcode: that follows an unrelated
colon. "meet at 10:30 :grinning:" and "https://example.org :grinning:"
previously came back unchanged, because the permissive :...: pattern
consumed the shortcode's opening colon; shortcode tokens are now matched on
the character set GitHub-style aliases actually use.emoji_pairs() and emoji_dfm() order glyphs in the C locale, so which
glyph lands in item1 and the order of a dfm's tied columns no longer depend
on the session's collation. Results are now reproducible across machines.emoji_lexicons() no longer reports the glyph column as a score dimension
for a lexicon registered with a by other than "emoji".emoji_emotion(long = TRUE) no longer drops a user column named
.row_number.emoji_ngrams(n = Inf) gives the documented error instead of a coercion
warning followed by "missing value where TRUE/FALSE needed".emoji_to_text() is several times faster: it locates emoji once for the
whole column rather than once per row.emoji_density() returns .emoji_per_token = 0 (not NA) for
whitespace-only text, matching .emoji_per_char and the documented
"no emoji -> 0" contract (#1).as_emoji() accepts the spaced Unicode names produced by as_emoji_name()
(routing through the reference table), so as_emoji(as_emoji_name(x))
round-trips instead of returning NA (#2).?emoji_sentiment_lexicon now explains which lexicon entries are stored as
unqualified, text-presentation code points (the bare heart U+2764 without
U+FE0F, the white smiling face, the heavy check mark, ...) and are
therefore not detected in text, and notes that the qualified form resolves to
the same entry (#3).?register_emoji_lexicon no longer points at an internal
development file.emoji_search() matches literally, so queries containing regex
metacharacters (for example the +1 alias) are safe and cannot error.emoji_to_text(format = "shortcode") now always emits the emoji's canonical
(first) GitHub-style alias -- the same one reported by emoji_frequency() and
as_emoji_shortcode() -- and the wrap template is honoured. Emoji with no
known name/shortcode are left in place rather than dropped from the text.emoji_to_text() and text_to_emoji() keep NA text entries as NA.emoji_emotion() and emoji_emotion_label() accept registered or
data-frame emotion lexicons (any subset of the eight Plutchik dimensions),
not just the bundled "emotag1200".register_emoji_lexicon(by = ) works with any glyph column name in
emoji_sentiment() and emoji_emotion().emoji_frequency() (and therefore top_n_emojis()) breaks count ties by
the glyph, making the output order deterministic.emoji_lexicons() no longer lists a custom lexicon's glyph/key columns among
its score dimensions.?tidyEmoji) documents the output and naming
contract shared by all verbs.doc_id argument); it was already a hard transitive dependency, so the
installed footprint is unchanged.emoji_pairs(), emoji_cooccurrence() or
emoji_dfm() warn that grouping is ignored -- use doc_id to express
per-group structure.U+FE0F variation
selector no longer get NA metadata, are no longer dropped by
emoji_categorize(), and no longer disappear from top_n_emojis(duplicated = TRUE).emoji_summary() and emoji_filter() use the same detection as the extraction
verbs.emoji_sentiment() gains .emoji_n_scored (emoji actually found in the
lexicon), distinct from .emoji_n.top_n_emojis(n =) counts distinct emoji rather than rows, breaks ties
deterministically, keeps emoji that have no GitHub-style alias, and preserves
the exact extracted glyph in duplicated mode (one row per distinct alias;
left_join instead of inner_join).emoji_extract_unnest() now uses .row_number (dotted) to avoid collision
with user columns and dplyr::row_number.emoji_summary() column names renamed from emoji_tweets/total_tweets to
n_with_emoji/n_total. The old names are no longer available in this
release.emoji_tweets() is soft-deprecated in favour of emoji_filter().emoji_sentiment() and emoji_categorize().emoji_summary(), emoji_frequency() and
top_n_emojis() now warn that grouping is ignored (per-group results land in
1.0).ata_tweets.rda (a CSV misnamed .rda) to
ata_tweets.csv and downsampled from 10k to 2k rows. Vignette language
updated to be less Twitter-specific.key column for normalised joins.tidyEmoji is now positioned as a general toolkit for emoji in any text column (social-media posts, reviews, chat logs, survey responses, ...), not just tweets.
emoji_sentiment() scores the emoji in each row using the bundled
emoji_sentiment_lexicon (the Emoji Sentiment Ranking of Kralj Novak et al.,
2015), returning a mean sentiment in [-1, 1].emoji_frequency() returns the count of every emoji in a text column, with
name, shortcode and category. top_n_emojis() is now a thin wrapper over it.emoji_tokens() expands data to one row per emoji occurrence with its name,
category and sentiment score -- a tidy, "one-token-per-row" shape.emoji_filter() is a clearer, text-agnostic name for emoji_tweets() (which
is kept as a synonym).emoji_sentiment_lexicon.top_n_emojis() in particular is dramatically faster on large inputs.top_n_emojis() no longer emits a many-to-many join warning, and reports the
emoji's canonical shortcode (e.g. mask) by default.emoji_tweets() previously
returned a plain data frame), and emoji_extract_unnest() no longer prints a
grouping message.data-raw/).tweet_tbl -> data and tweet_text -> text. Code
that passed these positionally (e.g. df %>% emoji_summary(text_col)) is
unaffected; update any calls that named the old arguments.top_n_emojis(duplicated_unicode = "yes"/"no") is deprecated in favour of the
logical duplicated = TRUE/FALSE. The old argument still works with a warning.