---
title: "Data you can trace: sources, vintages and provenance"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Data you can trace: sources, vintages and provenance}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", message = TRUE,
                      warning = TRUE, fig.width = 7, fig.height = 3.4,
                      dpi = 96)
library(countryatlas)
```

A number on a map is only as good as the answer to "where did it come from?"
Which provider, which series, in what unit, from which release, fetched when.
`countryatlas` records all of that with the data, carries it through the verbs
that reshape a frame, and prints it on the map. This vignette runs offline: the
network calls are shown but not run.

## Every column knows its source

The bundled snapshot was fetched from the World Bank, and says so:

```{r}
snap <- world_snapshot$countries
source_info(snap)[, c("column", "indicator", "unit", "vintage", "licence")]
```

The record travels with the column through joins, filters and geometry, and
the map verbs read it. With the default `footnote = "auto"`, the caption names
the source and the release:

```{r map, fig.alt = "World map of GDP per capita, with a caption naming the coverage and the World Bank release."}
p <- world_map(attach_geometry(snap), gdp_per_capita)
p
```

`map_provenance()` says the rest, and `map_citation()` lists the references for
exactly what this map used:

```{r}
map_provenance(p)
map_citation(p)[1:2]
```

## Fetching with the record attached

Every fetcher attaches the same record. These need the network, so they are not
run here:

```{r, eval = FALSE}
d <- world_data(2020, c(life_exp = "SP.DYN.LE00.IN"))   # World Bank
o <- fetch_indicator("owid", "life-expectancy")         # Our World in Data
w <- fetch_sdmx("imf", "WEO", key = "NGDP_RPCH")        # any SDMX provider
source_info(d)
```

## Pinning a release

The World Bank revises old values in every release: GDP for 2015 is not the
same number in the July 2023 and July 2024 releases. For a result that must
reproduce, pin the release, and see what changed between two of them:

```{r, eval = FALSE}
wdi_vintages()                                       # the releases on record
world_data(2015, vintage = "2024-07")                # this release, every time
compare_vintages("NY.GDP.PCAP.KD", c("2023-07", "2024-07"), year = 2015)
```

## Your own source, on the same footing

`register_country_source()` puts any data you can read behind the same verbs.
A fetcher that states its unit gets that unit into the record:

```{r}
survey <- function(indicator, countries = NULL, years = NULL) {
  out <- data.frame(iso3c = c("FRA", "DEU", "JPN"), year = 2020L,
                    lfp = c(61.2, 62.8, 62.0))
  source_info(out) <- data.frame(column = "lfp", source = "survey",
                                 unit = "percent", vintage = "2021 wave")
  out
}
register_country_source("survey", survey, meta = "A labour-force survey")
lfp <- fetch_indicator("survey", "lfp")
source_info(lfp)[, c("column", "source", "unit", "vintage")]
```

## Two sources, one indicator

`compare_sources()` lines two providers up on the ISO spine. When they state
different units, the difference is not a disagreement, so the comparison is
refused until you say otherwise:

```{r}
shares <- function(indicator, countries = NULL, years = NULL) {
  out <- data.frame(iso3c = c("FRA", "DEU", "JPN"), year = 2020L,
                    lfp = c(0.611, 0.629, 0.618))
  source_info(out) <- data.frame(column = "lfp", source = "shares",
                                 unit = "share")
  out
}
register_country_source("shares", shares)
try(compare_sources("lfp", sources = c("survey", "shares"), year = 2020))
```

```{r, include = FALSE}
remove_country_source(c("survey", "shares"))
```

## What the record holds

| Field | What it says |
| :- | :- |
| `source`, `indicator`, `label` | the provider, its series code and name |
| `unit` | the unit the provider states |
| `vintage`, `provider_updated` | the release, and when the provider last updated it |
| `fetched_at` | when you fetched it |
| `licence`, `citation` | how to reuse and credit it |