Diversity Indices

Abstract supertype for Shannon entropy estimators.

Plugin estimator for Shannon entropy.

This is the maximum-likelihood estimator (MLE) obtained by substituting the empirical probabilities into Shannon's definition.

Miller-Madow estimator for Shannon entropy.

For $S$ observed positive-abundance species and sample size $n$, the correction added to Shannon entropy is

\[\frac{S - 1}{2n\log b},\]

where $b$ is the logarithm base.

Hausser-Strimmer shrinkage estimator for Shannon entropy.

This shrinks empirical probabilities towards the uniform distribution over the known or observed support, then applies the plugin entropy estimator.

Basharin bias-reduced estimator for Shannon entropy.

For known support size $S$ and sample size $n$, the correction added to the plugin estimator is

\[\frac{S - 1}{n\log b},\]

where $b$ is the logarithm base.

AddGamma(gamma)

Add-gamma estimator for Shannon entropy.

With support size $S$ and counts $n_i$, probabilities are estimated as

\[\hat p_i = \frac{n_i + \gamma}{n + \gamma S}.\]

Chao-Shen estimator for Shannon entropy with possible unseen species.

This uses a Good-Turing coverage estimate and a Horvitz-Thompson adjustment. It is intended for samples where the true support may be larger than the observed support.

Species richness: the number of species/categories with positive abundance.

Shannon(; base=2, estimator=Plugin())

Shannon entropy/index. Use effective_diversity(Shannon(), x) for the corresponding Hill number, i.e. the effective number of species.

The default logarithm base is 2, so entropy is measured in bits unless another base is provided. Use estimator to choose among Plugin, MillerMadow, HausserStrimmer, Basharin, AddGamma, and ChaoShen.

\[H_b = -\sum_i p_i \log_b p_i\]

Renyi(q; base=2)

Renyi entropy of order q.

The default logarithm base is 2, so entropy is measured in bits unless another base is provided. At q = 1, Renyi entropy is evaluated as Shannon entropy.

\[H_q = \frac{1}{1-q}\log_b\left(\sum_i p_i^q\right)\]

Tsallis(q; base=2)

Tsallis entropy of order q, scaled to the requested logarithm base.

The default logarithm base is 2. At q = 1, Tsallis entropy is evaluated as Shannon entropy with the same base.

\[T_q = \frac{\sum_i p_i^q - 1}{(1-q)\log b}\]

Simpson concentration.

\[D = \sum_i p_i^2\]

Gini-Simpson diversity.

\[1 - D = 1 - \sum_i p_i^2\]

Greenberg's linguistic diversity index.

This is the probability that two randomly selected individuals have different mother tongues. It is mathematically the same quantity as Gini-Simpson diversity:

\[1 - \sum_i p_i^2\]

The index uses only relative mother-tongue frequencies. It does not account for second-language use, language vitality, or linguistic distance between languages.

Linguistic diversity index (LDI).

This is an alias-by-convention for GreenbergDiversityIndex, included for linguistic-demography workflows. It returns the probability that two randomly selected individuals have different mother tongues, equivalent to GiniSimpson.

Inverse Simpson diversity.

\[\frac{1}{D} = \frac{1}{\sum_i p_i^2}\]

Hill(q)

Hill diversity of order q. Orders 0, 1, and 2 correspond to richness, Shannon effective diversity, and inverse Simpson diversity.

\[{}^qD = \left(\sum_i p_i^q\right)^{1/(1-q)}\]

Chao1 asymptotic richness estimator.

For observed richness $S_{obs}$, singleton count $f_1$, and doubleton count $f_2$, this package uses the bias-corrected abundance form

\[\hat S_{Chao1} = S_{obs} + \frac{f_1(f_1 - 1)}{2(f_2 + 1)}.\]

ACE(; threshold=10)

Abundance-based Coverage Estimator (ACE) for richness.

Species with counts up to threshold are treated as rare. The default threshold=10 follows the usual ACE convention.

Good-Turing sample coverage estimate.

For sample size $n$ and singleton count $f_1$, sample coverage is

\[\hat C = 1 - \frac{f_1}{n}.\]

Pielou evenness, Shannon entropy divided by the maximum entropy for the observed richness.

\[J = \frac{H}{\log_b S}\]

Fisher's alpha diversity parameter.

Fisher's alpha solves

\[S = \alpha \log\left(1 + \frac{n}{\alpha}\right),\]

where $S$ is observed richness and $n$ is total abundance.

entropy(index, data; frequencies=true, support=nothing)

Evaluate an entropy index for data.

entropy is defined for Shannon, Renyi, and Tsallis. For dictionaries, values are abundances. Numeric vectors are abundance vectors by default. Non-numeric vectors are raw observations. Pass frequencies=false to treat a numeric vector as raw observations instead.

For Shannon, support may be an integer support size or a collection of known categories. Leave support=nothing when only the observed support is known, or when using ChaoShen for possible unseen categories.

For community matrices, rows are samples and columns are taxa/categories. The result is one entropy value per row.

entropy_variance(index, data; frequencies=true, support=nothing)

Estimate the variance of a Shannon entropy estimator.

Analytic variance estimates are available for Plugin, MillerMadow, Basharin, and ChaoShen. For ChaoShen, this is an approximate Horvitz-Thompson detection variance. Use bootstrap or jackknife for estimators without an analytic variance estimate.

entropy_confint(index, data; level=0.95, frequencies=true, support=nothing)

Return a normal-approximation confidence interval for Shannon entropy using entropy_variance.

diversity(index, data; frequencies=true, support=nothing)

Evaluate a diversity index for data.

For dictionaries, values are abundances. Numeric vectors are abundance vectors by default. Non-numeric vectors are raw observations. Pass frequencies=false to treat a numeric vector as raw observations instead.

For entropy-family indices, diversity returns the corresponding effective diversity. Use entropy when you want entropy units such as bits. For community matrices, rows are samples and columns are taxa/categories. The result is one diversity value per row.

effective_diversity(index, data; frequencies=true)

Return an index as an effective number of species when the transformation is standard. For Shannon entropy this is

\[{}^1D = b^H.\]

For community matrices, rows are samples and columns are taxa/categories. The result is one effective-diversity value per row.

richness(data; frequencies=true)

Return species richness for data.

shannon(data; base=2, estimator=Plugin(), frequencies=true, support=nothing)

Alias for shannon_entropy.

shannon_entropy(data; base=2, estimator=Plugin(), frequencies=true, support=nothing)

Return Shannon entropy for data.

The default logarithm base is 2, so entropy is measured in bits. Pass support when using finite-support estimators and the support is known.

shannon_variance(data; base=2, estimator=Plugin(), frequencies=true, support=nothing)

Return the estimated variance of a Shannon entropy estimator.

shannon_confint(data; base=2, estimator=Plugin(), level=0.95, frequencies=true, support=nothing)

Return a normal-approximation confidence interval for Shannon entropy.

shannon_diversity(data; base=2, estimator=Plugin(), frequencies=true, support=nothing)

Return Shannon effective diversity for data.

\[{}^1D = b^H\]

The default logarithm base is 2, matching Shannon.

bootstrap(index, data; nboot=1000, level=0.95, quantity=:entropy, rng=nothing)

Bootstrap a Shannon estimator from empirical abundance counts.

quantity=:entropy bootstraps entropy values. quantity=:diversity bootstraps effective Shannon diversity values. The returned named tuple contains the estimate, bootstrap variance, standard error, percentile interval, and replicates.

jackknife(index, data; level=0.95, quantity=:entropy)

Delete-one jackknife for Shannon entropy or Shannon effective diversity.

renyi(data, q; base=2, frequencies=true)

Alias for renyi_entropy.

renyi_entropy(data, q; base=2, frequencies=true)

Return Renyi entropy of order q for data.

The default logarithm base is 2, matching Renyi.

\[H_q = \frac{1}{1-q}\log_b\left(\sum_i p_i^q\right)\]

renyi_diversity(data, q; base=2, frequencies=true)

Return the effective diversity corresponding to Renyi entropy of order q.

\[{}^qD = b^{H_q}\]

tsallis(data, q; base=2, frequencies=true)

Alias for tsallis_entropy.

tsallis_entropy(data, q; base=2, frequencies=true)

Return Tsallis entropy of order q for data.

The default logarithm base is 2, matching Tsallis.

\[T_q = \frac{\sum_i p_i^q - 1}{(1-q)\log b}\]

tsallis_diversity(data, q; base=2, frequencies=true)

Return the effective diversity corresponding to Tsallis entropy of order q.

\[{}^qD = \left(1 + (1-q)(\log b)T_q\right)^{1/(1-q)}\]

hill_number(data, q; frequencies=true)

Return the Hill number of order q for data.

chao1(data; frequencies=true)

Return the Chao1 richness estimate for data.

ace(data; frequencies=true, threshold=10)

Return the abundance-based coverage estimator (ACE) for richness.

sample_coverage(data; frequencies=true)

Return the Good-Turing sample coverage estimate for data.

pielou_evenness(data; frequencies=true)

Return Pielou evenness, Shannon entropy divided by the maximum entropy for the observed richness.

fisher_alpha(data; frequencies=true)

Return Fisher's alpha diversity parameter.

simpson_index(data; frequencies=true)

Return Simpson concentration, $\sum_i p_i^2$, for data.

gini_simpson_index(data; frequencies=true)

Return Gini-Simpson diversity, $1 - \sum_i p_i^2$, for data.

greenberg_diversity_index(data; frequencies=true)

Return Greenberg's linguistic diversity index, the probability that two randomly selected individuals have different mother tongues.

This is equivalent to Gini-Simpson diversity, $1 - \sum_i p_i^2$.

linguistic_diversity_index(data; frequencies=true)

Return the linguistic diversity index (LDI), equivalent to greenberg_diversity_index.

index_of_linguistic_diversity(current, baseline; frequencies=true)

Return a Terralingua-style index of linguistic diversity (ILD) as the ratio of the current linguistic diversity index to a baseline linguistic diversity index.

A value of 1 indicates no change relative to the baseline; values below 1 indicate loss of linguistic diversity; values above 1 indicate an increase. This helper compares two assemblages and is therefore not itself a single-assemblage diversity index object.

inverse_simpson_index(data; frequencies=true)

Return inverse Simpson diversity, $1 / \sum_i p_i^2$, for data.