spine

Partition data into scale strata. One call: raw data in, strata out.

You supply a measure and an orientation. Those are the scientific input, and neither has a default. Everything after them is computed.

Syntax and options

Read from the function itself, so this page can never disagree with help(spine).

from entropy_spine import spine
help(spine)

Help on function spine in module entropy_spine.spine:

spine(
    data,
    measure=None,
    orientation=None,
    *,
    objects: 'str' = 'rows',
    k: 'int' = 5,
    object_ids=None,
    missing: 'str' = 'error',
    thin: 'bool' = False,
    **core_options
)
    spine: apply SPINE to raw data in one call.

        scores, result = spine(data, measure, orientation)
        scores, result = spine(data, measure, orientation, k=8, missing="omit")

    ``measure`` and ``orientation`` are both required. They are the scientific
    input: SPINE does not know what your data means, and choosing them is the
    modelling step. Neither has a default.

    Accepted data
        1-D array or list          one value per object
        2-D array                  rows are objects (``objects="columns"`` to flip)
        sequence of dicts          one dict per object; numeric scalar values are
                                   the features, a text value the identifiers
        pandas DataFrame           numeric columns are the features, the index or
                                   a text column the identifiers (pandas optional)
        text sequence              statefrequency only: each entry one observed state

    Options
        objects           "rows" (default) or "columns", for 2-D arrays
        k                 neighbour count for the knn measure (default 5)
        object_ids        identifiers, overriding anything found in the data
        missing           "error" (default) or "omit"
        thin              locate the cut on a deterministically thinned copy
        boundary_mode, profile_weights, bilateral_anchor
                          passed to the reference core unchanged

    Returns ``(scores, result)``: one score per object in YOUR object order, and a
    ``Result`` record. ``result.boundary_values`` is the transferable result — a
    value in your own measure's units, applied to new data or quoted in a paper.
    ``result.info`` is the complete reference record, for auditing; its ranks are
    one-based (the cross-language schema), while Result-level indices such as
    ``result.order`` and ``result.omitted`` are zero-based Python indices.

A first call

Two hundred objects from a standard normal cloud, and twelve sitting a long way out. Nothing tells SPINE that there are two groups.

%matplotlib inline
import numpy as np
from entropy_spine import spine, spine_summary

rng = np.random.default_rng(11)
X = np.vstack([rng.standard_normal((200, 2)),
               rng.standard_normal((12, 2)) + 9])

scores, result = spine(X, "dist2mean", "upper")
spine_summary(result);

SPINE   212 objects, 2 strata        measure dist2mean | orientation upper

  stratum  objects        measure range        score
        1      200      0.01647 to 3.423        0.000
        2       12        10.96 to 13.68        1.000

  cut at 3.423
  result.boundary_values applies these cuts to new data.

scores runs from 0 for the typical stratum to 1 for the most atypical, one value per object, in your object order.

from entropy_spine import spine_data_plot
spine_data_plot(result, title="a bulk and a far group");

How it works, in one picture

SPINE sorts your measure, then walks up it accumulating an incremental nested entropy: at each rank, how much uncertainty the prefix of objects seen so far carries. Adding another object of the same scale increases it. Adding an object from a genuinely larger scale does not: it concentrates the distribution, and the profile turns over.

A cut is placed where the profile turns over. That is the whole idea. There is no threshold and no target number of groups; the entropy-supported transition either happens or it does not.

import matplotlib.pyplot as plt

info = result.info
E = info["E"]
C = np.asarray(info["candidates"])
B = np.asarray(info["boundaries"])

fig, ax = plt.subplots(figsize=(6.4, 3.1))
ax.plot(np.arange(1, E.size + 1), E, lw=1.4, color="#0072B2", label="profile")
ax.plot(C, E[C - 1], "o", ms=7, mfc="none", mec="#D55E00", mew=1.4,
        label="candidate transition")
ax.plot(B, E[B - 1], "o", ms=7, color="#D55E00", label="retained cut")
ax.set_xlabel("rank in the sorted measure")
ax.set_ylabel("incremental nested entropy")
ax.legend(loc="lower right", frameon=False)
ax.spines[["top", "right"]].set_visible(False);

The turnover sits at rank 200 of 212, which is exactly where the twelve planted far objects begin. Nothing told SPINE there were two groups, or where to look.

print(f"cut placed at rank {info['boundaries'][0]} of {E.size}")

cut placed at rank 200 of 212

The cut is the transferable result

result.boundary_values is a value in your own measure's units, so it can be applied to new data or quoted in a paper. info["boundaries"], by contrast, are ranks into an internally sorted copy and mean nothing outside the call.

result.boundary_values

array([3.42303034])

Any container you already have

A numeric array, a sequence of dicts, or a pandas DataFrame. Identifiers carried by a DataFrame's index or a dict's text field come back in result.object_ids.

import pandas as pd

rng = np.random.default_rng(8)
sites = np.vstack([rng.standard_normal((55, 3)),
                   rng.standard_normal((5, 3)) + 7])
df = pd.DataFrame(sites, columns=["x", "y", "z"],
                  index=[f"site{i:02d}" for i in range(1, 61)])

s, r = spine(df, "robustrownorm", "upper")
[r.object_ids[i] for i in np.flatnonzero(s > 0)]

['site56', 'site57', 'site58', 'site59', 'site60']

Missing values

Real data has gaps. The default refuses rather than guessing.

Xm = X.copy()
Xm[3, 1] = np.nan
Xm[150, 0] = np.nan
try:
    spine(Xm, "dist2mean", "upper")
except Exception as err:
    print(err)

X must be finite (no NaN or Inf).

missing="omit" drops those objects and reports them. Their scores come back as NaN, so the result still lines up with your rows: row 150 is still row 150.

sm, rm = spine(Xm, "dist2mean", "upper", missing="omit")
print(f"omitted rows {rm.omitted.tolist()}, "
      f"scores returned {sm.size} for {Xm.shape[0]} input rows")

omitted rows [3, 150], scores returned 212 for 212 input rows

Very large collections

A profile that cannot be resolved at binary64 precision is refused rather than approximated. This follows the particular values, not the object count. When it happens, thin=True locates the cut on a deterministically thinned copy and applies it to every object; result.thinned records how many values were retained.

See also

spine_summary for what a partition found, spine_compare to check whether a different measure would have said something different, spine_measure for the ten named measures, and spine_data_plot and spine_plot for the figures.