spineCompare

Run the same data under several measures and see whether they agree.
The measure is the modelling step, and two defensible measures can call opposite objects atypical. Nothing else in the package lets you see that happening.

Syntax and options

help spineCompare
SPINE Compare Run the same data under several measures and see whether they agree. T = spineCompare(data, measures, orientation) [T, C] = spineCompare(data, measures, orientation, 'Name', Value, ...) spineCompare(...) prints, returns nothing The measure is the modelling step, and two defensible measures can call opposite objects atypical. Nothing else in the package lets you see that happening. This does. T is a table, one row per measure: how many strata it found, how many objects it flagged, and the first cut. C carries the detail: Results one Result per measure measures the names, in the order given agreement pairwise Jaccard overlap of the flagged sets flagged one logical column per measure AGREEMENT IS THE POINT. If two measures flag the same objects, the structure is robust to how you asked. If they disagree, the disagreement is telling you something real about your data, and choosing between them is a scientific decision rather than a default. Options 'Plot' false (default). When true, one data figure per measure. Anything else is passed through to spine, so 'K', 'Missing', 'Objects' and the rest work here unchanged. Author Abd AlRahman AlMomani Assistant Professor of Data Science and Mathematics, ERAU, AZ almomaniar@gmail.com

When measures disagree

Three planted groups, chosen so that "far from the centre" and "in a sparse neighbourhood" are genuinely different questions: a bulk, a halo displaced a little and spread widely, and an island displaced a long way and packed tight.
rng(4242);
bulk = randn(200,2);
halo = 5*[cos(2*pi*rand(30,1)), sin(2*pi*rand(30,1))] + 2*randn(30,2);
island = [12 12] + 0.25*randn(20,2);
D = [bulk; halo; island];
 
spineCompare(D, ["dist2mean" "robustrownorm" "knn" "dist2median"], "upper")
SPINE comparison over 4 measures measure strata flagged fraction first cut dist2mean 2 24 0.096 6.625 robustrownorm 4 42 0.168 3.122 knn 2 31 0.124 1.097 dist2median 4 42 0.168 4.081 agreement on which objects are atypical (Jaccard) dist2mean robustrownorm knn dist2median dist2mean - 0.57 0.08 0.57 robustrownorm 0.57 - 0.43 1.00 knn 0.08 0.43 - 0.43 dist2median 0.57 1.00 0.43 - knn and dist2mean overlap on only 8% of what they flag. They are answering different questions. Deciding which one you meant is the modelling step, not a detail. See docs/measure-guide.html.
Read the agreement matrix, not the strata counts. knn and dist2mean overlap on a small fraction of what they flag: the island is far away but densely packed, so its nearest-neighbour distances are small. Neither is wrong. They are answering different questions, and choosing between them is the modelling step.
Side by side, the difference is visible rather than tabulated.
[~, C] = spineCompare(D, ["dist2mean" "knn"], "upper");
spineDataPlot(C.Results{1}, 'Title', 'dist2mean: far from the centre');
spineDataPlot(C.Results{2}, 'Title', 'knn: in a sparse neighbourhood');

The profiles behind the disagreement

Each measure produces its own entropy profile, and each cut is a transition in that profile. Overlaying them shows why the measures disagree: they are not arguing about where to cut one curve, they are working on different curves.
[~, C4] = spineCompare(D, ["dist2mean" "knn" "robustrownorm"], "upper");
cols = [0 0.35 0.7; 0.85 0.33 0.10; 0.20 0.55 0.24];
 
figure('Color','w','Position',[100 100 640 340]); hold on
h = gobjects(1,3);
for k = 1:3
I = C4.Results{k}.Info;
h(k) = plot(I.E, 'LineWidth', 1.3, 'Color', cols(k,:));
plot(I.candidates, I.E(I.candidates), 'o', 'MarkerSize', 7, ...
'MarkerEdgeColor', cols(k,:), 'LineWidth', 1.2);
plot(I.boundaries, I.E(I.boundaries), 'o', 'MarkerSize', 7, ...
'MarkerFaceColor', cols(k,:), 'MarkerEdgeColor', cols(k,:));
end
xlabel('rank in the sorted measure'); ylabel('incremental nested entropy');
title('three measures, three profiles, three different transitions');
legend(h, C4.measures, 'Location', 'southeast', 'Box', 'off');
grid on; hold off
Open circles are candidate transitions, filled circles the cuts that were kept. knn turns over in a different place from dist2mean because the island is dense: on the neighbour-distance curve it never becomes atypical at all.

When measures agree

Here the structure is robust to how it was asked for, and the choice of measure does not matter much.
rng(7);
E = [randn(200,3); randn(15,3) + 12];
spineCompare(E, ["dist2mean" "dist2median" "robustrownorm"], "upper")
SPINE comparison over 3 measures measure strata flagged fraction first cut dist2mean 2 15 0.070 4.441 dist2median 2 15 0.070 3.525 robustrownorm 2 15 0.070 2.967 agreement on which objects are atypical (Jaccard) dist2mean dist2median robustrownorm dist2mean - 1.00 1.00 dist2median 1.00 - 1.00 robustrownorm 1.00 1.00 - All measures overlap on at least 100% of what they flag, so the structure is robust to how it was asked for.

What comes back

With outputs requested, the table carries what each measure found and the second output carries the detail.
[T, C] = spineCompare(D, ["dist2mean" "knn" "robustrownorm"], "upper");
disp(T)
measure nStrata nFlagged fracFlagged firstCut _______________ _______ ________ ___________ ________ "dist2mean" 2 24 0.096 6.6254 "knn" 2 31 0.124 1.0972 "robustrownorm" 4 42 0.168 3.1215
C.agreement is the pairwise Jaccard overlap of the flagged sets, C.Results holds one Result per measure, and C.flagged has one logical column per measure.
C.agreement
ans = 3×3
1.0000 0.0784 0.5714 0.0784 1.0000 0.4314 0.5714 0.4314 1.0000

See also

spine for a single partition, spineSummary for what one found, and docs/measure-guide.html for how to choose in the first place.
close all