Same Field, Opposite Winners: Representation-Dependent Recoverability in Spatial Reconstruction

The same physical field can be reconstructable or unreconstructable depending on which information channel the representation exposes.

Joe Poehlein

---

Update, July 2026: this paper was temporarily withdrawn along with the rest of this site's research corpus while every piece underwent a full re-audit against upgraded pipeline standards. This paper passed two independent re-derivation passes: every headline claim and all sixteen results-table values reproduce exactly from the archived pipeline. This revision corrects one descriptive constant the audit caught (field skewness), tightens two wording imprecisions, and adds the raw-field spatial-baseline controls from the audit hardening. No claim changed. Details in the Revision Note at the end.

---

Abstract

Reconstruction benchmarks often assume that method performance is determined primarily by algorithm family. We show that this framing is incomplete. Using a cosmic matter density field from the AbacusSummit N-body simulation suite, we demonstrate that the same physical field and identical observations produce opposite winning methods depending only on representation. On the raw overdensity field, both matrix completion and spatial interpolation fail. After log-density transformation, spatial methods achieve genuine predictive skill while matrix completion remains a zero predictor, indicating activation of a spatial correlation channel. We then formulate a two-channel recoverability model separating low-rank algebraic structure from spatial correlation structure. This model makes a falsifiable prediction: when the same field is constrained into a low-rank regime, matrix completion should outperform spatial interpolation. A rank-truncated control confirms this prediction, reversing the winner while preserving the underlying field identity. These results establish representation-dependent recoverability: reconstruction success is governed by which information channel a representation exposes, not by the nominal method family alone. We conclude with a practical method-selection doctrine based on channel diagnostics rather than algorithm labels.

The Setup

We pointed two reconstruction methods at the same physical field the cosmic matter density from an N-body simulation with identical observations. One method exploits low-rank algebraic structure (matrix completion). The other exploits spatial correlation (optimal interpolation). We expected one to win.

Instead, both won on different representations of the same data. And both lost on the other representation.

This note documents that result and what it means for choosing reconstruction methods in practice.

The Field

The cosmic matter density field from the AbacusSummit simulation suite (Maksimova et al. 2021), computed using the Abacus N-body code (Garrison et al. 2021). Specifically the small_c000_ph3000 box: a 500 Mpc/h periodic volume simulated with 1728³ dark matter particles at redshift z = 0.5. We voxelized the publicly distributed particle subsample (subsample A, ~144 million particles, ≈2.8% of the full set) onto a 128³ grid using Cloud-in-Cell assignment about 69 particles per cell and reshaped it into a 16,384 × 128 matrix.

This field is statistically extreme. Skewness 8.5, kurtosis 178, dynamic range 115×. It is dominated by a filamentary web of dense structures (galaxy clusters, walls) separated by vast voids. Its singular value spectrum decays slowly (rank 85 at 90% variance) and its incoherence parameter $\mu = 54$ far outside the regime where standard matrix completion recovery guarantees apply (Candès & Recht 2009).

We subsampled at 0.25%–50% to mimic sparse galaxy survey geometries and compared Soft-Impute (Mazumder et al. 2010) against thin-plate-spline radial basis function interpolation (OI), inverse distance weighting (IDW), and ordinary kriging.

Raw Field: Both Methods Fail

On the raw overdensity field, neither method demonstrates reconstruction skill.

MC converges to the field mean at every density (NRMSE ≈ 1.0, R² ≈ 0). It is the zero predictor it learns nothing beyond the global average.

OI performs worse than the mean predictor (R² < 0). The extreme kurtosis violates the smoothness assumption underlying the RBF kernel. Rare massive overdensities in galaxy clusters corrupt the local fit, producing predictions further from the truth than a constant.

This is the starting point: neither headline method can reconstruct this field in its raw form.

Audit-hardening controls added in revision: the two simple spatial baselines were also run on the raw field under the same observation masks. IDW reaches R² = +0.086 ± 0.003 and ordinary kriging R² = +0.108 ± 0.004 at 5% sampling weak positive skill, far below their log-density performance below. The raw field is not binary-unreconstructable for every spatial method; it is that no method family finds material skill until the representation exposes a channel.

Log-Density: The Spatial Channel Activates

The standard cosmological transform log1p, compressing the field to near-Gaussian (kurtosis 3.51) changes the picture dramatically.

At 5% sampling on the log-density field, OI achieves R² = +0.171 ± 0.003 (50 seeds, 57σ from zero). MC remains a zero predictor: R² = +0.002 ± 0.000 across the same 50 seeds (a representative single run gives −0.012; the multi-seed mean is the reliable figure and is what the table below reports).

The transform did not change the underlying spatial correlation structure. It changed the field's statistical compatibility with OI's kernel assumptions. The spatial information was always there the log transform exposed it to the spatial method.

MC gained nothing from the transform because MC's limitation is structural. It cannot exploit spatial correlation regardless of the field's marginal distribution (see our companion diagnostic: the permutation invariance test confirms MC is invariant to row/column reordering, while OI's skill is destroyed by permutation).

Across the full sampling range on the log-density field:

All ± values < 0.003 (10 seeds per condition). Kriging at 10% (R² = 0.362) achieves comparable skill to MC at 50% (R² = 0.348) a one-parameter spatial method with 5× fewer observations matches cross-validated matrix completion. Even zero-parameter IDW (R² = 0.139 at 5%) outperforms MC (R² = 0.002) because it can see where the points are.

Two Channels, One Field

The log-density result suggests that MC and OI are not merely different algorithms. They exploit fundamentally different information channels in the same data.

MC's channel: low-rank algebraic structure. Governed by effective rank × incoherence × sampling density. MC succeeds when the field can be well-approximated by a few singular vectors and sampling is dense enough for the incoherence condition to hold.

The spatial channel: correlation structure. Governed by field smoothness × sampling density × correlation length. Spatial methods succeed when nearby observations predict distant ones when the field varies smoothly over distances comparable to the observation spacing.

These channels are largely orthogonal under the present method classes: MC does not directly exploit spatial correlation, while spatial interpolators do not directly optimize global low-rank structure. The log-density transform activated the spatial channel without helping MC because MC's channel (low-rank algebraic structure) was not what the transform exposed.

If this two-channel model is correct, it makes a prediction: on the same field in a regime where the algebraic channel dominates instead of the spatial channel, MC should win and OI should fail. We tested this directly.

Prediction Test: The Algebraic Channel Activates

To directly test the channel model, we constrained the raw (not log-transformed) overdensity into a low effective-rank regime by truncating the SVD. At rank 5, MC achieves R² = +0.538 substantial skill. OI remains broken at R² ≈ −0.577, still defeated by the non-Gaussian statistics.

The prediction holds. Same physical field, same spatial structure, but the dominant exploitable structure has shifted from spatial correlation (which MC cannot see) to low-rank algebraic regularity (which OI cannot exploit on non-smooth fields). The two-channel model correctly predicts which method wins based on which channel the representation exposes.

Method Selection Doctrine

The distinction between these channels has a concrete implication for practitioners:

Before choosing a reconstruction method, determine which information channel your field representation actually exposes. Low rank does not guarantee MC will outperform spatial methods. Spatial correlation does not guarantee OI will outperform MC. The representation not just the raw field determines which channel is available.

A practical workflow:

1. Check the field's effective rank and incoherence (SVD). If rank is low and μ is small, MC has a viable channel.
2. Check the field's spatial correlation structure and smoothness (variogram, kurtosis). If the field is smooth and spatially correlated, spatial methods have a viable channel.
3. If the raw field fails both checks, explore standard transforms (log, Box-Cox, rank-based) before concluding the field is unreconstructable.
4. If a transform activates one channel but not the other, the choice is made for you.

Caveats

All results are from a single AbacusSummit box (500 Mpc/h, z = 0.5, 128³ grid). The regime boundaries are specific to these conditions extending to other field types, resolutions, and covariance structures is future work. The same representation-dependence principle appears in atmospheric science: our tornado outbreak precursor study found that a coarse-resolution representation (ERA5 at 0.25°) detects synoptic-scale multi-day outbreaks but leaves single-tornado mesoscale events unresolvable a resolution-limited detectability ceiling structurally analogous to the channel boundaries mapped here.

OI and kriging used default configurations (TPS kernel, exponential variogram). No hyperparameter search was performed. MC was cross-validated over its regularization parameter the asymmetry in tuning effort favors MC, making the spatial methods' dominance more striking.

The cosmic density field was natively 3D but flattened to a 2D matrix for MC. Tensor completion methods that preserve the 3D structure may exploit spatial adjacency along the flattened axis that matrix completion cannot.

The scope fence, stated precisely: all claims here concern vanilla flattened matrix completion (Soft-Impute) versus the tested spatial interpolators (TPS-RBF optimal interpolation, IDW, ordinary kriging) on this single AbacusSummit box, under the tested sampling regimes and transforms. Graph-regularized matrix completion, tensor completion, deep learned reconstructors, and hybrid interpolation-assisted MC variants are out of scope and untested the two-channel model makes no claim over them.

Revision Note

July 2026. After methodological flaws surfaced in the earliest project in this research program (the tornado-ERA5 work, since rewritten), every published piece on this site was withdrawn pending a full re-audit. This paper received the deepest scrutiny in the corpus: a four-packet adversarial deeper-dive (family positioning, claim re-audit, disposition), an external adversarial review, an independent re-derivation pass, a convergence test against held-back drafts, and a second independent re-derivation in July 2026 that re-ran the spatial-baseline sweep from scratch on fresh seeds. Every headline claim and all sixteen results-table values reproduce exactly. Four corrections were made in this revision:

1. Field skewness was published as 6.3; recomputation from the archived density grid gives 8.5. No claim depends on this constant (the field is more extreme than stated). Kurtosis, dynamic range, rank, and incoherence all verified as published.
2. The voxelization description now states that the distributed AbacusSummit particle subsample (~144M particles) was gridded, not the full 1728³ particle set.
3. The MC-at-5% figure in the text now quotes the 50-seed mean (+0.002) rather than a single representative run (−0.012), matching the table.
4. The raw-field IDW/kriging controls (added during audit hardening) are now reported, and the scope fence is stated explicitly in Caveats.

---

Author Note

The author's primary background is software engineering and systems architecture. This analysis was conducted using an AI-assisted research pipeline: Claude (Anthropic) served as research coordinator and managed project state across sessions; Claude Code (Anthropic) executed computational experiments autonomously on consumer hardware. The RBF interpolation workload that produced the adversarial validation results was accelerated 344× by applying voxel-engine chunk decomposition patterns to the spatial solves the architectural transfer is documented separately in our voxel-engine architecture transfer study. ChatGPT (OpenAI) provided independent adversarial review during the technical review process.

All scientific decisions hypothesis formation, experimental design, kill criteria, and interpretation were made by the author. The AI tools were used as execution and review instruments, not as originators of the research questions or methodology.

---

References

Candès, E. J. & Recht, B. (2009). Exact matrix completion via convex optimization. Foundations of Computational Mathematics, 9(6), 717–772.

Garrison, L. H., Eisenstein, D. J., Ferrer, D., Maksimova, N. A., & Pinto, P. A. (2021). The Abacus cosmological N-body code. Monthly Notices of the Royal Astronomical Society, 508(1), 575–596.

Maksimova, N. A., Garrison, L. H., Eisenstein, D. J., Hadzhiyska, B., Bose, S., & Satterthwaite, T. P. (2021). AbacusSummit: a massive set of high-accuracy, high-resolution N-body simulations. Monthly Notices of the Royal Astronomical Society, 508(3), 4017–4037.

Mazumder, R., Hastie, T. & Tibshirani, R. (2010). Spectral regularization algorithms for learning large incomplete matrices. JMLR, 11, 2287–2322.