When we attempt to map the sprawling, intricate facts of our world, we build models. The models we build are not the territories themselves, but simplified representations, often smooth and predictable, allowing us to find patterns and make predictions. Yet, the real landscape of information, particularly in the most advanced forms of machine learning, often defies such neat contours.
For decades, scientists have relied on statistical tools like the Bayesian Information Criterion, a method for judging which among several models best explains a set of observations. This criterion works well when the models themselves behave predictably, like a gently curving line. But modern machine learning, especially deep learning, often builds models far more complex. Imagine a map folded intricately upon itself, with unexpected creases and sharp, hidden corners. These are statistical models that do not satisfy the standard regularity conditions. These singular models, as they are called, refuse the neat, smooth assumptions of older statistical methods, leading to unreliable conclusions about their true complexity. This historical struggle to accurately assess model complexity and generalization for highly expressive, potentially overparameterized models meant that researchers needed new theoretical frameworks. The search for criteria capable enough to handle the complex geometric structure of singular parameter spaces continues to drive much contemporary research in this domain. To navigate these irregular landscapes, scientists developed newer tools, among them the Widely Applicable Bayesian Information Criterion. This advanced criterion depends on specific indicators of a model's true complexity, values known as learning coefficients. These values, it turns out, often align with what algebraic geometers call Real Log Canonical Thresholds—precise mathematical markers that describe the very nature of a model's irregularities. These thresholds provide a robust theoretical instrument for quantifying model complexity in deep learning and other singular models, offering a more accurate assessment than traditional approaches that assume model regularity.
Until now, determining these exact learning coefficients was a task largely confined to rare, simple cases. Researchers typically had to rely on estimation methods, akin to taking many scattered measurements across an unknown terrain and hoping to infer its true shape. These methods introduce an element of approximation, a degree of uncertainty.
A new path has opened. Recently, Grégoire Sergeant-Perthuis and colleagues described a groundbreaking development in a preprint appearing on arXiv: the first deterministic algorithm capable of precisely calculating these specific mathematical thresholds for certain types of irregular models. Unlike estimation techniques that involve an element of chance, this new algorithm is deterministic. Given the same input, it will always produce the exact same output, consistently revealing the precise values without any reliance on random sampling.
This precision, for now, applies to models of a particular design: those that are effectively two-dimensional, where the Kullback-Leibler distance—a measure of how one probability pattern differs from another—is similar to a polynomial expression. While this might seem a particular niche, it offers something profoundly important.
These exact computations provide a "ground truth—an unassailable reference point against which all previous estimation methods can be carefully measured and refined. More than that, they begin to expose the intrinsic algebraic structure of these complex models, a deeper understanding previously veiled by the approximations of sampling. This exact computation can, for shallow models, even outpace those sampling methods, demonstrating its practical efficiency. The algorithm has already shown its effectiveness across a wide range of models, including particular polynomial neural networks.
This new deterministic algorithm represents a significant advancement. While its current reach is limited to two-dimensional models, a critical future direction for this research involves generalizing these exact computational methods to higher-dimensional models. This will likely necessitate advancements in computational algebraic geometry or symbolic computation, pushing the boundaries of what is possible in exactly characterizing the singularities and complexity of real-world, high-dimensional singular models.
The work of mapping reality is never truly finished. Each new tool, each precise calculation, pulls back a small corner of the vast, intricate tapestry. The universe, in its deep learning models or its distant galaxies, continues to unfold its structures, revealing itself to patient, careful attention. And in that sustained gaze, we continue to find something far older, stranger, and more beautiful than we had remembered. ◆
Sourced from: "Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models" by Grégoire Sergeant-Perthuis, Elias Tsigaridas, and Jules Tsukahara