Brain's Representation of Visual and Auditory Textures

Textures are the “stuff” that fills a scene, distinct from discrete objects. They provide cues for material identification, shape inference, and scene segmentation. The brain appears to encode textures by averaging visual or auditory information over space or time, creating a statistical summary that captures the essence of the pattern.

Visual Texture Modeling

Early vision models applied V1‑like filters and measured the energy of their responses. The Heeger and Bergen model (1995) relied on marginal histograms of these filter outputs, but such statistics cannot reproduce textures with elongated structures like marble or hay. The failure stems from the loss of spatial continuity; marginal histograms ignore how orientations and locations co‑occur.

Portilla and Simoncelli (1999) introduced higher‑order statistics—correlations between filter responses at different orientations and spatial positions. By incorporating these dependencies, the model captures structural continuity and can synthesize more realistic textures. Typical synthesis systems now encode roughly 700 statistics to represent an image.

Auditory Texture Modeling

Stationary sounds—rain, fire, crowd noise—behave like visual textures: their essential properties remain constant over time. Auditory front‑ends decompose sound with cochlear‑like filters, extract amplitude envelopes, and apply modulation filters. Simple power‑spectrum models miss crucial information; vertical streaks in a cochleagram (e.g., fire crackles) reveal dependencies across frequency channels that require correlation matrices to capture.

Natural sound textures are often sparser than broadband noise, consisting of infrequent high‑amplitude events such as raindrops. Modeling these events demands statistics beyond marginal moments, including cross‑channel correlations and modulation‑domain measures.

Perceptual Experiments

When listeners hear short excerpts of a texture, unique temporal details allow discrimination between different samples. As excerpt duration grows to a few seconds, the measured statistics converge, and humans lose access to specific temporal cues, relying instead on the accumulated statistical summary. This creates a continuum from single‑source sounds with rich temporal structure to dense textures that are best described by statistics.

Biologically plausible models—those using log‑spaced cochlear filters, compression, and modulation analysis—consistently generate synthetic textures judged more realistic than models lacking these properties. Failures in synthesis, such as missing pitch, rhythm, or reverberation cues, highlight missing variables that the perceptual system may attend to.

Mechanisms and Explanations

Texture synthesis via optimization proceeds in four steps: (1) measure a set of statistics (marginal moments, correlations) from a natural source; (2) start with a noise signal; (3) define a loss function as the difference between the noise statistics and the source statistics; (4) iteratively adjust the noise using gradient‑based optimization to minimize the loss. If the brain’s representation has been correctly identified, the resulting synthetic texture should be perceptually indistinguishable from the original.

Exemplar discrimination operates on the principle that short excerpts contain enough unique detail to be discriminable, while long excerpts cause the statistics to converge, rendering different samples of the same texture perceptually identical.

“What makes a texture a texture is what it's like on average.”

“If we've correctly identified the brain's representation of texture, then if we measure that representation… then the synthetic textures, they should look like the example.”

“The intuitive explanation of the failure is that there are these higher‑order statistical properties that are present in these natural textures that we might plausibly be sensitive to that are missing from this model.”

“Synthesis failures are really interesting because they point the way to new variables that might be important for the perceptual system.”

“We're not quite this far in really any other aspect of perception at this point.”