SplatWeaver: Learning to Allocate Gaussian Primitives for Generalizable Novel View Synthesis

Abstract

In this work, we propose SplatWeaver, a feed-forward framework capable of allocating dynamic number of Gaussian primitives across spatial regions for generalizable novel view synthesis. In contrast to existing methods that typically predict uniform per-pixel or per-voxel Gaussian primitives which fail to adjust for spatially varying complexity, our approach is able to dynamically distribute Gaussians across different spatial regions, enabling more flexible and expressive 3D scene modeling. In particular, we introduce the concept of cardinality Gaussian experts, wherein each expert specializes in predicting a specific number of Gaussian primitives (0 to M). These experts are then orchestrated via a pixel-level routing scheme, enabling flexible allocation of Gaussians across the scene. Rather than directly regressing complete Gaussian parameters, each expert predicts a set of hidden Gaussians comprising positions and associated latent features, which are subsequently aggregated with spatial neighbors to predict the final parameters, yielding more coherent and precise primitive attributes. Additionally, we introduce the high-frequency prior with attendant guidance module and routing regularization to stabilize expert routing and facilitate a more complexity-aware allocation. Such a paradigm endows “dense where complex, sparse where smooth” allocation property, yielding superior efficiency and enhanced rendering quality. We highlight the superiority and effectiveness of SplatWeaver across a wide range of scenarios: it dominantly outperforms existing state-of-the-art methods by delivering more faithful renderings with economic primitives.

Method Overview

Our core insight is to adaptively allocate Gaussian primitives according to scene complexity, instead of predicting a uniform number of per-pixel or per-voxel Gaussians, thereby avoiding redundancy in simple regions and deficiency in complex areas. In particular, we advocate the concept of cardinality Gaussian experts, where each expert is responsible for predicting a specific number of Gaussian primitives (ranging from 0 to M). Allocation across regions is then achieved via pixel-level cardinality Gaussian expert routing.

Performance

SplatWeaver achieves consistent state-of-the-art performance across three benchmarks in pose-free generalizable novel view synthesis.

Visual Comparison

Comparison of predicted Gaussian distributions and novel view synthesis performance. SplatWeaver dynamically distributes Gaussians across different spatial regions in accordance with scene complexity. By concentrating primitives in intricate areas while maintaining sparsity in smooth regions, it achieves higher-quality rendering with a more compact representation.

BibTeX

@article{wan2026splatweaver,
  author    = {Yecong Wan and Fan Li and Mingwen Shao and Wangmeng Zuo},
  title     = {SplatWeaver: Learning to Allocate Gaussian Primitives for Generalizable Novel View Synthesis},
  journal   = {arxiv},
  year      = {2026},
}