Inference Optimization and Scalable AI: Insights from YC Paper Club

The YC Paper Club brings together founders and researchers to discuss cutting‑edge AI work. Hosted in a historic venue that once nurtured the Winter 16 batch and early OpenAI efforts, the club emphasizes technical depth and open dialogue.

Speculative Decoding (SSD)

Inference speed is presented as the “peak intelligence” a model can deliver, shifting the focus from cost reduction to capability. Speculative decoding traditionally runs sequentially, but SSD parallelizes token drafting with verification of previous drafts. By predicting verification outcomes from draft token distributions, SSD hides drafting latency. This design yields speedups for both latency and throughput, reaching 300 tokens per second on Llama 3 70B using four H100 GPUs.

Diffusion Model Predictive Control (DMPC)

Model Predictive Control enables agents to adapt to new rewards and dynamics at test time, but compounding errors limit robotics performance. DMPC addresses this by employing diffusion models to generate multi‑step action proposals and a learned dynamics model to evolve them. The factorized approach separates action proposal from dynamics, allowing modular adaptation and reducing error accumulation.

Latent World Models

World models learn to predict future observations from current states and actions. The SIGG regularizer—Sketching, Isotropic, Gaussian—prevents trivial representational collapse by enforcing Gaussian, isotropic latent embeddings through 1‑D slice losses across high‑dimensional space. Latent operations achieve planning that is up to 50× faster than competing methods, run on a 15 M‑parameter model with less than 24 GB VRAM, and support uncertainty quantification by detecting model error via perturbations.

Deep Learning Theory

Overparameterization improves generalization by guiding models toward more compressible solutions; flat minima are more compressible than sharp minima. Benign overfitting arises because regularization biases models toward lower‑order terms on structured data. Applying PAC‑Bayes bounds and soft inductive biases clarifies why scaling models often yields better performance.

Data‑Constrained Scaling

When data is limited but compute is abundant, traditional compute‑optimal scaling laws (e.g., Chinchilla) no longer apply. Aggressive regularization and ensembling provide substantial data‑efficiency gains, while distillation transfers test‑time compute into training‑time compute. Joint scaling recipes that combine ensembling, regularization, and distillation can deliver up to a five‑fold improvement in data efficiency, with continued pre‑training offering up to 17× gains.