nanogpt-experiments

What if a transformer didn't have to commit to a fixed forward pass? This fork of Karpathy's nanoGPT served as a sandbox for an adaptive-compute architecture: train each block individually as well as in sequence, have consecutive blocks vote via an LIF-style evidence accumulator on whether to emit a token, and let the stack recur until a stable winner emerges. The design splits blocks into sensory / internal (recurrent) / motor stages, injects noise as a Monte-Carlo-like search mechanism, and decomposes embeddings into sensory, internal, motor, and goal components - tying the architecture back to a broader cognitive-homeostasis / hvar–cvar framing.

The work predates the recent industry push toward test-time-compute and latent-reasoning recurrence, and lands on a similar intuition from a different direction. Most of the architectural exploration lives in the README as design notes rather than committed model forks, but the repo includes two custom toy datasets - abab and patternloop - built as minimal probes for the recurrence and pattern-loop behaviors being studied, alongside standard Shakespeare and OpenWebText preparation scripts and back-of-envelope scaling and sizing notebooks. Paused as a thinking sandbox rather than a polished system.