Arpit Mishra
Fullstack AI Developer
Socials
From ANNs to attention, positional encoding and training
Keys and Values never change once a token is processed — the KV cache is the simple trick that stops a transformer from recomputing them on every single generation step.
A model that memorizes its training set has learned nothing useful — regularization is the toolbox for pushing it toward the goal that actually matters: performing well on data it's never seen.
Two opposite ways a model can fail to generalize, and how to tell which one you're looking at from nothing more than a training curve.
Without a nonlinearity between layers, a neural network of any depth collapses into a single straight line — activation functions are what make depth actually mean something.
Before a single number reaches a neural network, raw text has to be cut into discrete units and cleaned into a consistent form — the unglamorous step that quietly shapes everything downstream.
The chain rule, run backward through a network, is the entire mechanism behind how a model learns — here's how that one idea scales into a real training loop.
Why GPT-style models write one token at a time, what that costs at inference time, and why parallel decoding is so hard to get right.
Same formula, different inputs — how swapping where Q, K, and V come from turns one attention mechanism into two very different tools.
The transformer threw away recurrence and processed every token in parallel — so how does it know which word came first? A look at the sinusoidal trick that solved it.
Why plain neural networks couldn't handle language, and how each fix — recurrence, gating, attention — quietly set up the next one, ending in the Transformer.
