Neural Net Quiz

Picture a judge at a talent show. Each performer (input) gets a score multiplied by how much the judge trusts their own taste in that genre (weight). The judge adds a personal bias — maybe they always lean generous — and then decides: does this act pass to the next round? That final yes/no decision is the activation function. Every neuron in a network does exactly this: weighted sum + bias + activation = output.

Weights are the numbers the network adjusts during training. Think of them as volume knobs on a mixing board. Some inputs get turned up loud (high weight = important), some get muted (low weight = unimportant), and some get inverted (negative weight = this input pushes the output down). Training is the process of finding the perfect setting for every knob.

Without bias, a neuron with all-zero inputs always outputs zero. Bias is like a thermostat's default setting — it shifts the point at which the neuron "fires." A positive bias means the neuron is eager to activate; a negative bias makes it harder to trigger. This gives the network flexibility to fit patterns that do not pass through the origin.

Activation functions introduce curves into what would otherwise be a straight-line calculation. ReLU (the modern standard) is like a floor at zero: negative signals get silenced, positive signals pass through unchanged. Sigmoid squashes everything into a 0-to-1 range — perfect for probabilities. Without these gates, stacking layers would be pointless — the whole network would collapse into a single linear equation.

The input layer receives raw data (pixels, numbers, text). Hidden layers transform that data through learned patterns — first layer finds edges, second finds shapes, third finds objects. The output layer makes the final prediction. More layers means the network can learn more complex representations, but also needs more data and compute to train.

Data flows from input through hidden layers to the output. Each neuron computes its weighted sum + bias + activation. The final output is the network's prediction — maybe "92% cat, 8% dog." On the first try, this prediction is essentially random because the weights have not been trained yet.

Compare the prediction to the correct answer using a loss function. If the network said "92% cat" but the image was a dog, the loss is high. If it said "95% dog," the loss is low. The loss function turns the error into a single number that the network can minimize.

Work backwards from the output to figure out which weights contributed most to the error. Each weight gets a "blame score" (technically called a gradient) that says how much it should change and in which direction. Weights that contributed a lot to the error get adjusted more.

Adjust every weight by a tiny amount in the direction that reduces the error. The size of the adjustment is controlled by the learning rate — too large and the network overshoots, too small and training takes forever. Then repeat: forward pass, loss, backprop, update. Millions of times.

Adding layers increases the network's capacity to learn complex patterns, but it also requires more training data and compute. A network that is too deep for the available data will overfit — it memorizes the training examples instead of learning general patterns. A 3-layer network trained well on enough data often beats a 100-layer network trained poorly.

A neural network that classifies cat photos does not "see" a cat the way you do. It detects statistical patterns in pixel values — edges, textures, shapes — that correlate with the label "cat." It has no concept of what a cat is, what it feels like to pet one, or that cats are alive. Pattern matching is powerful, but it is not understanding.

Architecture determines what the network can learn. Data determines what it does learn. A perfectly designed network trained on biased data will produce biased outputs. A network trained on too little data will overfit. A network trained on noisy, mislabeled data will learn noise. Data quality is at least as important as architecture quality.

Artificial neurons were inspired by biological neurons, but they are radically simplified. A biological neuron uses electrochemistry, has timing-dependent behavior, and connects to about 7,000 other neurons on average. An artificial neuron is pure math: multiply, add, threshold. The inspiration was useful, but modern AI has diverged far from neuroscience.