Introduction I have a collected a handful of GGUF models sitting in ~/.cache/llama.cpp. I wanted a straight answer to the simple question: how good are these local models at recalling facts and doing simple reasoning? So I built a small harness that runs my local models against the full MMLU test set (14,042 questions across 57 subjects) using llama.cpp, one model at a time, and records every single prediction. This post covers what MMLU is, how to get the questions, how the benchmark works, the example questions, and the results.

Introduction In the previous article we built engine.py: a complete reverse-mode automatic differentiation engine in about 80 lines, built around a single Value class. That engine knows how to record arithmetic on scalars and hand back the gradient of the final result with respect to every scalar that fed into it. This article builds the next layer on top: nn.py, a small neural-network library in roughly 60 lines (heres the repo).

Introduction In the first two articles of this series we built a reverse-mode autograd engine (engine.py) and a small neural-network library on top of it (nn.py). Together they are about 140 lines of pure Python with no numerical dependencies beyond the standard library. This article puts them to work on a real problem: predicting whether a breast tumor is malignant from measurements taken from a digitized image. The script is cancer_prediction.

Introduction engine.py is a complete reverse-mode automatic differentiation engine in about 80 lines (heres the repo). It has a single class, Value, that wraps one scalar and remembers how that scalar was computed, so that calling .backward() fills in the derivative of the final value with respect to every value that fed into it. It’s the same core idea as PyTorch’s autograd, just small enough to read in one sitting. The code is also strongly influenced by Karpathy’s micrograd repo.

Introduction A common way to learn how Convolutional Neural Networks (CNNs) work is to develop a tool that can classify whether an image is a cat or a dog (in this case all images are either of a cat or of a dog, no other options are allowed). I created an example that does just that. Since my background is in polymer mechanics, I wanted a more interesting version of this classification task: given an image of a stress-strain curve, decide whether it comes from a rubber material or a thermoplastic.

Different Types of Neural Networks (NN) There are several types of neural networks designed to perform specific tasks or process different kinds of data. Here are some of the most popular basic types: Feedforward Neural Networks (FFN) The simplest type of neural network, consisting of interconnected nodes. Convolutional Neural Networks (CNN) Primarily used for image and vision-related tasks such as object recognition. Reccurent Neural Networks (RRN) Designed to process sequential data, like time series, by maintaining internal state variables.