A short history of neural networks (and why it matters for your AI vendor decisions)

Before I started building on my own hardware, I did something most people skip when they get into AI: I read the history first. Not for trivia — for pattern recognition. If you’re a business owner or IT leader trying to figure out which parts of the current AI wave are durable and which are hype, the history is the fastest shortcut I know.

Here’s the short version.

The idea, and the first winter (1943–1986)

Neural networks are almost as old as computing itself. In 1943, McCulloch and Pitts described the first mathematical model of a neuron — weighted inputs, a threshold, an on/off output. By 1958, Rosenblatt had built the Perceptron, the first trainable neural network, as actual hardware at Cornell. It made headlines as “the machine that learns.”

Then, in 1969, Minsky and Papert proved that a single-layer perceptron couldn’t solve a problem as simple as XOR. Technically, they were right about that one architecture. But the finding got generalized into “neural networks are a dead end” — and funding for the whole approach collapsed for over a decade.

That’s lesson one: a real limitation of one specific architecture is not the same as a limitation of the underlying idea. It happened again, more than once.

The comeback, and the second winter (1986–2006)

Backpropagation — popularized in 1986 by Rumelhart, Hinton, and Williams — made it practical to train multi-layer networks. XOR, solved. CNNs (1989) and LSTMs (1997) followed, giving neural nets a real foothold in image and sequence data.

Then the field went quiet again. Support vector machines and other classical methods often won on the small, noisy datasets of the era, and deep networks were expensive to train on the hardware available at the time. A small group — Hinton, Bengio, LeCun, later nicknamed the “godfathers of deep learning” — kept pushing anyway.

The ignition point (2006–2017)

In 2012, a deep convolutional network called AlexNet won the ImageNet competition by a margin nobody expected — trained on GPUs. This is usually cited as the real turning point, and I’d argue it’s the most important lesson in the whole timeline: the ideas from the ’80s and ’90s hadn’t changed much. What changed was that compute and data finally caught up to the ideas. GPUs moved from gaming hardware to the center of AI research almost overnight.

Word embeddings, GANs, and attention mechanisms followed in 2013–2014, setting up the next jump.

The transformer era, and where we are now (2017–today)

“Attention Is All You Need” (2017) dropped recurrence entirely and used only attention, enabling massively parallel training. Every major LLM today — including the one that likely generated the first draft of a paragraph in this post — descends from that architecture.

GPT-3 (2019–2020) showed that scaling alone produced qualitatively new capabilities, without new architecture. ChatGPT (November 2022) wasn’t even a new model — it was a product wrapper around GPT-3.5 that made the whole category legible to the general public and to enterprises overnight.

And 2026 is where things get interesting for the reason I actually started this business: open-weight models are now closing the gap with the frontier, and running a genuinely capable model on consumer-grade hardware — a MacBook, a single workstation GPU — has gone from a hobbyist curiosity to a real deployment option. That shift is what makes “private AI,” in the literal sense of AI that runs on your own infrastructure under your own control, something a small or mid-size company can seriously consider today, not just a big enterprise with a data center budget.

The pattern worth taking away

Every “overnight” breakthrough in this timeline was really: an old, previously-abandoned idea, plus new compute or new data, equals a discontinuous jump. Perceptron → backprop → CNNs and LSTMs → deep learning → transformers → scale → and now, a real open-weight alternative to the closed frontier.

If you’re evaluating an AI vendor, a tool, or a strategy for your business, that pattern is a useful filter. Ask: is this genuinely new capability, or is it the same underlying idea from five or ten years ago, now viable because compute got cheaper or a model got better? Both can be worth adopting — but they carry very different risk profiles, and knowing which one you’re looking at changes how much you should bet on it staying relevant.

That’s the lens I bring to this practice, and it’s exactly why I think it’s worth understanding where all of this actually came from before deciding where to place your next AI investment.