Sahil Ahamad

AI Security: How We Got Here

Part 1 of a series on understanding AI from the ground up, so we can secure it.

I’m starting this series to write down what I’m learning about AI and machine learning from the ground up. I wanted to understand how we got here: the early ideas, the big breakthroughs, and how they led to the models we use today.

My interest in that history comes from security. If you want to secure an LLM, you need to understand what’s underneath it: how it learns, how it uses data, and why it behaves the way it does. Many of the security problems we face today make more sense once you understand how these systems developed.

This post follows that history, looking at what each generation of AI tried to solve, what changed when it worked, and what new problems came with it. It’s where I’m starting my own learning, and where I want to start this series.

1943: McCulloch and Pitts, the mathematical neuron

Could we describe a neuron using maths?

Warren McCulloch and Walter Pitts proposed a simple model: take several inputs, apply fixed rules, and return either 0 or 1. Their 1943 paper helped lay the foundations for artificial neural networks.

The mathematical neuron, 1943

The model couldn't learn. Its rules were fixed, and it left out most of the complexity of a real neuron. But connecting these units made it possible to represent logical operations, such as producing an output only when two inputs were active. That gave researchers a way to study networks without reproducing every detail of the brain.

The security connection comes later. In modern networks, learned weights help determine the output. Those weights, and the inputs we feed into the network, become things an attacker can target.

Resources

1950: Alan Turing, can machines think?

In Computing Machinery and Intelligence, Alan Turing proposed the imitation game. Could a machine answer questions well enough that a person couldn't reliably tell it apart from another human?

The exchange happened through written messages, so the judge couldn't rely on a voice or appearance. The focus was on what the machine could do in a conversation.

Turing also discussed machines that could learn, including the idea of teaching a simpler, child-like machine. The paper gave researchers a direction, though building such systems was still an open problem.

The imitation game, then and now

For this series, the security connection is convincing language. A machine that sounds human can also be used to impersonate one. LLMs make that concern much more immediate.

Resources

1957 to 1958: The perceptron, can a machine learn from examples?

Frank Rosenblatt's perceptron learned to separate inputs into groups by adjusting its weights. A programmer supplied examples instead of writing every rule. Each weight controlled how much an input contributed to the decision. When the perceptron made a mistake, its learning rule adjusted those weights to improve the answer.

But a single-layer perceptron could only separate groups with a straight line, or its equivalent in more dimensions. It couldn't learn XOR: output 1 when two binary inputs differ, and 0 when they match.

What one layer could and could not do

Minsky and Papert examined these limits in their 1969 book, Perceptrons. More complex patterns needed a different approach.

Learning from examples also gives us a security problem: someone who changes the training data may be able to change what the model learns. That's the basic idea behind data poisoning.

Where the trust boundary moved

The code still needs protecting. Now the training data does too. A program can run exactly as written and still learn the wrong thing from examples that someone has deliberately altered.

Resources

1959: Arthur Samuel, can machines improve through experience?

Writing rules for every possible checkers position would be a huge task. Arthur Samuel built a program that improved through experience, including games against itself. His 1959 paper helped establish the term “machine learning.”

The program still needed rules for checkers and a way to judge board positions. Learning helped it improve those judgements through play. It wasn't starting with no guidance at all.

More complex tasks still needed better algorithms and faster computers. But Samuel had shown that a programmer didn't have to supply every strategy.

Self-play later became central to AlphaGo Zero. For security, it leaves me with a question: if experience shapes a system's behaviour, what happens when someone can influence that experience?

Resources

1970s to 1986: Backpropagation, how does a network learn?

When a network gets an answer wrong, which weights should change?

Backpropagation calculates how each weight affects the error. A training algorithm uses that information to adjust the weights, then tries again. The method had earlier roots, but Rumelhart, Hinton and Williams' 1986 paper helped bring it to wider attention.

This mattered because a network could have several layers between its inputs and its answer. Backpropagation worked backwards through those layers, calculating how changing each weight would affect the error. The same calculation could be reused throughout the network.

What problem remained: training large networks still needed more data and computing power.

Why this matters for security: similar calculations can help an attacker find inputs that fool a model. Training changes the weights to reduce errors; a gradient-based attack changes the input to cause a mistake. Goodfellow and colleagues demonstrated this in their work on adversarial examples.

Same equation, two directions

The attacker doesn't necessarily need to change the model. They can search for an input it handles badly.

Resources

1980 to 1998: CNNs, can machines recognise images?

An image is a grid of numbers. Recognising what's in it means finding useful patterns in those numbers.

Kunihiko Fukushima's Neocognitron introduced ideas that helped shape convolutional neural networks, or CNNs. Yann LeCun and colleagues later trained networks to recognise handwriting. Their 1998 paper described LeNet-5 and systems for reading documents and cheques.

These networks learned visual features in layers, building from simple patterns towards more complex shapes. A CNN applies the same small filter across different parts of an image. That helps it recognise a pattern even when it appears in a different position, without learning a separate detector for every location.

What a vision model learns, and how it breaks

What problem remained: broader image recognition needed much more data and computation.

Why this matters for security: what looks obvious to us may not be obvious to a model. In work published in 2018, Eykholt and colleagues showed that carefully placed stickers could cause a road-sign classifier to misread physical signs.

A model can give the wrong answer with high confidence. That makes confidence alone a poor reason to trust it.

Resources

1996 and 1997: Deep Blue, can a machine beat a world chess champion?

Garry Kasparov beat IBM's Deep Blue in 1996. A year later, an improved version won the rematch, becoming the first computer to defeat a reigning world chess champion in a match under standard tournament conditions. IBM's account describes how the system combined fast search with chess expertise.

Deep Blue searched through possible moves and replies, then scored the positions it found. Chess knowledge helped guide that search. It was a powerful combination of computing speed and work by engineers and chess experts.

What problem remained: its ability was specific to chess. Winning the match didn't give it the ability to recognise an image or understand a conversation.

Which of these two approaches scales?

I keep this example in mind when looking at AI benchmarks. Doing very well at one task tells us little about how a system will handle another.

Resources

2009: ImageNet, what if machines had millions of examples?

Researchers needed larger, carefully labelled datasets to train and compare vision models.

Fei-Fei Li and her collaborators built ImageNet using images collected online and checked by workers on Amazon Mechanical Turk. Their 2009 paper described 3.2 million images. Human effort made that scale possible. Finding images was only part of the job: someone had to check that a picture labelled as a cat actually contained a cat. Using a shared dataset also made it easier to compare approaches on the same task.

What problem remained: training methods and hardware still had to make use of all that data.

Why this matters for security: collecting data from outside sources means trusting those sources, the labels, and the collection process.

The attack surface moves upstream

Later research showed what could go wrong. BadNets demonstrated models that behaved normally until an attacker supplied a particular trigger. In 2023, Carlini and colleagues showed how buying expired domains could let attackers replace content that web datasets pointed to.

The compromise can begin before training, then reach users through the finished model.

Resources

2007 onward: GPUs, what if we could train much faster?

Training a neural network involves repeating huge numbers of similar calculations. GPUs, originally designed for graphics, are well suited to running many of them at once.

Researchers used them to speed up training, making larger networks more practical. Raina, Madhavan and Ng's 2009 paper was an influential early demonstration.

The benefit went beyond fitting a bigger model. Shorter training runs also let researchers try more ideas and find out sooner which ones worked. The calculations were still demanding, but experiments that had been too slow became worth attempting.

What problem remained: more computing power only helped if the data and training methods could use it effectively.

Why this matters for security: faster computation also lets attackers test more possibilities when searching for inputs that fool a model.

Compute is dual use

For a public AI service, this is one reason to pay attention to repeated requests, usage limits and cost. Making a model available also gives people a way to test its weaknesses.

Resources

2012: AlexNet and the deep learning explosion

In 2012, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton entered AlexNet into the ImageNet competition. They trained their network using two GTX 580 GPUs.

The team also used techniques such as data augmentation, which creates variations of training images, and dropout, which helps reduce overfitting. The aim was to recognise new images, rather than just fit the examples used during training.

Their winning entry had a 15.3% top-five error rate, compared with 26.2% for the runner-up. That measures how often the correct label was missing from the system's five best guesses. The original paper reports the results.

Four things that had to arrive together

Backpropagation, CNNs, large datasets and GPUs came together in a result that was hard to ignore. None was new on its own, but the combination changed what researchers could achieve.

What problem remained: this was progress in image recognition. Building systems that could use language well was still a separate challenge.

Resources

What seventy years taught us

What stands out to me is how much these advances depended on one another. Better training methods needed more data. More data needed faster hardware. Together, they made learning approaches practical on a much larger scale.

Richard Sutton explored a related pattern in The Bitter Lesson: general methods that make use of more computation have repeatedly outperformed approaches built around carefully chosen human knowledge.

I want to explore what that means for security in the next post. It raises questions about defences built around lists of known attacks, though it doesn't prove that rules and filters are useless.

What we are actually defending

We now have more than code to protect. There is training data from outside sources, learned weights whose behaviour is difficult to fully explain, and inputs that may be designed to cause mistakes.

For me, the useful question is where an attacker can influence that process: the examples a model learns from, the model we download, or the input we send it. Looking only at the final answer misses much of what shaped it.

That's the background I wanted before getting to LLMs. From here, the series turns to the Bitter Lesson, then to language models and the security problems they bring.

Resources

  • Essay: Richard Sutton, The Bitter Lesson (2019). The short argument that the next post will explore.
  • Book: Ram Shankar Siva Kumar and Hyrum Anderson, Not with a Bug, But with a Sticker. Further reading on the security questions running through this post.
  • Documentary: The Thinking Game, available from Google DeepMind. A later view of AI research through Demis Hassabis and the DeepMind team.

← All posts