When an AI does something surprising—whether brilliant or baffling—it’s rarely just one thing that caused it. Data biases, choice of training goal, neural architectures, system-level guardrails, and conversation context all interact in complex ways. Understanding these components helps us move beyond “the AI is just weird” and toward a richer picture: one where misbehavior can be traced, mitigated, and, ideally, prevented.

The behaviors of a Large Language Model (LLM) are due to a combination of many components, including, but not limited to: 

• The data;
• the goal(s) the LLM is trying to achieve; 
• the architecture of the LLM; 
• the system the LLM is embedded in; and
• the context of inputs and outputs for the LLM.

In this piece, I will look at each of these components and explore their impacts on the behavior of LLMs. Below, Figure 1 shows an overview of how LLMs are trained. Figure 2 shows how they typically operate in deployment.

Figure 1: Training a Large Language Model

Figure 2: Operating a Large Language Model

Data

“Garbage in, garbage out” is a common saying in the machine learning community and problems with data quality account for a large portion of AI misbehavior. Intuitively, an image classification system that was trained only on pictures of cats is going to have a hard time positively classifying images of bicycles; a resume classification system trained mostly on resumes with male names will suffer when classifying resumes with female names; and a language model trained on mostly english text is likely to make grammatical errors in languages for which it has less data.

But these are just the more obvious cases. Sometimes the data used for training has subtle clues that teach the AI to “cheat” instead of learning more robust patterns in the data. For example, a medical diagnosis model trained on chest X-rays once learned to rely on hospital-specific watermarks rather than actual anatomy—predicting disease risk based on which hospital produced the image, rather than on the patient’s condition. Similarly, an early image generation system learned how to turn pictures of horses into pictures of zebras, but if the horse had a rider, then the zebra image would include a rider with zebra stripes.

Even slight imbalances in the data can lead to dramatic misbehavior. A dataset that included zebras and horses with riders and in a wider variety of environments would have likely prevented such failures.

Training Goal

You may have heard that LLMs are “next word predictors” or “next token predictors”—and that’s true. The training goal defines what “good” performance means. The training goal is often expressed as a “loss function,” and minimizing the loss function is the goal of training a model. 

During pre-training, an LLM is trained on the bulk of the data available. In this stage, advancing the training goal means minimizing the difference between the model’s prediction of the next token and the actual next token in the training data. For example, if the LLM confidently predicts that the text “to be or not to …” ends with the word “the”, then the loss function will be high. The training algorithm then apportions the punishment for getting the prediction wrong across the neurons in the LLM. This update will cause the LLM to avoid predicting “the” as the completion to the text “to be or not to …,” and become more likely to predict the word “be.” At the end of pre-training, an LLM is capable of predicting text, but will struggle with refusing harmful requests and will not have learned how to think through problems.

Most frontier models go through additional training stages, known as post-training. During post-training, secondary training objectives and loss functions may be introduced. Techniques like Reinforcement Learning from Human Feedback (RLHF) encourage the model to align its outputs with human preferences, while Chain of Thought (CoT) tuning uses extra data that demonstrate step-by-step reasoning to improve the model’s own ability to reason step-by-step. RLHF is commonly used to align the style and content preferences of an LLM with human preferences. For example, the LLM may be prompted to produce hate speech and, because it has not yet been aligned with a preference against such content, it complies. A human can then provide feedback indicating that this was a bad response to the prompt. That feedback is used to update the neurons of the LLM causing the LLM to be less likely to generate hate speech in the future.

These new goals reshape the model’s behavior, sometimes improving it, but also potentially introducing new quirks. For example, in April 2025, post-training changes in ChatGPT caused increased sycophancy, the model had to be rolled-back, and the secondary training objective was revised. And in April 2026, compounding effects from two post-training processes led GPT models to become strangely obsessed with goblins.

Neural Architecture

The “brain” of the AI matters too. In the early 2010s, recurrent neural networks (RNNs) and long short-term memory (LSTM) models were state of the art, but they struggled to capture long-range dependencies in text. A paragraph could drift off-topic because the model literally “forgot” what was written ten sentences earlier.

In 2017, the transformer architecture revolutionized text generation by introducing attention mechanisms, allowing models to “look back” across the entire context of a conversation. This architectural leap is what makes today’s large language models possible.

Even smaller architectural changes—like modifying attention spans—can meaningfully change how an AI reasons and what errors it makes.

System Architecture

An AI’s system architecture—the scaffolding around the raw neural network—also shapes its behavior.

When DeepSeek R1 was released, many users noticed it refused to criticize the Chinese ruling party. Yet when the raw neural network weights were downloaded and run independently, those refusals disappeared. The difference wasn’t in the model itself, but in the system wrapped around it—a layer filtering or rewriting responses.

System choices also determine how the next token is selected. Instead of always choosing the single most likely token (which can lead to repetitive or bland text), systems often use sampling strategies which choose from the most likely tokens, cut out improbable tokens, or even try building multiple potential sentences at once. Any choice can shift the model’s personality to be more deterministic, creative, or cautious. A common default sampling strategy is to simply select the most likely token, but this often leads to repetitive text when generating longer replies.

Other system-level features, such as retrieval-augmented generation (RAG) or external tool use (like code execution or web search), further change the AI’s behavior, sometimes dramatically. An AI that can use a calculator will do math much more reliably than one that only guesses numbers.

The impact of system architecture on the behavior of an AI system is especially important in the case of AI agents. AI agents use a scaffold of code and prompts to help the LLM solve complex, context-dependent, and recurrent problems. For AI agents the scaffold can be as impactful on the system’s behavior as the core LLM.

Context

Finally, AI systems don’t operate in a vacuum. Every conversation you have with a chatbot takes place within a context window—the running record of your inputs, the model’s outputs, and often a hidden “system prompt” that defines how the model should behave.

That system prompt can be surprisingly powerful. In some infamous cases, internal prompts leaked and revealed instructions such as “Never talk about goblins…” or even bizarre biases that distorted behavior (at one point, Grok’s chatbot responses would routinely devolve into conspiratorial nonsense about white genocide in South Africa due to system prompt sabotage).

The context window also makes the model sensitive to phrasing and history. A single offbeat instruction or misleading prior message can ripple forward, producing odd or harmful behavior many turns later. This dynamic is at the core of multi-turn attacks used to make LLMs misbehave.

Conclusion

The many components of an AI system can each fail individually, or interact in surprising ways to create surprising behavior. Fixing problems with these components can require a lot of time and research, and unfortunately, these fixes aren’t guaranteed to work. Because the causes of misbehavior in AI systems are so intertwined, fixing one problem might make the system vulnerable to a new kind of attack. Still, incremental progress in the performance of LLMs has shored up many of the flaws and quirks I have pointed to. So, while the question “Why did the AI do that?” doesn’t have a single answer—it does have many, and the future of trustworthy AI depends on asking it well.