AI’s Goldfish Memory (And How to Fix It).
We’ve all been there. I’m having a really productive and engaging conversation with my AI assistant of choice about a marketing idea. I’ve given it loads of context, asked it to put its social marketing experts hat on and we’ve talked through some scenarios, but then, mid-flow, it suffers some kind of memory loss and gives me a load of flannel that’s vaguely related but not really what I’ve asked about. Even worse, it’s also contradicted something it said two messages ago and now I’m not sure what’s fact and what’s hallucination. It’s frustrating and annoying, and it feels like a waste of my time, especially if I have to start again from the beginning. So why does it happen, and more importantly, how can we avoid it?
Let’s deal with the why of it first. It is baffling that a machine that has access to the whole conversation up to that point, not to mention a whole library of conversations we’ve had before, can forget what we’re talking about mid-flow. Is it a form of machine dementia? Is the AI assistant throwing a wobbly because it doesn’t like my tone? Or is it a money-making ploy to force me to pay for a subscription? In fact, it’s none of these things; I’ve just hit what is known as ‘context collapse’. And it’s all about how a large language model remembers things: it doesn’t have a ‘memory’ as we humans would define it, instead it relies on something called ‘tokens’.
What are tokens?
AI doesn’t read and understand the written word like a human does. Instead, it breaks these words, and any punctuation, down into small chunks for easy processing. Short words like ‘and’, ‘the’ or ‘one’ are usually left whole, while longer words can be split into a few pieces. For example, the word ‘pianist’ could become ‘pian’ and ‘ist’, as in this tokenised sentence:
The
enthus
iastic
pian
ist
care
fully
pract
ised
the
mel
ody
.
In reality, the exact splits are decided by the model’s internal rules and, with the addition of markers for spaces and special symbols, tokenised text looks pretty odd to our human eyes. But the idea is the same: long words are chopped into smaller pieces so the AI can reuse them across many different words. AI models are trained on huge datasets of these bite-sized bits of text, and they learn the most likely patterns in sequences of them. When I type a message into the prompt box and hit send, the AI breaks this into tokens, processes it and generates its response in tokens. It then converts these tokens back into text so I get a normal, readable reply in natural language.
And this is where issues can occur. AI models have a token limit, known as a ‘context window’. This context window only has room for a certain number of tokens – for some newer models, the upper limit is on the order of hundreds of thousands of tokens. Once the maximum number of pieces of text is in the context window anything above that has to be cut off, or it pushes older tokens out to make space. These limits contribute to forgetting earlier parts of the conversation (or ignoring parts of a very long prompt) because tokens at the start of the conversation drop out of the context window to make room for the newer input. Importantly, this token limit includes both the words and punctuation in the prompt and the documents we upload, and also the words and punctuation in the AI models reply. If your prompt uses around 99,000 tokens, there is only room left in the context window for roughly 101,000 more tokens of response.
This lack of a working memory isn’t so much of a problem if you want AI to check the weather or find a nice holiday destination, or if you use it for the odd rewrite of an email or to rustle up a social media post. But if you have a deadline and you need your AI assistant to read a 10,000 word document and then break it down into a 2000 word summary, you may run into problems. This forgetfulness is known as ‘context collapse’ and it manifests itself as the AI model becoming confused or going off topic. Information can be repeated and the model can contradict itself.
It’s important to keep this context window in mind when we’re working with an AI model on a more complex task. If you’re getting help with a difficult pitch or proposal or planning a complex launch campaign and your model of choice goes completely off piste it’s not only frustrating, it feels like a waste of precious time. Luckily there are ways to help keep the AI model on track and focussed – just like we need the odd coffee to keep us going, AI sometimes needs a bit of digital caffeine to keep it on track.
Keeping the Focus.
Talking to AI models can feel a lot like talking to people: if you dump too much information on them at once, or keep jumping between topics in a single conversation, they’re likely to lose track of what matters, get annoyed and give you vague or off‑target answers. But this isn’t AI getting stroppy with you because AI models are endlessly patient and good-humoured This is about helping AI stay on track with good prompting habits. Here is a list of the most important things to keep in mind when you’re next talking to an AI model:
Start with a clear goal - state what you want to achieve, who your target audience is, why you need the information and how you’d like the reply to be formatted.
Ask for one thing at a time – avoid stuffing your prompt with multiple different requests. Concentrate on one subject per conversation.
Format your prompt – break your text into sections such as ‘Context’, ‘Task’, ‘Examples’. Most AI text boxes work by sending the message when you press Enter. To get round this and add a new line without sending the prompt, use Shift + Enter on Windows/Linux, or Shift + Return on Mac.
Keep background details brief – context helps, but too much context can be confusing. Keep it short, relevant and to the point.
Summarise as you go – if you’re having a long conversation, recap every so often. ‘Quick summary: we agreed X, Y, Z. Next, please help with…’
Regularly repeat important constraints - if something really matters (tone, audience, word count, brand rules), repeat it briefly when you ask for a new step.
Break big documents into smaller sections - for long reports or articles paste a section at a time and ask for a summary or notes. Once you’ve got all of the summaries you can ask for ‘a summary of these summaries.’
AI loves a label – if you’re pasting in text tell the model what it is: ‘Here is a product description’, ‘Here is an email draft’, ‘Here is customer feedback.’ This helps the AI treat it correctly.
Start a new conversation for each new topic – make use of the library feature to keep each subject separate so you can find old conversations, carry on where you left off, and most importantly, prevent the AI getting confused and losing track. Mixing several projects in one thread makes it more likely that the AI will lose track.
Use a step‑by‑step process for complex tasks - ask it to outline first, then fill in each section. This uses the context window more efficiently and keeps the structure clear.
Set sensible length limits – keeping to a word count (500 words; 3-4 bullet points) helps to keep responses focused, saves space in the context window for the rest of your conversation, and makes it less likely the model will wander off into unnecessary detail.
Save and reuse good prompts - if a prompt works well, save it so that you can edit and reuse for future conversations. Reusing a proven structure is easier than reinventing it and helps keep future chats focused.
These twelve steps to refine and streamline the way you prompt may seem like a lot, but you can use your AI model to help with this too. You can start with a list of all of the things you want to talk about – a business idea, writing a Facebook post, which local MOT centre gets the best reviews etc, and ask the model to put it into a logical to-do list. For each item on the list you can start a new conversation and each conversation can be saved in your library so that all of that information is accessible and organised. Once you’re organised by subject it’s much easier to start refining your prompts and pinpointing exactly what and how you want the AI model to talk to you.
Of course, effective prompting is a little bit like learning a new language – its about trial, error and using the new language regularly. With time and practice you’ll be prompting like a pro in no time. For more tips and tricks check out the rest of the blogs, plus there are resources and tutorials in the members area.