Building a useful agent · Memory
Understanding an AI agent's memory through Harry Potter
August 7, 2026 · 3 min read
Compare what you remember from the Philosopher's Stone to what our agent remembers.
We talked about why and how memory is created in the previous posts (the overview, capturing, dreaming). But illustrating what an AI agent remembers is a bit tricky, as it depends on the experiences the agent has had.
Harry Potter and the Philosopher's Stone may be one of the most known works of art of our time. So we ran it through an agent's memory, letting it dream after each chapter. Below you can see the memory build up and change over time.
Harry Potter owns an invisibility cloak that originally belonged to his father, bequeathed to Harry by an unknown sender who had held it since James Potter's death.
→
Harry Potter's forehead scar resulted from a powerful dark curse that killed his mother and father but failed to affect Harry himself, which is the reason for his fame in the wizarding world.
Quirrell was serving as a host to Lord Voldemort, who was living with him having met and shaped him with his philosophy of power over good and evil.
Quirrell could not touch Harry's bare skin without suffering severe pain — contact caused Quirrell serious injury.
Hagrid's hut is located on the grounds of Hogwarts, reachable via a corridor whose entrance can be blocked.
Snape hated Harry Potter's father but was once saved by him, creating a debt; Snape resolved this by protecting Harry during Harry's first year at Hogwarts.
Harry has a thin lightning-bolt-shaped scar on his forehead that he has had for as long as he can remember; Aunt Petunia told him he got it in the car crash when his parents died.
→
The three-headed dog at Hogwarts is guarding Nicolas Flamel's Sorcerer's Stone, which Dumbledore is keeping safe on Flamel's behalf because the two are friends, while someone is actively attempting to steal it — a matter Hagrid insists does not concern Harry, Ron, or Hermione.
→
The Sorcerer's Stone, as of the events described, was being kept under protections that Professor McGonagall considered impossible to breach by theft.
→
Harry found his tea with Hagrid more thought-provoking than any of the lessons he had attended so far at the castle.
Harry Potter's invisibility cloak originally belonged to Harry's father and had been kept in Dumbledore's possession before being passed to Harry.
Snape was observed by Harry heading toward the Forbidden Forest in a furtive manner — hooded and moving swiftly to avoid being seen — while the rest of the castle was at dinner.
Mrs. Weasley knitted Harry Potter an emerald green sweater and included a large box of homemade fudge as a gift.
As of Dumbledore's revelation, the Sorcerer's Stone had been destroyed, and Nicolas and Perenelle Flamel had only enough Elixir of Life remaining to settle their affairs before dying.
Ginny Weasley recognised Harry Potter by sight at King's Cross station.
The Weasley family has at least three brothers who attended Hogwarts together: Percy, Fred, George, and Ron.
Fred and George Weasley noted that Mrs. Weasley puts more effort into sweaters she knits for non-family members than for her own children.
Scabbers is a gray rat whose whisker ends appear slightly lighter; a spell Ron attempted to turn him yellow produced no effect.
A few things to try. Pick "The hunt for the Stone" in the Focus menu and step through the final chapters: the agent spends most of the book convinced Snape is the thief, and you can watch that belief get overturned when the truth comes out.
It is not perfect, like how it states as fact that Snape got past the three-headed dog using a harp, when that was only the trio's confident guess. That is the nice thing about running a well-known work through the memory: you can spot the mistakes yourself.