Our Company Brain Looked Good in Tests. Then We Asked Real Questions.

Our company memory looked reassuring when we asked questions written to test it. Then we tested the way I actually work.

On the written questions, search achieved a recall score of 0.80, measuring whether the expected note appeared among the top 3 results. This measured finding a record, not answering correctly. On a set of real prompts taken from our conversations, it found the expected note in 4/20 cases. The automatic memory helper had the same result on that real set.

That gap changed the conversation about our second brain. I could describe the folders, the automation and the ambition. I could not honestly turn the better score into a promise about everyday use.

For a business owner, this is the part that matters. You are paying for work to move forward with less repetition. A memory that performs well when somebody knows how to ask the test question may still leave you explaining the business again.

What we mean by a company brain

Our second brain is a persistent memory that an AI session can start from. It holds working preferences, important decisions, project dossiers and lessons from errors. It gives the assistant somewhere to look beyond the current conversation.

A project dossier is the working record of a project: its index, unresolved questions, decisions, knowledge notes and supporting material. Keeping these together means a decision has a home. It does not depend on somebody remembering which chat contained it.

The error catalogue serves a different purpose. It preserves failures and what we learned from them, so an assistant has a route to earlier experience before repeating an investigation. That is useful company knowledge even when the original conversation is no longer in view.

The starting memory is deliberately selective. Our session-start mechanism loads preferences, core behavioural instructions, active decisions and recent context. It points to other material for the assistant to open when needed. Where the working folder matches a project's recorded code location, it can also load that project's dossier.

This is more practical than pasting the entire archive into every conversation. The assistant needs an orientation and a way to find evidence. A large pile of text alone does not tell it which decision is current or which project I mean.

Why I wanted this in the first place

The business reason is continuity. I want to carry a decision forward without delivering the same briefing every time work resumes. I want our preferences to remain available, and I want a project to retain its reasoning when we move between sessions.

Re-explaining is work. So is checking whether an assistant has quietly returned to an option we already rejected. Those activities consume attention that could go into delivery, sales or a decision only the owner can make.

That describes the value we are trying to create. It is not a measured claim about hours saved or revenue gained. Our dossier does not contain evidence for a commercial return, and I will not manufacture one from a retrieval score.

There is also a difference between keeping a record and using it. A decision can be safely stored and still fail to influence the next answer (in Bulgarian). The file exists, but the assistant does not find it, does not open it or does not recognise why it matters.

Our tests forced us to separate those problems. Building a place for knowledge was necessary. Proving that the right knowledge reached the work was a separate job.

The test that exposed the gap

Recall measures whether search returns the expected material. In our reported recall test, success meant finding the expected note among the top 3 results. It did not mean that the final answer was correct, or that the task was completed.

The early result of 0.80 came from 50 written questions. These were questions made for evaluation, with a known note to retrieve. They gave us a useful development check, but they did not represent the language of a working session well enough.

We then examined 20 real prompts taken from transcripts. Search returned the expected note in 4/20 cases, and the automatic recall hook did the same. A hook is a small program that runs at a defined point in the assistant's workflow; this one supplies pointers to potentially relevant memory.

The figures describe different question sets, so this was not a before-and-after decline in the same test. It was a change in what we measured. The polished result had concealed a much weaker result on the work we wanted the system to support.

Real prompts were often fragments. They referred to a decision without repeating its title, or expressed the business problem in Bulgarian while the note used English terms. The information could be present without sharing the words that search needed.

This is why I now want evaluation material taken from actual use. Otherwise we risk measuring how well we can write questions that our own system understands.

Why the obvious fixes were not enough

The diagnosis showed several ways the right note could disappear. Broad notes could outrank a specific decision because they mentioned many topics. Short word fragments could match unrelated material. Relevant terms could be dropped from a long prompt before search used them.

We tried structural changes intended to reduce the advantage of broad notes. They did not improve the real set. The explanation sounded reasonable; the result did not support keeping them.

Another idea was to favour notes from the active project. In ordinary language, if I am working on a project, its records should receive extra attention. But that depends on knowing which project the conversation concerns.

The transcripts undermined that assumption. I often opened a conversation from the brain's main folder and described the work immediately. The folder did not identify the project whose decision I needed. An automatic preference based on location could favour the wrong dossier.

The project preference was built behind an optional setting, measured and left off by default. It did not improve real recall. We also tested using the task board as a project signal; that did not provide enough benefit to add it to the hook.

I see this as a useful discipline for an owner (in Bulgarian). A plausible feature still has a maintenance cost. When the evidence does not show that it helps the actual work, leaving it disabled is a legitimate result.

What we changed, and what remains an experiment

We changed the evaluation rule: real prompts come from transcripts, and we report their results separately. We do not keep adjusting the system against those same real examples until they produce a flattering score. That would gradually turn them into another set of questions we had taught it to pass.

We also made a small change to how the hook turns a prompt into search terms. It improved the original real set from 4/20 to 5/20. After expanding the real set to 50 prompts, the hook found 5/50. The improvement was real, but narrow.

We tested a model-assisted rewrite offline. Instead of searching only with the original wording, the experiment added English search terms generated by Haiku. Its best variant reached 7/20 on the original real set, below the agreed threshold of 8/20 for proposing the next experiment. We did not deploy that rewrite into the hook.

A separate search experiment was more promising. An embedding model turns text into numerical representations so search can compare meaning rather than only shared words. In an offline test, BGE-M3 semantic search found the expected note in 14/50 real cases, compared with 7/50 for our live hybrid search, which combines keyword and semantic results.

That was an experiment on our own material, with the live index left unchanged. It was not a deployed improvement, and it did not fix the automatic hook. Those distinctions belong in the article because they belong in the buying decision.

The current picture is simple even if the experiments are not: the automatic helper found 5/50 real prompts, direct hybrid search found 7/50, and the offline semantic experiment found 14/50. The assistant can still navigate the organised files when automatic suggestions miss. The stronger experimental search is not yet running in the live system.

The work changed our understanding of the limitation. Sometimes the wording was the problem. Sometimes the prompt simply did not contain enough context for an automatic helper to select the right note.

Finding a note and finishing the task are different measures

A weak automatic pointer does not mean the assistant has no other way to use memory. It can navigate an index, open a dossier and follow the records. That takes work, but it can still lead to the right result.

Our later outcome evaluation used 14 real tasks. Both the hook-assisted and the no-hook runs completed 14/14. The hook-assisted run averaged 3.3 turns, compared with 4.1 without it. A turn is an interaction step in the evaluated conversation.

The no-hook run still had access to the underlying memory. This was a comparison of automatic recall assistance, not a comparison between a company brain and an assistant with no company context.

This is a small task set, not evidence that every kind of company work succeeds. These results suggest that automatic pointers can sometimes shorten the route. They do not establish that the hook improves final correctness, that the whole system saves a particular number of hours, or that fewer turns necessarily mean lower cost.

I want those distinctions kept visible. A business can care about speed, money and quality at the same time. A single attractive score should not stand in for all of them.

What I would ask before buying company memory

I would ask to see the questions people actually typed and how the system performed on them. I would ask whether the assistant opened the relevant records and whether the task finished correctly. I would also ask which improvements are running and which exist only in a test.

I would want the underlying records to remain understandable. Preferences, decisions, project context and known errors should have clear homes. If the automatic retrieval fails, there should still be a usable route through the company's knowledge.

Our chosen product direction is a repository template that a buyer can run locally with their own content. That is a direction, not evidence of a finished rollout. The roadmap still contains work needed to make the system suitable for somebody else's business.

I remain convinced that carrying company knowledge between sessions is valuable. The lesson from our own brain is that keeping knowledge and reliably bringing it into the next task require separate proof. The real questions exposed that gap early enough for us to change the work.

If your team keeps re-explaining decisions to AI, or you are considering a company memory, bring that process to a free audit. We can look at the context your work needs, where it currently gets lost, and what evidence would justify an investment.