Arnaldo Sepulveda
Career note · Applied AI

The knowledge was already there. The system to use it was not.

The support problem that pushed me toward applied AI was not a lack of answers. It was repeatedly rediscovering knowledge the organization had already paid to discover.

· 8 minute read

For much of my career at Genesys, I worked in enterprise technical support and Business Applications across knowledge systems, classification, digital interactions, customer and interaction data, agent applications, integrations, and production operations. That work regularly crossed team boundaries, bringing me together with customers, product specialists, product managers, developers, technical leaders, and other engineering teams.

One pattern kept bothering me.

A customer would encounter a difficult problem.

It might involve a particular product version, configuration, integration, deployment topology, or combination of components. A support analyst would investigate. Sometimes the case would move between several people. Sometimes we would involve a product specialist. Sometimes development or engineering would need to explain what the software was actually doing.

Eventually, someone would figure it out.

The customer would get an answer.

The case would close.

From the perspective of the individual ticket, the process had worked.

But that was not always the end of the problem.

Months later, another customer might encounter something very similar.

A different analyst would begin investigating.

Logs would be collected again. Configuration would be compared again. Someone might ask around to see whether anybody remembered seeing the behavior before. Development or engineering might get pulled into another conversation.

And eventually we would rediscover something the organization had already learned.

That was the part I could not stop noticing.

The answer existed somewhere

The organization often already possessed the relevant knowledge.

It might have been in a support case.

It might have been buried inside an engineering discussion.

It might have existed in product documentation but required very specific context to understand that it applied.

It might have been recorded in an internal chat between an analyst and a developer.

Or it might simply have remained in the memory of the people who had handled the original incident.

The problem was not always creating new knowledge.

The problem was turning existing knowledge into something operationally useful for the next person who needed it.

That distinction became increasingly important to me:

The organization already possessed the knowledge, but had not operationalized it so the right person could reliably get the right answer in the right context.

There is a large difference between storing information and making it usable.

A closed support ticket is stored information.

A developer's explanation in a chat is stored information.

A 200-page product manual is stored information.

None of those automatically gives the next analyst the correct answer for this customer, on this version, with this configuration, under these particular circumstances.

That is an operational systems problem.

Repetition is expensive in ways that are easy to miss

The obvious cost is time.

If an experienced support engineer spends two hours reconstructing something the company already worked out six months earlier, those two hours could have gone toward a genuinely new problem.

But the cost is larger than that.

The customer waits longer.

Development and engineering get interrupted by questions they may have answered before.

Different analysts reconstruct the same issue differently.

Useful context gets lost between teams.

One engineer knows that a particular workaround applies only after version X. Another remembers that it fails under a certain deployment configuration. A third knows that the apparent symptom is misleading and that the real problem is somewhere else.

The organization's knowledge becomes distributed across people rather than reliably available through the system.

This is not a criticism of support engineers.

In fact, it is almost the opposite.

Good support engineers are valuable because they can reason about unfamiliar environments, recognize when the obvious answer does not fit, interpret incomplete evidence, communicate under pressure, and know when another specialist needs to become involved.

Those are precisely the things I do not want to waste by repeatedly asking experienced people to rediscover known information.

The goal was never to replace support

When people talk about AI in customer support, the conversation often jumps immediately to automation or headcount reduction.

That was never the problem that interested me.

I did not look at an experienced support engineer and think, "An AI should replace this person."

I looked at what that engineer was spending time doing and thought:

Why are we making this person reconstruct knowledge the organization already has?

If an issue has been diagnosed before, a useful system should help the next analyst find that prior knowledge quickly.

It should preserve the evidence behind it.

It should understand enough context to distinguish a superficially similar problem from one where the prior answer really applies.

And when the evidence is weak or the situation is materially different, it should make that uncertainty visible rather than inventing an answer.

That moves the human toward the work where human judgment has the highest value.

The useful division is closer to:

Use the system to help recover what is already known.
Use the human to reason about what is actually new.

Why traditional search was not enough

Enterprise knowledge systems existed long before modern LLMs.

I worked around them.

Search indexes, knowledge bases, classification systems, documentation repositories, ticketing systems, and agent-facing knowledge tools were already part of enterprise support environments.

They helped.

But the deeper problem was often contextual.

The analyst did not merely need a document containing the right words.

They needed to know:

That is considerably harder than putting a search box in front of a document repository.

Modern retrieval and language models create better tools for attacking that problem, but they do not eliminate it.

They make some parts easier and introduce new failure modes of their own.

RAG was interesting because it attacked the problem I actually cared about

Retrieval-augmented generation caught my attention because it creates a useful separation.

Instead of asking a model to know everything, retrieve relevant evidence at the time of the request and use that evidence to construct the answer.

Conceptually, that is much closer to what a strong support engineer already does:

find the relevant information, understand the context, determine whether it applies, and explain the result.

But a production retrieval system still has to answer difficult questions.

What makes a source relevant?

What if lexical search and semantic similarity disagree?

What if the most semantically similar document belongs to a different customer, jurisdiction, role, or product version?

What if the retrieved evidence is weak?

What if the model produces a convincing answer that the evidence does not actually support?

What if the right behavior is to refuse or escalate?

Those questions are why my work through Keystone Applied Intelligence became much more than putting a chatbot in front of documents.

Building Keystone turned the original frustration into an engineering problem

Since late 2024, I have been working on different parts of this problem directly.

That includes hybrid retrieval combining lexical and semantic signals, authorization constraints before model-context construction, deterministic reranking, evidence thresholds, citations, refusal behavior, and structured evaluation.

A restricted SOP and checklist retrieval system was deployed for a pilot with a volunteer fire department. A small number of members tested the system, and their feedback informed subsequent workflow refinements.

That implementation does not prove that I have solved enterprise knowledge operationalization generally.

It gives me something more useful: a system where specific mechanisms can be implemented, exercised, evaluated, broken, and improved.

One of the most important lessons has been that retrieval quality alone is not enough.

A system can retrieve a highly relevant passage and still be wrong because the caller should not have received it.

It can enforce authorization correctly and still produce a poorly supported answer.

It can generate a convincing response from evidence that is technically related but contextually wrong.

And it can pass a demo while failing exactly the kinds of edge cases that matter in production.

That led me deeper into evaluation as well.

The point is not to prove that an AI system is universally correct.

It is to make failure visible enough that the implementation can improve.

The support engineer still matters

There is a tendency to describe AI systems as though the end state is maximum autonomy.

I do not think that is the right objective for many operational environments.

A strong system should know where automation is useful and where it should stop.

Sometimes the right result is an answer.

Sometimes it is a suggested diagnostic path.

Sometimes it is a summary that helps the next engineer understand what has already been tried.

Sometimes it is an escalation with the relevant evidence already attached.

And sometimes the correct output is simply:

I do not have enough evidence to answer this reliably.

That is not a failure of the product.

In many environments, that is the product behaving correctly.

The problem has expanded, but the motivation has not changed

My current work has expanded beyond retrieval into conversational AI, AI-assisted workflows, evaluation, APIs and integrations, observability, and, as a secondary research area, bounded runtime controls for consequential actions.

The questions are becoming more complex.

Once an AI system starts using tools or taking actions, the problem is no longer only whether it retrieved the right information.

Now we also have to ask what it is allowed to do, whether the conditions that justified an earlier decision are still current, what evidence should survive the action, and when a human should intervene.

But the motivation is still recognizable.

I am interested in systems that help people use organizational knowledge and capability more effectively without hiding uncertainty or removing the judgment that complex work still requires.

The original support problem remains a good test:

Are we making experienced people repeatedly solve things the organization already knows, or are we giving them reliable access to that knowledge so they can concentrate on what is genuinely new?

That is one of the reasons I moved from supporting enterprise systems toward building AI systems.

And it is still one of the problems I want to keep working on.

Related engineering notes