Skip to main content
ANVISoftware Solutions

AI Knowledge Assistant

Advanced35-50 hours

The flagship AI project: a full assistant with retrieval, tool use, conversation, guardrails, evaluation and cost tracking.

Technologies

The brief

What the finished thing needs to do.

  • A web interface for conversational questions
  • Retrieval over an internal document set
  • Tool calling so the assistant can look up live data from an API
  • Conversation context across turns, within a bounded window
  • Citations for every factual claim
  • Guardrails for out-of-scope and unsafe requests
  • Evaluation against a fixed question set that runs repeatedly
  • Token cost and latency tracked per conversation

How to structure it

Four separable parts: a web interface, an API that owns authentication and rate limiting, an orchestration layer that assembles prompts and decides whether to retrieve or call a tool, and the model itself. Keeping orchestration separate from the interface is what makes it testable and replaceable.

Every tool result and model response is treated as untrusted input. Validate tool arguments the model produces before executing anything, and validate structured output before acting on it. The model is not a trusted caller.

Prompt injection is a real concern the moment retrieved documents enter the prompt. A document containing instructions must not be able to redirect the assistant, which means clearly separating instructions from data and limiting what tools can do.

Evaluation must exist from early on. Without a fixed question set and recorded results, you cannot tell whether a prompt change improved things or simply changed them.

Structure of an AI-powered applicationA user interacts with a web interface, which calls an API. The API handles authentication and rate limiting, then passes the request to an orchestration layer. Orchestration assembles the prompt, decides whether to retrieve documents or call a tool, and sends the result to the language model. The model's response returns through the same path. Retrieval reads from a vector store; tools call internal APIs.UserWeb interfaceAPIauth, rate limitsOrchestrationprompt, routing, validationRetrieverVector storeToolsinternal APIsLanguage modelModel output and tool arguments aretreated as untrusted input throughout.

Why these technologies

Separate orchestration service
Prompt assembly and tool decisions are testable in isolation from the UI.
Validated tool calling
Model-generated arguments are untrusted input and must be checked before execution.
Retrieval with citations
Grounds answers in real content and makes them verifiable.
Fixed evaluation set
The only way to know whether a change helped rather than just differed.
Per-conversation cost tracking
Token spend is a real operational cost and must be visible, not discovered on an invoice.

Build it in this order

Each stage produces something that works. That matters — a project that only runs at the very end is a project people abandon.

  1. Single-turn question and answer

    No retrieval, no tools. Get the request path and error handling working.

  2. Add retrieval

    Ground answers in documents with citations.

  3. Add conversation context

    Multiple turns within a bounded context window; decide what to drop when it fills.

  4. Add one tool

    A single read-only lookup. Validate the arguments the model supplies before using them.

  5. Add tool selection

    Let the assistant choose between retrieving, calling a tool, or answering directly.

  6. Build the evaluation set

    Twenty to thirty questions with expected characteristics. Record a baseline.

  7. Add guardrails

    Refuse out-of-scope requests; ensure retrieved content cannot issue instructions.

  8. Add cost and latency tracking

    Per request and per conversation, visible in logs and to operators.

  9. Harden and deploy

    Authentication, rate limiting, containerisation and monitoring.

Done means

How to know it is finished

Check each of these before moving on. If one fails, the project is not done yet — and that is useful information rather than a setback.

  • Factual answers cite retrieved sources; the assistant declines when it lacks grounding
  • Tool arguments produced by the model are validated before execution
  • Instructions embedded in retrieved documents do not alter the assistant's behaviour
  • Conversation works across turns without exceeding the context window
  • The evaluation set runs on demand and results are comparable over time
  • Token cost and latency are recorded per conversation
  • Out-of-scope and unsafe requests are declined clearly

If you want to go further

Extensions worth attempting

Only once the core build meets every criterion above.

  • Add streaming responses for perceived speed
  • Add feedback collection and use it to extend the evaluation set
  • Add a second model provider and compare on the same evaluation set
  • Add per-user usage limits
  • Add an admin view of low-confidence and refused answers

Other projects at this level

Advanced

Document Management Platform

An architecture exercise: file storage, versioning, permissions and background processing, structured to stay maintainable.

40-60 hours
Advanced

Analytics Dashboard

A data-heavy interface that stays fast and accessible: filters, charts, large tables and honest loading states.

25-35 hours

Have a project worth talking through?

Tell us what you're building or what's slowing your current system down. We'll give you a direct read on scope and approach.

Start a ConversationExplore Our Services