Building a Low-Latency RAG Pipeline with Groq: From Ingestion to Grounded Answers

What low-latency RAG with Groq actually means

RAG with Groq usually combines two systems. A retrieval layer indexes and searches private or domain-specific knowledge. Groq runs the language model that interprets the selected evidence and generates the response. Frameworks such as LlamaIndex, LangChain, and Mastra can orchestrate this pattern, but the same architecture can be implemented directly.

Groq's role is broader than raw token generation. Its Chat Completions API supports streaming, service tiers, structured outputs, supplied documents, and citations for supported document or web-search workflows. Retrieval, source authorization, freshness, and indexing still need to remain explicit application concerns.

The objective is not the smallest benchmark number in isolation. The useful target is predictable end-to-end latency at realistic concurrency, with enough evidence to answer correctly and no more context than the model needs.

Model the complete RAG latency budget

A useful approximation is: end-to-end latency equals retrieval time plus reranking time plus prompt preparation plus network time plus model time. Groq further describes model latency as time to first token (TTFT) plus decoding time and network round trip. TTFT includes queueing and prompt prefill, while decoding grows with the number of output tokens.

Measure every boundary independently. A fast Groq server time can be hidden by a slow vector database, cross-region traffic, synchronous query rewriting, or a reranker that processes too many candidates. Conversely, fast retrieval cannot compensate for an oversized prompt and an unnecessarily long answer.

Track percentiles, not only averages. Median latency describes the common case; p95 and p99 expose queueing, cold connections, overloaded dependencies, and the requests most likely to feel broken.

• Query preparation: normalization, routing, rewriting, or decomposition
• Retrieval: vector search, lexical search, metadata filtering, and result merging
• Reranking: query-document scoring and duplicate removal
• Prompt construction: formatting evidence, citations, instructions, and conversation state
• Network time: application-to-retriever and application-to-Groq round trips
• Groq server time: queueing, prompt prefill, and token decoding
• Client rendering: streaming, markdown rendering, and citation UI

The latency metrics that matter

Groq exposes server-side timing and token-usage information, while your application should measure the full client-observed request. Comparing those numbers helps separate model time from network and orchestration overhead.

Keep ingestion and indexing off the query path

Document parsing, OCR, chunking, embedding, and index writes are generally offline operations. They affect how quickly new knowledge becomes searchable, but they should not run inside every user request. Treat ingestion latency as a freshness service-level objective and query latency as a separate serving objective.

Use incremental indexing. Reprocess only changed pages or sections, keep stable source identifiers, and remove deleted content from every index. Precompute embeddings, normalized metadata, lexical fields, and citation locators before traffic reaches the system.

For sources that must become available immediately, expose indexing state in the product. A file should move through accepted, processing, indexed, and searchable states instead of silently appearing in retrieval before its content and permissions are ready.

Make first-stage retrieval fast without sacrificing recall

The first retrieval stage should be inexpensive and recall-oriented. Vector search captures semantic similarity, while lexical search remains valuable for exact product names, error codes, versions, legal citations, and other rare tokens. Hybrid retrieval often gives a better candidate set than either method alone.

Run independent retrieval methods in parallel when possible. If vector and lexical search each take 40 milliseconds, parallel execution can keep the critical path closer to the slower branch instead of paying their sum. Merge results with reciprocal-rank fusion or another deterministic strategy before reranking.

Apply tenant, permission, language, product, status, and effective-date filters during retrieval. Filtering after generation is too late. Early filters reduce the search space, improve precision, and prevent unauthorized content from entering the prompt.

Chunk for retrieval quality, not a universal token target

There is no single correct chunk size. The retrieval unit should preserve the smallest self-contained piece of evidence that can answer a question. Support documentation often works well at the section level, FAQs as complete question-answer pairs, API references at the endpoint level, and tables with their headers and surrounding explanation.

Chunks that are too large increase index ambiguity, prompt tokens, and TTFT. Chunks that are too small separate claims from qualifiers and force the model to reconstruct meaning from fragments. Use parent-child retrieval or adjacent chunk expansion when a small matching unit needs broader context.

Avoid excessive overlap. Repeated passages can dominate the candidate list, waste reranker capacity, and send duplicate evidence to Groq. Deduplicate by source location or semantic similarity before prompt construction.

Use reranking selectively

Reranking improves precision by evaluating the query and candidate passage together, but it adds another network or model call. Do not rerank the entire corpus. Retrieve a bounded candidate set, rerank only those candidates, and send a smaller final evidence set to Groq.

Conditional reranking can reduce tail latency. Skip it when the top result has a strong score margin, the query contains an exact identifier, or a metadata filter leaves only a few candidates. Enable it for ambiguous, multi-intent, or high-risk questions.

Reranking should optimize downstream usefulness, not similarity alone. A passage can be topically related yet fail to contain the policy, number, exception, or procedure needed to answer.

Choose a Groq model by measured quality and latency

Groq's supported-models page currently lists production options including `openai/gpt-oss-20b`, `openai/gpt-oss-120b`, `llama-3.1-8b-instant`, and `llama-3.3-70b-versatile`. The smaller or faster model is not automatically the best RAG model, and the largest model is not automatically necessary.

Start with the fastest production model that meets your grounded-answer evaluation. Escalate difficult queries to a stronger model based on query complexity, retrieval confidence, policy risk, or prior failure patterns. This routing strategy can keep the common path fast without forcing every question through the most expensive reasoning configuration.

Model availability and performance change. Query Groq's models endpoint and pin tested model IDs in production rather than relying on a hard-coded list copied from an old tutorial.

Stream a grounded RAG answer with Groq

Streaming improves perceived latency by rendering tokens as soon as they arrive. It does not eliminate retrieval time or reduce the amount of decoding required, so begin the stream only after the evidence set is authorized and stable.

The example uses explicit source IDs because the pattern works with any retrieval backend. Groq's API also supports supplied documents and citation options for compatible workflows. Whichever citation mechanism you choose, preserve stable source metadata and make citations inspectable in the UI.

import os

from groq import Groq

client = Groq(api_key=os.environ["GROQ_API_KEY"])

retrieved_chunks = [

    {

        "source_id": "S1",

        "title": "Returns policy",

        "url": "https://example.com/returns",

        "text": "Unused products may be returned within 30 days of delivery."

    },

    {

        "source_id": "S2",

        "title": "Refund processing",

        "url": "https://example.com/refunds",

        "text": "Approved refunds are sent to the original payment method."

    }

]

evidence = "\n\n".join(

    f"[{chunk['source_id']}] {chunk['title']}\n"

    f"URL: {chunk['url']}\n"

    f"{chunk['text']}"

    for chunk in retrieved_chunks

)

system_prompt = """You answer only from the supplied evidence.

Cite factual claims with source IDs such as [S1].

If the evidence is incomplete, say what cannot be verified.

Do not invent policies, dates, or exceptions."""

stream = client.chat.completions.create(

    model=os.getenv("GROQ_MODEL", "openai/gpt-oss-20b"),

    service_tier="on_demand",

    messages=[

        {"role": "system", "content": system_prompt},

        {

            "role": "user",

            "content": (

                f"Evidence:\n{evidence}\n\n"

                "Question: Can I return an unused product, and how is the refund paid?"

            )

        }

    ],

    temperature=0.1,

    max_completion_tokens=500,

    stream=True

)

for chunk in stream:

    token = chunk.choices[0].delta.content

    if token:

        print(token, end="", flush=True)

Reduce prompt size before optimizing anything exotic

Input token count is a primary driver of TTFT. Remove duplicate chunks, navigation text, repeated disclaimers, irrelevant conversation history, and metadata the model does not need. Keep URLs, titles, page numbers, and source IDs when they are required for verification.

A large context window is capacity, not a retrieval strategy. Research on long-context language models has shown that useful evidence can be harder to use when it is buried in the middle of a long prompt. Rank evidence, group it by source or claim, and place the most important material deliberately.

Constrain output length as well. Sequential token generation is a major part of total response time. Ask for a direct answer first, then let the user request expansion rather than generating an essay for every question.

Use Groq prompt caching when the prefix really is reusable

Groq prompt caching automatically reuses recent computation for exact matching prompt prefixes on supported models. Current documentation lists the GPT-OSS models as supported, describes a two-hour inactivity expiration, and reports a discount for successfully cached input tokens.

Structure stable system instructions, tool definitions, schemas, examples, and genuinely shared context before dynamic user data. A timestamp, user ID, or query near the beginning of the prompt breaks the reusable prefix.

Caching is not a substitute for retrieval. Most RAG evidence changes from question to question. It is most useful for stable policies, repeated document sets, long tool definitions, and shared prompt scaffolding. Monitor `cached_tokens` instead of assuming that a cache hit occurred.

from groq import Groq

client = Groq()

STATIC_PREFIX = """You are a grounded support assistant.

Use only the approved policy below.

Cite the policy as [POLICY].

If the policy does not answer the question, say so.

[POLICY]

Unused products may be returned within 30 days of delivery.

Approved refunds are sent to the original payment method.

"""

def ask(question: str):

    response = client.chat.completions.create(

        model="openai/gpt-oss-20b",

        messages=[

            {"role": "system", "content": STATIC_PREFIX},

            {"role": "user", "content": question}

        ],

        temperature=0.1,

        max_completion_tokens=300

    )

    details = getattr(response.usage, "prompt_tokens_details", None)

    cached_tokens = getattr(details, "cached_tokens", 0) if details else 0

    print(response.choices[0].message.content)

    print({

        "prompt_tokens": response.usage.prompt_tokens,

        "cached_tokens": cached_tokens,

        "server_total_seconds": getattr(response.usage, "total_time", None)

    })

ask("Can I return an unused item?")

ask("Where will an approved refund be sent?")

Pick the right Groq service tier for each workload

Do not route every workload through the same path. Live question answering, bulk evaluation, document enrichment, and nightly regression tests have different latency and reliability requirements.

Configure explicit timeouts, bounded retries, and exponential backoff. Retries can improve reliability but also multiply tail latency, so retry only transient failures and keep the user-facing deadline visible to the orchestrator.

• Use `on_demand` for ordinary real-time user traffic that needs predictable processing.
• Use the enterprise `performance` tier when low and consistent p99 TTFT is business-critical.
• Use `flex` for high-throughput work that can tolerate best-effort capacity errors and retries.
• Use `auto` when the application should select the best available eligible tier.
• Use Batch for offline evaluations, enrichment, and other asynchronous workloads—not interactive questions.

Prompt for grounding, citations, and abstention

A grounded prompt should define the evidence boundary. Tell the model to use only supplied sources, cite claims, distinguish source facts from inference, and state when the evidence does not answer the question.

Give every chunk a stable source ID and enough citation metadata to resolve back to the original file, page, section, or URL. Citation generation becomes fragile when the prompt contains anonymous text fragments.

Do not ask the model to hide retrieval failure behind polished prose. If the retriever returns weak, contradictory, stale, or incomplete evidence, the correct response may be an abstention, a clarifying question, or an escalation.

Use structured outputs for application logic

When a RAG response feeds a workflow rather than a chat bubble, return a schema with fields such as `answer`, `supported`, `source_ids`, `missing_information`, and `next_action`. Groq supports structured outputs, including strict schema adherence on supported models.

Validate the schema at the boundary and keep retrieved evidence outside the model-generated object. That separation makes the system easier to trace, cache, replay, and audit.

Structured generation can add processing constraints, so benchmark it against plain text for the exact model and schema you plan to use.

Instrument the whole RAG request

The example accepts application-specific retrieval and reranking functions, then records each stage separately. The key comparison is Groq server time versus application-observed generation time and total end-to-end time.

Add trace IDs across the retriever, reranker, Groq request, and citation renderer. Without a shared trace, teams often optimize the model because it is visible while missing a slower dependency elsewhere in the request.

from time import perf_counter

from typing import Callable, Sequence

from groq import Groq

client = Groq()

def timed_rag_query(

    question: str,

    retrieve: Callable[[str, int], Sequence[dict]],

    rerank: Callable[[str, Sequence[dict], int], Sequence[dict]],

) -> dict:

    started = perf_counter()

    candidates = retrieve(question, 24)

    retrieved_at = perf_counter()

    selected = rerank(question, candidates, 6)

    reranked_at = perf_counter()

    evidence = "\n\n".join(

        f"[{item['source_id']}] {item['text']}"

        for item in selected

    )

    response = client.chat.completions.create(

        model="openai/gpt-oss-20b",

        service_tier="on_demand",

        messages=[

            {

                "role": "system",

                "content": (

                    "Answer only from the evidence. Cite every factual claim "

                    "with its source ID. Abstain when support is missing."

                )

            },

            {

                "role": "user",

                "content": f"Evidence:\n{evidence}\n\nQuestion: {question}"

            }

        ],

        temperature=0.1,

        max_completion_tokens=500

    )

    completed_at = perf_counter()

    usage = response.usage

    return {

        "answer": response.choices[0].message.content,

        "retrieval_ms": round((retrieved_at - started) * 1000, 1),

        "rerank_ms": round((reranked_at - retrieved_at) * 1000, 1),

        "generation_wall_ms": round((completed_at - reranked_at) * 1000, 1),

        "end_to_end_ms": round((completed_at - started) * 1000, 1),

        "groq_server_seconds": getattr(usage, "total_time", None),

        "prompt_tokens": usage.prompt_tokens,

        "completion_tokens": usage.completion_tokens

    }

Evaluate speed and grounding together

RAGAS and RAGChecker both emphasize evaluating retrieval and generation as separate modules. That separation is essential for latency work: a faster answer is not an improvement if retrieval recall or citation support drops.

Build a representative question set with expected evidence, acceptable answers, adversarial wording, permission boundaries, stale-source cases, and unanswerable questions. Run the same set whenever you change chunking, embedding models, candidate counts, reranking, prompts, Groq models, or service tiers.

Load test with realistic concurrency and token distributions. Single-request benchmarks hide queueing, connection limits, retriever saturation, and p99 behavior.

• Retrieval recall at k and precision at k
• Mean reciprocal rank or normalized discounted cumulative gain
• Reranker lift over first-stage retrieval
• Answer correctness and completeness
• Faithfulness and claim-level citation support
• Abstention correctness when evidence is missing
• TTFT at p50, p95, and p99
• End-to-end latency at p50, p95, and p99
• Prompt and completion tokens per answer
• Error, timeout, retry, and rate-limit rates
• Cache-hit rate for eligible prompts

Plan for production failure modes

Define behavior for each failure before launch. The application may fall back to first-stage results, switch models, return a partial answer, retry within a deadline, or abstain. Silent degradation is the worst option because it makes quality failures look authoritative.

Keep Groq API keys on a trusted backend, validate user input and model output, and enforce authorization before context reaches the model. Retrieval is part of the security boundary, not only a relevance feature.

• A relevant result is retrieved but does not contain enough evidence to answer.
• Multiple sources disagree or use different effective dates.
• A permission change has not propagated to every index.
• The reranker or vector service times out.
• A Groq request receives a rate-limit or transient server error.
• Streaming begins, then the connection closes before the answer finishes.
• A source contains prompt-injection instructions or malicious markup.
• The answer cites a chunk that the user cannot open.

Build the retrieval layer with Calypso

Groq is a strong option for teams that want to engineer and control their own low-latency generation path. Calypso takes a managed approach to the knowledge side of RAG: Buckets organize multimodal sources, Agents define retrieval and citation behavior, and Integrations deliver grounded answers through websites, APIs, MCP clients, n8n workflows, and product interfaces.

Calypso is powered by Gemini File Search for managed multimodal retrieval over PDFs, documents, screenshots, charts, diagrams, help content, and images. Teams can use Calypso when they prefer a hosted, citation-ready answer layer instead of operating chunking, indexing, retrieval, and source delivery themselves.

The architectural decision is therefore broader than model speed. Decide which parts of ingestion, retrieval, grounding, evaluation, and delivery your team wants to own—and which should be provided as a managed layer.

RAG with Groq: frequently asked questions

• What is Groq used for in RAG? Groq runs the language-model inference stage and can stream grounded answers from retrieved context at high generation speed.
• Does Groq replace a vector database? No. A typical RAG system still needs a retrieval or managed knowledge layer that indexes and selects evidence.
• Does Groq support citations? Groq's Chat Completions API supports citations for information retrieved from supplied documents or supported web-search workflows.
• Which Groq model is best for RAG? There is no universal best model. Start with the fastest production model that passes your answer, citation, and abstention evaluations.
• Does streaming make RAG faster? Streaming lowers perceived latency by exposing the first tokens earlier, but it does not remove retrieval or decoding work.
• Can Groq prompt caching speed up RAG? Yes, when requests share a sufficiently long exact prefix on a supported model. Dynamic retrieved evidence often limits the reusable portion.
• Should every RAG query use a reranker? No. Conditional reranking can preserve precision on difficult questions without adding latency to obvious or tightly filtered searches.
• Can I use the OpenAI SDK with Groq? Yes. Groq provides an OpenAI-compatible base URL, although not every OpenAI feature or parameter is supported.