What Is LLM Actually Doing? A Fellow Engineer's Take on Vectors, Next-Token Prediction, and Fail-back Routing

Why treating an LLM as a probability engine, not a brain, changes how you architect around it. The reasoning behind NEXT4I's AI layer.

#LLM #BuildinPublic #SystemArchitecture #AI #Model AI #AI Router #AI Stable

I used to wonder why the model confidently gives you a wrong number. It's not a bug in the traditional sense, it's the model doing exactly what it's built to do: predicting the next token from probability, with zero actual arithmetic happening underneath.

TLDR; An LLM (large language model) converts text into vectors (numeric coordinates in a high-dimensional meaning-space) and generates output via next-token prediction, sampled with parameters like top-k and temperature. Because it's fundamentally a probability engine and not a calculator or a database, I designed NEXT4I with automatic model fail-back routing and task-based model selection instead of trusting any single model as a source of truth.

The Mental Model: Vectors, Not Meaning

Every token gets embedded into a vector, often with hundreds or thousands of dimensions. Semantically similar tokens end up close together in that space. That's why semantic search (vector-based retrieval) can match "large flying animal consumes insects" to "big bird eats worms" even with zero shared keywords, unlike old-school lexical search (TF-IDF/BM25) which needs literal term overlap.

GPUs handle this well because they're already wired for massive parallel floating-point math (the same math used to shade millions of pixels per frame), so throwing billions of similarly-directed vectors at a GPU is a natural fit, not a coincidence.

Generation Is Sampling, Not Retrieval

Given a prompt, the model doesn't look up an answer, it samples one token at a time from a probability distribution. Two knobs matter in practice:

top_k: 2        # only sample from the top-2 most likely next tokens
temperature: 0.2  # low = deterministic/precise, high = creative/varied

Low temperature + low top_k gives you consistent, "boring" output, good for structured extraction. High temperature gives you variety, good for brainstorming, bad for anything requiring precision.

Why LLMs Hallucinate on Arithmetic

There's no calculator inside the model. It doesn't evaluate x * y, it predicts digits that are statistically plausible given the prompt, one token at a time. It gets 2 * 2 right because that pattern is everywhere in training data. It confidently botches large multiplication because it's still just sampling digits, not computing. This is exactly why production systems now delegate real math to a tool call (a Python sandbox, a calculator function) instead of trusting raw model output.

Core Value: A Generic Model-Tier Fail-back Router

Here's the simple pattern, stripped of any specific business logic, a reusable fail-back wrapper for any set of same-tier model clients:

package modelrouter

import (
	"context"
	"errors"
	"fmt"
)

// ModelClient is any backend that can answer a prompt.
type ModelClient interface {
	Name() string
	Complete(ctx context.Context, prompt string) (string, error)
}

// TieredRouter tries each client in a tier in order until one succeeds.
type TieredRouter struct {
	tier []ModelClient
}

func NewTieredRouter(clients ...ModelClient) *TieredRouter {
	return &TieredRouter{tier: clients}
}

// Complete attempts each model in the tier, fail-back on error.
func (r *TieredRouter) Complete(ctx context.Context, prompt string) (string, error) {
	var errs []error
	for _, client := range r.tier {
		resp, err := client.Complete(ctx, prompt)
		if err == nil {
			return resp, nil
		}
		errs = append(errs, fmt.Errorf("%s: %w", client.Name(), err))
	}
	return "", errors.Join(errs...)
}

This is intentionally boring: try the next model in the same tier on failure, return the first success. No retries with backoff yet, no circuit breaker, just the core fail-back idea. In NEXT4I's actual implementation, tiers are populated dynamically and health state feeds back into ordering, but that logic is abstracted here on purpose, the generic version above is what's actually useful to share.

Core Value: Task-Based Routing, the Simple Version

A minimal router that inspects task complexity before picking a tier:

package modelrouter

type Complexity int

const (
	Simple Complexity = iota
	Complex
)

func ClassifyAndRoute(task string, simpleTier, complexTier *TieredRouter) *TieredRouter {
	if estimateComplexity(task) == Simple {
		return simpleTier
	}
	return complexTier
}

func estimateComplexity(task string) Complexity {
	if len(task) < 100 {
		return Simple
	}
	return Complex
}

estimateComplexity can be as crude as a length/keyword heuristic or as sophisticated as a small classifier model, the point is the routing decision happens before the expensive call, not after.

Trade-offs I Made

• Fail-back within a tier, not across tiers. Swapping a cheap model in for an expensive one silently would change output quality without anyone noticing. Tiers exist specifically to avoid that.
• No cross-request state in the router. Keeps it stateless and trivially horizontally scalable, at the cost of not learning from past failures within a single request lifecycle.
• Routing is a toggle, not a mandate. Users can pin a specific model when they need deterministic behavior from one exact provider, the router only kicks in by default.

If there's one thing worth taking away: treat the model as a probability engine you can't fully trust, and let the system around it, fail-back, routing, tool calls for math, carry the reliability burden instead.

Thanks for reading all the way to the end, I'll keep working on more articles like this.