You write Java for a living. You have probably used a chatbot, and you may have wondered what happens inside it. This article answers that in terms you already know.

By the end, you will understand four ideas: tokens, the context window, sampling and embeddings. Everything else in this series builds on them. Each one gets a plain explanation, a Java comparison, and a warning about the mistake people make first.

You do not need any maths. You do not need Python. You will read one small Java program and look at three simple diagrams. If you have not read the map yet, start with Part 1, the roadmap. This article is the first step of Stage 1.

The short version

• A token is a small chunk of text. Models read and write tokens, not words.
• The context window is everything the model can see in one request. You send it again every time.
• Temperature sets how adventurous the next pick is. Low is predictable. High is varied.
• An embedding is a list of numbers for the meaning of a text. Close lists mean close meanings.
• A fluent answer is not always a true answer. Give the model facts, and check what it says.

What an LLM actually is

A large language model, or LLM, is a program trained on a huge amount of text. Its job is to predict the next small piece of text. That is the whole trick. Answering questions, summarising and writing code all come from doing that one thing over and over.

There are two phases. It helps to keep them apart.

Training is when the model learns. The maker feeds it a very large amount of text. The model practises predicting what comes next, and it adjusts itself each time it is wrong. A free online LLM course describes this as training on raw text to predict the next word.

Inference is when you use the finished model. You send text in. Text comes back. Nothing is learned during this step. The model does not remember your request later.

You will almost always use a model that someone else trained, just as you use a database engine you did not write. So this article is about using a model well.

Most LLMs you will meet use a design called the transformer, from the paper Attention Is All You Need. You do not need to read it.

Here is the honest part. The model predicts likely text. It does not look facts up. Ask for the capital of a country, and it writes the words that usually follow that question. That is usually right, because the right answer is common in its training text.

Now ask about something new, rare, or private to your company. The model has nothing to go on. It will still write something that sounds right. Unless you put the facts in the request, it is guessing from patterns. We come back to this in the section on why models make things up.

Tokens

The idea in plain words

A token is a small chunk of text. It is often a whole word. It can also be part of a word, a space or a punctuation mark. The model reads a list of tokens, and it writes a list of tokens.

One provider's documentation says tokens are commonly occurring sequences of characters. Its example splits a long word into two pieces, "token" and "ization".

Different models split text differently. The same sentence can give a different token count on a different model. Provider documentation says so directly: count your prompt again on the model you plan to use.

Both your prompt and the answer are counted in tokens. For English text, the same provider gives a rough guide: one token is about four characters, or three quarters of a word. Treat that as a loose estimate.

Figure 1 shows a model writing text one token at a time. The splits in it are made up to show the idea.

Figure 1: One token at a time. Simplified: real models split text in their own way, and the bars are only for show.

A Java parallel

Think about what happens when javac reads your source file. It does not work on letters. A lexer first turns the text into tokens: keywords, names, numbers and symbols. The Java Language Specification describes this step in its chapter on lexical structure. The parser then works on those tokens.

An LLM does something similar at the front door. A tokenizer cuts your text into tokens before the model sees anything.

There is one big difference. A Java lexer follows fixed rules from the language spec, so every compiler agrees. An LLM tokenizer is not tied to grammar. Its pieces are common chunks of text, and each model brings its own.

What it means for your code

Count tokens before you send a request. Cost, speed and limits are all measured in tokens. They are not measured in characters or words.

Many providers let you count tokens before a call, and their responses report how many tokens were used. Log both for every call. When a bill is higher than you expected, those numbers come first. You will meet this in Part 3, when your Spring Boot service sends its first request.

A common trap

The trap is assuming that one token equals one word. It does not. If you budget by word count, your numbers will be wrong, and you will not see it until a request fails or an invoice arrives.

The context window

The idea in plain words

The context window is the amount of text, counted in tokens, that a model can consider in one request. Think of it as the model's working memory for one call. A provider's guide describes it as all the text the model can reference when it writes a response, including the response itself.

Several things go inside: the system instructions (rules you give the model about how to behave), the conversation so far, any documents you paste in, and the new question.

The answer goes in the window too. In most APIs the prompt and the answer share the same space, so a prompt that fills nearly all of it leaves little room for the answer.

More is not always better. The same guide warns that accuracy and recall can get worse as the token count grows. So choose what you include. Do not stuff everything in.

Figure 2: What fits in one request. The proportions are illustrative.

A Java parallel

Think of a stateless HTTP request. Your Spring Boot controller does not remember the last caller. If it needs to know something, the client sends it again in the request body.

Most chat APIs work the same way. One provider's documentation puts it plainly: the API is stateless, so you always send the full conversation history. The list of messages is your request body. A "conversation" exists only because your code keeps sending it again.

What it means for your code

You own the conversation state. A chat feature is a simple loop. Keep the list of messages in your own code or database. On each turn, add the user's message, send the whole list, and add the answer to the list.

The list grows with every turn, so sooner or later you have to trim it. There are two common ways:

• Drop the oldest messages and keep the recent ones.
• Ask the model to write a short summary, and keep the summary in place of the old messages.

You will meet conversation state in Part 3.

A common trap

The trap is expecting the model to remember earlier calls. It does not. If your second request leaves out the first exchange, the model has never heard of it.

It is not forgetting. It never had that information, like an HTTP request that arrives without a session cookie.

Sampling and temperature

The idea in plain words

When the model predicts the next token, it does not produce just one answer. It scores many possible next tokens. Then it picks one. The API reference of one provider describes this as options for each next token, ranked by probability.

The pick is added to the text, and the process repeats. That is how a full answer appears, one token at a time. Figure 1 shows the loop.

The choice of pick is called sampling. Temperature is a setting that changes how adventurous the pick is. At a low temperature the model sticks to the likeliest tokens, so the output is more predictable. At a high temperature it picks less likely tokens more often, so the output is more varied.

The same API reference suggests a value near zero for analytical tasks and a value near the top of its range for creative ones. The range and the default differ from one provider to the next, so read the docs.

There is one more catch. Some providers limit which sampling settings you can change. The same reference says its newer models do not let you change temperature. Check the docs before you build on it.

A Java parallel

Picture a list of options, each with a weight. You pick one at random, in proportion to the weights. The heavy items win most of the time. The light items still win now and then.

Temperature is a knob on that list. Turn it down, and the top item wins nearly every time. Turn it up, and the other items get more chances.

What it means for your code

Use low values for extraction, classification and code, where you want the same kind of answer every time. Use higher values when variety helps, such as brainstorming names.

Never expect byte-identical output. Even at zero, the same reference says results are not fully deterministic. So do not write a test that compares an answer to an exact string. Check the shape and the facts instead.

You will meet this again in Part 4, which covers structured output.

A common trap

The trap is using a high temperature for structured output. Ask for JSON at a high temperature, and more varied picks mean more chances for a wrong field name or a missing brace. Your Jackson parser will tell you.

Embeddings

The idea in plain words

An embedding turns text into a list of numbers. Similar meanings get similar numbers. One provider's guide puts it simply: an embedding is a list of floating point numbers, and the distance between two lists measures how related they are.

Take "How do I reset my password?" and "I forgot my login". They share almost no words, but they mean nearly the same thing. Their embeddings land close together. The embedding for "banana" lands far away from both.

To compare two embeddings, you use cosine similarity. It measures the angle between two lists of numbers. The score runs from -1 to 1, and higher means the lists point the same way. Read it as "higher score, closer meaning".

Real embeddings are long. Provider documentation lists sizes from a few hundred to a few thousand numbers. Figure 3 shrinks them to two numbers so you can see the idea on a flat page.

You get embeddings from an embedding model, which is not the chat model.

Figure 3: Meaning as a map. Simplified to 2 dimensions. Real embeddings have hundreds or thousands.

A Java parallel

In Java, an embedding is a float[]. That is all it is. Comparing two embeddings is a method that takes two arrays and returns a double.

Here is the whole program. It is one file with no dependencies.

Cosine.java

public class Cosine {

    static double cosine(float[] a, float[] b) {
        double dot = 0, normA = 0, normB = 0;
        for (int i = 0; i < a.length; i++) {
            dot += a[i] * b[i];
            normA += a[i] * a[i];
            normB += b[i] * b[i];
        }
        return dot / (Math.sqrt(normA) * Math.sqrt(normB));
    }

    public static void main(String[] args) {
        // Made-up numbers for illustration. These are not real embeddings.
        float[] springBoot      = {0.90f, 0.80f, 0.10f, 0.05f};
        float[] springFramework = {0.85f, 0.75f, 0.20f, 0.10f};
        float[] banana          = {0.05f, 0.10f, 0.90f, 0.85f};

        System.out.printf("Spring Boot vs Spring Framework: %.3f%n", cosine(springBoot, springFramework));
        System.out.printf("Spring Boot vs banana:           %.3f%n", cosine(springBoot, banana));
        System.out.printf("Spring Framework vs banana:      %.3f%n", cosine(springFramework, banana));
    }
}

The numbers are made up for illustration. They are not real embeddings. Each list has four numbers, and a real one has hundreds or thousands.

I compiled and ran this file with OpenJDK 21.0.11 (Temurin), using javac Cosine.java and then java Cosine. This is the real output:

Spring Boot vs Spring Framework: 0.995
Spring Boot vs banana:           0.171
Spring Framework vs banana:      0.266

Spring Boot and Spring Framework score close to 1. Banana scores low against both. The pattern is the point: related things score high.

What it means for your code

This is how search by meaning works. You embed your documents once and store the lists. When a user asks a question, you embed the question and look for the stored lists that are closest to it.

That idea is the base of RAG, short for retrieval-augmented generation. You build it in Stage 2, in Part 5, with Spring AI and pgvector.

For now, only the idea matters. Skip vector databases and index tuning until then.

A common trap

The trap is mixing vectors from different embedding models. Each model builds its own map. Two models can even return lists of different lengths. One provider's guide shows its two models with different default sizes.

Even when the lengths match, treat vectors from different models as not comparable, unless your provider says they share a space. Store the model name next to every vector, and embed everything again if you switch.

Why models make things up, and what helps

A fluent answer is not the same as a true answer. The model is built to write likely text. Likely text reads well, correct or not.

People call a made-up answer a hallucination. A provider's guide says even the most advanced models can write text that is factually wrong or at odds with the context you gave.

It happens most when the model has little to go on, such as news, your internal systems or a niche library version. It writes made-up names and links in the same calm voice.

Here is what helps, in the order to try it.

1. Give the model the facts it needs in the prompt. Paste in the document or the policy, and tell the model to answer only from that text. The same guide suggests this, and also suggests letting the model say "I don't know".
2. Ask for sources. Ask the model to quote the line that supports each claim. If it cannot find a line, tell it to drop the claim. A quote does not prove the answer is right, but it gives you something to check.
3. Lower the temperature for factual tasks. Where you can set it, use a low value for questions with one right answer.
4. Check the output in code. Parse the JSON. Validate it against a schema. Confirm that an ID exists in your database. You are already good at this part.
5. Test with a fixed set of questions. Keep a list of questions with known answers. Run it after every change to a prompt or a model. One good answer proves nothing.

Warning

None of these steps removes the risk. The provider guide says its techniques reduce hallucinations but do not remove them, and it tells you to validate critical information. Never let an AI feature act on model output without a check.

Part 5 covers retrieval, which finds the right facts and puts them in the prompt for you. Part 8 turns that fixed list of questions into a real evaluation.

Cost, speed and limits

Tokens drive cost and speed. Providers bill by tokens. The prompt counts, and so does the answer, and many providers price the two at different rates.

Longer conversations cost more. The reason is the stateless design from earlier: on turn ten you send turns one to nine again. A longer answer also usually takes longer to arrive.

Models have limits too: the size of the context window, the length of one answer, and how many requests and tokens you can send per minute.

Prices and limits change, so this article gives none on purpose. Check your provider's pricing page for current prices, and the model's documentation for its limits, before you design around any number.

A cheat sheet for Java developers

Concept	Java parallel	What to remember
Tokens	A lexer splits code into tokens	Cost, speed and limits use tokens. A token is not a word.
Context window	A stateless HTTP request body	You send everything every time. The answer shares the space.
Sampling and temperature	Picking from a weighted list, with a randomness knob	Low for extraction and code. Never expect identical output.
Embeddings	A float[] compared with cosine similarity	Close lists mean close meanings. Never mix models.
Made-up answers	Unchecked user input	A fluent answer is not a checked answer. Validate it.

Five mistakes to avoid

Treating the model like a database. A database returns what you stored. A model writes likely text. If an answer must be exact, read it from a real database.

Sending the whole history every time without trimming. Costs climb, and one day the request is too long. Decide early how you will trim.

Using a high temperature for data extraction. Keep the setting low, and validate what comes back.

Judging quality by one lucky answer. Use a fixed set of questions and look at the results as a group.

Mixing embedding models. Vectors from different models are not comparable. Store the model name with every vector.

Check yourself

1. Your prompt is 800 words long. Is it 800 tokens?

Not necessarily. A token is often part of a word, and the count changes from model to model. Count with your provider's tool, using the model you plan to use.

2. Your chatbot forgets the user's name on the second request. Why?

Your code did not send the first exchange again. The API is stateless, so the model only knows what is in the current request.

3. You pull invoice numbers out of emails into JSON. High or low temperature?

Low. You want the same kind of answer each time. Validate the JSON anyway.

4. What is the risk when a prompt nearly fills the context window?

The prompt and the answer share the window, so the answer may be cut short. If the prompt alone is too long, the request can be refused.

5. Why not compare vectors from two different embedding models?

Each model builds its own map of meaning, and the lists can even differ in length. Treat them as not comparable, and embed everything again if you switch.

What comes next

Next is Part 3, "Call your first LLM from Spring Boot". It is not published yet.

Two of the ten parts are published. The order of the rest may change.

1. Java Developer to AI Engineer: The Complete Roadmap (published)
2. How LLMs work, for Java developers (this article)
3. Call your first LLM from Spring Boot (Stage 1 to 2)
4. Structured output and streaming (Stage 2)
5. RAG with Spring AI and pgvector (Stage 2)
6. Agents: the plan, act, observe loop (Stage 3)
7. MCP servers and clients in Java (Stage 3)
8. Evaluating and tracing LLM apps (Stage 4)
9. Guardrails, approvals and cost control (Stage 4)
10. Your portfolio and interviews (finish)

To follow along, subscribe to the blog feed in your RSS reader. You can also find us as @agentworkshq on X and Instagram.