THESIS: PROMPTS AND TOKENS IN AI — A FULL 15‑PAGE EXPLANATION

PAGE 1 — INTRODUCTION

A prompt and a token are the two most fundamental building blocks of modern Artificial Intelligence systems, especially Large Language Models (LLMs). A prompt is the instruction you give to an AI. A token is the unit of text the AI uses internally to read, interpret, and generate language.

Understanding these two concepts is essential because every AI interaction — from simple questions to complex multi‑step workflows — is governed by how prompts are written and how tokens are counted. LLMs do not think in words or sentences; they think in tokens, and they respond based on the structure and clarity of your prompt.

Sources confirm that LLMs operate as next‑token prediction machines, meaning they generate responses one token at a time, based on statistical patterns learned during training .

PAGE 2 — WHAT A PROMPT IS

A prompt is any input you give an AI model:

• A question
• A command
• A description
• A piece of text
• A role instruction (“Act as a financial analyst…”)

It is the bridge between human intention and machine output.

A prompt determines:

• The quality of the answer
• The accuracy
• The tone
• The format
• The depth
• The cost (because longer prompts use more tokens)

A prompt is not just text — it is a design element that shapes how the model interprets your request.

As explained by PromptedGrad, a prompt is simply the instruction you type into an AI tool, and the quality of the output depends almost entirely on the quality of the prompt .

PAGE 3 — WHY PROMPTS MATTER

Prompts matter because LLMs do not “understand” in the human sense. They follow patterns.

A clear prompt:

• Reduces ambiguity
• Minimises hallucinations
• Improves accuracy
• Controls tone and structure
• Saves tokens (and therefore money)
• Produces predictable results

A vague prompt leads to vague answers.

Example: Bad: “Explain tax.” Good: “Explain VAT input tax credit in 150 words for a first‑year commerce student.”

PAGE 4 — TYPES OF PROMPTS

1. Zero‑Shot Prompt

You ask directly without examples. Useful for simple tasks.

2. Few‑Shot Prompt

You provide examples to guide the model. Useful for style‑specific outputs.

3. Instruction Prompt

Clear command: “Summarise this text in 5 bullet points.”

4. Structured Prompt

Tables, JSON, bullet points.

5. Negative Prompt

Tell the AI what not to include.

These categories help shape the model’s behaviour and reduce uncertainty.

PAGE 5 — COMPONENTS OF A GOOD PROMPT

A strong prompt contains:

• Task — What you want
• Context — Background information
• Constraints — Length, tone, format
• Examples — If needed
• Role — “You are an expert…”

These elements reduce randomness and increase precision.

PAGE 6 — WHAT A TOKEN IS

A token is the smallest unit of text an AI model processes. It is not exactly a word — it is usually 3–4 characters in English, or about ¾ of a word.

Examples:

• “ChatGPT” = 2 tokens
• “Hello, how are you?” ≈ 5–6 tokens

LLMs convert your text into token IDs and process them numerically. This is why tokenisation matters: even a small change can alter token boundaries.

PAGE 7 — WHY TOKENS MATTER

Tokens matter because they determine:

• Cost
• Speed
• Memory limits
• Context window size
• Model performance

Every AI API call is measured and billed in tokens. Input tokens and output tokens are billed separately, and output tokens often cost 2–5× more than input tokens .

PAGE 8 — TOKEN COUNTING

Token counting includes:

• System prompt
• User messages
• AI responses
• Conversation history
• Attached files
• Images (converted to tokens internally)

This means even a long conversation can consume thousands of tokens.

Different languages tokenise differently — German and Japanese may consume 20–40% more tokens for the same text .

PAGE 9 — CONTEXT WINDOW

The context window is how much text an AI can “remember” at once.

It includes:

• All messages
• All responses
• All documents pasted in

Once the window is full, the model starts forgetting earlier parts. This is why long conversations sometimes lose context.

This behaviour is confirmed by PromptedGrad’s explanation of context windows .

PAGE 10 — HOW LLMs USE TOKENS INTERNALLY

LLMs operate as next‑token prediction systems. They generate responses one token at a time, based on:

• The prompt
• Previous tokens
• Model weights
• Sampling settings (temperature, top‑p, etc.)

This incremental generation explains why models sometimes change direction mid‑response or produce inconsistent paragraphs.

Knovo emphasises that LLMs do not think in full paragraphs — they build answers token by token under uncertainty .

PAGE 11 — PROMPT ENGINEERING

Prompt engineering is the practice of designing prompts to get reliable outputs.

It involves:

• Structuring inputs
• Reducing ambiguity
• Using constraints
• Providing examples
• Managing token usage
• Controlling model behaviour

Prompt engineering is essential for cost‑effective AI usage because poorly designed prompts waste tokens without improving output quality .

PAGE 12 — TOKEN ECONOMICS

Token economics refers to how tokens affect cost.

Key facts:

• Input tokens cost less
• Output tokens cost more
• Long outputs spike cost
• Redundant words waste tokens
• Choosing the right model reduces cost by 10–50×
• Local LLMs have zero per‑token cost but require hardware investment

Understanding token economics helps organisations budget AI usage effectively.

PAGE 13 — COMMON MISTAKES

1. Overly long prompts

Waste tokens without improving clarity.

2. Vague instructions

Lead to hallucinations.

3. Ignoring context window limits

Causes the model to forget earlier details.

4. Asking for unnecessary long outputs

Increases cost dramatically.

5. Using the wrong model

High‑end models cost more but are not always necessary.

PAGE 14 — BEST PRACTICES

For Prompts

• Be specific
• Use constraints
• Provide examples
• Define the role
• Use structured formats
• Avoid redundancy

For Tokens

• Keep prompts concise
• Limit output length
• Use summaries instead of full documents
• Choose smaller models when possible
• Track token usage
• Reset conversations when context window is full

PAGE 15 — CONCLUSION

Prompts and tokens form the foundation of modern AI interaction. A prompt is the instruction that guides the model, and a token is the unit of text the model uses to interpret and generate language.

Understanding these concepts allows you to:

• Improve accuracy
• Reduce cost
• Control output
• Prevent hallucinations
• Build reliable AI workflows
• Optimise enterprise‑level AI usage

LLMs are powerful, but they are not magic boxes. They are statistical next‑token prediction machines, and mastering prompts and tokens is the key to unlocking their full potential.