Understanding AI tokens with The Little Prince

When talking about artificial intelligence and language models like GPT-4, it's essential to understand what tokens are. To explain it in a simple way, let's take a well-known example from the book The Little Prince by Antoine de Saint-Exupéry.

A literary analogy: The Little Prince

Let's say you're reading this famous passage:

“Please... tame me! he said. “I want to,” replied the little prince, “but I don't have much time. I have friends to discover and lots of things to know.”

When we read this text, our brain processes every word, every space, every punctuation mark. For an AI model like GPT-4, it's exactly the same, except that it cuts the text into Tokens.

What is a token?

One Token is a unit of text that can be:

A whole word (for example: “Hello” = 1 token).
A part of a word (for example: “tame” can be divided into “learn” + “to hear” = 2 tokens).
Special characters, punctuations, or spaces (for example: “...” or “!” = 1 token each).

How does that work?

Each time you interact with an AI model, it breaks down your request into tokens.

Entry: What you ask him, whether it is a question or a specific instruction.
Output: What the model generates as a response.

These two elements consume tokens. For example, if you ask a 100 token question and the AI generates a 200 token response, the total request is 300 tokens.

Why is it important?

AI models like GPT-4 have a token limit per request. For example, GPT-4 can process up to 8192 tokens per interaction. The longer your message is, the fewer tokens are left available for a detailed response.

In API, this concept is particularly important because each token consumed has a cost. Optimizing your messages therefore allows you to reduce your costs while obtaining more relevant answers.

How to use the tokens properly?

Be clear and to the point. This allows the model to have more room to generate a relevant response.
Segment your queries. If your request is too long, break it up into several shorter requests.
Test your messages. You can check how your texts are divided into tokens using the tool OpenAI Tokenizer.

Conclusion

Understanding what a token is is essential to effectively interact with an artificial intelligence model. Whether it's to structure your requests or reduce your API costs, this concept helps you better control your exchanges with AI.