Tokens are essentially fragments of words. When you send a request to the API, it breaks down the input into tokens. These fragments can include parts of words, trailing spaces, and even sub-words. Here are some general guidelines to help you understand token lengths: 1 token ≈ 4 characters in English 1 token ≈ ¾ of a word 100 tokens ≈ 75 words For practical reference: 1-2 sentences ≈ 30 tokens 1 paragraph ≈ 100 tokens 1,500 words ≈ 2,048 tokens To put this in context, Wayne Gretzky’s famous quote "You miss 100% of the shots you don't take" is 11 tokens. The OpenAI charter is 476 tokens, and the US Declaration of Independence transcript is 1,695 tokens. Tokenization can vary by language. For instance, the Spanish phrase "¿Cómo estás?" (meaning "How are you?") has 5 tokens (for 10 characters), which indicates that non-English languages might have a higher token-to-character ratio, potentially increasing API usage costs. To explore tokenization further, you can use our interactive Tokenizer tool, which lets you calculate token counts and see how text is broken down. Remember, the tokenization process can differ between models. For example, GPT-3.5 and GPT-4 use a different tokenizer than earlier models, resulting in varied tokens for the same input. If you prefer a programmatic approach to tokenizing text, you can use Tiktoken, a fast BPE tokenizer designed for OpenAI models.
Different models have varying token limits for requests, with some handling up to 128,000 tokens shared between the prompt and completion. Models like GPT-4 Turbo have specific limits for input and output tokens. To stay within these limits, consider strategies like condensing your prompts or breaking text into smaller segments.
The API offers different models at various price points, with each model pricing requests differently. For detailed information on token pricing, you can visit our pricing page.
The API processes words based on their context within the corpus data. It converts the prompt into tokens, processes them, and then converts the predicted tokens back into the words we see in the response.
Words that seem identical to us may be tokenized differently based on their structure in the text. For example, the word 'red' can generate different tokens depending on its context:
' red' (lowercase with leading space) could be token “2266”
' Red' (uppercase with leading space) might be token “2296”
'Red' (uppercase at the beginning of a sentence) could be token “7738”
The token for a period remains the same ("13") in all contexts because its usage is consistent throughout the corpus data.
By understanding tokens and their context, you can better optimize your API usage and costs, ensuring efficient and effective communication with OpenAI models.