Finextra opinion highlights token costs and limits in AI systems

Stanley Epstein, an associate at Citadel Advantage Group, has set out how tokens underpin large language models, arguing in a Finextra community opinion that they affect AI processing costs, memory limits and multilingual performance. The post frames tokenisation as a core operating layer for systems increasingly used in business, financial services and software tasks.

Finextra labels the contribution as external author content that has not been edited by the publication and says it reflects the author’s views. Epstein’s central point is that AI systems do not read text as humans do. Instead, he writes, they break language into smaller units before converting those units into mathematical forms used by the model.

How tokenisation works

According to Epstein, a token can be a word, part of a word, punctuation, a number or a single character, depending on the model. He uses the example of a longer word being divided into components, allowing a model to recognise related terms built from similar fragments.

The mechanism matters because a model does not need to store every possible word as a separate item. Epstein argues that reusable fragments allow AI systems to handle new technical terms, slang, brand names and domain-specific vocabulary by drawing on known subcomponents.

After tokenisation, each token receives a numerical value and is transformed into an embedding, a mathematical representation in which related concepts sit near each other in a multidimensional space, Epstein writes. From there, the model generates output by predicting the next likely token based on the preceding sequence.

That next-token process, in Epstein’s description, applies across varied uses: drafting documents, explaining technical subjects, writing code or producing prose. The same token-based architecture can also support multiple human languages, programming languages and mathematical notation.

Cost, memory and operational risk

Epstein argues that tokens also function as an economic unit for commercial AI use. Providers charge largely on the basis of token consumption, he writes, so longer prompts increase input usage and longer answers add output usage. Extended conversations consume more of a model’s available context window.

Context windows, according to the post, define what a model can keep in short-term working memory. Epstein says some modern systems can process well over 100,000 tokens, enough for lengthy reports, books, research papers or sustained conversations. Even so, he adds, those limits can be reached, after which older material may be lost unless summarised or supplied again.

The post also identifies limits and risks. Technical documents may contain more tokens than their word counts suggest, increasing processing costs. Different models may split the same text differently, making direct comparison harder. Epstein also says long conversations can lose important context, while malicious users may try to exploit tokenisation methods to evade safety controls. Biases in recurring token patterns may also be reflected in generated output, according to his account.

Debate over future AI architectures

Epstein notes that researchers are examining alternatives, including character-level systems, byte-based models and approaches that use continuous representations rather than discrete tokens. He says these methods may offer advantages but carry computational trade-offs, with tokenisation remaining the current compromise between efficiency, scale and performance.

Rajeew Vishvakarma, a project manager at Infosys, commented on the post that token-based systems remain relevant but should be judged by more than efficiency. He said sectors including banking and fintech will need models that can reason across longer contexts, retain institutional memory, reduce hallucination risk and process domain-specific language more reliably.

Vishvakarma also linked token-level understanding to model risk management in regulated industries, citing explainability, security, cost-effectiveness and governance readiness as areas for further attention.

This story draws on original reporting from Finextra Research.