tokenizer

In natural language processing, this term refers to the critical first stage of a data pipeline. It transforms raw strings into a structured format that a machine learning model can interpret. The choice of tokenizer—whether word-based, character-based, or subword-based (like Byte Pair Encoding)—significantly impacts the model's ability to handle unknown words and vocabulary size.

In the context of cybersecurity and fintech, the term describes a security mechanism. Here, it functions as a proxy system that swaps sensitive data for a surrogate value. This ensures that if a database is breached, the stolen tokens are useless to an attacker because they lack the original sensitive mapping stored in a secure vault.