Tokenization : for freshers

Think tokenization as chopping vegetables to cook them as without cutting or chopping, how we’ll proceed with cooking.

It means converting big tasks into small ones for the ease of use.

Definition could be converting sentences into small words or char known as “Tokens”

Input: Hello World! How are you?

Tokens: “How”, “ “, “World”, “!”, “How“, “are“, “you“, “?”

Then these tokens are converted into numbers understandable by computers.