What is a token and a word type in NLP ?

    1. Token is a technical term given to a sequence of characters. Note that “:)” is also a token.
    2. Word type is a unique item of the vocabulary.
    3. For ex., in a phrase of “this is a sample text for this sample test”, word types are [“this”, “is”, “a”, “sample”, “text”, “for”, “test”]