Tokencito Game of the day Play

What is natural language processing (NLP)?

Natural language processing (NLP) is the branch of artificial intelligence that processes human language, written or spoken. It includes classifying text, translating, summarizing and generating answers. A spam filter, a translator and a chat with an LLM are all examples of NLP.

📏Hand-written rules📊Models that learn from examples💬LLM: many tasks through instructions

Put it to the test

A question from the app. Pick an answer.

You want to sort 5,000 reviews into positive and negative. Do you need generative AI?

Start playing on the web

Which tasks are NLP

Everything a machine does with words: filtering spam, telling whether a review is good or bad, pulling names and dates out of a contract, translating, summarizing, turning speech into text or answering questions. For decades, each task had its own system, built and trained just for that.

One spam filter, three eras

With rules: if the email says "you won" and "free", it is spam. It works until someone writes "fr3e". Someone had to write and maintain the rules by hand.

With machine learning: the model learns from thousands of emails people already marked as spam or not, and finds the clues on its own. Better, but it needs those examples and only knows that one task.

With an LLM: you paste the email and ask "is this spam? answer yes or no". You trained nothing, and the same model also translates, summarizes and extracts data. That is the big shift.

Did LLMs replace NLP?

No: an LLM is a tool within NLP. A tokenizer cuts the text into tokens, and the model works with numerical representations. Other models generate embeddings to find similar texts.

A small model trained for a single task still has its place: if you have to classify millions of similar texts every day, it is usually cheaper and faster than calling an LLM every time.

Not neuro-linguistic programming

Search for "NLP" and you may land on neuro-linguistic programming, a coaching and communication approach that has nothing to do with computers. Same three letters, different thing: in AI, NLP always means natural language processing.