A crawler is a program that walks through web pages by following their links to discover and read them, like Googlebot or the bots of AI companies. A scraper goes to a specific page and extracts concrete data, like prices or emails. The crawler explores; the scraper copies what it cares about.
A question from the app. Pick an answer.
Email: "I was charged twice this month. I need the refund today." Jev, with example numbers: Team: Billing 86% · Support 10% · Sales 4% Urgent?: 91% yes
How does an LLM build its answer to the same email?
It predicts the next token: it spreads probabilities over its whole vocabulary, picks one and repeats. Those are token probabilities. Jev gives them on your options.
If it writes percentages, it also generates them token by token, as text. Jev's numbers are its answer, on your options.
It can: it can name the team in a sentence, or even as structured data. Either way it generates it token by token.
Imagine you want to compare your competitors' prices. A crawler is what finds the pages: it enters the home page, follows every "products" link and builds the list. The scraper is what, on each one, pulls the name and the price and puts them in a spreadsheet. Many tools do both, which is why people mix them up.
Google works with a crawler: Googlebot walks the web so your site can show up in searches. If the data is available through an API, asking for it there is cleaner and more stable than scraping the page.
If you look at your site logs you will see visits from named bots. Two well known families: GPTBot (OpenAI) and ClaudeBot (Anthropic) collect content that may be used to train language models. Others, like OAI-SearchBot and Claude-SearchBot, are there so your page can show up in answers that use search.
A third type does not crawl anything: ChatGPT-User and Claude-User retrieve pages to answer user requests. It is what, for example, a RAG setup with web search does.
robots.txt is a file at the root of your site (yoursite.com/robots.txt) where you ask each bot, by name, which parts it should not walk. You can let a search bot in and close the door on a training one. But it is a request, not a lock: not all bots respect it. And it does not remove a page from Google or protect anything private: for that there are passwords.
If you are the one scraping, first check the site's terms of use, its robots.txt and whether personal data is involved. What is legal depends on the country, the site and what you do with the data, and it sometimes changes. This is not legal advice: for a business doubt, ask a lawyer.