Robots.txt is a text file at the root of a website that tells crawlers which parts of the site they may crawl.
It controls crawling, not indexing: a blocked URL can still appear in results if other sites link to it. Use noindex to keep a page out of search results.
Robots.txt is also where site owners allow or block AI crawlers such as GPTBot, ClaudeBot or PerplexityBot.
SEO is the work of improving a website so it appears higher in unpaid search results for the searches its customers make.
SEOTechnical SEO makes sure search engines can find, crawl, render and index a website’s pages without problems.
SEOOn-page SEO is optimising the content and HTML of individual pages so they match what searchers are looking for.
SEOOff-page SEO is everything done outside your own website to build its authority, mainly earning links and brand mentions.
