Technical SEO: A Complete Guide

Technical SEO is the work of making sure search engines can find, crawl, render and index a website’s pages, and that those pages load quickly and work well on every device. It is the foundation the rest of SEO stands on. Excellent content and strong links achieve little if Google can’t reach or process the pages they belong to.

This guide explains:

• what technical SEO covers
• how Google crawls and indexes sites
• how to use robots.txt, sitemaps, canonicals and redirects correctly
• what matters for JavaScript, speed, mobile, security, structured data and international sites
• how to audit a site step by step

What is technical SEO?

Technical SEO covers everything about how a website is built and served that affects whether search engines can access, understand and show its pages. It deals with the infrastructure of a site rather than its words or its links.

It is one of the four pillars of search engine optimisation, alongside on-page SEO, content and off-page SEO:

Technical SEO overlaps with on-page SEO in places, such as internal linking and structured data, but its focus is different. On-page SEO asks whether a page is good. Technical SEO asks whether search engines can get to it and process it properly in the first place.

Why does technical SEO matter?

Technical SEO matters because every page must pass through Google’s crawling and indexing systems before it can rank. A technical problem can stop a page at the first hurdle, however good the page is.

Google describes Search as working in three stages: crawling, indexing and serving results, and notes that not every page makes it through each stage. It also states that it doesn’t guarantee it will crawl, index or serve any page, even one that follows Google Search Essentials. Technical SEO is about removing the obstacles that keep good pages from getting through.

The main benefits are:

1. Pages get found. Clear internal links and accurate sitemaps help Google discover new and updated content.
2. Pages get indexed. Removing accidental blocks, duplicates and errors keeps valuable pages in Google’s index.
3. Signals consolidate. Canonicals and redirects focus ranking signals on one preferred URL instead of splitting them.
4. Users stay. Fast, stable, mobile-friendly pages give a better experience, which aligns with what Google’s ranking systems aim to reward.
5. Everything else works. Content and link building only pay off when the pages they support can be crawled and indexed.

How do search engines crawl and index a website?

Google finds pages by following links and reading sitemaps, downloads and renders them, then analyses and stores the ones it considers worth indexing. Understanding each step shows where technical problems can occur.

Crawling: discovering and downloading pages

Google finds new pages mainly by following links from pages it already knows, and from sitemaps submitted by site owners. Its crawler, Googlebot, decides algorithmically which sites to crawl, how often and how many pages to fetch. It tries not to crawl a site so fast that it overloads the server. Server errors, for example, tell it to slow down.

Crawling commonly fails because of server problems, network issues or robots.txt rules that block Googlebot.

Rendering: running JavaScript

During crawling, Google renders pages and runs their JavaScript using a recent version of Chrome, much as a browser would. Rendering matters because many sites rely on JavaScript to add content. Without rendering, Google might not see that content at all.

Indexing: understanding and storing pages

After crawling, Google analyses the page’s text, title, images, video and other elements. It groups pages with similar content and chooses a canonical, the version most likely to be shown in search results. Indexing isn’t guaranteed. Low-quality content, rules that block indexing, and site designs that make content hard to process can all prevent it.

Serving: choosing results

When someone searches, Google returns the indexed pages it judges most relevant and highest quality, adjusted for factors such as location, language and device. Technical SEO doesn’t decide relevance, but it makes sure your pages are eligible to be chosen.

How do you control crawling and indexing?

You control crawling and indexing with a small set of tools: robots.txt, noindex, canonical tags, redirects and status codes. Each does a different job, and mixing them up is one of the most common technical SEO mistakes.

robots.txt

A robots.txt file tells crawlers which URLs they may request. Google explains that it is used mainly to avoid overloading a site with requests, and that it is not a way to keep a page out of Google.

This distinction matters. A page blocked in robots.txt can still appear in search results if other pages link to it. It will usually show without a description, because Google can’t read its content. To keep a page out of search results, Google recommends using noindex or password protection instead.

Use robots.txt for:

• blocking crawl-heavy URL patterns that add no value, such as endless filter combinations or internal search results
• managing crawl load on large sites
• keeping media files out of search results

Avoid using robots.txt for:

• hiding sensitive pages. Use password protection.
• removing pages from search results. Use noindex.
• blocking CSS or JavaScript that Google needs to render your pages.

noindex

A noindex rule, set as a robots meta tag or an HTTP response header, tells Google not to show a page in search results. Google must be able to crawl the page to see the rule. If the same page is also blocked in robots.txt, Google never sees the noindex, and the URL can stay in the index.

Use noindex for pages that users need, but searchers don’t. Examples include thank-you pages, internal search results and thin tag archives.

Canonical tags

A canonical tag (rel=”canonical”) tells Google which URL you prefer among duplicate or very similar pages. Google lists three ways to indicate a canonical, in order of strength:

1. Redirects: a strong signal that the target should become canonical.
2. rel=”canonical” annotations: a strong signal that the specified URL should become canonical.
3. Sitemap inclusion: a weak signal.

Google’s guidance includes several best practices:

• add a self-referencing canonical to each canonical page
• use absolute URLs
• link internally to the canonical version
• never point different canonicalisation signals, such as the sitemap and the canonical tag, at different URLs for the same page

Google also advises against using robots.txt or noindex to choose canonicals within a site.

Canonicals are signals, not commands. Google can choose a different canonical if your signals conflict.

Redirects

A permanent (301) redirect moves a URL and tells Google the destination should replace it. Google treats all permanent redirect methods the same for Search, but server-side redirects take effect fastest. Use redirects when merging pages, moving a site or retiring duplicates.

Avoid redirect chains, where URL A redirects to B, then to C. Google notes that long redirect chains have a negative effect on crawling. Also avoid redirecting removed pages to unrelated pages such as the homepage. If there is no true replacement, a 404 or 410 is clearer.

HTTP status codes

Status codes tell Google what happened when it requested a page. Getting them right helps Google spend its time on pages that matter.

Two status-code problems deserve special attention:

• Soft 404s. A page that shows a “not found” message but returns a 200 OK status. Google keeps crawling soft 404s, which wastes crawl resources.
• Persistent 5xx errors. Server errors make Google slow its crawling. If they persist, pages can drop out of the index.

How do XML sitemaps work?

An XML sitemap is a file that lists the URLs you want search engines to know about. It helps Google discover pages, especially on large sites, new sites, or sites with pages that few internal links point to.

Sitemap rules

Google’s documentation sets out the main rules:

• Size limits. A single sitemap is limited to 50,000 URLs or 50MB uncompressed. Larger sites split their URLs across several sitemaps and can use a sitemap index file.
• Absolute URLs. Use full URLs, such as https://www.example.com/page/, not relative paths.
• Canonical URLs only. List the URLs you want to appear in search results, not duplicates, redirects or noindexed pages.
• Encoding and location. Sitemaps must be UTF-8 encoded. Google recommends placing them at the site root.

What Google uses, and what it ignores

Google ignores the priority and changefreq values in XML sitemaps. It uses lastmod only when the dates are consistently and verifiably accurate. Lastmod should reflect significant changes, such as updates to the main content, structured data or links, not trivial ones like a new copyright year.

Submitting a sitemap

Submit sitemaps in Google Search Console’s Sitemaps report, or reference them in robots.txt with a Sitemap line. Google notes that submitting a sitemap is a hint, not a guarantee that Google will use it. Most content management systems, including WordPress with SEO plugins, generate sitemaps automatically.

For individual pages that need indexing quickly, our guide on how to index a URL in Google Search Console explains the URL Inspection tool. For larger batches, our post on how to index multiple URLs with IndexMeNow covers one third-party approach.

What is crawl budget, and does it matter for your site?

Crawl budget is the set of URLs Google can and wants to crawl on a site. For most websites it isn’t a concern. It becomes important on very large or very fast-changing sites.

Who needs to worry about crawl budget

Google’s crawl budget guide is aimed mainly at:

• large sites with more than about 1 million unique pages whose content changes moderately often, around once a week
• medium or larger sites with more than about 10,000 unique pages whose content changes very rapidly, such as daily
• sites where a large share of URLs appear in Search Console as “Discovered – currently not indexed”

Google stresses that these numbers are rough estimates, not exact thresholds. For sites whose pages are crawled the same day they’re published, Google says keeping the sitemap up to date and checking the Page Indexing report is enough.

How crawl budget works

Google describes crawl budget as the combination of two things:

• Crawl capacity limit. How much Google can crawl without overloading the server. It rises when a site responds quickly and consistently, and falls with slow responses, server errors or rate limiting.
• Crawl demand. How much Google wants to crawl. It depends on the site’s size, how often it updates, its quality and its popularity.

How to use crawl budget efficiently

Google’s recommendations include:

• consolidating duplicate content so crawling focuses on unique pages
• blocking unimportant URL patterns, such as differently sorted versions of the same list, with robots.txt
• returning 404 or 410 for permanently removed pages
• eliminating soft 404s
• keeping sitemaps current with accurate lastmod dates
• avoiding long redirect chains
• making pages fast to load and supporting 304 Not Modified responses

Google also notes that noindex isn’t a crawl budget tool. Google still requests a noindexed page, then drops it, so the crawl time is spent anyway.

How does JavaScript affect technical SEO?

JavaScript can make content harder for search engines to process, because Google must render the page before it can see anything the scripts add. Most modern sites use JavaScript, so the goal is to use it in ways that keep content discoverable.

The main JavaScript SEO practices are:

1. Make important content available in the HTML where possible, through server-side rendering or static generation, so it doesn’t depend on client-side scripts.
2. Use real links. Google discovers pages through standard <a href> links. Navigation that only works through click handlers can hide pages from crawlers.
3. Don’t block resources. If robots.txt blocks the JavaScript or CSS files a page needs, Google may not render it correctly.
4. Handle lazy loading carefully. Content that loads only after scrolling or clicking should still be discoverable when the page renders.
5. Keep canonicals clear. Google recommends setting the canonical URL in the HTML source and making sure JavaScript doesn’t change it.

You can see how Google renders a page with the URL Inspection tool in Search Console, which shows the HTML Googlebot received.

What are Core Web Vitals and page experience?

Core Web Vitals are Google’s three metrics for real-world user experience: loading speed, responsiveness and visual stability. They are part of the broader idea of page experience.

The three Core Web Vitals

Google publishes the following thresholds for a good experience:

How much page experience affects ranking

Google states that Core Web Vitals are used by its ranking systems. It also stresses that there is no single page experience signal, and that good scores don’t guarantee top rankings. Google always seeks to show the most relevant content, even where page experience is weaker. A good page experience helps most when many pages are equally relevant.

Google’s self-assessment questions for page experience include:

• Do pages have good Core Web Vitals?
• Are they served securely over HTTPS?
• Do they display well on mobile?
• Do they avoid excessive ads and intrusive interstitials?
• Can visitors easily tell the main content apart from everything else?

Practical ways to improve speed

Common improvements include:

• serving images in modern formats at the right size
• deferring non-critical JavaScript
• reducing unused CSS
• using caching and a content delivery network
• improving server response time
• reserving space for images and embeds so the layout doesn’t shift

Google’s PageSpeed Insights and Search Console’s Core Web Vitals report show both lab and real-user data.

Speed work is a balance. Aggressive optimisation, such as stripping CSS that scripts add at runtime, can break layouts if it isn’t tested carefully. We have seen exactly this on our own WordPress sites, where a caching plugin’s unused-CSS feature removed styles that page scripts needed. Test every change from a logged-out browser, which is the view Google and most visitors get.

Why do mobile-first indexing and HTTPS matter?

Google primarily uses the mobile version of a site for indexing and ranking, and it prefers secure HTTPS pages. Both are now baseline requirements rather than optional extras.

Mobile-first indexing

With mobile-first indexing, the content, links and structured data on the mobile version of a page are what Google mainly evaluates. If the mobile version hides content, drops internal links or leaves out structured data that the desktop version has, that content may effectively be invisible to Google.

The simplest approach is responsive design: one URL and one set of HTML that adapts to every screen size.

HTTPS

HTTPS encrypts the connection between the browser and the server. Google’s page experience guidance includes secure serving, and Google prefers HTTPS URLs over HTTP equivalents when choosing canonicals, unless problems get in the way. Those problems include:

• an invalid certificate
• insecure dependencies on the page
• an HTTPS page that redirects to HTTP

To secure a site properly:

• redirect all HTTP URLs to HTTPS
• use a certificate that matches every host variant of the domain
• list only HTTPS URLs in sitemaps and hreflang annotations
• consider enabling HSTS so browsers always use HTTPS

How does structured data help?

Structured data is code, usually in JSON-LD format, that describes a page’s content in a way search engines can read reliably. It can make pages eligible for rich results, such as breadcrumbs, product details, recipes, events or review snippets.

Google’s structured data guidelines require markup to represent the visible content of the page accurately. Markup that describes content users can’t see, or that misrepresents the page, can make a page ineligible for rich results. Structured data helps Google understand a page. It isn’t a shortcut to higher rankings.

Practical tips:

• use the types relevant to your content, such as Organization, BreadcrumbList, Article or Product
• validate markup with Google’s Rich Results Test
• keep markup in sync with the visible content when pages change

Google also notes that no special structured data is needed to appear in AI Overviews or AI Mode. Its standard technical requirements and SEO best practices apply.

How do you handle international and multilingual sites?

International sites need separate URLs for each language or region, connected with hreflang annotations, so Google can show the right version to each searcher.

hreflang

Hreflang annotations tell Google which language and, optionally, which region each version of a page targets. Each version should list all the others and itself. Missing return links are among the most common hreflang errors.

Finland is a good example of why this matters. It has two official languages, so a Finnish site may need both fi-FI and sv-FI versions, as our guide to link building in Finland explains.

Locale-adaptive pages and geo-blocking

Some sites serve different content on the same URL depending on the visitor’s country or language. Google warns that it might not crawl, index or rank all the content of such pages, because Googlebot’s default IP addresses appear to be based in the US and it doesn’t send an Accept-Language header. It recommends separate locale URLs with hreflang instead.

Geo-blocking needs the same care. Google advises treating Googlebot like any other visitor from the country it appears to come from. A site that blocks US visitors will also block Googlebot when it crawls from the US. This is a common issue for regulated sites, as covered in our guide to casino SEO.

Native content for each market

Technical setup only works when each language version contains genuine native content, not machine translation. See our pages on Japanese content and content creation for more on writing for international audiences.

How do you run a technical SEO audit?

A technical SEO audit checks a site systematically for problems that stop pages from being crawled, indexed or served well. A practical audit follows a consistent order, from access through to experience.

Step 1: Start with Google Search Console

Search Console shows data directly from Google. Review:

• the Page Indexing report, to see which pages are indexed and why others aren’t
• the Sitemaps report, to check for processing errors
• the Core Web Vitals and HTTPS reports, for page experience issues
• Manual actions and security issues, for anything Google has flagged

Step 2: Crawl the site

Use a crawler such as Ahrefs Site Audit or Screaming Frog to find:

• broken links and redirect chains
• missing or conflicting canonicals
• noindex tags
• orphan pages with no internal links

Our Ahrefs site audit quick guide walks through one setup. When you’ve finished with a project, our post on how to delete an Ahrefs audit explains how to clean up.

Step 3: Check robots.txt, sitemaps and canonicals together

Make sure all three agree:

• robots.txt allows important pages
• sitemaps list only canonical, indexable URLs
• canonical tags point to the same URLs that internal links and sitemaps use

Step 4: Test rendering and mobile

Inspect key templates, such as the homepage, categories, articles and product pages, with the URL Inspection tool. Confirm that the main content, links and structured data appear in the rendered HTML and on the mobile version.

Step 5: Measure performance

Check Core Web Vitals field data in Search Console and PageSpeed Insights. Prioritise templates that serve many pages, because one fix there improves every page that uses them.

Step 6: Review internal linking

Make sure important pages are linked from relevant, strong pages with descriptive anchor text. Our post on internal link opportunities in Ahrefs shows a fast way to find gaps.

Step 7: Prioritise and fix

Rank issues by impact and effort. Blocked or deindexed important pages come first, followed by sitewide template problems, then individual page issues. Re-crawl after fixes to confirm they worked.

What are the most common technical SEO mistakes?

Most technical SEO mistakes come from small configuration errors with large effects. These are the ones we see most often.

1. Blocking the whole site after launch. A leftover “Disallow: /” in robots.txt or a sitewide noindex from a staging setup can remove an entire site from search.
2. Using robots.txt to hide pages. Blocked URLs can still be indexed. Use noindex or password protection instead.
3. Combining noindex with a robots.txt block. Google can’t see the noindex if it can’t crawl the page.
4. Conflicting canonical signals. The canonical tag, sitemap and internal links point to different URLs for the same content.
5. Redirect chains and loops. They waste crawl resources and dilute signals.
6. Soft 404s. Error pages returning 200 OK keep getting crawled.
7. Content hidden on mobile. Under mobile-first indexing, the mobile version is what counts.
8. JavaScript-only navigation. Links that aren’t real <a href> elements can hide pages from crawlers.
9. Mixed content and certificate errors. They undermine HTTPS and can affect which URL Google treats as canonical.
10. Open staging sites. Indexable test copies create duplicate content.

How does technical SEO connect with links and content?

Technical SEO, content and links depend on each other. Links help Google discover pages and are one signal of trust, but they only help pages that can be crawled and indexed. Content earns links and rankings only when it is technically accessible.

A few connections are worth knowing:

• Broken pages waste links. When a page with backlinks returns a 404, the value of those links is lost unless the URL is redirected to a relevant replacement.
• Canonicals consolidate links. Links pointing at duplicate URLs count towards the canonical when Google consolidates the cluster.
• Spammy links need monitoring. Hacked sites and spam attacks can create harmful links. Our guide on how to disavow links in Google Search Console explains when that tool is appropriate.

Our guide to off-page SEO explains link signals in more depth. Links obtained through our backlink marketplace work best when the target pages are technically sound.

How we use technical SEO

We run a portfolio of websites of our own, mostly on WordPress, across several markets. The practices in this guide are the ones we apply to them, including the occasional lesson learned from a caching setting that broke a page for logged-out visitors.

For clients, we offer two things: sponsored placements on established sites through our self-serve backlink marketplace, and native-language content for their own websites. Marketplace links carry rel=”sponsored” and are labelled as advertising on the publishing sites.

FAQ

What is the difference between technical SEO and on-page SEO?

Technical SEO makes sure search engines can access, crawl, render and index a site, covering things like robots.txt, sitemaps, canonicals, speed and HTTPS. On-page SEO makes each page clearly useful for its target search, through titles, headings, content and internal links. They overlap in places, but technical SEO comes first, because a page must be indexable before its content can rank.

Does robots.txt stop a page from being indexed?

No. Google states that robots.txt is not a mechanism for keeping a page out of Google. A blocked page can still appear in search results if other pages link to it. To keep a page out of search results, use a noindex rule or password protection, and make sure the page isn’t blocked in robots.txt, so Google can see the noindex.

Do Core Web Vitals affect rankings?

Yes, Google says Core Web Vitals are used by its ranking systems. It also says there is no single page experience signal, and that good scores don’t guarantee top rankings. Relevance comes first. Good Core Web Vitals help most when several pages are similarly relevant to a search.

Do I need to worry about crawl budget?

Most sites don’t. Google’s crawl budget guide is aimed mainly at very large sites, with around a million or more pages, or sites with tens of thousands of pages that change daily. For most sites, Google says keeping the sitemap up to date and checking the Page Indexing report is enough.

How often should I run a technical SEO audit?

Check Search Console regularly, ideally weekly, and run a full crawl after any major change, such as a redesign, migration, CMS update or new plugin. Many sites also run a full audit every few months. Technical problems often appear after changes, so checking straight afterwards catches most of them early.