Broken links are easy to dismiss as a hygiene problem — until they're a checkout flow that quietly stopped working, a documentation reference that's been 404 for six months, or an outbound vendor link that now resolves to a phishing kit. This is the long-form version of what broken-link checkers are, how the category works, what its architecture costs you, and what's worth running instead — or alongside.
What a broken-link checker is
A broken-link checker scans a site and reports the URLs that don't load. Mechanically: discover the links on each page, fire an HTTP request at each one, log the status code. Useful output is the list of links that returned 404, 500, timeout, or DNS-not-found.
Three operations, in order:
- Discovery. Walk pages, parse
<a>tags, build a graph - Verification. HEAD or GET each link, record the response
- Reporting. Output the list of failures, ideally with the page each one was found on
The category has existed since the 1990s. The first widely-used tool was Xenu's Link Sleuth in 1997. The market has gotten more polished but the architecture has barely changed.
Why broken links matter
The cost shows up in three places:
SEO. Search engines treat outbound-link health as a quality signal. Sites with measurable rot get downranked. Crawl budget on your own site gets wasted following dead internal links instead of indexing new ones. Internal link equity that should flow through <a href> chains hits dead ends.
UX and conversion. A broken footer link is a paper cut. A broken checkout link is a revenue hole. A broken "documentation" link in your support flow becomes a support ticket. The asymmetry matters: not all broken links are equal, and the worst ones are usually on the highest-traffic pages.
Compliance and trust. Regulated industries (finance, healthcare, education, public sector) get audited on link hygiene. A .gov or .edu site pointing at a parked-domain gambling redirect is a finding. Outside regulation, customers screenshot. Screenshots end up in escalations.
The numbers people quote about user impact ("88% of users won't return after a bad experience") trace back to a 2009 Akamai/Gomez study and are old enough to deserve a citation rather than confident quoting. The directional truth is intact: broken outbound links cost something measurable. The magnitude varies.
HTTP status codes in 30 seconds
A checker's output is mostly status codes:
| Range | Meaning | What it usually indicates |
|---|---|---|
| 2xx | Success | Link works |
| 3xx | Redirect | Link redirects; chain length matters |
| 4xx | Client error | Broken link (most commonly 404; also 403, 410) |
| 5xx | Server error | Destination's problem, possibly transient |
| Timeout / DNS / connection refused | Host unreachable | Could be permanent, could be flaky |
Good checkers distinguish hard failures from transient ones and rerun the latter before marking them dead. Bad checkers report timeouts as broken on the first try and produce a lot of false positives.
The categories of tool
There are five reasonably-distinct types. Each makes different trade-offs.
1. Web-based one-shot checkers
Examples: BrokenLinkCheck.com, DeadLinkChecker.com, Dr. Link Check.
Paste a URL, hit scan, get a list. No install, mostly free, capped at low page counts on the free tier. Crawler runs from their servers, so coverage is whatever an anonymous external bot can see: public pages, statically linked, no auth, no JS-rendered content.
Best for: quick audits of small public sites.
2. Browser extensions
Examples: Check My Links (Chrome), Link Checker (Firefox).
Run against the current page only. The browser is already authenticated, so they can check pages behind a login that crawlers can't reach — but only one page at a time, with no automation.
Best for: editors and authors verifying a specific page before publishing.
3. Desktop crawlers
Examples: Screaming Frog, Xenu's Link Sleuth.
Installed software that crawls from your machine. More configurable than the web-based tools, handles larger sites, exports detailed CSV/Excel reports. The good ones (Screaming Frog in particular) handle JS-rendered content via a headless-Chromium mode.
Best for: SEO professionals and audit consultants doing periodic deep-dives.
4. CMS plugins
Examples: the WordPress "Broken Link Checker" plugin and its variants.
Runs inside the CMS, scans content it has direct database access to, can check links in the background and surface issues in the admin. The downside is exactly the upside: it knows about content the CMS knows about, and doesn't see anything else.
Best for: WordPress-only sites that want a self-contained workflow inside the admin.
5. Continuous monitoring services
Examples: LinkSentry, certain SaaS Ahrefs/Semrush features.
Continuous checking instead of point-in-time scans. Some are crawler-based (just running the same checks more often). Others, like LinkSentry, replace the crawler with session-based discovery from real visitor traffic — coverage matches what users actually load, including behind login and inside dynamic content. Alerts fire when state changes, not when you next remember to run a scan.
Best for: any site where the outbound surface is ongoing infrastructure, not a one-time audit target.
Where traditional checkers stop short
Four limitations are worth naming explicitly.
1. Crawler accessibility. External crawlers can't sign in, can't render JS-loaded content reliably without a headless browser, and can't see personalized or role-gated links. On a modern web app, the crawlable surface is usually a small fraction of the actual surface. Empirically, this can easily mean missing the majority of links the site actually serves.
2. Snapshot-only detection. A check valid Monday is invalid Tuesday. Domains expire between scans. Vendors compromise their third-party scripts between scans. Re-registrations and content drift happen between scans. The longer the gap between checks, the more the report you're reading is already out of date.
3. False positives and negatives. Rate limiting, geo-blocking, bot detection, and aggressive WAFs cause working links to look broken. "Soft 404s" (servers returning HTTP 200 with a "page not found" body) cause broken pages to look working. Redirect chains and slow-loading pages add noise. Decent tools mitigate this; lazy ones don't.
4. Scope is just "broken." A checker tells you what's returning a 4xx. It doesn't tell you that a working 200 link is now a parked domain, a re-registered hijack, a phishing kit, an NSFW destination, or a script you didn't authorize. HTTP 200 with the wrong content is invisible to the entire category.
That last one is the gap that matters most, and it's why "link auditing" exists as a distinct thing from "broken-link checking."
Beyond broken: link auditing
Link auditing is the superset. The traditional checker asks: did this link return an error code? Link auditing also asks:
- Is the TLS chain still valid? Is the certificate about to expire?
- Has the destination's content materially changed since the link was first seen?
- Was the domain re-registered or transferred recently?
- Is the destination on a threat-intel list for malware, phishing, or scam delivery?
- Is the destination a parked domain serving ads?
- Is the destination category-appropriate (no NSFW or gambling on a healthcare site)?
- Is a new third-party script appearing on a sensitive page (checkout, login, account settings)?
A site running a quarterly broken-link audit and nothing else catches the dead URLs and misses every one of the other failure modes.
How to choose
A short decision tree:
- Site is small, mostly public, infrequently updated, budget is zero. A free web-based checker is the right answer.
- WordPress site, want it in the admin. The WordPress Broken Link Checker plugin.
- SEO team doing quarterly deep dives. Screaming Frog or the broken-link feature in Ahrefs/Semrush, depending on what you already pay for.
- The site has auth, dynamic content, or anything past a static page; outbound rot matters over time; the threat model includes anything past 404. A continuous monitoring service. LinkSentry is built specifically for this case.
- You want all of the above. Run a checker quarterly and continuous monitoring on the always-on side. They're complementary, not redundant.
Practices that work regardless of tool
- Cadence matched to traffic and revenue exposure. Daily monitoring for revenue-bearing flows, weekly for content sites, monthly is the floor for anything important.
- Triage by impact, not by alphabetical order. Broken checkout > broken homepage > broken archived blog post. Always.
- Fix at the source. Internal: update the URL. External: replace or remove, don't paper over with a redirect that you'll forget about.
- Avoid redirect chains. A → B → C → D is hostile to both crawlers and users. Collapse to A → D.
- Inventory your third-party dependencies. Analytics, fonts, polyfills, chat widgets, A/B tools, OAuth iframes. When one of them is compromised, you want to know what's on the list.
- Re-audit after structural changes. Migrations, CMS upgrades, theme changes, plugin swaps, large content imports — all of these break links by accident.
The short version
A broken-link checker tells you what's currently 404. That's useful and not nearly enough. Continuous link auditing tells you what's currently 404, what was 200 yesterday and isn't today, what's about to expire, what's been re-registered, what's been flagged by a feed your browser hasn't pulled yet, and what new third-party scripts your pages started loading without you noticing.
If your site is small and quiet, the cheap version of the cheap tool is fine. If your site is anything else, the continuous version is the right shape.
Related Posts: