Trust Users, Not Crawlers

The dominant model for monitoring a website's outbound links is to run a crawler against it: an automated bot that starts at the homepage, follows every <a> it can see, and reports what it finds. For a static brochure site in 2008, that worked. For a modern web app, it misses most of the surface area.

LinkSentry doesn't crawl. We collect outbound-link data from your actual visitors. Every link a real user encounters in their session gets reported back, deduplicated, and watched. The model is borrowed from a much older idea: crowdsourcing.

A quick aside on crowdsourcing

Crowdsourcing replaces a central data-collection operation with a network of contributors who each report a small piece of the picture. The classic examples all share the same shape:

• Google Maps traffic. No fleet of sensors. Anonymized speed data from the phones of people already driving. The map updates faster than any government traffic-monitoring system because the cars are the sensors.
• Waze. Real-time hazard reports from drivers — police, road closures, hazards, accidents — fed back into routing in seconds.
• reCAPTCHA. Every "select all images with a bus" you've ever solved trained a vision model, and a lot of them digitized old books along the way.
• Wikipedia. No editorial staff. The encyclopedia is a million volunteer edits a day.

The trick in every case is that the dataset is too big, too local, and too fast-changing for any one operator to collect. Distributing the collection is the only way to keep up.

Outbound links on a modern website are the same kind of problem.

Why crawlers don't see the half of your site that matters

Walk through what a crawler can actually reach:

• Login walls. Anything past /login — your dashboard, your admin, your customer portal, your support pages, your KB after the paywall — is invisible. The crawler doesn't have a session.
• Dynamic content. React, Vue, Svelte, Solid — anything that renders client-side. Naive crawlers see an empty <div id="root">. Headless-browser crawlers do better but still miss content gated behind hover states, tab switches, lazy-loaded modules, and interaction-triggered fetches.
• Personalization. Links that only appear for a specific user tier, role, region, or feature flag. A crawler that signs in as one account sees one slice. The other slices stay unmapped.
• Stateful flows. A multi-step checkout, a wizard, an onboarding sequence — the crawler doesn't fill the form.

Add it up and the crawler-visible portion of a real product site is usually a fraction of what's actually shipped to users. The links you most care about — the ones inside the product, where customers spend their time — are almost always on the invisible side.

What we do instead

When you install LinkSentry, you drop a small read-only script on your site. It does no fetching of its own, no DOM mutation, and no PII capture. What it does is observe the outbound destinations the user's session actually loads or hovers or clicks, and report them back.

The model has a few consequences worth naming:

• Coverage matches reality. If a user can reach a page, we can map its outbound links. No login flow to script, no JavaScript engine to babysit, no permission matrix to enumerate.
• Discovery happens at user speed. A link that appeared on the dashboard this morning is in your inventory by lunch — not on the next quarterly crawl.
• Scaling is free in the right direction. More traffic → faster discovery → more complete inventory. The opposite of crawler economics, where more pages means a longer crawl window.
• No origin load. We're observing a request the browser was going to make anyway. There's no recrawl, no per-link probe burst, no robots.txt etiquette to negotiate.

Once a link is in the inventory we watch it for the things that matter: dead, parked, NSFW, malware-flagged, low-reputation, recently re-registered, content-drifted from its first-seen baseline.

You get alerts when a destination changes — not the day a customer screenshots the parked-domain ad on your help page.

The short version

Crawlers see the parts of a site that don't matter very much. Real users see the parts that do. We collect from the users. That's the whole pitch.