We’ve made it easier to bring the right content into your knowledge base. When you add a URL, you can now tell the system whether you want just that page, everything under a section, or the whole site. The crawler also skips irrelevant pages like login screens and avoids indexing the same page twice, so you get cleaner, more relevant results faster.

Here’s what’s new.

  • Crawl scope selection: Choose “page”, “section”, or “site” when adding a source to control how much is crawled.
  • Duplicate page protection: The crawler now detects and skips pages it has already indexed.
  • App‑shell filtering: Common non‑content pages such as login screens are automatically ignored.
  • Help‑center URL handling: URLs that look like help or documentation pages are treated specially to keep crawling focused.

Crawl scope selection

When you add a new URL, a simple dropdown lets you pick the desired scope:

  • Page – only the exact URL you entered.
  • Section – the URL and everything underneath it (e.g., all articles in a folder).
  • Site – the entire domain.

This gives you precise control over how much content is pulled in, helping you avoid unnecessary data and keep your knowledge base tidy.

Duplicate page protection

The system now tracks pages it has already processed. If the crawler encounters the same page again, it skips it automatically, saving time and preventing redundant entries.

App‑shell filtering & help‑center handling

Common entry points like /login or other app shells are ignored during crawling, so you won’t waste resources on pages that don’t contain useful information. URLs that resemble help‑center or documentation sites are recognized and handled in a way that keeps the crawl focused on the most relevant content.

Other improvements

  • Updated onboarding wizard and source‑adding UI to include the new scope options and helpful hints.
  • Minor UI tweaks for clearer guidance when entering URLs.