Skip to main content

​
Website Crawling

Clarky can automatically crawl and learn from your website, keeping your agent’s knowledge up to date.

​
Adding a Website

1

Navigate to Knowledge > Websites

Access the website management section.
2

Enter Website URL

Provide your full website URL (e.g., https://yourbusiness.com).
3

Configure Crawling

Set crawl depth, included/excluded pages, and frequency.
4

Start Crawl

Clarky begins crawling and indexing your site.

​
What Gets Crawled

Clarky extracts:
  • Page text content
  • Headings and structure
  • Meta descriptions
  • FAQ sections
  • Product/service descriptions
  • Contact information
Does not extract:
  • Images (only alt text)
  • Videos
  • JavaScript-rendered content (limited)
  • Password-protected pages

​
Re-Crawling

Update your knowledge by re-crawling:
  • Manually trigger re-crawl anytime
  • Schedule automatic re-crawls
  • Get notified of significant changes

View Data Index

See what’s been indexed from your website