> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clarky.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Website Management

> Configure automatic website learning

## Website Crawling

Clarky can automatically crawl and learn from your website, keeping your agent's knowledge up to date.

## Adding a Website

<Steps>
  <Step title="Navigate to Knowledge > Websites">
    Access the website management section.
  </Step>

  <Step title="Enter Website URL">
    Provide your full website URL (e.g., [https://yourbusiness.com](https://yourbusiness.com)).
  </Step>

  <Step title="Configure Crawling">
    Set crawl depth, included/excluded pages, and frequency.
  </Step>

  <Step title="Start Crawl">
    Clarky begins crawling and indexing your site.
  </Step>
</Steps>

## What Gets Crawled

Clarky extracts:

* Page text content
* Headings and structure
* Meta descriptions
* FAQ sections
* Product/service descriptions
* Contact information

Does not extract:

* Images (only alt text)
* Videos
* JavaScript-rendered content (limited)
* Password-protected pages

## Re-Crawling

Update your knowledge by re-crawling:

* Manually trigger re-crawl anytime
* Schedule automatic re-crawls
* Get notified of significant changes

<Card title="View Data Index" icon="database" href="/features/knowledge/data-index">
  See what's been indexed from your website
</Card>