Skip to main content

​
What is the Data Index?

The Data Index shows all content that has been crawled from your website and indexed for your agent to use.

​
Viewing the Index

See what content your agent has access to:
  • Page URLs: All crawled pages
  • Content Previews: Snippet of extracted content
  • Last Updated: When each page was last crawled
  • Status: Successfully indexed or errors

​
Managing Indexed Content

​
Refreshing Specific Pages

Re-crawl individual pages:
  1. Find the page in the index
  2. Click “Refresh”
  3. Updated content is indexed

​
Removing Pages

Remove pages from the index:
  1. Select the page
  2. Click “Remove from Index”
  3. Content is no longer used by agent
Removing pages from the index means your agent won’t have that information available.

​
Index Statistics

View statistics about your indexed content:
  • Total pages indexed
  • Total content volume
  • Last crawl date
  • Crawl errors

​
Troubleshooting

If a page isn’t in the index:
  • Check it’s not excluded in crawl settings
  • Verify it’s publicly accessible
  • Ensure it’s linked from your site
  • Try manually adding the URL
If indexed content is old:
  • Trigger a re-crawl
  • Check crawl frequency settings
  • Verify the page still exists
If pages show errors:
  • Check if pages are accessible
  • Verify robots.txt isn’t blocking
  • Look for authentication requirements

Configure Websites

Manage website crawling settings