To make sure that your Chatbot does store old links, we would recommend that you ignore URLs that you do not want the chatbot to index.

<aside> 💡 The difference between ignoring URLs and deleting URLs is that the former will completely exclude those URLs from ever being crawled. This means that if you choose to reindex your chatbot, ignored URLs will not be crawled.

Deleting urls is a one off deletion, and deleted URLs are removed from your chatbot. However, in the event of a recrawl, webpages that were previously deleted will reappear

</aside>

How to ignore URLs from your site

  1. Click on “Create Bot” or “Edit Chatbot”

Untitled

  1. To ignore a single page url, copy and paste the url of the specific page you wish to ignore

  2. To ignore an entire subdirectory of pages such as https://wonderchat.io/blogs you will need to add an asterisk (*) behind the url name. So adding https//wonderchat.io/blogs/ will automatically exclude all the URLs in your website under the subdirectory /blogs.

  3. Once the link is ignored, it will not be recrawled for your monthly chatbot recrawl or if you press the recrawl button.