Skip to main content

Links

You can configure the AI Chatbot to crawl a set of URLs so that it has more information that it can use for generating a response.

You can add individual URLs/Links through the Individual Links tab in the popup that appears when you click on the Add Links button.

You can also set the Refresh Interval so that the chatbot recrawls the URL every regular period to get the latest information.

The URLs need to start with http:// or https://.

Most website URLs e.g. WordPress, WIX, Webflow, Squarespace present server side rendered HTML Content Type web pages. Cleiry also supports Single Page Applications (SPA Applications) that uses Javascript to render the entire website, e.g. React, Vue, Angular. For API responses, select JSON.

You can bulk add many URLs/Links through the URL Loader tab in the popup that appears when you click on the Add Links button. The same settings apply when adding URLs/Links Individually.

Many websites that are optimized for SEO have an XML sitemap that can be accessed via a url like https://website.com/sitemap.xml. This is especially true for WordPress websites that use SEO plugins like Yoast and All in One SEO. For websites like these, you can add all public URLs in the website via using the Sitemap Loader tab in the popup that appears when you click on the Add Links button.