How do I crawl a website?
- Go to the Knowledge Base of the Library you want to add knowledge to.
-
Click
Add Knowledge
button.

- Select Crawl website from the Add Knowledge window.


- Paste the URL of the site you want to crawl in the URL field.

- Click
Submit
button. Once submitted, it will take some time to crawl the website. You can check how many pages are currently found in the bottom left corner of the Crawl Website Section.

- Once the crawling is done, a window will show you the crawled pages and the number of credits that will be consumed to add the pages to the knowledge base.


- Click on the
Submit
button in the window to confirm and add it to your library.
It’s scraping unnecessary content. How do I refine what I scrape?
You can open theAdvanced
accordion and use selectors to select the content you want to scrape in the URL.
- Open the Advanced accordion by clicking on it.

- Input the selectors of the content you want to scrape in the text field. Thse selectors are used to select the content you want to scrape in the URL.
Sample HTML Code
CSS Selector
- Click on
Submit
button.


- Click on the
Submit
button in the window to confirm the scraping and add it to your library.

Advanced Scraping Output