Web URL Crawling

Summary: The Web URL Crawling feature in Antimanual allows your AI to learn from external public sources automatically. It filters out digital noise, ensuring your assistant stays updated, provides accurate answers, and improves customer support.

The Web URL Crawling feature in Antimanual helps your AI learn from external sources. It reads public web pages—such as guides, news, and helpful resources—so your AI can provide more accurate answers.

Feature Overview

This Pro version feature connects your AI to the internet. Instead of copying text by hand, the system automatically pulls and processes web data for you. This ensures your AI assistant stays up to date.

This is a great tool if you store guides on multiple platforms or want your AI to learn from industry experts. By adding URLs, your AI can:

• Check facts quickly.
• Provide better customer support.
• Show a deeper understanding of your specific industry.

Adding a Web URL

Adding a new link is simple:

1. Go to the Knowledge Base: Open your Antimanual dashboard and click the “Website URL” tab.
2. Enter the URL: Paste the public web link you want to add. (Note: The page must not require a login.)
3. Submit: Click the “Submit” button. The system will grab and process the page for you.

How We Extract Data

Web pages often contain “noise” like menus, sidebars, and ads. Antimanual uses smart filters to ignore this clutter and save only the main content.

• Smart Detection: The system finds the main article or content area.
• Noise Removal: It automatically ignores headers, cookie banners, and social buttons.
• Content Cleaning: It removes extra code to provide clear text that the AI can easily read.

This cleaning process ensures your AI focuses only on the helpful information, not on menu links like “About Us.”

Managing Your Links

You can view all your saved URLs in a table to track what information your AI uses. For each link, you will see:

• URL Source: The original link.
• AI Model: The engine used to process the data.
• Actions: A simple delete button to remove old URLs from your Knowledge Base.

Frequently Asked Questions