Hey Alex.
When a user gives the tool a link it doesn’t necessarily scrape it. Instead, if it has web-browsing enabled, it will search the internet and try to get information about a topic.
Websites in Knowledge Base
So importantly, when you add a website to the knowledge base it actually scrapes the website, turns the content into a text document, and then puts that in the knowledge base for the Pickaxe.
Websites in Chat
When you enable Web Browsing, something a little different happens. The Pickaxe looks at the user’s message, then searches the web looking for relevant content, grabs some, and uses it to inform its response. If an end-user drops a URL into a chat, the Pickaxe will search the web for it. Not all URLs work equally well. I’ve found there is relative parity between what websites work on our system vs. ChatGPT’s but it’s not an exact 1-to-1.