Scraping URL's to create business context

General
1

Hi there,

I’m trying to create a chatbot that takes in one or multiple URL’s to scrape and then create a business context that can be used to feed into another chatbot platform, to train the bot to answer questions about the business based on the context. The issue I’m encountering at the moment is that I get an error saying the URL couldn’t be scraped, but if I input the same URL into chatGPT it gets scraped no issues at all. are there any work arounds to scrape URL’s I though it would’ve done it straight out of the box. Just for completness sake I’m using another platform for some chatbots as I have the ability to set a webhook and send the data via email or save into a Google sheet. I really like the ability to just add the URL into a Pickaxe bot and it just does it but the fact there is no way to get the data out other than by copy and pasting it for reporting makes it not a good solution. thank you

2

Hey Alex.

When a user gives the tool a link it doesn’t necessarily scrape it. Instead, if it has web-browsing enabled, it will search the internet and try to get information about a topic.

Websites in Knowledge Base
So importantly, when you add a website to the knowledge base it actually scrapes the website, turns the content into a text document, and then puts that in the knowledge base for the Pickaxe.

Websites in Chat
When you enable Web Browsing, something a little different happens. The Pickaxe looks at the user’s message, then searches the web looking for relevant content, grabs some, and uses it to inform its response. If an end-user drops a URL into a chat, the Pickaxe will search the web for it. Not all URLs work equally well. I’ve found there is relative parity between what websites work on our system vs. ChatGPT’s but it’s not an exact 1-to-1.