Uploading list if links rather than pasting them individually

General
, ,
1

Hello i would like to know that can i upload list of links as txt files rather than pasting each link as i have a collection of 10000000 links to add to knowledgebase

2

Plus one ! And better website side scrapping options to scape an entire domain for example @admin_mike

3

Currently you can only add one website at a time. We are working on a better webscraping system.

It would be interesting to know what other options you would like to see with this feature.

4

Same here, I think if we could add the sitemap.xml or upload a txt with all the urls would be a great start.

5

Hi @Slabs,

You can upload https://<domain>/sitemap.xml into the knowledge base now.

1 Like
6

Hi @stephenasuncion ,

Thanks for the response, I have tried adding https://www.petsmart.ca/sitemap_1.xml and it did not work, is the tool only accepting urls ending on ‘/sitemap.xml’ ?

1 Like
7

Hi @Slabs,

You should be able to add URLs ending with sitemap_<index>.xml or sitemap.xml now

1 Like
8

Wonderful! Thanks for the fast update @stephenasuncion

9

Hi @stephenasuncion @admin_mike
I understand that you added a limit of 50 files per bot for the knowledge base and I appreciate your last updates to the KB setup, but why the URL is also limited to 50 links? Could we have a higher limit when it is only URLs? Depending on the case, 50 links is too little.

5 Likes
10

Hello Pickaxe team
Just two three two three one things to know from you

  1. are you planing for razorpay payment gateway as stripe is not yet available in India and I am not able to monitize my studio
  2. do you have plans for gemini api as they are offering ai startup credits
  3. can we have multiple options to use means knowledge base+ source llm or any plans for integration with pabbly?