Knowledge Base and Costs of Content Tokens

Hope someone can please help :slightly_smiling_face:

So I’ve built a Support Bot which works great but…

At first I tried to put all the Support Knowledge in as a Knowledge Base but it wasn’t providing very good answers.

So instead I started copying over the Support Knowledge INTO the main Prompt Frame in the Pickaxe.

The results were amazing!

But my costs are insane - over $40 a day just from me and some staff testing this thing!

I’m guessing this is because Im forcing GPT to pull in the entire Knowledge Base into the content window every time someone enters a query?

Please can someone help me either;

  1. Get an external knowledge base (google doc) to work properly (i did try altering the relevance scores etc)

  2. Lower content window and costs somehow?

Sorry if I’ve missed something obvious here, I am not a coder.

Many thanks

3 Likes

You were using a Google Doc as your knowledge source?

Hey Minchor, this is an excellent question from someone who is clearly a detailed oriented user.

So first let’s discuss the difference between putting information inside the prompt itself versus adding it as a knowledge base.

Prompt
The prompt is the actual instructions that the model reads. The prompt is the best piece of real estate with which to control and influence the model’s outputs. That’s because every time the model runs it will look at the entire prompt. It will do this every time it generates a new output. There are limits to how much you can put into the prompt, but the newer models allow for over 120k tokens, which is quite a lot. The more tokens you are asking the model to read for each run, the costlier the request becomes though because you’re asking the model to do a lot of work.

Knowledge Base
A Knowledge Base works differently. This takes a large document (hundreds of pages long or millions of tokens long) and indexes it into a series of easily queryable chunks. It organizes it with a special indexing system so that it can always find relevant chunks. So if a user asks about “food”, it can grab a chunk that’s all about sandwiches but doesn’t mention “food directly”. Then, whenever there is a request, the model will grab 1000 tokens worth of the most relevant stuff from the knowledge base and inject it directly into the prompt as additional context. These 1000 tokens from the knowledge will then be read as thoroughly as if they are part of the prompt originally. You can also have it inject 2000 tokens worth. Or 3000 tokens. Or 4000. You can set this number in the Advanced Settings under “Tokens available for knowledge base”.

Prompt vs. Knowledge Base
The advantage of using a knowledge base versus always cramming everything into a prompt is because the Knowledge Base lets you add 10 million tokens of knowledge and only use the relevant pieces when it’s relevant to the request. This saves time, save money, and makes for better requests because the prompt isn’t bogged down with extraneous information during every request.

Costs
To answer your final question about cost, I encourage you to look at OpenAI’s token pricing chart. The models are fairly cheap (fractions of cents) for most requests which have normal prompts of 100-1,000 tokens. But if you make massive requests with 100k+ tokens in each one, they quickly add up.

I hope that explains this a bit.

3 Likes

Great explanation to see how AI/LLMs are ticking :nerd_face:

1 Like

Ok thanks @mikegioia for this.

I’ve shifted 98% of the context now into knowledge base but the output is still hallucinating stuff and not pulling from the knowledge base (despite me setting relevance threshold to 0 and allowing 21000 tokens from Knowledge Base).

Is there something specific I can say in the initial Role Window that can direct the model to go to the knowledge base? I’ve tried repeatedly telling it in there to only use the Knowledge Base for replies, but should I refer to this in a specific way to get the desired outcome?

Thanks.