Message Insights: Look at Knowledge Base Citations 📣

[admin_mike]

Today we gave builders more visibility into how their Pickaxe is working. We released Message Insights. This is a small button that appears on all generated responses within the Builder’s testing chat. You can click it and see:

• Token Count: A quick overview of the tokens used
• Knowledge Chunks: The specific portions of the Knowledge Base it referenced for the generation.
• Actions Used: A list of which actions were used

Here’s a brief overview of the new feature.
.
.
MESSAGE INSIGHTS BUTTON

.
.
MESSAGE INISGHTS MODULE

WHY IT’S IMPORTANT
Many users upload tons of documents into the Knowledge Base without an understanding of how the knowledge is being processed and used in their Pickaxe. This is true across pretty much all AI products with a document upload portion. Most products don’t explain. They just saw “Upload your documents!” We want to provide insight into the process so users can build Pickaxes more effectively.

Importantly, your Pickaxe does not look at the entire Knowledge Base all the time. We use a state-of-the-art system called RAG (Retrieval Augmented Generation) where your Pickaxe grabs the most relevant parts of the Knowledge Base, reads those chunks, and then generates its answer.

These new insights will help users understand how their tools are working. We’ll be releasing multiple more features in this department soon to help you take action based on what you see in the Message Insights

[ab2308]

@admin_mike thank you! That’s very helpful.

[derekj1951]

This is a really useful addition. I wasted so much time in the past by uploading lots of documents only to have to remove them because of being unable to see which were being referenced.

[admin_mike]

Glad you find it useful! Soon you will be able to see a table of all your documents and the associated ‘chunks’ in the Learn tab. Coming soon!

[quangloclam]

Thank you, Mike. It is helpful.

[terryjames]

I’m sure this will be a useful feature. All it has highlighted for me so far is that the chunks taken from the KB to answer questions have no bearing or relevance to the question being asked of my assistant. It seems completely random and is therefore providing incorrect information (my information takes the form of a list of actions for a specific recipe) the assistant just pulls random steps from all recipes. I don’t get it.

[admin_mike]

That means you have too much information in the Knowledge Base probably. Less is more.

Many users assume the best approach is to dump all the information into the Knowledge Base, when in reality this is overwhelming to the LLM, the same way it would be for a human.

Curating your knowledge a bit is helpful.

Now we’ve also released functionality so that Pro users can delete and edit individual chunks.

[terryjames]

Thanks for that information. Can you elaborate on what effect dleting and editing individual chunks might have and best prectices to make the most of this feature?

[zerodot]

This is a great feature. I’d also request that users of pickaxe be able to view the document/chunk preview from where the question has been answered. I have uploaded my company policy but most users still use notebooklm because it can show the exact source chunk.

The feature is already there for the studio owner during testing but not to the end user. I’ll request you to make it a global setting.

[admin_mike]

End-user citations will come eventually. But first, we want builders (and us) to get a grip over the new Knowledge Base for a while before we start exposing it to end-users. It’s definitely where the product is headed.