Today we gave builders more visibility into how their Pickaxe is working. We released Message Insights. This is a small button that appears on all generated responses within the Builderās testing chat. You can click it and see:
Token Count: A quick overview of the tokens used
Knowledge Chunks: The specific portions of the Knowledge Base it referenced for the generation.
Actions Used: A list of which actions were used
Hereās a brief overview of the new feature.
.
. MESSAGE INSIGHTS BUTTON
WHY ITāS IMPORTANT
Many users upload tons of documents into the Knowledge Base without an understanding of how the knowledge is being processed and used in their Pickaxe. This is true across pretty much all AI products with a document upload portion. Most products donāt explain. They just saw āUpload your documents!ā We want to provide insight into the process so users can build Pickaxes more effectively.
Importantly, your Pickaxe does not look at the entire Knowledge Base all the time. We use a state-of-the-art system called RAG (Retrieval Augmented Generation) where your Pickaxe grabs the most relevant parts of the Knowledge Base, reads those chunks, and then generates its answer.
These new insights will help users understand how their tools are working. Weāll be releasing multiple more features in this department soon to help you take action based on what you see in the Message Insights
This is a really useful addition. I wasted so much time in the past by uploading lots of documents only to have to remove them because of being unable to see which were being referenced.
Iām sure this will be a useful feature. All it has highlighted for me so far is that the chunks taken from the KB to answer questions have no bearing or relevance to the question being asked of my assistant. It seems completely random and is therefore providing incorrect information (my information takes the form of a list of actions for a specific recipe) the assistant just pulls random steps from all recipes. I donāt get it.
That means you have too much information in the Knowledge Base probably. Less is more.
Many users assume the best approach is to dump all the information into the Knowledge Base, when in reality this is overwhelming to the LLM, the same way it would be for a human.
Curating your knowledge a bit is helpful.
Now weāve also released functionality so that Pro users can delete and edit individual chunks.
Thanks for that information. Can you elaborate on what effect dleting and editing individual chunks might have and best prectices to make the most of this feature?
