Tips for Quiz Generation from a Knowledge Base

I’m creating a chatbot to chat with an historical figure – in this case the 16th century Anabaptist leader, Menno Simons. I’ve loaded many of his writings into the knowledge base.
I discovered what seems like a pretty good way to add a quiz generation feature so people can easily test their knowledge of what the chatbot knows.
In the Prompt - Introductions section, I have two icebreaker questions, but I added a third one as a request for a quiz: “Quiz me about aspects of your life and theology.” (That’s the user telling Menno Simons to give them a quiz.)
In the “Configure - Training Dialogue” section, I created a User/Bot Response pair where the User Input was “Quiz me about aspects of your life and theology” and the “Bot Response” was as follows:

Response: 
Step 1: Ask the user whether they want questions about events in your life, or about your theology? Wait for a response.
Step 2: Based on the user's response, present a question. Number each question sequentially.
-- Example, if the user wants events: [#]. In what year did I become a priest?
-- Example, if the user wants theology: [#}. What did I believe regarding Holy Communion?
Step 3: Wait for the user to respond. 
Step 4: Comment briefly and concisely on how accurate the response was. Cite the knowledge file, but do not include further details that may give away the answer. Immediately present the next question.
[RETURN TO STEP 2 and REPEAT]

These instructions would probably work in the main prompt as well, but in my case, my main prompt is loaded with other instructions, and my bot seems to do a nice job following these instructions and examples in the Training Dialogue. It could of course be adapted to suit your use case.
One adaptation is to stipulate that the questions should be presented in multiple-choice format (e.g., A, B, C). I implemented a multiple-choice quiz today, but initially discovered that the correct answer was nearly always the same letter. I added: “After generating the answers, shuffle or reorder the answer choices randomly. Check the location of the correct answer of your previous question, and place the correct answer of the current question at a different location.”
Even that didn’t achieve much randomness. Finally, this further addition achieved a good level of randomness:

Example: 
-- If the previous correct answer was A, place this correct answer at B or C.  
-- If the previous correct answer was B, place this correct answer at A or C.   
-- If the previous correct answer was C, place this correct answer at A or B.

This isn’t truly random, of course, because following it strictly means that the same letter would never be used for the correct answer twice in a row. Anyone have a truly random way to do this?

1 Like

Hi Gene, I am not sure if Pickaxe can do this, but maybe let Pickaxe count the letters in the answers it generates and have the Answers be sorted by the number of letters the answer has. This way the shortest answer is always on top, but might not be the right answer … I like your Idea of hawing a Quiz. Let me know if it worked :wink: Best regards Andreas