Is there a way to get a response of unconstrained length from a pickaxe? In viewing the Output Length under Token Length, the Max Output Length seems to max at 4096. I’m getting responses cutting off. Isn’t there ways to increase this?
In searching the other older posts, I am not finding a resolution to this.
This limit is the underlying limit of the AI models themselves. So the OpenAI models themselves cannot produce outputs longer than 4096 tokens.
Okay…if using an OpenAI model then, is there a best practice to baking half of my pie at a time? When I’ve done this directly in ChatGPT, prompting with “please continue” seems to work fine, but for some reason within Pickaxe the ai seems to not retain the context of the previous (first part) of the output.
You probably need to increase the setting called “memory buffer” found under the Configure tab in the builder. This is how much of the conversation that the AI remembers. When you’re dealing with high token counts you need increase the default setting for this number.
thanks, Mike. Appreciate the help.
In that the memory buffer has what looks to be a token counter, I’m guessing whatever is in the memory buffer counts against your output tokens? Or am I conflating things?
It counts against your input tokens. And not all of it is used every time.
For example, let’s say you set memory buffer to be 1,000 tokens.
User Message #1 is 10 tokens.
The AI looks at the conversation which is 10 tokens long.
Then AI Response #1 is 10 tokens.
Now User Message #2 is 10 tokens.
Now the AI looks at the conversation which is 30 tokens long.
Then AI response #2 is 10 tokens.
Etc…
Now let’s say the the conversation is eventually 1,010 tokens long, the AI will stop including User Message #1 in the conversation history.