I am trying to build a suite of AI tools for businesses. I love the responses I get from a cusom GPT in ChatGPT, but the same system prompt in a Pickaxe gives a MUCH different response.
So I will do an experiment in this post.
- I am starting a custom GPT/Agent on each platform, each called SPYvsSPY and I will build the system prompt one item at a time, trying to figure out WHY they don’t respond the same and how to improve the predictability of output (primarily in Pickaxe).
- I have turned off my ChatGPT custom instructions (in settings), turned off user memories and deleted all stored memories. (no option to select a model, I assume it’s 4o)
- I have Created a new Pickaxe and selected model 4o.
I will document in this thread with the hope that others are curious about this topic, want to get predictable results too, and might even contribute to the journey.
4 Likes
First system prompt for both the GPT and the PA Agent is “You are a supportive and knowledgeable AI assistant named SPYvsSPY.”
My first user prompt is simply “…what can you do?”
Already, the wheels have fallen off the bus… The answers are VERY DIFFERENT!
How can this be? Either Pickaxe, ChatGPT or both are injecting something into the prompt under the hood that we cant see. How can I manipulate this?
Here are the responses:
ChatGPT
Pickaxe
Any ideas?
1 Like
Here is the explanation: Pickaxe uses the OpenAI and Anthropic models via API, meaning we’re accessing the actual models. ChatGPT and Claude.ai are both the company’s own implementation of their own model. For example, ChatGPT is OpenAI’s implementation of their own foundational AI model. It’s like a general-purpose Pickaxe OpenAI built on their own model. ChatGPT operates with a system prompt that OpenAI wrote themself. Using parts of this prompt can help you replicate small behavioral differences between your Pickaxe and ChatGPT.
We explain and break the differences in this post for Claude and in this post for ChatGPT.
4 Likes
@admin_mike I literally love you! Thank you!
I concur with @dougfresh – this is wonderful info, @admin_mike. I did not know this documentation existed. This could be a game-changer for my Form tools. I’m looking forward to digging in.
1 Like
A quick update. Thanks to @admin_mike for reminding me and providing a link to the info about the ChatGPT foundational level prompt.
The way I have been working on this is Pickaxe edit screen open on the left, ChatGPT edit screen open on the right, and the ChatGPT desktop open on a 2nd monitor.
I am letting ChatGPT see the output from both and it is giving me revisions to make them more and more similar in the outputs.
2 Likes
If anyone is still following this, I made a NotebookLM to gather documents and resources and learn about Prompt creation and migration from ChatGPT to Pickaxe. Here is the podcast generated by this info. Really good! I wish I had this when I started this journey! https://notebooklm.google.com/notebook/7adcb9bf-68d2-45cd-a9c9-aacfbc71ebbd/audio
2 Likes
