I’m looking for any tips and best practices anyone may have for the editing of chunks in a Pickaxe knowledge base.
Some specific questions:
– Is it the case that each chunk should ideally contain a complete, standalone set of information? How important is it to re-organize data that is “orphaned” from related data in an adjacent chunk?
– What is the maximum size you can have in a single chunk?
– If you edit chunks in a file, how do you deal with future major revisions of that file?
– What other tips will help optimize performance?
If you’re editing a chunk, you should have a very good reason. I usually edit my chunks after some testing and finding that key chunks are not being recalled when appropriate. In these cases, I will add some keywords to the chunks (almost like tags) that will help it get selected.
Another case when I edit chunks is when information is missing between chunks.
YOu don’t necessarily need each chunk to be a complete thought all the time. LLMs are very, very good at understanding text out of context. They have seen more words and more completions than either you or I can fathom.