-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Copilot functionality #258
Comments
Enhancing Search Capabilities with AI Copilot functionality. Main idea: Key Phases:
Example Workflow:
Possible Benefits: The copilot is doing exactly what a copilot is intended to do to copilot the search with you suggesting possible searches that would solve the search problem. Enhanced Accuracy: By iteratively refining the search process, Perplexica Copilot should ensure higher accuracy and relevance of search results. There is research showing that LLM are not currently able to truly self refine, but we try to push the limits as hard as possible. Large Language Models Cannot Self-Correct Reasoning Yet: https://arxiv.org/abs/2310.01798 Conclusion: |
|
|
I tried to make some modifications to the file webSearchAgent. By doing this I approached, sometimes exceeded the performance of copilot in Perplexity. However I used the local model of Microsoft Phi-3 Medium. I made some blind tests asking GPT4 to be the judge of performance. This new prompt forced the answer to be more in depth and more academic. Also I change the prompt in the web agent Sugestion generation agent, using self-refine approach. The result is more professional suggestions, while the vanilla goes towards mixed bag of theoretical and practical. Also I think for the pro suggestion the number of 5 is low, it has to be 12 as those are pretty narrow. Bear in mind that there are limitations of the search engine and the results regarding current events are somehow lacking sometimes. You need to update the file webSearchAgent in Perplexica/src/agents
|
What are effective strategies for managing stress? Perplexica with new prompt for more advanced response. Effective strategies for managing stress involve a variety of mental, emotional, and behavioral techniques that can help reduce its negative impacts on well-being. Here's an overview:
Remember, finding the best techniques for managing stress may take some experimenting and practice, but it's essential to keep looking for tools that will help you manage life's inevitable ups and downs in a healthy way [15]. Citations: Related
|
The results rated by gpt 4o. Perplexica (Phi 3 Medium with ollama) outperforms overall both vanilla Perplexity and google gemini.
|
This seems like a very well documented attempt @Zirgite, thank you for all of your answers i really appreciate your thoroughness. However, when i tried the example you provided it triggers an exceptions in the cosine-similarity calculation function, see below. I suspect this could either be improper handling of async data calls, but i have not found the fix for my issue yet so i am unable to test your findings. I am also using Ollama btw with llama3:8b-instruct-q8.
If the results of your tests are indeed valid, this would be a great improvement for a user looking for detailed information on a topic or running a more complex query, however there would be a need to abstract this away since from my understanding copilot mode needs to be a mode for Perplexica to function that is Agent agnostic. |
I cannot upload the files. So the easy thing is just to change the file file webSearchAgent.ts in Perplexica/src/agents |
Also further in the line i made a prompt for the suggestionGeneratorAgent. This tries to make more professional and targetted suggestions for next search. Those suggestions dig deeper into the topic, and I wish to have at least 8 to 12, as they are narrow and not broad like the the vanilla.
|
Another test with phi-3 model comparing Perplexica with deeper implementations copilot and Perplexity showed, that the perplexica is diving deeper in the subject compared to copilot perplexity, also the selected links are more into big dive into the topic. While Perplexity copilot remains more easier to grasp. Test 1: Perplexica with 'copilot' self define and self-refine 6CO2 + 12H2O + Light Energy → C6H12O6 + 6O2 + 6H2O The process involves two main stages: light-dependent reactions and light-independent reactions (Calvin cycle). Here's a step-by-step overview of how photosynthesis works in plants:
In summary, photosynthesis is an essential process for plants and other organisms capable of converting light energy into chemical energy. It involves two main stages: the light-dependent reactions generating ATP and NADPH while producing oxygen as a byproduct; and the Calvin cycle using those molecules to fix carbon dioxide into sugar, which can then be used for growth or stored (context [3][8][14]). Citations: Related Test 2: Perplexity with copilot How Photosynthesis Works in PlantsPhotosynthesis is the process by which plants convert light energy from the sun into chemical energy in the form of glucose. This process is essential for plant growth and survival, as well as for supporting many other lifeforms on Earth. Here's a detailed overview of how photosynthesis works in plants: The Light Reactions
The Calvin Cycle Oxygen Release Energy Storage In summary, photosynthesis in plants is a complex process that involves the absorption of light energy, the conversion of that energy into chemical energy in the form of ATP and NADPH, the fixation of carbon dioxide into organic compounds, and the release of oxygen as a byproduct. This process is essential for the survival and growth of plants, as well as for supporting the broader ecosystem on Earth.[1][2][3][4][5] Citations: What role do chloroplasts play in photosynthesis gpt4o arbiter:
Test 2:
Evaluation of Sources:
Test 2:
Quality of Related Searches:
Test 2:
Conclusion:
In summary, Test 1 Perplexica is more suitable for readers seeking a detailed, in-depth understanding of photosynthesis, while Test 2 Perplexity with copilot is ideal for those needing a clear and straightforward overview. Both answers are well-supported by reliable sources, but the depth and technical detail in Test 1 Perplexica provide a more comprehensive explanation. |
I see, i was particularily interested in the iterative search part, because although we sacrifice response time a model being able to self calibrate the query based on the responses and sources it finds seems like epitome of the Copilot mode. I was able to test some searches using our updated prompts and i could see that the answers provided more detail with the model offering more descriptive information about the things it found related to the query. This was not a very scientific test, i basically asked the initial web search agent and the one updated with the new prompts provided the same question and observed a difference in the depth of the information provided in the results. This is why i was initially asking if this feature was being actively developed, or if not what direction the project owner would like to take since things like the prompt engineering that you have showcased can go a long way in focusing the results of the model, but other steps would most likely need to be taken in order to achieve the functionality currently being described in the project readme. |
Thanks for the input. Right now it is toe to toe with Perplexity copilot. The perplexity focus more to be more usefriendly, why Perplexica with updated prompt makes pro searches and the results are for specialists and are less for the general audience looking for easy to understand results. |
I think I made a mistake yesterday as I think there is need to run the docker install again to intorduce the new agent into the docker. Please correct me if I am wrong. This is what I made today. The results are really good. When I asked to be simple it is simple when I ask to be complicated it is complicated and in more depth. |
The search seems to be performing better when using these new prompts, i know not very scientific but i have not had the time to properly test it like you have done. I have not had the problem of no references, but i did experience in the case of some more niche searches on topics that are less popular the references it gives for context are nonsensical and have nothing to do with the actual text, even if the generated response seems sound. This seems to be an issue where the model outputs a reference saying it found certain information in there, but the link it provides is completely unrelated so fact checking this information becomes more difficult. |
I update the prompt here: The idea is to follow the logic of Perplexica as it is. I made some wrong assumptions, the code above is just idea how to make a new function iterative search that is interesting for copilot functionality. Here instead to make an iterative search i make just bigger search looking with neighboring key words. const basicSearchRetrieverPrompt --- just formulated the search query const basicWebSearchResponsePrompt --- formulates the answer I made an evaluation prompt (also in the discussion) to compare with another AI (gpt 4o) as judge the performance between Perplexity and Perplexica. Some interesting finds:
ideas for later work: I may try to put the criteria in the prompt to evaluate: |
How are the sources for this version of the prompt for you, cause you previously stated that they were absent from the generated response. In my experience testing it today, it seems that it is able to give some ballpark references at the bottom of the generated text, but the sources listed at the top of the query, the indexed ones, are still way off the mark, and usually they direct to some websites or articles about rephrasing questions, or unrelated topics to the query. Have you had similar experiences with the sources stated at the top of the answer generated? |
This is my latest version.But before that Try different embedding as they make the model to do irrelevant searches. And when the searches are irrelevant I think the model starts to use its own knowledge.
|
I personally ended up reverting to a modified version of the regular prompt for more consistent source outputs. While testing your prompt i noticed that while the llm was context aware, in the sense that it was able to specify the correct reference in text form, whenever it came to introducing that reference as a link embedded in the text it usually failed and provided something totally unrelated. I am not sure what other tweaks you have done, particularly to the embeddings side i saw you metion, but that was the main issue for me with indexing the references and sorting their relevance, and often times getting back references that were nonsensical in the text. |
Debugging an ai + code is cream of the cream of difficulty. You never know where is the problem, in the prompt or in the hard code. I reversed back to the original prompt too to check but I get the same thing sometimes the references are perfectly aligned sometimes not. I went back to my last prompt and it is the same thing sometimes i get perfect reference alignment sometimes I don't. That means the AI prompt is the problem. |
Sometimes I need the vanilla suggestion that work like any other suggestions in Prplexity, Windows copilot, google etc. But sometimes I really want cutting suggestions into the topic. There is bug that is common with the normal prompt when going deeper into the topic the suggestions break down starting with level 5 of deep following of suggestions. #
|
Regarding the updated prompt. I made an update to the functionality of the prompt to incorporate the new features
|
Thank you very much for the excellent work and ideas on improving the copilot feature, I think you are both onto something indeed! I think the 2-steps self-define and self-refine process is part of the secret sauce of Perplexity's copilot feature, and the orchestration part is too. I would like to suggest a new prompting method called PLaG, which allows to improve significantly the orchestration of asynchronous tasks by representing them as a graph. This could maybe be the right tool we need to make the LLM define a step by step set of search requests. And since this method supports asynchronous, parallel tasks, it could potentially maybe be used to make the LLM generate parallel search requests, something that I think Perplexity is not yet able to do. Why would we want that? Because this would allow to implement simultaneous breadth-first and depth-first searches, so that a single copilot search would allow to not only dig deeper, but also cover a broader area of the topic. For example, we could ask the LLM to break down the topic into 3 main aspects (breadth-first search), and then do 3 define-refine searches for each in parallel (depth-first). In other words, since we would now be working with graphs, we are now converting the problem of searching for information on the web into a more tractable graph-search, and so we can now implement a variety of already existing algorithms. I think it's well established that the best strategy when whole graph size is unknown is to use a combination of breadth-first and depth-first, and there are a lot of different algorithms for that, such as Iterative Deepening A* (IDA*). Also I think the problem formulation matters here: should we view web search as a shortest path problem (ie, the user is looking for a specific answer that is somewhere in the graph of the whole web), or is it a coverage problem (ie, the user wants to learn more over a whole topic that is represented by a subgraph that we need to identify and extract)? I think these two different problem formulations are in fact co-existing, and is why some people will prefer how Perplexity works (answers with a more accessible language and with broader information at the expense of being less expert, less in-depth) versus why some others will prefer deepening models like @Zirgite suggest. And this preference may change even for the same user depending on the question they are searching. So I guess future models will have a module/prompt dedicated to first identify what is the user intent (broadly learning, or looking for a specific info/deepening their knowledge). For now, I think the easiest is to just offer the user the option to choose between broad or in-depth searchs. |
/home/perplexica/src/agents/webSearchAgent.ts I also play with the suggestionGeneratorAgent, which as in its file the following constant So I manually enter into the docker files to change those, but there was a bug recently that makes that difficult lately, or when the files are donwloaded with the command git clone https://github.com/ItzCrazyKns/Perplexica.git Here are different version of prompts, the results vary but are not very different so there is not a guaranteed improvement of the latest prompts as I tested it. |
lrq3000 |
Hey, I've been using this project for a while an creating different agents for random use cases. From the description of the copilot mode visiting the websites it finds as relevant results of splitting up a query into multiple rephrased variants and navigating those sites seems like a very good tool for searching through restaurant menus that usually come in many different forms.
From what I could gather this functionality is not currently implemented in the project, but I would like to assist in its development as it is a tool that would be very good for my use case. Is there any way I can contribute to this, or if there has not been any work done on this already could you provide any guidance as to where to start with the implementation?
The text was updated successfully, but these errors were encountered: