Here’s how i created two tools to do a/b testing in CrewAI

AI vs Google Search

It happened when we were discussing about incorrect responses by our AI crew.

The answer was available in the embeddings but somehow it wasn’t retrieved.

So, we decided to check it by Googling the query and we were able to grt the answer.

We saw some more instances wherein we realized Google search would give better results in some cases where RAG was failing.

So, I was tasked to create a Google Search tool as well for our AI agent.

For this, I had to simulate A/B testing scenario in the prompt.

Knowing that prompt is not the most reliable way to do this, I created a tool which was a function that returned head or tales by using the random function.

So the AI agent has to first call the tool to determine which functionality to use, either Google search or embeddings to search for the answer.

We tested it and I thought of two ways of using it going forward:

1. We’ll use both of them at the same time making sure we don’t miss out of any information.
2. We’ll use embeddings as the primary source and keep google search as a fallback. I’m more inclined towards this one.

We’re still testing this. So, let’s see how the results come up.

That’s it for today.

I’m creating a Fire-proof prompt engineering course including the secret tips that helped me generate amazing results at my job. If you’re interested, join the waitlist by clicking the link here: https://khushalkumar.kit.com/371a77e6e7