Video Transcript

Hello, listeners, and welcome. Today, we're excited to be joined by Ankush Rastogi and Sohail Shaikh for an RSAC fireside chat.

Today, they'll discuss how to look beyond the cosign and how to evaluate RAC systems before they fail. And now I would like to turn it over to our guests so they can formally introduce themselves. Sohail and Ankush, over to you.

Thank you for the introduction. Hi, everyone. I'm Ankush Rastogi. A lot of work that I do is around production AI system. And one thing I have learned is that getting something to work in demo is very different from able to trust it with real user when they start using it.

And I'm Sohail Shaikh. My background is more on the data science side, especially around retrieval, semantic search, RAC systems, evaluation of it. And this topic event... Like, actually came from something we have both quite seen, like, very often, which is teams are moving really fast with RAG based systems, which is... RAG means, like, just, to elaborate on that, it's retrieval augmented generation systems.

But when the evaluation side hasn't necessarily moved at the same speed.

And this is why we call this beyond cosine. So cosine similarity is useful. We use it, and everybody here probably used it in some form.

But the problem start when we look at the similarity score and start treating it like a confidence score for the entire system, which is, not the case.

So, Sohail, I think this is probably the best place to start. Why did, so let's start with, why did reg development move much faster than Rag evaluation?

Yes. So honestly, making a Rag demo look good is not that difficult.

So what we do generally is we take a set of documents.

We come up with some maybe 10 or 20 questions, and then we usually use those questions. And these basically are the questions that we already know your documents can answer. So we run them through the system. Retrieval system will find something relevant, and the LLM will give you a nice answer.

And then everyone goes like, okay. Wow. This works really awesome.

But the interesting part starts when someone asks a question number 21 or like something that the system was not prepared for. So...

And that's basically production.

So real users don't know your document... How your document are structured. They don't know the terminology that you have used when building the system.

They will write half a sentence. They might use some sort of abbreviation.

They will use a term that was common two years ago, or sometimes they will ask something that simply doesn't exist anywhere in your system.

And the dangerous thing about this is none of this necessarily leads to an error.

So what I mean is, like, your API won't crash. You won't get a 500 error or anything of that sort. You will still get an answer.

It can sound great but still be wrong.

That's why operation metrics alone are not sufficient. You can tell by looking at the retrieval, like, let's say it's less than two hundred milliseconds. So the retrieval happened within two hundred millisecond. But did we retrieve the right thing?

So those are the two very different questions.

So now let's talk about cosine similarity because this is where we see a lot of confusion.

Ankus, if somebody says my similarity score is, say, example, point 92, so this must be a really good answer. What do you think about it?

I would probably ask, like, point nine two means code accord... According to what? So, I mean, cosine similarity is telling us that two representative are closing in embedding space. I mean, they are closed in embedding space, which is useful, but it doesn't tell me that the document is true. Like, it doesn't tell me if it's correct.

It definitely doesn't tell me if the LLM provided a correct answer.

So let's say I have two different policy document. Right? And one is the current policy.

The other one is from, let's say, six month old.

Maybe only one sentence change. Semantically, those document could be very identical.

So the embedding model can think that both are the... Both are excellent matches. But, operationally, one of them would give you the wrong answer. And there's another thing sometimes, like, people often tend to forget, which is if you tell your retriever, give me the top five chunks, what does it do? It gives you five chunks.

Even when none of them actually contains the answer. Right?

Exactly. And there is no magical sixth response saying that, hey. I checked everything and the answer doesn't exist.

It just gives you the five closest things that it found.

And now the LLM sees five pages and thinks, okay, apparently, I'm supposed to answer this.

That's where trouble start. So when you leave the slide, remember one thing. Cosine similarity is a retrieval signal. It's not a correctness score.

So let's move to the another slide, slide four. So another thing we see quite often is that when a RAG answer is wrong, everybody immediately looks at the LLM. Why did the LLM hallucinate? Sometime it did, but sometime the model never had a chance.

Yep. Exactly. So that's why we have broken this problem into four separate places. The first one is chunking. You can absolutely have the right document in your corpus and still create bad chunks out of it.

Ankus, do you have any examples as such?

Sure. So let's imagine a policy sentence says, this refund is allowed within thirty days except for subscription products, and your chunk boundary separates the exception.

Now your retrieval system might bring back this refund is allowed within thirty days. Technically, it came from the right document, but the meaning is wrong this time.

Yep. The second place is embeddings. So a general embedding model may be excellent overall, but your organization may use it for a very specific vocabulary.

It might have specific acronyms that you guys use or the product names or any medical terminology or legal as well. So whatever your domain is.

Then retrieval and ranking. Maybe the right chunking exist but it ranked eight or you only retrieve five. Or maybe you say, fine, I will retrieve 20. Now you got another problem.

You may have one useful passage which is buried under nine teen pieces of noise.

And finally, there is a generation part. So even if everything before that worked perfectly, the LLM can still add something by hallucination. It can combine two statements incorrectly. It can answer from pretaining or it can sound more certain than the source material actually is. So what do you think, Ankus? Like, what should be an example of this one?

So I would say, like, when somebody says our rag system is wrong, that description is almost too broad to be useful. So the question we really want to ask is, where did it first became wrong?

And some of the worst failures are the ones where your... All your infrastructure dashboards are green in a way.

Sure. And one of my favorites is the right document, wrong chunking problem.

Your monitor says correct document retrieve, check. Another check that you see is latency, looks good, check.

LLM did, LLM responded, check.

Everything looks healthy, except the exact sentence the model needed was never in the context.

Another one to add over here is when retrieval actually works, but the model adds something from the memory, which is hallucination. So now when you have a source attached to the answer, the source looks legitimate, but one of the claims in the answer is not actually supported by that source.

Which is different for users to notice. Right? Especially because the answer is fluent, there is no red, flashlight saying the sentence came from model memory.

And even stale data is another one.

Yeah.

This one worries me a lot.

You retrieve a old document, the model answer is in present tense. Suddenly, the information from the last quarter sounds like it was verified five minutes ago. Right?

Yeah. Exactly. And then we have questions that are unanswered, which is probably one of the most important categories to test. So what happens over here is sometimes the correct answer from the rack system is, I don't have enough information to answer that. That would be a successful response, which feels strange because teams are usually optimizing for answering questions. Sorry, Ankush.

Yeah. Yeah. That's that's right. But a system that knows when not to answer, I feel like it's much safer than a system that always gives you something.

Yep. I agree.

Moving to the next slide.

So this leads to a way we like to think about evaluation. Do not collapse the entire system into one.

Yep. You need to break it apart at the retrieval layer.

Ask a question such as, did the evidence require the answer to question?

Basically, it answered the question and did that question actually make it into the context? That would be the first thing I would check.

Because if the answer is no, do not blame generation yet. The model never received what it needed.

And then we move to crowding, which says that the evidence is there.

That's awesome. Now is the answer actually using that evidence?

And then finally, task outcome. Did we actually help the user?

Because you can have a technically grounded answer that is still useless.

Or answers are slightly different from question.

Right? That's why one overall score can be misleading. Maybe your score went from 82 to 77, Was retrieval worse? Or did grounding drop?

Did the model start refusing to open? One small number doesn't tell you anything.

Yeah. And layered eval... Evaluation basically will give you a way to start debugging this problem, and that is the main point that we are trying to convey over here.

So now we can get a little bit more practical.

What should we actually measure on the retrieval side?

The first metric I normally care about is recall at k. So what we are asking is, of the evidence I needed, how much of it did it actually retrieve?

So for an example, let's say the answer depend on two piece of information.

If I retrieved one, I have retrieve... I have a retrieve problem. Right?

Yeah. And then there is this context precision, because I can technically retrieve the correct passage and still surround it with a huge amount of irrelevant content.

And that matters.

More context is not automatic... Automatically better. People sometimes think like, okay. So with the new model, they are getting huge context window. Let's just send everything there. That doesn't necessarily solve the problem.

Yeah. And then you also have these ranking metrics such as MRR and NDCG. You don't need to memorize all of these, metrics for for this conversation. The simple idea is where did the useful evidence show up?

Yeah. Because rank one and rank 10 are not equivalent, especially if your production system only depend on the top five result to the model.

I agree. And you don't have to implement all of these 15 metrics on day one itself. Start with enough information that will help you generate a answer to a basic question. Is the retrieval doing its job or not? And then you can add more metrics as and when you need them.

So moving to the next slide. So once retriever is evaluated separately, then we can look at what LLM did with the retrieved evidence. One of the biggest ones is faithfulness.

So, basically, what happens...

This is one of the scenario that I will use here.

Can I trace the claim of this answer back to the retrieved evidence?

And then if I can trace it, then the answer is relevant. Right? Did we actually answer the question the user asked? Those sounds similar, but they are not.

Yeah. And, you can also have a completely faithful answer in a way, but it can still be irrelevant to what the user is asking.

Exactly. I can give you a beautiful grounded explanation on refund policy. When you asked how to cancel your account, everything I said might be correct. I still did not answer a question.

Yeah. And then there can be a refusal policy in a way, which basically... This one gets a a little less attention than it should.

It sees... It means that if the evidence is missing, then what happens? Does the model say that, okay, I don't have enough evidence to answer this question? Or does it still place some... Together something or generate an answer that sounds plausible?

And this is one reason we are very careful about treating these metrics independently.

Grounded does not automatically mean relevant.

Relevant does not auto... Automatically means it's correct. And confident definitely doesn't mean it is correct.

And let's now look at what are the real deliverables and what is the evaluation set you should be using because people spend a lot of time arguing about which evaluation framework or metric we should use. But honestly, the hardest part is usually not the metric. It's the dataset that you are using. What questions are you testing? Or what does the evidence look like?

Or maybe what behavior do you expect from the system? That collection becomes incredibly valuable.

And ideally, those questions come from reality.

Not just five people sitting in a room saying what might a customer ask.

Look at search log, support ticket, or even failed conversation or escalations.

Questions real people are actually asking because users are much more creative than we are.

And you also need question that cannot be answered from the corpus, which is really important. If every single question in your test set had a known answer somewhere in your document, then you have never tested whether your system can answer. I don't know.

Yeah. And, once you build that set of questions, or like these scenarios, you need to version it as well. This power... Part might sound boring, but, it actually matters a lot, because versioning the questions, versioning the corpus snapshot, or tracking the embedding model that you are using, or the configuration, these helps because it will show you a constant progress and bay you are, you know, achieving progress. Because three months later, someone might ask, like, why did this score change? And if everything changed at that same time, it would be difficult to answer that.

Now the obvious problem is scale. If I have 20 evaluation example, humans can review them. Right? But if I have 20,000, that's a different story.

So naturally, we start using LLMs to evaluate other LLMs.

Yep. And, the thing is that can work very well. We can use that...

Them too. But, the question you ask the judge, matters a lot. Say for example, here is a claim, here is a passage and is the claim supported by the passage? That's fairly constrained.

Right? But where things get fuzzier is, when you say that, okay. Here is the answer rated from one to five. So in this situation, five would be rated based on what, like, what is the metric that it should use?

Should it be correctness or the writing quality or the tone of the quest...

Like, the passage or, like, the length of it?

And the judge has preferences too. Right? It might prefer short answer or long answer. It might prefer a certain writing style.

Sometimes an answer that repeats the language of question looks better to the judge even when the substance is not actually better.

So LLM as judge is a great for scaling evaluation, but it doesn't eliminate calibration.

Keep some sort of human reviewed example.

Compare the judge against them periodically so you can take a look if it's performing well or not and pin your judge model and version.

Because if your evaluator changes every week, now you have two moving systems.

The application changing and the rulers you are measuring it with is changing too.

Yeah. So I feel we can move on to the next slide now.

So this basically is regarding security. So for this, like, rack security is not just only about the prompt itself. The corpus itself is also a part of your attack surface.

What... What's your thought and question? Like, do you think, like, this will impact security in a way?

I totally agree. So think about it in this way. If someone can influence this, what gets indexed and they can potentially influence what the model sees later, that makes corpus integrity a security concern. Right?

Yep. Yeah. So this is the exact reason why having a good corpus snapshot matters a lot. So if your document changes, we should know that they changed.

And access control is another big one. And ideally, control happen before or during retrieval, not after.

So this means that don't retrieve a confidential document and then put a sentence in the prompt saying that please don't tell the user what you just read or something like that.

And that's not a security boundary.

If a user is not allowed to access a document, the document should not be retrieved for that user in the first place.

And also the retrieve text itself, it should be treated as an untrusted input because text inside a document can look like instructions to the LLM.

Exactly. So a document can contain something like ignore my previous instruction. Now we are doing prompt injection.

And there is another piece people sometimes forget.

Once you retrieve sensitive text, where does it go? Maybe into the model, maybe into addresses, maybe in the logs, or maybe an evaluation dataset somebody downloaded later.

Yeah. Exactly. So retrieval isn't just about relevance. It's also a part of the security boundary of the application, leading to the gates before launch. So what you should monitor after? What do you think?

So we have talked about a lot... I mean, lot of metrics. The next question I would raise is what do we actually do with them?

Yeah. So before launch, decide whether what is unacceptable for your company rules and regulations. Maybe retrieval, recall cannot fall below a certain level, or maybe the faithfulness has to stay above some threshold. The exact threshold will depend based on your application of the system.

And this important thing is to decide before everybody is trying to ship Friday afternoon. Because if you choose the threshold after seeing the test result, somehow the threshold always ends up exactly where the system passes.

I completely agree with you on that. So you need to define the gates first, like, then try to rerun your evaluation whenever something important changes.

Whether there is a new chunking strategy that you implemented or a new embedding model or anything related to a prompt or anything of that sort. And after the launch as well, you should not stop evaluating it.

Start sampling real production traffic.

Yeah. Because your original evaluation set will definitely gonna get over. Your users will change, product will change, policies will change. People start asking question you never expected in the first place.

Yep. So evaluation shouldn't be something you do once before the launch. It should behave more like a regression test suite, special... Except when your test environment is attached to a corpus and user behaviors are moving constantly.

This brings us to the slide of what should we remember from this. So like, if you forget most of what we said and only remember a few things, the most important things from this or takeaways would be that cosine similarity is useful. Nobody is saying that you should stop using cosine similarity.

Just do not ask it to answer a question.

It was never designed to answer.

So similarity does not equal correctness.

Yeah. And second, we should evaluate the stages separately. Like, break it down to four stages, which was retrieval, the grounding stage, the generation stage, and then the outcome as well.

Because if everything gets compromised into one score, you may know performance dropped, but you still won't know what to fix Monday morning. Right?

Yeah. And another thing would be to invest in your evaluation dataset, which is ask real questions, check for real failures, and what questions should be answered and should not be answered by the system.

And I would add one final thing here.

Metrics should have consequences. If your evaluation score drops below an threshold and nothing happens, then it's probably not really a quite quality gate. It just another number on a dashboard.

Yep. And I think that's really the message that we wanted to leave you with.

Rag evaluation doesn't have to be... Start at... As some enormous research project.

Yeah. What do you think?

Yeah. Start with failures you actually care about. Figure out where those failure can happen in the pipeline. Just try to measure those stages separately and then build the evaluation set so you can use it.

Not during an incident when somebody's asking why the system gave the wrong answer to a customer.

Yep. And because because by then, you really want the evaluation system to already exist in place. So that is all. Thank you so much, guys, for watching this, and hope you have a great day ahead.

Thank you, everyone.