Andy Karuza: Absolutely. How'd you meet your co-founder?

(Olivier): While you are trying to enter a new industry and you are not from the industry, you kind of try to work with the IT tech companies before, and you quickly realize the problem that people have there. By complete serendipitous facts, the problem I had in that tech startup was solved by someone that was introduced to me not for that reason but just as a piece of advice. When I found him, and he was in need of a co-founder, I jumped directly from that ship where I was to that one where I'm co-founding the company with him.

Andy Karuza: That's great. So, can you explain more, maybe to the uninitiated, what RAG stands for?

(Olivier): Right, it's called Retrieval Augmented Generation. So, you know, there are many techniques in AI, like machine learning, you have text-to-text, text-to-speech, text-to-audio, text-to-video, and computer vision. These are all techniques using neurological network intelligence. RAG is the technique to retrieve information and generate an answer. So, usually, it's based on training data. You train your model. Your model will know from the database what to answer and will think through the answer to generate a response. So, the retrieval is the retrieve of data. Augmented is the thinking that goes before the generation, and generation is the response. It's what we call GenAI, but GenAI is a wider term because GenAI can generate images, graphics, videos. So, generation is the response. RAG is the technique to retrieve information, augment, and generate.

Andy Karuza: Very cool. Thank you for the analysis there. So, could you share maybe the latest advancements in RAG evaluation techniques since you guys specialize in this?

(Olivier): Well, you know, there are many things that me and my competitor are working on. I say me and my competitor because we are trying to make an industry out of it and a sub-segment of the AI tools and DevOps tools. We worked on what we call human-in-the-loop feedback. One of the crucial things when you develop RAGs or whatever GenAI application is to be able to make sure that you can detect errors. The error detection is at the level of development, so you can test drive your development, but it's also on the live aspect of your LLM in production. That means you need to be able to monitor input and monitor output. Then, you need to be able to, because you're using AI to judge AI, so the output of your AI can be unreliable as a judge, which also creates more errors on top of the errors, which is not so great. But with us, we have a 95% human agreement. That means we have a high level of agreement on our judge to a human judge. So, what we're trying to do is to bridge that gap between the human and the jury of AI so they can work hand in hand and improve each other. Recently, my partner created what we call a human-in-the-loop loop. You know, so it scrubs the input of a RAG, scrubs the output, and tries to match them up. It does that on certain criteria that you would decide, that could be error detection, that could be how well you are doing on something else like a particular criteria for your industry. But for the sake of the error detection example, it will scrub the inputs, scrub the outputs, and every time it finds an error or something inappropriate in the generation that doesn't correspond to the input retriever, it will flag it. It will tell the human, "Am I right or wrong? Do you think the same?" The human will say, "Yes, you're right," and it will say why, or "No, you're wrong," and it will say why. Then, the human will also learn the mistakes of the AI, and the AI will reinforce its knowledge of what's an error and what's not. That is also allowing to fix outputs and improve them because you can always kind of like fix your output based on the error and find the right outputs, export that into a data set, and reimport that in your model to retrain it, so you fine-tune it. Does that make sense?

Andy Karuza: Oh, it definitely does. Yeah, it makes a lot of sense to have the human in the loop as well. I mean, who's better for performance evaluation than a human that could tell the AI whether it's right or wrong? I feel like a lot of people just go out and create an algorithm and think it's supposed to kind of work on its own, or they'll have too much automation where you have the AI judging an AI, but you know, let's be honest here, you still need that human in the loop to really help train the models.

(Olivier): Yeah. Well, what we say is we don't want to strip away the human in the loop. We want to put the human in a skybox so it can look over the result. You know, in the end, if you think of the most, I would say, biggest fantasy into human illustration of AI human collaboration, it's Jarvis is nothing without Iron Man, and Iron Man is nothing without Jarvis. So, you know, we think of the relationship between AI and human the same way. So, here for evaluation, testing, and improvements, we are, you know, into that Jarvis-Iron Man relationship.

Andy Karuza: Yep. Yeah, I totally agree with you there. So, how else would you say that RAG is measurable? Aside from the error outputs, what are you guys using to evaluate these models?

(Olivier): So, you know, evaluating models is a very important part of someone who is entering the LLM RAG business or the AI business because in the end, you'll be judged on the quality of your products. That quality is totally uncontrollable if there is no not only guardrails but a real mechanism and platform that is always there to be with you in making sure that your AI is the best there is. What we think is number one, retrieval, so being able to judge how well your development retrieves data because without good retrieval, you can't have good regeneration. It's called Retriever Augmented Generation. So, number one is retriever. Number two is metrics. Metrics, you know, need to be custom because you never know exactly what you're trying to achieve. If you take generic GenAI generic metrics, you might solve just a part of your problem. In order to be able to do it, you need to be able to customize the metrics you're going to use to judge the performance of your application because they're always very specific. They're not generic. Generic is for OpenAI. Generic is for Anthropic. Generic is for Gemini. That's their—that's what they're trying to get. It's AGI. What we're trying to unlock at RankMetrics is ASI, Artificial Specific Intelligence. Now, the third thing is being able to compare in seamless views. So, A/B/C/D/E/F testing is crucial as well. So, being able to say, you know, I compare two vector databases in a seamless way. I compare three vector databases with three different metrics. These combinations of testing greatly improve the result of your development. So, A/B testing is crucial. Then, obviously, being able to test more than just text. If you have data in text, you know, you'll be able to kind of measure that. Being able to measure agentic if you have agents as well is important. Yeah, all that are the tools that we provide at runmetrics but which are very crucial to any sort of development.

Andy Karuza: So, why do you think this has been so challenging to measure performance and whatnot in the past compared to today? I feel like a lot of people just kind of, you know, just want to jump in and start creating their own models, but you guys are kind of there needs to be a foundation around it, right? Like, like you said, you have to have the pick and the shovel, whereas, you know, in the gold rush, a lot of people just kind of went out there and didn't have any of the tools. They just wanted to be there. But then, you know, businesses like yourself kind of pop up and start selling the necessary tools that people actually need.

(Olivier): I think greed, you know, greed is what brings people to do things without thinking through. When there is such a gold mine like AI is, and AI is a gold mine, you know, it's definitely a gold mine. It's a gold mine for companies, and it's a gold mine for humans, you know, because it's a great chance to be able to augment what you're doing and to go faster and better at what you're doing. That's what AI is going to bring to humans. But when you rush it, it can quickly fall down the ladder of quality, and instead of augmenting you, it's going to downgrade you, and that's where you don't want to fall. But when there's a gold rush, it's called rush. People are rushing the gold, you know. So, that's the problem. The greed comes with a rush, which I understand. Intellects rush, greed rush, and a certain part of ego as well. I would say that ego is also driving the AI industry, meaning that we all have big egos because we're manipulating things which are extremely powerful and concepts that go above humans even sometimes, you know, that you kind of feel like, well, if I can develop it, I can control it, why do I need another AI to help me do that? Why do I need this tool? But I'm sure that if you compare that to the gold rush, the most successful miners were the ones who had the best tools. So, I think we need to take a step back and the best intel—like where to actually dig—be humble about it and say to ourselves, tools are here to help us, like AI that we build is here to help others. Why don't we help each other with AI? Also, I think there's a tremendous amount of cheap open-source tools on the market which can be decent, I would say, but decent doesn't bring you quality. So, if you're trying to build with quality in mind, you need to pay for that quality because, I mean, this is something, you know, anything of value is for sale in this world. So, you can't reconsider again by saying, "I'm going to take a cheap tool or free tool and say, hey, this is enough for me. I got enough," because once it hits you, you are going to regret it, you know. Obviously, I'm trying to preach for my own parole, you know, and I'm all for it, but good output creates good results, and everybody's happy in the best world, you know.

Andy Karuza: So, there's really no universal RAG score, right? So, you guys really focus on like...

(Olivier): There are normative RAG scores like DP Eval, which is a free open-source criteria library and evaluation free open-source that we have integrated, for example, which is the most popular one. So, we have integrated the library into rank metrics because it's very popular, and people are using it. Now, we also build pre-made criteria thinking of whatever we can think of. So, we make people's lives easy, not to code them, not to create them. But there's going to be a border case where you can do so much with what we offer, what DPA offers, and what we're going to keep constantly providing because you might have a corner case where you want to be able to measure something extremely specific that no one has thought of. That's why you need custom criteria, good custom criteria. In fact, the more customizable your criteria is, the more accurate you can fine-tune and do better results.

Andy Karuza: And you guys really specialize in helping companies get that those specific evaluation criteria.

(Olivier): Absolutely, here for helping them do that. Great. And we're also here to get go quick fast evaluation with whatever is popular that we could find and integrate in the market in a smart way while also using the library in the back that is always available.

Andy Karuza: That's great to hear. So, we talked earlier about humans being in the loop, and you guys being a big proponent of that.

(14:22) How do you see the human in the loop evolving over the next couple of years as more automated methods come out for judging and evaluating these LLMs?

(Olivier): I think the human will always stay on the side of the loop. So, instead of saying HITL, you'll be like a human-sided loop, you know, HSL or something like that. I think the human will intervene less in fixing and fine-tuning models where the judge and jury will help you do that.

(14:52) So, platforms like us will help fix and fine-tune models for humans while also proposing a sandbox version of that before it gets to release and production. So, humans will be, again, you know, we go back to the Jarvis and Iron Man relationship; they will be in that constant loop together, but the human will be more the boss of the AI, and the AI will be enslaved in a way. I think that's where you want to be with AI. You don't want AI to take over humans. The human would be a super expert in what they're trying to achieve, decision-makers, and AI would be workers that would achieve for humans. So, that's where I see the relationship going, and wherever it's fixing code, developing code, testing code, you know, that's where we're going to go.

Andy Karuza: And do you really see a future where, maybe 10 years from now, a lot of jobs, you know, people just don't really need to work? They could just interface with AI for pretty much...

(Olivier): That is a very controversial question, and it is a question to reply to. Again, it has more, I think, legs to just like, you know, are we going to be using AI only, or are you going to be using humans? I think a combination of the two is going to happen. Now, are there certain jobs that might not have value anymore? Probably. Are they going to be places where it's just going to augment humans and live side by side? Absolutely.

(16:26) You know, so, I'd say if you were a young person and you come and try to build your own career in your world, try to be a thinker more than being an executed piece, and try to think emotionally more than using just your IQ because your instinct is going to drive the IQ of the AI, and in the end, that's what's going to create value to both. So, you know, it's a very strange world that comes around because, you know, I think structurally, socially, economically, things are going to change. You know, we're in the middle of that tariff battle, and, you know, is that really appropriate or not? I don't know because the world is going to change to automation a lot, you know, and it's a new world that is coming where it's going to be a lot more fair for people but also a lot more unfair as well, you know. It's a real dystopian world that we're going to go into.

Andy Karuza: You know, I'm really interested to see how it plays out. Absolutely. I mean, you know, I hope I'll be old enough to see it. Yep. In a positive way, too. So, I guess how do measurable insights improve this iteration speed and model fine-tuning for RAG pipelines?

(Olivier): Well, when you are able to make choices, when you are able to detect winning numbers and quality, you make the right decision to improve it. That's ultimately what makes your development better. If you are equipped with the right set of tools to make it happen, you will go faster, more accurate than any other company in the world. If you start using a good judge, the problem is how good your judge is. That's why one of the things that we are doing that others don't is to grade our own jury. So, we are able to grade, and if you have good results and you're able to measure your success, you'll be able to know what do I need to improve. I can give you a simple example and say I'm building a customer support chatbot, and that chatbot has a functionality which is giving away a discount in certain situations or not to clients, and I want to measure how much time it gives away the discount or not. If I create a metric that allows me to measure that, I'll be able to measure the time, how much time it gives away or not, and in which situation. So, I'll be able to fix it by improving, for example, the system prompt. So, if I improve the system prompt, I'll be able to improve it. Maybe I'm not using the right retrieval strategy. Maybe if I change that and I can measure again in an iterative part, I'll be able to go and improve, you know, my development. Then, if I'm in production, I think everything is fine. I'll be able to keep watching it and detect errors that I can fix to improve quality because you have numbers, because you have measures, because you are able to detect and see and fix and understand what the right answer is. So, you know, all that we provide, you know, so but we don't say we're a magic wand; we need to have the human with us in order to be able to—you have to get a good human with a good AI judge.

Andy Karuza: Yep. Totally agreed. Could you share any other examples to kind of inspire people on how they could utilize your guys' services?

(Olivier): Well, you know, I think one of the first things we did as a POC was working with an AI tech startup in the chartered accounting world called Tenant AI. That AI tech startup was providing chatbots for accounting CPAs. So, you know, a job is for accounting, you know, it would help like, you know, cook the books, it would help build the books, it would help connect the Quickbooks as well. It would, you know, it would do complete assistance to the human, to the CPA human. One thing that happened to them is that, you know, the first of their big firms of CPA came to them and said, "Well, you know, I could take OpenAI for this, why do I need you?" This is something that commonly happened on the market for every sort of AI tech vendor or AI tech startup. They have to be challenged always by these big models and say, "Why do I need you? I'm already paying tokens to OpenAI. I'm paying for like Gemini or Azure. I don't need you." And he said, "But, you know, these people are like, you know, these big companies, they are AGI." Said, "What do you mean? I don't even understand what that means. It's general intelligence. They know everything, but they know nothing. We do very well with CPA. Yep. That's what we do. We provide you with intelligence on accounting." "Okay. Can you help me prove that?" Well, he didn't know where to go. The CEO got lost and said, "Where do I prove it?" He goes back to his R&D team and says, "Hey guys, can you give me measurable proof that we're better than OpenAI?" "I don't know." "Let me speak to another guy." And then they found us. When they found us, we were able to measure how good they were against every other large language model that helped them secure their first pilot and clients, and then it also helped them secure an investment of 3 million bucks.

Andy Karuza: Wow. So, that's a good story for anyone who wants to go and talk to us. That's a great story, and I love that you highlight the point that what you guys do and what, you know, people should be doing with their artificial intelligence models is that they need to be focusing on their niche angle, whereas if you're just doing an out-of-the-box like ChatGPT wrapper, you're not going to be really competitive, right? But if you could find your niche and you could be very good at that one thing, that's where a lot of the value is in AI right now, and I think that's where the all the AI industry is shaping up is that these big models become available to any sort of developer or tech startup in the valley where you see flourishing startups coming and, you know, "I'm building an AI agent to replace your go-to-market person. I am building like an AI content writer that will replace your blog person." And that thing is shaping up because if you think about it, you will need to have a human behind it to make sure that the content is right, that you might want to humanize it a bit so it doesn't look as plain. So, you know, and these people, they need to have a measure of success. They cannot just go to market like that because in the end, what's going to happen? I mean, I think you're aware that 87% of the old model is dying on the vine, you know. There is, you know, most of the development of LLM is dying on the vine, and what goes on is that there's no measure of quality, so people don't use it, you know, which is a shame. It's money that people throw in by the windows, and they come and say, "AI, AI," you know, and in the end, nothing is happening. So, we help unleash that value for a very small money that we ask. We are helping to shape up a safe and good AI future for the industry. And how do you guys, how does your pricing model work? How do you typically work with clients?

(Olivier): We have three tiers. We have a startup tier that starts at 750 a month. We have an emerging tier that starts at 3,000 a month. And we have an enterprise tier that starts at 20K a month that goes up to 25 when it's hosted on-premise. Our model is based on usage: how many LLM judgments you're going to have, how many custom metrics you're going to use, how many users you're going to have. There are a lot of common features across tiers which are present in multiple tiers because we want to give the value to everyone, but we decided to make a usage-based model so that would be easier and fairer for everyone.

Andy Karuza: Yep. I like that that you at least have a startup model and whatnot. Could you maybe you have a premium as well?

(Olivier): Oh, very nice. How does the what does the premium package look like? You have about 10 evaluations, one user, and you have testing. You can do a lot; you have one custom metric to test your custom metrics. You have the loop feedback in there. So, it's already a lot, and you have a Discord community where you can talk to me, talk to my partners, and they reply directly through the Discord. The more premium users we get, the better we are, but what we want is them to be active so they can give us feedback about the product, and we can improve the product while we are reading the feedback from them. There is a user guide and manual that is available online as well that they can use to start testing and connecting the LLM. If they have any issues, we're here to troubleshoot them, and it's free of charge for months.

Andy Karuza: Yeah. Yeah. Your product philosophy really aligns with your business philosophy as well. You seem really open to data and feedback from people.

(Olivier): Yeah. It's also like we don't want to—we don't want to say it's open source because something that is open source sometimes feels valueless. We want this to be open for the public to be able to improve it and to create the best product there is. To have that lower tier at 750 and headless in between allows people with smaller budgets to come in. You know, 750 a month is not a big deal for people to pay. You know, I pay multiple services much more than that as a subscription as a company, especially if it's core to their business. It's core to your business. You're able to prove to your client that you're good. You know how good it is, like, you know, how much time I take myself comparing myself to competition. I offer you a tool that allows you to compare right away and say, "Hey, you're able to compare to all your big models. You're also able to make the choice of what's the best tool to use for yourself, the best database, the best big model, you know, large language model." So, these are important choices for you.

Andy Karuza: Oh, very yeah, it's core to any any sort of operate. I had some remarks from the public saying like, "Oh, I use human in the loop because I pay them cheap. I have them in low labor costs." And but I said, "How much time does it take for you to release anything?" And are you sure of the measure of success? They're not really sure of the measure of success, you know? So, it's uh it's um it's something that they need to solve for sure. Can you tell us more about how you take some of the technical evaluation metrics and translate those into meaningful business KPIs? Like, for instance, with the accounting firm, or maybe one of the other groups that you've worked with, like how do you take some of those technical performance uh metrics and and show that in a business use case?

(Olivier): I mean, I think we discussed the percentage discount is a good one. I think we have a measure of child-appropriate language. We have, you know, we have a token price, you know, how much you pay. We have latency. We have many sorts of like metrics that will con that will convey to a business. So, it's uh it's something that is very scalable in a way, you know.

Andy Karuza: Yep. And where do you see RAG evaluation in the next couple of years? Where do you see it evolving to?

(Olivier): I think it will go to more than RAG. I think we will add more and more RAG tools. I think, you know, but I think in the end, what we want as an evaluation bench is to be able to evaluate pictures, videos, audio, so we can go through the entire scope, you know, and machine learning. So, checking the learning of machines all the time, you know, and be part of like the workflow in a constant manner and be a systematic tool that will keep, you know, judging quality all the time.

Andy Karuza: Yep. I like your guys' vision. You guys are off to the races already. Um, so you'd love to see your guys' company grow and what you guys are working on. Finally, somebody's actually building the infrastructure for AI as opposed to just trying to be AI. Um, so I guess what are some key takeaways, you know, from the interview today that you like customers to know about rag metrics and why they should pick you guys up today?

(Olivier): Well, I like I said, I want people to understand that, of course, we're here to monetize our company, but we're here to unleash what they do and help them out. We are here to make sure that what they do is right. We're here to help the public so that they don't get fooled by this technology, and we're here to help AI so it doesn't create a bubble as well. So, it becomes a stringent type of technology. It's not crypto. We're not in the metaverse. It's not one of these crazy bubbles that we've been through in the past couple of years. It is very much a true revolution like the industrial revolution, the gold revolution. It's one of these transformational times where it transforms society, social relationships, you know, work ethics, and everything that goes along with it, politics. So, we want to make that revolution very real and not dangerous for people, you know. It's not just saying like Elon is saying it can be dangerous, but next day he brings a big robot. This is something where we have no other interest than to serve people to make that revolution proper, and because we are being paid for it. So, we and we are completely agnostic as well. We have no dependency on OpenAI, no Anthropic. I mean, some competition already sold their soul to these big models, and we want to stay agnostic to that. We want to be able to have a judge that is a judge, and a judge is fair and a judge is square. And it relies on something that is not—it's not because you give us a bigger subscription that will give you bigger results. You know, if it's wrong, it's wrong, you know.

Andy Karuza: Well, love your guys' vision. I'm glad somebody's doing it the right way. Um, where can people learn more about ragmetrics and sign up?

(Olivier): They go to ragmetrics.ai, or they can write to me at Olivier@ragmetrics.ai. O L I V I E R at ragmetrics.ai. Sorry for my French accents.

Andy Karuza: We love it. Trust me. So, O L I V I E R at ragmetrics.ai, or they go to the website and fill up the forms. Excellent. And then we're also happy to be partners with you guys as well. RAM is also in the Nacho Nacho marketplace. Um, you know, really excited about this product. Uh, definitely one one one that really stands out from many of the software products and AI tools that I've seen out there. So, if you're building any sort of an artificial intelligence-based company, you need to be checking out rag rig metrics today. Um, make sure that your models are aligned and working properly because it's going to be a niche game going forward. It's, you know, don't just be a Chad GPT rapper. Uh, make sure that you have your LLMs in in order and you have a you're the best at what you do in your particular niche space to find success. And we're here to help them. We want to help them. Excellent. Well, Olivier, uh, thanks for coming on today. really appreciate chatting with you always and uh looking forward to growing our businesses together.

(Olivier): Thank you so much, Andy. Looking forward to it.

Andy Karuza: You too.