Added RAG Evaluation Code

README.md

@@ -14,16 +14,15 @@ license: mit
 
 Welcome to OpenPages IntelliBot, your intelligent and efficient chatbot powered by the state-of-the-art Retrieval-Augmented Generation (RAG) technique and Large Language Model (LLM).
 
-**Gradio:** [![Try it!](https://img.shields.io/badge/Try_it!-blue.svg)](https://huggingface.co/spaces/nikhilkomakula/llm-rag-op-chatbot)
-
-**Streamlit:** [![Try it!](https://img.shields.io/badge/Try_it!-blue.svg)](https://nk-openpages-intellibot.streamlit.app)
+****Streamlit:** [![Try it!](https://img.shields.io/badge/Try_it!-blue.svg)](https://nk-openpages-intellibot.streamlit.app)**
 
+**Gradio:** [![Try it!](https://img.shields.io/badge/Try_it!-blue.svg)](https://huggingface.co/spaces/nikhilkomakula/llm-rag-op-chatbot)
 
-## What is OpenPagesIntelliBot?
+## What is OpenPages IntelliBot?
 
 OpenPagesIntelliBot leverages cutting-edge AI technologies to provide you with instant and accurate responses about OpenPages, its features, solutions / modules it offers and its trigger framework. By combining the power of RAG and Zephyr LLM, OpenPagesIntelliBot ensures that you receive contextually relevant information.
 
-## How RAG Works?
+## How Retrieval-Augmented Generation (RAG) Works?
 
 ![RAG Diagram](images/RAG_workflow.png)
 
@@ -136,7 +135,18 @@ docker ps -a
 docker logs -f llm-rag-op-chatbot
 ```
 
-## REST API (Gradio):
+## RAG Evaluation:
+
+Used [DeepEval](https://github.com/confident-ai/deepeval) open-source LLM evaluation framework for evaluating the performance of the RAG pipeline. Below metrics are used to evaluate its performance:
+
+* **Answer Relevancy:** Measures the quality of your RAG pipeline's generator by evaluating how relevant the `actual_output` of your LLM application is compared to the provided `input`.
+* **Faithfulness:** Measures the quality of your RAG pipeline's generator by evaluating whether the `actual_output` factually aligns with the contents of your `retrieval_context`.
+* **Contextual Relevancy:** Measures the quality of your RAG pipeline's retriever by evaluating the overall relevance of the information presented in your `retrieval_context` for a given `input`.
+* **Hallucination:** Determines whether your LLM generates factually correct information by comparing the `actual_output` to the provided `context`.
+* **Bias:** Determines whether your LLM output contains gender, racial, or political bias.
+* **Toxicity:** Evaluates toxicness in your LLM outputs.
+
+## REST API (Gradio Only):
 
 **Note:** Navigate to the chat interface UI in the browser and locate `Use via API` and click on it. A fly over opens on the right hand side. Capture the URL under the title named `API documentation`.
 
@@ -150,17 +160,23 @@ docker logs -f llm-rag-op-chatbot
 * **Vector Database :** ChromaDB
 * **Orchestration Framework :** LangChain
 * **Embedding Model :** BAAI/bge-large-en-v1.5
-* **Large Language Model :** huggingfaceh4/zephyr-7b-beta
+* **Large Language Model :** huggingfaceh4/zephyr-7b-alpha
 * **UI Framework** : Streamlit & Gradio
 
+## CI/CD:
+
+* **Streamlit.io**
+  * Committing any changes to the git branch `deploy-to-streamlit` will deploy to streamlit.io.
+* **Hugging Face Spaces**
+  * Committing any changes to the git branch `deploy-to-hf-spaces` will deploy to Hugging Face Spaces as a Docker space.
+
 ## Streamlit.io Deployment:
 
 If you are encountering issues with `sqlite` version, then run the following steps:
 
 * Add the following dependency to `requirements.txt`:
 
-    `pysqlite3-binary==0.5.2.post3`
-
+  `pysqlite3-binary==0.5.2.post3`
 * Add the following block of code to `streamlit_app.py` at the beginning of the file:
 
 ```
@@ -172,6 +188,15 @@ sys.modules['sqlite3'] = sys.modules.pop('pysqlite3')
 
 **Note:** If running locally for Streamlit UI interace and if you hit any errors with `pysqlite3`, try removing whatever that is mentioned above.
 
+## Enhancements:
+
+* Different advanced retrieval methods could be used.
+* Context re-ranking can be implemented.
+* Latest LLMs could be used for better performance.
+* Could be converted to `conversational` AI chatbot.
+* Utilize better PDF parsers and experiment with `chunk_size` and `chunk_overlap` properties.
+* Fine-tuning the LLM on the proprietary dataset might improve the results.
+
 ## Contact Me:
 
 For any inquiries or feedback, please contact me at [nikhil.komakula@outlook.com](mailto:nikhil.komakula@outlook.com).

app.py

@@ -10,7 +10,7 @@ def run_streamlit_interface():
     subprocess.run(["streamlit", "run", "streamlit_app.py"])
     
 def run_rag_evaluate():
-    subprocess.run(["python", "eval.py"])
+    subprocess.run(["python", "eval_app.py"])
 
 # Main function to determine which interface to run
 def main():

eval.py → eval_app.py

@@ -3,7 +3,7 @@ from dotenv import find_dotenv, load_dotenv
 
 # import functions
 from src.test.eval_rag import evaluate_rag
-from src.generation.generate_response import get_qa_chain
+from src.generation.generate_response import get_qa_chain_eval
 
 def main():
 
@@ -12,7 +12,7 @@ def main():
     load_dotenv(find_dotenv())
 
     # get the qa chain
-    qa_chain = get_qa_chain()
+    qa_chain = get_qa_chain_eval()
     
     # evaluate the qa chain
     evaluate_rag("qa_chain", qa_chain)

requirements.txt

@@ -9,3 +9,4 @@ tensorflow==2.16.1
 gradio==4.21.0
 pandas==2.2.1
 streamlit==1.32.2
+deepeval==0.21.21

src/generation/generate_response.py

@@ -1,7 +1,9 @@
 # import libraries
 import time
 from typing import Optional
-from src.retrieval.retriever_chain import get_base_retriever, load_hf_llm, create_qa_chain
+
+# import functions
+from src.retrieval.retriever_chain import get_base_retriever, load_hf_llm, create_qa_chain, create_qa_chain_eval
 
 # constants
 HF_MODEL        = "huggingfaceh4/zephyr-7b-alpha"  # "mistralai/Mistral-7B-Instruct-v0.2" # "google/gemma-7b"
@@ -138,4 +140,24 @@ def has_global_variable():
     if 'global_qa_chain' in globals():
         return True
     
-    return False
+    return False
+
+# get the qa chain for evaluation
+def get_qa_chain_eval():
+    """
+    Instantiates QA Chain for evaluation.
+
+    Returns:
+        Runnable: Returns an instance of QA Chain.
+    """
+
+    # get retriever
+    retriever = get_base_retriever(embedding_model=EMBEDDING_MODEL, k=4, search_type="mmr")
+
+    # instantiate llm
+    llm = load_hf_llm(repo_id=HF_MODEL, max_new_tokens=512, temperature=0.4)
+
+    # instantiate qa chain
+    qa_chain = create_qa_chain_eval(retriever, llm)
+
+    return qa_chain

src/retrieval/retriever_chain.py

@@ -1,9 +1,9 @@
 # import libraries
-import os
 from langchain_core.prompts import ChatPromptTemplate
 from langchain_community.llms import HuggingFaceEndpoint
 from langchain_core.runnables import RunnablePassthrough
 from langchain_core.output_parsers import StrOutputParser
+from langchain_core.runnables import RunnableParallel
 
 # import functions
 from ..indexing.build_indexes import retrieve_indexes
@@ -90,7 +90,7 @@ def create_qa_chain(retriever, llm):
     Returns:
         Runnable: Returns qa chain.
     """
-    
+
     def format_docs(docs):
         return "\n\n".join(doc.page_content for doc in docs)
 
@@ -101,3 +101,33 @@ def create_qa_chain(retriever, llm):
         | StrOutputParser()
     )
     return qa_chain
+
+
+# define retrieval chain for evaluation
+def create_qa_chain_eval(retriever, llm):
+    """
+    Instantiates qa chain for evaluation.
+
+    Args:
+        retriever (VectorStoreRetriever): Vector store.
+        llm (HuggingFaceEndpoint): HuggingFace endpoint.
+
+    Returns:
+        Runnable: Returns qa chain.
+    """
+
+    def format_docs(docs):
+        return "\n\n".join(doc.page_content for doc in docs)
+
+    rag_chain_from_docs = (
+        RunnablePassthrough.assign(context=(lambda x: format_docs(x["context"])))
+        | create_prompt_template()
+        | llm
+        | StrOutputParser()
+    )
+
+    rag_chain_with_source = RunnableParallel(
+        {"context": retriever, "query": RunnablePassthrough()}
+    ).assign(result=rag_chain_from_docs)
+
+    return rag_chain_with_source

src/test/eval_custom_model.py

@@ -0,0 +1,100 @@
+# import functions
+from deepeval.models.base_model import DeepEvalBaseLLM
+from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric, ContextualRelevancyMetric, HallucinationMetric, BiasMetric, ToxicityMetric
+from deepeval.test_case import LLMTestCase
+
+class LLM(DeepEvalBaseLLM):
+    def __init__(
+        self,
+        model,
+        model_name
+    ):
+        self.model = model
+        self.model_name = model_name
+
+    def load_model(self):
+        return self.model
+
+    def generate(self, prompt: str) -> str:
+        model = self.load_model()
+        return model(prompt)
+
+    async def a_generate(self, prompt: str) -> str:
+        return self.generate(prompt)
+
+    def get_model_name(self):
+        return self.model_name
+    
+def eval_answer_relevancy_metric(llm: LLM, question: str, answer: str, context: list):
+    answer_relevancy_metric = AnswerRelevancyMetric(model=llm, threshold=0.5, include_reason=True)
+    test_case = LLMTestCase(
+        input=question,
+        actual_output=answer,
+        retrieval_context=context
+    )
+
+    answer_relevancy_metric.measure(test_case)
+    return answer_relevancy_metric.score
+
+def eval_faithfulness_metric(llm: LLM, question: str, answer: str, context: list):
+    faithfulness_metric = FaithfulnessMetric(model=llm, threshold=0.5, include_reason=True)
+    test_case = LLMTestCase(
+        input=question,
+        actual_output=answer,
+        retrieval_context=context
+    )
+
+    faithfulness_metric.measure(test_case)
+    return faithfulness_metric.score
+
+def eval_contextual_relevancy_metric(llm: LLM, question: str, answer: str, context: list):
+    contextual_relevancy_metric = ContextualRelevancyMetric(model=llm, threshold=0.5, include_reason=False)
+    test_case = LLMTestCase(
+        input=question,
+        actual_output=answer,
+        retrieval_context=context
+    )
+
+    contextual_relevancy_metric.measure(test_case)
+    return contextual_relevancy_metric.score
+
+def eval_hallucination_metric(llm: LLM, question: str, answer: str, context: list):
+    hallucination_metric = HallucinationMetric(model=llm, threshold=0.5, include_reason=True)
+    test_case = LLMTestCase(
+        input=question,
+        actual_output=answer,
+        context=context
+    )
+
+    hallucination_metric.measure(test_case)
+    return hallucination_metric.score
+
+def eval_bias_metric(llm: LLM, question: str, answer: str):
+    bias_metric = BiasMetric(model=llm, threshold=0.5, include_reason=True)
+    test_case = LLMTestCase(
+        input=question,
+        actual_output=answer
+    )
+
+    bias_metric.measure(test_case)
+    return bias_metric.score
+
+def eval_toxicity_metric(llm: LLM, question: str, answer: str):
+    toxicity_metric = ToxicityMetric(model=llm, threshold=0.5, include_reason=True)
+    test_case = LLMTestCase(
+        input=question,
+        actual_output=answer
+    )
+
+    toxicity_metric.measure(test_case)
+    return toxicity_metric.score
+
+def eval_rag_metrics(llm: LLM, question: str, answer: str, context: list) -> dict:
+    return {
+            "AnswerRelevancyMetric": eval_answer_relevancy_metric(llm, question, answer, context),
+            "FaithfulnessMetric": eval_faithfulness_metric(llm, question, answer, context),
+            "ContextualRelevancyMetric": eval_contextual_relevancy_metric(llm, question, answer, context),
+            # "HallucinationMetric": eval_hallucination_metric(llm, question, answer, context),
+            # "BiasMetric": eval_bias_metric(llm, question, answer),
+            # "ToxicityMetric": eval_toxicity_metric(llm, question, answer),
+        }

src/test/eval_rag.py

@@ -4,12 +4,31 @@ import time
 import datetime
 import pandas as pd
 
+# import functions
+from src.test.eval_custom_model import LLM, eval_rag_metrics
+from src.retrieval.retriever_chain import load_hf_llm
+
 # constants
+EVAL_LLM = "mistralai/Mistral-7B-v0.1" # "mistralai/Mistral-7B-Instruct-v0.2"
+EVAL_LLM_NAME = "Mistral 7B"
 EVAL_FILE_PATH = "./src/test/eval_questions.txt"
 EVAL_RESULTS_FILE_NAME = "eval_results_{0}.csv"
 EVAL_RESULTS_PATH = "./src/test"
 
 
+# format context documents as list
+def format_docs_as_list(docs):
+    """
+    Converts context documents as list
+    
+    Args:
+        docs (list): List of Document objects
+
+    Returns:
+        list: Returns list of documents.
+    """
+    return [doc.page_content for doc in docs]
+
 # load eval questions
 def load_eval_questions():
     """
@@ -42,25 +61,40 @@ def evaluate_rag(chain_name, rag_chain):
     columns = ["Chain", "Question", "Response", "Time"]
     df = pd.DataFrame(columns=columns)
 
+    # load evaluation questions
     eval_questions = load_eval_questions()
+    
+    # instantiate hf llm
+    hf_eval_llm = load_hf_llm(repo_id=EVAL_LLM, max_new_tokens=512, temperature=0.4)
+
+    # instantiate deepeval llm
+    eval_custom_model = LLM(model_name=EVAL_LLM_NAME, model=hf_eval_llm)
 
     for question in eval_questions:
 
         start_time = time.time()
-        answer = rag_chain.invoke(question)
+        response = rag_chain.invoke(question)
+        print("Response", response)
         end_time = time.time()
+        
+        query = response['query']
+        answer = response['result']
+        context = format_docs_as_list(response['context'])
+        metrics = eval_rag_metrics(eval_custom_model, question, answer, context)
 
         row = {
             "Chain": chain_name,
-            "Question": question,
+            "Question": query,
             "Response": answer,
             "Time": "{:.2f}".format(round(end_time - start_time, 2)),
+            "Metrics": metrics
         }
         
         print("*" * 100)
         print("Question:", question)
         print("Answer:", answer)
         print("Response Time:", "{:.2f}".format(round(end_time - start_time, 2)))
+        print("Metrics:", metrics)
         print("*" * 100)
 
         df = pd.concat([df, pd.DataFrame.from_records([row])])

src/test/eval_results_20240325111437.csv

@@ -1,86 +0,0 @@
-Chain,Question,Response,Time
-qa_chain,What is FastMap?," FastMap is a productivity tool that integrates with IBM OpenPages with Watson's export feature and automates the process of importing and batch processing object data into OpenPages with Watson. It uses a data load template, typically in Microsoft Excel format, to capture data for import. When importing data into OpenPages with Watson, FastMap validates the data and, if no errors are found, populates the repository with the new or updated records. It supports the import of various types of objects, but not File and Signature objects or the Comment field's system field. Some general rules apply when entering object data into a FastMap data load template, such as associating child objects using specific columns and not importing certain fields like read-only fields or those not in the Admin view for the object. Additionally, FastMap import does not evaluate field dependency rules, allowing users to stage data requiring them to enter required data during subsequent updates. Overall, FastMap helps streamline the process of importing large amounts of data into OpenPages with Watson, reducing manual input and increasing efficiency.",4.01
-qa_chain,What is a Role Template?," A Role Template in IBM OpenPages with Watson solutions is a predefined set of permissions and access controls that determine the level of access and functionality granted to a user in the system. It includes application permissions, feature access, and Object Access Controls (OACs) for specific object types. Role Templates help simplify the process of managing user access and ensure consistent security policies across the organization. By assigning a Role Template to a user, you can quickly and easily grant them the required permissions and access levels for their job function.",1.85
-qa_chain,What is the purpose of Object Reset?," The purpose of Object Reset in IBM OpenPages with Watson is to allow users to efficiently update multiple objects at once based on predefined rules. This feature helps to simplify repetitive data entry tasks and reduce errors associated with manually updating large numbers of objects. It also provides flexibility in terms of what objects are affected by the reset, as users can choose specific business entities and define rulesets to control the behavior of the reset. Overall, Object Reset helps to streamline data management processes and improve efficiency in IBM OpenPages with Watson.",2.56
-qa_chain,What is the purpose of Reporting Periods?," Reporting periods in IBM OpenPages with Watson serve as ""snapshots"" of the current state of the repository during specific points in time, typically at the end of a quarter or year. These periods allow organizations to archive and report on historical data while maintaining the ability to view and analyze current data. Reporting periods facilitate compliance and regulatory requirements, provide insights into trends and patterns, and enable comparisons between different periods. Additionally, they can be used to automate object resets, which modify object properties based on predefined rulesets. Overall, reporting periods help organizations manage and analyze their data more efficiently and effectively.",2.05
-qa_chain,List the system variables used in Expressions.,"1. $APPLICATION_URL$: A URL for OpenPages.
-    2. $COGNOS_URL$: A URL for IBM Cognos Analytics.
-    3. $ApplicationText/application text key$): Application text content.
-    4. $TASK_VIEW_URL$): A URL to an object task view.
-    5. $System Fields:Task View URL$): Deprecated. Use [$TASK_VIEW_URL$].
-    6. $Setting/OpenPages/...$): A registry setting value.
-    7. $END_USER$): The user name of the signed on user.
-    8. $TODAY$): Today's date.
-    9. $DaysFromNow/field group:field name$): A day count from today to a given date.
-    10. ${asset.id}$: Inserts the identifier of the underlying asset.
-    11. ${asset.name}$: Inserts the asset name of the underlying asset.
-    12. ${asset.description}$: Inserts the asset description of the underlying asset.",2.46
-qa_chain,Provide the steps to configure Watson Assistant in OpenPages?," Unfortunately, I don't have access to your specific environment or installation details. However, here are the general steps to configure Watson Assistant in OpenPages:
-
-1. Ensure that you have a Watson Assistant service instance created and configured with the necessary intents and entities.
-
-2. In OpenPages, go to the Administration Console > Integrations > Watson Assistant.
-
-3. Click on the ""Add"" button to create a new Watson Assistant integration.
-
-4. Enter a name and description for the integration and select the appropriate Watson Assistant service instance from the list.
-
-5. Configure the authentication settings by providing the API key and URL of your Watson Assistant service.
-
-6. Select the appropriate language and version of Watson Assistant.
-
-7. Map the Watson Assistant intents and entities to the corresponding OpenPages business objects and fields.
-
-8. Save the configuration and test the integration by running a sample request through the Watson Assistant REST API.
-
-9. Verify that the response is correctly mapped to the appropriate OpenPages business object and field values.
-
-10. Deploy the integration to make it available to users in the OpenPages application.
-
-Note: These steps may vary depending on your specific implementation and version of OpenPages and Watson Assistant. It's recommended to refer to the official documentation for detailed instructions and best practices.",1.94
-qa_chain,What is the difference between PRE and POST position in Triggers?,"
-    In Triggers, the position ""PRE"" refers to events that occur prior to the actual execution of an operation by the system. These triggers allow for additional processing of business logic before the operation is carried out. For example, during the creation of a GRC object, a PRE trigger has access to all the necessary information about the object to be created, but the system has not yet taken action to create the object and persist its values. PRE triggers are mandatory for deletions, associations, and disassociations.
-
-    On the other hand, the position ""POST"" refers to events that occur after an operation has been performed by the system but before the transaction is committed. These triggers provide further processing of additional business logic after the operation has been completed. POST triggers are mandatory for creating and updating operations.
-
-    In summary, PRE triggers execute before the operation takes place, while POST triggers execute after the operation has been completed.",2.56
-qa_chain,What are the features of Operational Risk Management in OpenPages?,"1. Loss Events: This feature allows organizations to track, assess, and manage both internal and external events that could lead to operational losses. Multiple impact events and recoveries associated with operational losses can also be managed through this feature.
-
-2. Risk and Control Self Assessments (RCSA): This feature helps in identifying, measuring, and mitigating risks, as well as testing and documenting internal controls.
-
-3. Key Risk Indicators (KRIs) and Key Performance Indicators (KPIs): These features enable tracking of performance metrics that may indicate the presence or state of a risk condition or trend.
-
-4. Scenario Analysis: This feature is used to identify and measure specific types of risks, particularly low-frequency, high-severity events.
-
-5. External Loss Events: This feature allows for the import of loss data from various sources such as IBM FIRST Risk Case Studies, ORX, and ORIC loss databases for scenario analysis, benchmarking, and report generation.
-
-6. Issue Management and Remediation (IMR): This feature includes issue creation and assignment, action creation and assignment, remediation performance, issue closure, and reporting.
-
-7. Reporting, Monitoring, and Analytics: This feature provides reporting, monitoring, and analytics capabilities for operational risk management.
-
-Note: The above features are part of IBM OpenPages Operational Risk Management. Other related features like Regulatory Compliance Management, Third Party Risk Management, and IT Governance with RiskLens are also available in IBM OpenPages.",2.56
-qa_chain,What are the different permissions that can be delegated to a user group administrator?," There are six security management permissions that can be delegated to a user group administrator: Manage, Lock, Unlock, Reset Password, Assign Role, and Browse. These permissions allow the administrator to perform various user-provisioning functions such as creating, modifying, and associating users and groups, locking and unlocking user accounts, resetting passwords, assigning roles to users and groups, and browsing users and groups within their respective groups. The specific permissions required for each user-provisioning function are listed in Table 38 provided in the text material. It's important to note that these permissions should be granted carefully and only to trusted individuals as they have significant impact on the overall security and functionality of the system.",8.57
-qa_chain,What are the different access controls available for non-participants for a standard stage within a workflow?," Table 142 in the text provides the different access controls available for non-participants for a standard stage within a workflow:
-
-     Access control for the stage
-     Can view the object when it’s at this stage
-     Can edit the object when it’s at this stage
-     Can see the Actions button in views
-     Strict
-     No
-     No
-     No
-     Read
-     Yes
-     No
-     No
-     Open
-     Depends on standard access controls
-     Depends on standard access controls
-     No
-     No Override
-     Depends on standard access controls
-     Depends on standard access controls
-     Yes
-
-     Note: For workflow participants (assignees, oversight users, and subscribers), the standard access controls apply, and they only know their part of the process. Non-participants may require additional access controls based on their roles and responsibilities within the organization. The ""Override"" option allows you to define whether to override these standard access controls for the workflow stage for non-participants.",41.15