Priti0210 commited on
Commit
a4ef927
ยท
1 Parent(s): d2ab7d2

made it gradio compatible

Browse files
.gradio/certificate.pem ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ -----BEGIN CERTIFICATE-----
2
+ MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
3
+ TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
4
+ cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
5
+ WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
6
+ ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
7
+ MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
8
+ h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
9
+ 0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
10
+ A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
11
+ T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
12
+ B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
13
+ B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
14
+ KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
15
+ OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
16
+ jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
17
+ qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
18
+ rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
19
+ HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
20
+ hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
21
+ ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
22
+ 3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
23
+ NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
24
+ ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
25
+ TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
26
+ jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
27
+ oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
28
+ 4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
29
+ mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
30
+ emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
31
+ -----END CERTIFICATE-----
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2025 humaninloop
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md CHANGED
@@ -1,13 +1,130 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
- title: MadGuard
3
- emoji: ๐Ÿ’ฌ
4
- colorFrom: yellow
5
- colorTo: purple
6
- sdk: gradio
7
- sdk_version: 5.0.1
8
- app_file: app.py
9
- pinned: false
10
- license: mit
 
 
 
 
 
 
 
 
 
11
  ---
12
 
13
- An example chatbot using [Gradio](https://gradio.app), [`huggingface_hub`](https://huggingface.co/docs/huggingface_hub/v0.22.2/en/index), and the [Hugging Face Inference API](https://huggingface.co/docs/api-inference/index).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ๐Ÿง  MADGuard AI Explorer
2
+
3
+ A diagnostic Gradio tool to simulate feedback loops in Retrieval-Augmented Generation (RAG) pipelines and detect **Model Autophagy Disorder (MAD)** risks.
4
+
5
+ ---
6
+
7
+ ## ๐Ÿ› ๏ธ Tool Description
8
+
9
+ - Toggle between **real** and **synthetic** input sources
10
+ - Visualize pipeline feedback loops with **Graphviz**
11
+ - Analyze training data via:
12
+ - Type-Token Ratio (TTR)
13
+ - Cosine Similarity
14
+ - Composite MAD Risk Score
15
+
16
+ ---
17
+
18
+ ## ๐Ÿš€ Run It Locally
19
+
20
+ ```bash
21
+ git clone <your-repo-url>
22
+ cd madguard
23
+ pip install -r requirements.txt
24
+ python app.py
25
+ ```
26
+
27
+ Then open [http://127.0.0.1:7860](http://127.0.0.1:7860) in your browser.
28
+
29
+ ---
30
+
31
+ ## ๐ŸŒ Deploy on Hugging Face Spaces
32
+
33
+ 1. Create a new Space (select **Gradio** as the SDK)
34
+ 2. Upload:
35
+ - `app.py`
36
+ - `requirements.txt`
37
+ - All files in the `visuals/` folder
38
+ 3. Hugging Face builds the app and gives you a public URL
39
+
40
+ ---
41
+
42
+ <details>
43
+ <summary>๐Ÿ“š Research Background</summary>
44
+
45
+ ### ๐Ÿ“„ Self-consuming LLMs: How and When Models Feed Themselves โ€“ Santurkar et al., 2023
46
+
47
+ This paper introduces and explores **Model Autophagy Disorder (MAD)** โ€” showing that large language models trained on their own outputs tend to lose performance and accumulate error over time.
48
+
49
+ **MADGuard implements several of the paperโ€™s proposed detection strategies:**
50
+
51
+ | Research Recommendation | MADGuard Implementation |
52
+ | ------------------------------------------- | ----------------------------------------- |
53
+ | Lexical redundancy analysis | โœ… via Type-Token Ratio (TTR) |
54
+ | Embedding-based similarity scoring | โœ… via SentenceTransformers + cosine |
55
+ | Warning system for feedback loop risk | โœ… risk score (Low / Medium / High) |
56
+ | Distinguishing real vs. synthetic inputs | โŒ not implemented (user-controlled only) |
57
+ | Multi-round retraining degradation tracking | โŒ not yet supported |
58
+
59
+ > โ€œMADGuard AI Explorer is inspired by key findings from this research, aligning with early warnings and pipeline hygiene practices recommended in their work.โ€
60
+
61
+ ๐Ÿ“Ž [Read Full Paper on arXiv](https://arxiv.org/abs/2307.01850)
62
+
63
+ </details>
64
+
65
  ---
66
+
67
+ <details>
68
+ <summary>๐Ÿ‘ฅ Who Is It For?</summary>
69
+
70
+ - **AI/ML Engineers**: Prevent model collapse due to training on synthetic outputs
71
+ - **MLOps Professionals**: Pre-retraining diagnostics
72
+ - **AI Researchers**: Study model feedback loops
73
+ - **Responsible AI Teams**: Audit data pipelines for ethical AI
74
+
75
+ ### Why Use It?
76
+
77
+ - Avoid data contamination
78
+ - Ensure model freshness
79
+ - Support data-centric decisions
80
+ - Provide audit-ready diagnostics
81
+
82
+ </details>
83
+
84
  ---
85
 
86
+ <details>
87
+ <summary>๐Ÿงฑ Limitations & Future Plans</summary>
88
+
89
+ ### ๐Ÿ”ธ Current Limitations
90
+
91
+ | Area | Missing Element |
92
+ | ------------------- | ----------------------------------------- |
93
+ | Multi-batch Uploads | No history or comparative dataset support |
94
+ | Real/Synthetic Tag | No auto-tagging or provenance logging |
95
+ | Visual Analytics | No charts, timelines, or embeddings view |
96
+ | Custom Thresholds | Fixed MAD score weightings |
97
+ | Provenance Tracking | No metadata or source history logging |
98
+
99
+ ### ๐Ÿ”ฎ Future Plans
100
+
101
+ - ๐Ÿ“Š Batch evaluations with historical trendlines
102
+ - ๐Ÿง  RAG framework integration (e.g., LangChain)
103
+ - ๐Ÿงฉ Live evaluation API endpoint
104
+ - ๐Ÿ”’ Source tracking and audit trails
105
+ - ๐Ÿงพ Exportable audit/compliance reports
106
+
107
+ </details>
108
+
109
+ ---
110
+
111
+ <details>
112
+ <summary>๐Ÿ“„ More Details</summary>
113
+
114
+ ### ๐Ÿ” Features Recap
115
+
116
+ - Simulates feedback loops in RAG pipelines
117
+ - Visualizes flow using Graphviz
118
+ - Accepts `.csv` or `.json` data
119
+ - Calculates TTR, cosine similarity, MAD score
120
+ - Classifies risk (Low / Medium / High)
121
+ - Offers human-readable suggestions
122
+ - Based on: [Santurkar et al., 2023 โ€“ arXiv:2307.01850](https://arxiv.org/abs/2307.01850)
123
+
124
+ ### ๐Ÿ“œ License
125
+
126
+ MIT License (see [LICENSE](LICENSE))
127
+
128
+ </details>
129
+
130
+ ---
app.py CHANGED
@@ -1,64 +1,264 @@
1
  import gradio as gr
2
- from huggingface_hub import InferenceClient
3
-
4
- """
5
- For more information on `huggingface_hub` Inference API support, please check the docs: https://huggingface.co/docs/huggingface_hub/v0.22.2/en/guides/inference
6
- """
7
- client = InferenceClient("HuggingFaceH4/zephyr-7b-beta")
8
-
9
-
10
- def respond(
11
- message,
12
- history: list[tuple[str, str]],
13
- system_message,
14
- max_tokens,
15
- temperature,
16
- top_p,
17
- ):
18
- messages = [{"role": "system", "content": system_message}]
19
-
20
- for val in history:
21
- if val[0]:
22
- messages.append({"role": "user", "content": val[0]})
23
- if val[1]:
24
- messages.append({"role": "assistant", "content": val[1]})
25
-
26
- messages.append({"role": "user", "content": message})
27
-
28
- response = ""
29
-
30
- for message in client.chat_completion(
31
- messages,
32
- max_tokens=max_tokens,
33
- stream=True,
34
- temperature=temperature,
35
- top_p=top_p,
36
- ):
37
- token = message.choices[0].delta.content
38
-
39
- response += token
40
- yield response
41
-
42
-
43
- """
44
- For information on how to customize the ChatInterface, peruse the gradio docs: https://www.gradio.app/docs/chatinterface
45
- """
46
- demo = gr.ChatInterface(
47
- respond,
48
- additional_inputs=[
49
- gr.Textbox(value="You are a friendly Chatbot.", label="System message"),
50
- gr.Slider(minimum=1, maximum=2048, value=512, step=1, label="Max new tokens"),
51
- gr.Slider(minimum=0.1, maximum=4.0, value=0.7, step=0.1, label="Temperature"),
52
- gr.Slider(
53
- minimum=0.1,
54
- maximum=1.0,
55
- value=0.95,
56
- step=0.05,
57
- label="Top-p (nucleus sampling)",
58
- ),
59
- ],
60
- )
61
 
 
 
 
 
 
62
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
  if __name__ == "__main__":
64
- demo.launch()
 
 
1
  import gradio as gr
2
+ import nltk
3
+ import pandas as pd
4
+ from nltk.tokenize import TreebankWordTokenizer
5
+ from sklearn.metrics.pairwise import cosine_similarity
6
+ from sentence_transformers import SentenceTransformer
7
+ import graphviz
8
+ from typing import Tuple, Optional
9
+ from visuals.score_card import render_score_card # Updated import
10
+ from visuals.layout import (
11
+ render_page_header,
12
+ render_core_reference,
13
+ render_pipeline,
14
+ render_pipeline_graph,
15
+ render_pipeline_warning,
16
+ render_strategy_alignment,
17
+ ) # Updated import
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
 
19
+ # Ensure NLTK data is downloaded
20
+ try:
21
+ nltk.download("punkt", quiet=True)
22
+ except Exception as e:
23
+ print(f"Error downloading NLTK data: {e}")
24
 
25
+ # Load SentenceTransformer model
26
+ model = SentenceTransformer("all-MiniLM-L6-v2")
27
+
28
+
29
+ def calculate_ttr(text: str) -> float:
30
+ """Calculates Type-Token Ratio (TTR) for lexical diversity."""
31
+ if not text:
32
+ return 0.0
33
+ words = text.split()
34
+ unique_words = set(words)
35
+ return len(unique_words) / len(words) if words else 0.0
36
+
37
+
38
+ def calculate_similarity(text1: str, text2: str) -> float:
39
+ """Calculates cosine similarity between two texts."""
40
+ embeddings = model.encode([text1, text2])
41
+ return cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
42
+
43
+
44
+ def calculate_mad_score(ttr: float, similarity: float) -> float:
45
+ """Calculates the MAD score."""
46
+ return 0.3 * (1 - ttr) + 0.7 * similarity
47
+
48
+
49
+ def get_risk_level(mad_score: float) -> str:
50
+ """Determines the risk level based on the MAD score."""
51
+ if mad_score > 0.7:
52
+ return "High"
53
+ elif 0.4 <= mad_score <= 0.7:
54
+ return "Medium"
55
+ else:
56
+ return "Low"
57
+
58
+
59
+ def process_data(file_obj, model_col: str, train_col: str, data_source: str) -> Tuple[
60
+ Optional[str],
61
+ Optional[bytes],
62
+ Optional[str],
63
+ Optional[str],
64
+ Optional[float],
65
+ Optional[float],
66
+ Optional[float],
67
+ ]:
68
+ """Processes the uploaded file and calculates metrics."""
69
+ try:
70
+ if not file_obj:
71
+ return "Error: No file uploaded.", None, None, None, None, None, None
72
+
73
+ file_path = file_obj.name
74
+ if file_path.endswith(".csv"):
75
+ df = pd.read_csv(file_path)
76
+ elif file_path.endswith(".json"):
77
+ df = pd.read_json(file_path)
78
+ else:
79
+ return (
80
+ "Error: Invalid file type. Please upload a CSV or JSON file.",
81
+ None,
82
+ None,
83
+ None,
84
+ None,
85
+ None,
86
+ None,
87
+ )
88
+
89
+ if model_col not in df.columns or train_col not in df.columns:
90
+ return (
91
+ "Error: Selected columns not found in the file.",
92
+ None,
93
+ None,
94
+ None,
95
+ None,
96
+ None,
97
+ None,
98
+ )
99
+
100
+ output_text = " ".join(df[model_col].astype(str))
101
+ train_text = " ".join(df[train_col].astype(str))
102
+
103
+ ttr_output = calculate_ttr(output_text)
104
+ ttr_train = calculate_ttr(train_text)
105
+ similarity = calculate_similarity(output_text, train_text)
106
+ mad_score = calculate_mad_score(ttr_output, similarity)
107
+ risk_level = get_risk_level(mad_score)
108
+
109
+ summary, details, explanation = render_score_card(
110
+ ttr_output, ttr_train, similarity, mad_score, risk_level
111
+ )
112
+ evaluation_markdown = summary + details + explanation
113
+
114
+ return (
115
+ None, # No error
116
+ render_pipeline_graph(data_source),
117
+ df.head().to_markdown(index=False, numalign="left", stralign="left"),
118
+ evaluation_markdown,
119
+ ttr_output,
120
+ ttr_train,
121
+ similarity,
122
+ )
123
+
124
+ except Exception as e:
125
+ return f"An error occurred: {str(e)}", None, None, None, None, None, None
126
+
127
+
128
+ def update_dropdowns(file_obj) -> Tuple[list, str]:
129
+ """Updates dropdown choices based on the uploaded file."""
130
+ if not file_obj:
131
+ return [], "No file uploaded."
132
+
133
+ file_path = file_obj.name
134
+ try:
135
+ if file_path.endswith(".csv"):
136
+ df = pd.read_csv(file_path)
137
+ elif file_path.endswith(".json"):
138
+ df = pd.read_json(file_path)
139
+ else:
140
+ return [], "Invalid file type."
141
+ columns = df.columns.tolist()
142
+ preview = df.head().to_markdown(index=False, numalign="left", stralign="left")
143
+ return columns, preview
144
+ except Exception as e:
145
+ return [], f"Error reading file: {e}"
146
+
147
+
148
+ def main_interface():
149
+ css = """
150
+ .gradio-container {
151
+ background: linear-gradient(-45deg, #e0f7fa, #e1f5fe, #f1f8e9, #fff3e0);
152
+ background-size: 400% 400%;
153
+ animation: oceanWaves 20s ease infinite;
154
+ }
155
+ @keyframes oceanWaves {
156
+ 0% { background-position: 0% 50%; }
157
+ 50% { background-position: 100% 50%; }
158
+ 100% { background-position: 0% 50%; }
159
+ }
160
+ """
161
+
162
+ with gr.Blocks(css=css, title="MADGuard AI Explorer") as interface:
163
+ gr.HTML(render_page_header())
164
+
165
+ with gr.Accordion("๐Ÿ“š Research Reference: arXiv:2307.01850", open=False):
166
+ gr.HTML(render_core_reference())
167
+
168
+ gr.Markdown("## 1. Pipeline Simulation")
169
+ data_source, description = render_pipeline()
170
+ gr.HTML(description)
171
+ pipeline_output = gr.Image(type="filepath", label="Pipeline Graph")
172
+ warning_output = gr.HTML()
173
+ data_source.change(
174
+ fn=render_pipeline_warning, inputs=data_source, outputs=warning_output
175
+ )
176
+ data_source.change(
177
+ fn=render_pipeline_graph, inputs=data_source, outputs=pipeline_output
178
+ )
179
+
180
+ gr.Markdown("## 2. Upload CSV or JSON File")
181
+ file_input = gr.File(
182
+ file_types=[".csv", ".json"], label="Upload a CSV or JSON file"
183
+ )
184
+
185
+ with gr.Row():
186
+ model_col_input = gr.Dropdown(
187
+ choices=[], label="Select column for model output"
188
+ )
189
+ train_col_input = gr.Dropdown(
190
+ choices=[], label="Select column for future training data"
191
+ )
192
+
193
+ file_preview = gr.Markdown(label="๐Ÿ“„ File Preview")
194
+
195
+ output_markdown = gr.Markdown(label="๐Ÿ” Evaluation Summary")
196
+
197
+ with gr.Accordion("๐Ÿ“‹ Research-Based Strategy Alignment", open=False):
198
+ gr.HTML(render_strategy_alignment())
199
+
200
+ with gr.Row():
201
+ ttr_output_metric = gr.Number(label="Lexical Diversity (Output)")
202
+ ttr_train_metric = gr.Number(label="Lexical Diversity (Training Set)")
203
+ similarity_metric = gr.Number(label="Semantic Similarity (Cosine)")
204
+
205
+ file_input.change(
206
+ update_dropdowns,
207
+ inputs=file_input,
208
+ outputs=[model_col_input, train_col_input, file_preview],
209
+ )
210
+
211
+ def process_and_generate(
212
+ file_obj, model_col_val: str, train_col_val: str, data_source_val: str
213
+ ):
214
+ error, graph, preview, markdown, ttr_out, ttr_tr, sim = process_data(
215
+ file_obj, model_col_val, train_col_val, data_source_val
216
+ )
217
+ if error:
218
+ return error, graph, warning_output, preview, None, None, None, None
219
+ return (
220
+ "",
221
+ graph,
222
+ render_pipeline_warning(data_source_val),
223
+ preview,
224
+ markdown,
225
+ ttr_out,
226
+ ttr_tr,
227
+ sim,
228
+ )
229
+
230
+ inputs = [file_input, model_col_input, train_col_input, data_source]
231
+ outputs = [
232
+ gr.Textbox(label="Error", visible=False), # Hidden error output
233
+ pipeline_output,
234
+ warning_output,
235
+ file_preview,
236
+ output_markdown,
237
+ ttr_output_metric,
238
+ ttr_train_metric,
239
+ similarity_metric,
240
+ ]
241
+ for input_component in inputs:
242
+ input_component.change(
243
+ fn=process_and_generate, inputs=inputs, outputs=outputs
244
+ )
245
+
246
+ gr.Markdown("---")
247
+ gr.Markdown(
248
+ """
249
+ **The upcoming Pro version of MADGuard will allow:**
250
+ - Bulk upload of .csv or folder of .txt files
251
+ - Automatic batch scoring and trend visualization
252
+ - Exportable audit reports
253
+
254
+ [**๐Ÿ“ฉ Join the waitlist**](https://forms.gle/your-form-link)
255
+ """
256
+ )
257
+
258
+ return interface
259
+
260
+
261
+ # Launch the Gradio interface
262
  if __name__ == "__main__":
263
+ interface = main_interface()
264
+ interface.launch(share=True)
requirements.txt CHANGED
@@ -1 +1,6 @@
1
- huggingface_hub==0.25.2
 
 
 
 
 
 
1
+ gradio
2
+ pandas
3
+ nltk
4
+ scikit-learn
5
+ sentence-transformers
6
+ graphviz
visuals/__pycache__/layout.cpython-313.pyc ADDED
Binary file (5.96 kB). View file
 
visuals/__pycache__/score_card.cpython-313.pyc ADDED
Binary file (3.08 kB). View file
 
visuals/layout.py ADDED
@@ -0,0 +1,140 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import gradio as gr
2
+ import graphviz
3
+ import pandas as pd
4
+ from typing import Tuple
5
+ import tempfile
6
+ import os
7
+
8
+
9
+ def render_page_header() -> str:
10
+ """Renders the page header."""
11
+ return """
12
+ <div style="text-align: center; margin-top: 1rem;">
13
+ <h1 style="margin-bottom: 0.25rem;">MADGuard AI Explorer</h1>
14
+ <h4 style="color: grey; font-weight: 400;">Robust Diagnostic Mode for RAG Pipeline Feedback Loops</h4>
15
+ </div>
16
+ """
17
+
18
+
19
+ def render_core_reference() -> str:
20
+ """Renders the research reference section."""
21
+ return """
22
+ <details>
23
+ <summary>๐Ÿ“š Research Reference: arXiv:2307.01850</summary>
24
+ <p>
25
+ <b>Self-consuming LLMs: How and When Models Feed Themselves</b> โ€“ <i>Santurkar et al., 2023</i><br>
26
+ This paper introduces and explores <b>Model Autophagy Disorder (MAD)</b> โ€” showing that large language models trained on their own outputs tend to lose performance and accumulate error over time.
27
+
28
+ The paper proposes detection strategies that MADGuard implements, including:
29
+ - Lexical diversity analysis
30
+ - Embedding-based similarity checks
31
+ - Warnings for training loop risks
32
+
33
+ <i>"MADGuard AI Explorer is inspired by key findings from this research, aligning with early warnings and pipeline hygiene practices recommended in their work."</i>
34
+
35
+ ๐Ÿ“Ž <a href="https://arxiv.org/pdf/2307.01850" target="_blank">Read Full Paper (arXiv)</a>
36
+ </p>
37
+ </details>
38
+ """
39
+
40
+
41
+ def render_pipeline(default: str = "Real User Inputs") -> Tuple[gr.Radio, str]:
42
+ """Renders the pipeline input selection."""
43
+ with gr.Row():
44
+ source = gr.Radio(
45
+ ["Real User Inputs", "Synthetic Generated Data"],
46
+ label="Select input source:",
47
+ value=default,
48
+ # Removed 'help' parameter to avoid TypeError with Gradio 4.44.0
49
+ )
50
+ description = """<center>โ„น๏ธ Real User Inputs reflect human queries. Synthetic Generated Data simulates model-generated text being reused for retraining.</center>"""
51
+ return source, description
52
+
53
+
54
+ def render_pipeline_graph(source: str) -> str:
55
+ """Generates a graph of the RAG pipeline and returns the image file path."""
56
+ dot = graphviz.Digraph(
57
+ graph_attr={"rankdir": "LR", "bgcolor": "transparent"},
58
+ node_attr={
59
+ "style": "filled",
60
+ "fillcolor": "#fefefe",
61
+ "color": "#888888",
62
+ "fontname": "Helvetica",
63
+ "fontsize": "12",
64
+ },
65
+ edge_attr={"color": "#999999"},
66
+ )
67
+ dot.edge("User Query", "Retriever")
68
+ dot.edge("Retriever", "LLM")
69
+ dot.edge("LLM", "Response")
70
+ dot.edge(
71
+ "Response",
72
+ "Retraining Set" if source == "Synthetic Generated Data" else "Embedding Store",
73
+ )
74
+
75
+ # Save to a temporary file and return the file path
76
+ with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as tmp_file:
77
+ output_path = tmp_file.name
78
+ dot.render(filename=output_path, format="png", cleanup=True)
79
+ return output_path + ".png"
80
+
81
+
82
+ def render_pipeline_warning(source: str) -> str:
83
+ """Renders a warning message based on the data source."""
84
+ if source == "Synthetic Generated Data":
85
+ return "<div style='color:red; font-weight:bold;'>โš ๏ธ High loop risk: Model may be learning from its own outputs.</div>"
86
+ else:
87
+ return "<div style='color:green; font-weight:bold;'>โœ… Healthy loop: Using diverse real inputs.</div>"
88
+
89
+
90
+ def render_strategy_alignment() -> str:
91
+ """Renders the strategy alignment table."""
92
+ data = {
93
+ "Strategy from Research": [
94
+ "Lexical redundancy (e.g., n-gram overlap)",
95
+ "Embedding-based similarity scoring",
96
+ "Flagging high similarity for retraining risk",
97
+ "Distinguishing real vs. synthetic data",
98
+ "Tracking degradation over retraining iterations",
99
+ ],
100
+ "Status in MADGuard": [
101
+ "โœ… Implemented via TTR",
102
+ "โœ… Implemented",
103
+ "โœ… Implemented (early warning)",
104
+ "โŒ Not implemented",
105
+ "โŒ Not implemented",
106
+ ],
107
+ "Explanation": [
108
+ "MADGuard uses Type-Token Ratio, a proxy for repetition.",
109
+ "Uses SentenceTransformers + cosine similarity.",
110
+ "Provides a risk score but doesn't block data.",
111
+ "Does not currently track source origin.",
112
+ "No multi-round training history/logs yet.",
113
+ ],
114
+ }
115
+ df = pd.DataFrame(data)
116
+ html = """
117
+ <style>
118
+ table { width: 100%; border-collapse: collapse; }
119
+ th, td { border: 1px solid #ddd; padding: 8px; text-align: left; }
120
+ th { background-color: #f2f2f2; }
121
+ </style>
122
+ <table>
123
+ <thead>
124
+ <tr><th>Strategy from Research</th><th>Status in MADGuard</th><th>Explanation</th></tr>
125
+ </thead>
126
+ <tbody>
127
+ """
128
+ for i in range(len(data["Strategy from Research"])):
129
+ html += f"""
130
+ <tr>
131
+ <td>{data["Strategy from Research"][i]}</td>
132
+ <td>{data["Status in MADGuard"][i]}</td>
133
+ <td>{data["Explanation"][i]}</td>
134
+ </tr>
135
+ """
136
+ html += """
137
+ </tbody>
138
+ </table>
139
+ """
140
+ return html
visuals/score_card.py ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import gradio as gr
2
+ from typing import Tuple
3
+
4
+
5
+ def render_score_card(
6
+ ttr_output: float,
7
+ ttr_train: float,
8
+ similarity: float,
9
+ mad_score: float,
10
+ risk_level: str,
11
+ ) -> Tuple[str, str, str]:
12
+ """Renders the evaluation summary and score details."""
13
+
14
+ color = {"High": "#e57373", "Medium": "#ffb74d", "Low": "#81c784"}[risk_level]
15
+
16
+ risk_explanations = {
17
+ "High": """
18
+ ๐Ÿšจ **High Risk Detected** Your model outputs are **very similar** to your planned training data.
19
+ This suggests a **strong feedback loop**, meaning the model is likely to reinforce existing patterns rather than learning new behaviors.
20
+ **What You Can Do**:
21
+ - Replace synthetic data with more **diverse real user input** - Use **paraphrasing techniques** before reuse
22
+ - Add **augmentation or filtering** before retraining
23
+ """,
24
+ "Medium": """
25
+ ๐ŸŸ  **Moderate Risk Identified** There is some overlap between your outputs and training content.
26
+ Your model may partially reinforce existing phrasing patterns.
27
+ **Suggestions**:
28
+ - Mix synthetic and real inputs carefully
29
+ - Monitor training logs for semantic redundancy
30
+ """,
31
+ "Low": """
32
+ ๐ŸŸข **Low Risk Score** Your model output and training data appear **diverse** and distinct.
33
+ This is a good sign that your model is learning from **new and varied sources**.
34
+ **Youโ€™re on the right track!**
35
+ """,
36
+ }
37
+
38
+ summary = f"""
39
+ ### ๐Ÿ” Evaluation Summary
40
+
41
+ **Lexical Diversity (Output):** {ttr_output:.2f}
42
+ TTR = unique words / total words
43
+
44
+ **Lexical Diversity (Training Set):** {ttr_train:.2f}
45
+ Broader content = higher TTR
46
+
47
+ **Semantic Similarity (Cosine):** {similarity:.2f}
48
+ Cosine similarity between embeddings
49
+
50
+ <div style="padding: 1rem; background-color: #fdfdfd; border-left: 6px solid {color}; border-radius: 6px;">
51
+ <strong>MAD Risk Score:</strong> {mad_score:.2f} โ†’ <span style='color: {color}; font-weight: bold;'>{risk_level} Risk</span>
52
+ </div>
53
+ <div style='margin-top: 0.5rem; width: 100%; background: #e0e0e0; border-radius: 10px; height: 16px;'>
54
+ <div style='width: {mad_score * 100:.0f}%; background: {color}; height: 100%; border-radius: 10px;'></div>
55
+ </div>
56
+ """
57
+
58
+ details = f"""
59
+ <details>
60
+ <summary>๐Ÿ“Š Score Breakdown</summary>
61
+ TTR Component (0.3 ร— (1 - TTR)): {(1 - ttr_output) * 0.3:.2f}
62
+ Similarity Component (0.7 ร— Cosine): {similarity * 0.7:.2f}
63
+ MAD Score = 0.3 ร— (1 - TTR) + 0.7 ร— Semantic Similarity
64
+ </details>
65
+ """
66
+
67
+ explanation = f"""
68
+ <details>
69
+ <summary>๐Ÿ” What does this score mean?</summary>
70
+ {risk_explanations[risk_level]}
71
+ </details>
72
+ """
73
+
74
+ return summary, details, explanation