Upload 19 files

README.md

@@ -4,12 +4,12 @@ emoji: 📚
 colorFrom: blue
 colorTo: indigo
 sdk: gradio
-sdk_version: 5.29.1
+sdk_version: 5.29.0
 python_version: 3.11
 app_file: app.py
 models:
-- buddhist-nlp/buddhist-sentence-similarity
-license: afl-3.0
+  - buddhist-nlp/buddhist-sentence-similarity
+  - fasttext-tibetan
 ---
 
 # Tibetan Text Metrics Web App
@@ -29,17 +29,55 @@ The Tibetan Text Metrics project aims to provide quantitative methods for assess
 -   **Easy File Upload**: Upload one or more Tibetan `.txt` files directly through the browser.
 -   **Automatic Segmentation**: Uses Tibetan section markers (e.g., `༈`) to automatically split texts into comparable chapters or sections.
 -   **Core Metrics Computed**:
-    -   **Jaccard Similarity (%)**: Measures vocabulary overlap between segments. *Common Tibetan stopwords are filtered out to focus on meaningful lexical similarity.*
+    -   **Jaccard Similarity (%)**: Measures vocabulary overlap between segments. *Common Tibetan stopwords can be filtered out to focus on meaningful lexical similarity.*
     -   **Normalized Longest Common Subsequence (LCS)**: Identifies the longest shared sequence of words, indicating direct textual parallels.
-    -   **Semantic Similarity (BuddhistNLP)**: Uses the [buddhist-nlp/buddhist-sentence-similarity](https://huggingface.co/buddhist-nlp/buddhist-sentence-similarity) model to compare the contextual meaning of segments. *Note: This metric is experimental and may not perform well for all texts. It is recommended to use it in combination with other metrics for a more comprehensive analysis.*
-    -   **TF-IDF Cosine Similarity**: Highlights texts that share important or characteristic terms by comparing their TF-IDF profiles. *Common Tibetan stopwords are excluded to ensure TF-IDF weights highlight genuinely characteristic terms.*
+    -   **Semantic Similarity**: Uses embedding models to compare the contextual meaning of segments. Users can select between:
+    -   A transformer-based model (buddhist-nlp/buddhist-sentence-similarity) specialized for Buddhist texts (experimental approach)
+    -   The official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors) pre-trained on a large corpus of Tibetan text
+    *Note: This metric works best when combined with other metrics for a more comprehensive analysis.*
+    -   **TF-IDF Cosine Similarity**: Highlights texts that share important or characteristic terms by comparing their TF-IDF profiles. *Common Tibetan stopwords can be excluded to ensure TF-IDF weights highlight genuinely characteristic terms.*
 -   **Handles Long Texts**: Implements automated chunking for semantic similarity to process texts exceeding the model's token limit.
+-   **Model Selection**: Choose from specialized embedding models for semantic similarity analysis:
+    -   **Buddhist-NLP Transformer** (Experimental): Pre-trained model specialized for Buddhist texts
+    -   **FastText**: Uses the official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors) with optimizations specifically for Tibetan language, including botok tokenization and TF-IDF weighted averaging
+-   **Stopword Filtering**: Three levels of filtering for Tibetan words:
+    -   **None**: No filtering, includes all words
+    -   **Standard**: Filters only common particles and punctuation
+    -   **Aggressive**: Filters all function words including particles, pronouns, and auxiliaries
 -   **Interactive Visualizations**:
     -   Heatmaps for Jaccard, LCS, Semantic, and TF-IDF similarity metrics, providing a quick overview of inter-segment relationships.
     -   Bar chart displaying word counts per segment.
+-   **Advanced Interpretation**: Get scholarly insights about your results with a built-in analysis engine that:
+    -   Examines your metrics and provides contextual interpretation of textual relationships
+    -   Generates a dual-layer narrative analysis (scholarly and accessible)
+    -   Identifies patterns across chapters and highlights notable textual relationships
+    -   Connects findings to Tibetan textual studies concepts (transmission lineages, regional variants)
+    -   Suggests questions for further investigation
 -   **Downloadable Results**: Export detailed metrics as a CSV file and save heatmaps as PNG files.
 -   **Simplified Workflow**: No command-line interaction or Python scripting needed for analysis.
 
+## Advanced Features
+
+### Using AI-Powered Analysis
+
+The application includes an "Interpret Results" button that provides scholarly insights about your text similarity metrics. This feature:
+
+1. Uses Mistral 7B Instruct via OpenRouter to analyze your results
+2. Requires an OpenRouter API key (set via environment variable)
+3. The AI will provide a comprehensive scholarly analysis including:
+   - Introduction explaining the texts compared and general observations
+   - Overall patterns across all chapters with visualized trends
+   - Detailed examination of notable chapters (highest/lowest similarity)
+   - Discussion of what different metrics reveal about textual relationships
+   - Conclusions suggesting implications for Tibetan textual scholarship
+   - Specific questions these findings raise for further investigation
+   - Cautionary notes about interpreting perfect matches or zero similarity scores
+
+### Data Processing
+
+- **Automatic Filtering**: The system automatically filters out perfect matches (1.0 across all metrics) that may result from empty cells or identical text comparisons
+- **Robust Analysis**: The system handles edge cases and provides meaningful metrics even with imperfect data
+
 ## Text Segmentation and Best Practices
 
 **Why segment your texts?**
@@ -73,11 +111,52 @@ Feel free to edit this list of stopwords to better suit your needs. The list is
 
 ### The application computes and visualizes the following similarity metrics between corresponding chapters/segments of the uploaded texts:
 
-1.  **Jaccard Similarity (%)**: This metric quantifies the lexical overlap between two text segments by comparing their sets of *unique* words. It essentially answers the question: 'Of all the distinct words found across these two segments, what proportion of them are present in both?' It is calculated as `(Number of common unique words) / (Total number of unique words in both texts combined) * 100`. Jaccard Similarity is insensitive to word order and word frequency; it only cares whether a unique word is present or absent. A higher percentage indicates a greater overlap in the vocabularies used in the two segments.
+1.  **Jaccard Similarity (%)**: This metric quantifies the lexical overlap between two text segments by comparing their sets of *unique* words, optionally **filtering out common Tibetan stopwords**. 
+It essentially answers the question: 'Of all the distinct, meaningful words found across these two segments, what proportion of them are present in both?' 
+It is calculated as `(Number of common unique meaningful words) / (Total number of unique meaningful words in both texts combined) * 100`. 
+Jaccard Similarity is insensitive to word order and word frequency; it only cares whether a unique meaningful word is present or absent. 
+A higher percentage indicates a greater overlap in the significant vocabularies used in the two segments.
+
+**Stopword Filtering**: Three levels of filtering are available:
+- **None**: No filtering, includes all words in the comparison
+- **Standard**: Filters only common particles and punctuation
+- **Aggressive**: Filters all function words including particles, pronouns, and auxiliaries
+
+This helps focus on meaningful content words rather than grammatical elements.
+
 2.  **Normalized LCS (Longest Common Subsequence)**: This metric measures the length of the longest sequence of words that appears in *both* text segments, maintaining their original relative order. Importantly, these words do not need to be directly adjacent (contiguous) in either text. For example, if Text A is '<u>the</u> quick <u>brown</u> fox <u>jumps</u>' and Text B is '<u>the</u> lazy cat and <u>brown</u> dog <u>jumps</u> high', the LCS is 'the brown jumps'. The length of this common subsequence is then normalized (in this tool, by dividing by the length of the longer of the two segments) to provide a score, which is then presented as a percentage. A higher Normalized LCS score suggests more significant shared phrasing, direct textual borrowing, or strong structural parallelism, as it reflects similarities in how ideas are ordered and expressed sequentially.
     *   *Note on Interpretation*: It's possible for Normalized LCS to be higher than Jaccard Similarity. This often happens when texts share a substantial 'narrative backbone' or common ordered phrases (leading to a high LCS), even if they use varied surrounding vocabulary or introduce many unique words not part of these core sequences (which would lower the Jaccard score). LCS highlights this sequential, structural similarity, while Jaccard focuses on the overall shared vocabulary regardless of its arrangement.
-3.  **Semantic Similarity (BuddhistNLP)**: Utilizes the `buddhist-nlp/bodhi-sentence-cased-v1` model to compute the cosine similarity between the semantic embeddings of text segments. This model is fine-tuned for Buddhist studies texts and captures nuances in meaning. For texts exceeding the model's 512-token input limit, an automated chunking strategy is employed: texts are divided into overlapping chunks, each chunk is embedded, and the resulting chunk embeddings are averaged (mean pooling) to produce a single representative vector for the entire segment before comparison.
-4.  **TF-IDF Cosine Similarity**: This metric first calculates Term Frequency-Inverse Document Frequency (TF-IDF) scores for each word in each text segment. TF-IDF gives higher weight to words that are frequent within a particular segment but relatively rare across the entire collection of segments. This helps to identify terms that are characteristic or discriminative for a segment. Each segment is then represented as a vector of these TF-IDF scores. Finally, the cosine similarity is computed between these vectors. A score closer to 1 indicates that the two segments share more of these important, distinguishing terms, suggesting they cover similar specific topics or themes.
+3.  **Semantic Similarity**: Computes the cosine similarity between semantic embeddings of text segments using one of two approaches:
+
+    **a. Transformer-based Model** (Experimental): Pre-trained model that understands contextual relationships between words.
+    - `buddhist-nlp/buddhist-sentence-similarity`: Specialized for Buddhist texts
+    - Processes raw Unicode Tibetan text directly (no special tokenization required)
+    - Note: This is an experimental approach and results may vary with different texts
+
+    **b. FastText Model**: Uses the official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors) pre-trained on a large corpus of Tibetan text. Falls back to a custom model only if the official model cannot be loaded.
+    - Processes Tibetan text using botok tokenization (same as other metrics)
+    - Uses the pre-tokenized words from botok rather than doing its own tokenization
+    - Better for texts with specialized Tibetan vocabulary
+    - More stable results for general Tibetan text comparison
+    - Optimized for Tibetan language with:
+      - Syllable-based tokenization preserving Tibetan syllable markers
+      - TF-IDF weighted averaging for word vectors (distinct from the TF-IDF Cosine Similarity metric)
+      - Enhanced parameters based on Tibetan NLP research
+
+    For texts exceeding the model's token limit, an automated chunking strategy is employed: texts are divided into overlapping chunks, each chunk is embedded, and the resulting embeddings are averaged to produce a single vector for the entire segment.
+4.  **TF-IDF Cosine Similarity**: This metric first calculates Term Frequency-Inverse Document Frequency (TF-IDF) scores for each word in each text segment, optionally **filtering out common Tibetan stopwords**. 
+TF-IDF gives higher weight to words that are frequent within a particular segment but relatively rare across the entire collection of segments. 
+This helps to identify terms that are characteristic or discriminative for a segment. When stopword filtering is enabled, the TF-IDF scores better reflect genuinely significant terms.
+Each segment is then represented as a vector of these TF-IDF scores. 
+Finally, the cosine similarity is computed between these vectors. 
+A score closer to 1 indicates that the two segments share more of these important, distinguishing terms, suggesting they cover similar specific topics or themes.
+
+**Stopword Filtering**: Three levels of filtering are available:
+- **None**: No filtering, includes all words in the comparison
+- **Standard**: Filters only common particles and punctuation
+- **Aggressive**: Filters all function words including particles, pronouns, and auxiliaries
+
+This helps focus on meaningful content words rather than grammatical elements.
 
 ## Getting Started (if run Locally)
 
@@ -113,21 +192,55 @@ Feel free to edit this list of stopwords to better suit your needs. The list is
 ## Usage
 
 1.  **Upload Files**: Use the file upload interface to select one or more `.txt` files containing Tibetan Unicode text.
-2.  **Run Analysis**: Click the "Run Analysis" button.
+2.  **Configure Options**: 
+    - Choose whether to compute semantic similarity
+    - Select an embedding model for semantic analysis
+    - Choose a stopword filtering level (None, Standard, or Aggressive)
+3.  **Run Analysis**: Click the "Run Analysis" button.
 3.  **View Results**:
     -   A preview of the similarity metrics will be displayed.
     -   Download the full results as a CSV file.
-    -   Interactive heatmaps for Jaccard Similarity, Normalized LCS, Semantic Similarity, and TF-IDF Cosine Similarity will be generated.
+    -   Interactive heatmaps for Jaccard Similarity, Normalized LCS, Semantic Similarity, and TF-IDF Cosine Similarity will be generated. All heatmaps use a consistent color scheme where darker colors represent higher similarity.
     -   A bar chart showing word counts per segment will also be available.
     -   Any warnings (e.g., regarding missing chapter markers) will be displayed.
 
+4.  **Get Interpretation** (Optional):
+    -   After running the analysis, click the "Help Interpret Results" button.
+    -   No API key or internet connection required! The system uses a built-in rule-based analysis engine.
+    -   The system will analyze your metrics and provide insights about patterns, relationships, and notable findings in your data.
+    -   This feature helps researchers understand the significance of the metrics and identify interesting textual relationships between chapters.
+
+## Embedding Models
+
+The application offers two specialized approaches for calculating semantic similarity in Tibetan texts:
+
+1. **Buddhist-NLP Transformer** (Default option):
+   - A specialized model fine-tuned for Buddhist text similarity
+   - Provides excellent results for Tibetan Buddhist texts
+   - Pre-trained and ready to use, no training required
+   - Best for general Buddhist terminology and concepts
+
+2. **FastText Model**:
+   - Uses the official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors)
+   - Pre-trained on a large corpus of Tibetan text from Wikipedia and other sources
+   - Falls back to training a custom model on your texts if the official model cannot be loaded
+   - Respects your stopword filtering settings when creating embeddings
+   - Uses simple word vector averaging for stable embeddings
+
+**When to choose FastText**:
+- When you want high-quality word embeddings specifically trained for Tibetan language
+- When you need a model that can handle out-of-vocabulary words through character n-grams
+- When you want to benefit from Facebook's large-scale pre-training on Tibetan text
+- When you need more control over how stopwords affect semantic analysis
+
 ## Structure
 
 -   `app.py` — Gradio web app entry point and UI definition.
 -   `pipeline/` — Modules for file handling, text processing, metrics calculation, and visualization.
     -   `process.py`: Core logic for segmenting texts and orchestrating metric computation.
     -   `metrics.py`: Implementation of Jaccard, LCS, and Semantic Similarity (including chunking).
-    -   `semantic_embedding.py`: Handles loading and using the sentence transformer model.
+    -   `semantic_embedding.py`: Handles loading and using the selected embedding models.
+    -   `fasttext_embedding.py`: Provides functionality for training and using FastText models.
     -   `tokenize.py`: Tibetan text tokenization using `botok`.
     -   `upload.py`: File upload handling (currently minimal).
     -   `visualize.py`: Generates heatmaps and word count plots.
@@ -137,6 +250,22 @@ Feel free to edit this list of stopwords to better suit your needs. The list is
 
 This project is licensed under the Creative Commons Attribution 4.0 International License - see the [LICENSE](../../LICENSE) file in the main project directory for details.
 
+## Research and Acknowledgements
+
+The FastText implementation for Tibetan text has been optimized based on research findings from several studies on Tibetan natural language processing:
+
+1. Di, R., Tashi, N., & Lin, J. (2019). Improving Tibetan Word Segmentation Based on Multi-Features Fusion. *IEEE Access*, 7, 178057-178069.
+   - Informed our syllable-based tokenization approach and the importance of preserving Tibetan syllable markers
+
+2. Tashi, N., Rabgay, T., & Wangchuk, K. (2020). Tibetan Word Segmentation using Syllable-based Maximum Matching with Potential Syllable Merging. *Engineering Applications of Artificial Intelligence*, 93, 103716.
+   - Provided insights on syllable segmentation for Tibetan text processing
+
+3. Tashi, N., Rai, A. K., Mittal, P., & Sharma, A. K. (2018). A Novel Approach to Feature Extraction for Tibetan Text Classification. *Journal of Information Processing Systems*, 14(1), 211-224.
+   - Guided our parameter optimization for FastText, including embedding dimensions and n-gram settings
+
+4. Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching Word Vectors with Subword Information. *Transactions of the Association for Computational Linguistics*, 5, 135-146.
+   - The original FastText paper that introduced the subword-enriched word embeddings we use
+
 ## Citation
 
 If you use this web application or the underlying TTM tool in your research, please cite the main project:
@@ -152,4 +281,4 @@ If you use this web application or the underlying TTM tool in your research, ple
 ```
 
 ---
-For questions or issues specifically regarding the web application, please refer to the main project's [issue tracker](https://github.com/daniel-wojahn/tibetan-text-metrics/issues) or contact Daniel Wojahn.
+For questions or issues specifically regarding the web application, please refer to the main project's [issue tracker](https://github.com/daniel-wojahn/tibetan-text-metrics/issues) or contact Daniel Wojahn.

app.py

@@ -2,7 +2,13 @@ import gradio as gr
 from pathlib import Path
 from pipeline.process import process_texts
 from pipeline.visualize import generate_visualizations, generate_word_count_chart
+from pipeline.llm_service import get_interpretation
 import logging
+import pandas as pd
+from dotenv import load_dotenv
+
+# Load environment variables from .env file
+load_dotenv()
 
 from theme import tibetan_theme
 
@@ -18,7 +24,7 @@ def main_interface():
     ) as demo:
         gr.Markdown(
             """# Tibetan Text Metrics Web App
-<span style='font-size:18px;'>A user-friendly web application for analyzing textual similarities and variations in Tibetan manuscripts, providing a graphical interface to the core functionalities of the [Tibetan Text Metrics (TTM)](https://github.com/daniel-wojahn/tibetan-text-metrics) project.</span>
+<span style='font-size:18px;'>A user-friendly web application for analyzing textual similarities and variations in Tibetan manuscripts, providing a graphical interface to the core functionalities of the [Tibetan Text Metrics (TTM)](https://github.com/daniel-wojahn/tibetan-text-metrics) project. Powered by Mistral 7B via OpenRouter for advanced text analysis.</span>
         """,
             elem_classes="gr-markdown",
         )
@@ -38,6 +44,10 @@ def main_interface():
                         file_types=[".txt"],
                         file_count="multiple",
                     )
+                    gr.Markdown(
+                        "<small>Note: Maximum file size: 10MB per file. For optimal performance, use files under 1MB.</small>",
+                        elem_classes="gr-markdown"
+                    )
             with gr.Column(scale=1, elem_classes="step-column"):
                 with gr.Group():
                     gr.Markdown(
@@ -47,12 +57,39 @@ def main_interface():
                         elem_classes="gr-markdown",
                     )
                     semantic_toggle_radio = gr.Radio(
-                        label="Compute semantic similarity?",
+                        label="Compute semantic similarity? (Experimental)",
                         choices=["Yes", "No"],
                         value="Yes",
                         info="Semantic similarity will be time-consuming. Choose 'No' to speed up analysis if these metrics are not required.",
                         elem_id="semantic-radio-group",
                     )
+                    
+                    model_dropdown = gr.Dropdown(
+                        label="Embedding Model",
+                        choices=[
+                            "buddhist-nlp/buddhist-sentence-similarity",
+                            "fasttext-tibetan"
+                        ],
+                        value="buddhist-nlp/buddhist-sentence-similarity",
+                        info="Select the embedding model for semantic similarity.<br><br>"
+                             "<b>Model information:</b><br>"
+                             "• <a href='https://huggingface.co/buddhist-nlp/buddhist-sentence-similarity' target='_blank'>buddhist-nlp/buddhist-sentence-similarity</a>: Specialized model fine-tuned for Buddhist text similarity.<br>"
+                             "• <b>fasttext-tibetan</b>: Uses the official Facebook FastText Tibetan model pre-trained on a large corpus. If the official model cannot be loaded, it will fall back to training a custom model on your uploaded texts.",
+                        visible=True,
+                        interactive=True
+                    )
+                    
+                    stopwords_dropdown = gr.Dropdown(
+                        label="Stopword Filtering",
+                        choices=[
+                            "None (No filtering)", 
+                            "Standard (Common particles only)", 
+                            "Aggressive (All function words)"
+                        ],
+                        value="Standard (Common particles only)",  # Default
+                        info="Choose how aggressively to filter out common Tibetan particles and function words when calculating similarity. This helps focus on meaningful content words."
+                    )
+
                     process_btn = gr.Button(
                         "Run Analysis", elem_id="run-btn", variant="primary"
                     )
@@ -69,25 +106,66 @@ def main_interface():
         metrics_preview = gr.Dataframe(
             label="Similarity Metrics Preview", interactive=False, visible=True
         )
-        word_count_plot = gr.Plot(label="Word Counts per Segment")
+        
+        # LLM Interpretation components
+        with gr.Row():
+            with gr.Column():
+                output_analysis = gr.Markdown(
+                    "## AI Analysis\n*The AI will analyze your text similarities and provide insights into patterns and relationships. Make sure to set up your OpenRouter API key for this feature.*",
+                    elem_classes="gr-markdown"
+                )
+                
+                # Add the interpret button
+                with gr.Row():
+                    interpret_btn = gr.Button(
+                        "Help Interpret Results",
+                        variant="primary",
+                        elem_id="interpret-btn"
+                    )
+                
+                # About AI Analysis section
+                with gr.Accordion("ℹ️ About AI Analysis", open=False):
+                    gr.Markdown("""
+                    ### AI-Powered Analysis
+                    
+                    The AI analysis is powered by **Mistral 7B Instruct** via the OpenRouter API. To use this feature:
+                    
+                    1. Get an API key from [OpenRouter](https://openrouter.ai/keys)
+                    2. Create a `.env` file in the webapp directory
+                    3. Add: `OPENROUTER_API_KEY=your_api_key_here`
+                    
+                    The AI will automatically analyze your text similarities and provide insights into patterns and relationships.
+                    """)
+                # Create a placeholder message with proper formatting and structure
+                initial_message = """
+## Analysis of Tibetan Text Similarity Metrics
+
+<small>*Click the 'Help Interpret Results' button above to generate an AI-powered analysis of your similarity metrics.*</small>
+"""
+                interpretation_output = gr.Markdown(
+                    value=initial_message,
+                    elem_id="llm-analysis"
+                )
+        
         # Heatmap tabs for each metric
         heatmap_titles = {
-            "Jaccard Similarity (%)": "Jaccard Similarity (%): Higher scores (brighter) mean more shared unique words.",
-            "Normalized LCS": "Normalized LCS: Higher scores (brighter) mean longer shared sequences of words.",
-            "Semantic Similarity (BuddhistNLP)": "Semantic Similarity (BuddhistNLP - using word embeddings/experimental): Higher scores (brighter) mean more similar meanings.",
-            "TF-IDF Cosine Sim": "TF-IDF Cosine Similarity: Higher scores mean texts share more important, distinctive vocabulary.",
+            "Jaccard Similarity (%)": "Jaccard Similarity (%): Higher scores (darker) mean more shared unique words.",
+            "Normalized LCS": "Normalized LCS: Higher scores (darker) mean longer shared sequences of words.",
+            "Semantic Similarity": "Semantic Similarity (using word embeddings/experimental): Higher scores (darker) mean more similar meanings.",
+            "TF-IDF Cosine Sim": "TF-IDF Cosine Similarity: Higher scores (darker) mean texts share more important, distinctive vocabulary.",
+            "Word Counts": "Word Counts: Shows the number of words in each segment after tokenization."
         }
 
         metric_tooltips = {
             "Jaccard Similarity (%)": """
 ### Jaccard Similarity (%)
-This metric quantifies the lexical overlap between two text segments by comparing their sets of *unique* words, after **filtering out common Tibetan stopwords**. 
-It essentially answers the question: 'Of all the distinct, meaningful words found across these two segments, what proportion of them are present in both?' 
-It is calculated as `(Number of common unique meaningful words) / (Total number of unique meaningful words in both texts combined) * 100`. 
-Jaccard Similarity is insensitive to word order and word frequency; it only cares whether a unique meaningful word is present or absent. 
-A higher percentage indicates a greater overlap in the significant vocabularies used in the two segments.
+This metric quantifies the lexical overlap between two text segments by comparing their sets of *unique* words, optionally filtering out common Tibetan stopwords. 
+
+It essentially answers the question: 'Of all the distinct words found across these two segments, what proportion of them are present in both?' It is calculated as `(Number of common unique words) / (Total number of unique words in both texts combined) * 100`. 
 
-**Stopword Filtering**: uses a range of stopwords to filter out common Tibetan words that do not contribute to the semantic content of the text.
+Jaccard Similarity is insensitive to word order and word frequency; it only cares whether a unique word is present or absent. A higher percentage indicates a greater overlap in the vocabularies used in the two segments.
+
+**Stopword Filtering**: When enabled (via the "Filter Stopwords" checkbox), common Tibetan particles and function words are filtered out before comparison. This helps focus on meaningful content words rather than grammatical elements.
 """,
             "Normalized LCS": """
 ### Normalized LCS (Longest Common Subsequence)
@@ -102,39 +180,85 @@ A higher Normalized LCS score suggests more significant shared phrasing, direct
 **Note on Interpretation**: It is possible for Normalized LCS to be higher than Jaccard Similarity. This often happens when texts share a substantial 'narrative backbone' or common ordered phrases (leading to a high LCS), even if they use varied surrounding vocabulary or introduce many unique words not part of these core sequences (which would lower the Jaccard score). LCS highlights this sequential, structural similarity, while Jaccard focuses on the overall shared vocabulary regardless of its arrangement.
 """,
             "Semantic Similarity": """
-### Semantic Similarity (Experimental)
-Utilizes the `<a href="https://huggingface.co/buddhist-nlp/buddhist-sentence-similarity">buddhist-nlp/buddhist-sentence-similarity</a>` model to compute the cosine similarity between the semantic embeddings of text segments. 
-This model is fine-tuned for Buddhist studies texts and captures nuances in meaning. 
-For texts exceeding the model's 512-token input limit, an automated chunking strategy is employed: texts are divided into overlapping chunks, each chunk is embedded, and the resulting chunk embeddings are averaged (mean pooling) to produce a single representative vector for the entire segment before comparison.
+### Semantic Similarity
+Computes the cosine similarity between semantic embeddings of text segments using one of two approaches:
+
+**1. Transformer-based Model** (Experimental): Pre-trained model that understands contextual relationships between words.
+   - `buddhist-nlp/buddhist-sentence-similarity`: Specialized for Buddhist texts
+   - Processes raw Unicode Tibetan text directly (no special tokenization required)
+   - Note: This is an experimental approach and results may vary with different texts
 
-**Note**: This metric is experimental and may not perform well for all texts. It is recommended to use it in combination with other metrics for a more comprehensive analysis.
+**2. FastText Model**: Uses the official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors) pre-trained on a large corpus of Tibetan text. Falls back to a custom model only if the official model cannot be loaded.
+   - Processes Tibetan text using botok tokenization (same as other metrics)
+   - Uses the pre-tokenized words from botok rather than doing its own tokenization
+   - Better for texts with specialized Tibetan vocabulary
+   - More stable results for general Tibetan text comparison
+   - Optimized for Tibetan language with:
+     - Syllable-based tokenization preserving Tibetan syllable markers
+     - TF-IDF weighted averaging for word vectors (distinct from the TF-IDF Cosine Similarity metric)
+     - Enhanced parameters based on Tibetan NLP research
+
+**Chunking for Long Texts**: For texts exceeding the model's token limit, an automated chunking strategy is employed: texts are divided into overlapping chunks, each chunk is embedded, and the resulting embeddings are averaged to produce a single vector for the entire segment.
+
+**Stopword Filtering**: When enabled (via the "Filter Stopwords" checkbox), common Tibetan particles and function words are filtered out before computing embeddings. This helps focus on meaningful content words. Transformer models process the full text regardless of stopword filtering setting.
+
+**Note**: This metric works best when combined with other metrics for a more comprehensive analysis.
 """,
             "TF-IDF Cosine Sim": """
 ### TF-IDF Cosine Similarity
-This metric first calculates Term Frequency-Inverse Document Frequency (TF-IDF) scores for each word in each text segment, **after filtering out common Tibetan stopwords**. 
-TF-IDF gives higher weight to words that are frequent within a particular segment but relatively rare across the entire collection of segments. 
-This helps to identify terms that are characteristic or discriminative for a segment. By excluding stopwords, the TF-IDF scores better reflect genuinely significant terms.
-Each segment is then represented as a vector of these TF-IDF scores. 
-Finally, the cosine similarity is computed between these vectors. 
-A score closer to 1 indicates that the two segments share more of these important, distinguishing terms, suggesting they cover similar specific topics or themes.
-
-**Stopword Filtering**: uses a range of stopwords to filter out common Tibetan words that do not contribute to the semantic content of the text.
+This metric calculates Term Frequency-Inverse Document Frequency (TF-IDF) scores for each word in each text segment, optionally filtering out common Tibetan stopwords. 
+
+TF-IDF gives higher weight to words that are frequent within a particular segment but relatively rare across the entire collection of segments. This helps identify terms that are characteristic or discriminative for a segment. When stopword filtering is enabled, the TF-IDF scores better reflect genuinely significant terms by excluding common particles and function words.
+
+Each segment is represented as a vector of these TF-IDF scores, and the cosine similarity is computed between these vectors. A score closer to 1 indicates that the two segments share more important, distinguishing terms, suggesting they cover similar specific topics or themes.
+
+**Stopword Filtering**: When enabled (via the "Filter Stopwords" checkbox), common Tibetan particles and function words are filtered out. This can be toggled on/off to compare results with and without stopwords.
 """,
         }
         heatmap_tabs = {}
         gr.Markdown("## Detailed Metric Analysis", elem_classes="gr-markdown")
+        
         with gr.Tabs(elem_id="heatmap-tab-group"):
+            # Process all metrics including Word Counts in a unified way
             for metric_key, descriptive_title in heatmap_titles.items():
                 with gr.Tab(metric_key):
-                    if metric_key in metric_tooltips:
-                        gr.Markdown(value=metric_tooltips[metric_key])
+                    # Set CSS class based on metric type
+                    if metric_key == "Jaccard Similarity (%)":
+                        css_class = "metric-info-accordion jaccard-info"
+                        accordion_title = "Understanding Vocabulary Overlap"
+                    elif metric_key == "Normalized LCS":
+                        css_class = "metric-info-accordion lcs-info"
+                        accordion_title = "Understanding Sequence Patterns"
+                    elif metric_key == "Semantic Similarity":
+                        css_class = "metric-info-accordion semantic-info"
+                        accordion_title = "Understanding Meaning Similarity"
+                    elif metric_key == "TF-IDF Cosine Sim":
+                        css_class = "metric-info-accordion tfidf-info"
+                        accordion_title = "Understanding Term Importance"
+                    elif metric_key == "Word Counts":
+                        css_class = "metric-info-accordion wordcount-info"
+                        accordion_title = "Understanding Text Length"
                     else:
-                        gr.Markdown(
-                            value=f"### {metric_key}\nDescription not found."
-                        )  # Fallback
-                    heatmap_tabs[metric_key] = gr.Plot(
-                        label=f"Heatmap: {metric_key}", show_label=False
-                    )
+                        css_class = "metric-info-accordion"
+                        accordion_title = f"About {metric_key}"
+                    
+                    # Create the accordion with appropriate content
+                    with gr.Accordion(accordion_title, open=False, elem_classes=css_class):
+                        if metric_key == "Word Counts":
+                            gr.Markdown("""
+                            ### Word Counts per Segment
+                            This chart displays the number of words in each segment of your texts after tokenization.
+                            """)
+                        elif metric_key in metric_tooltips:
+                            gr.Markdown(value=metric_tooltips[metric_key])
+                        else:
+                            gr.Markdown(value=f"### {metric_key}\nDescription not found.")
+                    
+                    # Add the appropriate plot
+                    if metric_key == "Word Counts":
+                        word_count_plot = gr.Plot(label="Word Counts per Segment", show_label=False)
+                    else:
+                        heatmap_tabs[metric_key] = gr.Plot(label=f"Heatmap: {metric_key}", show_label=False)
 
         # The outputs in process_btn.click should use the short metric names as keys for heatmap_tabs
         # e.g., heatmap_tabs["Jaccard Similarity (%)"]
@@ -145,7 +269,25 @@ A score closer to 1 indicates that the two segments share more of these importan
 
         warning_box = gr.Markdown(visible=False)
 
-        def run_pipeline(files, enable_semantic_str):
+        def run_pipeline(files, enable_semantic, model_name, stopwords_option="Aggressive (All function words)", progress=gr.Progress()):
+            """Run the text analysis pipeline on the uploaded files.
+
+            Args:
+                files: List of uploaded files
+                enable_semantic: Whether to compute semantic similarity
+                model_name: Name of the embedding model to use
+                stopwords_option: Stopword filtering level (None, Standard, or Aggressive)
+                progress: Gradio progress indicator
+
+            Returns:
+                Tuple of (metrics_df, heatmap_jaccard, heatmap_lcs, heatmap_semantic, heatmap_tfidf, word_count_fig)
+            """
+            # Initialize progress tracking
+            try:
+                progress_tracker = gr.Progress()
+            except Exception as e:
+                logger.warning(f"Could not initialize progress tracker: {e}")
+                progress_tracker = None
             # Initialize all return values to ensure defined paths for all outputs
             csv_path_res = None
             metrics_preview_df_res = None # Can be a DataFrame or a string message
@@ -172,6 +314,7 @@ A score closer to 1 indicates that the two segments share more of these importan
                     - semantic_heatmap (matplotlib.figure.Figure | None): Semantic similarity heatmap, or None.
                     - warning_update (gr.update): Gradio update for the warning box.
             """
+            # Check if files are provided
             if not files:
                 return (
                     None,
@@ -183,21 +326,76 @@ A score closer to 1 indicates that the two segments share more of these importan
                     None,  # tfidf_heatmap
                     gr.update(value="Please upload files.", visible=True),
                 )
+                
+            # Check file size limits (10MB per file)
+            for file in files:
+                file_size_mb = Path(file.name).stat().st_size / (1024 * 1024)
+                if file_size_mb > 10:
+                    return (
+                        None,
+                        f"File '{Path(file.name).name}' exceeds the 10MB size limit (size: {file_size_mb:.2f}MB).",
+                        None, None, None, None, None,
+                        gr.update(value=f"Error: File '{Path(file.name).name}' exceeds the 10MB size limit.", visible=True),
+                    )
 
             try:
+                if progress_tracker is not None:
+                    try:
+                        progress_tracker(0.1, desc="Preparing files...")
+                    except Exception as e:
+                        logger.warning(f"Progress update error (non-critical): {e}")
+                
+                # Get filenames and read file contents
                 filenames = [
                     Path(file.name).name for file in files
                 ]  # Use Path().name to get just the filename
-                text_data = {
-                    Path(file.name)
-                    .name: Path(file.name)
-                    .read_text(encoding="utf-8-sig")
-                    for file in files
-                }
-
-                enable_semantic_bool = enable_semantic_str == "Yes"
+                text_data = {}
+                
+                # Read files with progress updates
+                for i, file in enumerate(files):
+                    file_path = Path(file.name)
+                    filename = file_path.name
+                    if progress_tracker is not None:
+                        try:
+                            progress_tracker(0.1 + (0.1 * (i / len(files))), desc=f"Reading file: {filename}")
+                        except Exception as e:
+                            logger.warning(f"Progress update error (non-critical): {e}")
+                    
+                    try:
+                        text_data[filename] = file_path.read_text(encoding="utf-8-sig")
+                    except UnicodeDecodeError:
+                        # Try with different encodings if UTF-8 fails
+                        try:
+                            text_data[filename] = file_path.read_text(encoding="utf-16")
+                        except UnicodeDecodeError:
+                            return (
+                                None,
+                                f"Error: Could not decode file '{filename}'. Please ensure it contains valid Tibetan text in UTF-8 or UTF-16 encoding.",
+                                None, None, None, None, None,
+                                gr.update(value=f"Error: Could not decode file '{filename}'.", visible=True),
+                            )
+
+                # Configure semantic similarity
+                enable_semantic_bool = enable_semantic == "Yes"
+                
+                if progress_tracker is not None:
+                    try:
+                        progress_tracker(0.2, desc="Loading model..." if enable_semantic_bool else "Processing text...")
+                    except Exception as e:
+                        logger.warning(f"Progress update error (non-critical): {e}")
+                
+                # Process texts with selected model
+                # Convert stopword option to appropriate parameters
+                use_stopwords = stopwords_option != "None (No filtering)"
+                use_lite_stopwords = stopwords_option == "Standard (Common particles only)"
+                
                 df_results, word_counts_df_data, warning_raw = process_texts(
-                    text_data, filenames, enable_semantic=enable_semantic_bool
+                    text_data, filenames, 
+                    enable_semantic=enable_semantic_bool, 
+                    model_name=model_name,
+                    use_stopwords=use_stopwords,
+                    use_lite_stopwords=use_lite_stopwords,
+                    progress_callback=progress_tracker
                 )
 
                 if df_results.empty:
@@ -209,20 +407,43 @@ A score closer to 1 indicates that the two segments share more of these importan
                     warning_update_res = gr.update(value=warning_message, visible=True)
                     # Results for this case are set, then return
                 else:
+                    # Generate visualizations
+                    if progress_tracker is not None:
+                        try:
+                            progress_tracker(0.8, desc="Generating visualizations...")
+                        except Exception as e:
+                            logger.warning(f"Progress update error (non-critical): {e}")
+                    
                     # heatmap_titles is already defined in the outer scope of main_interface
                     heatmaps_data = generate_visualizations(
                         df_results, descriptive_titles=heatmap_titles
                     )
+                    
+                    # Generate word count chart
+                    if progress_tracker is not None:
+                        try:
+                            progress_tracker(0.9, desc="Creating word count chart...")
+                        except Exception as e:
+                            logger.warning(f"Progress update error (non-critical): {e}")
                     word_count_fig_res = generate_word_count_chart(word_counts_df_data)
+                    
+                    # Save results to CSV
+                    if progress_tracker is not None:
+                        try:
+                            progress_tracker(0.95, desc="Saving results...")
+                        except Exception as e:
+                            logger.warning(f"Progress update error (non-critical): {e}")
                     csv_path_res = "results.csv"
                     df_results.to_csv(csv_path_res, index=False)
+                    
+                    # Prepare final output
                     warning_md = f"**⚠️ Warning:** {warning_raw}" if warning_raw else ""
                     metrics_preview_df_res = df_results.head(10)
 
                     jaccard_heatmap_res = heatmaps_data.get("Jaccard Similarity (%)")
                     lcs_heatmap_res = heatmaps_data.get("Normalized LCS")
                     semantic_heatmap_res = heatmaps_data.get(
-                        "Semantic Similarity (BuddhistNLP)"
+                        "Semantic Similarity"
                     )
                     tfidf_heatmap_res = heatmaps_data.get("TF-IDF Cosine Sim")
                     warning_update_res = gr.update(
@@ -244,26 +465,68 @@ A score closer to 1 indicates that the two segments share more of these importan
                 lcs_heatmap_res,
                 semantic_heatmap_res,
                 tfidf_heatmap_res,
-                warning_update_res,
+                warning_update_res
             )
 
+        # Function to interpret results using LLM
+        def interpret_results(csv_path, progress=gr.Progress()):
+            try:
+                if not csv_path or not Path(csv_path).exists():
+                    return "Please run the analysis first to generate results."
+                
+                # Read the CSV file
+                df_results = pd.read_csv(csv_path)
+                
+                # Show detailed progress messages with percentages
+                progress(0, desc="Preparing data for analysis...")
+                progress(0.1, desc="Analyzing similarity patterns...")
+                progress(0.2, desc="Connecting to Mistral 7B via OpenRouter...")
+                
+                # Get interpretation from LLM (using OpenRouter API)
+                progress(0.3, desc="Generating scholarly interpretation (this may take 20-40 seconds)...")
+                interpretation = get_interpretation(df_results)
+                
+                # Simulate completion steps
+                progress(0.9, desc="Formatting results...")
+                progress(0.95, desc="Applying scholarly formatting...")
+                
+                # Completed
+                progress(1.0, desc="Analysis complete!")
+                
+                # Add a timestamp to the interpretation
+                from datetime import datetime
+                timestamp = datetime.now().strftime("%Y-%m-%d %H:%M")
+                interpretation = f"{interpretation}\n\n<small>Analysis generated on {timestamp}</small>"
+                return interpretation
+            except Exception as e:
+                logger.error(f"Error in interpret_results: {e}", exc_info=True)
+                return f"Error interpreting results: {str(e)}"
+        
         process_btn.click(
-            run_pipeline,
-            inputs=[file_input, semantic_toggle_radio],
+            fn=run_pipeline,
+            inputs=[file_input, semantic_toggle_radio, model_dropdown, stopwords_dropdown],
             outputs=[
                 csv_output,
                 metrics_preview,
                 word_count_plot,
                 heatmap_tabs["Jaccard Similarity (%)"],
                 heatmap_tabs["Normalized LCS"],
-                heatmap_tabs["Semantic Similarity (BuddhistNLP)"],
+                heatmap_tabs["Semantic Similarity"],
                 heatmap_tabs["TF-IDF Cosine Sim"],
                 warning_box,
-            ],
+            ]
+        )
+        
+        # Connect the interpret button
+        interpret_btn.click(
+            fn=interpret_results,
+            inputs=[csv_output],
+            outputs=interpretation_output
         )
+        
     return demo
 
 
 if __name__ == "__main__":
     demo = main_interface()
-    demo.launch()
+    demo.launch()

pipeline/fasttext_embedding.py

@@ -0,0 +1,410 @@
+"""
+FastText embedding module for Tibetan text.
+This module provides functions to train and use FastText models for Tibetan text.
+"""
+
+import os
+import math
+import logging
+import numpy as np
+import fasttext
+from typing import List, Optional
+from huggingface_hub import hf_hub_download
+
+# Set up logging
+logger = logging.getLogger(__name__)
+
+# Default parameters optimized for Tibetan
+DEFAULT_DIM = 100
+DEFAULT_EPOCH = 5
+DEFAULT_MIN_COUNT = 5
+DEFAULT_WINDOW = 5
+DEFAULT_MINN = 3
+DEFAULT_MAXN = 6
+DEFAULT_NEG = 5
+
+# Define paths for model storage
+DEFAULT_MODEL_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "models")
+DEFAULT_MODEL_PATH = os.path.join(DEFAULT_MODEL_DIR, "fasttext_model.bin")
+
+# Facebook's official Tibetan FastText model
+FACEBOOK_TIBETAN_MODEL_ID = "facebook/fasttext-bo-vectors"
+FACEBOOK_TIBETAN_MODEL_FILE = "model.bin"
+
+# Create models directory if it doesn't exist
+os.makedirs(DEFAULT_MODEL_DIR, exist_ok=True)
+
+def ensure_dir_exists(directory: str) -> None:
+    """
+    Ensure that a directory exists, creating it if necessary.
+    
+    Args:
+        directory: Directory path to ensure exists
+    """
+    if not os.path.exists(directory):
+        os.makedirs(directory, exist_ok=True)
+
+
+def train_fasttext_model(
+    corpus_path: str,
+    model_path: str = DEFAULT_MODEL_PATH,
+    dim: int = DEFAULT_DIM,
+    epoch: int = DEFAULT_EPOCH,
+    min_count: int = DEFAULT_MIN_COUNT,
+    window: int = DEFAULT_WINDOW,
+    minn: int = DEFAULT_MINN,
+    maxn: int = DEFAULT_MAXN,
+    neg: int = DEFAULT_NEG,
+    model_type: str = "skipgram"
+) -> fasttext.FastText._FastText:
+    """
+    Train a FastText model on Tibetan corpus using optimized parameters.
+    
+    Args:
+        corpus_path: Path to the corpus file
+        model_path: Path where to save the trained model
+        dim: Embedding dimension (default: 300)
+        epoch: Number of training epochs (default: 15)
+        min_count: Minimum count of words (default: 3)
+        window: Size of context window (default: 5)
+        minn: Minimum length of char n-gram (default: 3)
+        maxn: Maximum length of char n-gram (default: 6)
+        neg: Number of negatives in negative sampling (default: 10)
+        model_type: FastText model type ('skipgram' or 'cbow')
+        
+    Returns:
+        Trained FastText model
+    """
+    ensure_dir_exists(os.path.dirname(model_path))
+    
+    logger.info("Training FastText model with %s, dim=%d, epoch=%d, window=%d, minn=%d, maxn=%d...", 
+               model_type, dim, epoch, window, minn, maxn)
+    
+    # Preprocess corpus for Tibetan - segment by syllable points
+    # This is based on research showing syllable segmentation works better for Tibetan
+    try:
+        with open(corpus_path, 'r', encoding='utf-8') as f:
+            content = f.read()
+            
+        # Ensure syllable segmentation by adding spaces after Tibetan syllable markers (if not already present)
+        # This improves model quality for Tibetan text according to research
+        processed_content = content.replace('་', '་ ')
+        
+        # Write back the processed content
+        with open(corpus_path, 'w', encoding='utf-8') as f:
+            f.write(processed_content)
+        
+        logger.info("Preprocessed corpus with syllable segmentation for Tibetan text")
+    except Exception as e:
+        logger.warning("Could not preprocess corpus for syllable segmentation: %s", str(e))
+    
+    # Train the model with optimized parameters
+    if model_type == "skipgram":
+        model = fasttext.train_unsupervised(
+            corpus_path,
+            model="skipgram",
+            dim=dim,
+            epoch=epoch,
+            minCount=min_count,
+            wordNgrams=1,
+            minn=minn,
+            maxn=maxn,
+            neg=neg,
+            window=window
+        )
+    else:  # cbow
+        model = fasttext.train_unsupervised(
+            corpus_path,
+            model="cbow",
+            dim=dim,
+            epoch=epoch,
+            minCount=min_count,
+            wordNgrams=1,
+            minn=minn,
+            maxn=maxn,
+            neg=neg,
+            window=window
+        )
+    
+    # Save the model
+    model.save_model(model_path)
+    logger.info("FastText model trained and saved to %s", model_path)
+    
+    return model
+
+
+def load_fasttext_model(model_path: str = DEFAULT_MODEL_PATH) -> Optional[fasttext.FastText._FastText]:
+    """
+    Load a FastText model from file, with fallback to official Facebook model.
+    
+    Args:
+        model_path: Path to the model file
+        
+    Returns:
+        Loaded FastText model or None if loading fails
+    """
+    try:
+        # First try to load the official Facebook FastText Tibetan model
+        try:
+            # Try to download the official Facebook FastText Tibetan model
+            logger.info("Attempting to download and load official Facebook FastText Tibetan model")
+            facebook_model_path = hf_hub_download(
+                repo_id=FACEBOOK_TIBETAN_MODEL_ID,
+                filename=FACEBOOK_TIBETAN_MODEL_FILE,
+                cache_dir=DEFAULT_MODEL_DIR
+            )
+            logger.info("Loading official Facebook FastText Tibetan model from %s", facebook_model_path)
+            return fasttext.load_model(facebook_model_path)
+        except Exception as e:
+            logger.warning("Could not load official Facebook FastText Tibetan model: %s", str(e))
+            logger.info("Falling back to local model")
+        
+        # Fall back to local model
+        if os.path.exists(model_path):
+            logger.info("Loading local FastText model from %s", model_path)
+            return fasttext.load_model(model_path)
+        else:
+            logger.warning("Model path %s does not exist", model_path)
+            return None
+    except Exception as e:
+        logger.error("Error loading FastText model: %s", str(e))
+        return None
+
+
+def get_text_embedding(
+    text: str,
+    model: fasttext.FastText._FastText,
+    tokenize_fn=None,
+    use_stopwords: bool = True,
+    stopwords_set=None,
+    use_tfidf_weighting: bool = True,  # Enabled by default for better results
+    corpus_token_freq=None
+) -> np.ndarray:
+    """
+    Get embedding for a text using a FastText model with optional TF-IDF weighting.
+    
+    Args:
+        text: Input text
+        model: FastText model
+        tokenize_fn: Optional tokenization function or pre-tokenized list
+        use_stopwords: Whether to filter out stopwords before computing embeddings
+        stopwords_set: Set of stopwords to filter out (if use_stopwords is True)
+        use_tfidf_weighting: Whether to use TF-IDF weighting for averaging word vectors
+        corpus_token_freq: Dictionary of token frequencies across corpus (required for TF-IDF)
+        
+    Returns:
+        Text embedding vector
+    """
+    if not text.strip():
+        return np.zeros(model.get_dimension())
+    
+    # Handle tokenization
+    if tokenize_fn is None:
+        # Simple whitespace tokenization as fallback
+        tokens = text.split()
+    elif isinstance(tokenize_fn, list):
+        # If tokenize_fn is already a list of tokens, use it directly
+        tokens = tokenize_fn
+    elif callable(tokenize_fn):
+        # If tokenize_fn is a function, call it
+        tokens = tokenize_fn(text)
+    else:
+        # If tokenize_fn is something else (like a string), use whitespace tokenization
+        logger.warning(f"Unexpected tokenize_fn type: {type(tokenize_fn)}. Using default whitespace tokenization.")
+        tokens = text.split()
+    
+    # Filter out stopwords if enabled and stopwords_set is provided
+    if use_stopwords and stopwords_set is not None:
+        tokens = [token for token in tokens if token not in stopwords_set]
+    
+    # If all tokens were filtered out as stopwords, return zero vector
+    if not tokens:
+        return np.zeros(model.get_dimension())
+    
+    # Filter out empty tokens
+    tokens = [token for token in tokens if token.strip()]
+    
+    if not tokens:
+        return np.zeros(model.get_dimension())
+    
+    # Calculate TF-IDF weighted average if requested
+    if use_tfidf_weighting and corpus_token_freq is not None:
+        # Calculate term frequencies in this document
+        token_counts = {}
+        for token in tokens:
+            token_counts[token] = token_counts.get(token, 0) + 1
+        
+        # Calculate IDF for each token with improved stability
+        N = sum(corpus_token_freq.values())  # Total number of tokens in corpus
+        N = max(N, 1)  # Ensure N is at least 1 to avoid division by zero
+        
+        # Compute TF-IDF weights with safeguards against extreme values
+        weights = []
+        for token in tokens:
+            # Term frequency in this document
+            tf = token_counts.get(token, 0) / max(len(tokens), 1) if len(tokens) > 0 else 0
+            
+            # Inverse document frequency with smoothing to avoid extreme values
+            token_freq = corpus_token_freq.get(token, 0)
+            idf = math.log((N + 1) / (token_freq + 1)) + 1  # Add 1 for smoothing
+            
+            # TF-IDF weight with bounds to prevent extreme values
+            weight = tf * idf
+            weight = min(max(weight, 0.1), 10.0)  # Limit to reasonable range
+            weights.append(weight)
+        
+        # Normalize weights to sum to 1 with stability checks
+        total_weight = sum(weights)
+        if total_weight > 0:
+            weights = [w / total_weight for w in weights]
+        else:
+            # If all weights are 0, use uniform weights
+            weights = [1.0 / len(tokens) if len(tokens) > 0 else 0 for _ in tokens]
+            
+        # Check for NaN or infinite values and replace with uniform weights if found
+        if any(math.isnan(w) or math.isinf(w) for w in weights):
+            logger.warning("Found NaN or infinite weights in TF-IDF calculation. Using uniform weights instead.")
+            weights = [1.0 / len(tokens) if len(tokens) > 0 else 0 for _ in tokens]
+        
+        # Get vectors for each token and apply weights
+        vectors = [model.get_word_vector(token) for token in tokens]
+        weighted_vectors = [w * v for w, v in zip(weights, vectors)]
+        
+        # Sum the weighted vectors
+        return np.sum(weighted_vectors, axis=0)
+    else:
+        # Simple averaging if TF-IDF is not enabled or corpus frequencies not provided
+        vectors = [model.get_word_vector(token) for token in tokens]
+        return np.mean(vectors, axis=0)
+
+
+def get_batch_embeddings(
+    texts: List[str], 
+    model: fasttext.FastText._FastText,
+    tokenize_fn=None,
+    use_stopwords: bool = True,
+    stopwords_set=None,
+    use_tfidf_weighting: bool = True,  # Enabled by default for better results
+    corpus_token_freq=None
+) -> np.ndarray:
+    """
+    Get embeddings for a batch of texts with optional TF-IDF weighting.
+    
+    Args:
+        texts: List of input texts
+        model: FastText model
+        tokenize_fn: Optional tokenization function or pre-tokenized list of tokens
+        use_stopwords: Whether to filter out stopwords before computing embeddings
+        stopwords_set: Set of stopwords to filter out (if use_stopwords is True)
+        use_tfidf_weighting: Whether to use TF-IDF weighting for averaging word vectors
+        corpus_token_freq: Dictionary of token frequencies across corpus (required for TF-IDF)
+        
+    Returns:
+        Array of text embedding vectors
+    """
+    # If corpus_token_freq is not provided but TF-IDF is requested, build it from the texts
+    if use_tfidf_weighting and corpus_token_freq is None:
+        logger.info("Building corpus token frequency dictionary for TF-IDF weighting")
+        corpus_token_freq = {}
+        
+        # Process each text to build corpus token frequencies
+        for text in texts:
+            if not text.strip():
+                continue
+                
+            # Handle tokenization
+            if tokenize_fn is None:
+                tokens = text.split()
+            elif isinstance(tokenize_fn, list):
+                # In this case, tokenize_fn should be a list of lists (one list of tokens per text)
+                # This is not a common use case, so we'll just use the first one as fallback
+                tokens = tokenize_fn[0] if tokenize_fn else []
+            else:
+                tokens = tokenize_fn(text)
+            
+            # Filter out stopwords if enabled
+            if use_stopwords and stopwords_set is not None:
+                tokens = [token for token in tokens if token not in stopwords_set]
+            
+            # Update corpus token frequencies
+            for token in tokens:
+                if token.strip():  # Skip empty tokens
+                    corpus_token_freq[token] = corpus_token_freq.get(token, 0) + 1
+        
+        logger.info("Built corpus token frequency dictionary with %d unique tokens", len(corpus_token_freq))
+    
+    # Get embeddings for each text
+    embeddings = []
+    for i, text in enumerate(texts):
+        # Handle pre-tokenized input
+        tokens = None
+        if isinstance(tokenize_fn, list):
+            if i < len(tokenize_fn):
+                tokens = tokenize_fn[i]
+        
+        embedding = get_text_embedding(
+            text, 
+            model, 
+            tokenize_fn=tokens,  # Pass the tokens directly, not the function
+            use_stopwords=use_stopwords,
+            stopwords_set=stopwords_set,
+            use_tfidf_weighting=use_tfidf_weighting,
+            corpus_token_freq=corpus_token_freq
+        )
+        embeddings.append(embedding)
+    
+    return np.array(embeddings)
+
+
+def generate_embeddings(
+    texts: List[str],
+    model: fasttext.FastText._FastText,
+    device: str,
+    model_type: str = "sentence_transformer",
+    tokenize_fn=None,
+    use_stopwords: bool = True,
+    use_lite_stopwords: bool = False
+) -> np.ndarray:
+    """
+    Generate embeddings for a list of texts using a FastText model.
+    
+    Args:
+        texts: List of input texts
+        model: FastText model
+        device: Device to use for computation (not used for FastText)
+        model_type: Model type ('sentence_transformer' or 'fasttext')
+        tokenize_fn: Optional tokenization function or pre-tokenized list of tokens
+        use_stopwords: Whether to filter out stopwords
+        use_lite_stopwords: Whether to use a lighter set of stopwords
+        
+    Returns:
+        Array of text embedding vectors
+    """
+    if model_type != "fasttext":
+        logger.warning("Model type %s not supported for FastText. Using FastText anyway.", model_type)
+    
+    # Generate embeddings using FastText
+    try:
+        # Load stopwords if needed
+        stopwords_set = None
+        if use_stopwords:
+            from .tibetan_stopwords import get_stopwords
+            stopwords_set = get_stopwords(use_lite=use_lite_stopwords)
+            logger.info("Loaded %d Tibetan stopwords", len(stopwords_set))
+        
+        # Generate embeddings
+        embeddings = get_batch_embeddings(
+            texts, 
+            model, 
+            tokenize_fn=tokenize_fn,
+            use_stopwords=use_stopwords,
+            stopwords_set=stopwords_set,
+            use_tfidf_weighting=True  # Enable TF-IDF weighting for better results
+        )
+        
+        logger.info("FastText embeddings generated with shape: %s", str(embeddings.shape))
+        return embeddings
+    except Exception as e:
+        logger.error("Error generating FastText embeddings: %s", str(e))
+        # Return empty embeddings as fallback
+        return np.zeros((len(texts), model.get_dimension()))

pipeline/llm_service.py

@@ -0,0 +1,644 @@
+"""
+LLM Service for Tibetan Text Metrics
+
+This module provides a unified interface for analyzing text similarity metrics
+using both LLM-based and rule-based approaches.
+"""
+
+import os
+import json
+import logging
+import requests
+import pandas as pd
+import re
+
+# Set up logging
+logger = logging.getLogger(__name__)
+
+# Try to load environment variables
+ENV_LOADED = False
+try:
+    from dotenv import load_dotenv
+    load_dotenv()
+    ENV_LOADED = True
+except ImportError:
+    logger.warning("python-dotenv not installed. Using system environment variables.")
+
+# Constants
+DEFAULT_MAX_TOKENS = 4000
+DEFAULT_MODEL = "mistralai/mistral-7b-instruct"
+DEFAULT_TEMPERATURE = 0.3
+DEFAULT_TOP_P = 0.9
+
+class LLMService:
+    """
+    Service for analyzing text similarity metrics using LLMs and rule-based methods.
+    """
+    
+    def __init__(self, api_key: str = None):
+        """
+        Initialize the LLM service.
+        
+        Args:
+            api_key: Optional API key for OpenRouter. If not provided, will try to load from environment.
+        """
+        self.api_key = api_key or os.getenv('OPENROUTER_API_KEY')
+        self.model = DEFAULT_MODEL
+        self.temperature = DEFAULT_TEMPERATURE
+        self.top_p = DEFAULT_TOP_P
+    
+    def analyze_similarity(
+        self, 
+        results_df: pd.DataFrame, 
+        use_llm: bool = True,
+    ) -> str:
+        """
+        Analyze similarity metrics using either LLM or rule-based approach.
+        
+        Args:
+            results_df: DataFrame containing similarity metrics
+            use_llm: Whether to use LLM for analysis (falls back to rule-based if False or on error)
+            
+        Returns:
+            str: Analysis of the metrics in markdown format with appropriate fallback messages
+        """
+        # If LLM is disabled, use rule-based analysis
+        if not use_llm:
+            logger.info("LLM analysis disabled. Using rule-based analysis.")
+            return self._analyze_with_rules(results_df)
+            
+        # Try LLM analysis if enabled
+        try:
+            if not self.api_key:
+                raise ValueError("No OpenRouter API key provided. Please set the OPENROUTER_API_KEY environment variable.")
+                
+            logger.info("Attempting LLM-based analysis...")
+            return self._analyze_with_llm(results_df, max_tokens=DEFAULT_MAX_TOKENS)
+            
+        except Exception as e:
+            error_msg = str(e)
+            logger.error(f"Error in LLM analysis: {error_msg}")
+            
+            # Create a user-friendly error message
+            if "payment" in error_msg.lower() or "402" in error_msg:
+                error_note = "OpenRouter API payment required. Falling back to rule-based analysis."
+            elif "invalid" in error_msg.lower() or "401" in error_msg:
+                error_note = "Invalid OpenRouter API key. Falling back to rule-based analysis."
+            elif "rate limit" in error_msg.lower() or "429" in error_msg:
+                error_note = "API rate limit exceeded. Falling back to rule-based analysis."
+            else:
+                error_note = f"LLM analysis failed: {error_msg[:200]}. Falling back to rule-based analysis."
+            
+            # Get rule-based analysis
+            rule_based_analysis = self._analyze_with_rules(results_df)
+            
+            # Combine the error message with the rule-based analysis
+            return f"## Analysis of Tibetan Text Similarity Metrics\n\n*Note: {error_note}*\n\n{rule_based_analysis}"
+    
+    def _prepare_dataframe(self, df: pd.DataFrame) -> pd.DataFrame:
+        """
+        Prepare the DataFrame for analysis.
+        
+        Args:
+            df: Input DataFrame with similarity metrics
+            
+        Returns:
+            pd.DataFrame: Cleaned and prepared DataFrame
+        """
+        # Make a copy to avoid modifying the original
+        df = df.copy()
+        
+        # Clean text columns
+        text_cols = ['Text A', 'Text B']
+        for col in text_cols:
+            if col in df.columns:
+                df[col] = df[col].fillna('Unknown').astype(str)
+                df[col] = df[col].str.replace('.txt$', '', regex=True)
+        
+        # Filter out perfect matches (likely empty cells)
+        metrics_cols = ['Jaccard Similarity (%)', 'Normalized LCS', 'TF-IDF Cosine Sim']
+        if all(col in df.columns for col in metrics_cols):
+            mask = ~((df['Jaccard Similarity (%)'] == 100.0) & 
+                    (df['Normalized LCS'] == 1.0) & 
+                    (df['TF-IDF Cosine Sim'] == 1.0))
+            df = df[mask].copy()
+        
+        return df
+    
+    def _analyze_with_llm(self, df: pd.DataFrame, max_tokens: int) -> str:
+        """
+        Analyze metrics using an LLM via OpenRouter API.
+        
+        Args:
+            df: Prepared DataFrame with metrics
+            max_tokens: Maximum tokens for the response
+            
+        Returns:
+            str: LLM analysis in markdown format
+        """
+        # Prepare the prompt with data and instructions
+        prompt = self._create_llm_prompt(df)
+        
+        try:
+            # Call the LLM API
+            response = self._call_openrouter_api(
+                prompt=prompt,
+                system_message=self._get_system_prompt(),
+                max_tokens=max_tokens,
+                temperature=self.temperature,
+                top_p=self.top_p
+            )
+            
+            # Process and format the response
+            return self._format_llm_response(response, df)
+            
+        except Exception as e:
+            logger.error(f"Error in LLM analysis: {str(e)}")
+            raise
+    
+    def _analyze_with_rules(self, df: pd.DataFrame) -> str:
+        """
+        Analyze metrics using rule-based approach.
+        
+        Args:
+            df: Prepared DataFrame with metrics
+            
+        Returns:
+            str: Rule-based analysis in markdown format
+        """
+        analysis = ["## Tibetan Text Similarity Analysis (Rule-Based)"]
+        
+        # Basic stats
+        text_a_col = 'Text A' if 'Text A' in df.columns else None
+        text_b_col = 'Text B' if 'Text B' in df.columns else None
+        
+        if text_a_col and text_b_col:
+            unique_texts = set(df[text_a_col].unique()) | set(df[text_b_col].unique())
+            analysis.append(f"- **Texts analyzed:** {', '.join(sorted(unique_texts))}")
+        
+        # Analyze each metric
+        metric_analyses = []
+        
+        if 'Jaccard Similarity (%)' in df.columns:
+            jaccard_analysis = self._analyze_jaccard(df)
+            metric_analyses.append(jaccard_analysis)
+            
+        if 'Normalized LCS' in df.columns:
+            lcs_analysis = self._analyze_lcs(df)
+            metric_analyses.append(lcs_analysis)
+            
+        if 'TF-IDF Cosine Sim' in df.columns:
+            tfidf_analysis = self._analyze_tfidf(df)
+            metric_analyses.append(tfidf_analysis)
+        
+        # Add all metric analyses
+        if metric_analyses:
+            analysis.extend(metric_analyses)
+        
+        # Add overall interpretation
+        analysis.append("\n## Overall Interpretation")
+        analysis.append(self._generate_overall_interpretation(df))
+        
+        return "\n\n".join(analysis)
+    
+    def _analyze_jaccard(self, df: pd.DataFrame) -> str:
+        """Analyze Jaccard similarity scores."""
+        jaccard = df['Jaccard Similarity (%)'].dropna()
+        if jaccard.empty:
+            return ""
+            
+        mean_jaccard = jaccard.mean()
+        max_jaccard = jaccard.max()
+        min_jaccard = jaccard.min()
+        
+        analysis = [
+            "### Jaccard Similarity Analysis",
+            f"- **Range:** {min_jaccard:.1f}% to {max_jaccard:.1f}% (mean: {mean_jaccard:.1f}%)"
+        ]
+        
+        # Interpret the scores
+        if mean_jaccard > 60:
+            analysis.append("- **High vocabulary overlap** suggests texts share significant content or are from the same tradition.")
+        elif mean_jaccard > 30:
+            analysis.append("- **Moderate vocabulary overlap** indicates some shared content or themes.")
+        else:
+            analysis.append("- **Low vocabulary overlap** suggests texts are on different topics or from different traditions.")
+        
+        # Add top pairs
+        top_pairs = df.nlargest(3, 'Jaccard Similarity (%)')
+        if not top_pairs.empty:
+            analysis.append("\n**Most similar pairs:**")
+            for _, row in top_pairs.iterrows():
+                text_a = row.get('Text A', 'Text 1')
+                text_b = row.get('Text B', 'Text 2')
+                score = row['Jaccard Similarity (%)']
+                analysis.append(f"- {text_a} ↔ {text_b}: {score:.1f}%")
+        
+        return "\n".join(analysis)
+    
+    def _analyze_lcs(self, df: pd.DataFrame) -> str:
+        """Analyze Longest Common Subsequence scores."""
+        lcs = df['Normalized LCS'].dropna()
+        if lcs.empty:
+            return ""
+            
+        mean_lcs = lcs.mean()
+        max_lcs = lcs.max()
+        min_lcs = lcs.min()
+        
+        analysis = [
+            "### Structural Similarity (LCS) Analysis",
+            f"- **Range:** {min_lcs:.2f} to {max_lcs:.2f} (mean: {mean_lcs:.2f})"
+        ]
+        
+        # Interpret the scores
+        if mean_lcs > 0.7:
+            analysis.append("- **High structural similarity** suggests texts follow similar organizational patterns.")
+        elif mean_lcs > 0.4:
+            analysis.append("- **Moderate structural similarity** indicates some shared organizational elements.")
+        else:
+            analysis.append("- **Low structural similarity** suggests different organizational approaches.")
+        
+        # Add top pairs
+        top_pairs = df.nlargest(3, 'Normalized LCS')
+        if not top_pairs.empty:
+            analysis.append("\n**Most structurally similar pairs:**")
+            for _, row in top_pairs.iterrows():
+                text_a = row.get('Text A', 'Text 1')
+                text_b = row.get('Text B', 'Text 2')
+                score = row['Normalized LCS']
+                analysis.append(f"- {text_a} ↔ {text_b}: {score:.2f}")
+        
+        return "\n".join(analysis)
+    
+    def _analyze_tfidf(self, df: pd.DataFrame) -> str:
+        """Analyze TF-IDF cosine similarity scores."""
+        tfidf = df['TF-IDF Cosine Sim'].dropna()
+        if tfidf.empty:
+            return ""
+            
+        mean_tfidf = tfidf.mean()
+        max_tfidf = tfidf.max()
+        min_tfidf = tfidf.min()
+        
+        analysis = [
+            "### Thematic Similarity (TF-IDF) Analysis",
+            f"- **Range:** {min_tfidf:.2f} to {max_tfidf:.2f} (mean: {mean_tfidf:.2f})"
+        ]
+        
+        # Interpret the scores
+        if mean_tfidf > 0.8:
+            analysis.append("- **High thematic similarity** suggests texts share distinctive terms and concepts.")
+        elif mean_tfidf > 0.5:
+            analysis.append("- **Moderate thematic similarity** indicates some shared distinctive terms.")
+        else:
+            analysis.append("- **Low thematic similarity** suggests different conceptual focuses.")
+        
+        # Add top pairs
+        top_pairs = df.nlargest(3, 'TF-IDF Cosine Sim')
+        if not top_pairs.empty:
+            analysis.append("\n**Most thematically similar pairs:**")
+            for _, row in top_pairs.iterrows():
+                text_a = row.get('Text A', 'Text 1')
+                text_b = row.get('Text B', 'Text 2')
+                score = row['TF-IDF Cosine Sim']
+                analysis.append(f"- {text_a} ↔ {text_b}: {score:.2f}")
+        
+        return "\n".join(analysis)
+    
+    def _generate_overall_interpretation(self, df: pd.DataFrame) -> str:
+        """Generate an overall interpretation of the metrics."""
+        interpretations = []
+        
+        # Get metrics if they exist
+        has_jaccard = 'Jaccard Similarity (%)' in df.columns
+        has_lcs = 'Normalized LCS' in df.columns
+        has_tfidf = 'TF-IDF Cosine Sim' in df.columns
+        
+        # Calculate means for available metrics
+        metrics = {}
+        if has_jaccard:
+            metrics['jaccard'] = df['Jaccard Similarity (%)'].mean()
+        if has_lcs:
+            metrics['lcs'] = df['Normalized LCS'].mean()
+        if has_tfidf:
+            metrics['tfidf'] = df['TF-IDF Cosine Sim'].mean()
+        
+        # Generate interpretation based on metrics
+        if metrics:
+            interpretations.append("Based on the analysis of similarity metrics:")
+            
+            if has_jaccard and metrics['jaccard'] > 60:
+                interpretations.append("- The high Jaccard similarity indicates significant vocabulary overlap between texts, "
+                                     "suggesting they may share common sources or be part of the same textual tradition.")
+            
+            if has_lcs and metrics['lcs'] > 0.7:
+                interpretations.append("- The high LCS score indicates strong structural similarity, "
+                                     "suggesting the texts may follow similar organizational patterns or share common structural elements.")
+            
+            if has_tfidf and metrics['tfidf'] > 0.8:
+                interpretations.append("- The high TF-IDF similarity suggests the texts share distinctive terms and concepts, "
+                                     "indicating they may cover similar topics or themes.")
+            
+            # Add cross-metric interpretations
+            if has_jaccard and has_lcs and metrics['jaccard'] > 60 and metrics['lcs'] > 0.7:
+                interpretations.append("\nThe combination of high Jaccard and LCS similarities strongly suggests "
+                                     "that these texts are closely related, possibly being different versions or "
+                                     "transmissions of the same work or sharing a common source.")
+            
+            if has_tfidf and has_jaccard and metrics['tfidf'] < 0.5 and metrics['jaccard'] > 60:
+                interpretations.append("\nThe high Jaccard but lower TF-IDF similarity suggests that while the texts "
+                                     "share many common words, they may use them in different contexts or with different "
+                                     "meanings, possibly indicating different interpretations of similar material.")
+        
+        # Add general guidance if no specific patterns found
+        if not interpretations:
+            interpretations.append("The analysis did not reveal strong patterns in the similarity metrics. "
+                                 "This could indicate that the texts are either very similar or very different "
+                                 "across all measured dimensions.")
+        
+        return "\n\n".join(interpretations)
+    
+    def _create_llm_prompt(self, df: pd.DataFrame) -> str:
+        """
+        Create a prompt for the LLM based on the DataFrame.
+        
+        Args:
+            df: Prepared DataFrame with metrics
+            
+        Returns:
+            str: Formatted prompt for the LLM
+        """
+        # Format the CSV data for the prompt
+        csv_data = df.to_csv(index=False)
+        
+        # Create the prompt using the user's template
+        prompt = """You are a specialized text analysis interpreter with expertise in Tibetan textual studies. Your task is to analyze text similarity data from a CSV file and create a clear, narrative explanation for scholars who may not have technical expertise.
+
+<CONTEXT>
+This data comes from a text similarity analysis tool designed for various genres of Tibetan sources including historical, religious, literary, and philosophical texts. The tool compares texts using multiple linguistic metrics:
+- Jaccard Similarity (%): Measures word overlap between texts (higher % = more similar)
+- Normalized LCS: Longest Common Subsequence, measuring sequential text patterns
+- Semantic Similarity: Deep meaning comparison using sentence transformers or fasttext
+- TF-IDF Cosine Similarity: Term frequency-inverse document frequency comparison
+The "Chapter" column indicates which chapter/section of the texts is being compared.
+</CONTEXT>
+
+<INSTRUCTIONS>
+1. Begin by identifying the specific texts being compared in the data (e.g., "Japan13.txt vs Dolanji.txt").
+
+2. Create a dual-layer narrative analysis (800-1000 words) that includes:
+   a) A high-level overview of text similarity patterns accessible to non-technical readers
+   b) A more detailed analysis for scholars interested in specific textual relationships
+
+3. In your analysis:
+   - Summarize overall similarity patterns between the texts across all chapters
+   - Identify which chapters show strongest similarities and differences
+   - Explain whether similarities appear to be more lexical (Jaccard, LCS) or conceptual (Semantic)
+   - Interpret what these patterns might suggest about textual relationships, transmission, or variant histories
+   - Note any interesting anomalies (e.g., chapters with high semantic but low lexical similarity)
+
+4. Structure your analysis with:
+   - An introduction explaining the texts compared and general observations
+   - A section on overall patterns across all chapters with visualized trends
+   - A detailed examination of 2-3 notable chapters (highest/lowest similarity)
+   - A discussion of what different metrics reveal about textual relationships
+   - A conclusion suggesting what these patterns might mean for Tibetan textual scholarship
+   - 2-3 specific questions these findings raise for further investigation
+
+5. Connect your analysis to common interests in Tibetan textual studies such as:
+   - Textual transmission and lineages
+   - Regional variants and dialectical differences
+   - Potential historical relationships between texts
+   - Original vs. commentary material identification
+
+6. Consider using a "family tree" analogy to make the textual relationships more intuitive. For example:
+   - Texts with very high similarity (>80%) might be described as "siblings" from the same direct source
+   - Texts with moderate similarity (50-80%) could be "cousins" sharing a common ancestor but with separate development
+   - Texts with low similarity (<50%) might be "distant relatives" with only fundamental connections
+   Use this metaphor if it helps clarify the relationships, but don't force it if another explanation would be clearer.
+
+7. **Important note on perfect or zero similarity matches:**  
+   If you notice that all metrics indicate perfect or near-perfect similarity (for example, scores of 1.0/100 across all metrics for a chapter) or 0 for a complete mismatch, this may not indicate true textual identity or lack thereof. Instead, it likely means both corresponding text cells were empty or contained no content. In these cases, be sure to clarify in your narrative that such results are *artifacts of missing data, not genuine textual matches*, and should be interpreted with caution.
+
+8. Balance scholarly precision with accessibility, explaining technical concepts when necessary while keeping the overall narrative engaging for non-technical readers.
+</INSTRUCTIONS>
+
+Here is the CSV data to analyze:
+[CSV_DATA]
+"""
+        
+        # Replace [CSV_DATA] with the actual CSV data
+        prompt = prompt.replace("[CSV_DATA]", csv_data)
+        
+        return prompt
+    
+    def _get_system_prompt(self) -> str:
+        """Get the system prompt for the LLM."""
+        return """
+        You are a senior scholar of Tibetan Buddhist texts with expertise in textual criticism and 
+        comparative analysis. Your task is to analyze the provided similarity metrics and provide 
+        expert-level insights into the relationships between these Tibetan texts.
+        
+        CRITICAL INSTRUCTIONS:
+        1. Your analysis MUST be grounded in the specific metrics provided
+        2. Always reference actual text names and metric values when making claims
+        3. Focus on what the data shows, not what it might show
+        4. Be precise and avoid vague or generic statements
+        
+        ANALYSIS APPROACH:
+        1. Begin with a brief executive summary of the most significant findings
+        2. Group similar text pairs and explain their relationships
+        3. Highlight any patterns that suggest textual transmission or common sources
+        4. Note any anomalies or unexpected results that merit further investigation
+        5. Provide specific examples from the data to support your analysis
+        
+        TIBETAN TEXT-SPECIFIC GUIDANCE:
+        - Consider the implications of shared vocabulary in the context of Tibetan Buddhist literature
+        - Be aware that high LCS scores might indicate shared liturgical or formulaic language
+        - Note that texts with similar Jaccard but different LCS scores might share content but differ in structure
+        - Consider the possibility of text reuse, commentary traditions, or shared sources
+        
+        Your analysis should be scholarly but accessible, providing clear insights that would be 
+        valuable to researchers studying these texts.
+        """
+    
+    def _call_openrouter_api(
+        self,
+        prompt: str,
+        system_message: str = None,
+        max_tokens: int = None,
+        temperature: float = None,
+        top_p: float = None
+    ) -> str:
+        """
+        Call the OpenRouter API.
+        
+        Args:
+            prompt: The user prompt
+            system_message: Optional system message
+            max_tokens: Maximum tokens for the response
+            temperature: Sampling temperature
+            top_p: Nucleus sampling parameter
+            
+        Returns:
+            str: The API response
+            
+        Raises:
+            ValueError: If API key is missing or invalid
+            requests.exceptions.RequestException: For network-related errors
+            Exception: For other API-related errors
+        """
+        if not self.api_key:
+            error_msg = "OpenRouter API key not provided. Please set the OPENROUTER_API_KEY environment variable."
+            logger.error(error_msg)
+            raise ValueError(error_msg)
+        
+        url = "https://openrouter.ai/api/v1/chat/completions"
+        
+        headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "Content-Type": "application/json",
+            "HTTP-Referer": "https://github.com/daniel-wojahn/tibetan-text-metrics",
+            "X-Title": "Tibetan Text Metrics"
+        }
+        
+        messages = []
+        if system_message:
+            messages.append({"role": "system", "content": system_message})
+        messages.append({"role": "user", "content": prompt})
+        
+        data = {
+            "model": self.model,
+            "messages": messages,
+            "max_tokens": max_tokens or DEFAULT_MAX_TOKENS,
+            "temperature": temperature or self.temperature,
+            "top_p": top_p or self.top_p,
+        }
+        
+        try:
+            logger.info(f"Calling OpenRouter API with model: {self.model}")
+            response = requests.post(url, headers=headers, json=data, timeout=60)
+            
+            # Handle different HTTP status codes
+            if response.status_code == 200:
+                result = response.json()
+                if 'choices' in result and len(result['choices']) > 0:
+                    return result['choices'][0]['message']['content'].strip()
+                else:
+                    error_msg = "Unexpected response format from OpenRouter API"
+                    logger.error(f"{error_msg}: {result}")
+                    raise ValueError(error_msg)
+                    
+            elif response.status_code == 401:
+                error_msg = "Invalid OpenRouter API key. Please check your API key and try again."
+                logger.error(error_msg)
+                raise ValueError(error_msg)
+                
+            elif response.status_code == 402:
+                error_msg = "OpenRouter API payment required. Please check your OpenRouter account balance or billing status."
+                logger.error(error_msg)
+                raise ValueError(error_msg)
+                
+            elif response.status_code == 429:
+                error_msg = "API rate limit exceeded. Please try again later or check your OpenRouter rate limits."
+                logger.error(error_msg)
+                raise ValueError(error_msg)
+                
+            else:
+                error_msg = f"OpenRouter API error: {response.status_code} - {response.text}"
+                logger.error(error_msg)
+                raise Exception(error_msg)
+                
+        except requests.exceptions.RequestException as e:
+            error_msg = f"Failed to connect to OpenRouter API: {str(e)}"
+            logger.error(error_msg)
+            raise Exception(error_msg) from e
+            
+        except json.JSONDecodeError as e:
+            error_msg = f"Failed to parse OpenRouter API response: {str(e)}"
+            logger.error(error_msg)
+            raise Exception(error_msg) from e
+    
+    def _format_llm_response(self, response: str, df: pd.DataFrame) -> str:
+        """
+        Format the LLM response for display.
+        
+        Args:
+            response: Raw LLM response
+            df: Original DataFrame for reference
+            
+        Returns:
+            str: Formatted response with fallback if needed
+        """
+        # Basic validation
+        if not response or len(response) < 100:
+            raise ValueError("Response too short or empty")
+        
+        # Check for garbled output (random numbers, nonsensical patterns)
+        # This is a simple heuristic - look for long sequences of numbers or strange patterns
+        suspicious_patterns = [
+            r'\d{8,}',  # Long number sequences
+            r'[0-9,.]{20,}',  # Long sequences of digits, commas and periods
+            r'[\W]{20,}',  # Long sequences of non-word characters
+        ]
+        
+        for pattern in suspicious_patterns:
+            if re.search(pattern, response):
+                logger.warning(f"Detected potentially garbled output matching pattern: {pattern}")
+                # Don't immediately raise - we'll do a more comprehensive check
+        
+        # Check for content quality - ensure it has expected sections
+        expected_content = [
+            "introduction", "analysis", "similarity", "patterns", "conclusion", "question"
+        ]
+        
+        # Count how many expected content markers we find
+        content_matches = sum(1 for term in expected_content if term.lower() in response.lower())
+        
+        # If we find fewer than 3 expected content markers, it's likely not a good analysis
+        if content_matches < 3:
+            logger.warning(f"LLM response missing expected content sections (found {content_matches}/6)")
+            raise ValueError("Response does not contain expected analysis sections")
+        
+        # Check for text names from the dataset
+        # Extract text names from the Text Pair column
+        text_names = set()
+        if "Text Pair" in df.columns:
+            for pair in df["Text Pair"]:
+                if isinstance(pair, str) and " vs " in pair:
+                    texts = pair.split(" vs ")
+                    text_names.update(texts)
+        
+        # Check if at least some text names appear in the response
+        text_name_matches = sum(1 for name in text_names if name in response)
+        if text_names and text_name_matches == 0:
+            logger.warning("LLM response does not mention any of the text names from the dataset")
+            raise ValueError("Response does not reference any of the analyzed texts")
+        
+        # Ensure basic markdown structure
+        if '##' not in response:
+            response = f"## Analysis of Tibetan Text Similarity\n\n{response}"
+        
+        # Add styling to make the output more readable
+        response = f"<div class='llm-analysis'>\n{response}\n</div>"
+        
+        return response
+
+
+def get_interpretation(results_df: pd.DataFrame, use_llm: bool = True) -> str:
+    """
+    Get an interpretation of the similarity metrics.
+    
+    This is a convenience function that creates an LLMService instance
+    and calls analyze_similarity with default parameters.
+    
+    Args:
+        results_df: DataFrame containing similarity metrics
+        use_llm: Whether to use LLM for analysis (falls back to rule-based if False or on error)
+        
+    Returns:
+        str: Analysis of the metrics in markdown format
+    """
+    service = LLMService()
+    return service.analyze_similarity(results_df, use_llm=use_llm)

pipeline/metrics.py

@@ -7,9 +7,9 @@ import torch
 from .semantic_embedding import generate_embeddings
 from .tokenize import tokenize_texts
 import logging
-from sentence_transformers import SentenceTransformer
 from sklearn.feature_extraction.text import TfidfVectorizer
 from .stopwords_bo import TIBETAN_STOPWORDS, TIBETAN_STOPWORDS_SET
+from .stopwords_lite_bo import TIBETAN_STOPWORDS_LITE, TIBETAN_STOPWORDS_LITE_SET
 
 # Attempt to import the Cython-compiled fast_lcs module
 try:
@@ -107,6 +107,9 @@ def compute_semantic_similarity(
     tokens2: List[str],
     model,
     device,
+    model_type: str = "sentence_transformer",
+    use_stopwords: bool = True,
+    use_lite_stopwords: bool = False,
 ) -> float:
     """Computes semantic similarity using a sentence transformer model, with chunking for long texts."""
     if model is None or device is None:
@@ -122,9 +125,9 @@ def compute_semantic_similarity(
         return 0.0  # Or np.nan, depending on desired behavior for empty inputs
 
     def _get_aggregated_embedding(
-        raw_text_segment: str, botok_tokens: List[str], model_obj, device_str
+        raw_text_segment: str, botok_tokens: List[str], model_obj, device_str, model_type: str = "sentence_transformer", use_stopwords: bool = True, use_lite_stopwords: bool = False
     ) -> torch.Tensor | None:
-        """Helper to get a single embedding for a text, chunking if necessary."""
+        """Helper to get a single embedding for a text, chunking if necessary for transformer models."""
         if (
             not botok_tokens and not raw_text_segment.strip()
         ):  # Check if effectively empty
@@ -132,54 +135,147 @@ def compute_semantic_similarity(
                 f"Text segment is empty or only whitespace: {raw_text_segment[:100]}... Returning None for embedding."
             )
             return None
-
-        if len(botok_tokens) > MAX_TOKENS_PER_CHUNK:
-            logger.info(
-                f"Text segment with ~{len(botok_tokens)} tokens exceeds {MAX_TOKENS_PER_CHUNK}, chunking {raw_text_segment[:30]}..."
-            )
-            # Pass the original raw text and its pre-computed botok tokens to _chunk_text
-            text_chunks = _chunk_text(
-                raw_text_segment, botok_tokens, MAX_TOKENS_PER_CHUNK, CHUNK_OVERLAP
-            )
-            if not text_chunks:
-                logger.warning(
-                    f"Chunking resulted in no chunks for segment: {raw_text_segment[:100]}..."
+            
+        # For FastText, we don't need chunking as it processes tokens directly
+        if model_type == "fasttext":
+            if not raw_text_segment.strip():
+                logger.info(
+                    f"Text segment is empty or only whitespace: {raw_text_segment[:100]}... Returning None for embedding."
                 )
                 return None
-
-            logger.info(
-                f"Generated {len(text_chunks)} chunks for segment: {raw_text_segment[:30]}..."
+                
+            # Pass the raw text, pre-tokenized tokens, and stopword parameters
+            # Wrap the tokens in a list since generate_embeddings expects a list of token lists
+            embedding = generate_embeddings(
+                [raw_text_segment], 
+                model_obj, 
+                device_str, 
+                model_type, 
+                tokenize_fn=[botok_tokens], # Wrap in list since we're passing tokens for one text
+                use_stopwords=use_stopwords,
+                use_lite_stopwords=use_lite_stopwords
             )
-            chunk_embeddings = generate_embeddings(text_chunks, model_obj, device_str)
-
-            if chunk_embeddings is None or chunk_embeddings.nelement() == 0:
+            
+            if embedding is None or embedding.nelement() == 0:
                 logger.error(
-                    f"Failed to generate embeddings for chunks of text: {raw_text_segment[:100]}..."
+                    f"Failed to generate FastText embedding for text: {raw_text_segment[:100]}..."
                 )
                 return None
-            # Mean pooling of chunk embeddings
-            aggregated_embedding = torch.mean(chunk_embeddings, dim=0, keepdim=True)
-            return aggregated_embedding
+            return embedding  # Already [1, embed_dim]
+        
+        # For transformer models, check if all tokens are stopwords when filtering is enabled
+        elif use_stopwords:
+            # Filter stopwords to see if any content remains
+            filtered_tokens = []
+            if use_lite_stopwords:
+                from .stopwords_lite_bo import TIBETAN_STOPWORDS_LITE_SET
+                filtered_tokens = [token for token in botok_tokens if token not in TIBETAN_STOPWORDS_LITE_SET]
+            else:
+                from .stopwords_bo import TIBETAN_STOPWORDS_SET
+                filtered_tokens = [token for token in botok_tokens if token not in TIBETAN_STOPWORDS_SET]
+                
+            # If all tokens were filtered out as stopwords, return zero embedding
+            if not filtered_tokens:
+                logger.info("All tokens in text are stopwords. Returning zero embedding.")
+                # Create a zero tensor with the same dimension as the model's output
+                # For transformer models, typically 384 or 768 dimensions
+                embedding_dim = 384  # Default dimension for MiniLM models
+                return torch.zeros(1, embedding_dim)
+                
+            # Continue with normal processing if content remains after filtering
+            if len(botok_tokens) > MAX_TOKENS_PER_CHUNK:
+                logger.info(
+                    f"Text segment with ~{len(botok_tokens)} tokens exceeds {MAX_TOKENS_PER_CHUNK}, chunking {raw_text_segment[:30]}..."
+                )
+                # Pass the original raw text and its pre-computed botok tokens to _chunk_text
+                text_chunks = _chunk_text(
+                    raw_text_segment, botok_tokens, MAX_TOKENS_PER_CHUNK, CHUNK_OVERLAP
+                )
+                if not text_chunks:
+                    logger.warning(
+                        f"Chunking resulted in no chunks for segment: {raw_text_segment[:100]}..."
+                    )
+                    return None
+
+                logger.info(
+                    f"Generated {len(text_chunks)} chunks for segment: {raw_text_segment[:30]}..."
+                )
+                # Generate embeddings for each chunk using the model
+                chunk_embeddings = generate_embeddings(text_chunks, model_obj, device_str, model_type)
+
+                if chunk_embeddings is None or chunk_embeddings.nelement() == 0:
+                    logger.error(
+                        f"Failed to generate embeddings for chunks of text: {raw_text_segment[:100]}..."
+                    )
+                    return None
+                # Mean pooling of chunk embeddings
+                aggregated_embedding = torch.mean(chunk_embeddings, dim=0, keepdim=True)
+                return aggregated_embedding
+            else:
+                # Text is short enough for transformer model, embed raw text directly
+                if not raw_text_segment.strip():
+                    logger.info(
+                        f"Text segment is empty or only whitespace: {raw_text_segment[:100]}... Returning None for embedding."
+                    )
+                    return None
+
+                embedding = generate_embeddings([raw_text_segment], model_obj, device_str, model_type)
+                if embedding is None or embedding.nelement() == 0:
+                    logger.error(
+                        f"Failed to generate embedding for text: {raw_text_segment[:100]}..."
+                    )
+                    return None
+                return embedding  # Already [1, embed_dim]
         else:
-            # Text is short enough, embed raw text directly as per MEMORY[a777e6ad-11c4-4b90-8e6e-63a923a94432]
-            if not raw_text_segment.strip():
+            # No stopword filtering, proceed with normal processing
+            if len(botok_tokens) > MAX_TOKENS_PER_CHUNK:
                 logger.info(
-                    f"Text segment is empty or only whitespace: {raw_text_segment[:100]}... Returning None for embedding."
+                    f"Text segment with ~{len(botok_tokens)} tokens exceeds {MAX_TOKENS_PER_CHUNK}, chunking {raw_text_segment[:30]}..."
                 )
-                return None
+                # Pass the original raw text and its pre-computed botok tokens to _chunk_text
+                text_chunks = _chunk_text(
+                    raw_text_segment, botok_tokens, MAX_TOKENS_PER_CHUNK, CHUNK_OVERLAP
+                )
+                if not text_chunks:
+                    logger.warning(
+                        f"Chunking resulted in no chunks for segment: {raw_text_segment[:100]}..."
+                    )
+                    return None
 
-            embedding = generate_embeddings([raw_text_segment], model_obj, device_str)
-            if embedding is None or embedding.nelement() == 0:
-                logger.error(
-                    f"Failed to generate embedding for text: {raw_text_segment[:100]}..."
+                logger.info(
+                    f"Generated {len(text_chunks)} chunks for segment: {raw_text_segment[:30]}..."
                 )
-                return None
-            return embedding  # Already [1, embed_dim]
+                # Generate embeddings for each chunk using the model
+                chunk_embeddings = generate_embeddings(text_chunks, model_obj, device_str, model_type)
+
+                if chunk_embeddings is None or chunk_embeddings.nelement() == 0:
+                    logger.error(
+                        f"Failed to generate embeddings for chunks of text: {raw_text_segment[:100]}..."
+                    )
+                    return None
+                # Mean pooling of chunk embeddings
+                aggregated_embedding = torch.mean(chunk_embeddings, dim=0, keepdim=True)
+                return aggregated_embedding
+            else:
+                # Text is short enough for transformer model, embed raw text directly
+                if not raw_text_segment.strip():
+                    logger.info(
+                        f"Text segment is empty or only whitespace: {raw_text_segment[:100]}... Returning None for embedding."
+                    )
+                    return None
+
+                embedding = generate_embeddings([raw_text_segment], model_obj, device_str, model_type)
+                if embedding is None or embedding.nelement() == 0:
+                    logger.error(
+                        f"Failed to generate embedding for text: {raw_text_segment[:100]}..."
+                    )
+                    return None
+                return embedding  # Already [1, embed_dim]
 
     try:
-        # Pass raw text and its pre-computed botok tokens
-        embedding1 = _get_aggregated_embedding(text1_segment, tokens1, model, device)
-        embedding2 = _get_aggregated_embedding(text2_segment, tokens2, model, device)
+        # Pass raw text and its pre-computed botok tokens with stopword preference
+        embedding1 = _get_aggregated_embedding(text1_segment, tokens1, model, device, model_type, use_stopwords, use_lite_stopwords)
+        embedding2 = _get_aggregated_embedding(text2_segment, tokens2, model, device, model_type, use_stopwords, use_lite_stopwords)
 
         if (
             embedding1 is None
@@ -192,6 +288,11 @@ def compute_semantic_similarity(
             )
             return np.nan
 
+        # Check if both embeddings are zero vectors (which happens when all tokens are stopwords)
+        if np.all(embedding1.numpy() == 0) and np.all(embedding2.numpy() == 0):
+            # If both texts contain only stopwords, return 0 similarity
+            return 0.0
+            
         # Cosine similarity expects 2D arrays, embeddings are [1, embed_dim] and on CPU
         similarity = cosine_similarity(embedding1.numpy(), embedding2.numpy())
         return float(similarity[0][0])
@@ -204,7 +305,9 @@ def compute_semantic_similarity(
 
 
 def compute_all_metrics(
-    texts: Dict[str, str], model=None, device=None, enable_semantic: bool = True
+    texts: Dict[str, str], model=None, device=None, enable_semantic: bool = True, 
+    model_type: str = "sentence_transformer", use_stopwords: bool = True,
+    use_lite_stopwords: bool = False
 ) -> pd.DataFrame:
     """
     Computes all selected similarity metrics between pairs of texts.
@@ -220,14 +323,13 @@ def compute_all_metrics(
     Returns:
         pd.DataFrame: A DataFrame where each row contains the metrics for a pair of texts,
                       including 'Text Pair', 'Jaccard Similarity (%)', 'Normalized LCS',
-                      and 'Semantic Similarity (BuddhistNLP)'.
+                      and 'Semantic Similarity'.
     """
     files = list(texts.keys())
     results = []
     # Prepare token lists (always use tokenize_texts for raw Unicode)
     token_lists = {}
     corpus_for_tfidf = []  # For storing space-joined tokens for TF-IDF
-    tibetan_stopwords_set = set() # Initialize for Jaccard (and potentially LCS) filtering
 
     for fname, content in texts.items():
         tokenized_content = tokenize_texts([content])  # Returns a list of lists
@@ -245,36 +347,65 @@ def compute_all_metrics(
 
     # TF-IDF Vectorization and Cosine Similarity Calculation
     if corpus_for_tfidf:
-        # Using a dummy tokenizer and preprocessor as input is already tokenized (as space-separated strings)
-        # and we don't want further case changes or token modifications for Tibetan.
-        # Define Tibetan stopwords. These should match tokens produced by botok.
-        # Tibetan stopwords are now imported from stopwords_bo.py
-
-        vectorizer = TfidfVectorizer(
-            tokenizer=lambda x: x.split(),
-            preprocessor=lambda x: x,
-            token_pattern=None,
-            stop_words=TIBETAN_STOPWORDS
-        )
-        tfidf_matrix = vectorizer.fit_transform(corpus_for_tfidf)
-        # Calculate pairwise cosine similarity on the TF-IDF matrix
-        # This gives a square matrix where cosine_sim_matrix[i, j] is the similarity between doc i and doc j
-        cosine_sim_matrix = cosine_similarity(tfidf_matrix)
+        try:
+            # Using a dummy tokenizer and preprocessor as input is already tokenized (as space-separated strings)
+            # and we don't want further case changes or token modifications for Tibetan.
+            
+            # Select appropriate stopwords list based on user preference
+            if use_stopwords:
+                # Choose between regular and lite stopwords list
+                if use_lite_stopwords:
+                    stopwords_to_use = TIBETAN_STOPWORDS_LITE
+                else:
+                    stopwords_to_use = TIBETAN_STOPWORDS
+            else:
+                # If stopwords are disabled, use an empty list
+                stopwords_to_use = []
+                
+            vectorizer = TfidfVectorizer(
+                tokenizer=lambda x: x.split(),
+                preprocessor=lambda x: x,
+                token_pattern=None,
+                stop_words=stopwords_to_use
+            )
+            tfidf_matrix = vectorizer.fit_transform(corpus_for_tfidf)
+            # Calculate pairwise cosine similarity on the TF-IDF matrix
+            # This gives a square matrix where cosine_sim_matrix[i, j] is the similarity between doc i and doc j
+            cosine_sim_matrix = cosine_similarity(tfidf_matrix)
+        except ValueError as e:
+            if "empty vocabulary" in str(e):
+                # If vocabulary is empty after stopword removal, create a zero matrix
+                n = len(corpus_for_tfidf)
+                cosine_sim_matrix = np.zeros((n, n))
+            else:
+                # Re-raise other ValueError
+                raise
     else:
         # Handle case with no texts or all empty texts
-        cosine_sim_matrix = np.array(
-            [[]]
-        )  # Or some other appropriate empty/default structure
+        n = len(files) if files else 0
+        cosine_sim_matrix = np.zeros((n, n))
 
     for i, j in combinations(range(len(files)), 2):
         f1, f2 = files[i], files[j]
         words1_raw, words2_raw = token_lists[f1], token_lists[f2]
 
-        # Filter stopwords for Jaccard calculation using the imported TIBETAN_STOPWORDS_SET
-        # If TIBETAN_STOPWORDS_SET is empty (e.g., if stopwords_bo.py somehow yields an empty set), 
-        # filtering will have no effect, which is a safe fallback.
-        words1_jaccard = [word for word in words1_raw if word not in TIBETAN_STOPWORDS_SET]
-        words2_jaccard = [word for word in words2_raw if word not in TIBETAN_STOPWORDS_SET]
+        # Select appropriate stopwords set based on user preference
+        if use_stopwords:
+            # Choose between regular and lite stopwords sets
+            if use_lite_stopwords:
+                stopwords_set_to_use = TIBETAN_STOPWORDS_LITE_SET
+            else:
+                stopwords_set_to_use = TIBETAN_STOPWORDS_SET
+        else:
+            # If stopwords are disabled, use an empty set
+            stopwords_set_to_use = set()
+            
+        # Filter stopwords for Jaccard calculation
+        words1_jaccard = [word for word in words1_raw if word not in stopwords_set_to_use]
+        words2_jaccard = [word for word in words2_raw if word not in stopwords_set_to_use]
+
+        # Check if both texts only contain stopwords
+        both_only_stopwords = len(words1_jaccard) == 0 and len(words2_jaccard) == 0
 
         jaccard = (
             len(set(words1_jaccard) & set(words2_jaccard)) / len(set(words1_jaccard) | set(words2_jaccard))
@@ -290,7 +421,7 @@ def compute_all_metrics(
         if enable_semantic:
             # Pass raw texts and their pre-computed botok tokens
             semantic_sim = compute_semantic_similarity(
-                texts[f1], texts[f2], words1_raw, words2_raw, model, device
+                texts[f1], texts[f2], words1_raw, words2_raw, model, device, model_type, use_stopwords, use_lite_stopwords
             )
         else:
             semantic_sim = np.nan
@@ -300,8 +431,9 @@ def compute_all_metrics(
                 "Jaccard Similarity (%)": jaccard_percent,
                 "Normalized LCS": norm_lcs,
                 # Pass tokens1 and tokens2 to compute_semantic_similarity
-                "Semantic Similarity (BuddhistNLP)": semantic_sim,
+                "Semantic Similarity": semantic_sim,
                 "TF-IDF Cosine Sim": (
+                    0.0 if both_only_stopwords else
                     cosine_sim_matrix[i, j]
                     if cosine_sim_matrix.size > 0
                     and i < cosine_sim_matrix.shape[0]

pipeline/process.py

@@ -1,7 +1,7 @@
 import pandas as pd
 from typing import Dict, List, Tuple
 from .metrics import compute_all_metrics
-from .semantic_embedding import get_sentence_transformer_model_and_device
+from .semantic_embedding import get_model_and_device, train_fasttext_model, FASTTEXT_MODEL_ID
 from .tokenize import tokenize_texts
 import logging
 from itertools import combinations
@@ -10,119 +10,345 @@ logger = logging.getLogger(__name__)
 
 
 def process_texts(
-    text_data: Dict[str, str], filenames: List[str], enable_semantic: bool = True
+    text_data: Dict[str, str], 
+    filenames: List[str], 
+    enable_semantic: bool = True,
+    model_name: str = "buddhist-nlp/buddhist-sentence-similarity",
+    use_stopwords: bool = True,
+    use_lite_stopwords: bool = False,
+    progress_callback = None
 ) -> Tuple[pd.DataFrame, pd.DataFrame, str]:
     """
     Processes uploaded texts, segments them by chapter marker, and computes metrics between chapters of different files.
+    
     Args:
         text_data (Dict[str, str]): A dictionary mapping filenames to their content.
         filenames (List[str]): A list of filenames that were uploaded.
+        enable_semantic (bool, optional): Whether to compute semantic similarity metrics. 
+            Requires loading a sentence transformer model, which can be time-consuming. Defaults to True.
+        model_name (str, optional): The name of the sentence transformer model to use for semantic similarity.
+            Must be a valid model identifier on Hugging Face. Defaults to "buddhist-nlp/buddhist-sentence-similarity".
+        use_stopwords (bool, optional): Whether to use stopwords in the metrics calculation. Defaults to True.
+        use_lite_stopwords (bool, optional): Whether to use the lite stopwords list (common particles only)
+            instead of the comprehensive list. Only applies if use_stopwords is True. Defaults to False.
+        progress_callback (callable, optional): A callback function for reporting progress updates.
+            Should accept a float between 0 and 1 and a description string. Defaults to None.
+            
     Returns:
         Tuple[pd.DataFrame, pd.DataFrame, str]:
             - metrics_df: DataFrame with similarity metrics between corresponding chapters of file pairs.
+                Contains columns: 'Text Pair', 'Chapter', 'Jaccard Similarity (%)', 'Normalized LCS',
+                'Semantic Similarity' (if enable_semantic=True), and 'TF-IDF Cosine Sim'.
             - word_counts_df: DataFrame with word counts for each segment (chapter) in each file.
+                Contains columns: 'Filename', 'ChapterNumber', 'SegmentID', 'WordCount'.
             - warning: A string containing any warnings generated during processing (e.g., missing chapter markers).
+    
+    Raises:
+        RuntimeError: If the botok tokenizer fails to initialize.
+        ValueError: If the input files cannot be processed or if metrics computation fails.
     """
+    # Initialize model and device variables
     st_model, st_device = None, None
+    model_warning = ""
+    
+    # Update progress if callback provided
+    if progress_callback is not None:
+        try:
+            progress_callback(0.25, desc="Preparing for text analysis...")
+        except Exception as e:
+            logger.warning(f"Progress callback error (non-critical): {e}")
+            # Continue processing even if progress reporting fails
+    
+    # Load semantic model if enabled
     if enable_semantic:
-        logger.info(
-            "Semantic similarity enabled. Loading sentence transformer model..."
-        )
+        logger.info("Semantic similarity enabled. Loading embedding model...")
         try:
-            st_model, st_device = get_sentence_transformer_model_and_device()
-            logger.info(
-                f"Sentence transformer model loaded successfully on {st_device}."
-            )
+            logger.info("Using model: %s", model_name)
+            
+            # Check if this is a FastText model request
+            if model_name == FASTTEXT_MODEL_ID:
+                # Try to load the official Facebook FastText Tibetan model first
+                if progress_callback is not None:
+                    try:
+                        progress_callback(0.25, desc="Loading official Facebook FastText Tibetan model...")
+                    except Exception as e:
+                        logger.warning("Progress callback error (non-critical): %s", str(e))
+                
+                st_model, st_device, model_type = get_model_and_device(model_id=model_name)
+                
+                # If model is None, we need to train a fallback model
+                if st_model is None:
+                    if progress_callback is not None:
+                        try:
+                            progress_callback(0.25, desc="Official model unavailable. Training fallback FastText model...")
+                        except Exception as e:
+                            logger.warning("Progress callback error (non-critical): %s", str(e))
+                    
+                    # Collect all text data for training
+                    all_texts = list(text_data.values())
+                    
+                    # Train the model with standard parameters for stability
+                    st_model = train_fasttext_model(all_texts, dim=100, epoch=5)
+                    
+                    if progress_callback is not None:
+                        try:
+                            progress_callback(0.3, desc="Fallback FastText model trained successfully")
+                        except Exception as e:
+                            logger.warning("Progress callback error (non-critical): %s", str(e))
+                else:
+                    if progress_callback is not None:
+                        try:
+                            progress_callback(0.3, desc="Official Facebook FastText Tibetan model loaded successfully")
+                        except Exception as e:
+                            logger.warning(f"Progress callback error (non-critical): {e}")
+            else:
+                # For sentence transformers
+                st_model, st_device, model_type = get_model_and_device(model_id=model_name)
+                logger.info(f"Model {model_name} loaded successfully on {st_device}.")
+                
+                if progress_callback is not None:
+                    try:
+                        progress_callback(0.3, desc="Model loaded successfully")
+                    except Exception as e:
+                        logger.warning(f"Progress callback error (non-critical): {e}")
+                
         except Exception as e:
-            logger.error(
-                f"Failed to load sentence transformer model: {e}. Semantic similarity will not be available."
-            )
-            # Optionally, add a warning to the UI if model loading fails
-            # For now, keeping it as a logger.error. UI warning can be added later if desired.
-            pass # Explicitly noting that we are not changing the warning handling for UI here.
+            error_msg = str(e)
+            logger.error(f"Failed to load sentence transformer model: {error_msg}. Semantic similarity will not be available.")
+            
+            # Create a user-friendly warning message
+            if "is not a valid model identifier" in error_msg:
+                model_warning = f"The model '{model_name}' could not be found on Hugging Face. Semantic similarity will not be available."
+            elif "CUDA out of memory" in error_msg:
+                model_warning = "Not enough GPU memory to load the semantic model. Try using a smaller model or disable semantic similarity."
+            else:
+                model_warning = f"Failed to load semantic model: {error_msg}. Semantic similarity will not be available."
+                
+            if progress_callback is not None:
+                try:
+                    progress_callback(0.3, desc="Continuing without semantic model")
+                except Exception as e:
+                    logger.warning(f"Progress callback error (non-critical): {e}")
     else:
         logger.info("Semantic similarity disabled. Skipping model loading.")
+        if progress_callback is not None:
+            try:
+                progress_callback(0.3, desc="Processing text segments")
+            except Exception as e:
+                logger.warning(f"Progress callback error (non-critical): {e}")
 
-    # Detect chapter marker
+    # Detect chapter marker and segment texts
+    if progress_callback is not None:
+        try:
+            progress_callback(0.35, desc="Segmenting texts by chapters...")
+        except Exception as e:
+            logger.warning(f"Progress callback error (non-critical): {e}")
+        
     chapter_marker = "༈"
     fallback = False
     segment_texts = {}
-    for fname in filenames:
+    
+    # Process each file
+    for i, fname in enumerate(filenames):
+        if progress_callback is not None and len(filenames) > 1:
+            try:
+                progress_callback(0.35 + (0.05 * (i / len(filenames))), 
+                                desc=f"Segmenting file {i+1}/{len(filenames)}: {fname}")
+            except Exception as e:
+                logger.warning(f"Progress callback error (non-critical): {e}")
+            
         content = text_data[fname]
+        
+        # Check if content is empty
+        if not content.strip():
+            logger.warning(f"File '{fname}' is empty or contains only whitespace.")
+            continue
+            
+        # Split by chapter marker if present
         if chapter_marker in content:
             segments = [
                 seg.strip() for seg in content.split(chapter_marker) if seg.strip()
             ]
+            
+            # Check if we have valid segments after splitting
+            if not segments:
+                logger.warning(f"File '{fname}' contains chapter markers but no valid text segments.")
+                continue
+                
             for idx, seg in enumerate(segments):
                 seg_id = f"{fname}|chapter {idx+1}"
                 segment_texts[seg_id] = seg
         else:
+            # No chapter markers found, treat entire file as one segment
             seg_id = f"{fname}|chapter 1"
             segment_texts[seg_id] = content.strip()
             fallback = True
-    warning = ""
+            
+    # Generate warning if no chapter markers found
+    warning = model_warning  # Include any model warnings
     if fallback:
-        warning = (
+        chapter_warning = (
             "No chapter marker found in one or more files. "
             "Each file will be treated as a single segment. "
             "For best results, add a unique marker (e.g., ༈) to separate chapters or sections."
         )
+        warning = warning + " " + chapter_warning if warning else chapter_warning
+        
+    # Check if we have any valid segments
+    if not segment_texts:
+        logger.error("No valid text segments found in any of the uploaded files.")
+        return pd.DataFrame(), pd.DataFrame(), "No valid text segments found in the uploaded files. Please check your files and try again."
     # Group chapters by filename (preserving order)
+    if progress_callback is not None:
+        try:
+            progress_callback(0.4, desc="Organizing text segments...")
+        except Exception as e:
+            logger.warning(f"Progress callback error (non-critical): {e}")
+        
     file_to_chapters = {}
     for seg_id in segment_texts:
         fname = seg_id.split("|")[0]
         file_to_chapters.setdefault(fname, []).append(seg_id)
+        
     # For each pair of files, compare corresponding chapters (by index)
+    if progress_callback is not None:
+        try:
+            progress_callback(0.45, desc="Computing similarity metrics...")
+        except Exception as e:
+            logger.warning(f"Progress callback error (non-critical): {e}")
+        
     results = []
     files = list(file_to_chapters.keys())
+    
+    # Check if we have at least two files to compare
+    if len(files) < 2:
+        logger.warning("Need at least two files to compute similarity metrics.")
+        return pd.DataFrame(), pd.DataFrame(), "Need at least two files to compute similarity metrics."
+    
+    # Track total number of comparisons for progress reporting
+    total_comparisons = 0
+    for file1, file2 in combinations(files, 2):
+        chaps1 = file_to_chapters[file1]
+        chaps2 = file_to_chapters[file2]
+        total_comparisons += min(len(chaps1), len(chaps2))
+    
+    # Process each file pair
+    comparison_count = 0
     for file1, file2 in combinations(files, 2):
         chaps1 = file_to_chapters[file1]
         chaps2 = file_to_chapters[file2]
         min_chaps = min(len(chaps1), len(chaps2))
+        
+        if progress_callback is not None:
+            try:
+                progress_callback(0.45, desc=f"Comparing {file1} with {file2}...")
+            except Exception as e:
+                logger.warning(f"Progress callback error (non-critical): {e}")
+            
         for idx in range(min_chaps):
             seg1 = chaps1[idx]
             seg2 = chaps2[idx]
-            # Compute metrics for this chapter pair
-            # Use compute_all_metrics on just these two segments
-            pair_metrics = compute_all_metrics(
-                {seg1: segment_texts[seg1], seg2: segment_texts[seg2]},
-                model=st_model,
-                device=st_device,
-                enable_semantic=enable_semantic,
-            )
-            # Rename 'Text Pair' to show file stems and chapter number
-            # Set Text Pair and Chapter columns
-            pair_metrics.loc[:, "Text Pair"] = f"{file1} vs {file2}"
-            pair_metrics.loc[:, "Chapter"] = idx + 1
-            results.append(pair_metrics)
+            
+            # Update progress
+            comparison_count += 1
+            if progress_callback is not None and total_comparisons > 0:
+                try:
+                    progress_percentage = 0.45 + (0.25 * (comparison_count / total_comparisons))
+                    progress_callback(progress_percentage, 
+                                    desc=f"Computing metrics for chapter {idx+1} ({comparison_count}/{total_comparisons})")
+                except Exception as e:
+                    logger.warning(f"Progress callback error (non-critical): {e}")
+            
+            try:
+                # Compute metrics for this chapter pair
+                pair_metrics = compute_all_metrics(
+                    {seg1: segment_texts[seg1], seg2: segment_texts[seg2]},
+                    model=st_model,
+                    device=st_device,
+                    enable_semantic=enable_semantic,
+                    model_type=model_type if 'model_type' in locals() else "sentence_transformer",
+                    use_stopwords=use_stopwords,
+                    use_lite_stopwords=use_lite_stopwords
+                )
+                
+                # Rename 'Text Pair' to show file stems and chapter number
+                pair_metrics.loc[:, "Text Pair"] = f"{file1} vs {file2}"
+                pair_metrics.loc[:, "Chapter"] = idx + 1
+                results.append(pair_metrics)
+                
+            except Exception as e:
+                logger.error(f"Error computing metrics for {seg1} vs {seg2}: {e}")
+                # Continue with other comparisons instead of failing completely
+                continue
+    
+    # Create the metrics DataFrame
     if results:
         metrics_df = pd.concat(results, ignore_index=True)
     else:
         metrics_df = pd.DataFrame()
+        warning += " No valid metrics could be computed. Please check your files and try again."
 
     # Calculate word counts
+    if progress_callback is not None:
+        try:
+            progress_callback(0.75, desc="Calculating word counts...")
+        except Exception as e:
+            logger.warning(f"Progress callback error (non-critical): {e}")
+        
     word_counts_data = []
-    for seg_id, text_content in segment_texts.items():
+    
+    # Process each segment
+    for i, (seg_id, text_content) in enumerate(segment_texts.items()):
+        # Update progress
+        if progress_callback is not None and len(segment_texts) > 0:
+            try:
+                progress_percentage = 0.75 + (0.15 * (i / len(segment_texts)))
+                progress_callback(progress_percentage, desc=f"Counting words in segment {i+1}/{len(segment_texts)}")
+            except Exception as e:
+                logger.warning(f"Progress callback error (non-critical): {e}")
+            
         fname, chapter_info = seg_id.split("|", 1)
         chapter_num = int(chapter_info.replace("chapter ", ""))
-        # Use botok for accurate word count for raw Tibetan text
-        tokenized_segments = tokenize_texts([text_content])  # Returns a list of lists
-        if tokenized_segments and tokenized_segments[0]:
-            word_count = len(tokenized_segments[0])
-        else:
-            word_count = 0
-        word_counts_data.append(
-            {
-                "Filename": fname.replace(".txt", ""),
-                "ChapterNumber": chapter_num,
-                "SegmentID": seg_id,
-                "WordCount": word_count,
-            }
-        )
+        
+        try:
+            # Use botok for accurate word count for raw Tibetan text
+            tokenized_segments = tokenize_texts([text_content])  # Returns a list of lists
+            if tokenized_segments and tokenized_segments[0]:
+                word_count = len(tokenized_segments[0])
+            else:
+                word_count = 0
+                
+            word_counts_data.append(
+                {
+                    "Filename": fname.replace(".txt", ""),
+                    "ChapterNumber": chapter_num,
+                    "SegmentID": seg_id,
+                    "WordCount": word_count,
+                }
+            )
+        except Exception as e:
+            logger.error(f"Error calculating word count for segment {seg_id}: {e}")
+            # Add entry with 0 word count to maintain consistency
+            word_counts_data.append(
+                {
+                    "Filename": fname.replace(".txt", ""),
+                    "ChapterNumber": chapter_num,
+                    "SegmentID": seg_id,
+                    "WordCount": 0,
+                }
+            )
+    
+    # Create and sort the word counts DataFrame
     word_counts_df = pd.DataFrame(word_counts_data)
     if not word_counts_df.empty:
         word_counts_df = word_counts_df.sort_values(
             by=["Filename", "ChapterNumber"]
         ).reset_index(drop=True)
-
+    
+    if progress_callback is not None:
+        try:
+            progress_callback(0.95, desc="Analysis complete!")
+        except Exception as e:
+            logger.warning(f"Progress callback error (non-critical): {e}")
+        
     return metrics_df, word_counts_df, warning

pipeline/semantic_embedding.py

@@ -1,5 +1,6 @@
 import logging
 import torch
+from typing import List, Any
 from sentence_transformers import SentenceTransformer
 
 # Configure logging
@@ -11,8 +12,11 @@ logger = logging.getLogger(__name__)
 # Define the model ID for the fine-tuned Tibetan MiniLM
 DEFAULT_MODEL_NAME = "buddhist-nlp/buddhist-sentence-similarity"
 
+# FastText model identifier - this is just an internal identifier, not a HuggingFace model ID
+FASTTEXT_MODEL_ID = "fasttext-tibetan"
 
-def get_sentence_transformer_model_and_device(
+
+def get_model_and_device(
     model_id: str = DEFAULT_MODEL_NAME, device_preference: str = "auto"
 ):
     """
@@ -48,48 +52,95 @@ def get_sentence_transformer_model_and_device(
     else:  # Handles explicit "cpu" preference or fallback if preferred is unavailable
         selected_device_str = "cpu"
 
-    logger.info(f"Attempting to use device: {selected_device_str}")
+    logger.info("Attempting to use device: %s", selected_device_str)
 
     try:
-        logger.info(
-            f"Loading Sentence Transformer model: {model_id} on device: {selected_device_str}"
-        )
-        # SentenceTransformer expects a string like 'cuda', 'mps', or 'cpu'
-        model = SentenceTransformer(model_id, device=selected_device_str)
-        logger.info(f"Model {model_id} loaded successfully on {selected_device_str}.")
-        return model, selected_device_str
+        # Check if this is a FastText model request
+        if model_id == FASTTEXT_MODEL_ID:
+            try:
+                # Import here to avoid dependency issues if FastText is not installed
+                import fasttext
+                from .fasttext_embedding import load_fasttext_model
+                
+                # Try to load the FastText model
+                model = load_fasttext_model()
+                
+                if model is None:
+                    error_msg = "Failed to load FastText model. Semantic similarity will not be available."
+                    logger.error(error_msg)
+                    raise Exception(error_msg)
+                    
+                logger.info("FastText model loaded successfully.")
+                # FastText always runs on CPU
+                return model, "cpu", "fasttext"
+            except ImportError:
+                logger.error("FastText module not found. Please install it with 'pip install fasttext'.")
+                raise
+        else:
+            logger.info(
+                "Loading Sentence Transformer model: %s on device: %s",
+                model_id, selected_device_str
+            )
+            # SentenceTransformer expects a string like 'cuda', 'mps', or 'cpu'
+            model = SentenceTransformer(model_id, device=selected_device_str)
+            logger.info("Model %s loaded successfully on %s.", model_id, selected_device_str)
+            return model, selected_device_str, "sentence_transformer"
     except Exception as e:
         logger.error(
-            f"Error loading model {model_id} on device {selected_device_str}: {e}"
+            "Error loading model %s on device %s: %s",
+            model_id, selected_device_str, str(e)
         )
         # Fallback to CPU if the initially selected device (CUDA or MPS) failed
         if selected_device_str != "cpu":
             logger.warning(
-                f"Failed to load model on {selected_device_str}, attempting to load on CPU..."
+                "Failed to load model on %s, attempting to load on CPU...",
+                selected_device_str
             )
             fallback_device_str = "cpu"
             try:
-                model = SentenceTransformer(model_id, device=fallback_device_str)
-                logger.info(
-                    f"Model {model_id} loaded successfully on CPU after fallback."
-                )
-                return model, fallback_device_str
+                # Check if this is a FastText model request during fallback
+                if model_id == FASTTEXT_MODEL_ID:
+                    # Import here to avoid dependency issues if FastText is not installed
+                    from .fasttext_embedding import load_fasttext_model
+                    
+                    # Try to load the FastText model
+                    model = load_fasttext_model()
+                    
+                    if model is None:
+                        logger.error("Failed to load FastText model during fallback. Semantic similarity will not be available.")
+                        raise Exception("Failed to load FastText model. Please check if the model file exists.")
+                        
+                    logger.info("FastText model loaded successfully during fallback.")
+                    # FastText always runs on CPU
+                    return model, "cpu", "fasttext"
+                else:
+                    # Try to load as a sentence transformer
+                    model = SentenceTransformer(model_id, device=fallback_device_str)
+                    logger.info(
+                        "Model %s loaded successfully on CPU after fallback.",
+                        model_id
+                    )
+                    return model, fallback_device_str, "sentence_transformer"
             except Exception as fallback_e:
                 logger.error(
-                    f"Error loading model {model_id} on CPU during fallback: {fallback_e}"
+                    "Error loading model %s on CPU during fallback: %s",
+                    model_id, str(fallback_e)
                 )
                 raise fallback_e  # Re-raise exception if CPU fallback also fails
         raise e  # Re-raise original exception if selected_device_str was already CPU or no fallback attempted
 
 
-def generate_embeddings(texts: list[str], model, device: str):
+def generate_embeddings(texts: List[str], model: Any, device: str, model_type: str = "sentence_transformer", tokenize_fn=None, use_stopwords: bool = True, use_lite_stopwords: bool = False):
     """
-    Generates embeddings for a list of texts using the provided Sentence Transformer model.
+    Generates embeddings for a list of texts using the provided model.
 
     Args:
         texts (list[str]): A list of texts to embed.
-        model: The loaded SentenceTransformer model.
-        device (str): The device the model is on (primarily for logging, model.encode handles device).
+        model: The loaded model (SentenceTransformer or FastText).
+        device (str): The device to use ("cuda", "mps", or "cpu").
+        model_type (str): Type of model ("sentence_transformer" or "fasttext")
+        tokenize_fn: Optional tokenization function or pre-tokenized list for FastText
+        use_stopwords (bool): Whether to filter out stopwords for FastText embeddings
 
     Returns:
         torch.Tensor: A tensor containing the embeddings, moved to CPU.
@@ -102,14 +153,72 @@ def generate_embeddings(texts: list[str], model, device: str):
 
     logger.info(f"Generating embeddings for {len(texts)} texts...")
 
-    # The encode method of SentenceTransformer handles tokenization and pooling internally.
-    # It also manages moving data to the model's device.
-    embeddings = model.encode(texts, convert_to_tensor=True, show_progress_bar=True)
+    if model_type == "fasttext":
+        try:
+            # Import here to avoid dependency issues if FastText is not installed
+            from .fasttext_embedding import get_batch_embeddings
+            from .stopwords_bo import TIBETAN_STOPWORDS_SET
+            
+            # For FastText, get appropriate stopwords set if filtering is enabled
+            stopwords_set = None
+            if use_stopwords:
+                # Choose between regular and lite stopwords sets
+                if use_lite_stopwords:
+                    from .stopwords_lite_bo import TIBETAN_STOPWORDS_LITE_SET
+                    stopwords_set = TIBETAN_STOPWORDS_LITE_SET
+                else:
+                    from .stopwords_bo import TIBETAN_STOPWORDS_SET
+                    stopwords_set = TIBETAN_STOPWORDS_SET
+            
+            # Pass pre-tokenized tokens if available, otherwise pass None
+            # tokenize_fn should be a list of lists (tokens for each text) or None
+            embeddings = get_batch_embeddings(
+                texts, 
+                model, 
+                tokenize_fn=tokenize_fn, 
+                use_stopwords=use_stopwords, 
+                stopwords_set=stopwords_set
+            )
+            logger.info("FastText embeddings generated with shape: %s", str(embeddings.shape))
+            # Convert numpy array to torch tensor for consistency
+            return torch.tensor(embeddings)
+        except ImportError:
+            logger.error("FastText module not found. Please install it with 'pip install fasttext'.")
+            raise
+    else:  # sentence_transformer
+        # The encode method of SentenceTransformer handles tokenization and pooling internally.
+        # It also manages moving data to the model's device.
+        embeddings = model.encode(texts, convert_to_tensor=True, show_progress_bar=True)
+        logger.info("Sentence Transformer embeddings generated with shape: %s", str(embeddings.shape))
+        return (
+            embeddings.cpu()
+        )  # Ensure embeddings are on CPU for consistent further processing
+
 
-    logger.info(f"Embeddings generated with shape: {embeddings.shape}")
-    return (
-        embeddings.cpu()
-    )  # Ensure embeddings are on CPU for consistent further processing
+def train_fasttext_model(corpus_texts: List[str], **kwargs):
+    """
+    Train a FastText model on the provided corpus texts.
+    
+    Args:
+        corpus_texts: List of texts to use for training
+        **kwargs: Additional parameters for training (dim, epoch, etc.)
+        
+    Returns:
+        Trained model and path to the model file
+    """
+    try:
+        from .fasttext_embedding import prepare_corpus_file, train_fasttext_model as train_ft
+        
+        # Prepare corpus file
+        corpus_path = prepare_corpus_file(corpus_texts)
+        
+        # Train the model
+        model = train_ft(corpus_path=corpus_path, **kwargs)
+        
+        return model
+    except ImportError:
+        logger.error("FastText module not found. Please install it with 'pip install fasttext'.")
+        raise
 
 
 if __name__ == "__main__":
@@ -126,19 +235,21 @@ if __name__ == "__main__":
     try:
         # Forcing CPU for this example to avoid potential CUDA issues in diverse environments
         # or if CUDA is not intended for this specific test.
-        st_model, st_device = get_sentence_transformer_model_and_device(
+        model, device, model_type = get_model_and_device(
             device_preference="cpu"  # Explicitly use CPU for this test run
         )
 
-        if st_model:
-            logger.info(f"Test model loaded on device: {st_device}")
-            example_embeddings = generate_embeddings(test_texts, st_model, st_device)
+        if model:
+            logger.info("Test model loaded on device: %s, type: %s", device, model_type)
+            example_embeddings = generate_embeddings(test_texts, model, device, model_type)
             logger.info(
-                f"Generated example embeddings shape: {example_embeddings.shape}"
+                "Generated example embeddings shape: %s",
+                str(example_embeddings.shape)
             )
             if example_embeddings.nelement() > 0:  # Check if tensor is not empty
                 logger.info(
-                    f"First embedding (first 10 dims): {example_embeddings[0][:10]}..."
+                    "First embedding (first 10 dims): %s...",
+                    str(example_embeddings[0][:10])
                 )
             else:
                 logger.info("Generated example embeddings tensor is empty.")
@@ -146,6 +257,6 @@ if __name__ == "__main__":
             logger.error("Failed to load model for example usage.")
 
     except Exception as e:
-        logger.error(f"An error occurred during the example usage: {e}")
+        logger.error("An error occurred during the example usage: %s", str(e))
 
     logger.info("Finished example usage.")

pipeline/stopwords_lite_bo.py

@@ -0,0 +1,34 @@
+# -*- coding: utf-8 -*-
+"""Module for reduced Tibetan stopwords.
+
+This file provides a less aggressive list of Tibetan stopwords for use in the Tibetan Text Metrics application.
+It contains only the most common particles and punctuation that are unlikely to carry significant meaning.
+"""
+
+# Initial set of stopwords with clear categories
+PARTICLES_INITIAL_LITE = [
+    "ཏུ", "གི", "ཀྱི", "གིས", "ཀྱིས", "ཡིས", "ཀྱང", "སྟེ", "ཏེ", "ནོ", "ཏོ",
+    "ཅིང", "ཅིག", "ཅེས", "ཞེས", "གྱིས", "ན",
+]
+
+MARKERS_AND_PUNCTUATION = ["༈", "།", "༎", "༑"]
+
+# Reduced list of particles and suffixes
+MORE_PARTICLES_SUFFIXES_LITE = [
+    "འི་", "དུ་", "གིས་", "ཏེ", "གི་", "ཡི་", "ཀྱི་", "པས་", "ཀྱིས་", "ཡི", "ལ", "ནི་", "ར", "དུ", 
+    "ལས", "གྱིས་", "ས", "ཏེ་", "གྱི་", "དེ", "ཀ་", "སྟེ", "སྟེ་", "ངམ", "ཏོ", "དོ", "དམ་", 
+    "ན", "འམ་", "ལོ", "ཀྱིས", "བས་", "ཤིག", "གིས", "ཀི་", "ཡིས་", "གྱི", "གི"
+]
+
+# Combine all categorized lists
+_ALL_STOPWORDS_CATEGORIZED_LITE = (
+    PARTICLES_INITIAL_LITE +
+    MARKERS_AND_PUNCTUATION +
+    MORE_PARTICLES_SUFFIXES_LITE
+)
+
+# Final flat list of unique stopwords for TfidfVectorizer (as a list)
+TIBETAN_STOPWORDS_LITE = list(set(_ALL_STOPWORDS_CATEGORIZED_LITE))
+
+# Final set of unique stopwords for efficient Jaccard/LCS filtering (as a set)
+TIBETAN_STOPWORDS_LITE_SET = set(TIBETAN_STOPWORDS_LITE)

pipeline/tokenize.py

@@ -1,4 +1,16 @@
-from typing import List
+from typing import List, Dict
+import hashlib
+import logging
+
+# Configure logging
+logger = logging.getLogger(__name__)
+
+# Initialize a cache for tokenization results
+# Using a simple in-memory dictionary with text hash as key
+_tokenization_cache: Dict[str, List[str]] = {}
+
+# Maximum cache size (number of entries)
+MAX_CACHE_SIZE = 1000
 
 try:
     from botok import WordTokenizer
@@ -9,18 +21,40 @@ except ImportError:
     # Handle the case where botok might not be installed,
     # though it's a core dependency for this app.
     BOTOK_TOKENIZER = None
-    print("ERROR: botok library not found. Tokenization will fail.")
+    logger.error("botok library not found. Tokenization will fail.")
     # Optionally, raise an error here if botok is absolutely critical for the app to even start
     # raise ImportError("botok is required for tokenization. Please install it.")
 
 
+def _get_text_hash(text: str) -> str:
+    """
+    Generate a hash for the input text to use as a cache key.
+    
+    Args:
+        text: The input text to hash
+        
+    Returns:
+        A string representation of the MD5 hash of the input text
+    """
+    return hashlib.md5(text.encode('utf-8')).hexdigest()
+
+
 def tokenize_texts(texts: List[str]) -> List[List[str]]:
     """
-    Tokenizes a list of raw Tibetan texts using botok.
+    Tokenizes a list of raw Tibetan texts using botok, with caching for performance.
+    
+    This function maintains an in-memory cache of previously tokenized texts to avoid
+    redundant processing of the same content. The cache uses MD5 hashes of the input
+    texts as keys.
+    
     Args:
-        texts: List of raw text strings.
+        texts: List of raw text strings to tokenize.
+        
     Returns:
         List of tokenized texts (each as a list of tokens).
+        
+    Raises:
+        RuntimeError: If the botok tokenizer failed to initialize.
     """
     if BOTOK_TOKENIZER is None:
         # This case should ideally be handled more gracefully,
@@ -30,9 +64,42 @@ def tokenize_texts(texts: List[str]) -> List[List[str]]:
         )
 
     tokenized_texts_list = []
+    
+    # Process each text
     for text_content in texts:
-        tokens = [
-            w.text for w in BOTOK_TOKENIZER.tokenize(text_content) if w.text.strip()
-        ]
+        # Skip empty texts
+        if not text_content.strip():
+            tokenized_texts_list.append([])
+            continue
+            
+        # Generate hash for cache lookup
+        text_hash = _get_text_hash(text_content)
+        
+        # Check if we have this text in cache
+        if text_hash in _tokenization_cache:
+            # Cache hit - use cached tokens
+            tokens = _tokenization_cache[text_hash]
+            logger.debug(f"Cache hit for text hash {text_hash[:8]}...")
+        else:
+            # Cache miss - tokenize and store in cache
+            try:
+                tokens = [
+                    w.text for w in BOTOK_TOKENIZER.tokenize(text_content) if w.text.strip()
+                ]
+                
+                # Store in cache if not empty
+                if tokens:
+                    # If cache is full, remove a random entry (simple strategy)
+                    if len(_tokenization_cache) >= MAX_CACHE_SIZE:
+                        # Remove first key (oldest if ordered dict, random otherwise)
+                        _tokenization_cache.pop(next(iter(_tokenization_cache)))
+                    
+                    _tokenization_cache[text_hash] = tokens
+                    logger.debug(f"Added tokens to cache with hash {text_hash[:8]}...")
+            except Exception as e:
+                logger.error(f"Error tokenizing text: {e}")
+                tokens = []
+                
         tokenized_texts_list.append(tokens)
+        
     return tokenized_texts_list

pipeline/visualize.py

@@ -25,7 +25,6 @@ def generate_visualizations(metrics_df: pd.DataFrame, descriptive_titles: dict =
 
     # --- Heatmaps for each metric ---
     heatmaps = {}
-    # Using 'Reds' colormap as requested for a red/white gradient.
     # Chapter 1 will be at the top of the Y-axis due to sort_index(ascending=False).
     for metric in metric_cols:
         # Check if all values for this metric are NaN
@@ -41,19 +40,38 @@ def generate_visualizations(metrics_df: pd.DataFrame, descriptive_titles: dict =
             continue
 
         cleaned_columns = [col.replace(".txt", "") for col in pivot.columns]
-        cmap = "Reds"  # Apply 'Reds' colormap to all heatmaps
+        
+        # For consistent interpretation: higher values (more similarity) = darker colors
+        # Using 'Reds' colormap for all metrics (dark red = high similarity)
+        cmap = "Reds"  
+        
+        # Format values for display
         text = [
             [f"{val:.2f}" if pd.notnull(val) else "" for val in row]
             for row in pivot.values
         ]
+        
+        # Create a copy of the pivot data for visualization
+        # For LCS and Semantic Similarity, we need to reverse the color scale
+        # so that higher values (more similarity) are darker
+        viz_values = pivot.values.copy()
+        
+        # Determine if we need to reverse the values for consistent color interpretation
+        # (darker = more similar across all metrics)
+        reverse_colorscale = False
+        
+        # All metrics should have darker colors for higher similarity
+        # No need to reverse values anymore - we'll use the same scale for all
+        
         fig = go.Figure(
             data=go.Heatmap(
-                z=pivot.values,
+                z=viz_values,
                 x=cleaned_columns,
                 y=pivot.index,
                 colorscale=cmap,
-                zmin=float(np.nanmin(pivot.values)),
-                zmax=float(np.nanmax(pivot.values)),
+                reversescale=reverse_colorscale,  # Use the same scale direction for all metrics
+                zmin=float(np.nanmin(viz_values)),
+                zmax=float(np.nanmax(viz_values)),
                 text=text,
                 texttemplate="%{text}",
                 hovertemplate="Chapter %{y}<br>Text Pair: %{x}<br>Value: %{z:.2f}<extra></extra>",

requirements.txt

@@ -1,5 +1,5 @@
 # Core application and UI
-gradio==5.29.1
+gradio
 pandas==2.2.3
 
 # Plotting and visualization
@@ -14,9 +14,19 @@ torch==2.7.0
 transformers==4.51.3
 sentence-transformers==4.1.0
 numba==0.61.2
+fasttext==0.9.2
 
 # Tibetan language processing
 botok==0.9.0
 
 # Build system for Cython
-Cython==3.0.12
+Cython==3.0.12
+
+# HuggingFace integration
+hf_xet==0.1.0
+huggingface_hub
+
+# LLM integration
+python-dotenv==1.0.0
+requests==2.31.0
+tabulate

theme.py

@@ -169,6 +169,85 @@ class TibetanAppTheme(gr.themes.Soft):
                 "padding": "10px 15px !important",
                 "border-bottom": "2px solid transparent !important",
             },
+            
+            # Custom styling for metric accordions
+            ".metric-info-accordion": {
+                "border-left": "4px solid #3B82F6 !important",
+                "margin-bottom": "1rem !important",
+                "background-color": "#F8FAFC !important",
+                "border-radius": "6px !important",
+                "overflow": "hidden !important",
+            },
+            ".jaccard-info": {
+                "border-left-color": "#3B82F6 !important",  # Blue
+            },
+            ".lcs-info": {
+                "border-left-color": "#10B981 !important",  # Green
+            },
+            ".semantic-info": {
+                "border-left-color": "#8B5CF6 !important",  # Purple
+            },
+            ".tfidf-info": {
+                "border-left-color": "#F59E0B !important",  # Amber
+            },
+            ".wordcount-info": {
+                "border-left-color": "#EC4899 !important",  # Pink
+            },
+            
+            # Accordion header styling
+            ".metric-info-accordion > .label-wrap": {
+                "font-weight": "600 !important",
+                "padding": "12px 16px !important",
+                "background-color": "#F1F5F9 !important",
+                "border-bottom": "1px solid #E2E8F0 !important",
+            },
+            
+            # Accordion content styling
+            ".metric-info-accordion > .wrap": {
+                "padding": "16px !important",
+            },
+            
+            # Word count plot styling - full width
+            ".tabs > .tab-content > div[data-testid='tabitem'] > .plot": {
+                "width": "100% !important",
+            },
+            
+            # LLM Analysis styling
+            ".llm-analysis": {
+                "background-color": "#f8f9fa !important",
+                "border-left": "4px solid #3B82F6 !important",
+                "border-radius": "8px !important",
+                "padding": "20px 24px !important",
+                "margin": "16px 0 !important",
+                "box-shadow": "0 2px 8px rgba(0, 0, 0, 0.05) !important",
+            },
+            ".llm-analysis h2": {
+                "color": "#1e40af !important",
+                "font-size": "24px !important",
+                "margin-bottom": "16px !important",
+                "border-bottom": "1px solid #e5e7eb !important",
+                "padding-bottom": "8px !important",
+            },
+            ".llm-analysis h3, .llm-analysis h4": {
+                "color": "#1e3a8a !important",
+                "margin-top": "20px !important",
+                "margin-bottom": "12px !important",
+            },
+            ".llm-analysis p": {
+                "line-height": "1.7 !important",
+                "margin-bottom": "12px !important",
+            },
+            ".llm-analysis ul, .llm-analysis ol": {
+                "margin-left": "24px !important",
+                "margin-bottom": "16px !important",
+            },
+            ".llm-analysis li": {
+                "margin-bottom": "6px !important",
+            },
+            ".llm-analysis strong, .llm-analysis b": {
+                "color": "#1f2937 !important",
+                "font-weight": "600 !important",
+            },
         }
 
     def get_css_string(self) -> str: