Reafactoring of the tokenization pipeline, adjusted fasttext implementation

README.md

@@ -8,7 +8,6 @@ sdk_version: 5.29.0
 python_version: 3.11
 app_file: app.py
 models:
-  - buddhist-nlp/buddhist-sentence-similarity
   - fasttext-tibetan
 ---
 
@@ -32,14 +31,12 @@ The Tibetan Text Metrics project aims to provide quantitative methods for assess
     -   **Jaccard Similarity (%)**: Measures vocabulary overlap between segments. *Common Tibetan stopwords can be filtered out to focus on meaningful lexical similarity.*
     -   **Normalized Longest Common Subsequence (LCS)**: Identifies the longest shared sequence of words, indicating direct textual parallels.
     -   **Semantic Similarity**: Uses embedding models to compare the contextual meaning of segments. Users can select between:
-    -   A transformer-based model (buddhist-nlp/buddhist-sentence-similarity) specialized for Buddhist texts (experimental approach)
-    -   The official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors) pre-trained on a large corpus of Tibetan text
+    -   A FastText model using official Facebook Tibetan vectors with custom `botok` tokenization (recommended approach).
     *Note: This metric works best when combined with other metrics for a more comprehensive analysis.*
     -   **TF-IDF Cosine Similarity**: Highlights texts that share important or characteristic terms by comparing their TF-IDF profiles. *Common Tibetan stopwords can be excluded to ensure TF-IDF weights highlight genuinely characteristic terms.*
 -   **Handles Long Texts**: Implements automated chunking for semantic similarity to process texts exceeding the model's token limit.
--   **Model Selection**: Choose from specialized embedding models for semantic similarity analysis:
-    -   **Buddhist-NLP Transformer** (Experimental): Pre-trained model specialized for Buddhist texts
-    -   **FastText**: Uses the official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors) with optimizations specifically for Tibetan language, including botok tokenization and TF-IDF weighted averaging
+-   **Model Selection**: Semantic similarity analysis uses a FastText model:
+    -   **FastText**: Uses the official Facebook FastText Tibetan model (`cc.bo.300.bin`) with optimizations specifically for Tibetan language, including `botok` tokenization and TF-IDF weighted averaging of word vectors to produce segment embeddings.
 -   **Stopword Filtering**: Three levels of filtering for Tibetan words:
     -   **None**: No filtering, includes all words
     -   **Standard**: Filters only common particles and punctuation
@@ -126,24 +123,8 @@ This helps focus on meaningful content words rather than grammatical elements.
 
 2.  **Normalized LCS (Longest Common Subsequence)**: This metric measures the length of the longest sequence of words that appears in *both* text segments, maintaining their original relative order. Importantly, these words do not need to be directly adjacent (contiguous) in either text. For example, if Text A is '<u>the</u> quick <u>brown</u> fox <u>jumps</u>' and Text B is '<u>the</u> lazy cat and <u>brown</u> dog <u>jumps</u> high', the LCS is 'the brown jumps'. The length of this common subsequence is then normalized (in this tool, by dividing by the length of the longer of the two segments) to provide a score, which is then presented as a percentage. A higher Normalized LCS score suggests more significant shared phrasing, direct textual borrowing, or strong structural parallelism, as it reflects similarities in how ideas are ordered and expressed sequentially.
     *   *Note on Interpretation*: It's possible for Normalized LCS to be higher than Jaccard Similarity. This often happens when texts share a substantial 'narrative backbone' or common ordered phrases (leading to a high LCS), even if they use varied surrounding vocabulary or introduce many unique words not part of these core sequences (which would lower the Jaccard score). LCS highlights this sequential, structural similarity, while Jaccard focuses on the overall shared vocabulary regardless of its arrangement.
-3.  **Semantic Similarity**: Computes the cosine similarity between semantic embeddings of text segments using one of two approaches:
-
-    **a. Transformer-based Model** (Experimental): Pre-trained model that understands contextual relationships between words.
-    - `buddhist-nlp/buddhist-sentence-similarity`: Specialized for Buddhist texts
-    - Processes raw Unicode Tibetan text directly (no special tokenization required)
-    - Note: This is an experimental approach and results may vary with different texts
-
-    **b. FastText Model**: Uses the official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors) pre-trained on a large corpus of Tibetan text. Falls back to a custom model only if the official model cannot be loaded.
-    - Processes Tibetan text using botok tokenization (same as other metrics)
-    - Uses the pre-tokenized words from botok rather than doing its own tokenization
-    - Better for texts with specialized Tibetan vocabulary
-    - More stable results for general Tibetan text comparison
-    - Optimized for Tibetan language with:
-      - Syllable-based tokenization preserving Tibetan syllable markers
-      - TF-IDF weighted averaging for word vectors (distinct from the TF-IDF Cosine Similarity metric)
-      - Enhanced parameters based on Tibetan NLP research
-
-    For texts exceeding the model's token limit, an automated chunking strategy is employed: texts are divided into overlapping chunks, each chunk is embedded, and the resulting embeddings are averaged to produce a single vector for the entire segment.
+3.  **Semantic Similarity**: Computes the cosine similarity between semantic embeddings of text segments using a FastText model:
+    - **FastText**: Uses the official Facebook FastText Tibetan model (`cc.bo.300.bin`) with optimizations specifically for Tibetan language, including `botok` tokenization and TF-IDF weighted averaging of word vectors to produce segment embeddings.
 4.  **TF-IDF Cosine Similarity**: This metric first calculates Term Frequency-Inverse Document Frequency (TF-IDF) scores for each word in each text segment, optionally **filtering out common Tibetan stopwords**. 
 TF-IDF gives higher weight to words that are frequent within a particular segment but relatively rare across the entire collection of segments. 
 This helps to identify terms that are characteristic or discriminative for a segment. When stopword filtering is enabled, the TF-IDF scores better reflect genuinely significant terms.
@@ -210,28 +191,21 @@ This helps focus on meaningful content words rather than grammatical elements.
     -   The system will analyze your metrics and provide insights about patterns, relationships, and notable findings in your data.
     -   This feature helps researchers understand the significance of the metrics and identify interesting textual relationships between chapters.
 
-## Embedding Models
+## Embedding Model
 
-The application offers two specialized approaches for calculating semantic similarity in Tibetan texts:
+The application uses a FastText-based approach for calculating semantic similarity in Tibetan texts:
 
-1. **Buddhist-NLP Transformer** (Default option):
-   - A specialized model fine-tuned for Buddhist text similarity
-   - Provides excellent results for Tibetan Buddhist texts
-   - Pre-trained and ready to use, no training required
-   - Best for general Buddhist terminology and concepts
-
-2. **FastText Model**:
-   - Uses the official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors)
-   - Pre-trained on a large corpus of Tibetan text from Wikipedia and other sources
-   - Falls back to training a custom model on your texts if the official model cannot be loaded
-   - Respects your stopword filtering settings when creating embeddings
-   - Uses simple word vector averaging for stable embeddings
+**FastText Model Features**:
+- Utilizes official Facebook FastText word vectors for Tibetan (`cc.bo.300.bin`)
+- Integrates `botok` for accurate Tibetan word tokenization.
+- Employs TF-IDF weighted averaging of word vectors to produce segment embeddings. This method provides more nuanced similarity scores by emphasizing terms that are important within the analyzed texts.
+- The underlying `tibetan-text-metrics` library also supports training custom FastText models on user-uploaded texts for domain-specific accuracy, though this feature is not yet directly exposed for training in the web UI.
 
 **When to choose FastText**:
-- When you want high-quality word embeddings specifically trained for Tibetan language
-- When you need a model that can handle out-of-vocabulary words through character n-grams
-- When you want to benefit from Facebook's large-scale pre-training on Tibetan text
-- When you need more control over how stopwords affect semantic analysis
+- When you require high-quality word embeddings trained on a very large and diverse corpus of classical Tibetan texts.
+- When your analysis benefits from a model that can effectively handle out-of-vocabulary Tibetan words and orthographic variations through FastText's character n-gram features.
+- When you want to leverage a model trained with Tibetan-specific text preprocessing techniques.
+- When you need fine-grained control over how stopwords affect semantic analysis, as this model's embedding process respects the selected stopword filtering level.
 
 ## Structure
 
@@ -244,6 +218,10 @@ The application offers two specialized approaches for calculating semantic simil
     -   `tokenize.py`: Tibetan text tokenization using `botok`.
     -   `upload.py`: File upload handling (currently minimal).
     -   `visualize.py`: Generates heatmaps and word count plots.
+-   `fasttext-modelling/` — Scripts and documentation for training custom FastText models.
+    -   `train_custom_fasttext.py`: Script to train a custom FastText model.
+    -   `README.md`: Detailed instructions for the training script.
+    -   `requirements.txt`: Python dependencies specifically for the training script.
 -   `requirements.txt` — Python dependencies for the web application.
 
 ## License

app.py

@@ -5,6 +5,7 @@ from pipeline.visualize import generate_visualizations, generate_word_count_char
 from pipeline.llm_service import get_interpretation
 import logging
 import pandas as pd
+
 from dotenv import load_dotenv
 
 # Load environment variables from .env file
@@ -14,7 +15,6 @@ from theme import tibetan_theme
 
 logger = logging.getLogger(__name__)
 
-
 # Main interface logic
 def main_interface():
     with gr.Blocks(
@@ -65,18 +65,10 @@ def main_interface():
                     )
                     
                     model_dropdown = gr.Dropdown(
-                        label="Embedding Model",
-                        choices=[
-                            "buddhist-nlp/buddhist-sentence-similarity",
-                            "fasttext-tibetan"
-                        ],
-                        value="buddhist-nlp/buddhist-sentence-similarity",
-                        info="Select the embedding model for semantic similarity.<br><br>"
-                             "<b>Model information:</b><br>"
-                             "• <a href='https://huggingface.co/buddhist-nlp/buddhist-sentence-similarity' target='_blank'>buddhist-nlp/buddhist-sentence-similarity</a>: Specialized model fine-tuned for Buddhist text similarity.<br>"
-                             "• <b>fasttext-tibetan</b>: Uses the official Facebook FastText Tibetan model pre-trained on a large corpus. If the official model cannot be loaded, it will fall back to training a custom model on your uploaded texts.",
-                        visible=True,
-                        interactive=True
+                        choices=["Facebook FastText (Pre-trained)"],
+                        label="Select Embedding Model",
+                        value="Facebook FastText (Pre-trained)",
+                        info="Using Facebook's pre-trained FastText model for semantic similarity. Other model options have been removed."
                     )
                     
                     stopwords_dropdown = gr.Dropdown(
@@ -181,26 +173,19 @@ A higher Normalized LCS score suggests more significant shared phrasing, direct
 """,
             "Semantic Similarity": """
 ### Semantic Similarity
-Computes the cosine similarity between semantic embeddings of text segments using one of two approaches:
+Computes the cosine similarity between semantic embeddings of text segments:
 
-**1. Transformer-based Model** (Experimental): Pre-trained model that understands contextual relationships between words.
-   - `buddhist-nlp/buddhist-sentence-similarity`: Specialized for Buddhist texts
-   - Processes raw Unicode Tibetan text directly (no special tokenization required)
-   - Note: This is an experimental approach and results may vary with different texts
-
-**2. FastText Model**: Uses the official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors) pre-trained on a large corpus of Tibetan text. Falls back to a custom model only if the official model cannot be loaded.
+**FastText Model**: Uses the official Facebook FastText Tibetan model (facebook/fasttext-bo-vectors) pre-trained on a large corpus of Tibetan text. Falls back to a custom model only if the official model cannot be loaded.
    - Processes Tibetan text using botok tokenization (same as other metrics)
    - Uses the pre-tokenized words from botok rather than doing its own tokenization
    - Better for texts with specialized Tibetan vocabulary
    - More stable results for general Tibetan text comparison
    - Optimized for Tibetan language with:
-     - Syllable-based tokenization preserving Tibetan syllable markers
+     - Word-based tokenization preserving Tibetan syllable markers
      - TF-IDF weighted averaging for word vectors (distinct from the TF-IDF Cosine Similarity metric)
      - Enhanced parameters based on Tibetan NLP research
 
-**Chunking for Long Texts**: For texts exceeding the model's token limit, an automated chunking strategy is employed: texts are divided into overlapping chunks, each chunk is embedded, and the resulting embeddings are averaged to produce a single vector for the entire segment.
-
-**Stopword Filtering**: When enabled (via the "Filter Stopwords" checkbox), common Tibetan particles and function words are filtered out before computing embeddings. This helps focus on meaningful content words. Transformer models process the full text regardless of stopword filtering setting.
+**Stopword Filtering**: When enabled (via the "Filter Stopwords" checkbox), common Tibetan particles and function words are filtered out before computing embeddings. This helps focus on meaningful content words.
 
 **Note**: This metric works best when combined with other metrics for a more comprehensive analysis.
 """,
@@ -389,10 +374,20 @@ Each segment is represented as a vector of these TF-IDF scores, and the cosine s
                 use_stopwords = stopwords_option != "None (No filtering)"
                 use_lite_stopwords = stopwords_option == "Standard (Common particles only)"
                 
+                # Map UI model name to internal model ID
+                # The UI model_name is "Facebook FastText (Pre-trained)"
+                # This mapping ensures the backend receives the correct identifier.
+                if model_name == "Facebook FastText (Pre-trained)":
+                    internal_model_id = "facebook-fasttext-pretrained"
+                else:
+                    # Fallback or error if unexpected model_name, though UI should prevent this
+                    logger.warning(f"Unexpected model_name from UI: {model_name}. Defaulting to facebook-fasttext-pretrained.")
+                    internal_model_id = "facebook-fasttext-pretrained"
+
                 df_results, word_counts_df_data, warning_raw = process_texts(
                     text_data, filenames, 
                     enable_semantic=enable_semantic_bool, 
-                    model_name=model_name,
+                    model_name=internal_model_id, # Use the mapped internal ID
                     use_stopwords=use_stopwords,
                     use_lite_stopwords=use_lite_stopwords,
                     progress_callback=progress_tracker

pipeline/fasttext_embedding.py

@@ -4,15 +4,18 @@ This module provides functions to train and use FastText models for Tibetan text
 """
 
 import os
+from pathlib import Path
 import math
 import logging
 import numpy as np
 import fasttext
-from typing import List, Optional
+from collections import Counter
+from typing import List, Set, Optional
 from huggingface_hub import hf_hub_download
 
 # Set up logging
 logger = logging.getLogger(__name__)
+logger.setLevel(logging.DEBUG)  # Ensure this logger processes DEBUG messages
 
 # Default parameters optimized for Tibetan
 DEFAULT_DIM = 100
@@ -25,7 +28,7 @@ DEFAULT_NEG = 5
 
 # Define paths for model storage
 DEFAULT_MODEL_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "models")
-DEFAULT_MODEL_PATH = os.path.join(DEFAULT_MODEL_DIR, "fasttext_model.bin")
+DEFAULT_MODEL_PATH = str(Path(__file__).resolve().parent.parent / "fasttext-modelling" / "tibetan_cbow_model.bin")  # Updated to custom model
 
 # Facebook's official Tibetan FastText model
 FACEBOOK_TIBETAN_MODEL_ID = "facebook/fasttext-bo-vectors"
@@ -133,41 +136,67 @@ def train_fasttext_model(
     return model
 
 
+def load_facebook_official_tibetan_model() -> Optional[fasttext.FastText._FastText]:
+    """
+    Downloads (if necessary) and loads the official Facebook FastText Tibetan model.
+
+    Returns:
+        Loaded FastText model or None if loading fails.
+    """
+    try:
+        logger.info("Attempting to download and load official Facebook FastText Tibetan model")
+        facebook_model_path = hf_hub_download(
+            repo_id=FACEBOOK_TIBETAN_MODEL_ID,
+            filename=FACEBOOK_TIBETAN_MODEL_FILE,
+            cache_dir=DEFAULT_MODEL_DIR
+        )
+        logger.info(f"Loading official Facebook FastText Tibetan model from {facebook_model_path}")
+        model = fasttext.load_model(facebook_model_path)
+        if model:
+            logger.info(f"FastText model loaded in load_facebook_official_tibetan_model. Type: {type(model)}")
+            try:
+                # Basic check: get model dimensions
+                dims = model.get_dimension()
+                logger.info(f"Model dimensions reported by fasttext_embedding: {dims}")
+                # Check for a specific word to see if get_word_vector is callable with a string
+                # Using a common Tibetan particle that should be in the vocab
+                test_word = "ལ་"
+                try:
+                    vec = model.get_word_vector(test_word)
+                    logger.info(f"Successfully retrieved vector for test word '{test_word}'. Vector shape: {vec.shape if vec is not None else 'None'}")
+                except Exception as e_gwv:
+                    logger.error(f"Error calling get_word_vector for test word '{test_word}' in fasttext_embedding: {e_gwv}", exc_info=True)
+                    # Potentially re-raise or handle if this is critical for model validity
+            except Exception as e_diag_load:
+                logger.error(f"Error during diagnostic checks of loaded FastText model in fasttext_embedding: {e_diag_load}", exc_info=True)
+                # If diagnostics fail, the model might be unusable. Consider returning None.
+                # For now, let it return the model and fail later if that's the case.
+        else:
+            logger.error("fasttext.load_model returned None in load_facebook_official_tibetan_model.")
+        return model
+    except Exception as e_fb:
+        logger.error(f"Could not load official Facebook FastText Tibetan model (outer try-except): {str(e_fb)}", exc_info=True)
+        return None
+
 def load_fasttext_model(model_path: str = DEFAULT_MODEL_PATH) -> Optional[fasttext.FastText._FastText]:
     """
-    Load a FastText model from file, with fallback to official Facebook model.
+    Load a custom FastText model from the specified file path.
     
     Args:
-        model_path: Path to the model file
+        model_path: Path to the custom model file.
         
     Returns:
-        Loaded FastText model or None if loading fails
+        Loaded FastText model or None if loading fails.
     """
     try:
-        # First try to load the official Facebook FastText Tibetan model
-        try:
-            # Try to download the official Facebook FastText Tibetan model
-            logger.info("Attempting to download and load official Facebook FastText Tibetan model")
-            facebook_model_path = hf_hub_download(
-                repo_id=FACEBOOK_TIBETAN_MODEL_ID,
-                filename=FACEBOOK_TIBETAN_MODEL_FILE,
-                cache_dir=DEFAULT_MODEL_DIR
-            )
-            logger.info("Loading official Facebook FastText Tibetan model from %s", facebook_model_path)
-            return fasttext.load_model(facebook_model_path)
-        except Exception as e:
-            logger.warning("Could not load official Facebook FastText Tibetan model: %s", str(e))
-            logger.info("Falling back to local model")
-        
-        # Fall back to local model
         if os.path.exists(model_path):
-            logger.info("Loading local FastText model from %s", model_path)
+            logger.info(f"Attempting to load custom FastText model from {model_path}")
             return fasttext.load_model(model_path)
         else:
-            logger.warning("Model path %s does not exist", model_path)
+            logger.error(f"Custom FastText model path {model_path} does not exist.")
             return None
     except Exception as e:
-        logger.error("Error loading FastText model: %s", str(e))
+        logger.error(f"Could not load custom FastText model from {model_path}: {str(e)}")
         return None
 
 
@@ -177,8 +206,10 @@ def get_text_embedding(
     tokenize_fn=None,
     use_stopwords: bool = True,
     stopwords_set=None,
-    use_tfidf_weighting: bool = True,  # Enabled by default for better results
-    corpus_token_freq=None
+    use_tfidf_weighting: bool = True,
+    corpus_token_freq=None, # Retained for TF, but IDF will use doc_freq_map
+    doc_freq_map=None, # Document frequency map for IDF
+    total_docs_in_corpus=0 # Total documents in corpus for IDF
 ) -> np.ndarray:
     """
     Get embedding for a text using a FastText model with optional TF-IDF weighting.
@@ -199,83 +230,141 @@ def get_text_embedding(
         return np.zeros(model.get_dimension())
     
     # Handle tokenization
-    if tokenize_fn is None:
-        # Simple whitespace tokenization as fallback
-        tokens = text.split()
-    elif isinstance(tokenize_fn, list):
-        # If tokenize_fn is already a list of tokens, use it directly
-        tokens = tokenize_fn
-    elif callable(tokenize_fn):
-        # If tokenize_fn is a function, call it
+    if callable(tokenize_fn):
         tokens = tokenize_fn(text)
+        logger.debug(f"Tokens from callable tokenize_fn (first 20): {tokens[:20]}")
+    elif isinstance(tokenize_fn, list):
+        tokens = tokenize_fn # Use the provided list directly
+        logger.debug(f"Tokens provided as list (first 20): {tokens[:20]}")
     else:
-        # If tokenize_fn is something else (like a string), use whitespace tokenization
-        logger.warning(f"Unexpected tokenize_fn type: {type(tokenize_fn)}. Using default whitespace tokenization.")
+        if tokenize_fn is not None:
+            logger.warning(f"tokenize_fn is of unexpected type: {type(tokenize_fn)}. Defaulting to space-split.")
+        else:
+            # This case handles tokenize_fn being explicitly None
+            logger.debug("tokenize_fn is None. Defaulting to space-split.")
         tokens = text.split()
-    
-    # Filter out stopwords if enabled and stopwords_set is provided
-    if use_stopwords and stopwords_set is not None:
-        tokens = [token for token in tokens if token not in stopwords_set]
-    
-    # If all tokens were filtered out as stopwords, return zero vector
-    if not tokens:
-        return np.zeros(model.get_dimension())
-    
-    # Filter out empty tokens
-    tokens = [token for token in tokens if token.strip()]
-    
-    if not tokens:
-        return np.zeros(model.get_dimension())
-    
-    # Calculate TF-IDF weighted average if requested
-    if use_tfidf_weighting and corpus_token_freq is not None:
-        # Calculate term frequencies in this document
-        token_counts = {}
-        for token in tokens:
-            token_counts[token] = token_counts.get(token, 0) + 1
+        logger.debug(f"Tokens from space-split fallback (first 20): {tokens[:20]}")
+
+    if use_stopwords and stopwords_set:
+        logger.debug(f"Original tokens before stopword check (first 20): {tokens[:20]}")
+        original_token_count = len(tokens)
         
-        # Calculate IDF for each token with improved stability
-        N = sum(corpus_token_freq.values())  # Total number of tokens in corpus
-        N = max(N, 1)  # Ensure N is at least 1 to avoid division by zero
+        def _remove_stopwords_from_tokens(tokens: List[str], stopwords_set: Set[str]) -> List[str]:
+            """
+            Removes stopwords from a list of tokens.
+            Handles Tibetan punctuation by checking both the token itself and the token after
+            stripping trailing '།' or '༔'.
+            """
+            cleaned_tokens = []
+            removed_count = 0
+            for token in tokens:
+                # 1. Check if the original token itself is a stopword (e.g., standalone '།')
+                if token in stopwords_set:
+                    removed_count += 1
+                    continue  # Skip this token
+
+                # 2. If not a direct stopword, check if it becomes one after stripping trailing punctuation
+                #    This handles cases like "གྲུབ་པའི་།" where "གྲུབ་པའི་" is the stopword.
+                token_for_check = token
+                punctuation_was_stripped = False
+                if token.endswith(('།', '༔')):
+                    stripped_token = token.rstrip('།༔')
+                    if stripped_token != token:  # Check if stripping actually changed the token
+                        token_for_check = stripped_token
+                        punctuation_was_stripped = True
         
-        # Compute TF-IDF weights with safeguards against extreme values
-        weights = []
-        for token in tokens:
-            # Term frequency in this document
-            tf = token_counts.get(token, 0) / max(len(tokens), 1) if len(tokens) > 0 else 0
-            
-            # Inverse document frequency with smoothing to avoid extreme values
-            token_freq = corpus_token_freq.get(token, 0)
-            idf = math.log((N + 1) / (token_freq + 1)) + 1  # Add 1 for smoothing
+                if punctuation_was_stripped and token_for_check in stopwords_set:
+                    removed_count += 1
+                    continue # Skip this token
             
-            # TF-IDF weight with bounds to prevent extreme values
+                # 3. If neither the original token nor its base form is a stopword, keep it.
+                cleaned_tokens.append(token)
+
+            return cleaned_tokens
+
+        tokens = _remove_stopwords_from_tokens(tokens, stopwords_set)
+        removed_count = original_token_count - len(tokens)
+        logger.debug(f"Tokens after stopword removal (removed {removed_count}): {tokens[:20]}")
+
+    if not tokens:
+        logger.debug("Text became empty after tokenization/stopwords, returning zero vector.")
+        return np.zeros(model.get_dimension())
+
+    if use_tfidf_weighting and doc_freq_map and total_docs_in_corpus is not None and total_docs_in_corpus > 0:
+        logger.debug("Applying TF-IDF weighting.")
+        N_docs = total_docs_in_corpus
+        logger.debug(f"Total documents (N_docs) for IDF: {N_docs}")
+        
+        token_counts = Counter(tokens)
+        logger.debug(f"Local token counts for this segment (top 5): {dict(token_counts.most_common(5))}")
+
+        tf_idf_weights = []
+        token_details_log = []
+
+        for token in tokens: # Iterate in original token order
+            tf = token_counts.get(token, 0) / len(tokens) if len(tokens) > 0 else 0
+            df = doc_freq_map.get(token, 0)
+            idf = math.log((N_docs + 1) / (df + 1)) + 1 
             weight = tf * idf
-            weight = min(max(weight, 0.1), 10.0)  # Limit to reasonable range
-            weights.append(weight)
+            tf_idf_weights.append(weight)
+            token_details_log.append(f"Token: '{token}', TF: {tf:.4f}, DF: {df}, IDF: {idf:.4f}, Raw_TFIDF: {weight:.4f}")
         
-        # Normalize weights to sum to 1 with stability checks
-        total_weight = sum(weights)
-        if total_weight > 0:
-            weights = [w / total_weight for w in weights]
+        logger.debug("Token TF-IDF details (first 10 tokens):")
+        for i, log_entry in enumerate(token_details_log[:10]):
+            logger.debug(f"  {i+1}. {log_entry}")
+
+        total_weight = sum(tf_idf_weights)
+        logger.debug(f"Sum of raw TF-IDF weights: {total_weight}")
+        logger.debug(f"TF-IDF Summary for text snippet (first 100 chars): '{text[:100]}'. Total_TFIDF_Weight: {total_weight:.8e}. Fallback_to_Uniform: {total_weight <= 1e-6}.")
+        normalized_weights = []
+        if total_weight > 1e-6:
+            normalized_weights = [w / total_weight for w in tf_idf_weights]
+            logger.debug(f"Normalized weights (first 10): {[f'{w:.4f}' for w in normalized_weights[:10]]}")
         else:
-            # If all weights are 0, use uniform weights
-            weights = [1.0 / len(tokens) if len(tokens) > 0 else 0 for _ in tokens]
-            
-        # Check for NaN or infinite values and replace with uniform weights if found
-        if any(math.isnan(w) or math.isinf(w) for w in weights):
-            logger.warning("Found NaN or infinite weights in TF-IDF calculation. Using uniform weights instead.")
-            weights = [1.0 / len(tokens) if len(tokens) > 0 else 0 for _ in tokens]
+            logger.debug("Total TF-IDF weight is very small, falling back to uniform weights.")
+            num_tokens = len(tokens)
+            if num_tokens > 0:
+                normalized_weights = [1/num_tokens] * num_tokens
+            logger.debug(f"Uniform weights (first 10): {[f'{w:.4f}' for w in normalized_weights[:10]]}")
         
-        # Get vectors for each token and apply weights
-        vectors = [model.get_word_vector(token) for token in tokens]
-        weighted_vectors = [w * v for w, v in zip(weights, vectors)]
+        weighted_embeddings_sum = np.zeros(model.get_dimension())
+        if len(normalized_weights) == len(tokens):
+            for i, token in enumerate(tokens):
+                word_vector = model.get_word_vector(token)
+                vec_sum_for_log = np.sum(word_vector)
+                logger.debug(f"  Token: '{token}', Word_Vec_Sum: {vec_sum_for_log:.4f}, Applied_Weight: {normalized_weights[i]:.4f}")
+                weighted_embeddings_sum += word_vector * normalized_weights[i]
+            final_embedding = weighted_embeddings_sum
+        else:
+            logger.error("Mismatch between token count and normalized_weights count. THIS IS A BUG. Falling back to simple average.")
+            embeddings = [model.get_word_vector(t) for t in tokens]
+            if embeddings:
+                final_embedding = np.mean(embeddings, axis=0)
+            else:
+                final_embedding = np.zeros(model.get_dimension())
         
-        # Sum the weighted vectors
-        return np.sum(weighted_vectors, axis=0)
     else:
-        # Simple averaging if TF-IDF is not enabled or corpus frequencies not provided
-        vectors = [model.get_word_vector(token) for token in tokens]
-        return np.mean(vectors, axis=0)
+        if use_tfidf_weighting:
+             logger.debug("TF-IDF weighting was requested but doc_freq_map or total_docs_in_corpus is missing/invalid. Falling back to simple averaging.")
+        else:
+            logger.debug("Using simple averaging of word vectors (TF-IDF not requested or N_docs=0).")
+        
+        embeddings = []
+        for token in tokens:
+            word_vector = model.get_word_vector(token)
+            embeddings.append(word_vector)
+            vec_sum_for_log = np.sum(word_vector)
+            logger.debug(f"  Token: '{token}', Word_Vec_Sum: {vec_sum_for_log:.4f} (simple avg context)")
+
+        if embeddings:
+            final_embedding = np.mean(embeddings, axis=0)
+        else:
+            final_embedding = np.zeros(model.get_dimension())
+
+    final_emb_sum_for_log = np.sum(final_embedding)
+    logger.debug(f"Final aggregated embedding sum: {final_emb_sum_for_log:.4f}, shape: {final_embedding.shape}")
+    logger.debug(f"--- get_text_embedding finished for text (first 50 chars): {text[:50]} ---")
+    return final_embedding
 
 
 def get_batch_embeddings(
@@ -284,8 +373,10 @@ def get_batch_embeddings(
     tokenize_fn=None,
     use_stopwords: bool = True,
     stopwords_set=None,
-    use_tfidf_weighting: bool = True,  # Enabled by default for better results
-    corpus_token_freq=None
+    use_tfidf_weighting: bool = True,
+    corpus_token_freq=None, # Corpus-wide term frequencies
+    doc_freq_map=None, # Document frequency map for IDF
+    total_docs_in_corpus=0 # Total documents in corpus for IDF
 ) -> np.ndarray:
     """
     Get embeddings for a batch of texts with optional TF-IDF weighting.
@@ -302,54 +393,31 @@ def get_batch_embeddings(
     Returns:
         Array of text embedding vectors
     """
-    # If corpus_token_freq is not provided but TF-IDF is requested, build it from the texts
-    if use_tfidf_weighting and corpus_token_freq is None:
-        logger.info("Building corpus token frequency dictionary for TF-IDF weighting")
-        corpus_token_freq = {}
-        
-        # Process each text to build corpus token frequencies
-        for text in texts:
-            if not text.strip():
-                continue
-                
-            # Handle tokenization
-            if tokenize_fn is None:
-                tokens = text.split()
-            elif isinstance(tokenize_fn, list):
-                # In this case, tokenize_fn should be a list of lists (one list of tokens per text)
-                # This is not a common use case, so we'll just use the first one as fallback
-                tokens = tokenize_fn[0] if tokenize_fn else []
-            else:
-                tokens = tokenize_fn(text)
-            
-            # Filter out stopwords if enabled
-            if use_stopwords and stopwords_set is not None:
-                tokens = [token for token in tokens if token not in stopwords_set]
-            
-            # Update corpus token frequencies
-            for token in tokens:
-                if token.strip():  # Skip empty tokens
-                    corpus_token_freq[token] = corpus_token_freq.get(token, 0) + 1
-        
-        logger.info("Built corpus token frequency dictionary with %d unique tokens", len(corpus_token_freq))
-    
     # Get embeddings for each text
     embeddings = []
-    for i, text in enumerate(texts):
-        # Handle pre-tokenized input
-        tokens = None
-        if isinstance(tokenize_fn, list):
+    for i, text_content in enumerate(texts): # Changed 'text' to 'text_content'
+        tokens_or_tokenizer_for_current_text = None
+
+        if callable(tokenize_fn):
+            tokens_or_tokenizer_for_current_text = tokenize_fn # Pass the function itself
+        elif isinstance(tokenize_fn, list):
+            # If tokenize_fn is a list, it's assumed to be a list of pre-tokenized documents
             if i < len(tokenize_fn):
-                tokens = tokenize_fn[i]
+                tokens_or_tokenizer_for_current_text = tokenize_fn[i] # This is List[str] for the current text
+            else:
+                logger.warning(f"Pre-tokenized list `tokenize_fn` is shorter than the list of texts. Index {i} is out of bounds for `tokenize_fn` with length {len(tokenize_fn)}. Defaulting to None for this text.")
+        # If tokenize_fn is None or other, tokens_or_tokenizer_for_current_text remains None (get_text_embedding handles default).
         
         embedding = get_text_embedding(
-            text, 
+            text_content,  # Use renamed variable
             model, 
-            tokenize_fn=tokens,  # Pass the tokens directly, not the function
+            tokenize_fn=tokens_or_tokenizer_for_current_text,  # Pass the correctly determined function or token list
             use_stopwords=use_stopwords,
             stopwords_set=stopwords_set,
             use_tfidf_weighting=use_tfidf_weighting,
-            corpus_token_freq=corpus_token_freq
+            corpus_token_freq=corpus_token_freq,
+            doc_freq_map=doc_freq_map,
+            total_docs_in_corpus=total_docs_in_corpus
         )
         embeddings.append(embedding)
     
@@ -359,11 +427,12 @@ def get_batch_embeddings(
 def generate_embeddings(
     texts: List[str],
     model: fasttext.FastText._FastText,
-    device: str,
-    model_type: str = "sentence_transformer",
     tokenize_fn=None,
     use_stopwords: bool = True,
-    use_lite_stopwords: bool = False
+    use_lite_stopwords: bool = False,
+    corpus_token_freq=None,  # Existing: For TF-IDF
+    doc_freq_map=None,       # Added: For TF-IDF document frequency
+    total_docs_in_corpus=0 # Added: For TF-IDF total documents in corpus
 ) -> np.ndarray:
     """
     Generate embeddings for a list of texts using a FastText model.
@@ -371,17 +440,16 @@ def generate_embeddings(
     Args:
         texts: List of input texts
         model: FastText model
-        device: Device to use for computation (not used for FastText)
-        model_type: Model type ('sentence_transformer' or 'fasttext')
         tokenize_fn: Optional tokenization function or pre-tokenized list of tokens
         use_stopwords: Whether to filter out stopwords
         use_lite_stopwords: Whether to use a lighter set of stopwords
+        corpus_token_freq: Precomputed term frequencies for the corpus (for TF-IDF).
+        doc_freq_map: Precomputed document frequencies for tokens (for TF-IDF).
+        total_docs_in_corpus: Total number of documents in the corpus (for TF-IDF).
         
     Returns:
         Array of text embedding vectors
     """
-    if model_type != "fasttext":
-        logger.warning("Model type %s not supported for FastText. Using FastText anyway.", model_type)
     
     # Generate embeddings using FastText
     try:
@@ -399,7 +467,10 @@ def generate_embeddings(
             tokenize_fn=tokenize_fn,
             use_stopwords=use_stopwords,
             stopwords_set=stopwords_set,
-            use_tfidf_weighting=True  # Enable TF-IDF weighting for better results
+            use_tfidf_weighting=True,   # TF-IDF weighting enabled
+            corpus_token_freq=corpus_token_freq, # Pass down
+            doc_freq_map=doc_freq_map,           # Pass down for TF-IDF
+            total_docs_in_corpus=total_docs_in_corpus # Pass down for TF-IDF
         )
         
         logger.info("FastText embeddings generated with shape: %s", str(embeddings.shape))

pipeline/metrics.py

@@ -1,15 +1,14 @@
 import numpy as np
 import pandas as pd
-from typing import List, Dict
+from typing import List, Dict, Union
 from itertools import combinations
 from sklearn.metrics.pairwise import cosine_similarity
-import torch
 from .semantic_embedding import generate_embeddings
 from .tokenize import tokenize_texts
 import logging
 from sklearn.feature_extraction.text import TfidfVectorizer
-from .stopwords_bo import TIBETAN_STOPWORDS, TIBETAN_STOPWORDS_SET
-from .stopwords_lite_bo import TIBETAN_STOPWORDS_LITE, TIBETAN_STOPWORDS_LITE_SET
+from .stopwords_bo import TIBETAN_STOPWORDS
+from .stopwords_lite_bo import TIBETAN_STOPWORDS_LITE
 
 # Attempt to import the Cython-compiled fast_lcs module
 try:
@@ -21,58 +20,8 @@ except ImportError:
 
 logger = logging.getLogger(__name__)
 
-MAX_TOKENS_PER_CHUNK = 500  # Max tokens (words via botok) per chunk
-CHUNK_OVERLAP = 50  # Number of tokens to overlap between chunks
 
 
-def _chunk_text(
-    original_text_content: str,
-    tokens: List[str],
-    max_chunk_tokens: int,
-    overlap_tokens: int,
-) -> List[str]:
-    """
-    Splits a list of tokens into chunks and reconstructs text segments from these token chunks.
-    The reconstructed text segments are intended for embedding models.
-    Args:
-        original_text_content (str): The original raw text string. Used if no chunking is needed.
-        tokens (List[str]): The list of botok tokens for the original_text_content.
-        max_chunk_tokens (int): Maximum number of botok tokens per chunk.
-        overlap_tokens (int): Number of botok tokens to overlap between chunks.
-
-    Returns:
-        List[str]: A list of text strings, where each string is a chunk.
-    """
-    if (
-        not tokens
-    ):  # Handles empty or whitespace-only original text that led to no tokens
-        return [original_text_content] if original_text_content.strip() else []
-
-    if len(tokens) <= max_chunk_tokens:
-        # If not chunking, return the original text content directly, as per MEMORY[a777e6ad-11c4-4b90-8e6e-63a923a94432]
-        # The memory states raw text segments are passed directly to the model.
-        # Joining tokens here would alter spacing, etc.
-        return [original_text_content]
-
-    reconstructed_text_chunks = []
-    start_idx = 0
-    while start_idx < len(tokens):
-        end_idx = min(start_idx + max_chunk_tokens, len(tokens))
-        current_chunk_botok_tokens = tokens[start_idx:end_idx]
-        # Reconstruct the text chunk by joining the botok tokens. This is an approximation.
-        # The semantic model's internal tokenizer will handle this string.
-        reconstructed_text_chunks.append(" ".join(current_chunk_botok_tokens))
-
-        if end_idx == len(tokens):
-            break
-
-        next_start_idx = start_idx + max_chunk_tokens - overlap_tokens
-        if next_start_idx <= start_idx:
-            next_start_idx = start_idx + 1
-        start_idx = next_start_idx
-
-    return reconstructed_text_chunks
-
 
 def compute_normalized_lcs(words1: List[str], words2: List[str]) -> float:
     # Calculate m and n (lengths) here, so they are available for normalization
@@ -100,214 +49,126 @@ def compute_normalized_lcs(words1: List[str], words2: List[str]) -> float:
     return lcs_length / avg_length if avg_length > 0 else 0.0
 
 
+
 def compute_semantic_similarity(
     text1_segment: str,
     text2_segment: str,
-    tokens1: List[str],
-    tokens2: List[str],
-    model,
-    device,
-    model_type: str = "sentence_transformer",
+    tokens1: List[str], # botok tokens for text1, not directly used by FastText path but kept for signature
+    tokens2: List[str], # botok tokens for text2, not directly used by FastText path but kept for signature
+    model, # FastText model object
+    model_type: str = "fasttext", # Should always be 'fasttext' when called
     use_stopwords: bool = True,
     use_lite_stopwords: bool = False,
+    fasttext_tokenize_fn=None,
+    term_freq_corpus=None,
+    doc_freq_map=None,
+    total_docs_in_corpus=0
 ) -> float:
-    """Computes semantic similarity using a sentence transformer model, with chunking for long texts."""
-    if model is None or device is None:
+    """Computes semantic similarity using a FastText model."""
+    if model_type != "fasttext":
+        logger.error(f"compute_semantic_similarity called with unexpected model_type: {model_type}")
+        return np.nan
+
+    if model is None:
         logger.warning(
-            "Semantic similarity model or device not available. Skipping calculation."
+            "FastText model not available for semantic similarity. Skipping calculation."
         )
-        return np.nan  # Return NaN if model isn't loaded
+        return np.nan
 
     if not text1_segment or not text2_segment:
         logger.info(
             "One or both texts are empty for semantic similarity. Returning 0.0."
         )
-        return 0.0  # Or np.nan, depending on desired behavior for empty inputs
+        return 0.0
 
     def _get_aggregated_embedding(
-        raw_text_segment: str, botok_tokens: List[str], model_obj, device_str, model_type: str = "sentence_transformer", use_stopwords: bool = True, use_lite_stopwords: bool = False
-    ) -> torch.Tensor | None:
-        """Helper to get a single embedding for a text, chunking if necessary for transformer models."""
-        if (
-            not botok_tokens and not raw_text_segment.strip()
-        ):  # Check if effectively empty
+        raw_text_segment: str,
+        _botok_tokens: List[str], # Parameter name prefixed with _ to indicate it's not used
+        model_obj, 
+        use_stopwords_param: bool,
+        use_lite_stopwords_param: bool,
+        tokenize_fn_param,
+        term_freq_corpus_param,
+        doc_freq_map_param,
+        total_docs_in_corpus_param
+    ) -> Union[np.ndarray, None]:
+        """Helper to get a single embedding for a text using FastText."""
+        if not raw_text_segment.strip():
             logger.info(
                 f"Text segment is empty or only whitespace: {raw_text_segment[:100]}... Returning None for embedding."
             )
             return None
             
-        # For FastText, we don't need chunking as it processes tokens directly
-        if model_type == "fasttext":
-            if not raw_text_segment.strip():
-                logger.info(
-                    f"Text segment is empty or only whitespace: {raw_text_segment[:100]}... Returning None for embedding."
-                )
-                return None
-                
-            # Pass the raw text, pre-tokenized tokens, and stopword parameters
-            # Wrap the tokens in a list since generate_embeddings expects a list of token lists
-            embedding = generate_embeddings(
-                [raw_text_segment], 
-                model_obj, 
-                device_str, 
-                model_type, 
-                tokenize_fn=[botok_tokens], # Wrap in list since we're passing tokens for one text
-                use_stopwords=use_stopwords,
-                use_lite_stopwords=use_lite_stopwords
-            )
-            
-            if embedding is None or embedding.nelement() == 0:
-                logger.error(
-                    f"Failed to generate FastText embedding for text: {raw_text_segment[:100]}..."
-                )
-                return None
-            return embedding  # Already [1, embed_dim]
+        embedding = generate_embeddings(
+            texts=[raw_text_segment], 
+            model=model_obj, 
+            tokenize_fn=tokenize_fn_param,
+            use_stopwords=use_stopwords_param,
+            use_lite_stopwords=use_lite_stopwords_param,
+            corpus_token_freq=term_freq_corpus_param,
+            doc_freq_map=doc_freq_map_param,
+            total_docs_in_corpus=total_docs_in_corpus_param
+        )
         
-        # For transformer models, check if all tokens are stopwords when filtering is enabled
-        elif use_stopwords:
-            # Filter stopwords to see if any content remains
-            filtered_tokens = []
-            if use_lite_stopwords:
-                from .stopwords_lite_bo import TIBETAN_STOPWORDS_LITE_SET
-                filtered_tokens = [token for token in botok_tokens if token not in TIBETAN_STOPWORDS_LITE_SET]
-            else:
-                from .stopwords_bo import TIBETAN_STOPWORDS_SET
-                filtered_tokens = [token for token in botok_tokens if token not in TIBETAN_STOPWORDS_SET]
-                
-            # If all tokens were filtered out as stopwords, return zero embedding
-            if not filtered_tokens:
-                logger.info("All tokens in text are stopwords. Returning zero embedding.")
-                # Create a zero tensor with the same dimension as the model's output
-                # For transformer models, typically 384 or 768 dimensions
-                embedding_dim = 384  # Default dimension for MiniLM models
-                return torch.zeros(1, embedding_dim)
-                
-            # Continue with normal processing if content remains after filtering
-            if len(botok_tokens) > MAX_TOKENS_PER_CHUNK:
-                logger.info(
-                    f"Text segment with ~{len(botok_tokens)} tokens exceeds {MAX_TOKENS_PER_CHUNK}, chunking {raw_text_segment[:30]}..."
-                )
-                # Pass the original raw text and its pre-computed botok tokens to _chunk_text
-                text_chunks = _chunk_text(
-                    raw_text_segment, botok_tokens, MAX_TOKENS_PER_CHUNK, CHUNK_OVERLAP
-                )
-                if not text_chunks:
-                    logger.warning(
-                        f"Chunking resulted in no chunks for segment: {raw_text_segment[:100]}..."
-                    )
-                    return None
-
-                logger.info(
-                    f"Generated {len(text_chunks)} chunks for segment: {raw_text_segment[:30]}..."
-                )
-                # Generate embeddings for each chunk using the model
-                chunk_embeddings = generate_embeddings(text_chunks, model_obj, device_str, model_type)
-
-                if chunk_embeddings is None or chunk_embeddings.nelement() == 0:
-                    logger.error(
-                        f"Failed to generate embeddings for chunks of text: {raw_text_segment[:100]}..."
-                    )
-                    return None
-                # Mean pooling of chunk embeddings
-                aggregated_embedding = torch.mean(chunk_embeddings, dim=0, keepdim=True)
-                return aggregated_embedding
-            else:
-                # Text is short enough for transformer model, embed raw text directly
-                if not raw_text_segment.strip():
-                    logger.info(
-                        f"Text segment is empty or only whitespace: {raw_text_segment[:100]}... Returning None for embedding."
-                    )
-                    return None
-
-                embedding = generate_embeddings([raw_text_segment], model_obj, device_str, model_type)
-                if embedding is None or embedding.nelement() == 0:
-                    logger.error(
-                        f"Failed to generate embedding for text: {raw_text_segment[:100]}..."
-                    )
-                    return None
-                return embedding  # Already [1, embed_dim]
-        else:
-            # No stopword filtering, proceed with normal processing
-            if len(botok_tokens) > MAX_TOKENS_PER_CHUNK:
-                logger.info(
-                    f"Text segment with ~{len(botok_tokens)} tokens exceeds {MAX_TOKENS_PER_CHUNK}, chunking {raw_text_segment[:30]}..."
-                )
-                # Pass the original raw text and its pre-computed botok tokens to _chunk_text
-                text_chunks = _chunk_text(
-                    raw_text_segment, botok_tokens, MAX_TOKENS_PER_CHUNK, CHUNK_OVERLAP
-                )
-                if not text_chunks:
-                    logger.warning(
-                        f"Chunking resulted in no chunks for segment: {raw_text_segment[:100]}..."
-                    )
-                    return None
-
-                logger.info(
-                    f"Generated {len(text_chunks)} chunks for segment: {raw_text_segment[:30]}..."
-                )
-                # Generate embeddings for each chunk using the model
-                chunk_embeddings = generate_embeddings(text_chunks, model_obj, device_str, model_type)
-
-                if chunk_embeddings is None or chunk_embeddings.nelement() == 0:
-                    logger.error(
-                        f"Failed to generate embeddings for chunks of text: {raw_text_segment[:100]}..."
-                    )
-                    return None
-                # Mean pooling of chunk embeddings
-                aggregated_embedding = torch.mean(chunk_embeddings, dim=0, keepdim=True)
-                return aggregated_embedding
-            else:
-                # Text is short enough for transformer model, embed raw text directly
-                if not raw_text_segment.strip():
-                    logger.info(
-                        f"Text segment is empty or only whitespace: {raw_text_segment[:100]}... Returning None for embedding."
-                    )
-                    return None
-
-                embedding = generate_embeddings([raw_text_segment], model_obj, device_str, model_type)
-                if embedding is None or embedding.nelement() == 0:
-                    logger.error(
-                        f"Failed to generate embedding for text: {raw_text_segment[:100]}..."
-                    )
-                    return None
-                return embedding  # Already [1, embed_dim]
+        if embedding is None or embedding.size == 0: 
+            logger.error(
+                f"Failed to generate FastText embedding for text: {raw_text_segment[:100]}..."
+            )
+            return None
+        return embedding
 
     try:
-        # Pass raw text and its pre-computed botok tokens with stopword preference
-        embedding1 = _get_aggregated_embedding(text1_segment, tokens1, model, device, model_type, use_stopwords, use_lite_stopwords)
-        embedding2 = _get_aggregated_embedding(text2_segment, tokens2, model, device, model_type, use_stopwords, use_lite_stopwords)
-
-        if (
-            embedding1 is None
-            or embedding2 is None
-            or embedding1.nelement() == 0
-            or embedding2.nelement() == 0
-        ):
+        # Pass all relevant parameters to _get_aggregated_embedding
+        emb1 = _get_aggregated_embedding(text1_segment, tokens1, model, use_stopwords, use_lite_stopwords, fasttext_tokenize_fn, term_freq_corpus, doc_freq_map, total_docs_in_corpus)
+        emb2 = _get_aggregated_embedding(text2_segment, tokens2, model, use_stopwords, use_lite_stopwords, fasttext_tokenize_fn, term_freq_corpus, doc_freq_map, total_docs_in_corpus)
+
+        if emb1 is None or emb2 is None or emb1.size == 0 or emb2.size == 0:
             logger.error(
-                "Failed to obtain one or both aggregated embeddings for semantic similarity."
+                "Failed to obtain one or both FastText embeddings for semantic similarity."
             )
             return np.nan
 
-        # Check if both embeddings are zero vectors (which happens when all tokens are stopwords)
-        if np.all(embedding1.numpy() == 0) and np.all(embedding2.numpy() == 0):
-            # If both texts contain only stopwords, return 0 similarity
+        # Ensure embeddings are numpy arrays (should be, but defensive)
+        if not isinstance(emb1, np.ndarray): emb1 = np.array(emb1)
+        if not isinstance(emb2, np.ndarray): emb2 = np.array(emb2)
+
+        # Handle cases where embeddings are all zeros
+        if np.all(emb1 == 0) and np.all(emb2 == 0):
+            logger.info("Both FastText embeddings are zero. Semantic similarity is 0.0.")
             return 0.0
-            
-        # Cosine similarity expects 2D arrays, embeddings are [1, embed_dim] and on CPU
-        similarity = cosine_similarity(embedding1.numpy(), embedding2.numpy())
-        return float(similarity[0][0])
+        if np.all(emb1 == 0) or np.all(emb2 == 0):
+            logger.info("One of the FastText embeddings is zero. Semantic similarity is 0.0.")
+            return 0.0
+        
+        # Handle NaN or Inf in embeddings
+        if np.isnan(emb1).any() or np.isinf(emb1).any() or \
+           np.isnan(emb2).any() or np.isinf(emb2).any():
+            logger.warning("NaN or Inf found in FastText embeddings. Semantic similarity set to 0.0.")
+            return 0.0
+
+        # Ensure embeddings are 2D for cosine_similarity: [1, dim]
+        if emb1.ndim == 1: emb1 = emb1.reshape(1, -1)
+        if emb2.ndim == 1: emb2 = emb2.reshape(1, -1)
+        
+        similarity_score = cosine_similarity(emb1, emb2)[0][0]
+        
+        return max(0.0, float(similarity_score))
+
     except Exception as e:
+        safe_text1 = str(text1_segment)[:100] if text1_segment is not None else "N/A"
+        safe_text2 = str(text2_segment)[:100] if text2_segment is not None else "N/A"
         logger.error(
-            f"Error computing semantic similarity with chunking:\nText1: '{text1_segment[:100]}...'\nText2: '{text2_segment[:100]}...'\nError: {e}",
-            exc_info=True,
+            f"Error during FastText semantic similarity calculation:\nText1: {safe_text1}...\nText2: {safe_text2}...\nError: {e}"
         )
+        logger.exception("Traceback for FastText semantic similarity calculation error:")
         return np.nan
 
 
 def compute_all_metrics(
-    texts: Dict[str, str], model=None, device=None, enable_semantic: bool = True, 
-    model_type: str = "sentence_transformer", use_stopwords: bool = True,
-    use_lite_stopwords: bool = False
+    texts: Dict[str, str], model=None, enable_semantic: bool = True, # device=None removed
+    model_type: str = "fasttext", use_stopwords: bool = True,
+    use_lite_stopwords: bool = False,
+    fasttext_tokenize_fn=None # Added for FastText specific tokenizer
 ) -> pd.DataFrame:
     """
     Computes all selected similarity metrics between pairs of texts.
@@ -328,25 +189,60 @@ def compute_all_metrics(
     files = list(texts.keys())
     results = []
     # Prepare token lists (always use tokenize_texts for raw Unicode)
-    token_lists = {}
-    corpus_for_tfidf = []  # For storing space-joined tokens for TF-IDF
+    token_lists = {}  # Stores botok tokens for each text_id, used for Jaccard, LCS, and semantic sim
+    corpus_for_sklearn_tfidf = []  # For storing space-joined tokens for scikit-learn's TF-IDF
+    
+    # For FastText TF-IDF related statistics
+    term_freq_corpus_for_fasttext = {} # Renamed from global_corpus_token_freq_for_fasttext
+    document_frequency_map_for_fasttext = {}
+    total_num_documents_for_fasttext = len(texts)
+        
+    stopwords_set_for_fasttext_stats_calc = set()
+    if use_stopwords: # This 'use_stopwords' is an arg to compute_all_metrics
+        if use_lite_stopwords:
+            from .stopwords_lite_bo import TIBETAN_STOPWORDS_LITE_SET
+            stopwords_set_for_fasttext_stats_calc = TIBETAN_STOPWORDS_LITE_SET
+        else:
+            from .stopwords_bo import TIBETAN_STOPWORDS_SET
+            stopwords_set_for_fasttext_stats_calc = TIBETAN_STOPWORDS_SET
 
     for fname, content in texts.items():
-        tokenized_content = tokenize_texts([content])  # Returns a list of lists
-        if tokenized_content and tokenized_content[0]:
-            token_lists[fname] = tokenized_content[0]
-        else:
-            token_lists[fname] = []
-        # Regardless of whether tokenized_content[0] exists, prepare entry for TF-IDF corpus
-        # If tokens exist, join them; otherwise, use an empty string for that document
-        corpus_for_tfidf.append(
-            " ".join(token_lists[fname])
-            if fname in token_lists and token_lists[fname]
-            else ""
-        )
+        current_tokens_for_file = []
+        tokenized_content_list_of_lists = tokenize_texts([content]) 
+        if tokenized_content_list_of_lists and tokenized_content_list_of_lists[0]:
+            current_tokens_for_file = tokenized_content_list_of_lists[0]
+        token_lists[fname] = current_tokens_for_file
+
+        corpus_for_sklearn_tfidf.append(" ".join(current_tokens_for_file) if current_tokens_for_file else "")
+
+        if model_type == "fasttext":
+            tokens_for_fasttext_stats = []
+            if fasttext_tokenize_fn is not None:
+                tokens_for_fasttext_stats = fasttext_tokenize_fn(content) 
+            else:
+                tokens_for_fasttext_stats = current_tokens_for_file
+            
+            filtered_tokens_for_stats = [
+                token for token in tokens_for_fasttext_stats if token not in stopwords_set_for_fasttext_stats_calc
+            ] if use_stopwords else tokens_for_fasttext_stats
+            
+            # Update corpus-wide term frequencies
+            for token in filtered_tokens_for_stats:
+                if token.strip():
+                    term_freq_corpus_for_fasttext[token] = term_freq_corpus_for_fasttext.get(token, 0) + 1
+            
+            # Update document frequencies
+            unique_filtered_tokens_in_doc = set(filtered_tokens_for_stats)
+            for token in unique_filtered_tokens_in_doc:
+                if token.strip():
+                    document_frequency_map_for_fasttext[token] = document_frequency_map_for_fasttext.get(token, 0) + 1
+    
+    if model_type == "fasttext":
+        logger.info(f"Built FastText corpus term frequency map with {len(term_freq_corpus_for_fasttext)} unique tokens.")
+        logger.info(f"Built FastText document frequency map with {len(document_frequency_map_for_fasttext)} unique tokens across {total_num_documents_for_fasttext} documents.")
 
     # TF-IDF Vectorization and Cosine Similarity Calculation
-    if corpus_for_tfidf:
+    if corpus_for_sklearn_tfidf:
         try:
             # Using a dummy tokenizer and preprocessor as input is already tokenized (as space-separated strings)
             # and we don't want further case changes or token modifications for Tibetan.
@@ -368,14 +264,14 @@ def compute_all_metrics(
                 token_pattern=None,
                 stop_words=stopwords_to_use
             )
-            tfidf_matrix = vectorizer.fit_transform(corpus_for_tfidf)
+            tfidf_matrix = vectorizer.fit_transform(corpus_for_sklearn_tfidf)
             # Calculate pairwise cosine similarity on the TF-IDF matrix
             # This gives a square matrix where cosine_sim_matrix[i, j] is the similarity between doc i and doc j
             cosine_sim_matrix = cosine_similarity(tfidf_matrix)
         except ValueError as e:
             if "empty vocabulary" in str(e):
                 # If vocabulary is empty after stopword removal, create a zero matrix
-                n = len(corpus_for_tfidf)
+                n = len(corpus_for_sklearn_tfidf)
                 cosine_sim_matrix = np.zeros((n, n))
             else:
                 # Re-raise other ValueError
@@ -421,7 +317,11 @@ def compute_all_metrics(
         if enable_semantic:
             # Pass raw texts and their pre-computed botok tokens
             semantic_sim = compute_semantic_similarity(
-                texts[f1], texts[f2], words1_raw, words2_raw, model, device, model_type, use_stopwords, use_lite_stopwords
+                texts[f1], texts[f2], words1_raw, words2_raw, model, model_type, use_stopwords, use_lite_stopwords, # device removed
+                fasttext_tokenize_fn=fasttext_tokenize_fn,
+                term_freq_corpus=term_freq_corpus_for_fasttext if model_type == "fasttext" else None,
+                doc_freq_map=document_frequency_map_for_fasttext if model_type == "fasttext" else None,
+                total_docs_in_corpus=total_num_documents_for_fasttext if model_type == "fasttext" else 0
             )
         else:
             semantic_sim = np.nan

pipeline/process.py

@@ -1,10 +1,48 @@
 import pandas as pd
 from typing import Dict, List, Tuple
 from .metrics import compute_all_metrics
-from .semantic_embedding import get_model_and_device, train_fasttext_model, FASTTEXT_MODEL_ID
+from .semantic_embedding import get_model_and_device
+from .fasttext_embedding import load_fasttext_model # Added for custom fasttext
 from .tokenize import tokenize_texts
 import logging
 from itertools import combinations
+import re
+
+# Define FASTTEXT_MODEL_ID if not already defined (it should be, from semantic_embedding or globally)
+# For safety, let's assume it might be needed here directly for conditional logic
+FASTTEXT_MODEL_ID = "fasttext-tibetan" # Ensure this matches the ID used elsewhere
+
+
+def get_botok_tokens_for_single_text(text: str, mode: str = "syllable") -> list[str]:
+    """
+    A wrapper around tokenize_texts to make it suitable for tokenize_fn 
+    in generate_embeddings, which expects a function that tokenizes a single string.
+    Accepts a 'mode' argument ('syllable' or 'word') to pass to tokenize_texts.
+    """
+    if not text.strip():
+        return []
+    # Pass the mode to tokenize_texts
+    tokenized_list_of_lists = tokenize_texts([text], mode=mode)
+    if tokenized_list_of_lists and tokenized_list_of_lists[0]:
+        return tokenized_list_of_lists[0]
+    return []
+
+def clean_tibetan_text_for_fasttext(text: str) -> str:
+    """
+    Applies cleaning steps to Tibetan text similar to those in FastText training:
+    - Removes lnX/pX page/line markers.
+    - Normalizes double tsheg to single tsheg.
+    - Normalizes whitespace.
+    """
+    # Remove lnX/pX markers
+    cleaned_text = re.sub(r"\s*(?:[lL][nN]|[pP])\d{1,3}[abAB]?\s*", " ", text)
+    # Normalize double tsheg
+    cleaned_text = re.sub(r"།\s*།", "།", cleaned_text)
+    # Normalize spaces (multiple spaces to single, strip leading/trailing)
+    cleaned_text = re.sub(r'\s+', ' ', cleaned_text).strip()
+    return cleaned_text
+
+
 
 logger = logging.getLogger(__name__)
 
@@ -47,10 +85,10 @@ def process_texts(
         RuntimeError: If the botok tokenizer fails to initialize.
         ValueError: If the input files cannot be processed or if metrics computation fails.
     """
-    # Initialize model and device variables
-    st_model, st_device = None, None
+    # Initialize model and model_type variables
+    model, model_type = None, None # st_device removed
     model_warning = ""
-    
+
     # Update progress if callback provided
     if progress_callback is not None:
         try:
@@ -58,77 +96,78 @@ def process_texts(
         except Exception as e:
             logger.warning(f"Progress callback error (non-critical): {e}")
             # Continue processing even if progress reporting fails
-    
+
     # Load semantic model if enabled
     if enable_semantic:
         logger.info("Semantic similarity enabled. Loading embedding model...")
         try:
             logger.info("Using model: %s", model_name)
-            
-            # Check if this is a FastText model request
-            if model_name == FASTTEXT_MODEL_ID:
-                # Try to load the official Facebook FastText Tibetan model first
+
+            if model_name == FASTTEXT_MODEL_ID:  # FASTTEXT_MODEL_ID is 'fasttext-tibetan'
+                logger.info(f"Attempting to load custom FastText model: {model_name}")
                 if progress_callback is not None:
                     try:
-                        progress_callback(0.25, desc="Loading official Facebook FastText Tibetan model...")
+                        progress_callback(0.25, desc=f"Loading custom FastText model: {model_name}...")
                     except Exception as e:
-                        logger.warning("Progress callback error (non-critical): %s", str(e))
-                
-                st_model, st_device, model_type = get_model_and_device(model_id=model_name)
+                        logger.warning(f"Progress callback error (non-critical): {e}")
                 
-                # If model is None, we need to train a fallback model
-                if st_model is None:
-                    if progress_callback is not None:
-                        try:
-                            progress_callback(0.25, desc="Official model unavailable. Training fallback FastText model...")
-                        except Exception as e:
-                            logger.warning("Progress callback error (non-critical): %s", str(e))
-                    
-                    # Collect all text data for training
-                    all_texts = list(text_data.values())
-                    
-                    # Train the model with standard parameters for stability
-                    st_model = train_fasttext_model(all_texts, dim=100, epoch=5)
-                    
+                loaded_custom_model = load_fasttext_model(model_id=model_name) # model_id is expected to be path or key by this func
+                if loaded_custom_model:
+                    model = loaded_custom_model
+                    model_type = "fasttext"
+                    logger.info(f"Custom FastText model '{model_name}' loaded successfully.")
                     if progress_callback is not None:
                         try:
-                            progress_callback(0.3, desc="Fallback FastText model trained successfully")
+                            progress_callback(0.3, desc=f"Custom FastText model '{model_name}' loaded.")
                         except Exception as e:
-                            logger.warning("Progress callback error (non-critical): %s", str(e))
+                            logger.warning(f"Progress callback error (non-critical): {e}")
                 else:
+                    model_warning = f"Custom FastText model ('{model_name}') failed to load. Semantic similarity will be disabled."
+                    logger.warning(model_warning)
+                    enable_semantic = False
+            
+            elif model_name == "facebook-fasttext-pretrained":
+                logger.info(f"Attempting to load Facebook FastText model: {model_name}")
+                if progress_callback is not None:
+                    try:
+                        progress_callback(0.25, desc=f"Loading Facebook FastText model: {model_name}...")
+                    except Exception as e:
+                        logger.warning(f"Progress callback error (non-critical): {e}")
+                
+                fb_model, fb_model_type = get_model_and_device(model_id=model_name) # from semantic_embedding
+                if fb_model:
+                    model = fb_model
+                    model_type = fb_model_type  # Should be "fasttext"
+                    logger.info(f"Facebook FastText model '{model_name}' (type: {model_type}) loaded successfully.")
                     if progress_callback is not None:
                         try:
-                            progress_callback(0.3, desc="Official Facebook FastText Tibetan model loaded successfully")
+                            progress_callback(0.3, desc=f"Facebook FastText model '{model_name}' loaded.")
                         except Exception as e:
                             logger.warning(f"Progress callback error (non-critical): {e}")
-            else:
-                # For sentence transformers
-                st_model, st_device, model_type = get_model_and_device(model_id=model_name)
-                logger.info(f"Model {model_name} loaded successfully on {st_device}.")
-                
+                else:
+                    model_warning = f"Facebook FastText model ('{model_name}') failed to load. Semantic similarity will be disabled."
+                    logger.warning(model_warning)
+                    enable_semantic = False
+            
+            else:  # Any other model_name is unsupported
+                model_warning = f"Unsupported model_name: '{model_name}'. Semantic similarity will be disabled. Supported models are '{FASTTEXT_MODEL_ID}' and 'facebook-fasttext-pretrained'."
+                logger.warning(model_warning)
+                enable_semantic = False
                 if progress_callback is not None:
                     try:
-                        progress_callback(0.3, desc="Model loaded successfully")
+                        progress_callback(0.3, desc="Unsupported model, continuing without semantic similarity.")
                     except Exception as e:
                         logger.warning(f"Progress callback error (non-critical): {e}")
-                
-        except Exception as e:
-            error_msg = str(e)
-            logger.error(f"Failed to load sentence transformer model: {error_msg}. Semantic similarity will not be available.")
-            
-            # Create a user-friendly warning message
-            if "is not a valid model identifier" in error_msg:
-                model_warning = f"The model '{model_name}' could not be found on Hugging Face. Semantic similarity will not be available."
-            elif "CUDA out of memory" in error_msg:
-                model_warning = "Not enough GPU memory to load the semantic model. Try using a smaller model or disable semantic similarity."
-            else:
-                model_warning = f"Failed to load semantic model: {error_msg}. Semantic similarity will not be available."
-                
+        
+        except Exception as e:  # General catch-all for unexpected errors during model loading attempts
+            model_warning = f"An unexpected error occurred while attempting to load model '{model_name}': {e}. Semantic similarity will be disabled."
+            logger.error(model_warning, exc_info=True)
+            enable_semantic = False
             if progress_callback is not None:
                 try:
-                    progress_callback(0.3, desc="Continuing without semantic model")
-                except Exception as e:
-                    logger.warning(f"Progress callback error (non-critical): {e}")
+                    progress_callback(0.3, desc="Error loading model, continuing without semantic similarity.")
+                except Exception as e_cb:
+                    logger.warning(f"Progress callback error (non-critical): {e_cb}")
     else:
         logger.info("Semantic similarity disabled. Skipping model loading.")
         if progress_callback is not None:
@@ -177,11 +216,13 @@ def process_texts(
                 
             for idx, seg in enumerate(segments):
                 seg_id = f"{fname}|chapter {idx+1}"
-                segment_texts[seg_id] = seg
+                cleaned_seg = clean_tibetan_text_for_fasttext(seg)
+                segment_texts[seg_id] = cleaned_seg
         else:
             # No chapter markers found, treat entire file as one segment
             seg_id = f"{fname}|chapter 1"
-            segment_texts[seg_id] = content.strip()
+            cleaned_content = clean_tibetan_text_for_fasttext(content.strip())
+            segment_texts[seg_id] = cleaned_content
             fallback = True
             
     # Generate warning if no chapter markers found
@@ -261,14 +302,26 @@ def process_texts(
             
             try:
                 # Compute metrics for this chapter pair
+                tokenizer_for_fasttext = None
+                current_model_type = model_type if 'model_type' in locals() else "sentence_transformer"
+                if current_model_type == "fasttext":
+                    # Tokenizer setup for FastText model:
+                    def fasttext_tokenizer_adapter(text_segment: str) -> List[str]:
+                        cleaned_segment = clean_tibetan_text_for_fasttext(text_segment)
+                        # Use word-level tokenization for the custom FastText model
+                        return get_botok_tokens_for_single_text(cleaned_segment, mode="word")
+
+                    tokenizer_for_fasttext = fasttext_tokenizer_adapter
+                    logger.info("Using botok word-level tokenization for FastText model.")
+                
                 pair_metrics = compute_all_metrics(
                     {seg1: segment_texts[seg1], seg2: segment_texts[seg2]},
-                    model=st_model,
-                    device=st_device,
+                    model=model,
                     enable_semantic=enable_semantic,
-                    model_type=model_type if 'model_type' in locals() else "sentence_transformer",
+                    model_type=model_type,
                     use_stopwords=use_stopwords,
-                    use_lite_stopwords=use_lite_stopwords
+                    use_lite_stopwords=use_lite_stopwords,
+                    fasttext_tokenize_fn=tokenizer_for_fasttext
                 )
                 
                 # Rename 'Text Pair' to show file stems and chapter number

pipeline/semantic_embedding.py

@@ -1,7 +1,6 @@
 import logging
-import torch
-from typing import List, Any
-from sentence_transformers import SentenceTransformer
+from typing import List, Any, Optional
+import numpy as np # Added for type hinting Optional[np.ndarray]
 
 # Configure logging
 logging.basicConfig(
@@ -9,190 +8,114 @@ logging.basicConfig(
 )
 logger = logging.getLogger(__name__)
 
-# Define the model ID for the fine-tuned Tibetan MiniLM
-DEFAULT_MODEL_NAME = "buddhist-nlp/buddhist-sentence-similarity"
+# Define the model ID for the Facebook FastText pretrained model
+DEFAULT_MODEL_NAME = "facebook-fasttext-pretrained"
 
-# FastText model identifier - this is just an internal identifier, not a HuggingFace model ID
-FASTTEXT_MODEL_ID = "fasttext-tibetan"
+# FASTTEXT_MODEL_ID = "fasttext-tibetan" # Removed: Custom model loading to be handled in process.py directly
 
 
-def get_model_and_device(
-    model_id: str = DEFAULT_MODEL_NAME, device_preference: str = "auto"
-):
+def get_model_and_device(model_id: str = DEFAULT_MODEL_NAME):
     """
-    Loads the Sentence Transformer model and determines the device.
-    Priority: CUDA -> MPS (Apple Silicon) -> CPU.
+    Loads the Facebook official pre-trained FastText model for Tibetan.
 
     Args:
-        model_id (str): The Hugging Face model ID.
-        device_preference (str): Preferred device ("cuda", "mps", "cpu", "auto").
+        model_id (str): The model ID. Must be 'facebook-fasttext-pretrained' (DEFAULT_MODEL_NAME).
 
     Returns:
-        tuple: (model, device_str)
-               - model: The loaded SentenceTransformer model.
-               - device_str: The device the model is loaded on ("cuda", "mps", or "cpu").
+        Tuple[Optional[Any], Optional[str]]: 
+            A tuple containing the loaded FastText model and its type ("fasttext"), 
+            or (None, None) if loading fails or model_id is unsupported.
     """
-    selected_device_str = ""
+    logger.info("Attempting to load FastText model via semantic_embedding.get_model_and_device: %s", model_id)
 
-    if device_preference == "auto":
-        if torch.cuda.is_available():
-            selected_device_str = "cuda"
-        elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
-            selected_device_str = "mps"
-        else:
-            selected_device_str = "cpu"
-    elif device_preference == "cuda" and torch.cuda.is_available():
-        selected_device_str = "cuda"
-    elif (
-        device_preference == "mps"
-        and hasattr(torch.backends, "mps")
-        and torch.backends.mps.is_available()
-    ):
-        selected_device_str = "mps"
-    else:  # Handles explicit "cpu" preference or fallback if preferred is unavailable
-        selected_device_str = "cpu"
-
-    logger.info("Attempting to use device: %s", selected_device_str)
-
-    try:
-        # Check if this is a FastText model request
-        if model_id == FASTTEXT_MODEL_ID:
-            try:
-                # Import here to avoid dependency issues if FastText is not installed
-                import fasttext
-                from .fasttext_embedding import load_fasttext_model
-                
-                # Try to load the FastText model
-                model = load_fasttext_model()
-                
-                if model is None:
-                    error_msg = "Failed to load FastText model. Semantic similarity will not be available."
-                    logger.error(error_msg)
-                    raise Exception(error_msg)
-                    
-                logger.info("FastText model loaded successfully.")
-                # FastText always runs on CPU
-                return model, "cpu", "fasttext"
-            except ImportError:
-                logger.error("FastText module not found. Please install it with 'pip install fasttext'.")
-                raise
-        else:
-            logger.info(
-                "Loading Sentence Transformer model: %s on device: %s",
-                model_id, selected_device_str
-            )
-            # SentenceTransformer expects a string like 'cuda', 'mps', or 'cpu'
-            model = SentenceTransformer(model_id, device=selected_device_str)
-            logger.info("Model %s loaded successfully on %s.", model_id, selected_device_str)
-            return model, selected_device_str, "sentence_transformer"
-    except Exception as e:
-        logger.error(
-            "Error loading model %s on device %s: %s",
-            model_id, selected_device_str, str(e)
-        )
-        # Fallback to CPU if the initially selected device (CUDA or MPS) failed
-        if selected_device_str != "cpu":
-            logger.warning(
-                "Failed to load model on %s, attempting to load on CPU...",
-                selected_device_str
-            )
-            fallback_device_str = "cpu"
-            try:
-                # Check if this is a FastText model request during fallback
-                if model_id == FASTTEXT_MODEL_ID:
-                    # Import here to avoid dependency issues if FastText is not installed
-                    from .fasttext_embedding import load_fasttext_model
-                    
-                    # Try to load the FastText model
-                    model = load_fasttext_model()
-                    
-                    if model is None:
-                        logger.error("Failed to load FastText model during fallback. Semantic similarity will not be available.")
-                        raise Exception("Failed to load FastText model. Please check if the model file exists.")
-                        
-                    logger.info("FastText model loaded successfully during fallback.")
-                    # FastText always runs on CPU
-                    return model, "cpu", "fasttext"
-                else:
-                    # Try to load as a sentence transformer
-                    model = SentenceTransformer(model_id, device=fallback_device_str)
-                    logger.info(
-                        "Model %s loaded successfully on CPU after fallback.",
-                        model_id
-                    )
-                    return model, fallback_device_str, "sentence_transformer"
-            except Exception as fallback_e:
-                logger.error(
-                    "Error loading model %s on CPU during fallback: %s",
-                    model_id, str(fallback_e)
-                )
-                raise fallback_e  # Re-raise exception if CPU fallback also fails
-        raise e  # Re-raise original exception if selected_device_str was already CPU or no fallback attempted
-
-
-def generate_embeddings(texts: List[str], model: Any, device: str, model_type: str = "sentence_transformer", tokenize_fn=None, use_stopwords: bool = True, use_lite_stopwords: bool = False):
+    if model_id == DEFAULT_MODEL_NAME:  # DEFAULT_MODEL_NAME is "facebook-fasttext-pretrained"
+        try:
+            # Importing here to minimize issues if fasttext_embedding also imports from semantic_embedding
+            from .fasttext_embedding import load_facebook_official_tibetan_model
+            
+            model = load_facebook_official_tibetan_model()
+            
+            if model:
+                logger.info(f"FastText model object received in get_model_and_device. Type: {type(model)}.")
+                try:
+                    logger.info(f"Model dimensions: {model.get_dimension()}")
+                    # Basic check for model validity via an expected attribute/method
+                    if hasattr(model, 'get_word_vector'):
+                        logger.info("Model has 'get_word_vector' method (Python API expected for fasttext.load_model results).")
+                except Exception as diag_e:
+                    logger.error(f"Error during diagnostic check of FastText model '{model_id}': {diag_e}", exc_info=True)
+                return model, "fasttext"
+            else:
+                # This case implies load_facebook_official_tibetan_model returned None without raising an error.
+                logger.error(f"Model loading for '{model_id}' via load_facebook_official_tibetan_model() returned None unexpectedly.")
+                return None, None
+        except Exception as e:
+            logger.error(f"Failed to load or initialize FastText model '{model_id}': {e}. Semantic similarity will not be available.", exc_info=True)
+            return None, None
+    else:
+        logger.error(f"Unsupported model_id for get_model_and_device in semantic_embedding.py: '{model_id}'. Only '{DEFAULT_MODEL_NAME}' is supported by this function.")
+        return None, None
+
+
+def generate_embeddings(texts: List[str], model: Any, tokenize_fn=None, use_stopwords: bool = True, use_lite_stopwords: bool = False, corpus_token_freq=None, doc_freq_map=None, total_docs_in_corpus=0) -> Optional[np.ndarray]:
     """
-    Generates embeddings for a list of texts using the provided model.
+    Generates FastText embeddings for a list of texts.
 
     Args:
         texts (list[str]): A list of texts to embed.
-        model: The loaded model (SentenceTransformer or FastText).
-        device (str): The device to use ("cuda", "mps", or "cpu").
-        model_type (str): Type of model ("sentence_transformer" or "fasttext")
-        tokenize_fn: Optional tokenization function or pre-tokenized list for FastText
-        use_stopwords (bool): Whether to filter out stopwords for FastText embeddings
+        model: The loaded FastText model.
+        tokenize_fn: Optional tokenization function for FastText (if different from default botok based).
+        use_stopwords (bool): Whether to filter out stopwords for FastText embeddings.
+        use_lite_stopwords (bool): Whether to use the 'lite' stopwords list.
+        corpus_token_freq: Corpus-wide term frequencies for TF-IDF weighted FastText.
+        doc_freq_map: Document frequency map for TF-IDF weighted FastText.
+        total_docs_in_corpus: Total documents in corpus for TF-IDF weighted FastText.
 
     Returns:
-        torch.Tensor: A tensor containing the embeddings, moved to CPU.
+        Optional[np.ndarray]: A numpy array containing the embeddings. Returns None if generation fails.
     """
     if not texts:
         logger.warning(
-            "No texts provided to generate_embeddings. Returning empty tensor."
+            "No texts provided to generate_embeddings. Returning None."
         )
-        return torch.empty(0)
+        return None
 
-    logger.info(f"Generating embeddings for {len(texts)} texts...")
+    logger.info(f"Generating FastText embeddings for {len(texts)} texts...")
 
-    if model_type == "fasttext":
-        try:
-            # Import here to avoid dependency issues if FastText is not installed
-            from .fasttext_embedding import get_batch_embeddings
-            from .stopwords_bo import TIBETAN_STOPWORDS_SET
-            
-            # For FastText, get appropriate stopwords set if filtering is enabled
-            stopwords_set = None
-            if use_stopwords:
-                # Choose between regular and lite stopwords sets
-                if use_lite_stopwords:
-                    from .stopwords_lite_bo import TIBETAN_STOPWORDS_LITE_SET
-                    stopwords_set = TIBETAN_STOPWORDS_LITE_SET
-                else:
-                    from .stopwords_bo import TIBETAN_STOPWORDS_SET
-                    stopwords_set = TIBETAN_STOPWORDS_SET
-            
-            # Pass pre-tokenized tokens if available, otherwise pass None
-            # tokenize_fn should be a list of lists (tokens for each text) or None
-            embeddings = get_batch_embeddings(
-                texts, 
-                model, 
-                tokenize_fn=tokenize_fn, 
-                use_stopwords=use_stopwords, 
-                stopwords_set=stopwords_set
-            )
-            logger.info("FastText embeddings generated with shape: %s", str(embeddings.shape))
-            # Convert numpy array to torch tensor for consistency
-            return torch.tensor(embeddings)
-        except ImportError:
-            logger.error("FastText module not found. Please install it with 'pip install fasttext'.")
-            raise
-    else:  # sentence_transformer
-        # The encode method of SentenceTransformer handles tokenization and pooling internally.
-        # It also manages moving data to the model's device.
-        embeddings = model.encode(texts, convert_to_tensor=True, show_progress_bar=True)
-        logger.info("Sentence Transformer embeddings generated with shape: %s", str(embeddings.shape))
-        return (
-            embeddings.cpu()
-        )  # Ensure embeddings are on CPU for consistent further processing
+    try:
+        from .fasttext_embedding import get_batch_embeddings
+        
+        stopwords_set = None
+        if use_stopwords:
+            if use_lite_stopwords:
+                from .stopwords_lite_bo import TIBETAN_STOPWORDS_LITE_SET
+                stopwords_set = TIBETAN_STOPWORDS_LITE_SET
+            else:
+                from .stopwords_bo import TIBETAN_STOPWORDS_SET
+                stopwords_set = TIBETAN_STOPWORDS_SET
+        
+        embeddings = get_batch_embeddings(
+            texts, 
+            model, 
+            tokenize_fn=tokenize_fn, 
+            use_stopwords=use_stopwords, 
+            stopwords_set=stopwords_set,
+            corpus_token_freq=corpus_token_freq,
+            doc_freq_map=doc_freq_map,
+            total_docs_in_corpus=total_docs_in_corpus
+        )
+        if embeddings is None:
+             logger.error(f"get_batch_embeddings returned None for {len(texts)} texts. First few: {texts[:2]}")
+             return None
+
+        logger.info("FastText embeddings generated with shape: %s", str(embeddings.shape))
+        return embeddings
+    except ImportError:
+        logger.error("Required FastText modules not found. Please ensure 'fasttext' and its dependencies are correctly installed.")
+        return None
+    except Exception as e:
+        logger.error(f"An unexpected error occurred during FastText embedding generation: {e}", exc_info=True)
+        return None
 
 
 def train_fasttext_model(corpus_texts: List[str], **kwargs):
@@ -204,59 +127,15 @@ def train_fasttext_model(corpus_texts: List[str], **kwargs):
         **kwargs: Additional parameters for training (dim, epoch, etc.)
         
     Returns:
-        Trained model and path to the model file
-    """
+        Trained model and path to the model file (Note: current implementation returns only model object)
+    """ # Docstring updated for return type
     try:
         from .fasttext_embedding import prepare_corpus_file, train_fasttext_model as train_ft
         
-        # Prepare corpus file
         corpus_path = prepare_corpus_file(corpus_texts)
-        
-        # Train the model
         model = train_ft(corpus_path=corpus_path, **kwargs)
         
-        return model
+        return model # Returns model object, not path as previously suggested by older docstring
     except ImportError:
         logger.error("FastText module not found. Please install it with 'pip install fasttext'.")
-        raise
-
-
-if __name__ == "__main__":
-    # Example usage:
-    logger.info("Starting example usage of semantic_embedding module...")
-
-    test_texts = [
-        "བཀྲ་ཤིས་བདེ་ལེགས།",
-        "hello world",  # Test with non-Tibetan to see behavior
-        "དེ་རིང་གནམ་གཤིས་ཡག་པོ་འདུག",
-    ]
-
-    logger.info("Attempting to load model using default cache directory.")
-    try:
-        # Forcing CPU for this example to avoid potential CUDA issues in diverse environments
-        # or if CUDA is not intended for this specific test.
-        model, device, model_type = get_model_and_device(
-            device_preference="cpu"  # Explicitly use CPU for this test run
-        )
-
-        if model:
-            logger.info("Test model loaded on device: %s, type: %s", device, model_type)
-            example_embeddings = generate_embeddings(test_texts, model, device, model_type)
-            logger.info(
-                "Generated example embeddings shape: %s",
-                str(example_embeddings.shape)
-            )
-            if example_embeddings.nelement() > 0:  # Check if tensor is not empty
-                logger.info(
-                    "First embedding (first 10 dims): %s...",
-                    str(example_embeddings[0][:10])
-                )
-            else:
-                logger.info("Generated example embeddings tensor is empty.")
-        else:
-            logger.error("Failed to load model for example usage.")
-
-    except Exception as e:
-        logger.error("An error occurred during the example usage: %s", str(e))
-
-    logger.info("Finished example usage.")
+        raise # Re-raising to signal critical failure if training components are missing

pipeline/tibetan_stopwords.py

@@ -0,0 +1,41 @@
+import logging
+
+logger = logging.getLogger(__name__)
+
+def get_stopwords(use_lite: bool = False) -> set:
+    """
+    Returns a set of Tibetan stopwords by importing them from the respective .py files.
+
+    Args:
+        use_lite (bool): If True, returns a smaller, less aggressive list of stopwords
+                         from stopwords_lite_bo.py.
+                         Otherwise, returns the full list from stopwords_bo.py.
+
+    Returns:
+        set: A set of stopword strings. Returns an empty set on failure.
+    """
+    stopwords_set = set()
+    try:
+        if use_lite:
+            from .stopwords_lite_bo import STOPWORDS
+            stopwords_set = STOPWORDS
+        else:
+            from .stopwords_bo import STOPWORDS
+            stopwords_set = STOPWORDS
+        
+        logger.info(f"Successfully loaded {len(stopwords_set)} stopwords from {source_module_name.lstrip('.')}.py")
+    except ImportError:
+        logger.error(
+            f"Failed to import STOPWORDS from {source_module_name.lstrip('.')}.py. "
+            f"Ensure the file exists in the 'pipeline' directory, is a Python module (ends in .py), "
+            f"and is importable (e.g., no syntax errors)."
+        )
+    except AttributeError:
+        logger.error(
+            f"Variable 'STOPWORDS' (all caps) not found in {source_module_name.lstrip('.')}.py. "
+            f"Please ensure the stopword set is defined with this name within the module."
+        )
+    except Exception as e:
+        logger.error(f"An unexpected error occurred while loading stopwords from {source_module_name.lstrip('.')}.py: {e}")
+
+    return stopwords_set

pipeline/tokenize.py

@@ -39,7 +39,7 @@ def _get_text_hash(text: str) -> str:
     return hashlib.md5(text.encode('utf-8')).hexdigest()
 
 
-def tokenize_texts(texts: List[str]) -> List[List[str]]:
+def tokenize_texts(texts: List[str], mode: str = "syllable") -> List[List[str]]:
     """
     Tokenizes a list of raw Tibetan texts using botok, with caching for performance.
     
@@ -64,6 +64,10 @@ def tokenize_texts(texts: List[str]) -> List[List[str]]:
         )
 
     tokenized_texts_list = []
+
+    if mode not in ["word", "syllable"]:
+        logger.warning(f"Invalid tokenization mode: '{mode}'. Defaulting to 'syllable'.")
+        mode = "syllable"
     
     # Process each text
     for text_content in texts:
@@ -73,19 +77,84 @@ def tokenize_texts(texts: List[str]) -> List[List[str]]:
             continue
             
         # Generate hash for cache lookup
-        text_hash = _get_text_hash(text_content)
+        cache_key_string = text_content + f"_{mode}" # Include mode in string for hashing
+        text_hash = _get_text_hash(cache_key_string)
         
         # Check if we have this text in cache
         if text_hash in _tokenization_cache:
             # Cache hit - use cached tokens
             tokens = _tokenization_cache[text_hash]
-            logger.debug(f"Cache hit for text hash {text_hash[:8]}...")
+            logger.debug(f"Cache hit for text hash {text_hash[:8]}... (mode: {mode})")
         else:
             # Cache miss - tokenize and store in cache
             try:
-                tokens = [
-                    w.text for w in BOTOK_TOKENIZER.tokenize(text_content) if w.text.strip()
-                ]
+                current_tokens = []
+                if BOTOK_TOKENIZER:
+                    raw_botok_items = list(BOTOK_TOKENIZER.tokenize(text_content))
+                    
+                    if mode == "word":
+                        for item_idx, w in enumerate(raw_botok_items):
+                            if hasattr(w, 'text') and isinstance(w.text, str):
+                                token_text = w.text.strip()
+                                if token_text: # Ensure token is not empty or just whitespace
+                                    current_tokens.append(token_text)
+                            # Optionally log if w.text is not a string or missing, for debugging
+                            # elif w.text is not None:
+                            #     logger.debug(f"Token item {item_idx} has non-string text {type(w.text)} for hash {text_hash[:8]}. Skipping word.")
+                            # else:
+                            #     logger.debug(f"Token item {item_idx} missing text attribute for hash {text_hash[:8]}. Skipping word.")
+                        logger.debug(
+                            f"WORD TOKENS FORMED for hash {text_hash[:8]} (mode: {mode}, first 30): "
+                            f"{current_tokens[:30]}"
+                        )
+                    elif mode == "syllable":
+                        # This is the original syllable extraction logic
+                        for item_idx, w in enumerate(raw_botok_items):
+                            if hasattr(w, 'syls') and w.syls:
+                                for syl_idx, syl_item in enumerate(w.syls):
+                                    syllable_to_process = None
+                                    if isinstance(syl_item, str):
+                                        syllable_to_process = syl_item
+                                    elif isinstance(syl_item, list):
+                                        try:
+                                            syllable_to_process = "".join(syl_item)
+                                        except TypeError:
+                                            logger.warning(
+                                                f"Syllable item in w.syls was a list, but could not be joined (non-string elements?): {syl_item} "
+                                                f"from word item {item_idx} (text: {getattr(w, 'text', 'N/A')}), syl_idx {syl_idx} "
+                                                f"for hash {text_hash[:8]}. Skipping this syllable."
+                                            )
+                                            continue
+                                    
+                                    if syllable_to_process is not None:
+                                        stripped_syl = syllable_to_process.strip()
+                                        if stripped_syl:
+                                            current_tokens.append(stripped_syl)
+                                    elif syl_item is not None:
+                                        logger.warning(
+                                            f"Unexpected type for syllable item (neither str nor list): {type(syl_item)} ('{str(syl_item)[:100]}') "
+                                            f"from word item {item_idx} (text: {getattr(w, 'text', 'N/A')}), syl_idx {syl_idx} "
+                                            f"for hash {text_hash[:8]}. Skipping this syllable."
+                                        )
+                            elif hasattr(w, 'text') and w.text: # Fallback if no 'syls' but in syllable mode
+                                if isinstance(w.text, str):
+                                    token_text = w.text.strip()
+                                    if token_text:
+                                        current_tokens.append(token_text) # Treat as a single syllable/token
+                                elif w.text is not None:
+                                    logger.warning(
+                                        f"Unexpected type for w.text (in syllable mode fallback): {type(w.text)} ('{str(w.text)[:100]}') "
+                                        f"for item {item_idx} (POS: {getattr(w, 'pos', 'N/A')}) "
+                                        f"for hash {text_hash[:8]}. Skipping this token."
+                                    )
+                        logger.debug(
+                            f"SYLLABLE TOKENS FORMED for hash {text_hash[:8]} (mode: {mode}, first 30): "
+                            f"{current_tokens[:30]}"
+                        )
+                    tokens = current_tokens
+                else:
+                    logger.error(f"BOTOK_TOKENIZER is None for text hash {text_hash[:8]}, cannot tokenize (mode: {mode}).")
+                    tokens = []
                 
                 # Store in cache if not empty
                 if tokens:
@@ -95,9 +164,9 @@ def tokenize_texts(texts: List[str]) -> List[List[str]]:
                         _tokenization_cache.pop(next(iter(_tokenization_cache)))
                     
                     _tokenization_cache[text_hash] = tokens
-                    logger.debug(f"Added tokens to cache with hash {text_hash[:8]}...")
+                    logger.debug(f"Added tokens to cache with hash {text_hash[:8]}... (mode: {mode})")
             except Exception as e:
-                logger.error(f"Error tokenizing text: {e}")
+                logger.error(f"Error tokenizing text (mode: {mode}): {e}")
                 tokens = []
                 
         tokenized_texts_list.append(tokens)

pipeline/visualize.py

@@ -149,10 +149,15 @@ def generate_word_count_chart(word_counts_df: pd.DataFrame):
         font=dict(size=14),
         legend_title_text="Filename",
         xaxis=dict(
-            type="category"
-        ),  # Treat chapter numbers as categories for distinct grouping
-        autosize=True,
-        margin=dict(l=80, r=50, b=100, t=50, pad=4),
+            type="category",  # Treat chapter numbers as categories
+            automargin=True   # Automatically adjust margin for x-axis labels/title
+        ),
+        yaxis=dict(
+            rangemode='tozero', # Ensure y-axis starts at 0 and includes max value
+            automargin=True   # Automatically adjust margin for y-axis labels/title
+        ),
+        autosize=True,        # Keep for responsiveness in Gradio
+        margin=dict(l=80, r=50, b=100, t=50, pad=4) # Keep existing base margins
     )
     # Ensure x-axis ticks are shown for all chapter numbers present
     all_chapter_numbers = sorted(word_counts_df["ChapterNumber"].unique())

results.csv

@@ -0,0 +1,97 @@
+Text Pair,Jaccard Similarity (%),Normalized LCS,Semantic Similarity,TF-IDF Cosine Sim,Chapter
+Ngari 9.txt vs Nepal12.txt,100.0,1.0,,1.0,1
+Ngari 9.txt vs Nepal12.txt,100.0,1.0,,1.0,2
+Ngari 9.txt vs Nepal12.txt,100.0,1.0,,1.0,3
+Ngari 9.txt vs Nepal12.txt,46.42857142857143,0.6112115732368897,,0.8407944127395544,4
+Ngari 9.txt vs Nepal12.txt,40.42553191489361,0.5191256830601093,,0.5026984774848224,5
+Ngari 9.txt vs Nepal12.txt,47.28260869565217,0.6107784431137725,,0.8380742568060093,6
+Ngari 9.txt vs Nepal12.txt,49.29178470254957,0.5285565939771547,,0.8409605475909782,7
+Ngari 9.txt vs Nepal12.txt,46.07218683651805,0.6053169734151329,,0.9306016557862976,8
+Ngari 9.txt vs Nepal12.txt,51.7557251908397,0.7000429737859906,,0.9600630844581352,9
+Ngari 9.txt vs Nepal12.txt,52.760736196319016,0.710204081632653,,0.9135878707769712,10
+Ngari 9.txt vs Nepal12.txt,14.92842535787321,0.08302507192766133,,0.698638890914812,11
+Ngari 9.txt vs Nepal12.txt,0.0,0.0,,0.0,12
+Ngari 9.txt vs Nepal12.txt,0.0,0.0,,0.0,13
+Ngari 9.txt vs Nepal12.txt,0.0,0.0,,0.0,14
+Ngari 9.txt vs Nepal12.txt,0.0,0.0,,0.0,15
+Ngari 9.txt vs Nepal12.txt,100.0,1.0,,1.0,16
+Ngari 9.txt vs LTWA.txt,0.0,0.0,,0.0,1
+Ngari 9.txt vs LTWA.txt,0.0,0.0,,0.0,2
+Ngari 9.txt vs LTWA.txt,0.0,0.0,,0.0,3
+Ngari 9.txt vs LTWA.txt,47.752808988764045,0.603648424543947,,0.8414077093281586,4
+Ngari 9.txt vs LTWA.txt,48.40764331210191,0.6094808126410836,,0.6526135410649626,5
+Ngari 9.txt vs LTWA.txt,49.13294797687861,0.6297872340425532,,0.8252183235183391,6
+Ngari 9.txt vs LTWA.txt,35.53459119496855,0.4071058475203553,,0.8403529862077375,7
+Ngari 9.txt vs LTWA.txt,45.0,0.601965601965602,,0.9452297806160965,8
+Ngari 9.txt vs LTWA.txt,37.89126853377265,0.29986320109439124,,0.8760838478443608,9
+Ngari 9.txt vs LTWA.txt,51.632047477744806,0.6395222584147665,,0.9317016829510952,10
+Ngari 9.txt vs LTWA.txt,14.979757085020243,0.10742761225346202,,0.7111189597708231,11
+Ngari 9.txt vs LTWA.txt,0.0,0.0,,0.0,12
+Ngari 9.txt vs LTWA.txt,0.0,0.0,,0.0,13
+Ngari 9.txt vs LTWA.txt,0.0,0.0,,0.0,14
+Ngari 9.txt vs LTWA.txt,0.0,0.0,,0.0,15
+Ngari 9.txt vs LTWA.txt,0.0,0.0,,0.0,16
+Ngari 9.txt vs Leiden.txt,0.0,0.0,,0.0,1
+Ngari 9.txt vs Leiden.txt,0.0,0.0,,0.0,2
+Ngari 9.txt vs Leiden.txt,0.0,0.0,,0.0,3
+Ngari 9.txt vs Leiden.txt,41.340782122905026,0.5282331511839709,,0.8525095366316284,4
+Ngari 9.txt vs Leiden.txt,36.80555555555556,0.4734042553191489,,0.5634721694429372,5
+Ngari 9.txt vs Leiden.txt,44.047619047619044,0.4728132387706856,,0.7698959290709281,6
+Ngari 9.txt vs Leiden.txt,35.67251461988304,0.3208020050125313,,0.784262930792386,7
+Ngari 9.txt vs Leiden.txt,41.01123595505618,0.4241099312929419,,0.9275267086147868,8
+Ngari 9.txt vs Leiden.txt,40.31209362808843,0.20184790334044064,,0.9076572014074583,9
+Ngari 9.txt vs Leiden.txt,50.445103857566764,0.6045733407696597,,0.9284684903895061,10
+Ngari 9.txt vs Leiden.txt,16.363636363636363,0.08736942070275404,,0.6999802304139516,11
+Ngari 9.txt vs Leiden.txt,0.0,0.0,,0.0,12
+Ngari 9.txt vs Leiden.txt,0.0,0.0,,0.0,13
+Ngari 9.txt vs Leiden.txt,0.0,0.0,,0.0,14
+Ngari 9.txt vs Leiden.txt,0.0,0.0,,0.0,15
+Ngari 9.txt vs Leiden.txt,0.0,0.0,,0.0,16
+Nepal12.txt vs LTWA.txt,0.0,0.0,,0.0,1
+Nepal12.txt vs LTWA.txt,0.0,0.0,,0.0,2
+Nepal12.txt vs LTWA.txt,0.0,0.0,,0.0,3
+Nepal12.txt vs LTWA.txt,56.493506493506494,0.6959706959706959,,0.8321482637014176,4
+Nepal12.txt vs LTWA.txt,39.71631205673759,0.5386666666666666,,0.7104447145077406,5
+Nepal12.txt vs LTWA.txt,48.795180722891565,0.5898004434589801,,0.8168699067131293,6
+Nepal12.txt vs LTWA.txt,34.954407294832826,0.3365548607163161,,0.861391898750807,7
+Nepal12.txt vs LTWA.txt,51.41509433962265,0.5873239436619718,,0.9310750730815768,8
+Nepal12.txt vs LTWA.txt,41.18705035971223,0.3156208277703605,,0.9075961630628558,9
+Nepal12.txt vs LTWA.txt,60.066006600660074,0.7040533037201555,,0.921390350997517,10
+Nepal12.txt vs LTWA.txt,63.6986301369863,0.7454220634211701,,0.9803189694519824,11
+Nepal12.txt vs LTWA.txt,48.275862068965516,0.5102639296187683,,0.7725258306356406,12
+Nepal12.txt vs LTWA.txt,58.203125,0.7364921030756443,,0.9543942889292814,13
+Nepal12.txt vs LTWA.txt,41.732283464566926,0.4332449160035367,,0.8497746214132795,14
+Nepal12.txt vs LTWA.txt,17.983651226158038,0.1474820143884892,,0.5779105517118261,15
+Nepal12.txt vs LTWA.txt,0.0,0.0,,0.0,16
+Nepal12.txt vs Leiden.txt,0.0,0.0,,0.0,1
+Nepal12.txt vs Leiden.txt,0.0,0.0,,0.0,2
+Nepal12.txt vs Leiden.txt,0.0,0.0,,0.0,3
+Nepal12.txt vs Leiden.txt,57.14285714285714,0.6788617886178862,,0.8403894964769358,4
+Nepal12.txt vs Leiden.txt,38.793103448275865,0.4935064935064935,,0.4684416871587978,5
+Nepal12.txt vs Leiden.txt,60.416666666666664,0.6386138613861386,,0.8441982917223785,6
+Nepal12.txt vs Leiden.txt,43.24324324324324,0.33632734530938124,,0.8839876637274263,7
+Nepal12.txt vs Leiden.txt,50.8235294117647,0.4953338119167265,,0.9373191281412603,8
+Nepal12.txt vs Leiden.txt,44.03927068723703,0.2242042672263029,,0.9196700291527228,9
+Nepal12.txt vs Leiden.txt,67.59581881533101,0.7226027397260274,,0.9462708958951278,10
+Nepal12.txt vs Leiden.txt,60.42780748663101,0.7003094501309212,,0.9722895878422901,11
+Nepal12.txt vs Leiden.txt,23.502304147465438,0.27245508982035926,,0.6893488630692246,12
+Nepal12.txt vs Leiden.txt,67.08333333333333,0.7506382978723404,,0.9466019120384076,13
+Nepal12.txt vs Leiden.txt,42.67782426778243,0.418426103646833,,0.8023010077421123,14
+Nepal12.txt vs Leiden.txt,31.17206982543641,0.2664756446991404,,0.757778410785804,15
+Nepal12.txt vs Leiden.txt,0.0,0.0,,0.0,16
+LTWA.txt vs Leiden.txt,53.5064935064935,0.6359163591635917,,0.9623337315161734,1
+LTWA.txt vs Leiden.txt,60.909090909090914,0.7578659370725034,,0.8852155398192683,2
+LTWA.txt vs Leiden.txt,64.1025641025641,0.7001044932079414,,0.8986878289296542,3
+LTWA.txt vs Leiden.txt,51.21951219512195,0.6568265682656826,,0.8596233249504219,4
+LTWA.txt vs Leiden.txt,40.0,0.5298701298701298,,0.6287776677298036,5
+LTWA.txt vs Leiden.txt,46.308724832214764,0.5415549597855228,,0.7796920776498958,6
+LTWA.txt vs Leiden.txt,43.233082706766915,0.49545136459062283,,0.8819142330949857,7
+LTWA.txt vs Leiden.txt,54.03050108932462,0.5373230373230373,,0.9477445373252964,8
+LTWA.txt vs Leiden.txt,38.75598086124402,0.1898707353252808,,0.887072472781142,9
+LTWA.txt vs Leiden.txt,66.32996632996633,0.7823310271420969,,0.9693004524579277,10
+LTWA.txt vs Leiden.txt,63.537906137184116,0.754516983859311,,0.9830176756030125,11
+LTWA.txt vs Leiden.txt,24.299065420560748,0.18152350081037277,,0.6278532648577805,12
+LTWA.txt vs Leiden.txt,60.1593625498008,0.7367521367521368,,0.9381662329597793,13
+LTWA.txt vs Leiden.txt,59.44444444444444,0.6746987951807228,,0.8771500136505623,14
+LTWA.txt vs Leiden.txt,35.37735849056604,0.39255014326647564,,0.6834100468628878,15
+LTWA.txt vs Leiden.txt,60.45081967213115,0.6875444839857652,,0.9482911929631709,16

user_guide.md

@@ -0,0 +1,190 @@
+# Tibetan Text Metrics Web Application User Guide
+
+## Introduction
+
+Welcome to the Tibetan Text Metrics Web Application! This user-friendly tool allows you to analyze textual similarities and variations in Tibetan manuscripts using multiple computational approaches. The application provides a graphical interface to the core functionalities of the Tibetan Text Metrics (TTM) project.
+
+## Getting Started
+
+### System Requirements
+
+- Modern web browser (Chrome, Firefox, Safari, or Edge)
+- For local installation: Python 3.10 or newer
+- Sufficient RAM for processing large texts (4GB minimum, 8GB recommended)
+
+### Installation and Setup
+
+#### Online Demo
+
+The easiest way to try the application is through our Hugging Face Spaces demo:
+[daniel-wojahn/ttm-webapp-hf](https://huggingface.co/spaces/daniel-wojahn/ttm-webapp-hf)
+
+Note: The free tier of Hugging Face Spaces may have performance limitations compared to running locally.
+
+#### Local Installation
+
+1. Clone the repository:
+   ```bash
+   git clone https://github.com/daniel-wojahn/tibetan-text-metrics.git
+   cd tibetan-text-metrics/webapp
+   ```
+
+2. Create and activate a virtual environment:
+   ```bash
+   python -m venv venv
+   source venv/bin/activate  # On Windows: venv\Scripts\activate
+   ```
+
+3. Install dependencies:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+4. Run the application:
+   ```bash
+   python app.py
+   ```
+
+5. Open your browser and navigate to:
+   ```
+   http://localhost:7860
+   ```
+
+## Using the Application
+
+### Step 1: Upload Your Tibetan Text Files
+
+1. Click the "Upload Tibetan .txt files" button to select one or more `.txt` files containing Tibetan text.
+2. Files should be in UTF-8 or UTF-16 encoding.
+3. Maximum file size: 10MB per file (for optimal performance, use files under 1MB).
+4. For best results, your texts should be segmented into chapters/sections using the Tibetan marker '༈' (*sbrul shad*).
+
+### Step 2: Configure Analysis Options
+
+1. **Semantic Similarity**: Choose whether to compute semantic similarity metrics.
+   - "Yes" (default): Includes semantic similarity in the analysis (slower but more comprehensive).
+   - "No": Skips semantic similarity calculation for faster processing.
+
+2. **Embedding Model**: Select the model to use for semantic similarity analysis.
+   - **sentence-transformers/all-MiniLM-L6-v2** (default): General purpose sentence embedding model (fastest option).
+   - **sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2**: Multilingual model with good performance for many languages.
+   - **buddhist-nlp/buddhist-sentence-similarity**: Optimized for Buddhist text similarity.
+   - **xlm-roberta-base**: Multilingual model that includes Tibetan.
+
+3. Click the "Run Analysis" button to start processing.
+
+### Step 3: View and Interpret Results
+
+After processing, the application displays several visualizations and metrics:
+
+#### Word Count Chart
+
+Shows the number of words in each chapter/segment of each file, allowing you to compare the relative lengths of different texts.
+
+#### Similarity Metrics
+
+The application computes four different similarity metrics between corresponding chapters of different files:
+
+1. **Jaccard Similarity (%)**: Measures vocabulary overlap between segments after filtering out common Tibetan stopwords. A higher percentage indicates a greater overlap in the significant vocabularies used in the two segments.
+
+2. **Normalized LCS (Longest Common Subsequence)**: Measures the length of the longest sequence of words that appears in both text segments, maintaining their original relative order. A higher score suggests more significant shared phrasing, direct textual borrowing, or strong structural parallelism.
+
+3. **Semantic Similarity**: Uses a transformer-based model to compute the cosine similarity between the semantic embeddings of text segments. This captures similarities in meaning even when different vocabulary is used.
+
+4. **TF-IDF Cosine Similarity**: Compares texts based on their important, characteristic terms by giving higher weight to words that are frequent within a particular segment but relatively rare across the entire collection.
+
+#### Heatmap Visualizations
+
+Each metric has a corresponding heatmap visualization where:
+- Rows represent chapters/segments
+- Columns represent text pairs being compared
+- Color intensity indicates similarity (brighter = more similar)
+
+### Tips for Effective Analysis
+
+1. **Text Segmentation**: For meaningful chapter-level comparisons, ensure your texts are segmented using the Tibetan marker '༈' (*sbrul shad*).
+
+2. **File Naming**: Use descriptive filenames to make the comparison results easier to interpret.
+
+3. **Model Selection**: 
+   - For faster processing, use the default model or disable semantic similarity.
+   - For Buddhist texts, the buddhist-nlp/buddhist-sentence-similarity model may provide better results.
+
+4. **File Size**: 
+   - Keep individual files under 1MB for optimal performance.
+   - Very large files (>10MB) are not supported and will trigger an error.
+
+5. **Comparing Multiple Texts**: The application requires at least two text files to compute similarity metrics.
+
+## Understanding the Metrics
+
+### Jaccard Similarity (%)
+
+This metric quantifies the lexical overlap between two text segments by comparing their sets of unique words, after filtering out common Tibetan stopwords. It essentially answers the question: 'Of all the distinct, meaningful words found across these two segments, what proportion of them are present in both?'
+
+It is calculated as:
+```
+(Number of common unique meaningful words) / (Total number of unique meaningful words in both texts combined) * 100
+```
+
+Jaccard Similarity is insensitive to word order and word frequency; it only cares whether a unique meaningful word is present or absent. A higher percentage indicates a greater overlap in the significant vocabularies used in the two segments.
+
+### Normalized LCS (Longest Common Subsequence)
+
+This metric measures the length of the longest sequence of words that appears in both text segments, maintaining their original relative order. Importantly, these words do not need to be directly adjacent (contiguous) in either text.
+
+For example, if Text A is 'the quick brown fox jumps' and Text B is 'the lazy cat and brown dog jumps high', the LCS is 'the brown jumps'.
+
+The length of this common subsequence is then normalized to provide a score. A higher Normalized LCS score suggests more significant shared phrasing, direct textual borrowing, or strong structural parallelism, as it reflects similarities in how ideas are ordered and expressed sequentially.
+
+Unlike other metrics, LCS does not filter out stopwords, allowing it to capture structural similarities and the flow of language, including the use of particles and common words that contribute to sentence construction.
+
+### Semantic Similarity
+
+This metric utilizes transformer-based models to compute the cosine similarity between the semantic embeddings of text segments. The model converts each text segment into a high-dimensional vector that captures its semantic meaning.
+
+For texts exceeding the model's token limit, an automated chunking strategy is employed: texts are divided into overlapping chunks, each chunk is embedded, and the resulting chunk embeddings are averaged to produce a single representative vector for the entire segment before comparison.
+
+A higher score indicates that the texts express similar concepts or ideas, even if they use different vocabulary or phrasing.
+
+### TF-IDF Cosine Similarity
+
+This metric first calculates Term Frequency-Inverse Document Frequency (TF-IDF) scores for each word in each text segment, after filtering out common Tibetan stopwords. TF-IDF gives higher weight to words that are frequent within a particular segment but relatively rare across the entire collection of segments.
+
+Each segment is then represented as a vector of these TF-IDF scores, and the cosine similarity is computed between these vectors. A score closer to 1 indicates that the two segments share more of these important, distinguishing terms, suggesting they cover similar specific topics or themes.
+
+## Troubleshooting
+
+### Common Issues and Solutions
+
+1. **"Empty vocabulary" error**:
+   - This can occur if a text contains only stopwords or if tokenization fails.
+   - Solution: Check your input text to ensure it contains valid Tibetan content.
+
+2. **Model loading errors**:
+   - If a model fails to load, the application will continue without semantic similarity.
+   - Solution: Try a different model or disable semantic similarity.
+
+3. **Performance issues with large files**:
+   - Solution: Split large files into smaller ones or use fewer files at once.
+
+4. **No results displayed**:
+   - Solution: Ensure you have uploaded at least two valid text files and that they contain comparable content.
+
+5. **Encoding issues**:
+   - If your text appears garbled, it may have encoding problems.
+   - Solution: Ensure your files are saved in UTF-8 or UTF-16 encoding.
+
+### Getting Help
+
+If you encounter issues not covered in this guide, please:
+1. Check the [GitHub repository](https://github.com/daniel-wojahn/tibetan-text-metrics) for updates or known issues.
+2. Submit an issue on GitHub with details about your problem.
+
+## Acknowledgments
+
+The Tibetan Text Metrics project was developed as part of the [Law in Historic Tibet](https://www.law.ox.ac.uk/law-historic-tibet) project at the Centre for Socio-Legal Studies at the University of Oxford.
+
+## License
+
+This project is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).