chapter name feature added

academic_article.md

@@ -6,7 +6,7 @@
 
 ### 1.1. The Challenge of Tibetan Textual Scholarship
 
-The Tibetan literary corpus is one of the world's most extensive, encompassing centuries of philosophy, history, jurisprudence, and religious doctrine. The transmission of these texts has been a complex process, involving manual copying by scribes across different regions and monastic traditions. This has resulted in a rich but challenging textual landscape, characterized by numerous variations, scribal errors, and divergent manuscript lineages. For scholars, tracing the history of a text and understanding its evolution requires meticulous philological work. While traditional methods are indispensable, the sheer scale of the available material necessitates computational approaches that can identify patterns and relationships that are not immediately apparent to the human eye.
+The Tibetan literary corpus is one of the world's most extensive, encompassing centuries of philosophy, history, and religious doctrine. The transmission of these texts has been a complex process, involving manual copying that resulted in a rich but challenging textual landscape of divergent manuscript lineages. This challenge is exemplified by the development of the TTM application itself, which originated from the analysis of multiple editions of the 17th-century legal text, *The Pronouncements in Sixteen Chapters* (*zhal lce bcu drug*). An initial attempt to create a critical edition using standard collation software like CollateX proved untenable; the variations between editions were so substantial that they produced a convoluted apparatus that obscured, rather than clarified, the texts' relationships. It became clear that a different approach was needed—one that could move beyond one-to-one textual comparison to provide a higher-level, quantitative overview of textual similarity. TTM was developed to meet this need, providing a toolkit to assess relationships at the chapter level and reveal the broader patterns of textual evolution that traditional methods might miss.
 
 ### 1.2. Digital Humanities and Under-Resourced Languages
 
@@ -14,7 +14,7 @@ The rise of digital humanities has brought a wealth of computational tools to li
 
 ### 1.3. The Tibetan Text Metrics (TTM) Application
 
-This paper introduces the Tibetan Text Metrics (TTM) web application, a user-friendly, open-source tool designed to make sophisticated textual analysis accessible to scholars of Tibetan, regardless of their technical background. The application allows researchers to upload Tibetan texts, automatically segment them into meaningful sections, and compare them using a range of quantitative metrics. This article will detail the methodologies underpinning the TTM application, describe its key features, and demonstrate its practical utility through a case study. By doing so, we aim to show how TTM can serve as a valuable assistant in the scholar's toolkit, augmenting traditional research methods and opening up new possibilities for the study of Tibetan textual history.
+This paper introduces the Tibetan Text Metrics (TTM) web application, a user-friendly, open-source tool designed to make sophisticated textual analysis accessible to scholars of Tibetan, regardless of their technical background. The application empowers researchers to move beyond manual comparison by providing a suite of computational metrics that reveal distinct aspects of textual relationships—from direct lexical overlap (Jaccard similarity) and shared narrative structure (Normalized LCS) to thematic parallels (TF-IDF) and deep semantic connections (FastText and Transformer-based embeddings). This article will detail the methodologies underpinning these metrics, describe the application's key features—including its novel AI-powered interpretation engine—and demonstrate its practical utility through a case study. By doing so, we aim to show how TTM can serve as a valuable assistant in the scholar's toolkit, augmenting traditional research methods and opening up new possibilities for the study of Tibetan textual history.
 
 ## 2. Methodology: A Multi-faceted Approach to Text Similarity
 
@@ -28,7 +28,12 @@ Meaningful computational analysis begins with careful pre-processing. The TTM ap
 
 **Tokenization:** To analyze a text computationally, it must be broken down into individual units, or tokens. Given the syllabic nature of the Tibetan script, where morphemes are delimited by a *tsheg* (་), simple whitespace tokenization is inadequate. TTM leverages the `botok` library, a state-of-the-art tokenizer for Tibetan, which accurately identifies word boundaries, ensuring that the subsequent analysis is based on meaningful linguistic units.
 
-**Stopword Filtering:** Many words in a language (e.g., particles, pronouns) are grammatically necessary but carry little unique semantic weight. These "stopwords" can skew similarity scores by creating an illusion of similarity based on common grammatical structures. TTM provides two levels of optional stopword filtering: a *Standard* list containing the most common particles, and an *Aggressive* list that also removes function words. This allows researchers to focus their analysis on the substantive vocabulary of a text.
+**Stopword Filtering:** Many words in a language are grammatically necessary but carry little unique semantic weight. These "stopwords" can skew similarity scores by creating an illusion of similarity based on common grammatical structures. TTM provides two levels of optional stopword filtering to address this:
+
+*   The **Standard** list targets only the most frequent, low-information grammatical particles and punctuation (e.g., the instrumental particle `གིས་` (gis), the genitive particle `གི་` (gi), and the sentence-ending *shad* `།`).
+*   The **Aggressive** list includes the standard particles but also removes a wider range of function words, such as pronouns (e.g., `འདི` (this), `དེ་` (that)), auxiliary verbs (e.g., `ཡིན་` (is)), and common quantifiers (e.g., `རྣམས་` (plural marker)).
+
+This tiered approach allows researchers to fine-tune their analysis, either preserving the grammatical structure or focusing purely on the substantive vocabulary of a text.
 
 ### 2.2. Lexical and Thematic Similarity Metrics
 
@@ -49,3 +54,61 @@ To capture similarities in meaning that may not be apparent from lexical overlap
 **FastText Embeddings:** The application utilizes the official Facebook FastText model for Tibetan, which represents words as dense vectors in a high-dimensional space. A key advantage of FastText is its use of character n-grams, allowing it to generate meaningful vectors even for out-of-vocabulary words—a crucial feature for handling the orthographic variations common in Tibetan manuscripts. To create a single vector for an entire text segment, TTM employs a sophisticated TF-IDF weighted averaging of the word vectors, giving more weight to the embeddings of characteristic terms.
 
 **Hugging Face Models:** In addition to FastText, TTM integrates the `sentence-transformers` library, providing access to a wide range of pre-trained models from the Hugging Face Hub. This allows researchers to leverage powerful, context-aware models like LaBSE or XLM-RoBERTa, which can capture nuanced semantic relationships between entire sentences and paragraphs.
+
+## 3. The TTM Web Application: Features and Functionality
+
+The TTM application is designed to be a practical tool for researchers. Its features are built to facilitate an intuitive workflow, from data input to the interpretation of results.
+
+### 3.1. User Interface and Workflow
+
+Built with the Gradio framework, the application's interface is clean and straightforward. The workflow is designed to be linear and intuitive:
+
+1.  **File Upload:** Users begin by uploading one or more Tibetan `.txt` files.
+2.  **Configuration:** Users can then configure the analysis by selecting which metrics to compute, choosing an embedding model, and setting the desired level of stopword filtering.
+3.  **Execution:** A single "Run Analysis" button initiates the entire processing pipeline.
+
+This simple, step-by-step process removes the barriers of command-line tools and complex software setups, making the technology accessible to all scholars.
+
+### 3.2. Data Visualization
+
+Understanding numerical similarity scores can be challenging. TTM addresses this by providing rich, interactive visualizations:
+
+*   **Heatmaps:** For each similarity metric, the application generates a heatmap that provides an at-a-glance overview of the relationships between all text segments. Darker cells indicate higher similarity, allowing researchers to quickly identify areas of strong textual connection.
+*   **Bar Charts:** A word count chart for each text provides a simple but effective visualization of the relative lengths of the segments, which is important context for interpreting the similarity scores.
+
+These visualizations are not only useful for analysis but are also publication-ready, allowing researchers to easily incorporate them into their own work.
+
+### 3.3. AI-Powered Interpretation
+
+A standout feature of the TTM application is its AI-powered interpretation engine. While quantitative metrics are powerful, their scholarly significance is not always self-evident. The "Interpret Results" button addresses this challenge by sending the computed metrics to a large language model (Mistral 7B via the OpenRouter API).
+
+The AI then generates a qualitative analysis of the results, framed in the language of textual scholarship. This analysis typically includes:
+
+*   An overview of the general patterns of similarity.
+*   A discussion of notable chapters with particularly high or low similarity.
+*   An interpretation of what the different metrics collectively suggest about the texts' relationship (e.g., lexical borrowing vs. structural parallels).
+*   Suggestions for further scholarly investigation.
+
+This feature acts as a bridge between the quantitative data and its qualitative interpretation, helping researchers to understand the implications of their findings and to formulate new research questions.
+
+## 5. Discussion and Future Directions
+
+### 5.1. Interpreting the Metrics: A Holistic View
+
+The true analytical power of the TTM application lies not in any single metric, but in the synthesis of all of them. For example, a high Jaccard similarity combined with a low LCS score might suggest that two texts share a common vocabulary but arrange it differently, perhaps indicating a shared topic but different authorial styles. Conversely, a high LCS score with a moderate Jaccard similarity could point to a shared structural backbone or direct borrowing, even with significant lexical variation. The addition of semantic similarity further enriches this picture, revealing conceptual connections that might be missed by lexical and structural methods alone. The TTM application facilitates this holistic approach, encouraging a nuanced interpretation of textual relationships.
+
+### 5.2. Limitations
+
+While powerful, the TTM application has limitations. The quality of the analysis is highly dependent on the quality of the input texts; poorly scanned or OCR'd texts may yield unreliable results. The performance of the semantic models, while state-of-the-art, may also vary depending on the specific domain of the texts being analyzed. Furthermore, the AI-powered interpretation, while a useful guide, is not a substitute for scholarly expertise and should be treated as a starting point for further investigation, not a definitive conclusion.
+
+### 5.3. Future Work
+
+The TTM project is under active development, with several potential avenues for future enhancement. These include:
+
+*   **Integration of More Models:** Expanding the library of available embedding models to include more domain-specific options.
+*   **Enhanced Visualization:** Adding more advanced visualization tools, such as network graphs to show relationships between multiple texts.
+*   **User-Trainable Models:** Exposing the functionality to train custom FastText models directly within the web UI, allowing researchers to create highly specialized models for their specific corpora.
+
+## 6. Conclusion
+
+The Tibetan Text Metrics web application represents a significant step forward in making computational textual analysis accessible to the field of Tibetan studies. By combining a suite of powerful similarity metrics with an intuitive user interface and a novel AI-powered interpretation engine, TTM lowers the barrier to entry for digital humanities research. It provides scholars with a versatile tool to explore textual relationships, investigate manuscript histories, and generate new, data-driven insights. As such, TTM serves not as a replacement for traditional philology, but as a powerful complement, one that promises to enrich and expand the horizons of Tibetan textual scholarship.

app.py

@@ -48,6 +48,18 @@ def main_interface():
                         "<small>Note: Maximum file size: 10MB per file. For optimal performance, use files under 1MB.</small>",
                         elem_classes="gr-markdown"
                     )
+
+            with gr.Column(scale=1, elem_classes="step-column", elem_id="chapter-rename-column"):
+                chapter_rename_group = gr.Group(visible=False)
+                with chapter_rename_group:
+                    gr.Markdown(
+                        """## Step 1.5: Name Your Chapters (Optional)
+<span style='font-size:16px;'>Provide a name for each chapter below. These names will be used in the heatmaps and results.</span>
+                        """,
+                        elem_classes="gr-markdown",
+                    )
+                    chapter_names_ui = gr.Column()
+
             with gr.Column(scale=1, elem_classes="step-column"):
                 with gr.Group():
                     gr.Markdown(
@@ -257,9 +269,45 @@ Each segment is represented as a vector of these TF-IDF scores, and the cosine s
         # For now, this modification focuses on creating the plot object and making it an output.
         # The visual placement depends on how Gradio renders children of gr.Tab or if there's another container.
 
+        # Wire up the chapter renaming UI
+        file_input.upload(
+            setup_chapter_rename_ui,
+            inputs=[file_input],
+            outputs=[chapter_rename_group, chapter_names_ui, file_data_state]
+        )
+
         warning_box = gr.Markdown(visible=False)
 
-        def run_pipeline(files, enable_semantic, model_name, stopwords_option, batch_size, show_progress, progress=gr.Progress()):
+        # State to hold file info and chapter names
+        file_data_state = gr.State(value={})
+
+        def setup_chapter_rename_ui(files):
+            if not files:
+                return gr.update(visible=False), [], {}
+
+            file_data = {}
+            chapter_name_inputs = []
+            for file in files:
+                try:
+                    content = Path(file.name).read_text(encoding="utf-8-sig")
+                    segments = content.split('༈')
+                    num_chapters = len(segments)
+                    file_data[Path(file.name).name] = {'path': file.name, 'chapters': num_chapters}
+
+                    for i in range(num_chapters):
+                        default_name = f"{Path(file.name).name} - Chapter {i + 1}"
+                        chapter_name_inputs.append(gr.Textbox(label=f"Name for Chapter {i+1} in '{Path(file.name).name}'", value=default_name))
+                except Exception as e:
+                    logger.error(f"Error processing file {file.name} for chapter renaming: {e}")
+                    # Handle file read error gracefully
+                    pass
+
+            if not chapter_name_inputs:
+                return gr.update(visible=False), [], {}
+
+            return gr.update(visible=True), chapter_name_inputs, file_data
+
+        def run_pipeline(files, enable_semantic, model_name, stopwords_option, batch_size, show_progress, chapter_names_list, progress=gr.Progress()):
             """Run the text analysis pipeline on the uploaded files.
 
             Args:
@@ -340,6 +388,8 @@ Each segment is represented as a vector of these TF-IDF scores, and the cosine s
                     Path(file.name).name for file in files
                 ]  # Use Path().name to get just the filename
                 text_data = {}
+                # The chapter_names_list is a list of lists, flatten it
+                flat_chapter_names = [name for sublist in chapter_names_list for name in sublist]
                 
                 # Read files with progress updates
                 for i, file in enumerate(files):
@@ -390,15 +440,16 @@ Each segment is represented as a vector of these TF-IDF scores, and the cosine s
                     internal_model_id = "facebook-fasttext-pretrained"
 
                 df_results, word_counts_df_data, warning_raw = process_texts(
-                    text_data,
-                    filenames,
+                    text_data=text_data,
+                    filenames=filenames,
                     enable_semantic=enable_semantic_bool,
                     model_name=internal_model_id,
                     use_stopwords=use_stopwords,
                     use_lite_stopwords=use_lite_stopwords,
                     progress_callback=progress_tracker,
                     batch_size=batch_size,
-                    show_progress_bar=show_progress
+                    show_progress_bar=show_progress,
+                    chapter_names=flat_chapter_names
                 )
 
                 if df_results.empty:
@@ -504,9 +555,20 @@ Each segment is represented as a vector of these TF-IDF scores, and the cosine s
                 logger.error(f"Error in interpret_results: {e}", exc_info=True)
                 return f"Error interpreting results: {str(e)}"
         
+        # The `process_btn.click` call needs to be defined here.
+        # It will take inputs from all the configuration UI elements and the dynamic chapter name fields.
+        # The `chapter_names_ui` component, being a `gr.Column`, will pass the values of its children as a list.
         process_btn.click(
             fn=run_pipeline,
-            inputs=[file_input, semantic_toggle_radio, model_dropdown, stopwords_dropdown, batch_size_slider, progress_bar_checkbox],
+            inputs=[
+                file_input,
+                semantic_toggle_radio,
+                model_dropdown,
+                stopwords_dropdown,
+                batch_size_slider,
+                progress_bar_checkbox,
+                chapter_names_ui, # Pass the column containing dynamic textboxes
+            ],
             outputs=[
                 csv_output,
                 metrics_preview,
@@ -516,7 +578,7 @@ Each segment is represented as a vector of these TF-IDF scores, and the cosine s
                 heatmap_tabs["Semantic Similarity"],
                 heatmap_tabs["TF-IDF Cosine Sim"],
                 warning_box,
-            ]
+            ],
         )
         
         # Connect the interpret button

pipeline/process.py

@@ -57,7 +57,8 @@ def process_texts(
     use_lite_stopwords: bool = False,
     progress_callback = None,
     batch_size: int = 32,
-    show_progress_bar: bool = False
+    show_progress_bar: bool = False,
+    chapter_names: List[str] = None
 ) -> Tuple[pd.DataFrame, pd.DataFrame, str]:
     """
     Processes uploaded texts, segments them by chapter marker, and computes metrics between chapters of different files.
@@ -152,6 +153,7 @@ def process_texts(
     chapter_marker = "༈"
     fallback = False
     segment_texts = {}
+    chapter_name_counter = 0
     
     # Process each file
     for i, fname in enumerate(filenames):
@@ -181,14 +183,19 @@ def process_texts(
                 continue
                 
             for idx, seg in enumerate(segments):
-                seg_id = f"{fname}|chapter {idx+1}"
+                # Use custom chapter name if available
+                custom_name = chapter_names[chapter_name_counter] if chapter_names and chapter_name_counter < len(chapter_names) else f"Chapter {idx + 1}"
+                seg_id = f"{fname}|{custom_name}"
                 cleaned_seg = clean_tibetan_text_for_fasttext(seg)
-                segment_texts[seg_id] = cleaned_seg
+                segment_texts[seg_id] = (cleaned_seg, idx + 1) # Store text and original number
+                chapter_name_counter += 1
         else:
             # No chapter markers found, treat entire file as one segment
-            seg_id = f"{fname}|chapter 1"
+            custom_name = chapter_names[chapter_name_counter] if chapter_names and chapter_name_counter < len(chapter_names) else "Chapter 1"
+            seg_id = f"{fname}|{custom_name}"
             cleaned_content = clean_tibetan_text_for_fasttext(content.strip())
-            segment_texts[seg_id] = cleaned_content
+            segment_texts[seg_id] = (cleaned_content, 1)
+            chapter_name_counter += 1
             fallback = True
             
     # Generate warning if no chapter markers found
@@ -213,7 +220,7 @@ def process_texts(
             logger.warning(f"Progress callback error (non-critical): {e}")
 
     all_segment_ids = list(segment_texts.keys())
-    all_segment_contents = list(segment_texts.values())
+    all_segment_contents = [data[0] for data in segment_texts.values()]
     tokenized_segments_list = tokenize_texts(all_segment_contents)
 
     segment_tokens = dict(zip(all_segment_ids, tokenized_segments_list))
@@ -294,7 +301,7 @@ def process_texts(
                     logger.info("Using botok word-level tokenization for FastText model.")
                 
                 pair_metrics = compute_all_metrics(
-                    texts={seg1: segment_texts[seg1], seg2: segment_texts[seg2]},
+                    texts={seg1: segment_texts[seg1][0], seg2: segment_texts[seg2][0]},
                     token_lists={seg1: segment_tokens[seg1], seg2: segment_tokens[seg2]},
                     model=model,
                     enable_semantic=enable_semantic,
@@ -306,9 +313,10 @@ def process_texts(
                     show_progress_bar=show_progress_bar
                 )
                 
-                # Rename 'Text Pair' to show file stems and chapter number
+                # Rename 'Text Pair' to show file stems and chapter name
+                chapter_name = seg1.split('|', 1)[1]
                 pair_metrics.loc[:, "Text Pair"] = f"{file1} vs {file2}"
-                pair_metrics.loc[:, "Chapter"] = idx + 1
+                pair_metrics.loc[:, "Chapter"] = chapter_name
                 results.append(pair_metrics)
                 
             except Exception as e:
@@ -333,7 +341,7 @@ def process_texts(
     word_counts_data = []
     
     # Process each segment
-    for i, (seg_id, text_content) in enumerate(segment_texts.items()):
+    for i, (seg_id, (text_content, chapter_num)) in enumerate(segment_texts.items()):
         # Update progress
         if progress_callback is not None and len(segment_texts) > 0:
             try:
@@ -342,8 +350,7 @@ def process_texts(
             except Exception as e:
                 logger.warning(f"Progress callback error (non-critical): {e}")
             
-        fname, chapter_info = seg_id.split("|", 1)
-        chapter_num = int(chapter_info.replace("chapter ", ""))
+        fname, chapter_name = seg_id.split("|", 1)
         
         try:
             # Use botok for accurate word count for raw Tibetan text
@@ -356,6 +363,7 @@ def process_texts(
             word_counts_data.append(
                 {
                     "Filename": fname.replace(".txt", ""),
+                    "ChapterName": chapter_name,
                     "ChapterNumber": chapter_num,
                     "SegmentID": seg_id,
                     "WordCount": word_count,
@@ -367,6 +375,7 @@ def process_texts(
             word_counts_data.append(
                 {
                     "Filename": fname.replace(".txt", ""),
+                    "ChapterName": chapter_name,
                     "ChapterNumber": chapter_num,
                     "SegmentID": seg_id,
                     "WordCount": 0,