CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents
FabianKarl
Abstract
CRAWLDoc is a method for ranking linked web documents by embedding resources and metadata into a unified representation, enabling robust metadata extraction across diverse web layouts and formats.
Publication databases rely on accurate metadata extraction from diverse web sources, yet variations in web layouts and data formats present challenges for metadata providers. This paper introduces CRAWLDoc, a new method for contextual ranking of linked web documents. Starting with a publication's URL, such as a digital object identifier, CRAWLDoc retrieves the landing page and all linked web resources, including PDFs, ORCID profiles, and supplementary materials. It embeds these resources, along with anchor texts and the URLs, into a unified representation. For evaluating CRAWLDoc, we have created a new, manually labeled dataset of 600 publications from six top publishers in computer science. Our method CRAWLDoc demonstrates a robust and layout-independent ranking of relevant documents across publishers and data formats. It lays the foundation for improved metadata extraction from web documents with various layouts and formats. Our source code and dataset can be accessed at https://github.com/FKarl/CRAWLDoc.
Community
This paper introduces CRAWLDoc, a new method for contextual ranking of linked web documents. Additionally, a manually labeled dataset of 600 publications from six top computer science publishers is presented.
Both the source code and dataset are available at: https://github.com/FKarl/CRAWLDoc
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- A Unified Retrieval Framework with Document Ranking and EDU Filtering for Multi-document Summarization (2025)
- Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents (2025)
- Document Quality Scoring for Web Crawling (2025)
- Web Page Classification using LLMs for Crawling Support (2025)
- mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs (2025)
- Bridging Queries and Tables through Entities in Table Retrieval (2025)
- CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper