HALvesting: Harvests open scientific papers from HAL

HALvesting is a Python project designed to harvest research papers from Hyper Articles en Ligne (HAL) and turn them into a language modeling dataset.

The latest dump can be found on HuggingFace.

Quickstart

• Overview
  • Dataset Summary
• Installation
  • Requirements
  • Dependencies
  • Installing
• How to Use HALvesting
  • fetch_data.py
  • merge_data.py
  • enrich_data.py
  • filter_data.py

Code

• API
• Downloader
• Merger
• Enricher
• Filtering
• Preprocessing
• Postprocessing
• KenLM Utils
• Utils

About

• About

Indices and tables

• Index

• Module Index