Sommarie
View on GitHubAn Extractive Summarization for News Articles using natural language processing.
The Mission
Extractive text summariser for news articles. Scores every sentence by word frequency, selects the top-N highest-scoring sentences, and returns them in original document order. Extractive summarisation is a well-scoped NLP problem with a clear correctness criterion: the output must be factually identical to the source. It's a practical exercise in building a clean text processing pipeline with classical algorithms and no deep learning required.
The Problem
Long news articles contain filler and repetition. Readers want the key facts without the full read. An extractive summariser must identify the most informative sentences, though frequency scoring naively over-weights the opening paragraph, which introduces topic vocabulary at high density.
How I solve it
TF-IDF-style word frequency pipeline: tokenise into sentences and words, remove stop words, compute word weights, score each sentence by the sum of its word weights, apply a position penalty to discount early sentences, then select the top-N scoring sentences. N is a configurable ratio of total sentence count so the summary scales with article length. Sentences are returned in their original document order to maintain narrative coherence.
Key Features
- Extractive: verbatim source sentences only
- TF-IDF word frequency scoring
- Position-aware penalty to reduce lede bias
- Configurable summary ratio (default 30%)
- Handles varied article structures
- CPU-only with no GPU or deep learning dependencies
Architecture Overview
A linear Python pipeline. Raw article text flows through preprocessing (tokenisation, normalisation), frequency scoring (word weights computed across the full document), sentence scoring (each sentence scored by its token weights), and top-N selection (highest-scored sentences returned in document order).
Raw news article (plain string or file)
Condensed extractive summary
Sentence & word tokenisation, stop word removal, normalisation
TF-IDF-style word frequency computation across full document
Scores each sentence by sum of token weights + position penalty
Picks highest-scoring sentences, returns in document order