VARTA-2026

About the Track

The word Varta originates from Indian languages and denotes news, reporting, or public communication. News narratives frequently encompass multiple conflicting viewpoints and frame individual stakeholders in highly disparate ways. Resolving entity-specific sentiment profile tracking is key to structural breakthroughs in fields like computational journalism, media narrative tracking, and subtle media bias observation.

Traditional metrics look blindly at sentence or document levels. VARTA-2026 offers a specialized benchmark designed to extract highly granular, structured annotations centered around key stakeholders directly out of complex, long-context text environments.

Task Definition

Participants are tasked with evaluating a document and mapping out entity expressions using a structured 5-part architecture framework:

Sample Input / Output Format

To generalize across systems, the input paragraph is shown here in English. Submitted architectures must extract a list of 5-tuple blocks formatted cleanly in JSON.

[
  {
    "salient_entity": "Ministry of Finance",
    "opinion_by_entity": "confidently announced a new tax exemption policy for tech startups to encourage regional innovation",
    "opinion_targeted_to_target_entity": "welcomed enthusiastically by leading startup incubator",
    "polarity_of_opinion_targeted_to_target_entity": "Positive",
    "intensity_of_the_opinion": "High"
  },
  {
    "salient_entity": "Startup incubator",
    "opinion_by_entity": "enthusiastically welcomed the decision",
    "opinion_targeted_to_target_entity": "",
    "polarity_of_opinion_targeted_to_target_entity": null,
    "intensity_of_the_opinion": null
  },
  {
    "salient_entity": "Statistical bureau",
    "opinion_by_entity": "",
    "opinion_targeted_to_target_entity": "updated its quarterly index charts to reflect the policy change",
    "polarity_of_opinion_targeted_to_target_entity": "Factual",
    "intensity_of_the_opinion": "Low"
  }
]

Dataset

Reflecting the linguistic diversity of the Indian news ecosystem, our curated dataset contains source material spanning several languages:

Evaluation Metrics

System submissions will be benchmarked using a per-record evaluation strategy that jointly assesses entity recognition, semantic extraction quality, and classification accuracy:

• Salient Entity Alignment: Evaluated using token-level Overlap F1-Score rather than exact string matching. This ensures systems are fairly rewarded for capturing partial or synonymous entity mentions (e.g., matching "Ministry of Finance" with "Finance Ministry").
• Opinion Fields Correctness (By / Targeted To): Because opinions are expressed in natural language text, they are evaluated across two dimensions:
  • Semantic Correctness: Measured via BERTScore and embedding-based cosine similarity to verify that the extracted text accurately preserves the meaning of the ground truth.
  • Syntactic Correctness: Evaluated using language model Perplexity (PPL) to penalize fragmented "word-salad" outputs and reward grammatically coherent, fluent extractions.
• Polarity Classification: Evaluated using Macro F1-Score across the three distinct classes: Positive, Negative, and Factual.
• Intensity Accuracy: Evaluated using Mean Absolute Error (MAE) by mapping the ordinal scale labels to numeric weights (Low = 1, Medium = 2, High = 3), ensuring near-miss predictions are penalized less than complete opposites.

*Note: The evaluation script strictly enforces constraints; if an opinion_targeted_to_target_entity is correctly predicted as an empty string (""), the associated polarity and intensity metrics must match the null ground truth to avoid penalties.

Track Timeline

Our operation map coordinates with the master schedule outlined by the FIRE 2026 organizing committee:

Track Organizers

For any queries regarding the dataset, task criteria, or evaluation process, please contact the organizers or mail to varta2026.fire@gmail.com

• Dr. Prashant Kapil
  Assistant Professor, Bennett University, Uttar Pradesh, India
• Dr. Alapan Kuila
  Assistant Professor, Indian Institute of Information Technology Design and Manufacturing (IIITDM) Kurnool, Andhra Pradesh, India