Entity extraction is the NLP (natural language processing) technique used to identify specific named entities within a block of text. In the context of AI brand monitoring, entity extraction is the process of parsing an AI model’s response to determine which brand names appear in it, handling variations, abbreviations, and fuzzy matches.

Naive brand monitoring uses simple string search: “does the text contain the exact string ‘BrandName’?” This works for straightforward mentions but misses many real-world cases:

  • Abbreviations: “AR” for “AskRank”, “GH” for “GitHub”
  • Misspellings: “HubSopt” for “HubSpot”
  • Capitalization variations: “hubspot”, “Hubspot”, “HubSpot” should all match
  • Partial matches: “Ranks Tracker” mentioned in the context of discussing AskRank
  • Context-dependent disambiguation: “Notion” could refer to the software or the general concept

Good entity extraction handles these cases without producing false positives (incorrectly flagging a non-mention) or false negatives (missing a real mention).

How AskRank performs entity extraction

AskRank uses a two-stage approach:

Stage 1 - Regex-based matching: a fast pattern-matching pass that catches high-confidence matches (exact brand name, known abbreviations, simple case variations). This stage handles the majority of mentions efficiently.

Stage 2 - Model-based extraction: for responses where stage 1 finds ambiguous signals or low confidence, Claude Haiku is used as a secondary parser. It reads the raw response and determines whether the brand in question was genuinely mentioned, handling fuzzy matches, transliterations, and context-dependent disambiguation. This stage runs only when needed, keeping costs manageable.

The combination gives high precision (low false positives) and high recall (low false negatives) across the variety of outputs different AI models produce.

Entity extraction and confidence scores

Each detected mention gets a confidence score. High-confidence detections (exact match, known alias) are treated as definite. Low-confidence detections are either elevated to the model-based stage or flagged for manual review in the dashboard.

Confidence thresholds can be adjusted: a stricter threshold reduces false positives but may miss some genuine mentions; a looser threshold catches more real mentions but introduces more noise. AskRank defaults are calibrated for balanced precision and recall, with the option to adjust sensitivity in your account settings.

False positive and false negative rates

For a brand monitoring tool to be reliable, its mention detection needs to be accurate. AskRank maintains a test set of 50 manually-annotated AI responses across multiple LLMs and periodically measures the false positive and false negative rates of its entity extraction pipeline. The current target is a false positive rate below 5%.