Scraping sibling values with screen‑reader only classes

by

Summary

The issue involves a common scraping challenge: extracting specific data sibling or child nodes when the target text (e.g., “Price”) is encapsulated in a non-visual class (e.g., ud-sr-only) and multiple elements share the same CSS class. Using a naive selector like .ud-sr-only fails because it returns only the first match, leading to incorrect data extraction or logic errors in the scraper.

Root Cause

The failure stems from two primary factors:

  • Selector Ambiguity: Using a generic class selector like .ud-sr-only ignores the structural context. In modern web frameworks, utility classes like sr-only (screen-reader only) are reused dozens of times for different labels (Rating, Price, Stock, etc.).
  • Selector Behavior: The colly.OnHTML method with a generic selector triggers the callback for every match, but if the logic inside the callback is not conditioned to look for a specific relationship, the developer often inadvertently captures the first instance or the wrong sibling.

Why This Happens in Real Systems

  • Semantic HTML Abstraction: Developers use “screen-reader only” tags to provide accessibility without affecting the visual UI. This creates a dense, repetitive DOM structure that is difficult to parse with simple CSS selectors.
  • Component-Based Architecture: Modern front-ends (React/Vue) reuse identical HTML snippets for different data points. A “Price” component and a “Rating” component might share identical class names, making context-free selection impossible.

Real-World Impact

  • Data Pollution: Scrapers ingest the wrong values (e.g., grabbing “4.5” when the intent was “269,000”), leading to corrupted databases.
  • Brittle Pipelines: Small changes in the page layout or the order of elements cause the scraper to fail or return “garbage” data, increasing maintenance overhead.
  • High CPU/Latency: Attempting to solve this with massive, overly complex regex patterns or inefficient DOM traversals increases the computational cost of the scraping job.

Example or Code

package main

import (
    "fmt"
    "strings"

    "github.com/gocolly/colly/v2"
)

func main() {
    c := colly.NewCollector()

    // The goal is to find the text node that is a sibling/child 
    // relative to a specific label.
    c.OnHTML("span.ud-sr-only", func(e *colly.HTMLElement) {
        label := strings.TrimSpace(e.Text)

        if label == "Price" {
            // Move to the parent or next sibling to find the actual value
            value := e.DOM.Next().Text()
            fmt.Printf("Found Price: %s\n", value)
        } else if label == "Rating" {
            value := e.DOM.Next().Text()
            fmt.Printf("Found Rating: %s\n", value)
        }
    })

    html := `
        Rating 4.5 out of 5
        Price ₫269,000`

    c.LoadHTML(html)
}

How Senior Engineers Fix It

  • Relational Traversal: Instead of selecting by class alone, senior engineers use DOM Traversal. They identify the “Anchor” (the label with text “Price”) and then traverse to its parent, sibling, or child to locate the value.
  • XPath over CSS: When CSS selectors become too ambiguous, we switch to XPath. XPath allows for much more powerful queries, such as //span[contains(text(), "Price")]/following-sibling::span, which explicitly defines the relationship between the label and the value.
  • Robust Predicates: We implement logic that validates the format of the extracted data (e.g., checking if the extracted string contains a currency symbol) to ensure the integrity of the scraped data.

Why Juniors Miss It

  • Reliance on Single-Level Selectors: Juniors often assume that a CSS class is a unique identifier, which is rarely true in production-grade, modern web applications.
  • Lack of DOM Tree Awareness: Juniors treat the DOM as a flat list of elements rather than a hierarchical tree. They fail to account for how the label and the value are nested within parent containers.
  • Ignoring Accessibility Patterns: They don’t realize that sr-only classes are specifically designed to be repetitive for screen readers, making them terrible primary selectors for data extraction.

Leave a Comment