Product price parsing on Wildberries using Kotlin (In the Android app)

by

Summary

The core issue is an attempt to perform client-side web scraping of a modern e-commerce platform (Wildberries) directly from an Android application. The user encountered immediate request blocking, likely due to sophisticated bot detection, WAF (Web Application Firewall) rules, and TLS fingerprinting. The root cause is a fundamental architectural mismatch: treating a complex, security-hardened website as a simple static HTML source. This approach fails because the target site actively defends against automated access and requires a valid session context (cookies, headers) that is difficult to spoof from a non-browser environment.

Root Cause

The failure stems from three primary technical barriers implemented by Wildberries to prevent scraping:

  • TLS/JA3 Fingerprinting: Modern security systems analyze the unique byte sequence of the TLS handshake. The libraries used in Android (OkHttp, Jsoup) create a fingerprint that is easily identifiable as “non-human” traffic and is likely blacklisted.
  • Bot Mitigation & WAF: The site likely employs services like Cloudflare or custom WAFs that inspect headers (User-Agent, Accept-Language), IP reputation, and behavioral patterns. Requests missing specific headers or originating from known datacenter IP ranges (mobile carrier proxies often appear as such) are dropped immediately.
  • Dynamic Content & Anti-Scraping Measures: Wildberries is a Single Page Application (SPA) heavily reliant on JavaScript. The product price is often not present in the initial HTML response but is rendered client-side or fetched via internal APIs protected by dynamic tokens. Simple HTTP clients like Jsoup cannot execute JS or handle these dynamic tokens.

Why This Happens in Real Systems

In modern web development, Data Scraping is a constant cat-and-mouse game between site owners and scrapers.

  • Resource Protection: E-commerce sites protect their data to prevent competitors from undercutting prices or overloading servers with scraping bots.
  • Mobile App Context: Legitimate mobile apps typically consume data via dedicated APIs (GraphQL or REST), not HTML parsing. The HTML source is considered “presentation layer” only. If the official app needs the price, it calls api.wildberries.ru or a similar internal endpoint.
  • Legal & ToS: Scraping often violates Terms of Service. Sites implement aggressive countermeasures (like blocking IP ranges) to enforce these rules.

Real-World Impact

Attempting this approach in a production environment leads to:

  • Service Instability: The application becomes unreliable. As soon as the app is deployed, network requests start failing.
  • Development Deadlock: Engineers waste weeks trying to reverse-engineer complex headers and TLS fingerprints, which change frequently.
  • Account Bans: If authentication cookies are used, the associated user account risks being permanently banned for violating Terms of Service.
  • Performance Degradation: Parsing heavy HTML pages on a mobile device consumes significant CPU and battery, leading to poor UX and high crash rates.

Example or Code (Necessary)

To illustrate the complexity, here is an example of the networking setup required to attempt a connection (though it will likely still fail without rotating residential proxies and constant header updates). This is often the starting point where the failure is observed.

import okhttp3.OkHttpClient
import okhttp3.Request
import okhttp3.Response
import java.io.IOException
import java.util.concurrent.TimeUnit

fun fetchProductPage() {
    // A basic client is insufficient for modern anti-bot systems.
    // Even adding headers often fails due to TLS fingerprinting.
    val client = OkHttpClient.Builder()
        .connectTimeout(10, TimeUnit.SECONDS)
        .readTimeout(10, TimeUnit.SECONDS)
        .addInterceptor { chain ->
            val originalRequest = chain.request()
            val newRequest = originalRequest.newBuilder()
                .header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36")
                .header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
                .header("Accept-Language", "en-US,en;q=0.5")
                .header("Accept-Encoding", "gzip, deflate, br")
                .header("Connection", "keep-alive")
                .header("Upgrade-Insecure-Requests", "1")
                .build()
            chain.proceed(newRequest)
        }
        .build()

    val request = Request.Builder()
        .url("https://www.wildberries.ru/catalog/474837861/detail.aspx")
        .build()

    try {
        val response: Response = client.newCall(request).execute()
        if (response.isSuccessful) {
            val htmlBody = response.body?.string()
            // This body will likely be a captcha page or an error page, 
            // not the actual product HTML, due to bot detection.
            println("Success: $htmlBody") 
        } else {
            println("Failed: ${response.code}") // Likely 403 Forbidden or 429 Too Many Requests
        }
    } catch (e: IOException) {
        e.printStackTrace()
    }
}

How Senior Engineers Fix It

Senior engineers abandon the scraping approach entirely and solve the problem via official or architecture-compliant methods:

  1. Use Official APIs: Wildberries provides an Open API for partners (Data API). If access is granted, this is the only stable source of truth for product data.
  2. Server-Side Proxy/Scraper (The “Backend” Pattern): If no API is available, the scraping logic is moved to a backend server (e.g., AWS Lambda, Python backend). This server uses specialized tools (Puppeteer, Playwright, or Scrapy with scrapy-zyte-smartproxy) to render the page and extract data. The Android app then calls this backend to get the clean JSON data.
  3. Affiliate/Partner Integration: Registering as a partner to gain access to product feeds.

Key Takeaway: Do not fight the anti-bot systems on the client side. Move data extraction to a controlled environment or use official channels.

Why Juniors Miss It

  • Underestimation of Frontend Complexity: Assuming that “HTML source contains data” implies “I can easily get the data,” ignoring that the HTML is often just a shell for JavaScript to populate.
  • Lack of Security Awareness: Not being familiar with TLS fingerprinting, JA3 signatures, and WAFs.
  • Tool Misapplication: Reaching for Jsoup (a document parser) to solve a network security problem.
  • “It works in Postman” Fallacy: Many juniors test the URL in a browser or Postman (which has a legitimate browser TLS fingerprint) and assume it will work the same way in an Android app (which uses a different network stack).