JSON parse array of data

by

Summary

The provided code snippet attempts to parse a JSON array containing objects with varying structures and walk through the elements to extract specific fields. The root cause of the failure lies in a fundamental misunderstanding of Perl’s reference syntax and the structure of the parsed JSON data. Specifically, the attempt foreach ( $data->[] ) is syntactically invalid because $data is already an array reference (dereferencing it requires index notation, not an empty bracket suffix). Additionally, the inner structures (the data arrays) are heterogeneous, meaning not all objects contain the headline key. To produce a clean result, the code must handle missing keys gracefully and flatten the nested arrays to iterate over the leaf objects.

Root Cause

  1. Invalid Reference Dereferencing: In Perl, a scalar holding an array reference is dereferenced using @{$data} or indexed as $data->[0]. The syntax $data->[] is not valid Perl and will cause a compilation error. The user attempted to treat the reference as if it could be looped directly without expanding it.
  2. Ignoring JSON Structure: The JSON is a top-level array of objects. Inside the data key of each object is another array of objects. To “walk through” all elements, the logic must account for two levels of nesting (the outer array and the inner data arrays).
  3. Missing Key Handling: The objects within the data arrays are inconsistent. Some contain both headline and displayText, while others contain only displayText. A robust iterator must check for the existence of keys before accessing them to avoid undef warnings or errors.
  4. Variable Scope and Dumper Usage: The code uses Data::Dumper correctly to visualize the structure, but the subsequent loop attempts to iterate over the raw reference rather than iterating over the complex nested structure returned by decode.

Why This Happens in Real Systems

In production environments, this type of error occurs frequently when integrating with external APIs or microservices.

  • Schema Drift: JSON payloads are often generated dynamically. If a field becomes optional or is omitted under certain conditions (like the headline in the example), developers often fail to implement defensive checks. This leads to runtime errors when the code assumes a strict schema.
  • Ambiguous Data Structures: Developers sometimes conflate array references with lists. In Perl, distinguishing between scalar references and list data is crucial. Misunderstanding this boundary is a common source of “Not an ARRAY reference” runtime errors.
  • Over-reliance on Dumping: While Data::Dumper is excellent for debugging, seeing a complex structure printed doesn’t automatically translate to knowing how to access it programmatically. Junior engineers often struggle to map the visual output to the necessary dereferencing code.

Real-World Impact

  • Data Loss: If the parsing logic crashes, the entire batch of data (including the valid sent timestamps) is discarded or fails to process.
  • Silent Failures: If strict checking (use strict; use warnings;) is not enabled, accessing undefined variables might return empty strings, leading to corrupted database entries where displayText is missing without triggering an alert.
  • Application Instability: In a long-running service, unhandled exceptions during JSON parsing can crash worker threads or processes, reducing system availability and requiring manual restarts.

Example or Code

The following Perl script correctly parses the JSON, handles the missing headline field gracefully, and prints all sent dates and displayText values.

use strict;
use warnings;
use JSON qw( );
use Data::Dumper qw( );

my $json_text = '[ { "sent": "2026-01-16T17:00:00Z", "data": [ { "headline": "text1", "displayText": "text2" }, { "displayText": "text3" }, { "displayText": "text4" } ] }, { "sent": "2026-01-16T17:00:00Z", "data": [ { "headline": "text5", "displayText": "text6" }, { "displayText": "text7" }, { "displayText": "text8" }, { "headline": "text9", "displayText": "text10" } ] } ]';

my $json = JSON->new;
my $data = $json->decode($json_text);

# Iterate over the top-level array (using scalar reference syntax)
foreach my $item (@$data) {
    # Access the nested 'data' array
    my $inner_data = $item->{data};

    # Print the sent date for this group
    print "Sent Date: " . $item->{sent} . "\n";

    # Iterate over the inner array
    foreach my $record (@$inner_data) {
        # Defensive check for existence of 'displayText'
        if (exists $record->{displayText}) {
            print " - Display Text: " . $record->{displayText} . "\n";
        }

        # Check for optional 'headline'
        if (exists $record->{headline}) {
            print " - Headline: " . $record->{headline} . "\n";
        }
    }
    print "-" x 20 . "\n";
}

How Senior Engineers Fix It

Senior engineers approach this problem with a focus on robustness and defensive programming:

  1. Explicit Dereferencing: They use explicit syntax (@{$data} or @{ $item->{data} }) to clarify that a reference is being treated as a list. This prevents ambiguous parsing by the interpreter.
  2. Feature Checks: They utilize the exists function before accessing hash keys. This prevents runtime errors when encountering records that lack expected fields.
  3. Iterative Development: They write the code to print the Dumper output first, identify the data structure manually, and then write the loop logic. They do not attempt to write the loop logic while guessing the structure.
  4. Separation of Concerns: In larger systems, they would define a Schema or a validation class (e.g., using Moose or Type::Tiny) to normalize the data immediately after decoding, ensuring downstream logic only ever sees clean, predictable objects.

Why Juniors Miss It

  1. Reference vs. Value Confusion: Perl’s sigils ($, @, %) and reference arrows (->) are a common stumbling block. Juniors often struggle to remember that a hash reference accessed via $ref->{key} returns a scalar value, which might itself be an array reference requiring further dereferencing.
  2. Assumption of Uniformity: Junior developers often assume that all objects in an array share the exact same keys. They write code assuming $record->{headline} always exists, leading to failures when the data source changes slightly.
  3. Syntax Guessing: The syntax $data->[] suggests a misunderstanding of how to declare an empty index or how to iterate. It looks like a valid attempt to “start an array loop” but confuses the compiler.