Neural Networks with weight plus input instead of multiplication?

by

Summary

This postmortem analyzes an experimental neural‑network design that replaces the standard weighted multiplication (w_i x_i) with a weight–input addition (w_i + x_i). While the idea appears computationally attractive, the resulting model underperforms and introduces structural issues that make deeper architectures difficult to train—even with autograd.

Root Cause

The core issue is that addition destroys the expressive power of the linear transformation normally performed by matrix multiplication. Specifically:

  • No feature interaction occurs because addition does not mix inputs across dimensions.
  • The model collapses into a biased sum, making layers behave like repeated bias shifts rather than learned transformations.
  • Gradients become uninformative, because the derivative of (x + w) with respect to (w) is always 1, eliminating meaningful learning signals.
  • Layer stacking becomes ineffective, since each layer simply adds constants and applies nonlinearities without learning complex mappings.

Why This Happens in Real Systems

Real neural networks rely on affine transformations to create rich, high‑dimensional feature interactions. When this structure is removed:

  • The network cannot rotate, scale, or project data into new subspaces.
  • The model becomes functionally equivalent to a shallow additive model, regardless of depth.
  • Optimization algorithms lose the ability to shape the loss landscape because gradients are flat and uniform.

Real-World Impact

Systems built with additive “weights” instead of multiplicative weights typically show:

  • Poor accuracy on tasks requiring nonlinear feature interactions.
  • Inability to scale to deeper architectures.
  • Misleading performance improvements in forward/backward speed that mask the loss of representational capacity.
  • Difficulty debugging, because the model appears to “train” but never meaningfully improves.

Example or Code (if necessary and relevant)

Below is a minimal PyTorch implementation showing that autograd can compute gradients for the additive formulation, but the gradients are trivial and uninformative:

import torch
import torch.nn as nn

class AdditiveLayer(nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.W = nn.Parameter(torch.randn(in_dim, out_dim))

    def forward(self, x):
        # x: (batch, in_dim)
        # W: (in_dim, out_dim)
        # output: (batch, out_dim)
        return torch.sum(x.unsqueeze(2) + self.W, dim=1)

x = torch.randn(32, 11)
layer = AdditiveLayer(11, 50)
out = layer(x)
loss = out.mean()
loss.backward()

print(layer.W.grad)

This code works with autograd, but the gradient structure is constant, confirming the lack of learning capacity.

How Senior Engineers Fix It

Experienced engineers typically address this by:

  • Restoring multiplicative structure, e.g., standard matrix multiplication.
  • Using efficient linear algebra kernels rather than redesigning the math.
  • Applying low‑rank approximations, factorized layers, or kernel tricks when seeking computational savings.
  • Leveraging JIT compilation, fused operations, or quantization to reduce compute cost without sacrificing expressiveness.
  • Ensuring that any architectural change preserves gradient richness and feature mixing.

Why Juniors Miss It

Less‑experienced engineers often overlook:

  • The mathematical role of linear transformations in deep learning.
  • How feature interactions emerge from matrix multiplication.
  • The importance of gradient diversity for effective optimization.
  • That computational shortcuts can unintentionally collapse model capacity.
  • That autograd will happily compute gradients—even if those gradients are useless.

By understanding these pitfalls, engineers can better evaluate unconventional architectures and avoid designs that sacrifice expressive power for superficial efficiency gains.

Leave a Comment