class

`SushiBLAS::NonLinearOps`

Records activation functions and their backward passes on contiguous tensors.

Declared in: include/SushiBLAS/engine/math/nonlinear.hpp

Note

Every operand must have storage and dense strides, and the operands of a backward pass must share one shape: the library does not broadcast. Sigmoid and tanh accept real and complex dtypes; every other activation accepts real dtypes only.

Public member functions

explicit NonLinearOps(TaskRecorder &recorder) noexcept

Builds a view that records through recorder, which must outlive the view.

explicit NonLinearOps(Engine &engine) noexcept

Builds a view that records on engine's own graph.

NonLinearOps(const NonLinearOps &)=default

NonLinearOps & operator=(const NonLinearOps &)=delete

NonLinearOps(NonLinearOps &&)=default

NonLinearOps & operator=(NonLinearOps &&)=delete

~NonLinearOps()=default

TaskHandle relu(Tensor &t)

Rectified Linear Unit (ReLU).

Computes f(x) = max(0, x) element-wise.

Parameters

t: Input/Output tensor (modified in-place).

Returns

The handle of the recorded task.

TaskHandle relu_backward(const Tensor &dy, const Tensor &x, Tensor &dx)

ReLU Backward (Gradient).

Computes dx = dy * (x > 0 ? 1 : 0).

Parameters

dy: Output gradient (incoming).
x: Original forward input.
dx: Input gradient result.

Returns

The handle of the recorded task.

TaskHandle leaky_relu(Tensor &t, double alpha=0.01)

Leaky Rectified Linear Unit (LeakyReLU).

Computes f(x) = x if x > 0 else alpha * x.

Parameters

t: Input/Output tensor.
alpha: Small slope for negative values (default: 0.01).

Returns

The handle of the recorded task.

TaskHandle leaky_relu_backward(const Tensor &dy, const Tensor &x, Tensor &dx, double alpha=0.01)

Computes dx = dy * (x > 0 ?

1 : alpha), the LeakyReLU backward pass.

Parameters

x: The forward input.

TaskHandle sigmoid(Tensor &t)

Sigmoid Activation.

Computes f(x) = 1 / (1 + exp(-x)).

Parameters

t: Input/Output tensor.

Returns

The handle of the recorded task.

TaskHandle sigmoid_backward(const Tensor &dy, const Tensor &y, Tensor &dx)

Sigmoid Backward.

Computes dx = dy * y * (1 - y) where y = sigmoid(x).

Parameters

dy: Output gradient.
y: Forward output (the sigmoid result).
dx: Gradient result.

Returns

The handle of the recorded task.

TaskHandle tanh(Tensor &t)

Hyperbolic Tangent (Tanh).

Computes f(x) = tanh(x).

Parameters

t: Input/Output tensor.

Returns

The handle of the recorded task.

TaskHandle tanh_backward(const Tensor &dy, const Tensor &y, Tensor &dx)

Tanh Backward.

Computes dx = dy * (1 - y^2) where y = tanh(x).

Parameters

dy: Output gradient.
y: Forward output (the tanh result).
dx: Gradient result.

Returns

The handle of the recorded task.

TaskHandle elu(Tensor &t, double alpha=1.0)

Exponential Linear Unit (ELU).

Computes f(x) = x if x > 0 else alpha * (exp(x) - 1).

Parameters

t: Input/Output tensor.
alpha: ELU scale factor.

Returns

The handle of the recorded task.

TaskHandle elu_backward(const Tensor &dy, const Tensor &x, Tensor &dx, double alpha=1.0)

Computes dx = dy * (x > 0 ?

1 : alpha * exp(x)), the ELU backward pass.

Parameters

x: The forward input.

TaskHandle silu(Tensor &t)

Sigmoid Linear Unit (SiLU / Swish).

Computes f(x) = x * sigmoid(x).

Parameters

t: Input/Output tensor.

Returns

The handle of the recorded task.

TaskHandle silu_backward(const Tensor &dy, const Tensor &x, Tensor &dx)

SiLU Backward.

Computes dx = dy * (sig(x) * (1 + x * (1 - sig(x)))).

Parameters

dy: Output gradient.
x: Forward input.
dx: Gradient result.

Returns

The handle of the recorded task.

TaskHandle gelu(Tensor &t)

Gaussian Error Linear Unit (GELU).

Computes f(x) = x * P(X <= x) where X ~ N(0, 1). Approximated as: 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3))).

Parameters

t: Input/Output tensor.

Returns

The handle of the recorded task.

TaskHandle gelu_backward(const Tensor &dy, const Tensor &x, Tensor &dx)

GELU Backward.

Parameters

dy: Output gradient.
x: Forward input.
dx: Gradient result.

Returns

The handle of the recorded task.

TaskHandle softplus(Tensor &t)

Softplus activation.

Computes f(x) = max(x, 0) + log1p(exp(-|x|)), which equals ln(1 + exp(x)) without overflow.

Parameters

t: Input/Output tensor.

Returns

The handle of the recorded task.

TaskHandle softplus_backward(const Tensor &dy, const Tensor &x, Tensor &dx)

Softplus Backward.

Computes dx = dy * sigmoid(x).

Parameters

dy: Output gradient.
x: Forward input.
dx: Gradient result.

Returns

The handle of the recorded task.