libreasr.lib.layers.mish.MishAutoFn

class libreasr.lib.layers.mish.MishAutoFn

Bases: torch.autograd.function.Function

Methods

apply
backward	Defines a formula for differentiating the operation.
forward	Performs the operation.
mark_dirty	Marks given tensors as modified in an in-place operation.
mark_non_differentiable	Marks outputs as non-differentiable.
mark_shared_storage
register_hook
save_for_backward	Saves given tensors for a future call to backward().

Attributes

dirty_tensors
is_traceable
metadata
needs_input_grad
next_functions
non_differentiable
requires_grad
saved_tensors
saved_variables
to_save

__call__(*args, **kwargs)

Call self as a function.

static backward(ctx, grad_output)

Defines a formula for differentiating the operation.

This function is to be overridden by all subclasses.

It must accept a context ctx as the first argument, followed by as many outputs did forward() return, and it should return as many tensors, as there were inputs to forward(). Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input.

The context can be used to retrieve tensors saved during the forward pass. It also has an attribute ctx.needs_input_grad as a tuple of booleans representing whether each input needs gradient. E.g., backward() will have ctx.needs_input_grad[0] = True if the first input to forward() needs gradient computated w.r.t. the output.

static forward(ctx, x)

Performs the operation.

This function is to be overridden by all subclasses.

It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).

The context can be used to store tensors that can be then retrieved during the backward pass.

mark_dirty(*args)

Marks given tensors as modified in an in-place operation.

This should be called at most once, only from inside the forward() method, and all arguments should be inputs.

Every tensor that’s been modified in-place in a call to forward() should be given to this function, to ensure correctness of our checks. It doesn’t matter whether the function is called before or after modification.

mark_non_differentiable(*args)

Marks outputs as non-differentiable.

This should be called at most once, only from inside the forward() method, and all arguments should be outputs.

This will mark outputs as not requiring gradients, increasing the efficiency of backward computation. You still need to accept a gradient for each output in backward(), but it’s always going to be a zero tensor with the same shape as the shape of a corresponding output.

This is used e.g. for indices returned from a max Function.

save_for_backward(*tensors)

Saves given tensors for a future call to backward().

This should be called at most once, and only from inside the forward() method.

Later, saved tensors can be accessed through the saved_tensors attribute. Before returning them to the user, a check is made to ensure they weren’t used in any in-place operation that modified their content.

Arguments can also be None.