2
$\begingroup$

Is there a name for the class of functions whose derivative is expressed only in terms of the function value? One example is the exponential, another example is

\begin{align} s_1(t) = \frac{1}{1 + e^{-t}} \end{align}

with derivative

\begin{align} s'_1(t) = s_1(t)[1-s_1(t)]. \end{align}

Clarification

My question is related to writing about Neural Networks (NN). In neural networks you calculate the derivative of the output relative to the input by means of an algorithm called backpropagation, or backprop (which is really nothing but the chain rule expressed in a computationally-efficient manner).

An important computational advantage while doing backprop is to store the function value when propagating forward, and using that function value to compute the derivative when propagating backward. This is only possible if the derivative only depends on the function value (and not on, say, the variable value).

For example. Suppose that you have a working vector w:

# w is currently storing the value of t, the independent variable
w = [1, 2, 3]

in the first step you calculate the function value (you won't need the value of the independent variable $t$ anymore, so you overwrite the contents in memory)

# w is currently storing the value of s(t) = 1 / [1 + exp(-t)]
w = [0.7310585786300049, 0.8807970779778823, 0.9525741268224334]

in the next step you calculate the derivative value (you won't need the value of $s(t)$ anymore, so you overwrite the contents in memory)

# w is currently storing the value of s'(t) = s(t)[1-s(t)]
w = [0.19661193324148185, 0.10499358540350662, 0.045176659730912]

Notice that if $s'$ had a dependency on the value of $t$ (as opposed to only the value of $s(t)$) I would not be able to reuse the memory in w.

The specific paragraph I'm trying to improve reads as follows:

Pick an activation function whose derivative depends only on the function value, and not on the value of the independent variable. Such activation functions enable reusing the working memory during the backprop stage.

And I'd like it to know if this could be expressed more precisely: pick a ??? activation function.

  • 0
    They're solutions to [autonomous differential equations](https://en.wikipedia.org/wiki/Autonomous_system_(mathematics)), and if the graph of the function is a single piece you can characterize them as the [(weakly) monotonic](https://en.wikipedia.org/wiki/Monotonic_function#Monotonicity_in_calculus_and_analysis) differentiable functions. I'm not aware of a better name. – 2017-02-09
  • 0
    @epimorphic My problem is that the *differential equation* is autonomous, bu what do you call the function itself? I'm trying to express the following idea: "a function whose derivative does not depend on the independent variable is called: ???". Or, put it another way: "Every ??? function can be written as an autonomous differential equation". – 2017-02-09
  • 0
    I'm aware of that; my suggestions were "solutions to autonomous differential equations" and its variations, or in the certain aforementioned case, "(weakly) monotonic differentiable functions". If those don't do, you could always just define your own name in your work. I like the sound of "autonomously differentiable" (do note that it returns zero results on Google). – 2017-02-09
  • 0
    I like that name! It does seem to express the intent. If you add it as an answer I'll accept it. Thanks! – 2017-02-09

2 Answers 2

2

I think the answer to the literal question you ask is "no". I don't know of any name for that kind of function.

But behind that question, I think you are asking about describing a function as the solution to a (first order) differential equation, like $$ y' = Cy $$ for the exponential function or $$ y' = Cy(1-y) $$ for the logistic curve.

Solutions to first order equations often involve the exponential function.

  • 0
    Thank you. My question is more related to being able to *chain* calculations and reuse storage (from a computer perspective). Probably the edit I added to the question can clarify? I regret that my question is very unclear - apologies. – 2017-02-09
  • 0
    No need to apologize. Questions are often clarified here. @epimorphic 's comment is the best you'll get for a name for these functions. The wikipedia page he links to may help with numerical methods. – 2017-02-09
1

I think what you are looking for is the term "ordinary differential equations". These are equations that are defined with functions of a single variable as well as the derivatives of those functions. Wolfram MathWorld has a nice overview: http://mathworld.wolfram.com/OrdinaryDifferentialEquation.html

  • 0
    +1 Welcome to math.se. – 2017-02-09
  • 0
    Thank you, but that's not precisely what I'm asking. I'm asking for functions whose derivative only depends on the function value (and not on the variable itself). – 2017-02-09
  • 0
    I'm sorry, I misunderstood the nature of the question. Then I would have to agree with @EthanBolker. – 2017-02-09
  • 0
    @Addison Our answers are really the same. I just happened to pick examples of the form $y' = f(y)$. The OP specified that only in his comment. – 2017-02-09
  • 0
    @EthanBolker Definitely. Also thanks for the welcome! – 2017-02-09