Saturday, September 17, 2022

Logit and Sigmoid Functions and its Calculus

Logit and Sigmoid Functions and its Calculus



Definitions


The sigmoid function is defined as:


sigmoid(x) = 1 ÷ (1 + e^(-x))


The logit function is defined as:  


logit(p) = ln (p ÷ (1 - p))


For logit(p) to have a real number answer, 0 ≤ p < 1



Transform from the Sigmoid Function to the Logit Function


We can easily transform from the sigmoid function to the logit function.  


Let s = sigmoid(x). Then:


s = 1 ÷ (1 + e^(-x))

s * (1 + e^(-x)) = 1

s + s * e^(-x) = 1

s * e^(-x) = 1 - s

e^(-x) = (1 - s) ÷ s

e^x = s ÷ (1 - s)

x = ln(s ÷ (1 - s)) = logit(s)


To transform from the logit function to the sigmoid function, just go backwards.  



Sigmoid Function:  Derivative and Integral


Derivative


s = sigmoid(x)

s = 1 ÷ (1 + e^(-x))


Using the quotient rule of derivatives:

ds/dx = [(1 + e^(-x)) * 0 - 1 * -e^(-x)] ÷ (1 + e^(-x))^2

= -(-e^(-x)) ÷ (1 + e^(-x))^2

= -e^(-x) ÷ (1 + e^(-x))^2



Integral


s = sigmoid(x)

s = 1 ÷ (1 + e^(-x))


Multiply both sides by e^x ÷ e^x:


s * (e^x ÷ e^x) = (e^x ÷ e^x) * (1 ÷ (1 + e^(-x)))

s = e^x ÷ (e^x + 1)


Integral:

∫ e^x ÷ (e^x + 1) dx


Let u = e^x + 1.  Then du = e^x dx 

= ∫  du ÷ (u + 1) 

= ln (u + 1) + C

= ln (e^x + 1) + C


Summary:

d/dx sigmoid(x) = -e^(-x) ÷ (1 + e^(-x))^2

∫ sigmoid(x) dx = ln (e^x + 1) + C



Logit Function:  Derivative and Integral


Derivative


logit(p) = ln (p ÷ (1 - p))

L = ln (p ÷ (1 - p))


Derivative:

dL/dp =  [(1 - p) ÷ p] * d/dp ln (p ÷ (1 - p))

=  [(1 - p) ÷ p] * [(1 - p) * 1 - p * (-1)] ÷ [(1 - p)^2] 

=  [(1 - p) ÷ p] * [1 - p + p] ÷ [(1 - p)^2]

=  [(1 - p) ÷ p] * 1 ÷ (1 - p)^2

= 1 ÷ [p * (1 - p)]


Integral:

∫ ln (p ÷ (1 - p)) dp


By integration by parts:

u = ln (p ÷ (1 - p)) 

du = 1 ÷ [p * (1 - p)] dp


v = dp

v = p


Then:

∫u dv

= p * ln ( p ÷ (1 - p)) - ∫ p ÷ (1 - p) dp

= p * ln ( p ÷ (1 - p)) + ∫ -p ÷ (1 - p) dp

= p * ln ( p ÷ (1 - p)) + ln(1 - p) + C


In Summary:

d/dp logit(p) = 1 ÷ [p * (1 - p)]

∫ logit(p) dp = p * ln ( p ÷ (1 - p)) + ln(1 - p) + C


Eddie


All original content copyright, © 2011-2022.  Edward Shore.   Unauthorized use and/or unauthorized distribution for commercial purposes without express and written permission from the author is strictly prohibited.  This blog entry may be distributed for noncommercial purposes, provided that full credit is given to the author. 


Matrices in Python without Numpy: Part 1

Matrices in Python without Numpy:  Part 1 Introduction Python is a wonderful programming language and is a welcome addition to graphing calc...