3 ms·
He defines it pretty clearly. Logits are the inputs to a softmax layer / calculation, which turn the logits into normalized percentages (the percentages sum to
by Buttons840 4y ago
He defines it pretty clearly. Logits are the inputs to a softmax layer / calculation, which turn the logits into normalized percentages (the percentages sum to 1.0).
Before going through the softmax layer, the logits will be small numbers around 0, probably. Something like: [2.89, -4.53, 0.24, -1.556, 0.57]. Logits like this are natural outputs of a neural network, because they can be any real number and everything will still work.
The logits become percentage as follows:
julia> x = [2.89, -4.53, 0.24, -1.556, 0.57]
5-element Vector{Float64}:
2.89
-4.53
0.24
-1.556
0.57
julia> x = e.^x
5-element Vector{Float64}:
17.993309601550315
0.010780676072743085
1.2712491503214047
0.2109782988178321
1.768267051433735
julia> x / sum(x)
5-element Vector{Float64}:
0.8465613320288766
0.0005072164987105474
0.05981058503789324
0.009926248902037537
0.08319461753248213
Logits is an overloaded term though, and means different things in different contexts.