Nov
13
awarded Yearling
Nov
13
awarded Yearling
Oct
30
revised Cannot import scipy.misc.imread
removed the elaboration of the deprecation as it distracts from the main objective of the answer - to provide a way to work with scipy >= 1.3.0
Oct
23
awarded Yearling
Oct
23
awarded Yearling
Oct
23
revised Cannot import scipy.misc.imread
Made the wording more clear
Oct
3
revised Different optimization behaviours on delta vs non-delta targets
Continued the numbering of the headers
Sep
30
comment How to return 0 with divide by zero
I tried your method with the example arrays given in DStauffman's answer and it seems to result in very high numbers instead of np.inf, which remains at the final result
Sep
28
comment How to iterate over this n-dimensional dataset?
correction, these are functions of numpy and not of ndarray: ndenumerate, nditer
Sep
28
comment How to iterate over this n-dimensional dataset?
if it's an ndarray and you want to iterate over it without having the index iterated as well, you can use ndarray.nditer instead of ndarray.ndenumerate
Sep
21
revised Different optimization behaviours on delta vs non-delta targets
edited title
Sep
21
revised Different optimization behaviours on delta vs non-delta targets
Made the first section a bit more clear by fixing typos, summarizing and rearanging sentences
Sep
21
awarded Teacher
Sep
21
revised Different optimization behaviours on delta vs non-delta targets
Added a test to try to explain oscilations on non-delta dist with only learning rate
Sep
21
revised Different optimization behaviours on delta vs non-delta targets
Changed the title of each section. The sections are actually adressing the different distributions separetly
Sep
21
awarded Editor
Sep
21
revised Different optimization behaviours on delta vs non-delta targets
Added a more mathematically rigorous explanation to why lower alpha achieves higher loss for the delta distribution
Sep
20
answered Different optimization behaviours on delta vs non-delta targets
Sep
20
comment Different optimization behaviours on delta vs non-delta targets
I believe I found the answer to this. In the case that the target distribution is a delta function, we would like the output of the softmax to be [0, 1]. In order to reach that output the scores of the linear layer is required to have a difference of infinity. In addition, in order to achieve the same order of magnitude of loss that the non-delta case achieves, the linear model still has to supply huge differences between the scores. This requires a weight matrix with huge weights and it couldn't be achieved with iterations with a small learning rate.
Sep
20
comment Different optimization behaviours on delta vs non-delta targets
That indeed eliminates the ripples, but the achieved loss for the non-delta distribution is still lower by multiple orders of magnitude. Why can't the optimizer reach similar loss on the delta distribution?
1 2 3 4 5