3 ms·
SVMs are randomly initialized (with arbitrary priors) and then are deterministic. From "What Is the Random Seed on SVM Sklearn, and Why Does It Produce Differe
by westurner 3y ago
SVMs are randomly initialized (with arbitrary priors) and then are deterministic.
From "What Is the Random Seed on SVM Sklearn, and Why Does It Produce Different Results?" https://saturncloud.io/blog/what-is-the-random-seed-on-svm-sklearn-and-why-does-it-produce-different-results/ https://saturncloud.io/blog/what-is-the-random-seed-on-svm-s... :
> When you train an SVM model in sklearn, the algorithm uses a random initialization of the model parameters. This is necessary to avoid getting stuck in a local minimum during the optimization process.
> The random initialization is controlled by a parameter called the random seed. The random seed is a number that is used to initialize the random number generator. This ensures that the random initialization of the model parameters is consistent across different runs of the code
From "Random Initialization For Neural Networks : A Thing Of The Past" (2018)
https://towardsdatascience.com/random-initialization-for-neural-networks-a-thing-of-the-past-bfcdd806bf9e https://towardsdatascience.com/random-initialization-for-neu... :
> Lets look at three ways to initialize the weights between the layers before we start the forward, backward propagation to find the optimum weights.
> 1: zero initialization
> 2: random initialization
> 3: he-et-al initialization
Deep learning:
https://en.wikipedia.org/wiki/Deep_learning https://en.wikipedia.org/wiki/Deep_learning
SVM: https://en.wikipedia.org/wiki/Support_vector_machine https://en.wikipedia.org/wiki/Support_vector_machine
Is it guaranteed that SVMs converge upon a solution regardless of random seed?
- Dr_Birdbrain 3y agoAn SVM is a quadratic program, which is convex. This means that they should always converge and they should always converge to the same global optimum, regardless of initialization, as long as they are feasible, I.e. as long as the two classes can be separated by an SVM.
- steppi 3y agoThe soft-margin SVM which can handle misclassifications is also convex and has a unique global optimum [0]. [0] https://stackoverflow.com/a/12610455/992102 https://stackoverflow.com/a/12610455/992102
- westurner 3y ago> as long as the two classes can be separated by an SVM. Are the classes separable with e.g. the intertwined spiral dataset in the TensorFlow demo? Maybe only with a radial basis function kernel? Separable state https://en.wikipedia.org/wiki/Separable_state https://en.wikipedia.org/wiki/Separable_state : > In quantum mechanics, separable states are quantum states belonging to a composite space that can be factored into individual states belonging to separate subspaces. A state is said to be entangled if it is not separable. In general, determining if a state is separable is not straightforward and the problem is classed as NP-hard. An algorithm may converge upon the same wrong - or 'high error' - answer; regardless of a random seed parameter. It looks like there is randomization for SVMs for e.g. Platt scaling [1], though I had confused Simulated Annealing with SVMs. And then re-read Quantum Annealing; what is the ground state of the Hamiltonian any why would I use a hyperplane instead? [1] https://news.ycombinator.com/item?id=37369783 https://news.ycombinator.com/item?id=37369783
- steppi 3y agoThe article you’ve linked is incorrect. As Dr_Birdbrain said, fitting an SVM is a convex problem with unique global optimum. sklearn.SVC relies on libsvm which initializes the weights to 0 [0]. The random state is only used to shuffle the data to make probability estimates with Platt scaling [1]. Of the random_state parameter, the sklearn documentation for SVC [2] says Controls the pseudo random number generation for shuffling the data for probability estimates. Ignored when probability is False. Pass an int for reproducible output across multiple function calls. See Glossary. [0] https://github.com/scikit-learn/scikit-learn/blob/2a2772a87b6c772dc3b8292bcffb990ce27515a8/sklearn/svm/src/libsvm/svm.cpp#L1610 https://github.com/scikit-learn/scikit-learn/blob/2a2772a87b... [1] https://en.wikipedia.org/wiki/Platt_scaling https://en.wikipedia.org/wiki/Platt_scaling [2] https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html https://scikit-learn.org/stable/modules/generated/sklearn.sv...
- westurner 3y agoWhich article is incorrect? Indeed it looks like there is no random initialization in libsvm or thereby sklearn.svm.SVC or in sklearn.svm.*. I seem to have confused random initialization in Simulated Annealing with SVMs; though now TIL that there are annealing SVMs, and SVMs do work with wave functions (though it's optional to map the wave functions into feature space with quantum state tomography), and that there are SVMs for the D-Wave Quantum annealer QC. From "Support vector machines on the D-Wave quantum annealer" (2020) https://www.sciencedirect.com/science/article/pii/S001046551930342X https://www.sciencedirect.com/science/article/pii/S001046551... : Kernel-based support vector machines (SVMs) are supervised machine learning algorithms for classification and regression problems. We introduce a method to train SVMs on a D-Wave 2000Q quantum annealer and study its performance in comparison to SVMs trained on conventional computers. The method is applied to both synthetic data and real data obtained from biology experiments. We find that the quantum annealer produces an ensemble of different solutions that often generalizes better to unseen data than the single global minimum of an SVM trained on a conventional computer, especially in cases where only limited training data is available. For cases with more training data than currently fits on the quantum annealer, we show that a combination of classifiers for subsets of the data almost always produces stronger joint classifiers than the conventional SVM for the same parameters.