Swish Function

The swish function is a mathematical function used as an activation function in artificial neural networks, defined as swish_beta(x) equal to x times the sigmoid of beta times x, which can also be written as x divided by 1 plus e to the power of negative beta x, where beta is a constant or trainable parameter. It was introduced by researchers at Google in 2017, building on an earlier proposal of the same function under the name Sigmoid-weighted Linear Unit in reinforcement learning. On tests using the ImageNet dataset, swish improved performance compared to the ReLU and sigmoid activation functions, in part because it helps mitigate the vanishing gradient problem during the training of deep networks. This description is adapted from Wikipedia contributors under CC BY-SA 4.0; changes were made. https://creativecommons.org/licenses/by-sa/4.0/