GoKawiilSurvey of fast tanh approximations using Taylor series, Padé approximants, splines, and bitwise manipulation techniques like K-TanH and Schraudolph
The hyperbolic tangent function, \( tanh \), maps any real number to the range (-1, 1) with a smooth S-shaped curve. This property is useful as an activation function in neural networks, where it introduces non-linearity while keeping outputs bounded, and in audio signal processing, where it provides natural-sounding soft clipping for saturation and distortion effects.
In both contexts, speed matters. Neural network inference may evaluate \( tanh \) millions of times per forward pass, and audio processing demands real-time performance at sample rates of 44.1 kHz or higher. The accuracy provided by standard library implementations requires more computation than a tailored approximation.
This post surveys several approaches for approximating \( tanh \): traditional polynomial methods like Taylor series, Padé approximants, and splines, as well as more exotic techniques that exploit the IEEE-754 floating-point representation to achieve impressive speed without too much work.
§ Survey of Approximations
First, let's take a tour of some of the more common approaches to implementing fast approximations of functions like \( tanh \):
§ Taylor series
As you might recall from a distant Calculus class, a Taylor series is a polynomial expansion of a function that uses successive derivatives to generate a (infinite) summed sequence of fractions with \( x \) to successive degrees. By taking the first few terms of the sequence, we can find a rough approximation with relatively few operations.
pub fn tanhf ( x : f32 ) -> f32 { // Snap to 1 when polynomial begins to deviate at the tails if x. abs () > 1.365 { return 1 f32 . copysign (x); } let t1 = x; let t2 = x. powi ( 3 ) * ( 1. / 3. ); let t3 = x. powi ( 5 ) * ( 2. / 15. ); let t4 = x. powi ( 7 ) * ( 17. / 315. ); let t5 = x. powi ( 9 ) * ( 62. / 2835. ); let t6 = x. powi ( 11 ) * ( 1382. / 155925. ); t1 - t2 + t3 - t4 + t5 - t6 }
§ Padé Approximate
... continue reading