Theory of the Taylor Expansion

Suppose that the function f(x) has continuous derivatives of orders 1, 2, ..., n+1. From the definition of the integral as the anti-derivative of a function, we have:

$$f(x) - f(a) = \int_a^x f^{\prime}(t) dt$$

where $f^{\prime}(t)$ is the derivative of f(t) with respect to its argument, t in this case. The lower limit of the integral, where t = a, and f(x) takes the value f(a), is arbitrary as long as the function f(x) exists there and has the derivatives already specified.

The rule for the differentiation of the product of two functions, say u and v, is:

$$(u v)^{\prime} = u^{\prime} v + u v^{\prime}$$

which we can slightly re-arrange and integrate to obtain:

$$\int u dv = u v - \int v du.$$

This leads to the technique of integration by parts. The choice of u and v can often lead to a simpler integral to be evaluated.

In the case at hand, we can make the choices:

$$u = f^{\prime}(t),\ \ {\rm and}\ \ dv = dt.$$

Upon differentiating u with respect to t and integrating dv with respect to t, we have:

$$du = f^{\prime\prime}(t) dt,\ \ {\rm and}\ \ v = t - x,$$

where in the result v = t - x, we have introduced a constant of integration, x, which is indeed a constant when we integrate with respect to t, as we are doing here. Putting these results into the integration by parts formula, we obtain:

$$\int_a^x f^{\prime}(t) dt = \left [ (t - x) f^{\prime}(t) \... ...right ]_{t=a}^{t=x} + \int_a^x f^{\prime\prime}(t) (x - t) dt$$

Noting that the left hand side is just f(x) - f(a), and that the first term on the right hand side vanishes at the upper limit, t = x, and is simply $-(x - a)f^\prime(a)$ at the lower limit, we have the result:

$$f(x) = f(a) + f^{\prime}(a) (x-a) + \int_a^x f^{\prime\prime}(t) (x - t) dt.$$

Thus we have succeeded in writing the function f(x) in terms of its value f(a) at the point x = a, plus a linear term in (x-a) with coefficient $f^\prime (a)$, the first derivative at the same point x = a, with a remainder term which involves the second derivative of f(t) on the interval t = a to t = x.

The approach used to obtain the first level result above can now be applied again, but concentrating on the remaining integral, involving the second derivative $f^{\prime\prime}(t)$. We can use the same integration by parts technique again, but now choosing:

$$u = f^{\prime\prime}(t),\ \ {\rm and}\ \ dv = (x - t) dt$$

so that differentiating u with respect to t, and integrating dv with respect to t yields:

$$du = f^{(3)}(t) dt,\ \ {\rm and}\ \ v = -\frac{1}{2} (t - x)^2,$$

where the notation f⁽ⁿ⁾(x) indicates the n^th derivative of f(x) with respect to its argument. Note that the constant of integration for v has been taken to be zero at this level, as it will be taken at higher levels. The expression for f(x) becomes:

$$f(x) = f(a) + f^{\prime}(a) (x-a) + \frac{f^{\prime\prime}(a)}{2} (x-a)^2 + \frac{1}{2} \int_a^x f^{(3)}(t) (x - t)^2 dt$$

By repeating this integration by parts process on the remaining integral p times, one has the result:

$$f(x) = f(a) + f^{\prime}(a) (x-a) + \frac{f^{\prime\prime}(a... ...!} (x-a)^n + \frac{1}{n!} \int_a^x f^{(n+1)}(t) (x - t)^n dt,$$

where $n! = n \; (n-1) \; (n-2) \; \ldots \; 3 \; 2 \; 1$ is the factorial of n. This result is valid when both x and a are in an interval over which f(x) and its first n+1 derivatives are continuous, as defined in calculus courses.

The expression for f(x) given here is called Taylor's formula with integral remainder, derived by Brook Taylor (1685-1731). The last term is referred to as the remainder, R_n(x), since it contains the difference between the function f(x) and the representation of f(x) offered by the first n+1 terms of the Taylor formula. When the remainder reaches a limiting value of 0 as $n \rightarrow \infty$, we can represent f(x) by the infinite Taylor series:

$$f(x) = f(a) + f^{\prime}(a) (x-a) + \frac{f^{\prime\prime}(a... ...} (x-a)^2 + \cdots + \frac{f^{(n)}(a)}{n!} (x-a)^n + \cdots,$$

which we can also write as the infinite sum:

$$f(x) = \sum^\infty_{n=0} \frac{(x - a)^n}{n!} f^{(n)}(a),$$

where it is important to note that n! is taken to have the value 1 when n = 0. The remainder R_n(x) gives us the error we make in approximating f(x) when we truncate the series at the term in (x-a)ⁿ.

The remainder can be written in a number of ways, some of which are more usable than the integral form given above. For our purposes, the most useful form is that derived by J. L. Lagrange (1736-1813), resulting in the Taylor formula with derivative remainder:

$$f(x) = f(a) + f^{\prime}(a) (x-a) + \frac{f^{\prime\prime}(a... ...)}{n!} (x-a)^n + \frac{f^{(n+1)}(\xi)}{(n+1)!} (x - a)^{n+1},$$

where $\xi$ is some number between x and a. Thus the explicit value for the remainder in the integral form, which requires knowing the function f(x) well enough to be able to evaluate the integral, has been replaced by an form dependent only on the (n+1)^th derivative of f(x), but at some point, $\xi$, unknown to us, except that it lies between the values x and a.

When the point, a, about which we form the Taylor expansion is a = 0, the expansion is called the Maclaurin expansion. As a first example, we consider the Maclaurin expansion for the function $\sin x$. Using the familiar results that:

$$\frac{d}{dx} \sin x = \cos x,\ \ \ {\rm and}\ \ \ \frac{d}{dx} \cos x = -\sin x,$$

we have f(0) = 0, $f^{\prime}(0) = 1$, $f^{\prime\prime}(0) = 0$, f⁽³⁾(0) = -1, f⁽⁴⁾(0) = 0, f⁽⁵⁾(0) = 1, continuing in the sequence {0,1,0,-1,0,1,0,-1,...}. The Taylor formula for $\sin x$ about 0 is thus:

$$\sin x = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \cdots + \frac{x^n}{n!} + \cdots + R_n(x).$$

While it easy to see how to write any higher order terms, we still have to demonstrate that R_n(x) approaches 0 as $n \rightarrow \infty$. The derivative form of the remainder makes this quite easy, since we know that the (n+1)^th derivative is either $\sin x$ or $\cos x$, depending on whether n is odd or even. Thus f⁽ⁿ⁺¹⁾(x) always lies between -1 and +1, so that we can say that |f⁽ⁿ⁺¹⁾(x)|, the absolute value of f⁽ⁿ⁺¹⁾(x), always lies in the interval [0,1]. Thus the remainder satisfies:

$$\vert R_n \vert \leq \frac{ \vert x \vert ^{n+1} }{(n+1)!}.$$

Since the numerator of the right hand side is finite for finite x, and the denominator approaches infnty as $n \rightarrow \infty$, the remainder $\vert R_n\vert \rightarrow 0$ as $n \rightarrow \infty$. Thus we can conclude that the Taylor series for $\sin x$ about x = 0 will converge.

The Taylor series has provided us with a way of evaluating a function over an interval, but using only information about that function and its derivatives at one point. In addition, it allows us to write the function as a polynomial plus a remainder term, and when the Taylor series converges, we can always make the remainder term as small as we like, simply by taking a sufficient number of terms in the polynomial expression. This is particularly useful because a lot is known about polynomial functions, and leads to the basis for essentially all the numerical techniques we will use in numerical scientific computation. A related technique is to represent a function f as a series in the basic trigonometric functions, $\sin$ and $\cos$. Such a series is called a Fourier series, and is particularly useful in the analysis of periodic systems, such as vibrating strings, etc.

Even though we started off assuming that f is known only at a, Taylor series are actually very useful even if the function f is a known function. For example, we have found that with $f(x) = \sin(x)$ and a = 0:

$$\sin(x) \approx x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \ldots.$$

This approximation holds for all values of x, as long as enough terms in the series are used. (Calculators use this series to evaluate $\sin(x)$.) A similar approach yields $\cos(x), \tan(x), e^x, \log(x),$ etc. Besides giving a numerical value to the sine function at any desired point, this approximation shows that for small x, $\sin(x) \approx x$. This is because, the higher order terms all have powers of x greater than 1, which means that the numerator is already much smaller than x, and then they are divided by larger and larger denominators. So the higher order terms are smaller than the first term. The bottom line is: $\sin(x)$ looks like y=x for $x\ll 1$. There are many applications of this approximation. One is the function $\sin(x)/x$. At x=0, this function is singular (0/0). But the Taylor series for $\sin$ shows that this function is approximately 1 at x=0, as we can by dividing the Taylor series for $\sin x$ by x to yield:

$$\frac{\sin(x)}{x} \approx 1 - \frac{x^2}{3!} + \frac{x^4}{5!} - \frac{x^6}{7!} + \ldots.$$

Taking the limit as $x \rightarrow 0$, it is clear that the limit of the series is simply 1.

Another familiar example of the use of the $\sin$ Taylor series is in the consideration of the simple pendulum. In this case the restoring force arising from the resolution of the vertical force of gravity and the tension in the string is proportional to the $\sin$ of the angular displacement of the string from the vertical line. In this form the equation for the pendulum is not particularly simple to solve, but if one is limited to small displacements of the pendulum, then the $\sin$ of the angle can be replaced by the Taylor series, and if the displacement is very small, say limited to a few degrees, then the $\sin$ function needs only be replaced by the leading term of the $\sin$ series. This leads to a linear restoring force, the hallmark of a simple harmonic oscillator, a very useful approximation indeed, since it arises in so many physical systems.

It is frequently useful to write the Taylor series in the slightly different form:

$$f(a+h) = f(a) + h f^\prime(a) + \frac{h^2}{2!} f^{\prime\prime}(a) + \frac{h^3}{3!}f^{\prime\prime\prime}(a) + {\cal O}(h^4).$$

In writing this we have used the notation h = x - a, so that x no longer appears and the variable of interest is h, which measures the distance of the point of evaluation, x = a + h, from the point where the function and its derivatives are to be evaluated, ie. at x = a. We have also introduced the notion of the order of the Taylor series, indicating where the series has been truncated, indicated by the notation ${\cal O}(h^n)$. This means that all terms of order n or higher have been ignored, and are to be interpreted as being represented by the symbol ${\cal O}(h^n)$. In the example given above, one should consider only terms of order h³ or lower to be of significance when using the series in a calculation.

Taylor series are most useful when it is possible to stop the infinite sum at some point and use only the first few terms of the series. Since h is usually small (h<1), the higher order terms will all contain higher powers of h which should mean that they will be smaller than the first few terms. So truncating the series should lead to an approximation which is not too bad. It is frequently important to make it clear as to what order a series has been evaluated before it was truncated, so the ${\cal O}$ notation is very useful.