Pauls Online Notes : Linear Algebra

Paul's Online Math Notes

Linear Algebra

You can navigate through this E-Book using the menu to the left. For E-Books that have a Chapter/Section organization each option in the menu to the left indicates a chapter and will open a menu showing the sections in that chapter. Alternatively, you can navigate to the next/previous section or chapter by clicking the links in the boxes at the very top and bottom of the material.

Also, depending upon the E-Book, it will be possible to download the complete E-Book, the chapter containing the current section and/or the current section. You can do this be clicking on the E-Book, Chapter, and/or the Section link provided below.

For those pages with mathematics on them you can, in most cases, enlarge the mathematics portion by clicking on the equation. Click the enlarged version to hide it.


Fundamental Subspaces	E-Book Chapter Section	Orthonormal Basis

Inner Product Spaces

If you go back to the Euclidean n-space chapter where we first introduced the concept of vectors you’ll notice that we also introduced something called a dot product. However, in this chapter, where we’re dealing with the general vector space, we have yet to introduce anything even remotely like the dot product. It is now time to do that. However, just as this chapter is about vector spaces in general, we are going to introduce a more general idea and it will turn out that a dot product will fit into this more general idea. Here is the definition of this more general idea.

Definition 1 Suppose u, v, and w are all vectors in a vector space V and c is any scalar. An inner product on the vector space V is a function that associates with each pair of vectors in V, say u and v, a real number denoted by that satisfies the following axioms.

(a)

(b)

(c)

(d) and if and only if

A vector space along with an inner product is called an inner product space.

Note that we are assuming here that the scalars are real numbers in this definition. In fact we probably should have been using the terms “real vector space” and “real inner product space” in this definition to make it clear. If we were to allow the scalars to be complex numbers (i.e. dealing with a complex vector space) the axioms would change slightly.

Also, in the rest of this section if we say that V is an inner product space we are implicitly assuming that it is a vector space and that some inner product has been defined on it. If we do not explicitly give the inner product then the exact inner product that we are using is not important. It will only be important in these cases that there has been an inner product defined on the vector space.

Example 1 The Euclidean inner product as defined in the Euclidean n-space section is an inner product.

For reference purposes here is the Euclidean inner product. Given two vectors in , and , the Euclidean inner product is defined to be,

By Theorem 2 from the Euclidean n-space section we can see that this does in fact satisfy all the axioms of the definition. Therefore, is an inner product space.

Here are some more examples of inner products.

Example 2 Suppose that and are two vectors in and that , , … , are positive real numbers (called weights) then the weighted Euclidean inner product is defined to be,

It is fairly simple to show that this is in fact an inner product. All we need to do is show that it satisfies all the axioms from Definition 1.

So, suppose that u, v, and a are all vectors in and that c is a scalar.

First note that because we know that real numbers commute with multiplication we have,

So, the first axiom is satisfied.

To show the second axiom is satisfied we just need to run through the definition as follows,

and the second axiom is satisfied.

Here’s the work for the third axiom.

Finally, for the fourth axiom there are two things we need to check. Here’s the first,

Note that this is greater than or equal to zero because the weights , , … , are positive numbers. If we hadn’t made that assumption there would be no way to guarantee that this would be positive.

Now suppose that . Because each of the terms above is greater than or equal to zero the only way this can be zero is if each of the terms is zero itself. Again, however, the weights are positive numbers and so this means that

We therefore must have if .

Likewise if then by plugging in we can see that we must also have and so the fourth axiom is also satisfied.

Example 3 Suppose that and are two matrices in . An inner product on can be defined as,

where is the trace of the matrix C.

We will leave it to you to verify that this is in fact an inner product. This is not difficult once you show (you can do a direct computation to show this) that

This formula is very similar to the Euclidean inner product formula and so showing that this is an inner product will be almost identical to showing that the Euclidean inner product is an inner product. There are differences, but for the most part it is pretty much the same.

The next two examples require that you’ve had Calculus and so if you haven’t had Calculus you can skip these examples. Both of these however are very important inner products in some areas of mathematics, although we’re not going to be looking at them much here because of the Calculus requirement.

Example 4 Suppose that and are two continuous functions on the interval . In other words, they are in the vector space . An inner product on can be defined as,

Provided you remember your Calculus, showing this is an inner product is fairly simple. Suppose that f, g, and h are continuous functions in and that c is any scalar.

Here is the work showing the first axiom is satisfied.

The second axiom is just as simple,

Here’s the third axiom.

Finally, the fourth axiom. This is the only one that really requires something that you may not remember from a Calculus class. The previous examples all used properties of integrals that you should remember.

First, we’ll start with the following,

Now, recall that if you integrate a continuous function that is greater than or equal to zero then the integral must also be greater than or equal to zero. Hence,

Next, if then clearly we’ll have . Likewise, if we have then we must also have .

Example 5 Suppose that and are two vectors in and further suppose that is a continuous function called a weight. A weighted inner product on can be defined as,

We’ll leave it to you to verify that this is an inner product. It should be fairly simple if you’ve had calculus and you followed the verification of the weighted Euclidean inner product. The key is again the fact that the weight is a strictly positive function on the interval .

Okay, once we have an inner product defined on a vector space we can define both a norm and distance for the inner product space as follows.

Definition 2 Suppose that V is an inner product space. The norm or length of a vector u in V is defined to be,

Definition 3 Suppose that V is an inner product space and that u and v are two vectors in V. The distance between u and v, denoted by is defined to be,

We’re not going to be working many examples with actual numbers in them in this section, but we should work one or two so at this point let’s pause and work an example. Note that part (c) in the example below requires Calculus. If you haven’t had Calculus you should skip that part.

Example 6 For each of the following compute , and for the given pair of vectors and inner product.

(a) and in with the standard Euclidean inner product. [Solution]

(b) and in with the weighed Euclidean inner product using the weights , and . [Solution]

(c) and in using the inner product defined in Example 4. [Solution]

Solution

(a) and in with the standard Euclidean inner product.

There really isn’t much to do here other than go through the formulas.

[Return to Problems]

(b) and in with the weighed Euclidean inner product using the weights , and .

Again, not a lot to do other than formula work. Note however, that even though we’ve got the same vectors as part (a) we should expect to get different results because we are now working with a weighted inner product.

So, we did get different answers here. Note that in under this weighted norm u is “smaller” in some way than under the standard Euclidean norm and the distance between u and v is “larger” in some way than under the standard Euclidean norm.

[Return to Problems]

(c) and in using the inner product defined in Example 4.

Okay, again if you haven’t had Calculus this part won’t make much sense and you should skip it. If you have had Calculus this should be a fairly simple example.

[Return to Problems]

Now, we also have all the same properties for the inner product, norm and distance that we had for the dot product back in the Euclidean n-space section. We’ll list them all here for reference purposes and so you can see them with the updated inner product notation. The proofs for these theorems are practically identical to their dot product counterparts and so aren’t shown here.

Theorem 1 Suppose u, v, and w are vectors in an inner product space and c is any scalar. Then,

(a)

(b)

(c)

(d)

(e)

Theorem 2 Cauchy-Schwarz Inequality : Suppose u and v are two vectors in an inner product space then

Theorem 3 Suppose u and v are two vectors in an inner product space and that c is a scalar then,

(a)

(b) if and only if u=0.

(c)

(d) - Usually called the Triangle Inequality

Theorem 4 Suppose u, v, and w are vectors in an inner product space then,

(a)

(b) if and only if u=v.

(c)

(d) - Usually called the Triangle Inequality

There was also an important concept that we saw back in the Euclidean n-space section that we’ll need in the next section. Here is the definition for this concept in terms of inner product spaces.

Definition 4 Suppose that u and v are two vectors in an inner product space. They are said to be orthogonal if .

Note that whether or not two vectors are orthogonal will depend greatly on the inner product that we’re using. Two vectors may be orthogonal with respect to one inner product defined on a vector space, but not orthogonal with respect to a second inner product defined on the same vector space.

Example 7 The two vectors and in are not orthogonal with respect to the standard Euclidean inner product, but are orthogonal with respect to the weighted Euclidean inner product with weights , and .

We saw the computations for these back in Example 6.

Now that we have the definition of orthogonality out of the way we can give the general version of the Pythagorean Theorem of an inner product space.

Theorem 5 Suppose that u and v are two orthogonal vectors in an inner product space then,

There is one final topic that we want to briefly touch on in this section. In previous sections we spent quite a bit of time talking about subspaces of a vector space. There are also subspaces that will only arise if we are working with an inner product space. The following definition gives one such subspace.

Definition 5 Suppose that W is a subspace of an inner product space V. We say that a vector u from V is orthogonal to W if it is orthogonal to every vector in W. The set of all vectors that are orthogonal to W is called the orthogonal complement of W and is denoted by .

We say that W and are orthogonal complements.

We’re not going to be doing much with the orthogonal complement in these notes, although they will show up on occasion. We just wanted to acknowledge that there are subspaces that are only going to be found in inner product spaces. Here are a couple of nice theorems pertaining to orthogonal complements.

Theorem 6 Suppose W is a subspace of an inner product space V. Then,

(a) is a subspace of V.

(b) Only the zero vector, 0, is common to both W and .

(c) . Or in other words, the orthogonal complement of is W.

Here is a nice theorem that relates some of the fundamental subspaces that we were discussing in the previous section.

Theorem 7 If A is an matrix then,

(a) The null space of A and the row space of A are orthogonal complements in with respect to the standard Euclidean inner product.

(b) The null space of and the column space of A are orthogonal complements in with respect to the standard Euclidean inner product.

Fundamental Subspaces

Orthonormal Basis

[Contact Me] [Links] [Privacy Statement] [Site Map] [Terms of Use] [Menus by Milonic]