\(\DeclareMathOperator{\ad}{ad} \DeclareMathOperator{\dim}{dim} \DeclareMathOperator{\span}{span} \newcommand{glV}{\mathfrak{gl}(V)} \newcommand{gl}[1]{\mathfrak{gl}(#1)}\) In what follows, \(V\) is a finite-dimensional \(k\)-Vector space, \(x,y\) are elements of a respective Lie algebra, and \(v, w\) elements of a vector space.
We want to show the following:
Let \(L\) be \(\ad\)-nilpotent. If the lemma were to hold, then there were a \(d\) such that \([L,L_{d-1}] = 0\). In other words, setting \(A := L_{d-1}\), we have that \(\ad_x(a) = [xa] =0\) for all \(a\in A\). When viewed differently, this means that the induced lie algebra \(S := \{\ad_x\mid x\in L\} \leq \gl{L}\) (the \(\ad\)-representation) consists of nilpotent endomorphisms who annihilate \(A\).
In other words, the following lemma holds at least for such subalgebras of \(\glV\) which arise as \(\ad\)-representations:
As it turns out, this happens to be true in general – and this slightly stronger statement is enough to recover Engel's Theorem.
It took me some time to figure out what insights I was missing to fully understand the proof.
First, a construction similar to the \(\ad\)-representation:
Let \(K\unlhd L\), and \(M\) any subspace of \(L\) which normalizes \(K\). Then there is a representation \(\alpha\) of \(M\) on \(L/K\) by means of a projected \(\ad\)-representation: For an \(x\in M\), set
\begin{align*}\alpha_x(y+K) := \ad_x(y) + K = \mathrm{pr}_K \circ \ad_x.\end{align*}One simply verifies that well-definedness stems from the fact that \(\ad_m(K)\subseteq K\) for every \(m\in M\). In particular, this works for \(M=L\) if \(K\) is an ideal, or when we set \(M=K\).
Note that when \(\ad_x\) is nilpotent, so is \(\alpha_x\).
On the other hand, we have the following criterion for being an ideal:
It is easy to show using linearity and the jacobi identity that the normalizer
\begin{align*}N_L(K) := \{x\in L \mid [xK]\subseteq K\}\end{align*}is a subalgebra. Note that we have \(A\in N_L(A)\) for any subalgebra \(A\leq L\), and that the condition is equivalent to \(l\) being in the normalizer as well. Therefore, we must have
\begin{align*}\langle K, l \rangle \leq \langle N_L(K) \rangle = N_L(K)\leq L,\end{align*}where the angles denote the generated subalgebra. However, \(l\not\in K\) ensures that \(K\subsetneq \langle K, l \rangle\), which by maximality pins the latter to be \(L\).
To show codimension one, note that \(l\) normalizing \(K\) causes the linear sum \(K + k\cdot l\) to be closed under commutator, i.e. a subalgebra. But by maximality of \(K\), this can only be all of \(L\).
We are now ready for a proof.
(of the torsion lemma)
Induction over \(\dim L\). Let \(K\leq L\) be a maximal nontrivial subalgebra. As per the earlier construction, we can represent \(K\curvearrowright L/K\) by means of \(\alpha_k(l + K) := \ad_k(l) + K\). Because the \(\alpha_m\) are still nilpotent and \(\dim L/K < \dim L\), by induction there is a nonzero \(l+K\in L/K\) which is annihilated by all \(\alpha_k\). This means that for all \(k\in K\), \(0+K = \alpha_k(l) + K = [kl] + K\), ergo \([kl] \in K\). In other words, \(l\) normalizes \(K\) but is itself not contained in the latter.
But applying the previous lemma, we immediately know that \(K\) must be an ideal which is (linearly) complemented only by \(l\)!
By another induction argument, \(K\) itself must annihilate nonzero vectors, so the set \(W \leq V\) of vectors annihilated by all elements of \(K\) is nonempty. It is now easy to show that \(lW\leq W\), so we are allowed to restrict \(l\) to \(W\).
However, the restriction of a nilpotent endomorphism is still nilpotent, and in particular has nonempty kernel. So we can choose a \(w\in W\) such that \(0 = l\mid_W(w) =l(w)\), which consequently is annihilated by all of \(K + k\cdot l = L\).
If you re-read the proof, you might notice a funny situation: Since we later showed that \(K\unlhd L\), the representation \(K\curvearrowright L/K\) turns out to be trivial, since for every \(x\in K\), \(\ad_x(l)\) is always contained in \(K\)! Therefore, every \(\alpha_x\) maps only to \(K\), the zero element.
TODO show that there must be a flag of invariant subspaces – transported by all elements of \(L\) – eventually annihilated. Simple induction argument, since „common kernel“ is just a fancy kind of invariant subspace you can factor out.
TODO show that such a common flag is enough to prove nilpotency.