Geometric Schur duality of classical type, II

Abstract

We establish algebraically and geometrically a duality between the Iwahori-Hecke algebra of type $\mathbf{D}$ and two new quantum algebras arising from the geometry of $N$-step isotropic flag varieties of type $\mathbf{D}$. This duality is a type $\mathbf{D}$ counterpart of the Schur-Jimbo duality of type $\mathbf{A}$ and the Schur-like duality of type $\mathbf{B}/\mathbf{C}$ discovered by Bao-Wang. The new algebras play a role in the type $\mathbf{D}$ duality similar to the modified quantum $\mathfrak{gl}(N)$ in type $\mathbf{A}$, and the modified coideal subalgebras of quantum $\mathfrak{gl}(N)$ in type $\mathbf{B}/\mathbf{C}$. We construct canonical bases for these two algebras.

1. Introduction

Let $G$ be a classical linear algebraic group over an algebraically closed field. One of the milestones in geometric representation theory is the geometric realization of the associated Iwahori-Hecke algebra of $G$, by using the bounded derived category of $G$-equivariant constructible sheaves on the product variety of two copies of the complete flag variety of $G$. Via this realization, many problems related to the Iwahori-Hecke algebra of $G$ are solved. For example, the positivity conjecture for the structure constants of the Kazhdan-Lusztig bases (KL79) are settled by interpreting the basis elements as the (shifted) intersection cohomology complexes attached to $G$-orbit closures in the product variety.

One may wonder if the geometric approach can be adapted to study other objects in representation theory, besides Iwahori-Hecke algebras. Indeed, a modification by replacing the adjective ‘complete’ in the construction by ‘partial’ already yields highly nontrivial results, as we explain in the following.

If $G$ is of type $\mathbf{A}$, i.e., $G=\mathrm{GL}(d)$, and the complete flag variety is replaced by the $N$-step partial flag variety of $\mathrm{GL}(d)$ with $N$ bearing no relation to $d$, then an analogous construction provides a geometric realization of the $v$-Schur quotient of the quantum $\mathfrak{ gl}(N)$ in the classic work BLM90. Moreover, the quantum $\mathfrak{gl}(N)$ can then be realized in the projective limit of the $v$-Schur quotients (as $d$ goes to infinity). Remarkably, an idempotented version of the quantum $\mathfrak{gl}(N)$ is discovered inside the projective limit as well, admitting a canonical basis. The role of the canonical basis for the modified quantum $\mathfrak{gl}(N)$ is similar to that of Kazhdan-Lusztig bases for Iwahori-Hecke algebras. Subsequently, the Schur-Jimbo duality, as a bridge connecting the Iwahori-Hecke algebra of $\mathrm{GL}(d)$ and the (modified) quantum $\mathfrak{gl}(N)$, is realized geometrically by considering the product variety of the complete flag variety and the $N$-step partial flag variety of $\mathrm{GL}(d)$ in GL92. The modified quantum $\mathfrak{sl}(N)$ (a variant of the quantum $\mathfrak{gl}(N)$) and its canonical basis are further categorified in the works La10 and KhLa10, which play a fundamental role in higher representation theory and the categorification of knot invariants.

If $G$ is of type $\mathbf{B} /\mathbf{C}$, i.e., $G=\mathrm{SO}(2d+1)/\mathrm{SP}(2d)$, and the variety involved is replaced by the $N$-step isotropic flag variety of $\mathrm{SO}(2d+1)/\mathrm{SP}(2d)$, then one gets a geometric realization of the modified forms of two coideal subalgebras $\mathbf{U}^{\imath }$ and $\mathbf{U}^{\jmath }$ of the quantum $\mathfrak{gl}(N)$ in BKLW14 by mimicking the approach in BLM90. Moreover, the canonical bases of these modified coideal subalgebras are constructed and studied for the first time. Along the way, a duality of Bao-Wang in BW13 relating the (modified) coideal subalgebras and the Iwahori-Hecke algebra of type $\mathbf{B}/ \mathbf{C}$ associated to $\mathrm{SO}(2d+1)/\mathrm{SP}(2d)$ is also geometrically realized in a similar manner as the type-$\mathbf{A}$ case. (See also G97 for a duality closely related to the duality of Bao-Wang.) The canonical basis theory for these coideal subalgebras is initiated in the seminal work BW13, and is used substantially to give simultaneously a new formulation of the Kazhdan-Lusztig conjecture of type $\mathbf{B}/\mathbf{C}$ on the irreducible character problem and the resolution of the analogous problem for the ortho-symplectic Lie superalgebras.

To this end, it is compelling to ask what happens to the remaining classical case: $G=\mathrm{SO}(2d)$ of type $\mathbf{D}$. The purpose of this paper is to provide an answer to this question, as a sequel to BKLW14. More precisely, we obtain two quantum algebras $\mathcal{K}$ and $\mathcal{K}^m$ via the geometry of the $N$-step isotropic flag variety of type $\mathbf{D}$ and a stabilization process following BLM90 and BKLW14. We show that both algebras possess three distinguished bases, i.e., the standard, monomial and canonical bases, similar to the results in type $\mathbf{ABC}$. We further establish new dualities between these two algebras and the Iwahori-Hecke algebra of type $\mathbf{D}$ attached to $\mathrm{SO}(2d)$ algebraically and geometrically.

Unlike type $\mathbf{ABC}$, the algebras $\mathcal{K}$ and $\mathcal{K}^m$ are not modified forms of some known quantum algebras in literature, even though they resemble the modified forms $\dot{\mathbf{U}}^{\imath }$, $\dot{\mathbf{U}}^{\jmath }$ of coideal subalgebras of quantum $\mathfrak{gl}(N)$. It is natural to ask for a presentation of the two algebras by generators and relations. We have a complete answer for the algebra $\mathcal{K}^m$, and partial results for $\mathcal{K}$. We show that the algebra $\mathcal{K}^m$ admits defining relations similar to those of $\dot{\mathbf{U}}^{\imath }$, but with the size of the set of idempotent generators doubled, after extending the underlying ring to the field of rational functions. Despite all the similarities, we caution the reader that $\dot{\mathbf{U}}^{\imath }$ is not a subalgebra of $\mathcal{K}^m$. The presentation for $\mathcal{K}^m$ is obtained by showing that (the complexification of) $\mathcal{K}^m$ is isomorphic to the modified form of a new unital associative algebra $\mathbf{U}^m$ containing the coideal subalgebra $\mathbf{U}^{\imath }$ and two additional idempotents. The appearance of the new idempotents reflects the geometric fact that there are two connected components for maximal isotropic Grassmannians in the type $\mathbf{D}$ geometry. As for the bigger algebra $\mathcal{K}$, we formulate another new unital associative algebra $\mathbf{U}$ containing the coideal subalgebra $\mathbf{U}^{\jmath }$ and three extra idempotents, and expect its modified form to be isomorphic to $\mathcal{K}$ after a suitable field extension. As evidence in support of this expectation, we show that $\mathbf{U}$ and the Iwahori-Hecke algebra of type $\mathbf{D}$ satisfy a double centralizer property. Notice that the commuting actions between $\mathbf{U}^{\imath }$, $\mathbf{U}^{\jmath }$ and the Iwahori-Hecke algebra of type $\mathbf{D}$ are first observed in ES13b, 7.8 (see also ES13a), so this result can be thought of as an enhancement of those in loc. cit.

As an application, we expect that the type-$\mathbf{D}$ duality and the canonical basis theory for the new algebras $\mathcal{K}$ and $\mathcal{K}^m$ developed in this paper will shed light on the type-$\mathbf{D}$ problems similar to those addressed in BW13, currently under investigation by H. Bao.

Since our results are governed in principle by the (parabolic) Kazhdan-Lusztig polynomials of type $\mathbf{D}$, they are obviously different from those in type $\mathbf{ABC}$ in BLM90 and BKLW14. Furthermore, the geometry of type $\mathbf{D}$ is more challenging to handle. In particular, there are mainly three new technical barriers in our type $\mathbf{D}$ setting that we overcome. The first one is that there are two connected components for the maximal isotropic Grassmannian associated to $\mathrm{SO}(2d)$. This forces us to parametrize the $\mathrm{SO}(2d)$-orbits by using $signed$ $matrices$ instead of matrices in type $\mathbf{ABC}$. The second one is that the number of isotropic lines in a given quadratic space of even dimension over a finite field depends on its isometric class; we choose to work in the case when the group is split, which has a remarkable hereditary property (see Lemma 3.1.1). The last one is that during the stabilization process, one cannot subtract/add by an (even) multiple of the identity matrix as in loc. cit., because the signs of the matrices may change. To circumvent this difficulty, we subtract/add an even multiple of a matrix obtained by changing the middle entry to be zero in the diagonal of the identity matrix. All these factors make the computations and arguments more involved than those in previous cases.

As this paper provides a complete picture for the cases of the classical groups, the problem of whether a similar picture exists for exceptional groups is still wide open. Meanwhile, for $G$ replaced by a loop group of type $\mathbf{A}$, there exists a similar geometric theory involving affine Iwahori-Hecke algebras of type $\mathbf{A}$ and affine quantum $\mathfrak{gl}(N)$ in Lu99, GV93, SV00 and M10. The investigation for $G$ being a loop group of type $\mathbf{BCD}$ will be presented in a separate article.

2. Schur dualities of type $\mathbf{D}_d$

In this section, we shall introduce the algebras $\mathbf{U}$ and $\mathbf{U}^m$, and formulate algebraically the dualities between these two algebras and the Iwahori-Hecke algebras of type $\mathbf{D}_{d}$.

2.1. The algebra $\mathbf{U}$ and the first duality

Let $[a, b]$ denote the set of integers between $a$ and $b$. Let $\mathbf{U}$ be the unital associative $\mathbb{Q}(v)$-algebra generated by the symbols

$$\begin{equation*} E_i, F_i, H_a^{\pm 1}, \ \text{and} \ J_{\alpha }, \quad \forall i\in [1,n], a\in [1, n+1], \alpha \in \{ +,0, -\}, \end{equation*}$$

satisfying the following relations:

$$\begin{align*} J_++J_0+J_- &= 1,\quad J_{\alpha }J_{\beta }=\delta _{\alpha ,\beta }J_{\alpha }, \\ J_{\alpha }H_a &= H_aJ_{\alpha },\\ J_{\pm }E_i &= (1-\delta _{in})E_iJ_{\pm },\\ F_iJ_{\pm } &= (1-\delta _{in})J_{\pm }F_i,\\ J_{\pm }F_nE_n-F_nE_nJ_{\mp } &=\frac{H_n^{-1}H_{n+1}-H_nH_{n+1}^{-1}}{v-v^{-1}}(J_{\pm }-J_{\mp }),\\ H_aH_b &=H_b H_a,\quad H_aH_a^{-1}=1,\\ H_aE_i &=v^{ \delta _{a i} -\delta _{a,i+1} - \delta _{a,n+1} \delta _{i,n}}E_iH_a, \\ H_aF_i &=v^{- \delta _{ai} + \delta _{a,i+1} + \delta _{a,n+1}\delta _{i,n}}F_iH_a,\\ E_iF_j-F_jE_i &=\delta _{ij}\frac{H_iH_{i+1}^{-1}-H_i^{-1}H_{i+1}}{v-v^{-1}}, && \text{if $(i, j)\neq (n,n)$},\\ E_iE_j &=E_jE_i,\quad F_iF_j=F_jF_i, && \text{if $|i-j|>1$},\\ E_i^2E_j-(v+v^{-1})E_iE_jE_i+E_jE_i^2 &=0, && \text{if $|i-j|=1$},\\ F_i^2F_j-(v+v^{-1})F_iF_jF_i+F_jF_i^2 &=0, && \text{if $|i-j|=1$},\\ E^2_nF_n+F_nE_n^2 &=(v+v^{-1})(E_nF_nE_n-E_n(vH_nH_{n+1}^{-1}+v^{-1}H_n^{-1}H_{n+1})),\\ F_n^2E_n+E_nF_n^2 &=(v+v^{-1})(F_nE_nF_n-(vH_nH_{n+1}^{-1}+v^{-1}H_n^{-1}H_{n+1})F_n), \end{align*}$$

where $i$, $j\in [1,n]$, $a$, $b\in [1,n+1]$, and $\alpha \in \{ +, 0, -\}$. Notice that the subalgebra $\mathbf{U}^{\jmath }$ generated by $E_i$, $F_i$, and $H_a^{\pm 1}$ for any $i\in [1, n]$, $a\in [1,n+1]$ is the coideal subalgebra in the same notation in BKLW14. See also Le02 and ES13b.

Let $\mathbf{V}$ be a vector space over $\mathbb{Q}(v)$ of dimension $2n+1$. We fix a basis $( \mathbf{v}_i)_{ 1\leq i\leq 2n+1}$ for $\mathbf{V}$. Let $\mathbf{V}^{\otimes d}$ be the $d$-th tensor space of $\mathbf{V}$. Thus we have a basis $(\mathbf{v}_{r_1}\otimes \dots \otimes \mathbf{v}_{r_d})$, where $r_1$, …, $r_d\in [1, 2n+1]$, for the tensor space $\mathbf{V}^{\otimes d}$.

For a sequence $\mathbf{r}=(r_1,\dots , r_d)$, we write $\mathbf{v}_{\mathbf{r}}$ for $\mathbf{v}_{r_1}\otimes \dots \otimes \mathbf{v}_{r_d}$. For a sequence $\mathbf{r}=(r_1,\dots , r_d)$, it defines uniquely a sequence of length $2d$ of the form

$$\begin{equation*} \tilde{ \mathbf{r}}=(r_1, \dots , r_d, 2n+2-r_d, 2n+2-r_{d-1},\dots , 2n+2-r_1) \end{equation*}$$

such that $r_i+r_{2d+1-i} =2n+2$. We shall identify $\mathbf{r}$ with $\tilde{\mathbf{r}}$ in what follows.

For a sequence $\mathbf{r}$ and a fixed integer $p\in [1, 2d]$, we define the sequences $\mathbf{r}'_p$ and $\mathbf{r}''_p$ by

$$\begin{equation*} (\mathbf{r}'_p)_j= \begin{cases} r_j, & j\neq p, 2d+1-p,\\ r_p +1, & j=p,\\ r_{2d+1 -p}-1, &j=2d+1-p, \end{cases} \end{equation*}$$

and

$$\begin{equation*} (\mathbf{r}''_p)_j= \begin{cases} r_j, & j\neq p, 2d+1-p,\\ r_p -1, & j=p,\\ r_{2d+1 -p}+1, &j=2d+1-p. \end{cases} \end{equation*}$$

Lemma 2.1.1.

We have a left $\mathbf{U}$-action on $\mathbf{V}^{\otimes d}$ defined by, for any $i\in [1, n]$, $a\in [1, n+1]$,

$$\begin{align} E_i\cdot \mathbf{v}_{\mathbf{r}} &=v^{-\sum _{1\leq j\leq 2d} \delta _{i+1,r_j}} \sum _{1\leq p\leq 2d: r_p=i}v^{2\sum _{j<p}\delta _{i+1,r_j}} \ \mathbf{v}_{\mathbf{r}'_p}, \\ F_i \cdot \mathbf{v}_{\mathbf{r}} &=v^{-\sum _{1\leq j\leq 2d} \delta _{i,r_j}} \sum _{1\leq p\leq 2d:r_p=i+1}v^{2\sum _{j>p}\delta _{i, r_j}} \mathbf{v}_{\mathbf{r}''_p}, \\ H_a^{\pm 1} \cdot \mathbf{v_r} &=v^{\mp \sum _{1\leq j\leq 2d} \delta _{a, r_j} } \mathbf{v_r}, \\ J_{+}\cdot \mathbf{v}_{\mathbf{r}} &= \begin{cases} \mathbf{v}_{\mathbf{r}}, & r_i\neq n+1,\forall i, \#\{ j\in [1,d]| r_j\geq n+1\} \text{ is even},\\ 0, &\text{otherwise}, \end{cases}\\ J_{-}\cdot \mathbf{v}_{\mathbf{r}} &= \begin{cases} \mathbf{v}_{\mathbf{r}}, & r_i\neq n+1,\forall i, \#\{ j\in [1,d]| r_j\geq n+1\} \text{ is odd},\\ 0, &\text{otherwise}, \end{cases} \\ J_{0}\cdot \mathbf{v}_{\mathbf{r}} &= \begin{cases} \mathbf{v}_{\mathbf{r}}, & r_i= n+1,\ \text{for some} \ i,\\ 0, &\text{otherwise}. \end{cases} \end{align}$$

The lemma follows from 13, Proposition 4.1.1, and Corollary 4.4.1.

Recall that the Iwahori-Hecke algebra $\mathbf{H}_d$ of type $\mathbf{D}_d$ is a unital associative algebra over $\mathbb{Q}(v)$ generated by $\tau _i$ for $i\in [1, d]$ and subject to the relations

$$\begin{align*} \tau _i^2 &= (v^2-1) \tau _i + v^2 , &&1\leq i \leq d,\\ \tau _j \tau _{j+1} \tau _j &= \tau _{j+1} \tau _j \tau _{j+1}, &&1\leq j\leq d-2,\\ \tau _i \tau _j &= \tau _j \tau _i, &&1 \leq j\leq i-2\leq d-3,\\ \tau _d \tau _l &= \tau _l \tau _d, &&l\neq d-2.\\ \tau _d \tau _{d-2} \tau _d &= \tau _{d-2} \tau _d \tau _{d-2}. \end{align*}$$

Lemma 2.1.2.

We have a right $\mathbf{H}_d$-action on $\mathbf{V}^{\otimes d}$ given by, for $1\leq j\leq d-1$,

$$\begin{align} \mathbf{v}_{r_1 \dots r_{2d}} \tau _j &= \begin{cases} \mathbf{v}_{r_1\dots r_{j-1} r_{j+1} r_j\dots r_{2d-j} r_{2d-j-1} r_{2d-j+1} \dots r_{2d}} , & r_j < r_{j+1},\\ v^2 \mathbf{v}_{r_1\dots r_{2d}}, & r_j = r_{j+1},\\ (v^2-1) \mathbf{v}_{r_1\dots r_{2d}} + v^2 \mathbf{v}_{r_1 \dots r_{j-1} r_{j+1} r_j \dots r_{2d-j} r_{2d-j-1} r_{2d-j+1} \dots r_{2d}}, & r_j> r_{j+1}, \end{cases} \tag{1}\\[3.0pt] \mathbf{v}_{r_1 \dots r_{2d}} \tau _d &= \begin{cases} \mathbf{v}_{r_1\dots r_{d-2} r_{d+1} r_{d+2}r_{d-1}r_d r_{d+3} \dots r_{2d}} , & r_{d-1}+r_d < N+1,\\ v^2 \mathbf{v}_{r_1\dots r_{2d}}, & r_{d-1}+r_d =N+1,\\ (v^2-1) \mathbf{v}_{r_1\dots r_{2d}} + v^2 \mathbf{v}_{r_1\dots r_{d-2} r_{d+1} r_{d+2}r_{d-1}r_d r_{d+3} \dots r_{2d}}, &r_{d-1}+r_d > N+1. \end{cases} \tag{2} \end{align}$$

Here we identify the sequence $\mathbf{r}$ with the associated sequence $\tilde{ \mathbf{r}}$.

This lemma follows from 13 and Lemmas 3.4.1 and 3.4.2. We now can state the first duality.

Proposition 2.1.3.

The left $\mathbf{U}$-action in Lemma 2.1.1 and the right $\mathbf{H}_d$-action in Lemma 2.1.2 on $\mathbf{V}^{\otimes d}$ are commuting. They form a double centralizer for $n\geq d$, i.e.,

$$\begin{equation*} \mathbf{H}_d\simeq \operatorname {End}_{\mathbf{U}} (\mathbf{V}^{\otimes d}) \quad \text{and} \quad \mathbf{U}\to \operatorname {End}_{\mathbf{H}_d} (\mathbf{V}^{\otimes d}) \ \text{is surjective}. \end{equation*}$$

The proposition follows from the previous two lemmas, Lemma 3.2.1, Proposition 4.1.1 and Corollary 4.6.6.

2.2. The algebra $\mathbf{U}^m$ and the second duality

Let $\mathbf{U}^m$ be an associative $\mathbb{Q}(v)$-algebra with unit generated by the symbols

$$\begin{equation*} E_i, F_i, H_a^{\pm 1}, T, J_{\alpha }, \quad \forall i\in [1, n-1], a\in [1, n], \alpha \in \{ +, -\}, \end{equation*}$$

and subject to the defining relations

$$\begin{align*} J_++J_- &= 1,\quad J_{\alpha }J_{\beta }=\delta _{\alpha ,\beta }J_{\alpha }, \\ J_{\pm }H_a &= H_aJ_{\pm },\quad J_{\pm }E_i=E_iJ_{\pm },\quad F_iJ_{\pm }= J_{\pm }F_i,\\ J_{\pm }T &= TJ_{\mp },\\ H_aH_b &= H_b H_a,\quad H_a H_a^{-1}=1,\\ H_aE_i &= v^{-\delta _{a,i+1} + \delta _{a i} }E_iH_a,\quad \\ H_aF_i &= v^{- \delta _{ai} + \delta _{a,i+1} }F_iH_a,\\ H_i T &= TH_i,\\ E_iF_j-F_jE_i &= \delta _{ij}\frac{H_iH_{i+1}^{-1}-H_i^{-1}H_{i+1}}{v-v^{-1}},\\ E_iE_j &= E_jE_i,\quad F_iF_j=F_jF_i, && \text{if $|i-j|>1$},\\ E_i^2E_j-(v+v^{-1})E_iE_jE_i+E_jE_i^2 &=0, && \text{if $|i-j|=1$},\\ F_i^2F_j-(v+v^{-1})F_iF_jF_i+F_jF_i^2 &=0, && \text{if $|i-j|=1$},\\ TE_i &= E_iT,\ TF_i=F_iT, && \text{if $i\leq n-2$},\\ E_{n-1}^2T-(v+v^{-1}) E_{n-1}TE_{n-1}+TE_{n-1}^2 &= 0,\\ F_{n-1}^2T- (v+v^{-1} F_{n-1}TF_{n-1}+TF_{n-1}^2 &= 0,\\ T^2E_{n-1}-(v+v^{-1})TE_{n-1}T+E_{n-1}T^2 &= E_{n-1},\\ T^2F_{n-1}-(v+v^{-1})TF_{n-1}T+F_{n-1}T^2 &= F_{n-1}. \end{align*}$$

Note that the subalgebra $\mathbf{U}^{\imath }$ generated by $E_i$, $F_i$, $H_a^{\pm 1}$, and $T$ is the algebra in the same notation in BKLW14, 5.3. See also Le02 and ES13b.

Let $\mathbf{W}$ be the subspace of $\mathbf{V}$ spanned by the basis elements $\mathbf{v}_i$ for $i\neq n+1$. Its $d$-th tensor space $\mathbf{W}^{\otimes d}$ is naturally a subspace of $\mathbf{V}^{\otimes d}$ spanned by the vectors $\mathbf{v_r}$ such that $r_i\neq n+1$ for any $i$. Then we see that $\mathbf{W}^{\otimes d}$ is a vector space of dimension $(2n)^d$.

By Lemma 2.1.2, the $\mathbf{H}_d$-action on $\mathbf{V}^{\otimes d}$ induces an $\mathbf{H}_d$-action on $\mathbf{W}^{\otimes d}$. Moreover,

Lemma 2.2.1.

We have a $\mathbf{U}^m$-action on $\mathbf{W}^{\otimes d}$ given by the same formulae for $E_i$, $F_i$, $H_a^{\pm 1}$, and $J_{\pm }$ for $i\in [1, n-1]$ and $a\in [1, n]$ as in Lemma 2.1.1, together with

$$\begin{equation*} T\cdot \mathbf{v_r}= \left(F_nE_n + \frac{H_n H_{n+1}^{-1} - H_n^{-1} H_{n+1}}{v-v^{-1}} \right)\cdot \mathbf{v_r}. \end{equation*}$$

This lemma is proved by 75, 77, Lemma 6.1.2 and Proposition 6.2.1. We can now state the second duality.

Proposition 2.2.2.

The $\mathbf{U}^m$-action and $\mathbf{H}_d$-action on $\mathbf{W}^{\otimes d}$ are commuting. They enjoy the double centralizer property when $n\geq d$.

The proof is given by Lemma 2.2.1, Lemma 6.1.1, Proposition 6.2.1 and Proposition 6.3.2.

3. A geometric setting

We now turn to the geometric setting in order to prove the above results among others.

3.1. Preliminaries

We start by recalling some results on counting isotropic subspaces in an even-dimensional quadratic space over a finite field. We refer to W93 and the references therein for more details.

Let $\mathbb{F}_q$ be a finite field of $q$ elements and of odd characteristic. Recall that $d$ is a fixed positive integer, and we set

$$\begin{equation*} D=2d. \end{equation*}$$

On the $D$-dimensional vector space $\mathbb{F}_q^D$, we fix a nondegenerate symmetric bilinear form $Q$ whose associated matrix is

$$\begin{align} \begin{bmatrix} 0 &I_d\\ I_d & 0 \end{bmatrix} {\tag{3}} \end{align}$$

under the standard basis of $\mathbb{F}_q^D$. By convention, $W^{\perp }$ stands for the orthogonal complement of a vector subspace $W$ in $\mathbb{F}_q^D$ with respect to the bilinear form $Q$. Moreover, we call $W$ isotropic if $W\subseteq W^{\perp }$. We write $|W|$ for the dimension of $W$.

For any isotropic subspace $W$, the bilinear form $Q$ induces a nondegenerate symmetric bilinear form $Q|_{W^{\perp }/W}$ on $W^{\perp }/W$. One of the reasons that we fix $Q$ of the form 3 is its hereditary property in the following lemma, which can be proved inductively.

Lemma 3.1.1.

The associated matrix of $Q|_{W^{\perp }/W}$ is of the form 3 with rank $d-2|W|$ under a certain basis.

Note that this hereditary property does not always hold for an arbitrary nondegenerate symmetric bilinear form. By using Lemma 3.1.1, we can count the number of isotropic lines inductively to get the following lemma.

Lemma 3.1.2.

The cardinality, $S_d$, of the set of isotropic lines in $\mathbb{F}_q^D$ is $\frac{q^{2d-1}-1}{q-1}+q^{d-1}$.

We will need the following lemma later. We write $W\overset{a}{\subset } V$ if $W\subseteq V$ and $|V/W|=a$.

Lemma 3.1.3.

Let $V=(V_i)_{1\leq i\leq 5}$ be a fixed flag of $\mathbb{F}_q^D$ such that $V_{i-1} \subset V_{i}$, $V_i=V_{5-i}^{\bot }$, $|V_i/V_{i-1}|=a_i$, and $a_i \geq 0$, for any $i\in [1,5]$. Consider the sets

$$\begin{equation*} Z_i=\{\, U\subset V_i \mid |U|=1, U\ \text{is isotropic}, \ U\not \subseteq V_{i-1} \,\}, \quad \forall i\in [1, 5]. \end{equation*}$$

We have

$$\begin{equation*} (\mathrm{i}) \ \# Z_3=q^{a_1+a_2} \left(\frac{q^{a_3-1}-1}{q-1}+q^{\frac{a_3}{2}-1} \right) \quad \text{and} \quad (\mathrm{ii}) \ \# Z_4=q^{a_1+a_2+a_3-1}\frac{q^{a_4}-1}{q-1}. \end{equation*}$$

Proof.

To prove (i), we consider the set $Z_3'=\{W\subset V_3| V_2 \overset{1}{\subset } W, W\ \text{is isotropic}\}$. Let $\phi : Z_3\rightarrow Z'_3$ be the map defined by $U\mapsto V_2+U$. Clearly, the map $\phi$ is surjective. Observe that the order of each fiber is $q^{|V_2|}=q^{a_1+a_2}$ and, moreover, $Z_3'$ gets identified with the set of all isotropic lines in $V_3/V_2$. By Lemma 3.1.2, we have $\#Z_3'=\frac{q^{a_3-1}-1}{q-1}+q^{\frac{a_3}{2}-1}$. This proves (i).

We now prove (ii). Consider the set $Z_4'=\{\, W\subset V_4 \mid V_1 \overset{1}{\subset } W$, $W\not \subset V_3$, $W$ is isotropic$\,\}$. We define a map $\phi ': Z_4\rightarrow Z'_4$ by $U\mapsto V_1+U$. This is a well-defined and surjective map and the cardinality of each fiber is $q^{|V_1|}=q^{a_1}$. To calculate the cardinality of $Z_4'$, we set

$$\begin{equation*} \tilde{Z}_i=\{\, U\subset V_i/V_1 \mid |U|=1, \text{$U$ is isotropic}\,\}. \end{equation*}$$

By a similar calculation in (i), we have

$$\begin{align*} \#Z'_4 &=\#\tilde{Z}_4-\#\tilde{Z}_3=\#\tilde{Z}_4-(\#\tilde{Z}_3\setminus \tilde{Z}_2+\#\tilde{Z}_2)\\ &=\frac{q^{a_2+a_3+a_4-1}-1}{q-1}+q^{a_2+\frac{a_3}{2}-1}- \left(q^{a_2} (\frac{q^{a_3-1}-1}{q-1}+q^{\frac{a_3}{2}-1})+\frac{q^{a_2}-1}{q-1}\right)\\ &=q^{a_2+a_3-1}\frac{q^{a_4}-1}{q-1}. \end{align*}$$

This proves (ii).

■

3.2. The first double centralizer

We fix another positive integer $n$ and let $N=2n+1.$ We fix a maximal isotropic vector subspace $M_d$ in $\mathbb{F}_q^D$ (of dimension $d$). Consider the following sets:

•

The set $\mathscr{X}$ of $N$-step flags $V=( V_i)_{ 0\leq i\leq N}$ in $\mathbb{F}_q^D$ such that $V_i\subseteq V_{i+1}$, $V_i= V_j^{\perp }$, for any $i+j = N$.

•

The set $\mathscr{Y}$ of complete flags $F= (F_i)_{0\leq i\leq D}$ in $\mathbb{F}_q^D$ such that $F_i\subset F_{i+1}$, $|F_i|=i$ and $F_i = F_j^{\perp }$, for any $i+ j = D$, and $|F_d\cap M_d|\equiv d \bmod 2$.

Let $G=\mathrm{SO}(D)$ be the special orthogonal group attached to $Q$. The sets $\mathscr{X}$ and $\mathscr{Y}$ admit naturally a $G$-action from the left. Moreoever, $G$ acts transitively on $\mathscr{Y}$ thanks to the condition $|F_d\cap M_d|\equiv d \bmod 2$. Let $G$ act diagonally on the product $\mathscr{X}\times \mathscr{X}$ (resp. $\mathscr{X}\times \mathscr{Y}$ and $\mathscr{Y}\times \mathscr{Y}$). Set

$$\begin{equation} \mathcal{A}=\mathbb{Z}[v, v^{-1}]. {\tag{4}} \end{equation}$$

Let

$$\begin{align} \mathcal{S}_{\mathscr{X}}=\mathcal{A}_G(\mathscr{X}\times \mathscr{X}) {\tag{5}} \end{align}$$

be the set of all $\mathcal{A}$-valued $G$-invariant functions on $\mathscr{X}\times \mathscr{X}$. Clearly, the set $\mathcal{S}_{\mathscr{X}}$ is a free $\mathcal{A}$-module. Moreover, $\mathcal{S}_{\mathscr{X}}$ admits an associative $\mathcal{A}$-algebra structure ‘$*$’ under a standard convolution product as discussed in BKLW14, 2.3. In particular, when $v$ is specialized to $\sqrt q$, we have

$$\begin{equation} f * g(V, V')=\sum _{V''\in \mathscr{X}}f(V, V'')g(V'',V'), \quad \forall \ V,V'\in \mathscr{X}. {\tag{6}} \end{equation}$$

Similarly, we define the free $\mathcal{A}$-modules

$$\begin{equation} \mathcal{V}=\mathcal{A}_G(\mathscr{X}\times \mathscr{Y}) \quad \text{and} \quad \mathcal{H}_{\mathscr{Y}}=\mathcal{A}_G(\mathscr{Y}\times \mathscr{Y}). {\tag{7}} \end{equation}$$

A similar convolution product gives an associative algebra structure on $\mathcal{H}_{\mathscr{Y}}$ and a left $\mathcal{S}_{\mathscr{X}}$-action and a right $\mathcal{H}_{\mathscr{Y}}$-action on $\mathcal{V}$. Moreover, these two actions commute and hence we have the following $\mathcal{A}$-algebra homomorphisms:

$$\begin{equation*} \mathcal{S}_{\mathscr{X}} \to \operatorname {End}_{\mathcal{H}_{\mathscr{Y}}} (\mathcal{V}) \quad \text{and}\quad \mathcal{H}_{\mathscr{Y}} \to \operatorname {End}_{\mathcal{S}_{\mathscr{X}}} (\mathcal{V}). \end{equation*}$$

By P09, Theorem 2.1, we have the following double centralizer property.

Lemma 3.2.1.

$\operatorname {End}_{\mathcal{H}_{\mathscr{Y}}}(\mathcal{V})\simeq \mathcal{S}_{\mathscr{X}}$ and $\operatorname {End}_{\mathcal{S}_{\mathscr{X}}}(\mathcal{V}) \simeq \mathcal{H}_{\mathscr{Y}}$, if $n\geq d$.

We note that the result in P09, Theorem 2.1 is obtained over the field $\mathbb{C}$ of complex numbers, but the proof can be adapted to our setting over the ring $\mathcal{A}$.

3.3. $G$-orbits on $\mathscr{X}\times \mathscr{Y}$ and $\mathscr{Y}\times \mathscr{Y}$

We shall give a description of the $G$-orbits on $\mathscr{X}\times \mathscr{Y}$ and $\mathscr{Y}\times \mathscr{Y}$. The description of the $G$-orbits on $\mathscr{X}\times \mathscr{X}$ is more subtle, and postponed until Section 4.2.

We start by introducing the following notation associated to a matrix $M=(m_{ij})_{1\leq i, j \leq c}$:

$$\begin{align} \begin{split} \operatorname {ro}(M) & = \left(\sum _{j=1}^{c} m_{ij} \right)_{1\leq i\leq c}, \\ \operatorname {co}(M) & = \left(\sum _{i=1}^{c} m_{ij} \right)_{1\leq j \leq c}, \\ \operatorname {ur}(M) & = \sum _{i\leq c/2, j\geq c/2+1} m_{ij}. \end{split} {\tag{8}} \end{align}$$

We also write $\operatorname {ro}(M)_i$ and $\operatorname {co}(M)_j$ for the $i$-th and $j$-th component of the row vectors of $\operatorname {ro}(M)$ and $\operatorname {co}(M)$, respectively.

To a pair $(F, F') \in \mathscr{Y}\times \mathscr{Y}$, we can associate a $D\times D$ matrix $\sigma (F, F') \equiv \sigma =(\sigma _{ij})_{1\leq i, j \leq D}$ by setting

$$\begin{equation} \sigma _{ij} = \dim \frac{F_{i-1}+ F_i \cap F_j'}{F_{i-1} +F_i \cap F_{j-1}'}, \quad \forall 1\leq i, j\leq D. {\tag{9}} \end{equation}$$

It is obvious that any two pairs of flags in the same $G$-orbit produce the same matrix. In particular, we can assume that $F_d =M_d$ since $G$ acts transitively on $\mathscr{Y}$. Under this assumption, we have $\operatorname {ur}(\sigma (F, F')) = | M_d/M_d\cap F'_d| \equiv 0$ mod $2$. So, the assignment in 9 defines a map $G \backslash \mathscr{Y}\times \mathscr{Y} \to \Sigma$, where $\Sigma$ is the set of all matrices $\sigma \equiv (\sigma _{ij})$ in $\operatorname {Mat}_{D\times D} (\mathbb{N})$ such that

$$\begin{align*} \operatorname {ro}(\sigma )_i = 1, \quad \operatorname {co}(\sigma )_j =1, \quad \sigma _{ij} = \sigma _{D+1-i, D+1-j},\quad \operatorname {ur}(\sigma ) \equiv 0 \bmod 2,\quad \forall i, j\in [1, D]. \end{align*}$$

Moreover this map is bijective:

$$\begin{align} G \backslash \mathscr{Y}\times \mathscr{Y} \simeq \Sigma . {\tag{10}} \end{align}$$

Indeed, let $F^0$ be a fixed flag in $\mathscr{Y}$ and let $B$ be the stabilizer of $F^0$ in $G$. Let $B$ act on $\mathscr{Y}$ via the $G$-action on $\mathscr{Y}$. The assignment $F \mapsto (F^0, F)$ defines a bijection of the set $B\backslash \mathscr{Y}$ of the $B$-orbits in $\mathscr{Y}$ and the set of $G$-orbits in $\mathscr{Y} \times \mathscr{Y}$. By the standard Bruhat decomposition, the former set $B\backslash \mathscr{Y}$ is parametrized by the Weyl group of type $D$ which is $\Sigma$ as a subset in $G$ (see S74, 2.14(iii) or BB05, 8.2 for details). We thus have 10.

Replacing the range of the index $i$ in 9 to $1\leq i \leq N$, we have a bijection

$$\begin{align} G \backslash \mathscr{X}\times \mathscr{Y} \simeq \Pi , {\tag{11}} \end{align}$$

where the set $\Pi$ consists of all matrices $B=(b_{ij})$ in $\mathrm{Mat}_{N\times D} (\mathbb{N})$ subject to

$$\begin{equation*} \operatorname {co}(B)_j =1;\quad b_{ij} = b_{N+1- i, D+1-j},\ \forall \ i\in [1, N],\ j\in [1, D]. \end{equation*}$$

(For a proof, see the proof of Lemma 4.2.1.)

Moreover, we have

$$\begin{equation} \begin{split} \# \Sigma = 2^{d-1} \cdot d! \quad \text{and}\quad \# \Pi = (2n+1)^d. \end{split} \tag{12} \end{equation}$$

3.4. $\mathcal{H}_{\mathscr{Y}}$-action on $\mathcal{V}$

We shall provide an explicit description of the action of $\mathcal{H}_{\mathscr{Y}}$ on $\mathcal{V}$. For any $1\leq j\leq d-1$, we define a function $\tau _j$ in $\mathcal{H}_{\mathscr{Y}}$ by

$$\begin{align*} \tau _j (F, F') &= \begin{cases} 1, & \text{if} \; F_i = F_i', \ \forall i\in [1, d]\backslash \{j\}, F_j\neq F_j',\\ 0, & \text{otherwise}, \end{cases}\\ \tau _d &= e_{(d-1, d+1)(d, d+2)}, \end{align*}$$

where $e_{(d-1, d+1)(d, d+2)}$ is the characteristic function of the $G$-orbit corresponding to the permutation matrix $(d-1, d+1)(d, d+2)$. Then we have the following well-known result.

Lemma 3.4.1.

The assignment of sending the functions $\tau _j$, for $1\leq j\leq d$, in the algebra $\mathcal{H}_{\mathscr{Y}}$ to the generators of $\mathbf{H}_d$ in the same notation is an isomorphism.

Given $B=(b_{ij})\in \Pi$, let $r_c$ be the unique number in $[1,N]$ such that $b_{r_c, c} =1$ for each $c\in [1, D]$. The correspondence $B\mapsto \tilde{\mathbf{r}}= (r_1,\dots , r_D)$ defines a bijection between $\Pi$ and the set of all sequences $(r_1,\dots , r_d)$ such that $r_i+ r_{D+1-i} = N+1$ for any $i\in [1, D]$. Denote by $e_{r_1 \dots r_D}$ the characteristic function of the $G$-orbit corresponding to the matrix $B$ in $\mathcal{V}$. It is clear that the collection of these characteristic functions provides a basis for $\mathcal{V}$.

Recall from Section 2.1 that we have the space $\mathbf{V}^{\otimes d}$ spanned by vectors $\mathbf{v_r}$ and to each sequence $\mathbf{r}$ a sequence $\tilde{\mathbf{r}}$ is uniquely defined. Thus we have an isomorphism of vector spaces over $\mathbb{Q}(v)$:

$$\begin{align} \mathbf{V}^{\otimes d} \to \mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{V}, \quad \mathbf{v}_{\mathbf{r}} \mapsto e_{r_1,\dots , r_D}. {\tag{13}} \end{align}$$

Moreover, we have

Lemma 3.4.2.

The action of $\mathcal{H}_{\mathscr{Y}}$ on $\mathcal{V}$ is described as follows. For $1\leq j\leq d-1$, we have

$$\begin{align} e_{r_1 \dots r_D} \tau _j &= \begin{cases} e_{r_1\dots r_{j-1} r_{j+1} r_j\dots r_{D-j} r_{D-j-1} r_{D-j+1} \dots r_D} , & r_j < r_{j+1},\\ v^2 e_{r_1\dots r_D}, & r_j = r_{j+1},\\ (v^2-1) e_{r_1\dots r_D} + v^2 e_{r_1 \dots r_{j-1} r_{j+1} r_j \dots r_{D-j} r_{D-j-1} r_{D-j+1} \dots r_D}, & r_j> r_{j+1}, \end{cases} {\tag{14}}\\[3.0pt] e_{r_1 \dots r_{D}} \tau _d &= \begin{cases} e_{r_1\dots r_{d-2} r_{d+1} r_{d+2}r_{d-1}r_d r_{d+3} \dots r_D} , & r_{d-1}+r_d < N+1,\\ v^2 e_{r_1\dots r_D}, & r_{d-1}+r_d =N+1,\\ (v^2-1) e_{r_1\dots r_D} + v^2 e_{r_1\dots r_{d-2} r_{d+1} r_{d+2}r_{d-1}r_d r_{d+3} \dots r_D}, &r_{d-1}+r_d > N+1. \end{cases} {\tag{15}} \end{align}$$

Proof.

Formula 14 agrees with the one in GL92, 1.12, whose proof is also the same as the one for the type-$\mathbf{A}$ case. We shall prove 15. It suffices to show the result by specializing $v$ to $\sqrt q$. By the definition of convolution product, we have

$$\begin{equation*} e_{r_1 \dots r_{D}} \tau _d (V, F) =\sum _{F'\in \mathscr{Y}} e_{r_1 \dots r_{D}} (V, F') \tau _d(F', F). \end{equation*}$$

By the definition of $\tau _d$, we have $F'_i=F_i$ if $i\neq d-1$, $d$, $d+1$ or $d+2$. So the calculation is reduced to the case when $D=4$. Note also that it is enough to calculate the case when $n=2$, which we will assume.

If two of $r_1$, $r_2$, $r_3$, $r_4$ are equal, then the calculation can be reduced further to the case when $n=1$. In this case, we have

$$\begin{align*} e_{2132}\tau _2 &= e_{3221}, & e_{2222} \tau _2 &= v^2e_{2222}, & e_{2312}\tau _2 &= (v^2-1)e_{2312}+v^2e_{1223},\\ e_{1223}\tau _2 &= e_{2312}, & e_{1313} \tau _2 &= v^2e_{1313}, & e_{3221}\tau _2 &= (v^2-1)e_{3221}+v^2e_{2132},\\ e_{1133}\tau _2 &= e_{3311}, & e_{3131} \tau _2 &= v^2e_{3131}, & e_{3311}\tau _2 &= (v^2-1)e_{3311}+v^2e_{1133}. \end{align*}$$

For the case when $r_1, r_2, r_3, r_4$ are all distinct, we have

$$\begin{align*} e_{2514} \tau _2&=(v^2-1)e_{2514}+v^2e_{1425}, & e_{4512} \tau _2&=(v^2-1)e_{4512}+v^2e_{1245},\\ e_{5241} \tau _2&=(v^2-1)e_{5241}+v^2e_{4152}, & e_{5421} \tau _2&=(v^2-1)e_{5421}+v^2e_{2154},\\ e_{1245} \tau _2&=e_{4512}, \quad e_{1425} \tau _2=e_{2514}, & e_{2154} \tau _2&=e_{5421}, \quad e_{4152} \tau _2=e_{5241}. \end{align*}$$

Formula 15 follows from the above computations.

■

4. Calculus of the algebra $\mathcal{S}$

Recall from the previous section that $\mathcal{S}_{\mathscr{X}}$ is the convolution algebra on $\mathscr{X}\times \mathscr{X}$ defined in 5. For simplicity, we shall say $\mathcal{S}$ instead of $\mathcal{S}_{\mathscr{X}}$. In this section, we determine the generators for $\mathcal{S}$ and the associated multiplication formula. Furthermore, we provide a (conjectural) algebraic presentation of $\mathcal{S}$ and deduce various bases.

4.1. Defining relations of $\mathcal{S}$

For any $i\in [1, n]$, $a\in [1, n+1]$, we set

$$\begin{align} \begin{split} E_i (V, V') &= \begin{cases} v^{-|V'_{i+1}/V'_i|}, &\text{if}\; V_i\overset{1}{\subset } V_i', V_j=V_{j'},\forall j\in [1,n]\backslash \{i\}, \\ 0, &\text{otherwise}, \end{cases} \\ F_i (V, V') &= \begin{cases} v^{-|V'_i/V'_{i-1}|}, &\text{if}\; V_i\overset{1}{\supset } V_i', V_j=V_{j'},\forall j\in [1,n]\backslash \{i\}, \\ 0, &\text{otherwise}, \end{cases} \\ H_a^{\pm 1} (V, V') & = \begin{cases} v^{\mp |V_a'/V_{a-1}'|} , &\text{if}\; V=V',\\ 0, & \text{otherwise}, \end{cases} \\ J_+(V,V') &= \begin{cases} 1 , &\text{if}\; V=V',\ |V_n|=d,\ \text{and}\ |V_n\cap M_d|\equiv d\bmod 2,\\ 0, & \text{otherwise}, \end{cases} \\ J_-(V,V') &= \begin{cases} 1 , &\text{if}\; V=V',\ |V_n|=d,\ \text{and}\ |V_n\cap M_d|\equiv d-1\bmod 2,\\ 0, & \text{otherwise}, \end{cases} \\ J_0&=1-J_+-J_-. \end{split} \tag{16} \end{align}$$

It is clear that these functions are elements in $\mathcal{S}$.

Proposition 4.1.1.

The functions $E_i, F_i$, $H_a^{\pm 1}$, and $J_{\alpha }$ in $\mathcal{S}$, for any $i\in [1,n]$, $a\in [1,n+1]$, and $\alpha \in \{ \pm , 0\}$, satisfy the defining relations of the algebra $\mathbf{U}$ in Section 2.1, together with the following relations:

$$\begin{equation} H_{n+1}\prod _{i=1}^n H_i\,^2=v^{-D}\quad \text{and}\quad \prod _{l=1}^d(H_j-v^{-l})=0,\quad \forall j\in [1,n]. {\tag{17}} \end{equation}$$

Proof.

The proofs of the identities in the first four rows of the defining relations of $\mathbf{U}$ are straightforward. We show the identity in the fifth row. Let $\lambda '_i=|V'_i/V'_{i-1}|$. We have

$$\begin{equation*} F_nE_{n}(V,V')=\begin{cases} \frac{v^{2\lambda _n'}-1}{v^2-1}v^{-\lambda _n'-\lambda '_{n+1}+1}, & \text{if $V=V'$},\\[3.0pt] v^{-\lambda _n'-\lambda '_{n+1}+1}, & \text{if $|V_n\cap V_n'|=|V_n|-1=|V_n'|-1$},\\[3.0pt] 0,&\text{otherwise}. \end{cases} \end{equation*}$$

We set

$$\begin{equation*} \mathscr{X}^3=\{\, V\in \mathscr{X} \mid |V_n|=d,\ |V_n\cap M_d|\equiv d-1\bmod 2 \,\}. \end{equation*}$$

It is a $G$-orbit and we have $|V_n\cap V_n'|\equiv d\bmod 2$ for any $V, V'\in \mathscr{X}^3$. Therefore,

$$\begin{equation*} (J_+F_nE_{n}-F_nE_nJ_-)(V,V')=\begin{cases} \frac{v^{\lambda _n'}-v^{-\lambda _n'}}{v-v^{-1}},& \text{if}\ V=V'\not \in \mathscr{X}^3,\\[3.0pt] -\frac{v^{\lambda _n'}-v^{-\lambda _n'}}{v-v^{-1}},& \text{if}\ V=V'\in \mathscr{X}^3,\\[3.0pt] 0,&\text{otherwise}. \end{cases} \end{equation*}$$

It is easy to check that the right-hand side is equal to

$$\begin{equation*} \frac{H_n^{-1}H_{n+1}-H_nH_{n+1}^{-1}}{v-v^{-1}}(J_{+}-J_{-})(V,V'). \end{equation*}$$

We now show the penultimate identity. By a direct calculation, we have

$$\begin{align*} E_nF_nE_n(V,V')&= \begin{cases} \lefteqn{(\frac{v^{2\lambda '_n}-1}{v^2-1}+\frac{v^{2\lambda _{n+1}'+2}-1}{v^2-1} +v^{D-2|V_n|-2}-1)v^{-\lambda '_n-2\lambda _{n+1}'+1}},\\[3.0pt] & \text{if $V_n \overset{1}{\subset } V'_n$},\\[3.0pt] v^{-\lambda '_n-2\lambda _{n+1}'+1}, & \text{if $|V_n \cap V'_n|=|V_n|-1$ and $V_n\not \subset V'_{n+1}$},\\[3.0pt] (v^2+1)v^{-\lambda '_n-2\lambda _{n+1}'+1}, & \text{if $|V_n \cap V'_n|=|V_n|-1$ and $V_n \subset V'_{n+1}$},\\ 0, &\text{otherwise}, \end{cases} \\[3.0pt] E_n^2F_n(V,V') &= \begin{cases} \lefteqn{(\frac{v^{2\lambda _{n+1}'-2}-1}{v^2-1} +v^{D-2|V'_n|-2})(v^2+1)v^{-\lambda '_n-2\lambda _{n+1}'+2},}\\[3.0pt] & \text{if $V_n \overset{1}{\subset } V'_n$ and $|V'_n|<d$}, \\[3.0pt] (v^2+1)v^{-\lambda '_n-2\lambda _{n+1}'+2} , & \text{if $|V_n \cap V'_n|=|V_n|-1$, $|V'_n|<d$, and $V_n \subset V'_{n+1}$},\\[3.0pt] 0, & \text{otherwise}, \end{cases} \\[3.0pt] F_n E_n^2(V,V')&=\begin{cases} \frac{v^{2\lambda _{n}'-2}-1}{v^2-1} (v^2+1)v^{-\lambda '_n-2\lambda _{n+1}'}, & \text{if $V_n \overset{1}{\subset } V'_n$},\\[3.0pt] (v^2+1)v^{-\lambda '_n-2\lambda _{n+1}'}, & \text{if $|V_n \cap V'_n|=|V_n|-1$},\\[3.0pt] 0, & \text{otherwise}. \end{cases} \end{align*}$$

The penultimate identity follows.

To prove the last identity, we define a map $\rho : \mathcal{S} \rightarrow \mathcal{S}$ such that $\rho (f)(V,V')=f(V',V)$. It is clear that $\rho$ is an anti-automorphism. Moreover, we have

$$\begin{align} \rho (E_n)=v^{-1}H_n^{-1}H_{n+1}F_n,\quad \rho (F_n)=v^{-2}H_nH_{n+1}^{-1} E_n, \quad \text{and}\quad \rho (H_n^{\pm 1})=H_n^{\pm 1}. {\tag{18}} \end{align}$$

Applying $\rho$ to both sides of the penultimate identity, we get the last identity. The rest of the relations are reduced to the type-$\mathbf{A}$ case, and will not be reproduced here.

■

4.2. Parametrization of $G$-orbits on $\mathscr{X}\times \mathscr{X}$

In order to describe the structure of the algebra $\mathcal{S}$, we need to parametrize the $G$-orbits in $\mathscr{X}\times \mathscr{X}$. Recall from Section 3.2 that $\mathscr{X}$ is the set of $N$-step flags in $\mathbb{F}_q^D$ such that $V_i= V_j^{\perp }, \forall \ i+j = N$. For any pair $(V, V')$ of flags in $\mathscr{X}$, we can assign an $N$-by-$N$ matrix as in 9 whose $(i,j)$-entry is equal to $\dim \frac{V_{i-1}+ V_i\cap V_j'}{V_{i-1} + V_i\cap V_{j-1}'}$. It is clear that this assignment is $G$-invariant. Thus we have a map

$$\begin{align} \tilde{\Phi }: G\backslash \mathscr{X}\times \mathscr{X} \to \Xi , {\tag{19}} \end{align}$$

where $\Xi$ is the set of all $N\times N$ matrices $A$ with entries in $\mathbb{N}$ subject to

$$\begin{equation*} \sum _{i,j\in [1,N]}a_{ij}=D,\ a_{ij} = a_{N+1 -i, N+1-j}, \quad \forall i, j\in [1, N]. \end{equation*}$$

This map is surjective, but not injective. We need to refine it.

Recall that $M_d$ is a fixed maximal isotropic subspace in $\mathbb{F}_q^D$ and

$$\begin{equation*} \mathscr{X}^3=\{\, V \in \mathscr{X} \mid |V_n|=d\ \text{and}\ |V_n\cap M_d| \equiv d-1\bmod 2\,\}. \end{equation*}$$

We set

•

$\mathscr{X}^1=\{\, V \in \mathscr{X} \mid |V_n|<d \,\}$,

•

$\mathscr{X}^2=\{\, V \in \mathscr{X} \mid |V_n|=d\ \text{and}\ |V_n\cap M_d| \equiv d\bmod 2 \,\}$.

We have a partition of $\mathscr{X}$:

$$\begin{equation*} \mathscr{X} =\mathscr{X}^1\sqcup \mathscr{X}^2 \sqcup \mathscr{X}^3. \end{equation*}$$

Let $O(D)$ be the orthogonal group associated to $Q$. For any $g\in O(D)\setminus G$, the map $\psi _g: \mathscr{X}^2 \rightarrow \mathscr{X}^3$, defined by $V\mapsto g\cdot V$, is a bijection, which yields the following bijections:

$$\begin{equation} \begin{split} & G \backslash \mathscr{X}^1\times \mathscr{X}^2 \rightarrow G \backslash \mathscr{X}^1\times \mathscr{X}^3, \quad G \backslash \mathscr{X}^2\times \mathscr{X}^1 \rightarrow G \backslash \mathscr{X}^3\times \mathscr{X}^1, \\ & G \backslash \mathscr{X}^2\times \mathscr{X}^2 \rightarrow G \backslash \mathscr{X}^3\times \mathscr{X}^3, \quad G \backslash \mathscr{X}^2\times \mathscr{X}^3 \rightarrow G \backslash \mathscr{X}^3\times \mathscr{X}^2. \end{split} {\tag{20}} \end{equation}$$

Moreover, corresponding pairs on both sides under the bijections in 20 get sent to the same matrix by $\tilde{\Phi }$. Corresponding to 20, we define a sign function

$$\begin{align} \operatorname {sgn}(i, j) = \begin{cases} 0, & (i, j) =(1, 1),\\ +, & (i, j)= (1,2), (2, 1), (2,2), (2,3),\\ -, & (i, j)= (1, 3), (3, 1), (3,3), (3,2). \end{cases} {\tag{21}} \end{align}$$

Recalling the notation $\operatorname {ro}(A)$ and $\operatorname {co}(A)$ from 8, we set

$$\begin{align} \begin{split} \Xi ^0 &=\{ \, A\in \Xi \mid \operatorname {ro}(A)_{n+1} > 0, \ \operatorname {co}(A)_{n+1}> 0 \, \}\times \{0\}, \\ \Xi ^+ &=\Xi \backslash \Xi ^0 \times \{+\}, \\ \Xi ^- &=\Xi \backslash \Xi ^0\times \{ - \}. \end{split} {\tag{22}} \end{align}$$

For convenience, we sometimes write $A^{\pm }$ for $(A,\pm )\in \Xi ^{\pm }$ and $A^0$ for $(A, 0)$ in $\Xi ^0$. We further set

$$\begin{align} \Xi _{\mathbf{D}}=\Xi ^+\sqcup \Xi ^0 \sqcup \Xi ^-. {\tag{23}} \end{align}$$

Elements in $\Xi _{\mathbf{D}}$ will be called $signed$ $matrices$. We have the following lemma.

Lemma 4.2.1.

The map $\tilde{\Phi }$ in 19 induces a bijection

$$\begin{align} \Phi : G\backslash \mathscr{X}\times \mathscr{X}\to \Xi _{\mathbf{D}}, \quad G.(V, V') \mapsto \mathfrak{a}\equiv (A,\alpha ), {\tag{24}} \end{align}$$

such that $\tilde{\Phi }(G.(V,V') )=A$ and $\alpha =\operatorname {sgn}(i, j)$ if $(V, V') \in \mathscr{X}^i\times \mathscr{X}^j$. Moreover, we have

$$\begin{equation} \begin{split} \# \Xi _{\mathbf{D}}=\binom{2n^2+2n+d}{d}+2\binom{2n^2+n+d-1}{d}-\binom{2n^2+d-1}{d}. \end{split} {\tag{25}} \end{equation}$$

Proof.

Similar to BKLW14, Lemma 2.1, there is a bijection

$$\begin{equation*} \phi : O(D) \backslash \mathscr{X} \times \mathscr{X} \rightarrow \Xi ^0 \sqcup \Xi ^+. \end{equation*}$$

Here we identify $\Xi$ with $\Xi ^0 \sqcup \Xi ^+$. Let

$$\begin{equation*} \mathscr{X}^m = \mathscr{X}^2 \sqcup \mathscr{X}^3 \quad \text{and}\quad \mathscr{Z} = \mathscr{X}^1\times \mathscr{X}^m \sqcup \mathscr{X}^m\times \mathscr{X}^1 \sqcup \mathscr{X}^m\times \mathscr{X}^m. \end{equation*}$$

By the construction of $\phi$, we have $\phi (O(D)\backslash \mathscr{X}^1 \times \mathscr{X}^1) = \Xi ^0$ and $\phi (O(D)\backslash \mathscr{Z}) = \Xi ^+$. By 20, for any $x\in \mathscr{Z}$, each $O(D)$-orbit $O(D)\cdot x$ can be split into two $G$-orbits. This gives a copy of $\Xi ^+$, which we denote by $\Xi ^-$. The bijection 24 follows.

We now calculate $\#\Xi _{\mathbf{D}}$. From 22 $\#\Xi _{\mathbf{D}}=\#\Xi +\#\Xi ^-$. We have

$$\begin{equation} \begin{split} \#\Xi &=\#\left\{a_{ij}, i\in [1,n], \forall j; a_{n+1,j},j\in [1,n]|\sum _{ i\leq n; j}a_{ij} +\sum _{j\leq n}a_{n+1,j}=d-\frac{a_{n+1,n+1}}{2} \right\}\\ &=\sum _{l=0}^d\binom{2n^2+2n+d-l-1}{d-l}=\binom{2n^2+2n+d}{d}. \end{split} {\tag{26}} \end{equation}$$

Denote $\Xi ^-_1=\{A\in \Xi ^-|a_{n+1,j}=0,\forall j\}$ and $\Xi ^-_2=\{A\in \Xi ^-|a_{i,n+1}=0,\forall i\}$. Then $\Xi ^-=\Xi ^-_1\sqcup \Xi ^-_2$ and

$$\begin{equation} \#\Xi ^-=\#\Xi ^-_1+\#\Xi ^-_2-\#\Xi ^-_1\cap \Xi ^-_2=2\binom{2n^2+n+d-1}{d}-\binom{2n^2+d-1}{d}. {\tag{27}} \end{equation}$$

The equality 25 follows from 26 and 27. The lemma is proved.

■

4.3. Multiplication formulas in $\mathcal{S}$

For each signed matrix $\mathfrak{a}\in \Xi _{\mathbf{D}}$, we denote by $\mathcal{O}_{\mathfrak{a}}$ the associated $G$-orbit. We introduce the following notation:

$$\begin{align} \begin{split} \sup (\mathfrak{a}) & = (i, j),\quad \text{if}\ \mathcal{O}_{\mathfrak{a}} \subseteq \mathscr{X}^i\times \mathscr{X}^j, \\ \operatorname {sgn}(\mathfrak{a}) & = \operatorname {sgn}(\sup (\mathfrak{a})), \\ \operatorname {ro}(\mathfrak{a}) & = \operatorname {ro}(A), \\ \operatorname {co}(\mathfrak{a}) & = \operatorname {co}(A), \\ \operatorname {ur}(\mathfrak{a}) &= \operatorname {ur}(A), \\ \mathfrak{a}+B &= A+B, \quad \text{if} \ \mathfrak{a}=(A,\alpha ) \ \text{and}\ B \ \text{a matrix}, \\ \operatorname {p}( \mathfrak{a}) & = \begin{cases} 1, & \text{if $\operatorname {ur}(\mathfrak{a})$ is odd},\\ 0, & \text{otherwise}. \end{cases} \end{split} {\tag{28}} \end{align}$$

We note that $\mathfrak{a}+B$ is a matrix instead of a signed matrix. For a signed matrix $\mathfrak{a}\in \Xi _{\mathbf{D}}$, we define

$$\begin{equation} \begin{split} s_l(\mathfrak{a})&= \begin{cases} 1, & \text{if $\operatorname {ro}(\mathfrak{a})_{n+1} > 0$},\\ 2, & \text{if $\operatorname {ro}(\mathfrak{a})_{n+1} = 0$, $\operatorname {sgn}(\mathfrak{a}) =+$},\\ 3, & \text{if $\operatorname {ro}(\mathfrak{a})_{n+1} = 0$, $\operatorname {sgn}(\mathfrak{a}) = -$}, \end{cases} \\[3.0pt] s_r(\mathfrak{a})&= \begin{cases} 1, & \text{if $\operatorname {co}(\mathfrak{a})_{n+1}>0$},\\ 3- \operatorname {p}(\mathfrak{a})\delta _{0, \operatorname {ro}(\mathfrak{a})_{n+1}}, & \text{if $\operatorname {co}(\mathfrak{a})_{n+1} =0$, $\operatorname {sgn}(\mathfrak{a}) = -$},\\ 2+ \operatorname {p}(\mathfrak{a})\delta _{0, \operatorname {ro}(\mathfrak{a})_{n+1}}, & \text{if $\operatorname {co}(\mathfrak{a})_{n+1} =0$, $\operatorname {sgn}(\mathfrak{a}) = +$}. \end{cases} \end{split} {\tag{29}} \end{equation}$$

Then, we have

Lemma 4.3.1.

$\sup (\mathfrak{a})=( s_l(\mathfrak{a}), s_r(\mathfrak{a}))$ and $\operatorname {sgn}(\mathfrak{a}) = \operatorname {sgn}(s_l(\mathfrak{a}), s_r(\mathfrak{a}))$, for all $\mathfrak{a}\in \Xi _{\mathbf{D}}$.

For any $n\in \mathbb{Z}, k\in \mathbb{N}$, we set

$$\begin{equation*} (n)_v=\frac{v^{2n}-1}{v^2-1} \quad \text{and}\quad \begin{pmatrix} n\\ k \end{pmatrix}_{v}= \prod _{i=1}^k\frac{(n+1-i)_v}{(i)_v}. \end{equation*}$$

Let

$$\begin{equation*} E_{ij}^{\theta }=E_{ij}+E_{N+1-i, N+1-j}, \end{equation*}$$

where $E_{ij}$ is the $N\times N$ matrix whose $(i,j)$-entry is 1 and all other entries are 0. Let $e_{\mathfrak{a}}$ be the characteristic function of the $G$-orbit corresponding to $\mathfrak{a}\in \Xi _{\mathbf{D}}$. It is clear that the set $\{e_{\mathfrak{a}}| \mathfrak{a}\in \Xi _{\mathbf{D}}\}$ forms a basis of $\mathcal{S}$. For convenience, we set

$$\begin{equation*} e_{\mathfrak{a}} =0,\quad \text{if}\ \mathfrak{a}\not \in \Xi _{\mathbf{D}}. \end{equation*}$$

Recall the notation, such as $\mathfrak{a}+ B$, from 28. We have

Proposition 4.3.2.

Suppose that $\mathfrak{a}\equiv A^{\alpha }$, $\mathfrak{b}$, $\mathfrak{c}\in \Xi _{\mathbf{D}}$, and $h\in [1, n]$.

(a) If $\mathfrak{b}$ is chosen such that $\mathfrak{b}-E_{h,h+1}^{\theta }$ is diagonal, $\operatorname {co}(\mathfrak{b})= \operatorname {ro}(\mathfrak{a})$, and $s_r(\mathfrak{b})=s_l(\mathfrak{a})$, then

$$\begin{equation} \begin{split} e_{\mathfrak{b}} * e_{\mathfrak{a}}=\sum _{1\leq p \leq N} v^{2\sum _{j>p}a_{h,j}} & (1+ a_{h,p})_v\ e_{\mathfrak{a}_p}, \quad \text{where}\\ & \mathfrak{a}_p =(A+E^{\theta }_{h,p}-E^{\theta }_{h+1,p}, \operatorname {sgn}(s_l(\mathfrak{b}), s_r(\mathfrak{a}))) \in \Xi _{\mathbf{D}}. \end{split}{\tag{30}} \end{equation}$$

(b) If $h\neq n$ and $\mathfrak{c}$ has that $\mathfrak{c}-E_{h+1,h}^{\theta }$ is diagonal, $\operatorname {co}(\mathfrak{c})= \operatorname {ro}(\mathfrak{a})$, and $s_r(\mathfrak{c})=s_l(\mathfrak{a})$, then

$$\begin{align} e_{\mathfrak{c}} * e_{\mathfrak{a}}=\sum _{1\leq p\leq N} v^{2\sum _{j<p} a_{h+1,j}} & (1+a_{h+1,p})_v\ e_{\mathfrak{a}(h, p)}, \quad \text{where}{\tag{31}}\\ & \mathfrak{a}(h, p) = (A-E^{\theta }_{h, p}+E^{\theta }_{h+1,p}, \operatorname {sgn}(s_l(\mathfrak{c}), s_r(\mathfrak{a}))) \in \Xi _{\mathbf{D}}. \end{align}$$

(c) If the condition $h\neq n$ in (b) is replaced by $h=n$, then we have

$$\begin{align} e_{\mathfrak{c}} * e_{\mathfrak{a}} = & \sum _{p\neq n+1, a_{n,p}\geq 1}v^{2\sum _{j<p}a_{n+1,j}} (1+a_{n+1,p})_v\ e_{\mathfrak{a}(n, p)} {\tag{32}}\\ & +v^{2\sum _{j<n+1}a_{n+1,j}} ((1+a_{n+1,n+1})_v +(1-\delta _{0, \operatorname {ro}(\mathfrak{a})_{n+1}}) v^{a_{n+1,n+1}})\ e_{\mathfrak{a}(n, n+1)}. \end{align}$$

Remark 4.3.3.

Although the formulas 30, 31, and 32 look similar to formulas (3.9), (3.10), and (3.11) in BKLW14, they are different in many ways. For example, if we take $h=n$ and $\mathfrak{a}$ in 30 to be such that $A - E^{\theta }_{n+1, n}$ is diagonal, then we have

$$\begin{equation*} e_{\mathfrak{b}} * e_{\mathfrak{a}} = \begin{cases} (1+ a_{nn})_v \ e_{\mathfrak{a}_n}, & \text{if}\ \operatorname {sgn}(\mathfrak{b}) =\operatorname {sgn}(\mathfrak{a}),\\ e_{\mathfrak{a}_{n+2}}, & \text{otherwise}. \end{cases} \end{equation*}$$

While in BKLW14, the product $e_B * e_A$ is a sum of the terms in the right-hand side of the above identity in a similar situation.

Proof.

The proofs of (a) and (b) are the same as the proof for Lemma 3.2 in BLM90. We must show (c), which can be reduced to analogous results at the specialization of $v$ to $\sqrt q$. We first deal with the case when $\operatorname {ro}(\mathfrak{a})_{n+1}>0$. Let $V=(V_i)_{ 1\leq i\leq N}$ and $V'=(V'_i)_{ 1\leq i\leq N}$ be two flags such that $(V,V')\in \mathcal{O}_{\mathfrak{a}'}$ for a signed matrix $\mathfrak{a}'=(a'_{ij}, \operatorname {sgn}(s_l(\mathfrak{c}), s_r(\mathfrak{a})))$. Set $Z=\{U| V_n\overset{1}{\subset } U\subset V_{n+1}, U\ \text{is isotropic}\}$. Let $V_U$ be the flag obtained by replacing $V_n$ (resp. $V_{n+1}$) in $V$ by $U$ (resp. $U^{\bot }$). Then $(V, V_U)\in \mathcal{O}_{\mathfrak{c}}$ if and only if $U\in Z$. Let

$$\begin{equation*} Z_p=\{U\in Z|V_n\cap V_j'=U\cap V_j'\ \text{if}\ j<p; V_n\cap V_j'\neq U\cap V_j'\ \text{if}\ j\geq p\}, \end{equation*}$$

so that $\{Z_p|1\leq p\leq N\}$ forms a partition of $Z$. Moreover, if $U\in Z_p$ and $(V_U, V')\in \mathcal{O}_{\mathfrak{a}}$, then

$$\begin{equation*} a_{n,p}=a'_{n,p}+1\quad \text{and}\quad a_{n+1,p}=a'_{n+1,p}-1. \end{equation*}$$

Hence $\mathfrak{a}' = \mathfrak{a}(n,p)$. In particular, we have

$$\begin{equation*} e_{\mathfrak{c}} * e_{\mathfrak{a}}=\sum _p \#Z_p\ e_{\mathfrak{a}(n,p)}. \end{equation*}$$

Observe that

$$\begin{equation*} Z_p \simeq \{U\ \text{is isotropic}|V_n\overset{1}{\subset }U \subset V_n+V_{n+1}\cap V_p',\ U\not \subset V_n+V_{n+1}\cap V_{p-1}'\}. \end{equation*}$$

If $p\leq n$ and $V_n+V_{n+1}\cap V_p'$ is isotropic, then

$$\begin{equation*} \#Z_p=(q-1)^{-1}(q^{\sum _{1\leq j\leq p}a'_{n+1,j}}-q^{\sum _{1\leq j\leq p-1}a'_{n+1,j}}) =q^{\sum _{1\leq j< p}a_{n+1,j}}\frac{q^{1+a_{n+1,p}}-1}{q-1}. \end{equation*}$$

This matches the coefficient of the first term on the right-hand side of 32 for $p\leq n$.

We now compute the number $\# Z_p$ for $p\geq n+1$. We set $W_i= \frac{V_n+V^{\bot }_{n}\cap V'_{i}}{V_n}$ and consider the flags

$$\begin{equation*} \begin{array}{ll} 0\subset W_{N-p} \subset W_{N-p+1} \subset W_{p-1} \subset W_{p} \subset W_N,& \text{if}\ p>n+1, \\[3.0pt] 0\subset W_n \subset W_{n+1} \subset W_N,& \text{if}\ p=n+1. \end{array} \end{equation*}$$

So $Z_p\simeq \{U\ \text{is isotropic}\ |\ U\subset W_p, U\not \subset W_{p-1}, |U|=1\}$ if $p\geq n+1$. From this observation and applying Lemma 3.1.3, we have that $\# Z_p$ matches the coefficients of the terms in 32. Therefore, we have (c) when $\operatorname {ro}(\mathfrak{a})_{n+1} >0$.

Finally, we assume that $\operatorname {ro}(\mathfrak{a})_{n+1} =0$. In this case, $\operatorname {sgn}(\mathfrak{a}) = +$ or $-$, and hence $e_{\mathfrak{c}} e_{\mathfrak{a}} (V,V')=\sum _{V_U} e_{\mathfrak{c}} (V,V_U) e_{\mathfrak{a}} (V_U,V')$ and $V_U$ runs as follows:

$$\begin{equation*} \begin{cases} V_U\in \mathscr{X}^2,& \text{if $\operatorname {sgn}(\mathfrak{a}) = +$}, \\ V_U\in \mathscr{X}^3,& \text{if $\operatorname {sgn}(\mathfrak{a}) = -$}. \end{cases} \end{equation*}$$

Moreover, if $(V_U,V')\in \mathcal{O}_{\mathfrak{a}}$, then $|U|=d$ and $|V_n|=d-1$. Given such $V_n$, there exists a unique maximum isotropic vector subspace $U$ such that $V_n \overset{1}{\subset } U$ and $|U \cap M_d| \equiv d \bmod 2$ (or $|U \cap M_d| \not \equiv d\bmod 2$, but not both), where $M_d$ is the fixed maximum isotropic subspace in Section 3.2. In this case, the coefficient of $e_{\mathfrak{a}(n, n+1)}$ is equal to 1 in both cases. Therefore, we have (c) for the case $\operatorname {ro}(\mathfrak{a})_{n+1} =0$.

■

Recall the notation from 28. By Proposition 4.3.2 and an induction process, we have the following corollary.

Corollary 4.3.4.

Suppose that $\mathfrak{a}=A^{\alpha }$, $\mathfrak{b}$, $\mathfrak{c}\in \Xi _{\mathbf{D}}$, $h\in [1, n]$, and $r\in \mathbb{N}$.

(a) If $\operatorname {co}(\mathfrak{b})= \operatorname {ro}(\mathfrak{a})$, $s_r(\mathfrak{b})=s_l(\mathfrak{a})$, and $\mathfrak{b}-rE_{h,h+1}^{\theta }$ is diagonal, then we have

$$\begin{align} e_{\mathfrak{b}} * e_{\mathfrak{a}} =\sum _{t =(t_u)\in \mathbb{N}^N: \sum _{u=1}^Nt_u=r} v^{2\sum _{j>u}a_{hj}t_u}\prod _{u=1}^N\begin{pmatrix} a_{hu}+t_u \\ t_u\end{pmatrix}_{v} e_{\mathfrak{a}_t}, \quad \text{where} {\tag{33}}\\ \mathfrak{a}_t=\left(A+\sum _{u=1}^Nt_u(E^{\theta }_{hu}-E^{\theta }_{h+1,u}),\operatorname {sgn}(s_l(\mathfrak{b}), s_r(\mathfrak{a})) \right) \in \Xi _{\mathbf{D}}. \end{align}$$

(b) If $h\neq n$, $\operatorname {co}(\mathfrak{c})= \operatorname {ro}(\mathfrak{a})$, $s_r(\mathfrak{c})=s_l(\mathfrak{a})$, and $\mathfrak{c}-rE_{h+1,h}^{\theta }$ is diagonal, then

$$\begin{align} e_{\mathfrak{c}} * e_{\mathfrak{a}} =\sum _{t =(t_u)\in \mathbb{N}^N: \sum _{u=1}^Nt_u=r} v^{2\sum _{j < u}a_{h+1,j}t_u}\prod _{u=1}^N\begin{pmatrix} a_{h+1,u}+t_u\\ t_u\end{pmatrix}_{v}e_{\mathfrak{a}(h, t)}, {\tag{34}}\\ \text{where}\quad \mathfrak{a}(h, t) =\left(A-\sum _{u=1}^Nt_u(E^{\theta }_{hu}-E^{\theta }_{h+1,u}), \operatorname {sgn}(s_l(\mathfrak{c}), s_r(\mathfrak{a})) \right) \in \Xi _{\mathbf{D}}. \end{align}$$

(c) If the condition $h\neq n$ in (b) is replaced by $h=n$, then we have

$$\begin{align} e_{\mathfrak{c}} * e_{\mathfrak{a}} &= \sum _{t=(t_u)\in \mathbb{N}^N: \sum _{u=1}^N t_u=r} v^{\tilde{\beta }(t)} \ \mathcal{G} \ e_{\mathfrak{a}(n, t)},\quad \text{where} \ {\tag{35}}\\ \tilde{\beta }(t) & = 2\sum _{j<u}a_{n+1,j}t_u+2\sum _{N+1-j<u<j}t_jt_u+\sum _{u>n+1}t_u(t_u-1), \\ \mathcal{G} &= \prod _{u<n+1} \begin{pmatrix} a_{n+1,u}+t_u+t_{N+1-u}\\ t_u \end{pmatrix}_{v} \cdot \prod _{u> n+1} \begin{pmatrix} a_{n+1,u}+t_u \\ t_u\end{pmatrix}_{v} \cdot \prod _{i=0}^{t_{n+1}-1} L_i, \\ L_i & = \frac{(a_{n+1,n+1}+1+2i)_v+(1-\delta _{0,i}\delta _{0, \operatorname {ro}(\mathfrak{a})_{n+1} }) v^{a_{n+1,n+1}+2i}}{(i+1)_v}. \end{align}$$

Note that $L_i\in \mathcal{A}$ since $a_{n+1, n+1}$ is even, which in turn is due to the facts that $D$ is even and $a_{ij}= a_{N+1-i, N+1-j}$.

Proof.

The proofs of (a) and (b) are the same as the similar proof for Lemma 3.4 in BLM90.

We now show (c) by induction on $r$. We rewrite $\mathfrak{c}$ as $\mathfrak{c}_r$ to emphasize the dependence on $r$. Let $d_{t'} (\mathfrak{a})$ be the coefficient of $\mathfrak{a}(n, t')$ in the product $e_{\mathfrak{c}_{r}}e_{\mathfrak{a}}$. Let $\underline{p} \in \mathbb{N}^N$ be the vector whose $p$-th entry is 1 and 0 elsewhere. Statement (c) is reduced to showing that for any $t=(t_1,t_2,\dots , t_N)\in \mathbb{N}^N$ such that $\sum _ut_u=r+1$, we have

$$\begin{align} \sum _{t', \underline{p}} d_{t'} (\mathfrak{a}) \ d_{\underline{p}}(\mathfrak{a}(n, t')) = (r+1)_v \ d_t(\mathfrak{a}), {\tag{36}} \end{align}$$

where the sum runs over pairs $(t', \underline{p})$ such that $\sum t'_u= r$ and $t'+\underline{p}= t$.

We shall prove 36 by induction. When $r=0$, statement 36 holds automatically. We first deal with the case when $\operatorname {ro}(\mathfrak{a})_{n+1}>0$. By the induction assumption, we have

$$\begin{align*} \sum _{t', \underline{p}} d_{t'}(\mathfrak{a}) d_{\underline{p}}(\mathfrak{a}(n, t')) &= \sum _{t', \underline{p}} v^{2 \sum _{j<u}a_{n+1,j}t'_u + 2\sum _{N+1-j<u<j}t_j't_u' + \sum _{u>n+1}t_u'(t_u'-1)}\\ &\qquad {}\cdot \prod _{u<n+1} \begin{pmatrix} a_{n+1,u}+t'_u+t'_{N+1-u}\\ t'_u \end{pmatrix}_{v}\\ &\qquad {}\cdot \prod _{i=0}^{t'_{n+1}-1} \frac{(a_{n+1,n+1}+1+2i)_v+(1-\delta _{0,i}\delta _{0,\operatorname {ro}(\mathfrak{a})_{n+1}}) v^{a_{n+1,n+1}+2i}}{(i+1)_v}\\ &\qquad {}\cdot \prod _{u> n+1} \begin{pmatrix} a_{n+1,u}+t'_u \\ t'_u\end{pmatrix}_{v} (a_{n+1,p}+1+t_p'+t_{N+1-p}')_v \ v^{2\sum _{j<p}a_{n+1,j}+t_j'+t_{N+1-j}'}. \end{align*}$$

Since $t'+\underline{p}=t$, we have

$$\begin{equation*} t_i=t_i'+\delta _{ip}. \end{equation*}$$

We can compute the quotient $\sum _{t', \underline{p}} d_{t'}(\mathfrak{a}) d_{\underline{p}} (\mathfrak{a}(n, t')) / d_t(\mathfrak{a})$. We first calculate the power of $v^2$ for each $p$ in this quotient, which is

$$\begin{align*} \lefteqn{\sum _{j<u}a_{n+1,j}t'_u+\sum _{N+1-j<u<j}t_j't_u'+ \sum _{u>n+1}t_u'(t_u'-1)/2+\sum _{j<p}(a_{n+1,j}+t_j'+t_{N+1-j}')}\qquad \\ &\qquad {}-\sum _{j<u}a_{n+1,j}t_u- \sum _{N+1-j<u<j}t_jt_u- \sum _{u>n+1}t_u(t_u-1)/2\\ &= -\sum _{N+1-p<u<p}t_u-\sum _{N+1-j<p<j}t_j+\sum _{j<p}(t_j+t_{N+1-j})+ \begin{cases} -t_p, & \text{if $p>n+1$},\\ 0, & \text{otherwise}, \end{cases} \\[3.0pt] &=\sum _{j<p}t_j. \end{align*}$$

We then calculate the coefficients containing $v$-numbers for each $p$ in the above quotient, which can be broken into the following three cases. If $p<n+1$, then the coefficient involving $v$-numbers is

$$\begin{equation*} (a_{n+1,p}+t_p+t_{N+1-p})_v \cdot \begin{pmatrix} a_{n+1,u}+t_p+t_{N+1-p}-1 t_p-1 \end{pmatrix}_{v}/\begin{pmatrix} a_{n+1,u}+t_p+t_{N+1-p}\\ t_p \end{pmatrix}_{v}=(t_p)_v. \end{equation*}$$

If $p=n+1$, then the term is

$$\begin{multline*} \frac{(t_{n+1})_v}{(a_{n+1,n+1}+2t_{n+1}-1)_v+q^{1/2a_{n+1,n+1}+t_{n+1}}}\\ \cdot ((a_{n+1,n+1}+2t_{n+1}-1)_v+ v^{a_{n+1,n+1}+ 2 t_{n+1}})=(t_p)_v. \end{multline*}$$

If $p>n+1$, then the term is

$$\begin{multline*} \frac{\begin{pmatrix} a_{n+1,N+1-p}+t_p+t_{N+1-p}-1\\ t_{N+1-p} \end{pmatrix}_{v}}{\begin{pmatrix} a_{n+1,N+1-p}+t_p+t_{N+1-p}\\ t_{N+1-p} \end{pmatrix}_{v}} \frac{\begin{pmatrix} a_{n+1,p}+t_p-1 \\ t_p-1\end{pmatrix}_{v}}{\begin{pmatrix} a_{n+1,p}+t_p \\ t_p\end{pmatrix}_{v}}\\ \cdot (a_{n+1,p}+t_p+t_{N+1-p})_v=(t_p)_v. \end{multline*}$$

Summing up, we have

$$\begin{align*} \sum _{t', \underline{p}} d_{t'} (\mathfrak{a}) d_{\underline{p}} (\mathfrak{a}(n, t')) / d_t(\mathfrak{a})&=\sum _p v^{2\sum _{j<p}t_j} (t_p)_v\\ & =\frac{\sum _p (v^{2\sum _{j\leq p}t_j}-v^{2\sum _{j\leq p-1}t_j})}{v^2-1}=(r+1)_v. \end{align*}$$

This proves 36 under the assumption that $\operatorname {ro}(\mathfrak{a})_{n+1} > 0$. The proof of 36 for the case of $\operatorname {ro}(\mathfrak{a})_{n+1}=0$ is similar and skipped.

■

4.4. $\mathcal{S}$-action on $\mathcal{V}$

A degenerate version of Proposition 4.3.2 gives us an explicit description of the $\mathcal{S}$-action on $\mathcal{V}=\mathcal{A}_G(\mathscr{X}\times \mathscr{Y})$ as follows. For any $r_j\in [1,N]$, we denote $\check{r}_j=r_j+1$ and $\hat{r}_j=r_j-1$.

Corollary 4.4.1.

For any $1\leq i\leq n$, we have

$$\begin{align} E_i * e_{r_1\dots r_D} &=v^{-\sum _{1\leq j \leq D} \delta _{i+1,r_j}} \sum _{p,\ r_p=i} v^{2\sum _{j<p}\delta _{i+1,r_j}} \ e_{r_1\dots \check{r}_p\dots \hat{r}_{D+1-p} \dots r_{D}}, \\ F_i * e_{r_1\dots r_D} &=v^{-\sum _{1\leq j \leq D} \delta _{i,r_j}} \sum _{p,\ r_p=i+1} v^{2\sum _{j>p}\delta _{i, r_j}} e_{r_1\dots \hat{r}_p\dots \check{r}_{D+1-p} \dots r_{D}}, \\ H_i^{\pm 1} * e_{r_1\dots r_D} & = v^{\mp \sum _{1\leq j \leq D} \delta _{a, r_j} } e_{r_1\dots r_D}, \quad \text{and}\\ J_{+} * e_{r_1,\dots , r_D} &= \begin{cases} e_{r_1,\dots , r_D}, & r_i\neq n+1,\forall i, \#\{ j\in [1,d]| r_j\geq n+1\} \text{ is even},\\ 0, &\text{otherwise}, \end{cases}\\ J_{-} * e_{r_1,\dots , r_D} &= \begin{cases} e_{r_1,\dots , r_D}, & r_i\neq n+1,\forall i, \#\{ j\in [1,d]| r_j\geq n+1\} \text{ is odd},\\ 0, &\text{otherwise}, \end{cases} \\ J_{0} * e_{r_1,\dots , r_D} &= \begin{cases} e_{r_1,\dots , r_D}, & r_i= n+1,\ \text{for some} \ i,\\ 0, &\text{otherwise}. \end{cases} \end{align}$$

Proof.

Since the number of columns of the matrix associated to $e_{r_1\dots r_D}$ is $D=2d$, the second term in 32 disappears when we calculate the $E_n$ action on $e_{r_1\dots r_D}$. The first two identities follow directly from Proposition 4.3.2. The last four identities are straightforward.

■

4.5. Standard basis of $\mathcal{S}$

In this subsection, we assume that the ground field is an algebraic closure $\overline{\mathbb{F}}_q$ of $\mathbb{F}_q$ when we talk about the dimension of a $G$-orbit or its stabilizer. We set

$$\begin{equation*} d(\mathfrak{a})= \dim \mathcal{O}_{\mathfrak{a}} \quad \text{and}\quad r(\mathfrak{a})= \dim \mathcal{O}_{\mathfrak{b}}, \quad \forall \mathfrak{a}\in \Xi _{\mathbf{D}}, \end{equation*}$$

where $\mathfrak{b}=(b_{ij})^{\epsilon }$ is the signed diagonal matrix such that $b_{ii}=\sum _ka_{ik}$ and $\epsilon = \operatorname {sgn}(s_l(\mathfrak{a}), s_l(\mathfrak{a}))$. Denote by $\mathrm{C}_{G}(V,V')$ the stabilizer of $(V,V')$ in $G$.

Lemma 4.5.1.

We have

$$\begin{equation*} \begin{split} \dim \mathrm{C}_{G}(V,V') & =\frac{1}{2} \left( \sum _{i\geq k, j\geq l}a_{ij}a_{kl}-\sum _{i\geq n+1, j\geq n+1}a_{ij} \right), \quad \text{if}\ (V,V') \in \mathcal{O}_{\mathfrak{a}},\\ \dim \mathcal{O}_{\mathfrak{a}} &=\frac{1}{2} \left(\sum _{i<k\ \text{or}\ j<l}a_{ij}a_{kl}-\sum _{i<n+1 \ \text{or}\ j<n+1}a_{ij} \right),\\ d(\mathfrak{a})-r(\mathfrak{a}) &=\frac{1}{2} \left(\sum _{i\geq k, j<l}a_{ij}a_{kl}-\sum _{i\geq n+1>j}a_{ij} \right). \end{split} \end{equation*}$$

Notice that the above dimensions are independent of the sign of $\mathfrak{a}$.

Proof.

Let $Z_{ij}$ be subspaces of $\mathbb{F}_q^D$ with $\dim Z_{ij}=a_{ij}$ such that $V_r=\bigoplus _{i\leq r,j}Z_{ij}$, $V'_s=\bigoplus _{i, j\leq s}Z_{ij}$ for all $r,s\in [1,n]$, and $\mathbb{F}^D_q=\bigoplus _{i,j}Z_{ij}$. With respect to the decomposition, an endomorphism $T$ of $\mathbb{F}_q^D$ is determined by a family of linear maps $T_{(ij), (kl)}: Z_{ij}\rightarrow Z_{kl}$. Similar to BKLW14, 3.4, the Lie algebra of $\mathrm{C}_{G}(V,V')$ is the space of such $T$ satisfying the following conditions:

(a)

$T_{(ij),(kl)}\neq 0$ implies that $i\geq k$ and $j\geq l$;

(b)

$T_{(ij),(kl)}=-{}^tT_{(N+1-k,N+1-l),(N+1-i,N+1-j)}, \quad \forall i,j,k,l\in [1,N]$.

Note that $T_{(ij),(kl)}=-{}^tT_{(ij),(kl)}$ if and only if $i+k=N+1$ and $j+l=N+1$. In this case, $a_{ij}=a_{kl}$ and the dimension of such $T_{(ij),(kl)}$ is $\frac{1}{2}(a_{ij}a_{kl}-a_{ij})$, from which the first equality follows.

By using $\dim G=\frac{1}{2}D(D-1)=\frac{1}{2}(\sum _{i,j,k,l}a_{ij}a_{kl}-\sum _{i,j}a_{ij})$, we have the second equality. The third equality follows from the previous two equalities.

■

For any $\mathfrak{a}\in \Xi _{\mathbf{D}}$, let

$$\begin{equation*} [\mathfrak{a}]=v^{-(d(\mathfrak{a})-r(\mathfrak{a}))}e_{\mathfrak{a}}. \end{equation*}$$

We define a bar involution ‘$-$’ on $\mathcal{A}$ by $\bar{v}= v^{-1}$. By Lemma 4.5.1, Corollary 4.3.4 can be rewritten in the following form.

Corollary 4.5.2.

Suppose that $\mathfrak{a}=A^{\alpha }$, $\mathfrak{b}$, $\mathfrak{c}\in \Xi _{\mathbf{D}}$, $h\in [1, n]$, and $r\in \mathbb{N}$.

(a) If $\operatorname {co}(\mathfrak{b})= \operatorname {ro}(\mathfrak{a})$, $s_r(\mathfrak{b})=s_l(\mathfrak{a})$, and $\mathfrak{b}-rE_{h,h+1}^{\theta }$ is diagonal, then we have

$$\begin{align} [\mathfrak{b}] * [\mathfrak{a}] & =\sum _{t: \sum _{u=1}^N t_u=r} v^{\beta (t)}\prod _{u=1}^N \overline{\begin{pmatrix} a_{hu}+t_u\\ t_u \end{pmatrix}}_{v}\ [\mathfrak{a}_{t}], \ \text{where} {\tag{37}}\\ \beta (t) & = \sum _{j\geq l} a_{hj} t_l - \sum _{j>l} a_{h+1, j} t_l + \sum _{j<l} t_j t_l + \frac{1}{2}\delta _{hn}(\sum _{j+l<N+1}t_jt_l+\sum _{j<n+1}t_j), \\ \mathfrak{a}_t & =\left(A+\sum _{u=1}^Nt_u(E^{\theta }_{hu}-E^{\theta }_{h+1,u}),\operatorname {sgn}(s_l(\mathfrak{b}), s_r(\mathfrak{a})) \right) \in \Xi _{\mathbf{D}}. \end{align}$$

(b) If $h\neq n$, $\operatorname {co}(\mathfrak{c})= \operatorname {ro}(\mathfrak{a})$, $s_r(\mathfrak{c})=s_l(\mathfrak{a})$, and $\mathfrak{c}-rE_{h+1,h}^{\theta }$ is diagonal, then

$$\begin{align} [\mathfrak{c}] * [\mathfrak{a}] & =\sum _{t: \sum _{u=1}^N t_u=r}v^{\beta '(t)}\prod _{u=1}^N \overline{\begin{pmatrix} a_{h+1,u}+t_u\\ t_u\end{pmatrix}}_{v} \ [\mathfrak{a}(h, t) ], \ \text{where}{\tag{38}}\\ \beta '(t) & =\sum _{j\leq l} a_{h+1,j} t_l - \sum _{j<l} a_{hj} t_l + \sum _{j<l} t_j t_l, \\ \mathfrak{a}(h, t) &=\left(A-\sum _{u=1}^Nt_u(E^{\theta }_{hu}-E^{\theta }_{h+1,u}), \operatorname {sgn}(s_l(\mathfrak{c}), s_r(\mathfrak{a})) \right) \in \Xi _{\mathbf{D}}. \end{align}$$

(c) If the condition $h\neq n$ in (b) is replaced by $h=n$, then we have

$$\begin{align} [\mathfrak{c}] * [\mathfrak{a}] &= \sum _{t: \sum _{u=1}^N t_u=r}v^{\beta ''(t)} \ \overline{\mathcal{G}}\ [\mathfrak{a}(n,t)], \ \text{where}{\tag{39}}\\ \beta ''(t)&=\sum _{j\leq l}a_{n+1,j}t_l-\sum _{j<l} a_{nj} t_l\\ &\quad +\sum _{j<l, j+l\geq N+1}t_jt_l + r(r-1)/2-\sum _{j<n+1}t_j (t_j-1) / 2, \\ \mathcal{G} &= \prod _{u<n+1} \begin{pmatrix} a_{n+1,u}+t_u+t_{N+1-u}\\ t_u \end{pmatrix}_{v} \cdot \prod _{u> n+1} \begin{pmatrix} a_{n+1,u}+t_u \\ t_u\end{pmatrix}_{v} \cdot \prod _{i=0}^{t_{n+1}-1} L_i, \\ L_i & = \frac{(a_{n+1,n+1}+1+2i)_v+(1-\delta _{0,i}\delta _{0, \operatorname {ro}(\mathfrak{a})_{n+1} }) v^{a_{n+1,n+1}+2i}}{(i+1)_v}. \end{align}$$

Proof.

The proof involves lengthy mechanical computations. We shall sketch a proof of part (a) for the sake of completeness and to give a flavor of how the proofs for the rest shall go. By the definition of $[\mathfrak{a}]$ and Corollary 4.3.4(a), we have

$$\begin{equation*} [\mathfrak{b}]* [\mathfrak{a}] = \sum _{t: \sum _{u=1}^N t_u=r} v^{\tilde{\beta }(t)}\prod _{u=1}^N \overline{\begin{pmatrix} a_{hu}+t_u\\ t_u \end{pmatrix}}_{v}\ [\mathfrak{a}_{t}], \end{equation*}$$

where $\tilde{\beta }(t) = d(\mathfrak{a}_t) - r(\mathfrak{a}_t) -d(\mathfrak{a}) +r(\mathfrak{a}) - d(\mathfrak{b}) + r(\mathfrak{b}) + 2 \sum _{j \geq u} a_{hj} t_u$. Let $a_{ij}'$ be the $(i,j)$-th entry of $\mathfrak{a}_t$. By using $a_{ij}' = a_{ij} + (\delta _{hi}-\delta _{h+1,i}) t_j + (\delta _{N+1-h,i} -\delta _{N-h,i}) t_{N+1-j}$ and Lemma 4.5.1, we have

$$\begin{equation*} \begin{split} d(\mathfrak{b})-r(\mathfrak{b}) &= \sum _{j,u} a_{hj} t_u, \\ d(\mathfrak{a}_t) - r(\mathfrak{a}_t) &=\frac{1}{2} ( \sum _{i \geq k, j<l} a_{ij}a_{kl} + 2 \sum _{j<l} a_{hj} t_l -2 \sum _{j>l} a_{h+1,j}t_l + 2 \sum _{j<l} t_jt_l \\ &- \sum _{i \geq n+1 >j} a_{ij}- \sum _{j<n+1} \delta _{hn} t_j + \sum _{j+l<N+1} \delta _{hn}t_jt_l). \end{split} \end{equation*}$$

Combining all these together, we see that the coefficient $\tilde{\beta }(t)$ is the same as $\beta (t)$ in 37. This proves (a). The proofs of (b) and (c) are similar.

■

4.6. Generators of $\mathcal{S}$

Define a partial order “$\leq$” on $\Xi _{\mathbf{D}}$ by $\mathfrak{a}\leq \mathfrak{b}$ if $\mathcal{O}_{\mathfrak{a}} \subset \overline{\mathcal{O}}_{\mathfrak{b}}$. For any $\mathfrak{a}=(a_{ij})^{\alpha }$ and $\mathfrak{b}=(b_{ij})^{\epsilon }$ in $\Xi _{\mathbf{D}}$, we say that $\mathfrak{a}\preceq \mathfrak{b}$ if and only if $\alpha =\epsilon$ and the following two conditions hold:

$$\begin{align} \sum _{r\leq i, s\geq j} a_{rs} & \leq \sum _{r\leq i, s\geq j} b_{rs}, \quad \forall i<j.{\tag{40}}\\ \sum _{\substack{r\leq n,\\s>N+1-j}} a_{rs} & \equiv \sum _{\substack{r\leq n,\\s>N+1-j}} b_{rs} \ \bmod 2, \quad \text{if}\tag{41}\\ \operatorname {ro}(\mathfrak{a})_{n+1} & = \operatorname {co}(\mathfrak{a})_{n+1} = 0, \quad \operatorname {ro}(\mathfrak{b})_{n+1} = \operatorname {co}(\mathfrak{b})_{n+1} =0, \\ \sum _{\substack{i\leq r\leq N+1-i,\\ j\leq s\leq N+1-j}} a_{rs} & = \sum _{\substack{i\leq r\leq N+1-i,\\ j\leq s\leq N+1-j}} b_{rs} = 0, \quad \text{and}\\ \sum _{\substack{r<i,\\s>N+1-j}} a_{rs} &= \sum _{\substack{r<i,\\s>N+1-j}} b_{rs}, \quad \forall i, j\in [1,n]. \end{align}$$

The relation “$\preceq$” defines a second partial order on $\Xi _{\mathbf{D}}$. We say that $\mathfrak{a}\prec \mathfrak{b}$ if $\mathfrak{a}\preceq \mathfrak{b}$ and at least one of the inequalities in 40 is strict. By BB05, Theorem 8.2.8 and BKLW14, Lemma 3.8, we have the following lemma.

Lemma 4.6.1.

$\mathfrak{a}\leq \mathfrak{b}$ if and only if $\mathfrak{a}\preceq \mathfrak{b}$ for any $\mathfrak{a}, \mathfrak{b}\in \Xi _{\mathbf{D}}$.

We shall denote by “$[ \mathfrak{m}]$+ lower terms” an element in $\mathcal{S}$ which is equal to $[\mathfrak{m}]$ plus a linear combination of $[\mathfrak{m}']$ with $\mathfrak{m}' \prec \mathfrak{m}$. By Corollary 4.5.2, we have

Corollary 4.6.2.

Fix integers $r > 0$, $c>0$ with $c$ even, and $h\in [1,n]$.

(a) Assume that $\mathfrak{a}=(a_{ij})^{\alpha } \in \Xi _{\mathbf{D}}$ satisfies one of the following two conditions:

(1)

$a_{hj}=0$, $\forall j\geq k$, $a_{h+1,k}=r$, $a_{h+1,j}=0$, $\forall j>k$, if $h<n$;

(2)

$a_{nj}=0$, $\forall j\geq k$, $a_{n+1,k}=r+(r+c)\delta _{n+1,k}$, $a_{n+1,j}=0$, $\forall j>k$, if $h=n$, $k\geq n+1$.

If $\mathfrak{b}$ is subject to $\mathfrak{b}-rE_{h,h+1}^{\theta }$, which is diagonal, $s_r(\mathfrak{b})=s_l(\mathfrak{a})$, and $\operatorname {co}(\mathfrak{b})=\operatorname {ro}(\mathfrak{a})$, then

$$\begin{equation*} [\mathfrak{b}] * [\mathfrak{a}]=[\mathfrak{a}_{t(k)}] +{\text{ lower terms}},\quad \text{where} \quad t(k)_u = r\delta _{u, k} . \end{equation*}$$

(b) Assume that $\mathfrak{a}=(a_{ij})^{\alpha } \in \Xi _{\mathbf{D}}$ satisfies one of the following conditions:

(1)

$a_{hj}=0$, $\forall j< k$, $a_{hk}=r$, $a_{h+1,j}=0$, $\forall j\leq k$, if $h<n$, or

(2)

$a_{nj}=0$, $\forall j< k$, $a_{nk}=r$, $a_{n+1,j}=0$, $\forall j\leq k$, if $h=n$, $k\leq n$.

If $\mathfrak{c}$ satisfies that $\mathfrak{c}-rE_{h+1,h}^{\theta }$ is diagonal, $s_r(\mathfrak{c})=s_l(\mathfrak{a})$, and $\operatorname {co}(\mathfrak{c})=\operatorname {ro}(\mathfrak{a})$, then

$$\begin{equation*} [\mathfrak{c}] * [\mathfrak{a}]=[ \mathfrak{a}(h, t(k)) ]+{\text{ lower terms}}. \end{equation*}$$

We define an order on $\mathbb{N}\times \mathbb{N}$ by

$$\begin{equation} (i,j)<(i',j')\quad \text{if and only if $j'-i'<j-i$ or $j'-i'=j-i$, $i'<i$}. {\tag{42}} \end{equation}$$

By using Corollary 4.6.2, we are able to prove the following theorem.

Theorem 4.6.3.

For any $\mathfrak{a}=(a_{ij})^{\alpha } \in \Xi _{\mathbf{D}}$, we set $R_{ij}=\sum _{k=1}^ia_{kj}$. There exist signed matrices $\mathfrak{m}(i, j)$ such that $\mathfrak{m}(i, j) - R_{ij}E^{\theta }_{i,i+1}$ is diagonal and

$$\begin{equation*} \prod _{1\leq i<j \leq N} [\mathfrak{m}(i, j) ] = [\mathfrak{a}]+{\text{ lower terms}}, \end{equation*}$$

where the product is taken in the order 42. The product has $N(N-1)/2$ terms.

Proof.

For any $A = (a_{ij})$, we denote $\lambda (A) = \min \{k \geq 0|a_{ij}=0\ \text{for all $|i-j| > k$}\}$. Moreover, we set $\lambda (\mathfrak{a}) = \lambda (A)$ if $\mathfrak{a}=(A, \alpha )$. We shall prove the theorem by induction on $\lambda (\mathfrak{a})$. If $\lambda (\mathfrak{a}) =0$, then $\mathfrak{a}$ is a diagonal signed matrix. There is nothing to show.

Assume that the theorem is true for all $\mathfrak{a}$ with $\lambda (\mathfrak{a}) \leq k$. We shall show that the theorem holds for all $\mathfrak{a}$ with $\lambda (\mathfrak{a}) =k + 1$.

Given $\mathfrak{a}=(A, \alpha )$ with $\lambda (\mathfrak{a}) =k+1$, we define a matrix $B_0 = (b_{ij}) \in \Xi _D$ by

$$\begin{equation*} b_{ij}= \begin{cases} a_{ij}, &\text{if}\ 0\leq j-i < k,\\ R_{ij}, &\text{if}\ j-i = k,\\ 0, & \text{if} \ j-i > k. \end{cases} \end{equation*}$$

Note that the lower triangle part of $B_0$ is uniquely determined by the symmetry, i.e., $b_{ij} = b_{N+1-i, N+1-j}$. It is clear that $\lambda (B_0) = k$ from the definition.

For $i=1$, …, $N-k-1$, we define $B_i$ inductively by the conditions that $B_{i}-a_{i,i+k+1}E^{\theta }_{i,i+1}$ is a diagonal matrix and $\operatorname {co}(B_{i})= \operatorname {ro}(B_{i-1})$. We set

$$\begin{equation*} \mathfrak{b}_{i} = (B_{i}, \operatorname {sgn}( s_l(\mathfrak{b}_{i}), s_r(\mathfrak{b}_{i})) ),\ i=0,1,\dots , N-k-1, \end{equation*}$$

where $s_r(\mathfrak{b}_0) = s_r(\mathfrak{a})$, $s_l(\mathfrak{b}_0)$, $s_r(\mathfrak{b}_{i})$, and $s_l(\mathfrak{b}_i)$ are defined as follows.

If $k > n$, we set $s_l(\mathfrak{b}_i) = s_r(\mathfrak{b}_i) = s_l(\mathfrak{b}_0) = s_l(\mathfrak{a})$ for all $i=1$, …, $N-k-1$.

If $k = n$, we set $s_l(\mathfrak{b}_n) = s_l(\mathfrak{a})$ and

$$\begin{equation*} \begin{cases} s_r(\mathfrak{b}_n) = s_l(\mathfrak{b}_i) = s_r(\mathfrak{b}_i) = s_l(\mathfrak{b}_0) =1,\ 1 \leq i < n, & \text{if}\ a_{n,N} \neq 0,\\ s_r(\mathfrak{b}_n) = s_l(\mathfrak{b}_i) = s_r(\mathfrak{b}_i) = s_l(\mathfrak{b}_0) = s_l(\mathfrak{a}),\ 1\leq i < n, & \text{if}\ a_{n,N} = 0. \end{cases} \end{equation*}$$

If $k < n$, we set $s_l(\mathfrak{b}_{n+1}) = s_r(\mathfrak{b}_{i}) = s_l(\mathfrak{b}_{i}) = s_l(\mathfrak{a})$ for all $i > n+1$ and

$$\begin{equation*} \begin{cases} s_r(\mathfrak{b}_{n}) = s_l(\mathfrak{b}_i)= s_r(\mathfrak{b}_i) = s_l(\mathfrak{b}_0) =1, 1 \leq i < n, & \text{if}\ a_{n,n+k+1}\neq 0,\\ s_r(\mathfrak{b}_{n}) = s_l(\mathfrak{b}_i)= s_r(\mathfrak{b}_i) = s_l(\mathfrak{b}_0) = s_r(\mathfrak{b}_{n+1}),\ 1\leq i < n, & \text{if}\ a_{n,n+k+1} = 0, \end{cases} \end{equation*}$$

where $s_r(\mathfrak{b}_{n+1}) = s_l(\mathfrak{b}_n)$ satisfies

$$\begin{equation*} \begin{cases} s_l(\mathfrak{b}_n) = 1 & \text{if}\ \operatorname {ro}(A)_{n+1} -2a_{n+1,n+k+2} \neq 0,\\ s_l(\mathfrak{b}_n) = 2 & \text{if}\ \operatorname {ro}(A)_{n+1} -2a_{n+1,n+k+2} = 0. \end{cases} \end{equation*}$$

By Corollary 4.6.2, we have

$$\begin{equation*} [\mathfrak{b}_{N-k-1}] * \dots * [\mathfrak{b}_{1}] * [\mathfrak{b}_{0}] =[\mathfrak{a}]+{\text{lower terms}}. \end{equation*}$$

By the induction assumption, the theorem follows.

■

We have immediately

Corollary 4.6.4.

The products $\mathfrak{m}_{\mathfrak{a}}=\prod _{1\leq i<j \leq N} [\mathfrak{m}(i, j) ]$ for any $\mathfrak{a}\in \Xi _{\mathbf{D}}$ in Theorem 4.6.3 form a basis for $\mathcal{S}$. (It is called a monomial basis of $\mathcal{S}$.)

By 37, 38, 39, and Corollary 4.6.4, we have

Corollary 4.6.5.

The algebra $\mathcal{S}$ (resp. $\mathbb{Q}(v) \otimes _{\mathcal{A}} \mathcal{S}$) is generated by the elements $[\mathfrak{e}]$ such that $\mathfrak{e} - RE^{\theta }_{i, i+1}$ (resp. either $\mathfrak{e}$ or $\mathfrak{e}-E^{\theta }_{i, i+1}$) is diagonal for some $R\in \mathbb{N}$ and $i\in [1, N-1]$.

Observe that

$$\begin{equation*} E_i =\sum [\mathfrak{c}], \ F_i = \sum [\mathfrak{b}], \ H^{\pm 1}_a =\sum v^{\mp d_a} [\mathfrak{d}],\quad \forall i\in [1, n], a\in [1, n+1], \end{equation*}$$

where $\mathfrak{b}$, $\mathfrak{c}$, and $\mathfrak{d}$ run over all signed matrices in $\Xi _{\mathbf{D}}$ such that $\mathfrak{b}-E^{\theta }_{i, i+1}$, $\mathfrak{c}-E^{\theta }_{i+1, i}$, and $\mathfrak{d}$ are diagonal, respectively, and $d_a$ is the $(a, a)$-entry of the matrix in $\mathfrak{d}$. We have the following corollary by Corollary 4.6.5.

Corollary 4.6.6.

The algebra $\mathbb{Q}(v) \otimes _{\mathcal{A}} \mathcal{S}$ is generated by the functions $E_i$, $F_i$, $H_a^{\pm 1}$, $J_{\alpha }$ for any $i\in [1, n]$, $a\in [1, n+1]$, and $\alpha \in \{ +, 0, -\}$.

Remark 4.6.7.

The order 42 in Theorem 4.6.3 is different from the orders in BKLW14, Theorem 3.6.1 and BLM90, 3.9. It can be shown that using the latter orders, one can construct a different monomial basis for the algebra $\mathcal{S}$.

4.7. Canonical basis of $\mathcal{S}$

In this subsection, we assume that the ground field is an algebraic closure $\overline{\mathbb{F}}_q$ of the finite field $\mathbb{F}_q$. Let $IC_{\mathfrak{a}}$ be the intersection cohomology complex of $\overline{\mathcal{O}}_{\mathfrak{a}}$, normalized so that the restriction of $IC_{\mathfrak{a}}$ to $\mathcal{O}_{\mathfrak{a}}$ is the constant sheaf on $\mathcal{O}_{\mathfrak{a}}$. Since $IC_{\mathfrak{a}}$ is a $G$-equivariant complex and the stabilizers of the points in $\overline{\mathcal{O}}_{\mathfrak{a}}$ are connected, the restriction of the $i$-th cohomology sheaf $\mathscr{H}_{\mathcal{O}_{\mathfrak{b}}}^i(IC_{\mathfrak{a}})$ of $IC_{\mathfrak{a}}$ to $\mathcal{O}_{\mathfrak{b}}$ for $\mathfrak{b}\leq \mathfrak{a}$ is a trivial local system. We denote by $n_{\mathfrak{b},\mathfrak{a},i}$ the rank of this local system. We set

$$\begin{equation} \{\mathfrak{a}\}=\sum _{\mathfrak{b}\leq \mathfrak{a}} P_{\mathfrak{b},\mathfrak{a}}[\mathfrak{b}],\quad \text{where}\quad P_{\mathfrak{b},\mathfrak{a}}=\sum _{i\in \mathbb{Z}} n_{\mathfrak{b}, \mathfrak{a},i}v^{i-d(\mathfrak{a})+d(\mathfrak{b})}. {\tag{43}} \end{equation}$$

The polynomials $P_{\mathfrak{b}, \mathfrak{a}}$ satisfy

$$\begin{equation} P_{\mathfrak{a},\mathfrak{a}}=1\quad \text{and}\quad P_{\mathfrak{b},\mathfrak{a}}\in v^{-1}\mathbb{Z}[v^{-1}] \quad \text{for any $\mathfrak{b}<\mathfrak{a}$}. {\tag{44}} \end{equation}$$

Since $\{[\mathfrak{a}]| \mathfrak{a}\in \Xi _{\mathbf{D}}\}$ is an $\mathcal{A}$-basis of $\mathcal{S}$, by 43 and 44, we have

Lemma 4.7.1.

The set $\{\{\mathfrak{a}\}| \mathfrak{a}\in \Xi _{\mathbf{D}}\}$ forms an $\mathcal{A}$-basis of $\mathcal{S}$, called the $canonical$ $basis$.

By the sheaf-function principle, we have

Corollary 4.7.2.

The structure constants of $\mathcal{S}$ with respect to the canonical basis $\{\{\mathfrak{a}\}| \mathfrak{a}\in \Xi _{\mathbf{D}}\}$ are in $\mathbb{N}[v, v^{-1}]$.

4.8. Inner product on $\mathcal{S}$

We shall define an inner product on $\mathcal{S}$ following M10, Section 3 and BKLW14, 3.7. Since the arguments and statements are very similar, we shall be sketchy.

Denote by ${}^t\!A$ the transposition matrix of $A$. For any signed matrix $\mathfrak{a}=(A,\epsilon )$, we define ${}^t\! \mathfrak{a}= ({}^t\!A, \epsilon ')$, where

$$\begin{equation*} \epsilon '= \begin{cases} \epsilon , & \text{if} \sup (\mathfrak{a}) \neq (2, 3), (3,2),\\ -\epsilon , & \text{otherwise}. \end{cases} \end{equation*}$$

For any $\mathfrak{a}\in \Xi _{\mathbf{D}}$, we set

$$\begin{equation} d_{\mathfrak{a}}=d(\mathfrak{a})-r(\mathfrak{a})\quad \text{and}\quad \mathscr{X}^V_{\mathfrak{a}}=\{V'\in \mathscr{X}|(V,V')\in \mathcal{O}_{\mathfrak{a}}\}. \tag{45} \end{equation}$$

We define a bilinear form

$$\begin{equation*} (- , - )_D: \mathcal{S}\times \mathcal{S}\rightarrow \mathcal{A} \end{equation*}$$

by

$$\begin{equation*} (f_1, f_2)_D=\sum _{V,V' \in \mathscr{X}}v^{\sum _i|V_i/V_{i-1}|^2-\sum _i|V'_i/V'_{i-1}|^2}f_1(V,V') f_2(V,V'),\quad \forall f_1, f_2\in \mathcal{S}. \end{equation*}$$

In particular,

$$\begin{equation*} (e_{\mathfrak{a}},e_{\mathfrak{b}})_D=\delta _{\mathfrak{a},\mathfrak{b}}v^{2(d_{\mathfrak{a}}-d_{^t\mathfrak{a}})} \# \mathscr{X}^V_{{}^t\!\mathfrak{a}},\quad \forall \mathfrak{a},\mathfrak{b}\in \Xi _{\mathbf{D}}, \end{equation*}$$

where $V$ is any element in $\mathscr{X}$ such that $|V_i/V_{i-1}|=\operatorname {co}(\mathfrak{a})_i$. By the definition of $d_{\mathfrak{a}}$ and Lemma 4.5.1, we have

$$\begin{equation*} d_{\mathfrak{a}}-d_{{}^t\!\mathfrak{a}}=\frac{1}{4} \sum _i( \operatorname {ro}(\mathfrak{a})_i^2-\operatorname {co}(\mathfrak{a})_i^2)-\frac{1}{4}( \operatorname {ro}(\mathfrak{a})_{n+1}-\operatorname {co}(\mathfrak{a})_{n+1}). \end{equation*}$$

This implies that

$$\begin{equation} d_{\mathfrak{a}}-d_{{}^t\!\mathfrak{a}}+d_{\mathfrak{b}}-d_{{}^t\!\mathfrak{b}}=d_{\mathfrak{c}}-d_{{}^t\!\mathfrak{c}} {\tag{46}} \end{equation}$$

if $\operatorname {ro}(\mathfrak{a})= \operatorname {ro}(\mathfrak{c})$, $\operatorname {co}(\mathfrak{a})= \operatorname {ro}(\mathfrak{b})$, and $\operatorname {co}(\mathfrak{b})=\operatorname {co}(\mathfrak{c})$. By using 46 and the same argument as the one proving Proposition 3.2 in M10, we have the following proposition.

Proposition 4.8.1.

For any $\mathfrak{a}, \mathfrak{b}, \mathfrak{c}\in \Xi _{\mathbf{D}}$, we have

$$\begin{equation*} ([\mathfrak{a}]e_{\mathfrak{b}}, e_{\mathfrak{c}})_D=v^{d_{\mathfrak{a}} -d_{^t\mathfrak{a}}}(e_{\mathfrak{b}}, [{}^t\!\mathfrak{a}]e_{\mathfrak{c}} )_D. \end{equation*}$$

Moreover, the following proposition holds from Proposition 4.8.1.

Proposition 4.8.2.

For any $\mathfrak{b}, \mathfrak{c}\in \Xi _{\mathbf{D}}$, we have

$$\begin{equation*} ([\mathfrak{b}],[\mathfrak{c}])_D \in \delta _{\mathfrak{b}, \mathfrak{c}}+v^{-1}\mathbb{Z}[v^{-1}] \quad \text{and} \quad (x \mathfrak{b}, \mathfrak{c})_D =(\mathfrak{b}, \rho (x) \mathfrak{c})_D, \quad \forall x\in \mathcal{S}, \end{equation*}$$

where $\rho$ is defined in 18.

We define a bar involution $\bar{}: \mathcal{S}\to \mathcal{S}$ by

$$\begin{equation*} \bar{v} =v^{-1} \quad \text{and} \quad \overline{[\mathfrak{e}]}=[\mathfrak{e}], \end{equation*}$$

for any $\mathfrak{e}$ in $\Xi _{\mathbf{D}}$ such that $\mathfrak{e}-RE^{\theta }_{i, i+1}$ for some $R\in \mathbb{N}$ and $i \in [1, N-1]$. By an argument similar to BLM90 and BKLW14, we have $\overline{\{\mathfrak{a}\}} =\{\mathfrak{a}\}$. By Proposition 4.8.2 and this observation, we have

Corollary 4.8.3.

The canonical basis $\{ \{\mathfrak{a}\}| \mathfrak{a}\in \Xi _{\mathbf{D}} \}$ of $\mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{S}$ is characterized up to sign by the properties

$$\begin{equation*} \{\mathfrak{a}\} \in \mathcal{S}, \quad \overline{\{\mathfrak{a}\}} =\{\mathfrak{a}\}, \quad \text{and} \quad ( \{ \mathfrak{a}\} , \{ \mathfrak{a}' \})_D \in \delta _{\mathfrak{a}, \mathfrak{a}'} +v^{-1}\mathbb{Z}[v^{-1}]. \end{equation*}$$

5. The limit algebra $\mathcal{K}$ and its canonical basis

We shall apply the stabilization process to the algebras $\mathcal{S}$ in 5 as $D$ goes to $\infty$, following BLM90. We write $\mathcal{S}_D$ to emphasize the dependence on $D$, and $\Xi _{\mathbf{D}}(D)$ for the set $\Xi _{\mathbf{D}}$ in 23 for the same reason.

5.1. Stabilization

Let $I'=I-E_{n+1,n+1}$, where $I$ is the identity matrix. We set

$$\begin{equation*} {}_pA=A+2pI' \quad \text{and} \quad {}_p\mathfrak{a}=( {}_pA, \alpha ), \quad \text{if}\ \mathfrak{a}=(A,\alpha ). \end{equation*}$$

Let

$$\begin{align} \widetilde{\Xi }_{\mathbf{D}} =\{\mathfrak{a}=(A, \alpha ) \in \mathrm{Mat}_{N\times N}(\mathbb{Z})\times \{+, 0, -\} | {}_p \mathfrak{a}\in \Xi _{\mathbf{D}}(D) \ \text{for some}\ p\in \mathbb{Z}, D\in \mathbb{N}\}. {\tag{47}} \end{align}$$

For any matrix $\mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}}$, the notation introduced in 28 and 29 is still well defined and will be used freely in the following. Moreover, we observe that $\operatorname {sgn}(\mathfrak{a})=\operatorname {sgn}({}_p\mathfrak{a})$. Let

$$\begin{equation*} \mathcal{K}= \operatorname {span}_{\mathcal{A}} \{ [\mathfrak{a}] | \mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}} \}, \end{equation*}$$

where the notation $[\mathfrak{a}]$ is a formal symbol bearing no geometric meaning. Let $v'$ be a second indeterminate and

$$\begin{align} \mathfrak{R}=\mathbb{Q}(v)[v', v'^{-1}]. {\tag{48}} \end{align}$$

We have

Proposition 5.1.1.

Suppose that $\mathfrak{a}_1$, $\mathfrak{a}_2$, …, $\mathfrak{a}_r$ ($r\geq 2$) are signed matrices in $\widetilde{\Xi }_{\mathbf{D}}$ such that $\operatorname {co}(\mathfrak{a}_i)= \operatorname {ro}(\mathfrak{a}_{i+1})$ and $s_r(\mathfrak{a}_i)=s_l(\mathfrak{a}_{i+1})$ for $1\leq i \leq r-1$. There exist $\mathfrak{z}_1$, …, $\mathfrak{z}_m\in \widetilde{\Xi }_{\mathbf{D}}$, $G_j(v,v')\in \mathfrak{R}$, and $p_0\in \mathbb{N}$ such that in $\mathcal{S}_D$ for some $D$, we have

$$\begin{equation*} [{}_p \mathfrak{a}_1] * [{}_p\mathfrak{a}_2] * \dots *[{}_p \mathfrak{a}_r]=\sum _{j=1}^mG_j(v,v^{-p})[{}_p \mathfrak{z}_j],\quad \forall p \in \mathbb{N}, p\geq p_0. \end{equation*}$$

Moreover, $\operatorname {sgn}( {}_p\mathfrak{z}_j ) =\operatorname {sgn}(s_l ({}_p\mathfrak{a}_1), s_r({}_p\mathfrak{a}_r))$.

Proof.

The proof is essentially the same as the one for Proposition 4.2 in BLM90 by using Corollary 4.5.2 and Theorem 4.6.3. The main difference is that when $h=n$, the twists $\beta (t)$ and $\beta ''(t)$ in 37 and 39, respectively, change when $\mathfrak{a}$ is replaced by ${}_p \mathfrak{a}$. To remedy this difference, we adjust these two twists as follows:

$$\begin{equation*} \gamma (t)=\beta (t)- a_{nn} \sum _{l\leq n} t_l\quad \text{and}\quad \gamma ''(t)=\beta ''(t)+ a_{nn} \sum _{n<l}t_l,\quad \text{if}\ h=n. \end{equation*}$$

Then the new twists $\gamma (t)$ and $\gamma ''(t)$ remain the same when $\mathfrak{a}$ is replaced by ${}_p\mathfrak{a}$. For example, when $r=2$ and $\mathfrak{a}_1$ is chosen such that $\mathfrak{a}_1-RE_{n,n+1}^{\theta }$ is a diagonal with $R\in \mathbb{N}$, the structure constant $G_{t}(v,v')$ is defined by

$$\begin{equation*} \begin{split} G_{t}(v,v')&=v^{\gamma (t)}\prod _{\substack{1\leq u\leq N\\ u\neq n}}\overline{\begin{pmatrix} a_{nu}+t_u\\ t_u\end{pmatrix}}_{v}\\ &\quad \cdot \prod _{1\leq i\leq t_n}\frac{v^{-2(a_{nn}+t_n-i+1)}v'^2-1}{v^{-2i}-1} v^{\sum _{l\leq n}a_{nn}t_l}v'^{-\sum _{l\leq n}t_l}. \end{split} \end{equation*}$$

Similarly, if $r=2$ and $\mathfrak{a}_1$ is chosen such that $\mathfrak{a}_1-RE_{n+1,n}^{\theta }$ is diagonal with $R\in \mathbb{N}$, the structure constant $G_{t}(v,v')$ is defined by

$$\begin{equation*} \begin{split} G_{t}(v,v')&=v^{\gamma ''(t)}v^{-\sum _{n<l}a_{nn}t_l} \prod _{1\leq u\leq N,u\neq n+1}\overline{\begin{pmatrix} a_{n+1,u}+t_u\\ t_u\end{pmatrix}}_{v}\\ &\quad \cdot \prod _{1\leq i\leq t_{n+1}}\frac{v^{-2(a_{n+1,n+1}+t_{n+1}-i+1)}v'^2-1}{v^{-2i}-1} \cdot v'^{\sum _{n<l}t_l}. \end{split} \end{equation*}$$

For the case when $\mathfrak{a}_1$ is chosen such that $\mathfrak{a}_1- R E^{\theta }_{h, h+1}$ or $\mathfrak{a}_1-RE^{\theta }_{h+1, h}$ is diagonal for some $h<n$, then the structural constant $G_t(v,v')$ is defined similarly as in the proof of Proposition 4.2 in BLM90, i.e.,

$$\begin{equation*} G_{t}(v,v')=v^{\beta (t)}\prod _{\substack{1\leq u\leq N\\ u\neq h}}\overline{\begin{pmatrix} a_{hu}+t_u\\ t_u\end{pmatrix}}_{v}\prod _{1\leq i\leq t_h}\frac{v^{-2(a_{hh}+t_h-i+1)}v'^2-1}{v^{-2i}-1}, \end{equation*}$$

for $\mathfrak{a}_1$ such that $\mathfrak{a}_1- R E^{\theta }_{h, h+1}$ is diagonal for some $h<n$, and

$$\begin{equation*} G_{t}(v,v')=v^{\beta '(t)} \prod _{\substack{1\leq u\leq N\\ u\neq h+1}} \overline{\begin{pmatrix} a_{h+1,u}+t_u\\ t_u\end{pmatrix}}_{v} \prod _{1\leq i\leq t_{h+1}}\frac{v^{-2(a_{h+1,h+1}+t_{h+1}-i+1)}v'^2-1}{v^{-2i}-1}, \end{equation*}$$

for $\mathfrak{a}_1$ such that $\mathfrak{a}_1- R E^{\theta }_{h+1, h}$ is diagonal for some $h<n$. Bearing in mind the above modifications, the rest of the proof for Proposition 4.2 in BLM90 can be repeated here.

■

By specialization $v'$ at $v'=1$, we have

Corollary 5.1.2.

There is a unique associative $\mathcal{A}$-algebra structure on $\mathcal{K}$, without unit, where the product is given by

$$\begin{equation*} [\mathfrak{a}_1] \cdot [\mathfrak{a}_2]\cdot \dots \cdot [\mathfrak{a}_r] =\sum _{j=1}^m G_j(v,1)[\mathfrak{z}_j], \end{equation*}$$

if $\mathfrak{a}_1$, …, $\mathfrak{a}_r$ are as in Proposition 5.1.1.

By Corollary 5.1.2 and comparing the $G_t(v,1)$’s with 37, 38, and 39, the structure of $\mathcal{K}$ can be determined by the following multiplication formulas. Recall the notation from 28.

Let $\mathfrak{a}$ and $\mathfrak{b}\in \widetilde{\Xi }_{\mathbf{D}}$ be chosen such that $\mathfrak{b}-rE_{h,h+1}^{\theta }$ is diagonal for some $1\leq h\leq n, r\in \mathbb{N}$ satisfying $\operatorname {co}(\mathfrak{b})= \operatorname {ro}(\mathfrak{a})$ and $s_r(\mathfrak{b})=s_l(\mathfrak{a})$. Then we have

$$\begin{equation} [ \mathfrak{b}] \cdot [\mathfrak{a}] =\sum _{t} v^{\beta (t)}\prod _{u=1}^N \overline{\begin{pmatrix} a_{hu}+t_u\\ t_u \end{pmatrix}}_{v}\ [\mathfrak{a}_{t}], {\tag{49}} \end{equation}$$

where the sum is taken over all $t=(t_u)\in \mathbb{N}^N$ such that $\sum _{u=1}^Nt_u=r$, $\beta (t)$ is defined in 37, and $\mathfrak{a}_{t} \in \widetilde{\Xi }_{\mathbf{D}}$ is in 33.

Similarly, if $\mathfrak{a}, \mathfrak{c}\in \widetilde{\Xi }_{\mathbf{D}}$ are chosen such that $\mathfrak{c}-rE_{h+1,h}^{\theta }$ is diagonal for some $1\leq h< n, r\in \mathbb{N}$ satisfying $\operatorname {co}(\mathfrak{c})= \operatorname {ro}(\mathfrak{a})$ and $s_r(\mathfrak{c})=s_l(\mathfrak{a})$, then we have

$$\begin{align} [\mathfrak{c}] \cdot [\mathfrak{a}] =\sum _{t}v^{\beta '(t)}\prod _{u=1}^N \overline{\begin{pmatrix} a_{h+1,u}+t_u\\ t_u\end{pmatrix}}_{v} \ [\mathfrak{a}(h, t) ], {\tag{50}} \end{align}$$

where the sum is taken over all $t=(t_u)\in \mathbb{N}^N$ such that $\sum _{u=1}^Nt_u=r$, $\beta '(t)$ is defined in 38, and $\mathfrak{a}(h, t) \in \widetilde{\Xi }_{\mathbf{D}}$ is in 34.

If $\mathfrak{a}, \mathfrak{c}\in \widetilde{\Xi }_{\mathbf{D}}$ are chosen such that $\mathfrak{c}-rE_{n+1,n}^{\theta }$ is diagonal for some $r\in \mathbb{N}$ satisfying $\operatorname {co}(\mathfrak{c})= \operatorname {ro}(\mathfrak{a})$ and $s_r(\mathfrak{c})=s_l(\mathfrak{a})$, then we have

$$\begin{equation} [\mathfrak{c}] \cdot [\mathfrak{a}] = \sum _{t: \sum _{u=1}^N t_u=r}v^{\beta ''(t)} \ \overline{\mathcal{G}}\ [\mathfrak{a}(n,t)], {\tag{51}} \end{equation}$$

where the sum is taken over all $t=(t_u)\in \mathbb{N}^N$ such that $\sum _{u=1}^Nt_u=r$, $\mathcal{G}$ and $\beta ''(t)$ are in 39, and $\mathfrak{a}(n, t)\in \widetilde{\Xi }_{\mathbf{D}}$.

Given $\mathfrak{a}, \mathfrak{a}'\in \widetilde{\Xi }_{\mathbf{D}}$, we shall denote $\mathfrak{a}' \sqsubseteq \mathfrak{a}$ if $\mathfrak{a}' \preceq \mathfrak{a}$, $\operatorname {co}(\mathfrak{a}')=\operatorname {co}(\mathfrak{a})$, $\operatorname {ro}(\mathfrak{a}')= \operatorname {ro}(\mathfrak{a})$, $s_l(\mathfrak{a}')=s_l(\mathfrak{a})$, and $s_r(\mathfrak{a}')=s_r(\mathfrak{a})$.

By using 49, 50, and 51 and arguing in a similar way as in the proof of Theorem 4.6.3, we have the following proposition.

Proposition 5.1.3.

For any $\mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}}$, there exist signed matrices $\mathfrak{m}(i,j)$ such that $\mathfrak{m}(i,j) - R_{ij} E^{\theta }_{i, i+1}$ is diagonal with $R_{ij} =\sum _{k=1}^i a_{kj}$ and

$$\begin{equation} \mathfrak{m}_{\mathfrak{a}} \equiv \prod _{1\leq i<j \leq N}[\mathfrak{m}(i, j) ]=[\mathfrak{a}]+\sum _{\mathfrak{a}'\sqsubseteq \mathfrak{a}, \mathfrak{a}'\neq \mathfrak{a}} \gamma _{\mathfrak{a}', \mathfrak{a}} [\mathfrak{a}'], {\tag{52}} \end{equation}$$

where $\gamma _{\mathfrak{a}', \mathfrak{a}}\in \mathcal{A}$ and the product is taken in the order 42.

As a consequence of the above proposition, we have

Proposition 5.1.4.

The algebra $\mathcal{K}$ is generated by the elements $[\mathfrak{e}]$ such that $\mathfrak{e} - rE^{\theta }_{i,i+1}$ is diagonal for some $r\in \mathbb{N}$ and $i$ such that $1\leq i< N$.

We set

$$\begin{align} [k]_v = \frac{v^{k} -v^{-k}}{v-v^{-1}}. \tag{53} \end{align}$$

By applying 49, 50, and 51, we have

$$\begin{align} [\mathfrak{e}] \cdot [\mathfrak{e}'] = [r+1]_v [\mathfrak{e}'' ], {\tag{54}} \end{align}$$

if $\mathfrak{e}- E^{\theta }_{i, i+1}$, $\mathfrak{e}'-rE^{\theta }_{i, i+1}$, and $\mathfrak{e}'' -(r+1) E^{\theta }_{i,i+1}$ are diagonal for some $i\in [1, N-1]$, $s_r(\mathfrak{e})= s_l(\mathfrak{e}')$, $s_l(\mathfrak{e})= s_l(\mathfrak{e}'')$ and $s_r(\mathfrak{e}') =s_r(\mathfrak{e}'')$. From this observation, we have the following corollary.

Corollary 5.1.5.

The algebra $\mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{K}$ is generated by the elements $[\mathfrak{e}]$ such that either $\mathfrak{e}$ or $\mathfrak{e} - E^{\theta }_{i,i+1}$ is diagonal for some $i$ such that $1\leq i<N$.

5.2. Bases of $\mathcal{K}$

We define a bar involution $^- \ : \mathcal{K}\to \mathcal{K}$ by

$$\begin{equation*} \bar{v} = v^{-1}, \quad \overline{[\mathfrak{e}]} = [\mathfrak{e}], \end{equation*}$$

for any $\mathfrak{e}$ such that $\mathfrak{e} - RE^{\theta }_{i, i+1}$ is diagonal for some $R\in \mathbb{N}$ and $i\in [1, N-1]$. By using 52, we have

$$\begin{equation*} \overline{[\mathfrak{a}]} =[\mathfrak{a}] + \sum _{\mathfrak{a}': \mathfrak{a}' \sqsubseteq \mathfrak{a}, \mathfrak{a}'\neq \mathfrak{a}} c_{\mathfrak{a}', \mathfrak{a}} [\mathfrak{a}'], \quad \text{for some} \ c_{\mathfrak{a}', \mathfrak{a}} \in \mathcal{A}. \end{equation*}$$

By a standard argument similar to the proof of Proposition 4.7 in BLM90, we have the following proposition.

Proposition 5.2.1.

For any $\mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}}$, there exists a unique element $\{\mathfrak{a}\}$ in $\mathcal{K}$ such that

$$\begin{equation*} \overline{\{\mathfrak{a}\}}=\{\mathfrak{a}\},\quad \{\mathfrak{a}\}=[\mathfrak{a}]+\sum _{\mathfrak{a}'\sqsubset \mathfrak{a}, \mathfrak{a}'\neq \mathfrak{a}}\pi _{\mathfrak{a}', \mathfrak{a}}[\mathfrak{a}'], \quad \pi _{\mathfrak{a}', \mathfrak{a}} \in v^{-1} \mathbb{Z} [v^{-1}]. \end{equation*}$$

By Propositions 5.1.3 and 5.2.1, we have

Corollary 5.2.2.

The algebra $\mathcal{K}$ possesses a standard basis $\{[\mathfrak{a}] | \mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}} \}$, a monomial basis $\{\mathfrak{m}_{\mathfrak{a}} | \mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}}\}$, and a canonical basis $\{ \{ \mathfrak{a}\} | \mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}}\}$.

5.3. From $\mathcal{K}$ to $\mathcal{S}$

We define an $\mathcal{A}$-linear map

$$\begin{align} \Psi : \mathcal{K} \to \mathcal{S} {\tag{55}} \end{align}$$

by

$$\begin{equation*} \Psi ([\mathfrak{a}]) = \begin{cases}[\mathfrak{a}] , & \text{if} \ \mathfrak{a}\in \Xi _{\mathbf{D}},\\ 0, & \text{otherwise}. \end{cases} \end{equation*}$$

By comparing the multiplication formulas 37, 38, and 39 with 49, 50, and 51, respectively, and following an argument in Fu12 and BKLW14, Appendix A, we have

Theorem 5.3.1.

The map $\Psi$ in 55 is a surjective algebra homomorphism. Moreover we have

$$\begin{equation*} \Psi (\{\mathfrak{a}\}) = \begin{cases} \{ \mathfrak{a}\}, & \text{if} \ \mathfrak{a}\in \Xi _{\mathbf{D}},\\ 0, & \text{otherwise}. \end{cases} \end{equation*}$$

Now the algebra $\mathcal{K}$ acts on the $\mathcal{A}$-module $\mathcal{V}$ in 7 via $\Psi$ and the $\mathcal{S}$-action. By Lemma 3.2.1, we have

Proposition 5.3.2.

The algebra $\mathcal{K}$ and $\mathcal{H}_{\mathscr{Y}}$ form a double centralizer, i.e.,

$$\begin{equation*} \operatorname {End}_{\mathcal{K}} (\mathcal{V}) \simeq \mathcal{H}_{\mathscr{Y}}, \quad \text{if} \ n\geq d,\ \text{and} \quad \mathcal{K}\to \operatorname {End}_{\mathcal{H}_{\mathscr{Y}}}(\mathcal{V}) \text{ is surjective.} \end{equation*}$$

5.4. Towards a presentation of $\mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{K}$

We make an observation of the signed diagonal matrices in $\widetilde{\Xi }_{\mathbf{D}}$ in 47. We denote by $D_{\lambda }$ the diagonal matrix whose $i$-th diagonal entry is $\lambda _i$, for any $\lambda =(\lambda _i) \in \mathbb{Z}^N$. We have

Lemma 5.4.1.

Suppose that $\mathfrak{d}=(D_{\lambda }, \epsilon )$ is a signed diagonal matrix in $\widetilde{\Xi }_{\mathbf{D}}$. Then we have $\lambda _i = \lambda _{N+1-i}$ and $\lambda _{n+1}\in 2\mathbb{N}$. Moreover,

$$\begin{equation*} \lambda _{n+1} = \begin{cases} 0, & \text{if} \ \operatorname {sgn}(\mathfrak{d}) = \pm , \\ \geq 2, & \text{if} \ \operatorname {sgn}(\mathfrak{d}) =0. \end{cases} \end{equation*}$$

For any signed diagonal matrix $\mathfrak{d}$, we set

$$\begin{equation*} \begin{split} &E_h \mathfrak{d}= [(\mathfrak{d}- E^{\theta }_{h, h} + E^{\theta }_{h+1,h},\operatorname {sgn}(\mathfrak{d}) )],\quad \forall h\in [1,n], \\ &F_h \mathfrak{d}= [(\mathfrak{d}-E^{\theta }_{h+1, h+1} + E^{\theta }_{h,h+1}, \operatorname {sgn}(\mathfrak{d})) ],\quad \forall h\in [1,n-1]. \end{split} \end{equation*}$$

For a signed diagonal matrix $\mathfrak{d}=(D_{\lambda }, 0)$ of sign $0$, we set

$$\begin{equation*} F_n \mathfrak{d}= \begin{cases}[( \mathfrak{d}- E^{\theta }_{n+1, n+1} + E^{\theta }_{n,n+1}, 0) ] , & \text{if} \ \lambda _{n+1} \geq 4,\\ [(\mathfrak{d}- E^{\theta }_{n+1, n+1} + E^{\theta }_{n,n+1}, +)]\\ \quad +[(\mathfrak{d}- E^{\theta }_{n+1, n+1} + E^{\theta }_{n,n+1}, -)], & \text{if} \ \lambda _{n+1} =2. \end{cases} \end{equation*}$$

For any element $y\in \mathbf{U}$ in Section 2.1 and singed diagonal matrix $\mathfrak{d}$, we shall define the notation $y\mathfrak{d}$. We may assume that $y$ is homogeneous. We assume that $x\mathfrak{d}$ is defined for all homogenous $x\in \mathbf{U}$ of degree strictly less than $y$, and then we define

$$\begin{equation} E_j x \mathfrak{d}= \sum [\mathfrak{e}_j] \cdot x\mathfrak{d}, \quad \forall j\in [1,n], {\tag{56}} \end{equation}$$

where the sum runs over all signed matrices $\mathfrak{e}_j$ in $\widetilde{\Xi }_{\mathbf{D}}$ such that $\mathfrak{e}_j - E^{\theta }_{j+1,j}$ is diagonal. Although this is an infinite sum, there are only finitely many nonzero terms, hence it is well defined. Similarly, we can define $F_j x \mathfrak{d}$ for any $j\in [1,n]$. Therefore, the notation $y \mathfrak{d}$ for $y\in \mathbf{U}$ is well defined.

Proposition 5.4.2.

For any signed diagonal matrices $\mathfrak{d}= (D_{\lambda },\epsilon )$, $\mathfrak{d}' =(D_{\lambda '}, \epsilon ')$ in $\widetilde{\Xi }_{\mathbf{D}}$, we have the following relations in $\mathcal{K}$:

$$\begin{align} \mathfrak{d}\mathfrak{d}' &= \delta _{\mathfrak{d}, \mathfrak{d}'} \mathfrak{d}. {\tag{57}}\\ \mathfrak{d}' E_n \mathfrak{d}&= 0, \quad \text{if $\operatorname {sgn}(\mathfrak{d}') =\pm $}, {\tag{58}}\\ \mathfrak{d}' F_n \mathfrak{d}&= 0, \quad \text{if $\operatorname {sgn}(\mathfrak{d}) = \pm $}, \\ \mathfrak{d}E_h [(\mathfrak{d}- E^{\theta }_{h, h} + E^{\theta }_{h+1,h+1}, \operatorname {sgn}(\mathfrak{d}) ) ] &= E_h [(\mathfrak{d}- E^{\theta }_{h, h} + E^{\theta }_{h+1,h+1},\operatorname {sgn}(\mathfrak{d}))], \\ \mathfrak{d}F_h [(\mathfrak{d}+ E^{\theta }_{h, h} - E^{\theta }_{h+1,h+1}, \operatorname {sgn}(\mathfrak{d}))] & = F_h [(\mathfrak{d}+ E^{\theta }_{h, h} - E^{\theta }_{h+1,h+1}, \operatorname {sgn}(\mathfrak{d}))], &&\text{if $h\neq n$}, \\ F_nE_n \mathfrak{d}- \mathfrak{d}' F_n E_n \mathfrak{d}&= [\lambda _n]_v \mathfrak{d}, && \text{if $\lambda =\lambda '$, $\epsilon = - \epsilon ' \neq 0$}, {\tag{59}}\\ \mathfrak{d}' F_n E_n \mathfrak{d}&=0, \quad \mathfrak{d}F_n E_n \mathfrak{d}' =0, && \text{if $\operatorname {sgn}(\mathfrak{d}) =0$, $\operatorname {sgn}(\mathfrak{d}') =\pm $}, \\ {\tag{60}}\\ (E_iF_j-F_jE_i) \mathfrak{d}&=0, && \text{if $i\neq j$},\tag{61}\\ (E_iF_i-F_iE_i) \mathfrak{d}&= [\lambda _{i+1}- \lambda _i]_v \mathfrak{d}, && \text{if $i\neq n$}, \\ \ (E_iE_iE_j- [2]_v E_iE_jE_i+E_jE_iE_i) \mathfrak{d}&=0, && \text{if $|i-j|=1$},{\tag{62}}\\ (F_iF_iF_j- [2]_v F_iF_jF_i+F_jF_iF_i) \mathfrak{d}&=0, && \text{if $|i-j|=1$}, \\ \ (E_iE_j-E_jE_i) \mathfrak{d}&= 0, \quad (F_iF_j-F_jF_i)\mathfrak{d}=0, && \text{if $|i-j|>1$},{\tag{63}}\\ \ (E^2_nF_n+F_nE_n^2) \mathfrak{d}&= [2]_v (E_nF_nE_n-E_n (v^{\lambda _{n+1} - \lambda _n+1} +v^{- \lambda _{n+1} +\lambda _n-1} )) \mathfrak{d},{\tag{64}}\\ F_n^2E_n+E_nF_n^2) \mathfrak{d}&= [2]_v (F_nE_nF_n- (v^{\lambda _{n+1}-\lambda _n-2} +v^{-\lambda _{n+1} + \lambda _n+2} ) F_n) \mathfrak{d}. \end{align}$$

Proof.

The proofs of the identities in 57 and 58 are straightforward. We now show 59. By the multiplication formula 49, we have

$$\begin{equation*} \begin{split} F_n E_n \mathfrak{d}&=[\lambda _n]_v [\mathfrak{d}]+[( D_{\lambda '}+E_{n,n+2}^{\theta }, - \epsilon )],\\ \mathfrak{d}' F_nE_n \mathfrak{d}&=[(D_{\lambda '} +E_{n,n+2}^{\theta }, -\epsilon )], \quad \quad \quad \text{where}\ \lambda _i'=\lambda _i-\delta _{i,n}-\delta _{i, n+2}. \end{split} \end{equation*}$$

So the first identity in 59 holds. Observe that if $\operatorname {sgn}(\mathfrak{d})=0$, then $\lambda _{n+1}\neq 0$; and if $\operatorname {sgn}(\mathfrak{d})\neq 0$, then $\lambda _{n+1}=0$. We have the second identities in 59 by this observation.

For the remaining relations, they can be proved by the following principle. Suppose that $x\mathfrak{d}= \sum C_{x\mathfrak{d}, \mathfrak{a}} \mathfrak{a}$ with $C_{x\mathfrak{d}, \mathfrak{a}}\in \mathcal{A}$. We can pick a large enough $p$ such that $_p\mathfrak{d}$ and $_p\mathfrak{a}$ all have nonnegative entries. For an appropriate $D'$, we have an element in $\mathcal{S}_{D'}$ of the form $x\, {}_p\mathfrak{d}$ defined in a similar way as that in $\mathcal{K}$. We can write

$$\begin{equation*} x \, {}_p \mathfrak{d}= \sum {}_pC_{x {}\mathfrak{d}, {}\mathfrak{a}} (v, v')|_{v'=v^{-p}} \ {}_p\mathfrak{a}\quad \text{in} \ \mathcal{S}_{D'}, \end{equation*}$$

where ${}_pC_{x {}\mathfrak{d}, {}\mathfrak{a}} (v, v') \in \mathfrak{R}$ in 48. If $x$ is of the form in the remaining relations, we have

$$\begin{equation*} C_{x\mathfrak{d}, \mathfrak{a}} = {}_pC_{x {}\mathfrak{d}, {}\mathfrak{a}} (v, v')|_{v'=1}. \end{equation*}$$

This follows from the comparison of 49, 50, and 51 in $\mathcal{K}$ with 37, 38, and 39 in $\mathcal{S}_{D'}$, respectively. Now the remaining relations all hold in $\mathcal{S}_{D'}$ for all $D'$ large enough by Proposition 4.1.1, so are those relations without specializing $v'$. Now relations in $\mathcal{K}$ are obtained by specializing $v'=1$.

■

5.5. The algebra $\mathcal{U}$

In this subsection, we shall define a new algebra $\mathcal{U}$ in the completion of $\mathcal{K}$ similar to BLM90, Section 5. We show that $\mathcal{U}$ is a quotient of the algebra $\mathbf{U}$ defined in Section 2.1.

Let $\hat{\mathcal{K}}$ be the $\mathbb{Q}(v)$-vector space of all formal sums $\sum _{\mathfrak{a}\in \tilde{\Xi }_{\mathbf{D}}}\xi _{\mathfrak{a}} [\mathfrak{a}]$ with $\xi _{\mathfrak{a}}\in \mathbb{Q}(v)$ and a locally finite property, i.e., for any ${\mathbf{t}}\in \mathbb{Z}^N$, the sets $\{\mathfrak{a}\in \tilde{\Xi }_{\mathbf{D}}| \operatorname {ro}(\mathfrak{a})={\mathbf{t}}, \xi _{\mathfrak{a}} \neq 0\}$ and $\{\mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}} | \operatorname {co}(\mathfrak{a})={\mathbf{t}}, \xi _{\mathfrak{a}} \neq 0\}$ are finite. The space $\hat{\mathcal{K}}$ becomes an associative algebra over $\mathbb{Q}(v)$ when equipped with the following multiplication:

$$\begin{equation*} \sum _{\mathfrak{a}\in \tilde{\Xi }_{\mathbf{D}}} \xi _{\mathfrak{a}} [\mathfrak{a}] \cdot \sum _{\mathfrak{b}\in \tilde{\Xi }_{\mathbf{D}}} \xi _{\mathfrak{b}} [\mathfrak{b}] =\sum _{\mathfrak{a}, \mathfrak{b}} \xi _{\mathfrak{a}} \xi _{\mathfrak{b}} [\mathfrak{a}] \cdot [\mathfrak{b}], \end{equation*}$$

where the product $[\mathfrak{a}] \cdot [\mathfrak{b}]$ is taken in $\mathcal{K}$. This is shown in exactly the same way as BLM90, Section 5.

Observe that the algebra $\hat{\mathcal{K}}$ has a unit element $\sum \mathfrak{d}$, the summation of all diagonal signed matrices.

We define the following elements in $\hat{\mathcal{K}}$. For any nonzero signed matrix $\mathfrak{a}=(A, \epsilon )\in \widetilde{\Xi }_{\mathbf{D}}$, let $\hat{\mathfrak{a}}=(\hat{A}, \epsilon )$, where $\hat{A}$ is the matrix obtained by replacing diagonal entries of $A$ by zeroes. We set

$$\begin{equation*} \hat{\Xi }_{\mathbf{D}}= \{ \hat{\mathfrak{a}} | \mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}} \}, \end{equation*}$$

For any $\hat{\mathfrak{a}}$ in $\hat{\Xi }_{\mathbf{D}}$ and ${\mathbf{j}}=(j_1,\dots , j_N)\in \mathbb{Z}^N$, we define

$$\begin{equation} \hat{\mathfrak{a}} ({\mathbf{j}})=\sum _{\lambda }v^{\lambda _1j_1+\dots +\lambda _{n+1}j_{n+1}}[ (\hat{\mathfrak{a}} + D_{\lambda }, \operatorname {sgn}(\hat{\mathfrak{a}})) ], {\tag{65}} \end{equation}$$

where the sum runs through all $\lambda =(\lambda _i)\in \mathbb{Z}^N$ such that $(\hat{\mathfrak{a}} + D_{\lambda }, \operatorname {sgn}(\hat{\mathfrak{a}})) \in \widetilde{\Xi }_{\mathbf{D}}$.

For any $i\in [1,n]$, there exists $\mathfrak{a}=(A, \epsilon )$ such that $\hat{\mathfrak{a}}=(E_{i+1, i}^{\theta }, \epsilon )$ (resp. $\hat{\mathfrak{a}}=(E_{i, i+1}^{\theta }, \epsilon )$). So by 65, the elements $E_{i+1,i}^{\theta , \epsilon }(\mathbf{j})$ (resp. $E_{i, i+1}^{\theta , \epsilon }(\mathbf{j})$) are well defined, for any $\mathbf{j}\in \mathbb{Z}^N$. Moreover, this definition is independent of the choice of $\hat{\mathfrak{a}}$. For $i\in [1,n]$, let

$$\begin{equation*} E_i=E_{i+1,i}^{\theta , +}(0)+E_{i+1,i}^{\theta , 0}(0)+E_{i+1,i}^{\theta , -}(0)\quad \text{and}\quad F_i=E_{i, i+1}^{\theta ,+}(0)+E_{i, i+1}^{\theta ,0}(0)+E_{i, i+1}^{\theta ,-}(0). \end{equation*}$$

For simplicity, we write $E_i^{\epsilon }(\mathbf{j})$ (resp. $F_i^{\epsilon }(\mathbf{j})$) instead of $E_{i+1,i}^{\theta , \epsilon }(\mathbf{j})$ (resp. $E_{i,i+1}^{\theta , \epsilon }(\mathbf{j})$).

We also define

$$\begin{equation*} \begin{split} 0(\mathbf{j})&=0^+(\mathbf{j})+0^0(\mathbf{j})+0^-(\mathbf{j}),\quad \text{where}\\ 0^{\epsilon }(\mathbf{j})&=\sum v^{\lambda _1j_1+\dots +\lambda _{n+1}j_{n+1}}[\mathfrak{d}], \end{split} \end{equation*}$$

where the sum runs through all diagonal matrices $\mathfrak{d}$ with sign $\epsilon$ and the $\lambda _i$’s are diagonal entries of $\mathfrak{d}$.

Let $\mathcal{U}$ be the subalgebra of $\hat{\mathcal{K}}$ generated by $E_i, F_i, 0(\mathbf{j}), 0^+(0), 0^0(0)$, and $0^-(0)$, for all $i\in [1,n]$ and $\mathbf{j}\in \mathbb{Z}^N$.

Proposition 5.5.1.

The following relations hold in $\mathcal{U}$:

$$\begin{align} 0(\mathbf{j})0(\mathbf{j}') &= 0 (\mathbf{j}')0(\mathbf{j}),\ 0^{\pm }(0)0(\mathbf{j})=0(\mathbf{j})0^{\pm }(0),\ 0^{\pm }(0)^2=0^{\pm }(0),{\tag{66}}\\ 0^+(0)+0^0(0)+0^-(0) &= 1,\ 0^{\alpha }(0)0^{\epsilon }(0)=\delta _{\alpha ,\epsilon }0^{\alpha }(0), \\ 0(\mathbf{j})F_h&=v^{j_h-j_{h+1}-\delta _{hn}j_{n+1}}F_h0(\mathbf{j}),\ 0(\mathbf{j})E_h=v^{-j_h+j_{h+1}+\delta _{hn}j_{n+1}}E_h0(\mathbf{j}),{\tag{67}}\\ 0^{\pm }(0)E_h &= (1-\delta _{hn})E_h 0^{\pm }(0),\ F_h0^{\pm }(0)=(1-\delta _{hn})0^{\pm }(0)F_h,{\tag{68}}\\ 0^{\pm }(0)F_nE_n-F_nE_n0^{\mp }(0) &= \frac{0(\underline{n})-0(-\underline{n})}{v-v^{-1}}(0^{\pm }(0)-0^{\mp }(0)),{\tag{69}}\\ F_hE_h-E_hF_h &= (v-v^{-1})^{-1}(0(\underline{h}-\underline{h+1})-0(\underline{h+1} -\underline{h})),{\tag{70}}\\ E_iF_n &= F_nE_i,\quad F_iE_n=E_nF_i, && \text{if $i\in [1,n-2]$}, \\ E_iE_j &=E_jE_i,\quad F_iF_j=F_jF_i, && \text{if $|i-j|>1$},{\tag{71}}\\ E^2_nF_n+F_nE_n^2 &= [2]_v (E_nF_nE_n -E_n(v0(\underline{n+1}-\underline{n})+v^{-1}0(\underline{n}- \underline{n+1}))),{\tag{72}}\\ F_n^2E_n+E_nF_n^2 &= [2]_v (F_nE_nF_n -(v0(\underline{n+1}-\underline{n})+v^{-1}0(\underline{n}-\underline{n+1}))F_n), \\ E_i^2E_j- [2]_v E_iE_jE_i+E_jE_i^2 &= 0, && \text{if $|i-j|=1$, $i$, $j\in [1,n-1]$},{\tag{73}}\\ F_i^2F_j-[2]_v F_iF_jF_i+F_jF_i^2 &= 0, && \text{if $|i-j|=1$, $i$, $j\in [1,n-1]$}, \end{align}$$

where $\mathbf{j}$, $\mathbf{j}'\in \mathbb{Z}^N$, $\alpha$, $\epsilon \in \{ \pm , 0\}$, $h, i, j\in [1, n]$, and $\underline{i} \in \mathbb{N}^N$ is the vector whose $i$-th entry is $1$ and $0$ elsewhere.

Proof.

We show 67. By checking the values of functions $s_l$ and $s_r$ defined in 29 at $0(\mathbf{j})$ and $F_n$, we have

$$\begin{align*} 0(\mathbf{j})F_n &= 0^+(\mathbf{j})F^+_n(0)+0^0(\mathbf{j})F^0_n(0)+0^-(\mathbf{j})F^-_n(0)\\ &= \sum _{\lambda ,\lambda '}v^{\sum \lambda _kj_k}[D_{\lambda }^+] [( E_{n,n+1}^{\theta ,+}+D_{\lambda '}, +)]\\ & \qquad {} +\sum _{\lambda ,\lambda '}v^{\sum \lambda _kj_k}[D_{\lambda }^-] [( E_{n,n+1}^{\theta ,-}+D_{\lambda '}, -)]\\ & \qquad {} +\sum _{\lambda ,\lambda '}v^{\sum \lambda _kj_k}[D_{\lambda }^0] [(E_{n,n+1}^{\theta ,0}+D_{\lambda '}, 0)]\\ &= v^{j_n}(F_n^+(\mathbf{j})+F_n^0(\mathbf{j})+F_n^-(\mathbf{j}))=v^{j_n}F_n(\mathbf{j}), \end{align*}$$

where the sums run through in an obvious range by the definition in 65.

$$\begin{align*} F_n 0(\mathbf{j}) &= F^+_n(0)0^0(\mathbf{j})+F^0_n(0)0^0(\mathbf{j})+F^-_n(0)0^0(\mathbf{j})\\ &= \sum _{\lambda , \lambda ' }v^{\sum \lambda _kj_k}[(E_{n,n+1}^{\theta ,+}+D_{\lambda '}, +) ][D_{\lambda }^0]\\ & \qquad {}+\sum _{\lambda ,\lambda '}v^{\sum z_kj_k} [(E_{n,n+1}^{\theta ,0}+D_{\lambda }, 0)] [D_{\lambda }^0]\\ & \qquad {}+\sum _{\lambda , \lambda '}v^{\sum \lambda _kj_k}[(E_{n,n+1}^{\theta ,-}+D_{\lambda '}, -)][D_{\lambda }^0]\\ &=v^{2j_{n+1}}(F_n^+(\mathbf{j})+F_n^0(\mathbf{j})+F_n^-(\mathbf{j}))=v^{2j_{n+1}}F_n(\mathbf{j}). \end{align*}$$

So we have the first identity in 67 for the case of $h=n$. Other cases for the first identity and all other identities in 66 and 67 can be shown similarly.

We show 68. By the definition of $0^+(0)$ and $F_h$ for $h<n$, we have

$$\begin{equation*} \begin{split} 0^+(0)F_h &=0^+(0)F_h^+(0) = \sum _{\lambda ,\lambda '}[D_{\lambda }^+] [(D_{\lambda '}^+ + E_{h,h+1}^{\theta }, +)]\\ &=\sum _{\lambda '} [( D_{\lambda '}^+ + E_{h+1,h}^{\theta }, +)]=F_h0^+(0). \end{split} \end{equation*}$$

The other identities in 68 can be shown similarly.

We show 69. By Proposition 5.4.2 59, we have

$$\begin{equation*} \begin{split} &0^+(0)F_nE_n= \sum _{\lambda }[D_{\lambda }^+]F_nE_n =\sum _{\lambda }([\lambda _n]_v [D^+_{\lambda _n}]+[(D_{\lambda }^++E_{n,n+2}^{\theta }, +)]),\\ &F_nE_n0^-(0)= \sum _{\lambda }F_nE_n[D_{\lambda }^-] =\sum _{\lambda }([\lambda _n]_v [D^-_{\lambda _n}]+[(D_{\lambda }^++E_{n,n+2}^{\theta }, +)]). \end{split} \end{equation*}$$

Therefore,

$$\begin{equation*} \begin{split} 0^+(0)F_nE_n-F_nE_n0^-(0)&={\sum _{\lambda }} [\lambda _n]_v ([D^+_{\lambda }]-[D^-_{\lambda }])\\ & =\frac{0(\underline{n})-0(- \underline{n})}{v-v^{-1}}(0^{+}(0)-0^{-}(0)). \end{split} \end{equation*}$$

We now show 72. By definition, we have

$$\begin{equation*} v0(\underline{n+1}- \underline{n})F_n=F_n 0 (\underline{n+1}- \underline{n})= \sum _{\lambda }v^{\lambda _{n+1}-\lambda _n-2}F_nD_{\lambda }^0. \end{equation*}$$

Similarly, $v^{-1}0(\underline{n}- \underline{n+1}) F_n= \sum _{\lambda }v^{\lambda _n-\lambda _{n+1}+2}F_nD_{\lambda }^0$. Moreover,

$$\begin{equation*} F_n^2E_n+E_nF_n^2-[2]_v F_nE_nF_n=\sum _{\lambda } (F_n^2E_n+E_nF_n^2-[2]_v F_nE_nF_n)D_{\lambda }^0. \end{equation*}$$

The identity 72 follows from Proposition 5.4.2. All other identities in 70, 71, 72, and 73 can be shown similarly.

■

Proposition 5.5.2.

The assignment $E_i\mapsto E_i$, $F_i\mapsto F_i$, $H_a \mapsto 0(- \underline{a})$, and $J_{\alpha }\mapsto 0^{\alpha }(0)$, for any $i\in [1,n]$, $a\in [1, n+1]$, and $\alpha \in \{0, +, -\}$, defines a surjective algebra homomorphism $\Upsilon : \mathbf{U}\rightarrow \mathcal{U}$, where $\mathbf{U}$ is defined in Section 2.1.

Proof.

Under the map $\Upsilon$, all defining relations of $\mathbf{U}$ map to the corresponding relations in $\mathcal{U}$ given in Proposition 5.5.1 except the commutator relation between $J_{\pm }$ and $F_nE_n$. Since

$$\begin{equation*} 0(\underline{n+1}-\underline{n})0^{\pm }(0)=\sum _{\lambda }v^{\lambda _{n+1}-\lambda _n}[D_{\lambda }^{\pm }] =\sum _{\lambda }v^{-\lambda _n}[D_{\lambda }^{\pm }]=0(- \underline{n})0^{\pm }(0), \end{equation*}$$

we have

$$\begin{equation*} \begin{split} &\Upsilon (J_{\pm }F_nE_n-F_nE_nJ_{\mp }-\frac{H_n^{-1}H_{n+1} -H_n H_{n+1}^{-1} }{v-v^{-1}} (J_{\pm }-J_{\mp }))\\ & =0^{\pm }(0)F_nE_n-F_nE_n0^{\mp }(0)-\frac{0( \underline{n})-0(- \underline{n})}{v-v^{-1}}(0^{\pm }(0)-0^{\mp }(0))=0. \end{split} \end{equation*}$$

This shows that $\Upsilon$ is an algebra homomorphism. The surjectivity is clear.

■

Remark 5.5.3.

It is not clear if $\ker \Upsilon =0$.

6. Case II

In this section, we turn to the case when all flags at the $n$-th step are assumed to be maximal isotropic.

6.1. The second double centralizer

We define $\mathscr{X}^m$ to be the subset of $\mathscr{X}$ in Section 3.2 subject to the condition that the $n$-th step of the flags is maximal isotropic. In particular, we have $V_n = V_{n+1}$ for any $V\in \mathscr{X}^m$, and thus

$$\begin{equation*} \mathscr{X}^m = \mathscr{X}^2 \sqcup \mathscr{X}^3. \end{equation*}$$

Similar to the definition of the algebra $\mathcal{S}$ in Section 4.2, we consider the convolution algebra

$$\begin{equation*} \mathcal{S}^m=\mathcal{A}_G (\mathscr{X}^m \times \mathscr{X}^m) \end{equation*}$$

on $\mathscr{X}^m \times \mathscr{X}^m$ and the free $\mathcal{A}$-module

$$\begin{equation*} \mathcal{W}=\mathcal{A}_G(\mathscr{X}^m \times \mathscr{Y}). \end{equation*}$$

Under the convolution product, $\mathcal{W}$ has an $\mathcal{S}^m$-$\mathcal{H}_{\mathscr{Y}}$-bimodule structure. By P09, Theorem 2.1, we have

Lemma 6.1.1.

The triple $(\mathcal{S}^m, \mathcal{H}_{\mathscr{Y}}; \mathcal{W})$ satisfies the double centralizer property, i.e.,

$$\begin{align} \operatorname {End}_{\mathcal{S}^m}(\mathcal{W})=\mathcal{H}_{\mathscr{Y}}\quad \text{and}\quad \operatorname {End}_{\mathcal{H}_{\mathscr{Y}}}(\mathcal{W})=\mathcal{S}^m, \quad \text{if}\; n\geq d. {\tag{74}} \end{align}$$

Let $\Pi ^m=\{B\in \Pi |b_{n+1,j}=0, \forall j\}$, where $\Pi$ is defined in Section 3.3. A restriction of the bijection 11 in Section 3.3 yields a bijection

$$\begin{equation*} G \backslash \mathscr{X}^m \times \mathscr{Y} \xrightarrow {\sim } \Pi ^m. \end{equation*}$$

Moreover, the isomorphism 13 restricts to an isomorphism

$$\begin{align} \mathbf{W}^{\otimes d} \overset{\nobreakspace }{\to } \mathbb{Q}(v)\otimes \mathcal{W}, {\tag{75}} \end{align}$$

where $\mathbf{W}^{\otimes d}$ is defined in Section 2.2.

Observe that the algebra $\mathcal{S}^m$ is naturally a subalgebra of $\mathcal{S}$, while $\mathcal{W}$ is an $\mathcal{A}$-submodule of $\mathcal{V}$ in 7. So we can define the functions $E_i$, $F_i$, $H_a^{\pm }$, for $i\in [1, n-1]$, $a\in [1, n]$, and $J_{\pm }$ in $\mathcal{S}^m$ to be the restrictions of the functions in $\mathcal{S}$ under the same notation, respectively. We further define

$$\begin{equation} T(V,V')= \begin{cases} v^{1-\lambda '_n}, & \text{if}\ |V_n\cap V_n'|= d-1,\ V_j=V_j',\ \forall j\in [1,n-1],\\ 0,&\text{otherwise}. \end{cases} \tag{76} \end{equation}$$

By definition, we have

$$\begin{equation} T= \left( F_nE_n+\frac{H_nH_{n+1}^{-1} - H_n^{-1}H_{n+1} }{v-v^{-1}} \right)|_{\mathscr{X}^m \times \mathscr{X}^m}. {\tag{77}} \end{equation}$$

We see immediately

Lemma 6.1.2.

The actions of $E_i$, $F_i$, $H_a^{\pm 1}$, and $J_{\pm }$ for $i\in [1, n-1]$, $a\in [1,n]$ on $\mathcal{W}$ are given by the formulas in Corollary 4.4.1, with the action of $T$ on $\mathcal{W}$ given by $F_nE_n+\frac{H_nH_{n+1}^{-1} - H_n^{-1}H_{n+1} }{v-v^{-1}}$ from Corollary 4.4.1 again.

6.2. Relations for $\mathcal{S}^m$

We now determine the relations for the algebra $\mathcal{S}^m$. By using Proposition 4.1.1 and 77, we have

Proposition 6.2.1.

The functions $E_i$, $F_i$, and $H_a^{\pm 1}$, for $i\in [1, n-1]$, $a\in [1,n]$ together with the functions $J_{\pm }$ and $T$ in $\mathcal{S}^m$ satisfy the defining relations of the algebra $\mathbf{U}^m$ in Section 2.2.

Remark 6.2.2.

The function $T$ has a geometric interpretation. More preciely, we set

$$\begin{equation*} S(T)=\{\, (V,V') \mid |V_n\cap V_n'|=d-1,\ V_j=V_j', \forall j\in [1,n-1] \,\}. \end{equation*}$$

By 40, we see that $S(T)$ is a smooth closed subvariety of $\mathscr{X}^m\times \mathscr{X}^m$ over the algebraic closure of the field $\mathbb{F}_q$. So the function $T$ is the function version of the intersection complex associated to the variety $S(T)$, up to a shift.

The rest of this subsection is devoted to giving another more direct proof of Proposition 6.2.1.

We first define an auxiliary function $\widetilde{T}$ by

$$\begin{equation} \widetilde{T}(V,V')=\begin{cases} v^{1-\lambda '_n}, & \text{if}\ |V_n\cap V_n'|\geq d-1,\ V_j=V_j', \forall j\in [1,n-1],\\ 0,&\text{otherwise}, \end{cases} \tag{78} \end{equation}$$

where $\lambda '_n=|V'_n|-|V'_{n-1}|$. Moreover, we have

$$\begin{equation} \widetilde{T}=T+ vK_n = (F_nE_n+v\frac{vK_n-v^{-1}K_n^{-1}}{v-v^{-1}})|_{\mathscr{X}^m \times \mathscr{X}^m}. {\tag{79}} \end{equation}$$

By a direct computation, we have

$$\begin{align*} E^2_{n-1}\widetilde{T}(V,V') &=\begin{cases} (v^2+1)v^{-3\lambda _n'},& \text{if}\ V_{n-1}\overset{2}{\subset }V_{n-1}'\subset V_n,\ |V_n\cap V_n'|\geq d-1,\\[3.0pt] 0,&\text{otherwise}, \end{cases} \\[3.0pt] \widetilde{T} E^2_{n-1}(V,V')&=\begin{cases} (v^2+1)v^{-3\lambda _n'-2},& \text{if}\ V_{n-1}\overset{2}{\subset }V_{n-1}',\ |V_n\cap V_n'|\geq d-1,\\[3.0pt] 0,&\text{otherwise}, \end{cases} \\[3.0pt] E_{n-1}\widetilde{T} E_{n-1}(V,V') &= \begin{cases} (v^2+1)v^{-3\lambda _n'-1},& \text{if}\ V_{n-1}\overset{2}{\subset } V_{n-1}' \subset V_n,\ |V_n \cap V_n'|\geq d-1,\\[3.0pt] v^{-3\lambda _n'-1},& \text{if}\ V_{n-1}\overset{2}{\subset }V_{n-1}' \not \subset V_n,\ |V_n \cap V_n'| \geq d-1,\\[3.0pt] 0,&\text{otherwise}. \end{cases} \end{align*}$$

So we have

$$\begin{equation*} E_{n-1}^2\widetilde{T}-[2]_v E_{n-1}\widetilde{T}E_{n-1}+\widetilde{T}E_{n-1}^2=0, \end{equation*}$$

and by 79 it implies that

$$\begin{align} E_{n-1}^2 T-[2]_v E_{n-1} TE_{n-1}+TE_{n-1}^2=0. {\tag{80}} \end{align}$$

A direct computation shows that we have

$$\begin{align*} \widetilde{T}^2E_{n-1}(V,V')&=\begin{cases} (\frac{v^{2\lambda '_n+2}-1}{v^2-1}+1)v^{-3\lambda '_n},& \text{if} \ V_{n-1}\overset{1}{\subset }V'_{n-1},\ V_n=V'_n, \\[3.0pt] (v^2+1)v^{-3\lambda '_n},& \text{if}\ V_{n-1}\overset{1}{\subset }V'_{n-1},\ |V_n\cap V_n'|=d-2,\\[3.0pt] 2v^{-3\lambda '_n},& \text{if}\ V_{n-1}\overset{1}{\subset }V'_{n-1},\ |V_n\cap V_n'|=d-1,\\[3.0pt] 0,&\text{otherwise}, \end{cases} \\[3.0pt] E_{n-1}\widetilde{T}^2(V,V')&=\begin{cases} (\frac{v^{2\lambda '_n}-1}{v^2-1}+1)v^{-3\lambda '_n+2},& \text{if} \ V_{n-1} \overset{1}{\subset } V'_{n-1} \subset V_n,\ V_n=V'_n, \\[3.0pt] (v^2+1)v^{-3\lambda '_n+2},& \text{if}\ V_{n-1}\overset{1}{\subset } V'_{n-1} \subset V_n,\ |V_n\cap V_n'|=d-2,\\[3.0pt] 2v^{-3\lambda '_n+2},& \text{if}\ V_{n-1} \overset{1}{\subset } V'_{n-1} \subset V_n,\ |V_n\cap V_n'|=d-1,\\[3.0pt] 0,&\text{otherwise}, \end{cases} \\[3.0pt] \widetilde{T}E_{n-1}\widetilde{T}(V,V') &= \begin{cases} (\frac{v^{2\lambda '_n}-1}{v^2-1}+1)v^{-3\lambda '_n+1},& \text{if} \ V_{n-1} \overset{1}{\subset } V'_{n-1} \subset V_n,\ V_n=V'_n, \\[3.0pt] (v^2+1)v^{-3\lambda '_n+1},& \text{if}\ V_{n-1} \overset{1}{\subset } V'_{n-1} \subset V_n,\ |V_n \cap V_n'|=d - 2,\\[3.0pt] v^{-3\lambda '_n+1},& \text{if}\ V_{n-1} \overset{1}{\subset } V'_{n-1} \not \subset V_n,\ |V_n \cap V_n'|=d - 2,\\[3.0pt] 2v^{-3\lambda '_n+1},& \text{if}\ V_{n-1} \overset{1}{\subset } V'_{n-1} \subset V_n,\ |V_n \cap V_n'|=d - 1,\\[3.0pt] v^{-3\lambda '_n+1},& \text{if}\ V_{n-1} \overset{1}{\subset } V'_{n-1} \not \subset V_n,\ |V_n \cap V_n'|=d - 1,\\[3.0pt] 0,&\text{otherwise}, \end{cases} \\[3.0pt] E_{n-1}\widetilde{T}(V,V') &= \begin{cases} v^{-2\lambda '_n+1},& \text{if}\ V_{n-1}\overset{1}{\subset }V'_{n-1}\subset V_n,\ |V_n\cap V_n'|\geq d-1,\\[3.0pt] 0,&\text{otherwise}, \end{cases} \\[3.0pt] \widetilde{T} E_{n-1}(V,V') &= \begin{cases} v^{-2\lambda '_n},& \text{if}\ V_{n-1}\overset{1}{\subset }V'_{n-1},\ |V_n\cap V_n'|\geq d-1,\\[3.0pt] 0,&\text{otherwise}. \end{cases} \end{align*}$$

This implies that we have

$$\begin{equation*} \widetilde{T}^2E_{n-1}-[2]_v\widetilde{T}E_{n-1}\widetilde{T}+E_{n-1}\widetilde{T}^2=E_{n-1}-(v-v^{-1})(v\widetilde{T}E_{n-1}-E_{n-1}\widetilde{T})H_nH_{n+1}^{-1}, \end{equation*}$$

which implies again by 79 that

$$\begin{equation} T^2E_{n-1}- [2]_v TE_{n-1}T+E_{n-1}T^2=E_{n-1}. {\tag{81}} \end{equation}$$

Now apply the map $\rho$ in 18 to 80 and 81. We get

$$\begin{align*} & F_{n-1}^2 T - [2]_v F_{n-1}TF_{n-1}+ TF_{n-1}^2=0,\\ & T^2F_{n-1} - [2]_vTF_{n-1}T+F_{n-1}T^2=F_{n-1}. \end{align*}$$

The other defining equations of $\mathbf{U}^m$ are straightforward to check and are skipped. This finishes the proof of Proposition 6.2.1.

6.3. Generators and bases for $\mathcal{S}^m$

We consider the following subset of $\Xi _{\mathbf{D}}$:

$$\begin{equation*} \Xi _{\mathbf{D}}'=\{ \mathfrak{a}\in \Xi _{\mathbf{D}}| \operatorname {ro}(\mathfrak{a})_{n+1}=\operatorname {co}(\mathfrak{a})_{n+1}=0\}. \end{equation*}$$

We then have $\operatorname {sgn}(\mathfrak{a}) \in \{ +, -\}$ if $\mathfrak{a}\in \Xi _{\mathbf{D}}'$. Moreover, we have a bijection

$$\begin{align} G\backslash \mathscr{X}^m\times \mathscr{X}^m \simeq \Xi _{\mathbf{D}}', \tag{82} \end{align}$$

inherited from the bijection 24.

Recall from Theorem 4.6.3 that we set $R_{ij}=\sum _{k=1}^ia_{kj}$ for a signed matrix $\mathfrak{a}=(A, \epsilon )$. Let $\mathfrak{e}_{i,t}$ denote a signed matrix such that $\mathfrak{e}_{i, t} -R_{i, i+t} E^{\theta }_{i, i+1}$ is diagonal. For a sequence $a_s$, $a_{s+1}$, …, $a_{r}$ with $s\leq r$, we set

$$\begin{equation*} \overset{s}{\underset{i=r}{\bigsqcap }} a_i=a_ra_{r-1}\dots a_s. \end{equation*}$$

Theorem 6.3.1.

For any $\mathfrak{a}=A^{\epsilon } \in \Xi '_{\mathbf{D}}$, there exists a product of signed matrices $\mathfrak{e}_{i, t}$,

$$\begin{equation} \begin{split} \mathfrak{n}_{\mathfrak{a}} &= \left( \overset{n+1}{\underset{t=N-1}{\bigsqcap }} \overset{1}{\underset{i=N-t}{\bigsqcap }} [\mathfrak{e}_{i, t}] \right)\\ &\quad \cdot \overset{1}{\underset{t=n}{\bigsqcap }} \left( \overset{n+2}{\underset{i=N-t}{\bigsqcap }} [\mathfrak{e}_{i, t}] ([\mathfrak{e}_{n, t+1}][\mathfrak{e}_{n+1, t}]) \overset{n-t+1}{\underset{i=n-1}{\bigsqcap }} [\mathfrak{e}_{i, t+1}] \overset{1}{\underset{i=n-t}{\bigsqcap }} [\mathfrak{e}_{i,t}] \right), \end{split} {\tag{83}} \end{equation}$$

such that

$$\begin{equation} \mathfrak{n}_{\mathfrak{a}} =[\mathfrak{a}]+\text{lower terms}, \tag{84} \end{equation}$$

where the matrices $\mathfrak{e}_{i, t}$ are completely determined by the conditions $\operatorname {ro}( \mathfrak{e}_{1, N-1})=\operatorname {ro}( \mathfrak{a})$ and $\operatorname {co}(\mathfrak{e}_{1, 1}) = \operatorname {co}(\mathfrak{a})$ and the signs of $\mathfrak{e}_{i, t}$ are inductively determined by the conditions that $s_r(\mathfrak{e}_{1,1}) = s_r (\mathfrak{a})$ and $s_l (\mathfrak{e}_{i, t}) = s_r (\mathfrak{e}_{i, t} ) + (-1)^{ s_r (\mathfrak{e}_{i, t} ) } \operatorname {p}( \mathfrak{ e}_{i , t} )$.

Proof.

The proof is a modification of the proof of Theorem 4.6.3. Recall that $\lambda (\mathfrak{a})$ is defined in the proof of Theorem 4.6.3. We shall prove the theorem by induction on $\lambda (\mathfrak{a})$. When $\lambda (\mathfrak{a})=0$, the statement is trivial.

Fix $\mathfrak{a}$ with $\lambda (\mathfrak{a}) = k+1$. We want to show that the theorem is true for $\mathfrak{a}$ under the induction assumption and the theorem holds for all $\mathbf{b}$ with $\lambda (\mathbf{b}) \leq k$. If $k \geq n+1$, then the argument is exactly the same as that in the proof of Theorem 4.6.3. So we may and will assume that $k \leq n$. Without loss of generality, we further assume that $\operatorname {ur}(\mathfrak{a})$ is even. In this case, $\mathfrak{a}=(A, +)$ with $\lambda (\mathfrak{a}) =k+1$. We set

$$\begin{equation*} \mathfrak{b}_0 = \begin{cases} (B_0, +), & \text{if $a_{n, n+k+1}$ is even},\\ (B_0, -), & \text{if $a_{n, n+k+1}$ is odd}, \end{cases} \end{equation*}$$

where $B_0 = (b_{ij}) \in \Xi _D'$ is defined by

$$\begin{equation*} b_{ij}= \begin{cases} a_{ij}, & \text{if $0\leq j-i < k-1$ or $j-i=k-1$, $i=1$, …, $n$},\\ R_{ij}, & \text{if $j-i = k$, $i\leq n$, or $j-i=k-1$, $i> n+1$},\\ 0, & \text{if $j-i > k$ or $j-i = k$, $i> n+1$}. \end{cases} \end{equation*}$$

By definition, $\lambda (\mathfrak{b}_0) = k$.

For $i=1$, …, $n-1$, let $B_i$ be the matrix such that $B_i-a_{i,i+k+1}E^{\theta }_{i,i+1}$ is a diagonal matrix and $\operatorname {co}(B_i)= \operatorname {ro}(B_{i-1})$. Let $M$ be the matrix such that $M - a_{n, n+k+1} E^{\theta }_{n,n+2}$ is diagonal and $\operatorname {co}(M)= \operatorname {ro}(B_{n-1})$. For $i = n+2$, …, $N-k$, let $B_i$ be the matrix such that $B_i- a_{i,i+k+1}E^{\theta }_{i+1,i+2}$ is a diagonal matrix, $\operatorname {co}(B_i)= \operatorname {ro}(B_{i-1})$, and $\operatorname {co}(B_{n+2}) = \operatorname {ro}(M)$. For $i = n+2$, …, $N-k$, let $C_i$ be the matrix such that $C_i- a_{i,i+k+1}E^{\theta }_{i,i+1}$ is a diagonal matrix, $\operatorname {co}(C_i)= \operatorname {ro}(C_{i-1})$, and $\operatorname {co}(C_{n+2}) = \operatorname {ro}(B_{N-k})$. We set

$$\begin{equation*} \begin{split} \mathfrak{b}_i &= \begin{cases} (B_i, +),& \text{if $a_{n,n+k+1}$ is even},\\ (B_i, -),& \text{if $a_{n,n+k+1}$ is odd}, \end{cases}\quad \forall i=1,\dots ,n-1,\\ \mathfrak{m}&= (M, +),\\ \mathfrak{b}_i &= (B_i, +)\quad \text{and}\quad \mathfrak{c}_i = (C_i, +),\ \forall i=n+2, \dots , N-k. \end{split} \end{equation*}$$

Now applying Corollary 4.6.2, we have

$$\begin{equation*} \overset{n+2}{\underset{i=N-k}{\bigsqcap }}[\mathfrak{c}_i] * \overset{n+2}{\underset{i=N-k}{\bigsqcap }}[\mathfrak{b}_i] * [\mathfrak{m}] * \overset{1}{\underset{i=n-1}{\bigsqcap }}[\mathfrak{b}_{i}] * [\mathfrak{b}_{0}]=[\mathfrak{a}] + \text{lower terms}. \end{equation*}$$

By the induction assumption, the theorem holds for $\mathfrak{a}=(A, +)$. The proof for $\mathfrak{a}= (A, -)$ is entirely similar. This finishes the proof.

■

By Theorem 6.3.1, we can deduce the following results for $\mathcal{S}^m$ similar to those for $\mathcal{S}$.

Proposition 6.3.2.

(a) The algebra $\mathcal{S}^m$ is generated by $[\mathfrak{e}]$ such that either $\mathfrak{e} - R E^{\theta }_{n, n+2}$, $\mathfrak{e}-R E^{\theta }_{i, i+1}$, or $\mathfrak{e} - R E^{\theta }_{i+1,i}$ is diagonal for some $R\in \mathbb{N}$ and $i\in [1, n-1]$.

(b) The algebra $\mathcal{S}^m$ admits a standard basis $\{ [\mathfrak{a}]|\mathfrak{a}\in \Xi '_{\mathbf{D}}\}$, a monomial basis $\{ \mathfrak{n}_{\mathfrak{a}} | \mathfrak{a}\in \Xi '_{\mathbf{D}}\}$, and the canonical basis $\{\{ \mathfrak{a}\} | \mathfrak{a}\in \Xi '_{\mathbf{D}}\}$, where $\mathfrak{n}_{\mathfrak{a}}$ is in 83.

(c) The algebra $\mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{S}^m$ is generated by the functions $E_i, F_i$, $H_a^{\pm 1}$, $J_{\pm }$, and $T$ for any $i\in [1, n-1]$ and $a\in [1, n]$.

Remark 6.3.3.

The basis $\{\mathfrak{n}_{\mathfrak{a}} \}$ of $\mathcal{S}^m$ is not a subset of the basis $\{ \mathfrak{m}_{\mathfrak{a}}\}$ of $\mathcal{S}$ in Corollary 4.6.4. But the canonical basis $\{ \{\mathfrak{a}\} | \mathfrak{a}\in \Xi '_{\mathbf{D}}\}$ of $\mathcal{S}^m$ is a subset of the basis $\{ \{ \mathfrak{a}\} | \mathfrak{a}\in \Xi _{\mathbf{D}} \}$ of $\mathcal{S}$.

6.4. The algebra $\mathcal{K}^m$

Recall that $\Xi _{\mathbf{D}}'=\{\mathfrak{a}\in \Xi _{\mathbf{D}}|\operatorname {ro}(\mathfrak{a})_{n+1} = \operatorname {co}(\mathfrak{a})_{n+1} =0 \}$. Let

$$\begin{equation*} \widetilde{\Xi }'_{\mathbf{D}}=\{ \mathfrak{a}\in \widetilde{\Xi }_{\mathbf{D}}|a_{n+1,j}=a_{j,n+1}=0, \forall j\}. \end{equation*}$$

Let $\mathcal{K}^m$ be the subalgebra of $\mathcal{K}$ spanned by the elements $[\mathfrak{a}]$ for any $\mathfrak{a}\in \widetilde{\Xi }'_{\mathbf{D}}$. Notice that $\mathcal{K}^m$ can be obtained via a stabilization similar to Section 5.1 by using the algebras $\mathcal{S}^m$. Similar to Theorem 6.3.1, we have

$$\begin{equation*} \mathfrak{n}_{\mathfrak{a}} = \mathfrak{a}+\text{lower terms}, \quad \forall \mathfrak{a}\in \widetilde{\Xi }'_{\mathbf{D}}, \end{equation*}$$

where $\mathfrak{n}_{\mathfrak{a}}$ is defined in 83. Moreover, by 49, we have

$$\begin{equation*} [(D_{\lambda } + E_{n,n+1}^{\theta }, \epsilon )]\cdot [(D_{\lambda } +E_{n+1,n}^{\theta }, \epsilon ')] = \begin{cases}[(D_{\lambda }+ E^{\theta }_{n,n}, \epsilon )] , & \text{if} \ \epsilon = \epsilon ',\\ [(D_{\lambda } + E^{\theta }_{n, n+2}, \epsilon )], & \text{if} \ \epsilon \neq \epsilon '. \end{cases} \end{equation*}$$

From this observation, we have the following results for $\mathcal{K}^m$ and $\mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{K}^m$ similar to those for $\mathcal{K}$ and $\mathbb{Q}(v) \otimes _{\mathcal{A}} \mathcal{K}$.

Proposition 6.4.1.

(a) The algebra $\mathcal{K}^m$ is generated by the elements $[\mathfrak{e}]$ such that either $\mathfrak{e} - R E^{\theta }_{n, n+2}$, $\mathfrak{e}- R E^{\theta }_{i, i+1}$, or $\mathfrak{e} - R E^{\theta }_{i+1, i}$ is diagonal for some $i\in [1, n-1]$ and $R\in \mathbb{N}$.

(b) The algebra $\mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{K}^m$ is generated by the elements $[\mathfrak{e}]$ such that either $\mathfrak{e}$, $\mathfrak{e} - E^{\theta }_{n, n+2}$, $\mathfrak{e} -E^{\theta }_{i,i+1}$, or $\mathfrak{e} - E^{\theta }_{i+1,i}$ is diagonal for some $i \in [1,n-1]$.

(c) The algebras $\mathcal{K}^m$ and $\mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{K}^m$ possess three bases: the standard basis $\{ [\mathfrak{a}] | \mathfrak{a}\in \widetilde{\Xi }'_{\mathbf{D}}\}$, the monomial basis $\{ \mathfrak{n}_{\mathfrak{a}} | \mathfrak{a}\in \widetilde{\Xi }'_{\mathbf{D}}\}$, and the canonical basis $\{ \{ A\} | \mathfrak{a}\in \widetilde{\Xi }'_{\mathbf{D}}\}$.

(d) The restriction of $\Psi$ in Theorem 5.3.1 defines a surjective algebra homomorphism $\Psi ^m: \mathcal{K}^m \to \mathcal{S}^m$ such that $\Psi ^m ([\mathfrak{a}]) = [\mathfrak{a}]$ if $\mathfrak{a}\in \Xi _{\mathbf{D}}'$ and $0$ otherwise. Moreover $\Psi ^m (\{ \mathfrak{a}\})=\{\mathfrak{a}\}$ if $\mathfrak{a}\in \Xi '_{\mathbf{D}}$ and $0$ otherwise.

6.5. A presentation of $\mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{K}^m$

To a diagonal signed matrix $\mathfrak{d}=(D_{\lambda }, \epsilon )$ in $\widetilde{\Xi }'_{\mathbf{D}}$, we set

$$\begin{equation} T \mathfrak{d}= F_n E_n \mathfrak{d}- [\lambda _n]_v \mathfrak{d}, {\tag{85}} \end{equation}$$

where $F_nE_n \mathfrak{d}$ is defined in 56 and lies in $\mathcal{K}^m$. Note that $\lambda _{n+1}=0$ in this case.

Proposition 6.5.1.

Let $\mathfrak{d}=(D_{\lambda }, \epsilon )$ and $\mathfrak{d}' = (D_{\lambda '}, \epsilon ')$ be two signed diagonal matrices in $\widetilde{\Xi }'_{\mathbf{D}}$. The following relations hold in $\mathcal{K}^m$:

$$\begin{align*} \mathfrak{d}\mathfrak{d}' &= \delta _{\mathfrak{d}, \mathfrak{d}' } \mathfrak{d},\\ T \mathfrak{d}- \mathfrak{d}' T \mathfrak{d}&= 0, && \text{if $\lambda =\lambda '$, $\epsilon =\epsilon '$},\\ (E_iF_j-F_jE_i) \mathfrak{d}&= 0, && \text{if $i\neq j$},\\ (E_iF_i-F_iE_i) \mathfrak{d}&= [\lambda _{i+1}-\lambda _{i}]_v \mathfrak{d}, && \text{if $\neq n$},\\ (E_{n-1}^2T-[2]_v E_{n-1}TE_{n-1}+TE_{n-1}^2) \mathfrak{d}&= 0,\\ (F_{n-1}^2T-[2]_v F_{n-1}TF_{n-1}+TF_{n-1}^2) \mathfrak{d}&= 0,\\ (T^2E_{n-1}-[2]_v TE_{n-1}T+E_{n-1}T^2) \mathfrak{d}&= E_{n-1} \mathfrak{d},\\ (T^2F_{n-1}-[2]_v TF_{n-1}T+F_{n-1}T^2) \mathfrak{d}&= F_{n-1} \mathfrak{d}, \\ (E_iE_j-E_jE_i) \mathfrak{d}=0, \quad (F_iF_j-F_jF_i) \mathfrak{d}&= 0, && \text{if $|i-j|>1$},\\ (E_iE_iE_j-[2]_v E_iE_jE_i+E_jE_iE_i) \mathfrak{d}&= 0, && \text{if $|i-j|=1$},\\ (F_iF_iF_j-[2]_v F_iF_jF_i+F_jF_iF_i) \mathfrak{d}&= 0, && \text{if $|i-j|=1$}. \end{align*}$$

Proof.

The proposition can be shown by using 85 and Proposition 5.4.2. One could prove them directly by using the same argument as we make for Proposition 5.4.2. More precisely, all identities can be reduced into $\mathcal{S}^m$ by replacing $[\mathfrak{a}]$ by $[{}_p \mathfrak{a}]$. The proposition then follows from Proposition 6.2.1.

■

6.6. The identification $\mathcal{K}^m = \dot{\mathbf{U}}^m$

Recall the algebra $\mathbf{U}^m$ from Section 2.2. Following Lu94, Section 23, we shall define the modified form $\dot{\mathbf{U}}^m$ of $\mathbf{U}^m$. We set

$$\begin{equation*} \Lambda ^m=\{\lambda \in \mathbb{Z}^N|\lambda _i = \lambda _{N+1-i}, \lambda _{n+1}=0\}. \end{equation*}$$

For any $\lambda , \lambda ' \in \Lambda ^m$, we define

$$\begin{equation*} {}_{\lambda }\mathbf{U}^m_{\lambda '} = \mathbf{U}^m / (\sum _{a=1}^{n+1}(H_a-v^{-\lambda _a}) \mathbf{U}^m +\sum _{a=1}^{n+1} \mathbf{U}^m (H_a-v^{-\lambda _a})). \end{equation*}$$

Let $\pi _{\lambda , \lambda '}:\mathbf{U}^m \rightarrow {}_{\lambda } \mathbf{U}^m_{\lambda '}$ be the canonical projection. We set $\operatorname {sgn}(\pi _{\lambda , \lambda } (J_+))=+$ and $\operatorname {sgn}(\pi _{\lambda , \lambda } (J_-) )= -$. Set

$$\begin{equation*} \dot{\mathbf{U}}^m = \bigoplus _{\lambda ,\lambda '\in \Lambda ^m} {}_{\lambda } \mathbf{U}^m_{\lambda '}. \end{equation*}$$

Similarly, we can define $\dot{\mathbf{U}}^{\imath }$ by replacing $\mathbf{U}^m$ by its subalgebra $\mathbf{U}^{\imath }$. (See also BKLW14, Section 5.6.) Following Lu94, Section 23, we have

$$\begin{equation} \dot{\mathbf{U}}^m =\sum _{\mathfrak{d}} \mathbf{U}^m \mathfrak{d}=\sum _{\operatorname {sgn}(\mathfrak{d}) =+} \mathbf{U}^{\imath } \mathfrak{d}\oplus \sum _{\operatorname {sgn}(\mathfrak{d})=-} \mathbf{U}^{\imath } \mathfrak{d}\simeq \dot{\mathbf{U}}^{\imath } \oplus \dot{\mathbf{U}}^{\imath }, {\tag{86}} \end{equation}$$

as vector spaces, where the sum runs over all elements $\mathfrak{d}$ of the form $\pi _{\lambda , \lambda }(J_+)$ or $\pi _{\lambda , \lambda } (J_-)$ for $\lambda \in \Lambda ^m$.

Let $\mathrm{A}_{\mathbf{D}}$ be the associative $\mathbb{Q}(v)$-algebra without unit generated by $E_i \mathfrak{d}, F_i \mathfrak{d}$, $T \mathfrak{d}$, and $\mathfrak{d}$ for all $i\in [1,n-1]$ and $\mathfrak{d}$ runs over all diagonal signed matrices in $\widetilde{\Xi }'_{\mathbf{D}}$, subject to the relations in Proposition 6.5.1. We have

Proposition 6.6.1.

The map $\phi : \mathrm{A}_{\mathbf{D}} \rightarrow \dot{\mathbf{U}}^m$ sending generators in $\mathrm{A}_{\mathbf{D}}$ to the respective elements in $\dot{\mathbf{U}}^m$ is an algebra isomorphism.

Proof.

Observe that all relations in $\mathbf{U}^m$ can be transformed into corresponding relations in $\dot{\mathbf{U}}^m$ by adjoining diagonal signed matrices. By comparing the defining relations of $\mathbf{U}^m$ and those in Proposition 6.5.1, we have that $\dot{\mathbf{U}}^m$ is an associative $\mathbb{Q}(v)$-algebra generated by $E_i \mathfrak{d}$, $F_i \mathfrak{d}$, $T \mathfrak{d}$, and $\mathfrak{d}$ for all $i\in [1,n-1]$, and $\mathfrak{d}$ diagonal signed matrices in $\widetilde{\Xi }'_{\mathbf{D}}$ and subject to the defining relations of $\mathrm{A}_{\mathbf{D}}$. So we see that the map $\phi$ is a surjective algebra homomorphism.

It is left to show $\phi$ is injective. By using the same argument of 86, we have $\mathrm{A}_{\mathbf{D}}\simeq \dot{\mathbf{U}}^{\imath }\oplus \dot{\mathbf{U}}^{\imath }$, as vector spaces. So the map $\phi$ is injective. We are done.

■

Theorem 6.6.2.

The assignment of sending generators in $\dot{\mathbf{U}}^m$ to the respective generators in $\mathcal{K}^m$ defines an algebra isomorphism $\Upsilon ': \dot{\mathbf{U}}^m \rightarrow \mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{K}^m$.

Proof.

By Propositions 6.5.1 and 6.6.1, we see that $\Upsilon '$ is a surjective algebra homomorphism. We observe that $\mathbb{Q}(v)\otimes _{\mathcal{A}} \mathcal{K}^m$ is a direct sum of two copies of $\dot{\mathbf{U}}^{\imath }$ as $\mathbb{Q}(v)$ vector spaces. So we have the injectivity.

■

6.7. The algebra $\mathcal{U}^m$

Recall the algebra $\hat{\mathcal{K}}$ and the notation $0^{\pm }$ from Section 5.5 and $\hat{\mathfrak{a}} (\mathbf{j})$ from 65. We consider the following elements in $\hat{\mathcal{K}}$:

$$\begin{equation*} \begin{split} \mathrm{O} (\mathbf{j}) & =0^+(\mathbf{j})+0^-(\mathbf{j}),\quad \forall \mathbf{j}\in \mathbb{Z}^N,\\ E_i &=E_{i+1,i}^{\theta , +}(0)+E_{i+1,i}^{\theta , -}(0),\\ F_i & =E_{i, i+1}^{\theta ,+}(0)+E_{i, i+1}^{\theta ,-}(0),\quad \forall i\in [1,n-1],\\ T &= \sum _{\lambda } ([D_{\lambda }^++E_{n,n+2}^{\theta }]+[D_{\lambda }^-+E_{n,n+2}^{\theta }]). \end{split} \end{equation*}$$

Let $\mathcal{U}^m$ be the subalgebra of $\hat{\mathcal{K}}$ generated by $E_i, F_i, T, \mathrm{O}(\mathbf{j}), 0^+(0)$, and $0^-(0)$ for all $i\in [1,n-1]$ and $\mathbf{j}\in \mathbb{Z}^N$. By a similar argument as Proposition 5.5.1, we have the following proposition.

Proposition 6.7.1.

The following relations hold in $\mathcal{U}^m$:

$$\begin{align*} \mathrm{O}(\mathbf{j}) \mathrm{O}(\mathbf{j}') &= \mathrm{O}(\mathbf{j}') \mathrm{O}(\mathbf{j}),\ 0^{\pm }(0) \mathrm{O}(\mathbf{j}) = \mathrm{O}(\mathbf{j})0^{\pm }(0),\ 0^{\alpha }(0)0^{\epsilon }(0)=\delta _{\alpha ,\epsilon }0^{\alpha }(0),\\ \mathrm{O}(\mathbf{j})F_h &= v^{j_h-j_{h+1}}F_h \mathrm{O}(\mathbf{j}),\ \mathrm{O} (\mathbf{j})E_h = v^{-j_h+j_{h+1}}E_h \mathrm{O}(\mathbf{j}), \\ E_hT &= TE_h, \quad F_hT=TF_h, \quad && \text{if $h\in [1,n-2]$},\\ 0^{\pm }(0)E_h &= E_h 0^{\pm }(0),\ F_h0^{\pm }(0)=0^{\pm }(0)F_h,\ \mathrm{O}(\mathbf{j})T=T \mathrm{O}(\mathbf{j}),\ 0^{\pm }(0)T=T0^{\mp }(0),\\ F_hE_h-E_hF_h &= (v-v^{-1})^{-1}(0( \underline{h}- \underline{h+1})-0( \underline{h+1} - \underline{h})),\\ E_{n-1}^2T-[2]_v E_{n-1}TE_{n-1}+TE_{n-1}^2 &= 0,\\ F_{n-1}^2T-[2]_v F_{n-1}TF_{n-1}+TF_{n-1}^2 &= 0,\\ T^2E_{n-1}-[2]_v TE_{n-1}T+E_{n-1}T^2 &= E_{n-1},\\ T^2F_{n-1}-[2]_v TF_{n-1}T+F_{n-1}T^2 &= F_{n-1},\\ E_iE_j &= E_jE_i,\quad F_iF_j=F_jF_i, && \text{if $|i-j|>1$},\\ E_i^2E_j-[2]_v E_iE_jE_i+E_jE_i^2 &= 0, && \text{if $|i-j|=1$},\\ F_i^2F_j-[2]_v F_iF_jF_i+F_jF_i^2 &= 0, && \text{if $|i-j|=1$}. \end{align*}$$

By comparing the defining relations and graded dimensions, we have

Corollary 6.7.2.

We have a unique algebra isomorphism $\mathbf{U}^m \to \mathcal{U}^m$ defined by $E_i\mapsto E_i$, $F_i\mapsto F_i$, $T\mapsto T$, $H_a \mapsto \mathrm{O}(- \underline{a})$, and $J_{\pm }\mapsto 0^{\pm }(0)$, for any $i\in [1,n-1]$ and $a\in [1, n]$.