This is a neat calculation that was pointed out to me by Vikas Sindhwani. The exposition here is from the book Reproducing Kernel Hilbert Spaces in Probability and Statistics. $ \newcommand{\abs}[1]{| #1 |} \newcommand{\ind}{\mathbf{1}} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\ip}[2]{\langle #1, #2 \rangle} \newcommand{\R}{\mathbb{R}} \newcommand{\C}{\mathbb{C}} \newcommand{\Z}{\mathbb{Z}} \newcommand{\T}{\mathsf{T}} \newcommand{\Proj}{\mathcal{P}} \newcommand{\Tr}{\mathrm{Tr}} \newcommand{\A}{\mathcal{A}} $

Let $\mathcal{H}$ be a reproducing kernel Hilbert space (RHKS) of functions $f : X \longrightarrow \R$ generated by a positive-definite kernel $k : X \times X \longrightarrow \R$. For a fixed $x \in X$ with $k(x, x) \neq 0$, define the subspace $\mathcal{H}_x \subseteq \mathcal{H}$ as $$ \mathcal{H}_x := \{ f \in \mathcal{H} : \ip{k_x}{f} = f(x) = 0 \} \:. $$ Of course, $\mathcal{H}_x$ is itself a RKHS. What is its associated kernel $k_x$?

More generally, for a $\phi \in \mathcal{H}$ with $\norm{\phi} = 1$, consider the one-dimensional subspace $$ \mathcal{H}_\phi = \mathrm{span}\{ \phi \} \:. $$

Lemma: The reproducing kernel of $\mathcal{H}_\phi$ is $(x, y) \mapsto \phi(x)\phi(y)$.

Proof: We do this by checking the properties of a reproducing kernel. For any $x \in X$, we have $\phi(x) \phi(\cdot) \in \mathcal{H}_\phi$. Now, let $f \in \mathcal{H}_\phi$ and $x \in X$. By definition, we can write $f = \alpha \phi$ for some $\alpha \in \R$. On the other hand, $$ \ip{f}{ \phi(x)\phi} = \ip{\alpha \phi}{\phi(x) \phi} = \alpha \phi(x) \norm{\phi}^2 = \alpha \phi(x) = f(x) \:. $$ And hence, the reproducing property is satisfied. $\square$

We can now describe the reproducing kernel for $\mathcal{H}_x^\perp$. We have that $$ \mathcal{H}_x^\perp = \mathrm{span}\{ k_x/\norm{k_x} \} \:. $$ Also, recall that by the reproducing property, $$ \norm{k_x}^2 = \ip{k_x}{k_x} = k(x, x) \:. $$ Hence, applying the above lemma, the reproducing kernel for $\mathcal{H}_x^\perp$ is $$ (s, t) \mapsto k(s, x) k(x, x)^{-1} k(x, t) \:. $$

We now state a composition rule of reproducing kernels.

Lemma: Let $\mathcal{H} = S \oplus S^\perp$, where $S,S^\perp$ are RKHSes with kernels $k_1, k_2$. The reproducing kernel for $\mathcal{H}$ is $k = k_1 + k_2$.

Proof: Let $f \in \mathcal{H}$. By the orthogonal decomposition, we have that $f = f_1 + f_2$, where $f_1 \in S$ and $f_2 \in S^\perp$. Fix an $x \in X$. We have that $k(x, \cdot) = k_1(x, \cdot) + k_2(x, \cdot) \in \mathcal{H}$. Furthermore, $$ \begin{align*} \ip{f}{k(x, \cdot)} &= \ip{f}{k_1(x, \cdot)} + \ip{f}{k_2(x, \cdot)} \\ &= \ip{f_1}{k_1(x, \cdot)} + \ip{f_2}{k_2(x, \cdot)} \\ &= f_1(x) + f_2(x) \\ &= f(x) \:. \end{align*} $$ Above, the second equality uses the orthogonal decomposition. Hence, $k$ satisfies the reproducing property. $\square$

From this lemma, we conclude that the reproducing kernel of $\mathcal{H}_x$ is $$ (s, t) = k(s, t) - k(s, x) k(x, x)^{-1} k(x, t) \:. $$