The relative trace formula is a tool for studying automorphic representations on symmetric spaces. In this work two relative trace formulas are studied: One for GSp(2) relative to a form of SO(4) and one for PGL(n) relative to GL(n-1).
The first trace formula is used to study the Saito-Kurokawa lifting of automorphic representations from PGL(2) to PGSp(2), thus characterizing the image as the representations with a nonzero period for the special orthogonal group SO(4,E/F) associated to a quadratic extension E of the base field F, and a nonzero Fourier coefficient for a generic character of the unipotent radical of the Siegel parabolic subgroup.
The second trace formula is used to study the representations pi of PGL(n), which have a nonzero linear form invariant under the Levi subgroup GL(n-1), and a certain nonvanishing degenerate Fourier coefficient, over local and global fields. The result is that these pi are of the form I(1n-2,pi'), namely normalizedly induced from the parabolic subgroup of type (n-2, 2), trivial representation on the GL(n-2)-factor and cuspidal pi' on the PGL(2)-factor. The analysis of the formula is intricate our main achievement is rigorously establishing the convergence of the formula. New in this case is that none of the representations of PGL(n) involved are cuspidal, they rather occur in the continuous
spectrum, but discretely in our formula.