The spectral cocycle is an extension of the Rauzy-Veech cocycle, introduced by A. I. Bufetov and B. Solomyak. It is motivated by the study of twisted Birkhoff sums, which have been the focus of many recent works on the spectral measures of dynamical systems.

We study the behavior of the top Lyapunov exponents of the spectral cocycle associated to bijective (in particular, of constant length) substitutions on two letters and their topological subshift factors. We prove that for every topological factor coming from a substitution the top Lyapunov exponent does not increase. We also give an explicit sub-exponential deviation from the expected exponential growth of the spectral cocycle.