If the front end feel is changing after use on bumpy roads, I suspect your fork oil is overdue for a replacement. Suspension use warms the oil, which results in it thinning out, which explains the change in feel.
The stock fork springs are awfully soft. While spacers are an improvement over stock, your best (moderate budget) option for the forks is new springs and Gold Valve Emulators. There might be a company other than Racetech doing something similar to the GVEs, not sure. Springs are $130, GVEs $180, oil 2L @ $14/L. You might be able to find a better deal through a friendly local shop vs direct - I know I've had luck with that.
Proper springs and GVEs are a huge improvement over just spacers.