Morgan Stanley was widely seen as the winner in the deal announced last week to purchase Citigroup Inc.’s stake in Morgan Stanley Smith Barney LLC. But the company could end up on the losing side of its longer-term challenge: Expanding the retail-brokerage business.

Problems with MSSB’s technology platform pose the immediate challenge, but brokers and other observers say the bigger issue is a culture clash that has left legacy Smith Barney reps feeling demoralized, as well as continuing cost-cutting that could cause more of the firm’s brokers to jump ship.

One former Smith Barney broker,…