I had similar issues with mine.
Pulled it out (MAJOR job) and found the brushes in poor condition. They cannot be purchased as a part from Sherco but the starter is a common one on many bikes and found some for a KTM that are the same. Fitted them and problem solved.
Side note .... to make the starter removal easy I ground away the small bracing strut on the clutch housing that stops the starter coming out without lifting the motor. Works fine and so far no issues with that brace removed.