Top federal regulators told a Senate committee that they had investigated Bernard L. Madoff’s brokerage firm in the past, but found no evidence of a vast fraud.