I've searched the NCAA rulebook and could not find any such rule that restricts the lineman from going down field before the punt.

Unless I missed it, you can leave as soon as the ball is snapped.

I browsed the rules also and did not see anything that prevents anyone from going downfield in college.

The punter is at risk once he runs with the ball and the roughing the penalty call is taken out of the mix as long as the guy hit him while trying to block the punt. If he hit him after he had already punted by more than just a milisecond I'm sure they would call a personal foul.

I honestly don't ever remember seeing a penalty in college football for someone leaving too soon on a punt.

And I watch a lot of college football...

I say it because I know that is an NFL rule and NOT a NCAA rule. Please tell me where you've seen this rule. Watch the college games closely this weekend when teams punt. Then report back here what you see. Then, watch when NFL teams punt and note the difference. I coached pee wee football for years. We followed the NCAA rules and there was no rule restricting how many or which players could be downfield before the ball was punted.

And the reason the punter does not wait until everyone is 10 or 20 yards downfield is because there are also 11 guys on the other side, some of whom are likely trying to get to him and knock his block off or block his kick! This is why college teams should ALWAYS try to put some pressure on the punter! I can't stand when LSU's punt return team members all lay back to try and set up a return. Not only does it allow the punter to take his time punting the ball, but there are many times when there is a bad snap and the punter has time to recover because there was no pressure. If a few guys had been pushing hard, the punter would not have been able to kick it. And, rushing the punter also occupies blockers for the kick team who, therefore, cannot get downfield to cover.