I think there's a middle ground. I think cars need to be able to off track safely in order to take avoiding action when someone tries to pass when they shouldn't (and probably get penalized).
Penalizing someone for going off track when they're not passing/defending seems like it should be okay to me.
I also like the idea of there being a penalty for going off road. Low grip surface, curbs that slow a car. I don't like cars being beached in gravel, or curbs that will damage a car, just because that requires a safety car.
I don't really have a problem with the current rules though. The limits are the limits; if you can't drive in the limits, then you're not doing your job. If no one can, then the track designers and stewards aren't doing their job.