First - I wouldn't consider a Vitara with AllGrip a 4x4, or even a 4WD, it's AWD or All Wheel Drive, and yes, there is a difference. No, the AllGrip can not be turned off, and even if it could, it wouldn't change the negatives that are mentioned below, it would simply remove the positives.
Second, and I recognize that your idea of going "off road" may be different from mine, but I wouldn't consider it as suitable for going "off road" - primarily because of the lack of clearance underneath the vehicle - mine cannot clear a standard oil drain pan, something which is not an issue with any of the previous generations of Vitaras or Grand Vitaras.
I can't comment on taxation or insurance aspects, those are going to be country specific, but, yes, they are more expensive to maintain, not so much as in the routine maintenance, but from the point of view that there is more to go wrong. One aspect of the routine maintenance stuff that is different between 2WD & AWD/4WD, a potentially expensive one is that with 2WD you can replace tires in pairs, but with AWD/4WD, you're going to want to do in sets of four or five, if you have a full sized spare.
I have not driven a 2WD version of the Vitara, so I can't give you a comparison, but I would expect the consumption on 2WD to be slightly less, it may or may not be enough to make a difference, comfort of the drive is subjective and will be impacted by road conditions which are certainly not going to be the same here as where ever you are. I don't expect there to be a big difference in comfort between the 2WD and the AWD versions, however there will be a difference in the way they handle, especially on a wet or slippery surface, AWD versions will be more "sure footed" pulling away from a stop, especially if you're turning at the same time.
Perhaps approach it as aussie11 has done, do you need the AllGrip or is it just nice to have - in my mind the lack of clearance prevents the Vitara from being considered as suitable for anything other use in the city and the suburbs, and I don't need AWD for that.