Liberals and libertarians need to think very carefully about the kinds of preferences/harms should be considered valid policy concerns. The are obvious cases: I wrong you when kick you in the shin, but not when I wear clothes you find distasteful. It seems that this is so even when you have a very high tolerance for shin pain and a low tolerance for fashion crimes, and the harm/disutility is equal in each case. Most people find it reasonable that people have a presumptive right not to be physically attacked, but no such right not to be visually offended by poor taste. There is a large grey area in between these two cases. Utilitarianism as a moral theory is incapable of considering this question, or even admitting that it is a problem. This, more than anything else, is why I am not a utilitarian. The Mill of On Liberty was not a utilitarian in this respect either. On some readings, not even the Mill of Utilitarianism was really a utilitarian.
I'll disagree. Buchanan and Stubblebine provide a very nice framework for thinking about the problem. Sure, the offense I feel when I see someone in a Che Guevara t-shirt is real. And it can be viewed as an external harm imposed upon me just as my wearing my Hayek t-shirt may impose similar harms on others. But the fact of an external effect isn't sufficient to make it policy relevant. Rather, the externality has to be Pareto-relevant: my willingness to pay to avoid seeing Che t-shirts has to be higher than the other guy's willingness to pay to wear the shirt. If the aggregate sum of all willingness-to-pay-weighted distaste caused by Che shirts is higher than the aggregate sum of all willingness-to-pay-weighted pleasure caused by the wearing of such shirts, then a ban could be efficient. Of course, we have no way of extracting that kind of information about preferences outside of market transactions. But in principle there's no conflict with utilitarianism: it's just the generalized problem of the absence of reliable hedonometers.
In some cases we can draw reasonable conclusions though. In the absence of regulations forbidding or mandating the practice, we'd expect that evidence that a bus company allows smelly people on the bus to be evidence that the money-weighted preferences of those imposed-upon aren't high enough to overcome the transactions costs of enforcing a "no smelly people" rule and the losses to the smelly people. Otherwise, the bus company could earn higher profits by banning smelly folks from the bus and charging a higher ticket price for the better service. Similarly, if a grocery store bans entry of customers wearing neo-nazi t-shirts, they must reckon such bans are worth the effort of enforcement and the lost custom from skinheads. In both cases, the common third party (bus service, grocery store) transforms all into internalities.
In general, for externalities the magnitude of which or the enforcement costs of which are very difficult to ascertain apriori, a rule allowing property owners to make the call is most efficient: Coase operates. Where it's pretty likely to weigh one way or the other and transactions costs among all affected parties are likely to be very high, blanket bans or blanket allowances may be more efficient. I'm certainly not saying that such an approach positively explains the pattern of regulation anywhere in the world; rather that such an approach is consistent with utilitarianism.
