If I put a number and then an apostrophe in the body of my blog, it breaks the RSS and Atom feed but not the RSS2 feed.
1′ breaks the feed
2′ breaks the feed
a’ doesn’t break the feed.
(you might notice that my RSS and Atom feeds are currently showing an “Error: XML Parse Error” error. That’s because of this post.)
WP 1.5 just came out this week so I won’t bug the developers about it. In a month or three (after they’ve worked the bugs out of it… there are always bugs) I’ll probably install it.