Thanks to Per Wigren (aka Tuxie), I got a good pointer to why the ratings at the Rails plugin directory were wrong (everything was a 3): bots.
In my attempt to make the functionality accessible, I provided a fallback to the AJAX ratings. Unfortunately, I chose simple links, aka GETs, as the fallback. Apparently I needed a personal experience with the truism of never letting a GET request make a change to your database.
So, I changed the links to be AJAX only, changed the controller to return 403 unless the request is via XHR, and then recalculated the ratings to exclude all the bot-submitted ones. Now not every plugin is rated a 3, and there are about 20,000 fewer votes in the database. :)