Since capistrano creates a symlink for the assets that is moved across deploys, you really don't need to compile those assets for any deploy where the assets didn't change. Instead, all you need to do is move the symlink. However, the default capistrano for compiling the assets does compile them every time, regardless of whether any assets were changed in the set of commits that you are deploying. The trick, then, is to check the list of files that were changed in the range of commits that are being deployed, and compile the assets only if assets show up in that list. And here is a code snippet that does exactly that:
It only supports git, as that's what I use, so if you use git, just drop that snippet into deploy.rb and enjoy quicker deployments.