Problems with Slug size on Heroku when using Asset Sync Gem

Problems with Slug size on Heroku when using Asset Sync Gem

I have a Rails 3.2 app that has a lot of assets. That's OK because I'm
using the [Asset Sync Gem(https://github.com/rumblelabs/asset_sync) which
pushes all my compiled assets to an S3 bucket.
Problem is that these assets are still included in the slug size even
though the application uses the versions on S3. This is causing my slug
size to end up going over the 300MB limit.
I can't use a .slugignore file as this would prevent the files from being
synced, so how can I delete all the assets once AssetSync has pushed them
to S3 and before the slug is compiled?