I have been trying recently to reencode a bunch of video files to make the file size smaller but not let quality suffer. The source videos are of varying formats, .avi, .wmv, .flv, but I want to standardize my library into .mp4 for compatibility with my phone. .mkv would be ok too since I do use VLC but most are .mp4. So I got handbrake and have been running them through but some turn out bigger than the source file size. Anyone have recommendations on settings or a better program to use, proferably opensource or at least free, that I can just toss a bunch of videoes in and walk away for awhile? I do use an nVidia GTX 680 so if there is CUDA support to make the process faster I would be interested, but really I can just set it and walk away to do something else.

Handbrake is the best video encoder I've found. I'd recommend just playing around with settings and seeing what works and what doesn't. But, generally, make sure the format you are exporting to has the same or less FPS than the source, use this same philosophy for resolution and audio quality. No point in making a 15 FPs file 30 FPS, you will just make it larger and won't improve it. I can't think of anything else off the top of my head, just experiment with a couple of files and be careful of setting the quality higher than the source file, this will just make the file larger.