But if you had to guess what would be the maximum size the file could handle per mod?
With regard to the file format: It's not bounded, at least not strongly. There is a record count, which has a maximum of 4 billion records (IIRC), and each record must be under 4GB + 8 bytes, but that allows for a significant amoutn of data.
With regard to memory: With LAA (the "4-gig patch"), slightly under 4GB. Without, slightly under 2. The game would not be playable with the file loaded, due to memory use. This is a hard limit, with no workaround.
As noted by a few people, the only way to get a data file that large is a huge amount of landscape, far more than currently exists. It's very unlikely you'll ever hit either limit.