Unless a herd of Guar come around and eat all your plants. Or the crop fails due to insects, harmful fungi, or poor weather.
Personally, I'm not interested in planting and farming in a TES game. It seems like such a passive activity, I'd rather be out making political mischief and sneaking into people's closets to steal their underwear.
In Ultima Online it is possble to grow and cross polinate your own plants, the process works like this, you water the soil twice to make it soft, plant your seed, add some strength potion and leave it for a day while you go adventuring (the number of plants you plant really affects how busy you will be.), the next day you come back and water the plant again and check for infection, if the plant is sick or has fungus you give it a cure or a poison potion, and if the plant is wilted you give it a heal potion. After 3 days the plant shows some visible form of growth, after 6 days the plant pops out of its pot so it looks like a semi grown plant, then you keep looking after it for another week and during that time the plant produces 8 seeds and/or 8 reagents(ingredients). Also, at the 6 day mark you are able to polinate the plant with any other plant of any other species in order to have the possibility of producing an entirely new and as yet unseen plant species with various properties, you can then use the old plants as decoration or sell them, or let them wilt and die.
If I were to raise 100 plants at a time it is the most time consuming process I have ever seen and I rarely have time to go hunt monsters. That's why in TESV if they include this farming process it should be possible to hire peasants to look after your farm and train them, for if their skill level is low and you have too many plants then most of your crops will fail, if you have highly trained highly skilled farmer peasants then you pay them more and they yield great harvests for you. You can always go in and cross polinate your plants if you want to experiment or you could go without farmers so that you do all the work, up to you and how you want to play, you might choose to never start a farm at all.
Stephen.