Grass is a double hit on performance. On the GPU to draw the grass in the first place, on the CPU to Shadow it.
This is true for everything.
The performance hit from grass can be lots of things. Make sure you don't use transparency AA for one. I believe Skyrim uses instancing for some grass (at least Boris, the author of ENB, believes so) so having tons of grass polys on screen is never the actual issue... until you add stuff like transparency AA. On the CPU side, it seems the Skyrim engine buckles under the weight of having thousands upon thousands of objects in its scene graph. The more objects, the harder the scene graph is to traverse, the longer it takes to update, etc. etc.
What would make sense is if grass meshes were made which consist of several clumps of grass. So if you had a grass NIF with 10 clumps of grass, instead of placing 100,000 grass objects around, you could maybe get away with 10,000. Of course this runs into the issue of looking repetitive.