So i'm making a house that has those little windows to let "the light" in. Only problem is all I can see is that default blue/gray color. How do I change that? Is it something that gets drowned out by light beam lighting affects?

You can drown it out by beams and other FX (I strongly recommend looking at vanilla levels for examples of what objects to use, and some ideas on how to use them), or you can actually enable a skybox and set it to a matching weather region. It's been a while since I've used the CK, so I forgot exactly how it's one, so forgive my vagueness, but:

Somewhere in the cell properties there is a checkbox that says something like 'Use Sky' or 'Has Sky' or something. If you check that, it enables a dropdown. Then you can pick a region from this dropdown and it will simulate that region's weather in the skybox. If your interior cell is located somewhere near Falkreath, you'd want to pick the weather region that's something like WeatherPineForest. (You can look at 'World > Regions' to see what regions are used where in Tamriel, but the names are usually pretty self explanatory)

Hope that helps!