Ok I was sick of disconnecting my PS4 from the network. I wanted to use my pipboy app on my tablet, so for that I needed network feature. I did lots of testing and stuff, played around with the router and all.
After a while I finally found out a FIX!!!
Here is the fix:
Go to network setting and setup your internet connection.
Choose advance settings
Choose manual setup.
manually input your PS4 ip address and Subnet Mask. (just check internet connection status first to find out your PS4 address and Subnet Mask)
Here's the tricky part!!
on the DNS line use the Google DNS server address 8.8.8.8 or 8.8.4.4
on the second line use your old DNS.
Or you can also Edit your Router's connection settings to use the Google DNS. (as mentioned by Pobatti) (I tested it on the console and the router as well, both work)
So the problem lies with DNS resolvers I guess. I am with Bell Canada and the Bell DNS make the game freeze.