It is all about safety and consideration for other road users. I was always taught that if I could see another car on the road ahead, whether it was travelling away from me, or towards me, I should dip my lights. I also always dip lights if I am approaching a crossing or junction, in case a car approaches the junction from another direction.
Also be aware of other road users. A pedestrian blinded by high beams is in danger. Likewise cyclists, riders etc.
Under normal circumstances, normal beams should be enough for you to travel safely, high beams provide better visibility when there are no other road users around who would be affected by them. If you are unable to see properly when driving on normal beams, slow down, quite simply.
It may also be a good idea to consult your optician. I have difficulty in the UK because the lights shine at me from the right and it blinds me. Whereas I don't have the problem in mainland Europe. I have mild astigmatism and I've had to learn to adjust my driving for the UK as a result.
I now drive in Europe and I ALWAYS have my lights on, on a dipped-light setting as it helps other road users to see me. However, I have found that many people don't consider dipping their headlights when coming up behind another car and the dazzling light makes driving very difficult.
What concerns me, is that you said " I heard somewhere that you should turn the headlights down....." This should have been a part of your driving lessons and test. I suggest you look at the link below, it is for the Highway Code. It is always advisable to check this regularly as we all forget things, and the rules change!
Contact the AA or Rospa for additional training. Not only will it improve your confidence on the roads, which I believe is a major cause of your need for high beams, but it may also bring your insurance premiums down!