Reducing the size of the light source resulted in an even darker image since now fewer rays found their way into the light source. Adding a back wall behind the camera did not seem to help much.
Now using the cos density sampling, and increasing samples to 100 it gets almost worse:
