What is making the noise when you turn the key until the
picture on the dashboard (of fuel pump) goes off?

I'm sure it is not the fuel pump making the noise because
fuel pump is located underneath the rear seats, and the
noise is comming somewhere around fuel pressure regulator
valve.

Is that regulator making the noise?