Carry the pot: 1 min, but the other three people wait 1 min.

So 1*4=4 minutes.

Carry two pots: 2 minutes, but wait for the other two for 2 minutes each.

2*3=6 minutes

Carry three pots: three minutes, and one person will wait for him for three minutes.

3*2=6 minutes

Carry four pots for four minutes. No one is waiting for him.

Total: 4+6+6+4=20 minutes