Add -o Acquire::ForceIPv4=true when running apt-get.
If you want to make the setting persistent just create /etc/apt/apt.conf.d/99force-ipv4 and put Acquire::ForceIPv4 "true"; in it.