Vermiculite is used in potting soil as it has excellent water retention properties, so it should probably come as no surprise that the insulation mix using Vermiculite will take quite a few days to dry.
Perlite on the other hand does not retain water like Vermiculite, so the drying time of the Perlite / Portland mix is considerably shorter.