Several oven builders on YouTube have used Castable Refractory / Insulating Castable Refractory or Vermiculite / Perlite instead of Stucco to create the exterior pizza oven “shell”.
Castable Refractory, Insulating Castable Refractory and Refractory Mortar don’t fully cure (harden) until it’s been fired (500°+) over several cycles. So applying any refractory materials on the OUTSIDE of your oven is not recommended as it will not properly cure (and remain soft and pliable).
As for the Vermiculite / Perlite & Portland mix - While this combo is GREAT under the firebrick hearth surface, it is just too fragile for a long-term and rock-hard exterior shell that is exposed to the elements.
Stucco on the other hand has thousands upon thousands of fiberglass fibers in the mix which act as tiny rebar in the dry stucco finish. Once the stucco is fully cured, it is nearly impossible to break! And even with the slight chance that the stucco does crack or break, the fiberglass fibers will hold the stucco together with minimal damage.
Finally - the cost. You are going to need 200 lbs of dry stucco for the exterior of your oven at cost of about $40. The same amount of Insulating Castable Refractory would cost you over $200… easily!