Hi Richard and welcome to the BrickWood forums.
You are correct about the heat coming from the oven, since I presume the opening will be away from the fence.
But I do have a couple of concerns, which I’m expressing without seeing your yard layout.
Three feet clearance between the rear of your oven and the fence is about a doorway’s width of clearance. You will find it is tight during your build, and it may prove to be just the wrong amount of space once your oven is built: too much to ignore, but not enough to be comfortable walking through or to store anything substantial. That’s partially practical and partially aesthetic.
The other concern is the one you brought up about smoke reaching the neighbor, because just 3 feet from the fence and 5 feet from the chimney, it’s guaranteed to do that. If you have more space to pull it away from the fence, I’d strongly recommend you do so. Think of the area in between oven and fence as a buffer space, usable for anything from a woodpile to a low storage shed for your cooking supplies.
Every locality is different, and like yours, mine has some specific rules about setbacks. We have to have any outbuildings set back 10 feet from the property line, or 14 feet if the structure is over a certain height. One wonderful provision exempts structures intended for residential recreation from setback requirements (and the example they give is even for non-commercial cooking). So technically I could butt my oven as close to the fence as possible and the town would be silent about it.
But I have the space, so I set it back 14 feet as though it were a taller outbuilding. My neighbor to the East will still get some smoke but in 16 feet the chances are that it will have already started to blend with fresh air and not be a problem except for the delicious aroma.
Once your slab is poured you will lock yourself into a location, so if your layout allows it consider bringing it back from the fence at least a few feet.
Good luck with your slab and for sure we want to hear about your build!