I noticed when I was cleaning that there were little chucks of mortar and when I looked up I could see pieces falling off. I pulled a few down but there are still loose sections, should I bang/scrape these down?
No! (but a qualified “no”)
In general, do not scrape or bang inside the oven. One of those chunks could pull out a whole joint with it, and then you have a bigger repair job. That’s problematic because of the risk that the repair job itself could fail. (There are special compounds that can be used for mortar repair; search these forums.)
If you are concerned that some of what looks like “flash” may fall into your food, try a wire brush or your mason’s coarse-bristled brush that is one of the recommended tools. But you must resist the temptation to start picking at the joints, testing them, and so on. Just brush gently and remember that firebrick is fragile and a little brittle.
Do not approach this with a lot of energy, or you will cause some damage that you are sure to regret.