I finally figured out what to use for the window sandwich, paper! Specifically some heavy weight drawing paper. I tried it out and was mostly pleased with the result. I used the stained glass cut out for each window as a template and cut out the outline. I then held it in place and marked the areas I needed to cut out from the other side.
I glued the plastic stained glass panel in place with CA and also used CA to glue the paper on top of it. There is a "stain" on the paper from the CA soaking through a bit but paint will cover that. The paper sits lower than the built of "frame" around the window so it looks pretty good. It is a bit tedious with all of the cut and fit involved.
|Cut to fit|
|Cut outs marked|
|Glued in place|