The model only works on the rendered image and motion vectors. Other than the image it has no information about the lighting in the scene, weather or anything else. So in the current form it really doesn’t have much to do with realism.

Nah it’s just that they have blocks of the same size. It probably synchronizes terrain using analogous block types where possible, and player position. Basically it has to convert everything into something similar.