Dear Daniel,
I just saw your post. Does the problem still exist?
In your screenshot from microDisplay, I see that you are using the Optronis Cyclone 1HS 3500 camera. This delivers a maximum of 3.518 fps at full resolution.
In both designs you reduce the parallelism to 4. With the given image format (1.280x860) you can process a maximum of 1.100 fps.
So the question is, at what frame rate is the camera running in your application? Please test it with less than 1.100fps. If the design works with that, the problem is the parallelism. To use the full bandwidth of the camera, you need to increase the parallelism.
Best regards,
Oliver