I've found what causes the issue. It seems that if I turn the recording on, the video starts to have problems. When I shut the recording off, the video stabilizes and is OK. When the recording starts, the video jumps up and down and the lighting goes up and down as well.
I'm leaning toward the cheap dvr card, I have another PC and will give that a shot before I buy another card. From what I've read the card can do 30fps on 1 camera, adding more cameras, the bandwidth gets divided for each camera, so the recording may put an extra strain on that as well. If I slowed the recording frame rate down, it seems to help a bit but not much.
We're getting camera installed at our workplace, so I asked one of the installers if he's seen that before and asked me if I tried it with the recording off. Smart guy.