VSO Software You are using the following software?
With your specs you should be able to encode without too much of a problem but I would suspect that the encoding software will keep spiking the system.
If it's what's suggest below doesn't help the only other time I had your problem was when my PSU was failing and the system used to freeze. Once I got a new PSU my problems disappeared. Video encoding is very CPU intensive and will really stress the system if any of the components are failing then you will see freezing.
Prime95http://www.ultimatebootcd.com/If you don't want to run the ultimatebootcd I would suggest running prime95 on your system and see if it freezes, if it does to me that would suggest a hardware problem and my system tweaks etc.... below won't work.
If you can get and run the ultimatebootcd run the memory and cpu burn in tests, if these fail again it's your system.
If you have a spare PSU I would advise swapping and see if your system freezes whilst encoding.
Can you confirm what security software you have active like AV, firewall etc.... It may be wise to exclude your video workarea folder from scanning whilst you encode.
Are you certain your PC is malware free? Is your AV, spyware software up to date and regularly run?
speedxpI've tweaked my system at home with this software it requires no installing.
I just have the top 3 and number 7 selected, have 262144 in the top box. See if these tweaks make any difference.
Process explorerDownload this and have it running before you try encoding see if this software can tell you what's spiking the system.
It will be normal for the video encoder to be running at 99% it could be that you won't be able to do anything with your system until it's finished, have you left your system for several hours to see if you get a encoded file?
I've never used the software you have but I have done plenty of video encoding and my PC is basically unusable while it's running and my PC only has a 2800+ Athlon inside.