http://support.microsoft.com/kb/905865/

according to KB Article 905865 if you're running Windows Server 2003, there is an issue (fixed in Service Pack 2) regarding Working Set trimming that takes place whenever you log on via RDP (even with the /console switch).  If you log on at the physical console, this trimming does not occur.  The problem occurs on W2K3SP1 or W2K3 servers with the patch from KB834293 applied.