When K2Server.exe eat up memory > Others app down

  • 12 February 2007
  • 20 replies
  • 7 views

Badge +13

Which SQL Server are you running? Are you running both SQL Server, IIS, and K2 Server service on the same machine? If so how much memory are you allocating to SQL Server?

When you look at Task Manager do you see very high Total/Peak commit charge for the swap file? How about the available physical memory remaining?

Even if the company that sold you K2 does not support it anymore you can probably find a regional office/company to help you migrate to later release?

Since you are using SP2 you can't use .NET FW 2.0.

If you have a lot more physical memory, you could try putting /3GB option in boot.ini and that might allow the service exe to use up to 3GB of memory.


20 replies

Badge +3
Problem are solved, thanks all posted in here



Hi

I have a problem with my server which is running K2.net Server (K2Server.exe).
When K2Server.exe eat up memory to around 1 Gb, all website which running on this server will be down.
I try to find out about the problem and found that before all the website down, it will have the following msg in Event Viewer when system is running at peak time (Lots of user entering and working in workspace). :shock: :shock:

ISAPI 'C:WINDOWSMicrosoft.NETFrameworkv1.1.4322aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.

After I restart the K2Server.exe, others would be working fine.
Have any one had face this problem?


After that I try to find the solution in Microsoft Website and found this article:
http://support.microsoft.com/default.aspx?scid=kb;en-us;821268

After read through above article and config as it said. Our problem still persist but with no msg in Event Viewer.

Any Suggestion?

Update
Currently error when K2Server.exe eat up memory and I open web that is in the same server.


System.Web.HttpRuntime.RejectRequestInternal(HttpWorkerRequest wr) +148







System spec for K2Server SP2 (1cpu license):
P4 Xeon 3.6
Ram 3 GB (normally use 2gb but after had error I add 1 more gb)
Raid 5 Hdd
Windows 2003 std Edition
System spec for DB of K2Server
P4 3.0 Xeon
Ram 2 GB
Raid 5 Hdd
Windows 2003 std Edition




PS. My English might not be good.... Cuz I'm not native...
Badge +13
How are you determining it's a memory issue? Did you use Task Manager to check to see how much memory k2server.exe is consuming?

Server too busy sounds like a CPU bottleneck and there aren't enough threads to work either from IIS application pool or ASP .NET setting.

Even though you have 1 CPU license do you have more than 1 CPU on the box?
Badge +3
How are you determining it's a memory issue? Did you use Task Manager to check to see how much memory k2server.exe is consuming?

Yes, I use Task Manager and saw process K2Server.exe use memory up to 1-1.3 gb before it make others website down.

Server too busy sounds like a CPU bottleneck and there aren't enough threads to work either from IIS application pool or ASP .NET setting.

Can you tell me how to determine that this problem is a CPU bottleneck problem or threads problem?
Should I try add more cpu first?

Even though you have 1 CPU license do you have more than 1 CPU on the box?

I do have only one cpu with Hyper-Threading in box(server case)
If I add up one more cpu, do you think it might solve the problem?
Please read below info first :wink:

added up info:
Long time ago, in taskmanager and K2 property (when you right click in K2Service manager and choose property) saw this cpu with HT as 2 cpu and config K2 to use 2 cpu. Once the server down, we need to replace it with a new server which spec of server is as I mentioned in previous one.
New server cannot boot up with an old disk, so I design to repair windows.
After that it make taskmanager and K2 property see cpu with HT as only one cpu (In device manager, it still saw 2 cpu). I don't know whether this will be a problem or not?

I would like to re-install windows and k2 but after an old server is down, it have to change Motherboard so I think it might not beable to use old serial.
Oh!!! Our company donot have any MA because it's expired, so I came here and looking for some help.


Any answer or info you give here would be very good for me th solve problem (may be to others that have same problem as me)
Thanks.




Update Log Before K2Server.exe down with all other app

14-02-50 12:20:45 K2Sql.SaveProcInst: Exception of type System.OutOfMemoryException was thrown.
14-02-50 12:20:45 K2Sql.SaveProcInst: Exception of type System.OutOfMemoryException was thrown.
14-02-50 12:20:45 K2Sql.SaveProcInst: Exception of type System.OutOfMemoryException was thrown.
14-02-50 12:20:45 RunLoadedProcess: A database error occured
Badge +3
I have had a case with K2 open for over six months for this exact issue. K2server.exe slowly consuming memory until 1.1GB is used, at which point K2 becomes unresponsive (processes throut out of memory exception) and the service usually terminates. This happens like clockwork every four business days or so.

Unfortunately, there has been no solution so far. Trace files, network captures, perfmon, etc has yielded no answer to the memory consumption.
Badge +13
Are you on K2 SP4?
Does it matter if you assign K2 to use .NET FW 1.1 or FW 2.0?
What does event log say?

Are you using Win2003 SP1?

In your IIS6 application pool do you have it defined so that if it reaches certain memory threshold it recycles the pool?
Badge +3
We're on K2SP3 currently, but will be upgrading to Sp4 this weekend to see if it helps any. We're on .Net 1.1.

Event log basically just registers that the k2server terminated unexpectedly once it reaches 1.1GB of memory.

Adjusting application pool web garden or recycling settings wouldn't help in this case because it's the k2server.exe process that is eating memory, not the w3wp.exe process.
Badge +3
Are you on K2 SP4?
No, I'm using K2 SP2. I cannot upgrade to newer version one because dealer that sold K2 to our company have no further support this product.

Does it matter if you assign K2 to use .NET FW 1.1 or FW 2.0?
I cannot boot k2 up with FW2. May be some thing wrong about setting. So our K2 is still running on FW1.14

What does event log say?
It said notthing after I set as the solution of MS(about .Net deadlock).

Are you using Win2003 SP1?
Yes

In your IIS6 application pool do you have it defined so that if it reaches certain memory threshold it recycles the pool?
No



Today log
22-02-50 01:58:37 ExecEscalation: A database error occured
22-02-50 01:58:37 K2Sql.LoadProcess: Exception of type System.OutOfMemoryException was thrown.
22-02-50 01:58:37 K2Sql.CloseConnection: Internal connection fatal error.
22-02-50 01:58:37 K2Sql.LoadProcess: Exception of type System.OutOfMemoryException was thrown.
22-02-50 01:58:37 K2Sql.CloseConnection: Internal connection fatal error.
22-02-50 01:58:37 K2Sql.LoadProcess: Exception of type System.OutOfMemoryException was thrown.
22-02-50 01:58:37 K2Sql.CloseConnection: Internal connection fatal error.
22-02-50 01:58:37 K2Sql::LoadProcInst: A database error occured
22-02-50 01:58:37 ExecEscalation: A database error occured
22-02-50 01:58:37 K2Sql::LoadProcInst: Exception of type System.OutOfMemoryException was thrown.
22-02-50 01:58:37 K2Sql.CloseConnection: Internal connection fatal error.
22-02-50 01:58:37 K2Sql::LoadProcInst: Exception of type System.OutOfMemoryException was thrown.
22-02-50 01:58:37 K2Sql.OpenConnection: Exception of type System.OutOfMemoryException was thrown.

in Event Viewer said
The transaction database is offline.

Is this mean it have a database problem?
After I restart K2.net service, it's working fine with cpu usage 100% and memory growing up.
Today, I have to restart it for 3 times. -_-'


Memory Usage before down ~ 1.4 GB


this will be the 4th time I restart K2.net -_-'log as follow

22-02-50 03:47:14 RunProcess: A database error occured
22-02-50 03:47:15 K2Sql::LoadProcInst: Exception of type System.OutOfMemoryException was thrown.
22-02-50 03:47:16 K2Sql::LoadProcInst: Exception of type System.OutOfMemoryException was thrown.
22-02-50 03:47:16 K2Sql.CloseConnection: Internal connection fatal error.
22-02-50 03:47:16 K2Sql::LoadProcInst: Exception of type System.OutOfMemoryException was thrown.
22-02-50 03:47:17 K2Sql.CloseConnection: Internal connection fatal error.
22-02-50 03:47:17 RunProcess: A database error occured
22-02-50 03:47:47 K2Sql::LoadProcInst: Exception of type System.OutOfMemoryException was thrown.
22-02-50 03:47:48 RunProcess: An error occured in the compilation of the process
22-02-50 03:47:48 RunProcess: An error occured in the compilation of the process
22-02-50 03:47:48 RunProcess: An error occured in the compilation of the process
22-02-50 03:47:48 RunProcess: An error occured in the compilation of the process
22-02-50 03:47:48 RunProcess: An error occured in the compilation of the process
22-02-50 03:47:48 RunLoadedProcess: An error occured in the compilation of the process
22-02-50 03:47:48 K2DestQueue.RunDestQueue: The root element is missing.
22-02-50 03:47:48 K2DestQueue.RunDestQueue: The root element is missing.
22-02-50 03:47:48 RunLoadedProcess: An error occured in the compilation of the process
22-02-50 03:47:48 K2DestQueue.RunDestQueue: The root element is missing.
22-02-50 03:47:48 K2DestQueue.RunDestQueue: The root element is missing.
22-02-50 03:47:48 K2DestQueue.RunDestQueue: The root element is missing.
22-02-50 03:47:48 K2DestQueue.RunDestQueue: The root element is missing.
22-02-50 03:47:48 K2DestQueue.RunDestQueue: The root element is missing.
22-02-50 03:47:49 K2DestQueue.RunDestQueue: The root element is missing.
22-02-50 03:57:01 K2DestQueue.RunDestQueue: The root element is missing.
22-02-50 04:05:52 RunLoadedProcess: An error occured in the compilation of the process
22-02-50 04:07:01 K2DestQueue.RunDestQueue: The root element is missing.

memory usage 1.3GB

notthing in event log viewer
Badge +3
This is one possible reason for your memory problem.


32 bit app and operating system has a 2 gb memory per process

The 2 gb memeory is shared by the os and apps.

Calling the K2ROM inside your K2 process WILL CAUSE K2 to have memory leak. This memory leak will not be GC. Eventually the process will exceed 2 gb and K2server.exe will hang or CRASH

Walk through your server event and client events and make sure your are not calling K2ROM or a class wrapper with K2ROM calls

If your process needs to call the K2ROM then
Call a web service that makes K2ROM calls to accomplish this (make sure the web service is on a different box.

Newbie, I feel your pain. Been there myself. Had to get with the people who wrote K2 to solve this.
Badge +13
Wouldn't the web service run in context of w3wp.exe process, thus it's OK for the web service to be on the same server?

-----------------
"Call a web service that makes K2ROM calls to accomplish this (make sure the web service is on a different box. "
Badge +3
Peter Yao
Which SQL Server are you running? Are you running both SQL Server, IIS, and K2 Server service on the same machine? If so how much memory are you allocating to SQL Server?

I seperate into 2 servers
1st is running K2.net with IIS.
2nd is running SQL Server 2000. SQL
total mem is 2 GB
usage mem around 1.3-1.7 GB

When you look at Task Manager do you see very high Total/Peak commit charge for the swap file? How about the available physical memory remaining?
Total/Peak commit charge for the swap file ???
Did you mean Commit Charge(K) ?
If that so...
Right now at normal condition...
Commit Charge(K)
Total 948312
Limit 5088724
Peak 3468372
Physical Memory
Total 3145204
Avaliable 201xxxx
System cache 1192980

I'll update above info again when it's down.

Even if the company that sold you K2 does not support it anymore you can probably find a regional office/company to help you migrate to later release?
Yes, we do ask them for help but our M.A. is expired. Right now our company is working on re-nigotiate for M.A.

Since you are using SP2 you can't use .NET FW 2.0.
OK, thanks. :wink:

If you have a lot more physical memory, you could try putting /3GB option in boot.ini and that might allow the service exe to use up to 3GB of memory.
Our server have total physical memory 3GB.
How or which line to put /3GB in boot.ini?
I would like to try this one. Can you give me an example?

Update
I've got an example of setting.
Here is website
http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx
and more setting on boot.ini
http://www.microsoft.com/technet/sysinternals/information/bootini.mspx
I'll try it on.


lchoing

Err... I don't know how to call K2ROM from k2 process. I just use K2.ProcessInstance.DataFields["something"].Value = some value;
Is this code call to K2ROM ?
In my project properties, there are none of reference to K2ROM.
Badge +9
You are most likely not utilizing K2 context objects exposed by K2 for use within the process definition, which is different from K2ROM.

If you were using K2ROM, you would have to have made an explict reference to it, then programatically open a connection.
Badge +3
Um..

I hope this might help others.

I'm looking at config file of K2 in workspace folder and see
"TimeOut" in Web.configfile .
So I set it to timeout="9999" (default is timeout="99999").
And right now there is no memory problem
I have to see that this setting really effect the memory usage,
so I'll confirm this solution again on friday if server is not down again.


info
my server is running fine for 2 work days + weekend (2 days)
with memory usage around 200-400 mb.

:wink:
Badge +13
How many users do you have on your system?

That's the session state so on server IIS it will maintain the state for 27.7 hours for each user that opens the workspace.

You changed to 2.7 hours, which is still longer than the typical 20 minutes default.

You can also define in the IIS Application Pool for Workspace to recycle when maximum used memory reaches 900 meg.
Badge +3
icon-quote.gifPeter Yao:
How many users do you have on your system?

More than 1,000 users.
icon-quote.gifPeter Yao:
That's the session state so on server IIS it will maintain the state for 27.7 hours for each user that opens the workspace.
You changed to 2.7 hours, which is still longer than the typical 20 minutes default.

So you mean that default setting of K2 is 27.7 hours? and I changed it to 2.7 hours? Am I correct?
How could you compute this??
If I'm correct, why K2 has to set session state for this long?
icon-quote.gifPeter Yao:
You can also define in the IIS Application Pool for Workspace to recycle when maximum used memory reaches 900 meg.

Would this setting effect user when IIS recycle memory?


update situation
Our server still running fine ^^

:wink:
Badge +13
Actually 9999 minutes = 6.9 days.
I am not sure why this number is so high. Let us know in a week if it's still up.

Bump the number down to 720 if it occurs again for a 12 hour session.
Badge +3
Alright, I confirm that my server is still up
from last Friday until now with memory usage ~ 250 - 400 MB


:wink:


If any one have same problem as me, you may try config as what I do.
Your problem may solve!!! :o

Another solution is looking at K2 Service manager and see any long running process(process that stuck in running status when you search) may make a problem with your DB. So, if you have log -> go and check it, if not then try to see you own code that could make unlimit loop.

:wink:


Hope this help!!! :o
Badge +13
How many days have you had this problem prior to the 2/12/2007 post?
Badge +3
icon-quote.gifPeter Yao:
How many days have you had this problem prior to the 2/12/2007 post?


Around 2 weeks before posted in this forum. :o
Badge +9
Just a small pointer here.

I noticed from your error logs that you have this error.

22-02-50 03:47:48 K2DestQueue.RunDestQueue: The root element is missing.

Could you check if there are issues with your destination queues? The best way to verify this in a testing environment is to run the K2.net Server in console mode and do a manual refresh of each queue in K2.net Service Manager. You can see the error in the console if the refreshed queue has an invalid definition. If you manage to isolate which queue has the error, try fixing the error in the K2 solution and re-export the queue again (i.e. the solution with the specified queue).
Badge +3
icon-quote.gifjohnny:
Just a small pointer here.

I noticed from your error logs that you have this error.

22-02-50 03:47:48 K2DestQueue.RunDestQueue: The root element is missing.

Could you check if there are issues with your destination queues? The best way to verify this in a testing environment is to run the K2.net Server in console mode and do a manual refresh of each queue in K2.net Service Manager. You can see the error in the console if the refreshed queue has an invalid definition. If you manage to isolate which queue has the error, try fixing the error in the K2 solution and re-export the queue again (i.e. the solution with the specified queue).



thanks, but that error cause by other flow process which I do not have code to fix it. -_-'
Anyway thanks again :wink:

Reply