Applies to Exchange 2000, Exchange 2003, Exchange 2007.
So in Version Store 623 errors, Version Store gets ‘clogged’, if you will, and will fail to process transactions.
624 errors on the other hand, are caused by a lack of available virtual memory on the server. Sometimes this has no impact and the server corrects itself, but in a memory leak condition, this can be the sign your Exchange server is no longer accepting client connections and is in need of some assistance.
In the particular instance where I have seen this occur, the 624 event comes after a series of errors:
First we throw a MSExchangeDSAccess 2104 event.
Event ID : 2104
Raw Event ID : 2104
Record Nr. : 4802384
Category : None
Source : MSExchangeDSAccess
Type : Error
Generated : 9/7/2008 12:27:27 PM
Written : 9/7/2008 12:27:27 PM
Machine : JAHUMBALABAH
Message : Process STORE.EXE (PID=636). All the DS Servers in domain are not responding.
Shortly thereafter you’ll see a MSExchangeDSAccess 2102.
Event ID : 2102
Raw Event ID : 2102
Record Nr. : 4802387
Category : None
Source : MSExchangeDSAccess
Type : Error
Generated : 9/7/2008 12:28:15 PM
Written : 9/7/2008 12:28:15 PM
Machine : JAHUMBALABAH
Message : Process MAD.EXE (PID=2588). All Domain Controller Servers in use are not responding:
JAHUMBALABAH-DC
Then we will see a MSExchangeSA 9152.
Event ID : 9152
Raw Event ID : 9152
Record Nr. : 4802391
Category : None
Source : MSExchangeSA
Type : Error
Generated : 9/7/2008 12:31:15 PM
Written : 9/7/2008 12:31:15 PM
Machine : JAHUMBALABAH
Message : Microsoft Exchange System Attendant reported an error ‘0x8007000e’ in its DS Monitoring thread.
This particular error is an out of memory error. Uh oh.
Then DSAccess has another problem…. a 9154.
Event ID : 9154
Raw Event ID : 9154
Record Nr. : 4802392
Category : None
Source : MSExchangeSA
Type : Error
Generated : 9/7/2008 12:31:20 PM
Written : 9/7/2008 12:31:20 PM
Machine : JAHUMBALABAH
Message : DSACCESS returned an error ‘0x80004005’ on DS notification. Microsoft Exchange System Attendant will re-set DS notification later.
This means a call failed, due to lack of memory…
Then the error you’ve all been waiting for, a 624 gets thrown by ESE.
Event ID : 624
Raw Event ID : 624
Record Nr. : 4802473
Category : None
Source : ESE
Type : Error
Generated : 9/7/2008 12:32:58 PM
Written : 9/7/2008 12:32:58 PM
Machine : JAHUMBALABAH
Message : Information Store (636) Storage Group 1 (First Storage Group): The version store for this instance (1) cannot grow because it is receiving Out-Of-Memory errors from the OS. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until the long-running transaction has been completely committed or rolled back.
Current version store size for this instance: 1Mb
Maximum version store size for this instance: 249Mb
Global memory pre-reserved for all version stores: 1Mb
Possible long-running transaction:
SessionId: 0xBD345AC0
Session-context: 0x00000000
Session-context ThreadId: 0x000015AC
Cleanup: 1
So what can cause this? Check your task manager. Do you see any handle leaks or processes with out of control handles? In the instance I saw for this, it was a mixture of stale messages stuck in the SMTP temp tables and a third-party AV scanner that had an apparent memory leak. Both Inetinfo and Store were over 2 gig and had 32k handles each. Once we resolved the issue Store was around 6k handles and Inetinfo around 3k.
What is happening is a memory leak is consuming all the virtual memory space in Store and Inetinfo, at least in our case here. Yours may differ in what is causing the leak, but I’d bet more than likely its going to be something that ties into Store, such as Anti-Virus, something gumming up IIS and then Epoxy, or something along those lines.
Because you run out of memory, DSAccess starts to fail, then you see the string of errors above.
If you see this, what should you do first and foremost? Give PSS a call so we can help you debug it.
More information on this can be found here:
http://technet.microsoft.com/en-us/library/bb218083(EXCHG.80).aspx
Leave a Reply