syncblocks, perfmon counter keeps going up

[nickdu]

While watching the CLR Memory counters for my application I noticed that the
syncblocks in use counter for the most part continually climbs. It does go
down, but over time it continues to grow. Not sure what upper limit it may
go to. I assume this is related to my hashtable implementation I have.

My question is, once an object is allocated a syncblock does it ever go away
before the object is collected? My hashtable is a cache and therefore sticks
around for the life of the program. So I'm wondering whether the syncblocks
will hit the limit of the number of objects in my cache.
--
Thanks,
Nick

(e-mail address removed)
remove "nospam" change community. to msn.com

 

[nickdu]

Hmm, there seems to be something else going on. I attached WinDbg to my app
while it was running and dumped out the sync blocks. There were 315 of them.
Here are the first couple:

0:005> !syncblk -all
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
1 0018b00c 0 0 00000000 none 012989cc
System.Threading.Thread
2 0018b03c 0 0 00000000 none 012be764
System.Threading.Thread
3 0018b06c 0 0 00000000 none 01299648
Company.Framework.Utility.Common.Holder`1[[Company.Framework.Utility.Common.Pair`2[[System.Threading.ReaderWriterLock,
mscorlib],[IdCache`1+BucketNode[[System.UInt64, mscorlib]], TestIdCache]],
Company.Framework.Utility.Common]]
4 0018b09c 0 0 00000000 none 0129aeac
Company.Framework.Utility.Common.Holder`1[[Company.Framework.Utility.Common.Pair`2[[System.Threading.ReaderWriterLock,
mscorlib],[IdCache`1+BucketNode[[System.UInt64, mscorlib]], TestIdCache]],
Company.Framework.Utility.Common]]
5 0018b0cc 0 0 00000000 none 01298164
System.Object
6 0018b0fc 0 0 00000000 none 012b7b7c
Company.Framework.Utility.Common.Holder`1[[Company.Framework.Utility.Common.Pair`2[[System.Threading.ReaderWriterLock,
mscorlib],[IdCache`1+BucketNode[[System.UInt64, mscorlib]], TestIdCache]],
Company.Framework.Utility.Common]]
7 0018b12c 0 0 00000000 none 0129b220
Company.Framework.Utility.Common.Holder`1[[Company.Framework.Utility.Common.Pair`2[[System.Threading.ReaderWriterLock,
mscorlib],[IdCache`1+BucketNode[[System.UInt64, mscorlib]], TestIdCache]],
Company.Framework.Utility.Common]]
8 0018b15c 0 0 00000000 none 0129a5d8
Company.Framework.Utility.Common.Holder`1[[Company.Framework.Utility.Common.Pair`2[[System.Threading.ReaderWriterLock,
mscorlib],[IdCache`1+BucketNode[[System.UInt64, mscorlib]], TestIdCache]],
Company.Framework.Utility.Common]]


All of them show 0 for MonitorHeld. This code is from a cache class I wrote
that uses a LRU list and my own hashtable. Once you reach the size limit of
the cache I use the node at the end of the list to hold the new item and then
put that item at the front of the list and also update the hashtable. When
the max size of the cache is not exceeded syncblocks stays at 3. When the
max size of the cache is exceeded, thus invoking this reuse of the node at
the end of the list, the syncblocks increase to the number of buckets in my
hashtable. I'm guessing there is something wrong with my code, but don't
know what tools I can use to help me diagnose the problem.
--
Thanks,
Nick

(e-mail address removed)
remove "nospam" change community. to msn.com

Hello Nick

My question is, once an object is allocated a syncblock does it ever go
away before the object is collected?

Yes. The object SyncBlock is reclaimed in certain conditions even if the
object is still alive. There are several articles that illustrate the point
in detail:

http://www.bluebytesoftware.com/blog/2007/06/24/CLRMonitorsAndSyncBlocks.asp
x

<quote>
So now back to the question: when are sync blocks reclaimed?
When a GC is triggered, objects in the reachability traversal may have
their sync blocks reclaimed, even if the object in question is still alive,
and made available again in the system-wide pool of reusable sync blocks.
This reclamation can happen so long as the sync block isn't needed
permanently (as would be the case if COM interop information was stored
inside of it) and the sync block isn't marked precious. A sync block is
precious anytime there is a thread inside of the object's critical region,
when a thread is waiting to enter the critical region, or when at least one
thread has registered its event into the associated condition variable
list. Notice that orphaning monitors can thus lead to leaking events,
because they will remain precious, unless the monitor object itself becomes
unreachable. When a sync block is reclaimed in this fashion, certain
reusable state is kept, like the critical region event object, so that the
next monitor to use the sync block can reuse it.
</quote>

http://msdn.microsoft.com/en-us/magazine/cc188793.aspx

<quote>
When an object is constructed, the object's SyncBlockIndex is initialized
to a negative value to indicate that it doesn't refer to any SyncBlock at
all. Then, when a method is called to synchronize access to the object, the
CLR finds a free SyncBlock in its cache and sets the object's
SyncBlockIndex to refer to the SyncBlock. In other words, SyncBlocks are
associated with an object on the fly when the object needs the
synchronization fields. When no more threads are synchronizing access to
the object, the object's SyncBlockIndex is reset to a negative number, and
the SyncBlock is free to be associated with another object in the future.
</quote>

<quote>
By the way, the SyncBlock cache is able to create more SyncBlocks if
necessary so you shouldn't worry about the system running out of them if
many objects are being synchronized simultaneously.
</quote>

Therefore, if the SyncBlock does not contain COM interop information and it
is not marked as precious, GC can reclaim the SyncBlock. Besides, the
SyncBlock cache can grow if more SyncBlock is needed, so you don't need to
worry the SyncBlocks will hit the limit of the numbers of objects in your
cache.

For additional information on what a SyncBlock contains, you can see
http://msdn.microsoft.com/en-us/magazine/cc163791.aspx.

If you have any questions, please be free to let me know. Have a nice day!

Regards,
Jialiang Ge ([email protected], remove 'online.')
Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/en-us/subscriptions/aa948868.aspx#notifications.

MSDN Managed Newsgroup support offering is for non-urgent issues where an
initial response from the community or a Microsoft Support Engineer within
2 business day is acceptable. Please note that each follow up response may
take approximately 2 business days as the support professional working with
you may need further investigation to reach the most efficient resolution.
The offering is not appropriate for situations that require urgent,
real-time or phone-based interactions. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/en-us/subscriptions/aa948874.aspx
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Jialiang Ge [MSFT]]

Hello Nick

According to the dump, it appears that you have a list of Common.Pair
objects. The type of the list is Common.Holder. Each Common.Pair object is
a pair of ReaderWriterLock and Cache. It seems that all Common.Pair objects
in the list were once locked. When you dump out the sync blocks, all those
locks are freed (but the syncblock are not freed because the Common.Pair
objects are still alive). In other words, no threads are owning those
Common.Pair objects.

To figure out the cause of the problem, I think that we need to first
detect who (what thread) locked those objects and why they were locked.
Because you have the direct access to the source code, can you find out the
places that uses lock? How are you using lock in the list?

If it's not easy to locate the problem using the source code, I think that
a debugger stepping though the codes + SOS!syncblk -all should be the right
diagnostic tools. If you are not comfortable with windbg, you can use
Visual Studio that facilitates "stepping through". Visual Studio can use
SOS too. See:
http://blogs.msdn.com/vijaysk/archive/2007/11/15/sos-your-visual-studio.aspx
..

In addition, the reference source code of .NET can be downloaded from

Shared Source Common Language Infrastructure 2.0 Release
http://www.microsoft.com/downloads/details.aspx?FamilyID=8c09fd61-3f26-4555-
ae17-3121b4f51d4d&displaylang=en

The code of syncblock is in the files clr\src\vm\syncblk.h/cpp. Although
SSCLI is different from the retail bits of .NET Framework, the reference
source can give a rough picture of how syncblock works.

Last but not least, the below analysis is based on a small test of 'lock'
on my side. It's for your references.

lock(obj1)
{ (pos1)
....
} (pos2)

When a thread runs to (pos1), this will set obj1 to be owned by the current
thread. A syncblock will be created for obj1 if thinlock does not apply,
and the thread ownership information will be stored there. In other words,
at this moment the Owning Thread column of !syncblk -all for the object is
not 000000 and MonitorHeld is not 0 either.

MonitorHeld and Owning Thread turn 0 when the thread runs to (pos2), in
other words, when the lock block quits. The syncblock then remains the
state like:

3 002b304c 0 0 00000000 none 01e059b4
_43334936.MyObject

, so that when a new thread attempts to enter the lock block of obj1, the
existing syncblock can be reused.

When obj1 is GC-ed or in other conditions that triggers the free of the
block (this replies on the implementation details of syncblock), the
syncblock entry will be:

3 00000000 0 0 00000000 none 0 Free

Then the free block can be used by other objects like obj2.

Regards,
Jialiang Ge ([email protected], remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

 

[nickdu]

Thanks Jialiang. As I mentioned, this is a cache. All threads hold onto
this one cache object which in turn has this hashtable and list. This
hashtable is an array of these holder objects which are the objects I'm
locking on. Since these are my cache they don't get collected and stick
around for the lifetime of the application. That's why I was asking
originally when should I see the syncblocks perfmon counter go down, when the
lock is over or when the object is collected.

The odd thing is that when I set my cache size to a larger value, larger
than the number of items I put into the cache, the syncblock perfmon counter
stays at 3 even though I'm hitting the same lock while I'm filling up the
cache. So if there are 2500 unique items (words in this case) and my cache
size is 4096, the sync block counter stays at 3 even though this lock is hit
2500 times (I assume on a variety of holder objects: which I'll check into
now). However, when my cache size is 2048 the sync block counter continues
to climb to the number of holder objects I have (which is the cache size /
4). I'll continually hit the lock statement because items get removed from
the cache since the cache size is smaller than the number of unique items.
-----------
Thanks,
Nick

(e-mail address removed)
remove "nospam" change community. to msn.com

 

[nickdu]

I've got a bit more information as I've been debugging. I'm running this on
my single proc laptop machine. When I run two threads I notice this problem.
When I run a single thread the number of sync blocks in use stay at 1.
Also, when I ran with 2 threads and added a Console.WriteLine() to emit some
debug information the sync blocks in use counter shot up to the maxium value
I've been experiencing in just a second or so as opposed to gradually. So it
appears thread contention has something to do with this.
--
Thanks,
Nick

(e-mail address removed)
remove "nospam" change community. to msn.com

Hello Nick

According to the dump, it appears that you have a list of Common.Pair
objects. The type of the list is Common.Holder. Each Common.Pair object is
a pair of ReaderWriterLock and Cache. It seems that all Common.Pair objects
in the list were once locked. When you dump out the sync blocks, all those
locks are freed (but the syncblock are not freed because the Common.Pair
objects are still alive). In other words, no threads are owning those
Common.Pair objects.

To figure out the cause of the problem, I think that we need to first
detect who (what thread) locked those objects and why they were locked.
Because you have the direct access to the source code, can you find out the
places that uses lock? How are you using lock in the list?

If it's not easy to locate the problem using the source code, I think that
a debugger stepping though the codes + SOS!syncblk -all should be the right
diagnostic tools. If you are not comfortable with windbg, you can use
Visual Studio that facilitates "stepping through". Visual Studio can use
SOS too. See:
http://blogs.msdn.com/vijaysk/archive/2007/11/15/sos-your-visual-studio.aspx
.

In addition, the reference source code of .NET can be downloaded from

Shared Source Common Language Infrastructure 2.0 Release
http://www.microsoft.com/downloads/details.aspx?FamilyID=8c09fd61-3f26-4555-
ae17-3121b4f51d4d&displaylang=en

The code of syncblock is in the files clr\src\vm\syncblk.h/cpp. Although
SSCLI is different from the retail bits of .NET Framework, the reference
source can give a rough picture of how syncblock works.

Last but not least, the below analysis is based on a small test of 'lock'
on my side. It's for your references.

lock(obj1)
{ (pos1)
....
} (pos2)

When a thread runs to (pos1), this will set obj1 to be owned by the current
thread. A syncblock will be created for obj1 if thinlock does not apply,
and the thread ownership information will be stored there. In other words,
at this moment the Owning Thread column of !syncblk -all for the object is
not 000000 and MonitorHeld is not 0 either.

MonitorHeld and Owning Thread turn 0 when the thread runs to (pos2), in
other words, when the lock block quits. The syncblock then remains the
state like:

3 002b304c 0 0 00000000 none 01e059b4
_43334936.MyObject

, so that when a new thread attempts to enter the lock block of obj1, the
existing syncblock can be reused.

When obj1 is GC-ed or in other conditions that triggers the free of the
block (this replies on the implementation details of syncblock), the
syncblock entry will be:

3 00000000 0 0 00000000 none 0 Free

Then the free block can be used by other objects like obj2.

Regards,
Jialiang Ge ([email protected], remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================

 

[nickdu]

Here is a sample program which demonstrates the issue:

using System;
using System.Threading;
using System.IO;

public class Pair<F, S>
{
private F _first;
private S _second;

public F First
{
get
{
return this._first;
}
set
{
this._first = value;
}
}

public S Second
{
get
{
return this._second;
}
set
{
this._second = value;
}
}
}

public class Holder<T>
{
private T _object = default(T);

public T Object
{
get
{
return this._object;
}
set
{
this._object = value;
}
}
}

public class Cache<T>
{
private class BucketNode
{
public uint Hash;
public char[] Value;
public T Id;
public BucketNode Next;
public BucketNode Prev;
}

private Holder<Pair<int, BucketNode>>[] _hashtable;

public Cache(int size)
{
this._hashtable = new Holder<Pair<int, BucketNode>>[PowerOf2((uint) size)
/ 4];
}

private static uint PowerOf2(uint value)
{
// Get next higher power of 2 if the value is not already a
// power of 2.

value -= 1;
value = value | (value >> 1);
value = value | (value >> 2);
value = value | (value >> 4);
value = value | (value >> 8);
value = value | (value >> 16);
value += 1;

return value;
}

private static bool AreEqual(char[] v1, char[] v2, int length)
{
if (v1.Length < length)
return false;
if (v1.Length < length)
return false;

for (int i = 0; i < length; ++i)
{
if (v1 != v2)
return false;
}
return true;
}

private T Locate(uint hash, char[] mutableValue, string value)
{
int index = (int) (hash % (uint) this._hashtable.Length);
Holder<Pair<int, BucketNode>> holder = this._hashtable[index];

if (holder == null)
{
lock(this._hashtable)
{
holder = this._hashtable[index];
if (holder == null)
{
holder = new Holder<Pair<int, BucketNode>>();
holder.Object = new Pair<int, BucketNode>();
holder.Object.First = 0;
this._hashtable[index] = holder;
}
}
}

lock(holder)
{
BucketNode bucketNode;
for (bucketNode = holder.Object.Second; bucketNode != null; bucketNode =
bucketNode.Next)
{
if (bucketNode.Hash == hash)
{
if (AreEqual(bucketNode.Value, mutableValue, mutableValue.Length))
{

// If not at front of list then remove. Will
// be added to front shortly.

if (holder.Object.Second != bucketNode)
{
bucketNode.Prev.Next = bucketNode.Next;
if (bucketNode.Next != null)
bucketNode.Next.Prev = bucketNode.Prev;
}
break;
}
}
}

// If we didn't find it then we need to insert or morph.

if (bucketNode == null)
{
T id;

id = default(T);

// If we haven't reached the limit for this bucket then
// allocate new node.

if (holder.Object.First < 4)
{
holder.Object.First++;
bucketNode = new BucketNode();
}

// Else take over the last node.

else
{
bucketNode = holder.Object.Second;
while (bucketNode.Next != null)
bucketNode = bucketNode.Next;
bucketNode.Prev.Next = null;
}

bucketNode.Hash = hash;
if ((bucketNode.Value == null) || (bucketNode.Value.Length <
mutableValue.Length))
bucketNode.Value = new char[PowerOf2((uint) mutableValue.Length)];

// Copy in value.

Array.Copy(mutableValue, bucketNode.Value, mutableValue.Length);
bucketNode.Id = id;
}

// Move to front of list if it's not there already.

if (holder.Object.Second != bucketNode)
{
bucketNode.Next = holder.Object.Second;
if (bucketNode.Next != null)
bucketNode.Next.Prev = bucketNode;
bucketNode.Prev = null;
holder.Object.Second = bucketNode;
}

return bucketNode.Id;
}
}

public T this[string value]
{
get
{
char[] mutableValue = new char[value.Length];
value.CopyTo(0, mutableValue, 0, value.Length);
return Locate((uint) value.GetHashCode(), mutableValue, value);
}
}
}

public class Application
{
private class Worker<T>
{
private Cache<T> _cache;
private string _file;
private WaitHandle _shutdown;

public Worker(Cache<T> cache, string file, WaitHandle shutdown)
{
this._cache = cache;
this._file = file;
this._shutdown = shutdown;
}

public void DoWork()
{
T id;

while (this._shutdown.WaitOne(0, false) == false)
{
StreamReader reader = new StreamReader(this._file);
string line;
while ((line = reader.ReadLine()) != null)
{
string[] tokens = line.Split(' ', '\t', '.', ',', '"', '(', ')', ';',
':', '[', ']', '{', '}', '?',
'!', '_', '&', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0');
if (tokens != null)
{
for (int i = 0; i < tokens.Length; ++i)
{
if (tokens.Length != 0)
{
id = this._cache[tokens];
}
}
}
}
}
}
}

public static void Main(string[] args)
{
ManualResetEvent shutdown = new ManualResetEvent(false);

if (args.Length != 2)
Console.WriteLine("Error, missing cache size and file name.");
else
{
Cache<int> cache = new Cache<int>(int.Parse(args[0]));
Thread[] threads = new Thread[2];
Worker<int> worker = new Worker<int>(cache, args[1], shutdown);
for (int i = 0; i < threads.Length; ++i)
{
threads = new Thread(worker.DoWork);
threads.Start();
}
shutdown.WaitOne();
Console.WriteLine("Waiting for threads to stop...");
for (int i = 0; i < threads.Length; ++i)
{
threads.Join();
}
Console.WriteLine("Threads stopped.");
}
}
}

It takes two command line arguments. The first is the size of the cache and
the second is a text file that contains some sort of words. The code
continues to read through the text file and parses out the works and inserts
them into the cache. The code doesn't do anything useful but you will see
the sync blocks increase to the size of the hashtable array. If I replace
all the statements within the lock(holder) with return default(T); then I
don't see the sync blocks increase to the size of the hashtable array.
--
Thanks,
Nick

(e-mail address removed)
remove "nospam" change community. to msn.com

 

[Jialiang Ge [MSFT]]

Hello Nick

I've been experiencing in just a second or so as opposed to gradually. So
it appears thread contention has something to do with this.

Yes. I have a simpler way to prove that thread contention is related to the
creation of the syncblocks:

static class Program
{
static object obj = new object();

/// <summary>
/// The main entry point for the application.
/// </summary>
static void Main()
{
Thread thread1 = new Thread(new ThreadStart(ThreadProc));
Thread thread2 = new Thread(new ThreadStart(ThreadProc));

thread1.Start();
Thread.Sleep(4000); // avoid thread contention
thread2.Start();

// wait for the quits of the two threads
thread1.Join();
thread2.Join();
}

static void ThreadProc()
{
lock (obj)
{
Thread.Sleep(2000); // cause thread contention
//Debug.WriteLine(obj.GetHashCode());
}
}
}

This example creates two threads that contends to lock obj.

1. If you comment the line "Thread.Sleep(4000)" between the two
thread.Start, you will find one syncblock is created for obj. The reason is
that, if thread1 first locks obj and enters the lock block, it goes to
sleep due to Thread.Sleep(2000). Then, the thread context is switched to
thread2, who attempts to enter the lock block. Because obj is already owned
by thread 1, contention happens, and thus a syncblock is created.

2. If you uncomment the line "Thread.Sleep(4000)", you will find that no
syncblock is created for obj. That's because when thread 2 starts to run,
thread 1 already quits. Thus no contention happens. As an optimization, no
syncblock is created.

Another reason for the creation of syncblock is the call of GetHashCode. It
always generates a syncblock for the obj if none was created in the past.

Please also note that the above analysis is completely based on tests.
Microsoft does not document the behavior of syncblock because its
implementation may change in future.

Regards,
Jialiang Ge ([email protected], remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================