Using BlockingCollection<T> with Parallel.ForEach

[Travis Parks]

Hello:

We're in the process of upgrading to .NET 4.0. I am using
Parallel.ForEach with a BlockingCollection<T>. The action executed
triggers more items to be added to the BlockingCollection<T>. I don't
know why, but it appears that jobs will simply stop running, so that
the BlockingCollection<T> stops filling up and so my program locks up.

Strangely, all the items thus far in the collection are consumed; some
simply never get run. It is like Parallel.ForEach is buffering, taking
more than one item at a time. However, I can't predict how many items
I need to add until after job finishes running.

I am right that there is "buffering" going on and is this behavior
expected? Should I find another way of looping through my data?

In my example, I have a dependency graph. I start by plucking off the
leaves and running them, generating new leaves as each node finishes
running. So, I can't add items until after a node runs and I don't
know what nodes will become available until then either.

Any help would be nice. I will try to come up with a simple example
that illustrates the "buffering" issue, if I can.

Thanks,
Travis

 

[Travis Parks]

[...]
Any help would be nice. I will try to come up with a simple example
that illustrates the "buffering" issue, if I can.

You must.  Your question doesn't really make much sense without a
concise-but-complete code example to show what code you've actually
written, what it does, and what you expect it to do instead.

I suspect that you simply misunderstand what a BlockingCollection<T> is
supposed to do, making an assumption that the enumeration of the
collection would block by default, when in fact you have to call
GetConsumingEnumerable() to get that behavior.

But without code to look at, there's no way to know for sure.

Pete

I thought about how to demonstrate this today and came up with a
simply idea. Since I am trying to show that there is some kind of
buffer, I had to have items getting added and a call to CompleteAdding
at different times.

Here is a simple example that only works some times. Most of the time
it never runs the last item. It is like it is waiting for something.
However, without running the last item, it has no way of completing. I
don't know why it would delay unless either the BlockingCollection<T>
or Parallel.ForEach is purposefully buffering.

// break iterations across levels
int[][] jagged = new int[][]
{
new int[] { 1, 2, 3 },
new int[] { 4, 5, 6, 7, 8, },
new int[] { 9 },
};
int level = 0;
object sync = new object();
BlockingCollection<int> collection = new
BlockingCollection<int>();
// just to get us started... 1 - 3
for (int i = 0; i != jagged[level].Length; ++i)
{
collection.Add(jagged[level]);
}
int count = 0;
Parallel.ForEach(collection.GetConsumingEnumerable(), i =>
{
Console.Out.WriteLine(i); // do something with i
lock (sync)
{
++count;
// something happens in the process that adds
more items to BlockingCollection<T>
if (count == 3 || count == 8)
{
++level;
for (int j = 0; j != jagged[level].Length;
++j)
{
collection.Add(jagged[level][j]); //
add 4 - 8
}
}
// something happens to add more items - but
we're out!
else if (count == 9)
{
collection.CompleteAdding(); // we're
done!
}
}
});


In my real-world scenario, even if I detect that there are no more
items, the code will lock between a different level (seems to be based
on the number of items added); for example, calling CompleteAdding
when count = 8. I think it works in the example because there are 5
items in the second array. Changing the size of the arrays or number
of arrays generates different results.

Hopefully, this is more meaningful.

Thanks,
Travis Parks

 

[Travis Parks]

[...]
Any help would be nice. I will try to come up with a simple example
that illustrates the "buffering" issue, if I can.

You must.  Your question doesn't really make much sense without a
concise-but-complete code example to show what code you've actually
written, what it does, and what you expect it to do instead.

I suspect that you simply misunderstand what a BlockingCollection<T> is
supposed to do, making an assumption that the enumeration of the
collection would block by default, when in fact you have to call
GetConsumingEnumerable() to get that behavior.

But without code to look at, there's no way to know for sure.

Pete

I thought about how to demonstrate this today and came up with a
simply idea. Since I am trying to show that there is some kind of
buffer, I had to have items getting added and a call to CompleteAdding
at different times.

Here is a simple example that only works some times. Most of the time
it never runs the last item. It is like it is waiting for something.
However, without running the last item, it has no way of completing. I
don't know why it would delay unless either the BlockingCollection<T>
or Parallel.ForEach is purposefully buffering.

            // break iterations across levels
            int[][] jagged = new int[][]
            {
                new int[] { 1, 2, 3 },
                new int[] { 4, 5, 6, 7, 8, },
                new int[] { 9 },
            };
            int level = 0;
            object sync = new object();
            BlockingCollection<int> collection = new
BlockingCollection<int>();
            // just to get us started... 1 - 3
            for (int i = 0; i != jagged[level].Length; ++i)
            {
                collection.Add(jagged[level]);
            }
            int count = 0;
            Parallel.ForEach(collection.GetConsumingEnumerable(), i =>
                {
                    Console.Out.WriteLine(i); // do something with i
                    lock (sync)
                    {
                        ++count;
                        // something happens in the process that adds
more items to BlockingCollection<T>
                        if (count == 3 || count == 8)
                        {
                            ++level;
                            for (int j = 0;j != jagged[level].Length;
++j)
                            {
                                collection.Add(jagged[level][j]); //
add 4 - 8
                            }
                        }
                        // something happens to add more items - but
we're out!
                        else if (count == 9)
                        {
                            collection.CompleteAdding(); // we're
done!
                        }
                    }
                });

In my real-world scenario, even if I detect that there are no more
items, the code will lock between a different level (seems to be based
on the number of items added); for example, calling CompleteAdding
when count = 8. I think it works in the example because there are 5
items in the second array. Changing the size of the arrays or number
of arrays generates different results.

Hopefully, this is more meaningful.

Thanks,
Travis Parks- Hide quoted text -

- Show quoted text -

I actually went the next step and wrote my own parallel foreach-
statement. It isn't nearly as functional and it probably doesn't run
as efficiently, but at least it works against a BlockingCollection<T>.

private static void parallelForEach<T>(IEnumerable<T> tasks,
int maximumConcurrentTasks, Action<T> action)
{
SemaphoreSlim semaphore = new
SemaphoreSlim(maximumConcurrentTasks, maximumConcurrentTasks);
List<Exception> exceptions = new List<Exception>();
foreach (T task in tasks)
{
lock (exceptions)
{
if (exceptions.Count > 0)
{
break;
}
}
AsyncCallback callback = result =>
{
try
{
action.EndInvoke(result);
}
catch (Exception exception)
{
lock (exceptions)
{
exceptions.Add(exception);
}
}
finally
{
semaphore.Release();
}
};
semaphore.Wait();
action.BeginInvoke(task, callback, null);
}
for (int finishedJobs = 0; finishedJobs !=
maximumConcurrentTasks; ++finishedJobs)
{
semaphore.Wait();
}
if (exceptions.Count > 0)
{
throw new AggregateException(exceptions);
}
}

 

[Travis Parks]

[...]
Here is a simple example that only works some times.  [...]

Honestly, is it really so hard to post a complete program?  Assuming the
code you posted actually operates in an actual program, you already have
the code.  You already copied and pasted _some_ code.  Why not just past
the complete program?

Is it so hard to create a Main method? That code is complete in that
it fully demonstates what I am doing and it requires nothing extra. I
do not perceive your difficulty...

 

[Travis Parks]

It's not that hard.  But I'm volunteering my time and have better things
to do than waste it adding stuff to _your_ code that you should have
included in the first place, on the off-chance that that's really all
that's needed.


Well, good luck with that.  If you can't do the simplest things to help
people who would like to help you, you won't get the help you seek.

And you've been around here for awhile.  You should know better.
Frankly, if you were brand new to this newsgroup, and if I hadn't
already specifically asked for a _complete_ code example, I probably
would've even given you the benefit of the doubt and taken a look.  But
a person gets frustrated doing other people's work for them after awhile.

No doubt you'll think I'm just being a dick.  But that's only because
you aren't looking at the issue from anyone's point of view other than
your own.

Pete

You've also go to realize that you're the only person who seems to
answer my questions. We've had friction in the past. If I had a way of
sending zip files on this forum, I'd probably do that instead. Knowing
whether to include a bunch of using statements, namespaces, class
names, etc. isn't something written in the C# forum handbook.

Some times I think you may be taking on too much responsibility. You
seem to be answering questions for people 24/7. You must have a
personal life. If pasting code into the default console project
template code is too much, I think you needn't worry yourself and take
a break. No one asked you to be the savior of the forum. It is
unfortunate that you seem to be one of the few posters to really ever
respond regularly.

It isn't that I don't think about you whenever you reply. I do! I was
just talking about you with my wife last night. I talk about you with
my coworkers often, too. It has always amazed me that you seem to be
answering my questions in less than a hour on average, regardless of
when I post. I always wonder if you do this for a living or whether
you do this for entertainment. For the former, that would be pretty
awesome. In the other case, I'd say helping people on a forum is more
of a pain as any emotional reward that is provides.

I used to post of the moderated C++ forum. However, I had to stop
because one of the moderators seemed to hate me (as you're starting
to). Either I would ask hypothetical or theoretical questions (rather
than the typical "help me" variety) or I would break one little rule
by accident. You'll notice that I rarely help anyone else; that is
because it feels like most of the time you end up fighting with the
ungrateful bastard you were trying to help or some know-it-all MSMVP
would go off on a tangent or would start a meaningless fight that just
confuses the original poster.

So, sorry about not posting examples most of the time. I am hoping it
rings a bell for someone. I spent close to 2 hours yesterday just
coming up with a way of quickly demonstating my issue. Otherwise, I
would have needed to send you a graph class, a bocking graph class, a
"job" interface and a FileSystemJob, not to mention that Program.cs
file. At my job, it is very hard to isolate the exact code needing
help with. I also have to be careful about what I put on the Internet
because I work for a Federal agency.

So, saying I don't think about other people isn't quite right.
Egocentric, definitely! I just wanted you to know that I don't take
you for granted. So, don't stress yourself. You're allowed to say, "I
don't feel like helping you anymore. Goodbye."

 

[Travis Parks]

[long tangential, speculative, and off-base essay snipped]

This post stopped being about threading quite some time ago. It is
about how to post examples. I'd say the tangent is the only thing
relevant in this post!

Ah, but you do.  That's exactly what you say when you refuse to do the
simplest things to post a code example that is useful as-is.  You are
telling us all (not just me, but the half-dozen or more other people who
also answer questions here) that you value the assistance so little that
you can't be bothered to put even the smallest effort into formulating a
question that is completely self-contained.

Actions speak louder than words.

If it were my whole life to make perfect examples, I would do that.
I've got better things to do, and so I don't expect the best answers.
Honestly, when I create tickets on sites like Telerik (with actual
support) I create entirely new projects that directly show what my
issue is. Zip it up and send it off. There I have an issue keeping me
from getting my job done. I don't have that luxury here. I mean, if
you want me to write this:

using System;

namespace Example
{
class Program
{
static void Main()
{
...
}
}
}

I can do that. However, you get that for free just by creating a new
console application. Why should anyone bother with boiler plate? It
just means more scrolling. Do you want me to include AssemblyInfo.cs,
too? I can't know what you expect at a minimum until you request it.
But what about Bob? What does he expect? There is no such thing as a
self-contained C# program that can be copied an pasted on this forum;
someone has to be willing to open up an editor or VS... and I pray
someone with an MVP status has VS.

And I think you missed the entire point of my last post. Honestly, I
don't want your help if you think it is too much trouble or preaching
to me about proper posting etiquette. Posting is such a small piece of
my life, it is a wonder I spend time responding to you at all. No one
is forcing you to help people, so stop complaining when they don't
meet your standards. As you can see, I was able to figure out a
solution without anyone's help. I was more interested in whether this
was a known issue and whether there was a workaround.

You seem to need to belittle people and to always have the last word.
I don't know you and we've never talked in person; however, I get the
impression you prefer situations where you are the answer guy. When it
has come to knowledge about .NET, your answers are always absolute.
You're a .NET guru! But, when it comes to things like style,
convention, etc., I almost always disagree with you. You and I have
totally different experiences working in the same industry, which I
find really odd. I some times wonder if you are one of those people
who know everything but have no idea how to use it.

Perhaps you're not the control freak like you seem. Perhaps you're the
nicest, sweetest guy in the world. However, my overall opinion of you
is not a positive one. Everytime you respond, the answer is not worth
the "attitude" I seem to pick up. For instance, take a look at your
initial response: you immediately claim my question doesn't make
sense. Then you proceed to state that I probably don't understand
BlockingCollection<T>. You are being very rude, are you not? You
assume I am incapable of formulating a question and then you assume I
lack knowledge. So, in other words you think I'm some kind of idiot.
Well, I'm not. I wrote my own BlockingCollection, for goodness sakes!
I some times wonder if it is you who lacks the ability to understand
questions that require more than one sentence to write...

I think you really need to evaluate your goal, here. I assume you are
trying to help people. You're not helping me; you're condemning me.
Actions speak louder than words.

So, why should I appreciate your feedback? You don't ask help from
someone who belittles you; you avoid them.

 

[Travis Parks]

[...]
I can do that. However, you get that for free just by creating a new
console application. Why should anyone bother with boiler plate? It
just means more scrolling. Do you want me to include AssemblyInfo.cs,
too?

Why would I?  You don't need it in order to compile a self-contained C#
program.

I can't know what you expect at a minimum until you request it.

I have, over and over, posted several different links to articles on the
web that explain in precise detail what constitutes a
"concise-but-complete" program, and why it's so important that people
asking questions be willing to post one.

Your lack of knowledge of the topic is due to nothing other than your
own failure to read the information provided to you.

What you post on some external website is irrelevant. If the forum has
not laid down guidelines as to how to post questions, you have no
authority to dictate how. You seem to think you own this forum. Your
participation does not give you any inherent rights over anyone else.

That's simply not true.  I post self-contained programs all the time.
You can compile them from the command line if you like, having copied
and pasted them into an empty text file.  You can even do it for Forms
applications, simply by moving all of the relevant code into a single file.

You still need assemblies, references to them, the compiler, the
knowledge to compile and run the code. Sure, you can send someone some
text, but it is that person's responsibility to make use of it. You
are missing the point again. It is not a matter of "yes" and "no", but
of degree. What constitutes enough? Some website isn't the standard I
chose to post by. There is something called discretion.

More to the point though, when you _don't_ post a complete program, the
reader has no way to know how much effort it will be to get the thing
working.  It might just be a matter of pasting it into an empty console
program.  Or it could be missing a huge amount of supporting structure.
  There's no way to know until you get into the middle of trying to get
it to work, wasting one's time if it's not simple.

When you perceive something as complete is your own limitation. Some
people might have been willing to simply take my original post as
enough and create their own program to demonstrate it. Others wouldn't
have helped me if I gave them a .zip file and documentation. You are
playing the stock market, here, trying to determine at what point it
is worth your investment. Perhaps you've found ways to better your
chances. But in any case, you can never know ahead of time. Suppose my
example had been "complete", but had some obvious error in it. Would
that have been worth the effort you put into it? To the person you
post to, sure. To you? I doubt any thing you do on this forum benefits
you much beyond a fancy title and ego boosting.

You may think it's not a big deal, but it gets old real fast.  You might
ask one question a month.  But the person answering the question is
often answer more than one question a day.

Stop complaining or stop "helping" people. You put this onto yourself.

[pointless, hostile, ad hominem attack deleted]

You should review what you type carefully. I cannot interpret your
responses in any way BUT demeaning. Marking blocks of my posts with
your personal opinions is clear evidence of that. You seem incapable
of understanding my questions and you seem incapable of absorbing my
criticisms. You may think you are reading clearly what I am trying to
say, but you are not. This post has become more about our relationship
and the obvious breakdown in communication. You continue to avoid that
and seem to think you hold something over me, as if I need you. In the
past few months, you have provided very little help, except to
motivate me to find my own answers.

There's been no belittling of you by me or anyone else here.  In fact,
I've never written a single thing that should be construed as any sort
of attack on you at all, in contrast to your own willingness to devolve
into insults and insinuations.  You should get that chip off your shoulder.

You may be unable to tell that you are belittling me, but I can sense
it in every response you have. Perhaps you come from a different
culture than me, unaware that your use of the english language results
in hostility. I don't know the reason. Your denying it can only mean
that 1) you are delusional, 2) you are expected to perform a certain
way, 3) you are unaware of how to formulate sentences in a condusive
way or 4) you are just trying to become the victim in a situation that
you provoked.

Pete

Someone needs to tell you that your method of assisting people is
poor. In all of our experience, I cannot see why anyone would nominate
you for an MVP. You should not make the amount of work you put on
yourself everyone else’s problem. Helping people is voluntary. If you
don’t wish to assist someone, simply don’t. If your MVP status or your
job depends on it, I’d recommend resigning that position.

If creating a new project is too cumbersome for you, don’t do it. I
get the impression that you are extremely anal, but you must realize
that people are fallible. You deal with dozens of posts a day, which I
know is frustrating, so you already know that. I used to make regular
posts to the C++ forums. However, I stopped because I realized how
little benefit there was to it, how much emotional baggage there was
dealing with people like you and how much of my personal life it took
away. Helping people requires more patience than I can produce, simply
put.

I have communicated with many of the common posters on this forum. I
have never had a problem with any of them, except you. I find that
most people have an open and fair way of responding. Perhaps it is
just your choice of words, but I have always found you offensive. I
find it unprofessional and disheartening to criticize someone on
something as impersonal as a forum, and I apologize. I want my
criticism to be constructive, but I am failing to get through to you.

It doesn’t help that I feel this way when you are being fair. I am
ready to start fighting before I even submit the first post. Almost
guaranteed, you will respond and that response will be demeaning. In
one sense, it is really comforting to know that someone will at a
minimum respond. However, it is troubling to know that that person
will be you and that somehow you will find a way to fault my post. At
one point I used to post to this forum quite often, but now only do so
when I am generally curious or lost. Otherwise, it isn’t worth the
battle and criticism.

I think you enjoy arguing with me. I think you enjoy it more than
answering questions, even. Wouldn’t it be a wonderful feeling if I
submitted to your reasoning and admitted how right you are? Well, you
know people aren’t like that; most people on issues concerning
personal failings will never back down, regardless of how hopeless
their position is. I honestly believe that while you are trying to
help me, you are being far too anal about completeness. When you
complain about dealing with as many people as you do, it just tells me
that you have risen yourself to too high a responsibility. Suddenly,
you think everyone should bend to what best suites you. Dude, you’re
not that important; the masses always outweigh the one. Get it through
you head: you do not control this forum and you do not control me.

If you want to help me, fine; do it. If you want me to slap a few
extra lines on there, ask. If I don’t comply, don’t help. It is better
to go without help than to argue with a tyrant.

Personally, I would prefer it if you didn’t answer my questions from
now on. Perhaps seeing that no one has responded, someone with better
interpersonal skills will help instead. I would find it a much
friendlier forum if you would not try to help me. However, if you wish
to continue this on-going blather, I am glad to keep responding.
However, we should probably rename the post to something like: two
nerds duke it out over meaningless forum argument. You talk to me
about perspective… try looking away from your monitor for a minute.