Using CSharpCodeProvider to create MANY assemblies

[Bilz]

I am planning to use the CSharpCodeProvider to generate some compiled
functions in my app.

In my current implementation, All of these functions are generated in
one swipe... thus they all invoke the compiler once, and create only
one DLL. My new requirement tells me that I need to be more
dynamic... that I can't queue up all of these functions... I actually
need to do them more on demand.

The number of times this happens can be as many as 10,000 times...
each with a different calculation. This would mean that I invoke the
compiler 10,000 times and create 10,000 in-memory DLLs. This idea
scares me... can .NET handle that many DLLs being linked to the app?

Does anyone know about the performance overhead of such an endeavor?
I am about to write some tests to gather data, but I was wondering if
anyone out there has experience with this type of execution. Am I
going to run into performance penalties for invoking the compiler that
many times? Am I going to run into performance penalties for having
that many DLLs in the app domain? If so, are there any workarounds
that anyone knows of?

Thanks,
Brian

 

[Nicholas Paldino [.NET/C# MVP]]

Brian,

I would say you are probably going to start seeing some issues when you
try and load 10,000 assemblies into an app-domain. The assemblies
themselves are small, but the CLR is still going to have overhead in the
form of assembly, module, and type info for each assembly.

As for your requirement, does it state that you have to have separate
assemblies, or that you have to be able to call them dynamically (and not
queue it all up).

If the case is the former, then I would say revisit the requirements.
What's the justification for generating all those assemblies?

If the case is the former, I would look into creating an in-memory
assembly, with a single module, and add new global methods/types as needed
and implementing them using IL, instead of the codedom provider and
compiling separately.

 

[Nicholas Paldino [.NET/C# MVP]]

Sorry, the second "former" should be "latter".


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Brian,

I would say you are probably going to start seeing some issues when you
try and load 10,000 assemblies into an app-domain. The assemblies
themselves are small, but the CLR is still going to have overhead in the
form of assembly, module, and type info for each assembly.

As for your requirement, does it state that you have to have separate
assemblies, or that you have to be able to call them dynamically (and not
queue it all up).

If the case is the former, then I would say revisit the requirements.
What's the justification for generating all those assemblies?

If the case is the former, I would look into creating an in-memory
assembly, with a single module, and add new global methods/types as needed
and implementing them using IL, instead of the codedom provider and
compiling separately.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

I am planning to use the CSharpCodeProvider to generate some compiled
functions in my app.

In my current implementation, All of these functions are generated in
one swipe... thus they all invoke the compiler once, and create only
one DLL. My new requirement tells me that I need to be more
dynamic... that I can't queue up all of these functions... I actually
need to do them more on demand.

The number of times this happens can be as many as 10,000 times...
each with a different calculation. This would mean that I invoke the
compiler 10,000 times and create 10,000 in-memory DLLs. This idea
scares me... can .NET handle that many DLLs being linked to the app?

Does anyone know about the performance overhead of such an endeavor?
I am about to write some tests to gather data, but I was wondering if
anyone out there has experience with this type of execution. Am I
going to run into performance penalties for invoking the compiler that
many times? Am I going to run into performance penalties for having
that many DLLs in the app domain? If so, are there any workarounds
that anyone knows of?

Thanks,
Brian

 

[Bilz]

Brian,

I would say you are probably going to start seeing some issues when you
try and load 10,000 assemblies into an app-domain. The assemblies
themselves are small, but the CLR is still going to have overhead in the
form of assembly, module, and type info for each assembly.

No, there is no requirement to create multiple DLLs... just that the
functions be dynamically generated. The functions look to the user
like math equations... wrap their math equations with C# methods,
compile them, and run them against their inputs.

I will look into the idea of inserting methods via IL. I have never
done anything like that. If I can't use C# code, it might defeat the
purpose, since I am using this method to avoid parsing their
functions. They write something like ((input1 * input2)/input3) and I
write that out directly as a C# method with 3 parameters.

Thanks,
B

 

[Nicholas Paldino [.NET/C# MVP]]

Brian,

Well, you would have to parse it anyways to get the parameter names.
Beyond that, you would have to map the operators and whatnot to IL (which
might be no small task).

You might be able to get away with compiling to memory and then taking
the IL from the compiled unit and then placing it dynamically in your type.

I would test creating separate assemblies, and see if you see any
noticable change in your app for the load you would expect to normally
handle. If it works, then I would leave it at that. Only if it doesn't
would I venture down this path.

 

[Bilz]

I would test creating separate assemblies, and see if you see any
noticable change in your app for the load you would expect to normally
handle. If it works, then I would leave it at that. Only if it doesn't
would I venture down this path.

Yes, I agree with that... I am really just investigating right now. I
just don't want to go down a design path that falls apart as it
scales.

I looked into the DynamicMethod class... I thought about what you
said... compile it into a new assembly and transfer the IL over to the
primary assembly. Is there a way to unload the temp assembly? I
don't know that I have ever UNloaded an assembly dynamically.

Thanks for ALL the help,
B

 

[Nicholas Paldino [.NET/C# MVP]]

Bilz,

If you compile the assembly, I don't know that it gets loaded by the
CLR. There might be a managed representation of it in memory, but I don't
know that it's considered loaded.

If it is loaded into memory though, you are going to have to do it in a
separate app domain, so that you can unload it.

Is the parsing that difficult to do? I ask, because if it is not, then
I would recommend building some sort of parser/graph tree for your
expressions and then creating the IL from that.

If you are using .NET 3.5/C# 3.0, you could even use Expressions to help
build your methods for you, as it should make it much, much easier.

 

[Martin Carpella]

I looked into the DynamicMethod class... I thought about what you
said... compile it into a new assembly and transfer the IL over to the
primary assembly. Is there a way to unload the temp assembly? I
don't know that I have ever UNloaded an assembly dynamically.

Assemblies are only unloaded, if the (last) AppDomain using it is
unloaded. So temporary assemblies should be loaded only in an own
AppDomain to allow later unloading.

Best regards,
Martin