Configuration parameter in SQL database

  • Thread starter Thread starter StephaneVarin
  • Start date Start date
S

StephaneVarin

Hi all,

For some reasons, we've had to move away from the configuration files and
are now storing all the configuration parameters for our applications in a
SQL table. We have developped a library to manage the access to these
parameters and all our projects are now using this component. Our
applications vary can be web applications, web services, windows services,
even MS Office Addins.

Now here is the question that we are asking ourselves.

Shoud we or should we not keep the value of the parameters in static members?

Some developpers in the team would rather use dynamic members so that any
change to the underlying parameter in the table would be immediately
reflected without the need to restart the application pool in IIS or the
windows service.

Any comment or advice would be much appreciated to help us make our decision.

Cheers,

Stephane Varin
 
Well, we have the same basic concept here with one of our tables that stores
general configuration data across our applications as well.

If you're storing those values inside static members on the objects the only
way you're going to allow them to update when the database changes is by
reloading the appdomain they're hosted in. Typically this involves restarting
the application or service that is running unless you're managing the
appdomains yourself, which can get very complicated.

I would agree with the developers on your team that want to store the
configuration data dynamically. Your solutions will end up requiring less
maintenance, which in turn gives you time to work on other things. When the
data has to change (which it probably will) you won't need someone to go
around restarting everything.

Just my opinion, but I'd have to agree with your team.
 
There is a middle ground; you could cache the values with a moderate
duration - even a minute or two would save a *lot* of round-trips, while
negating the need to reset the app each time. There are various cache
layers that could do this for you; even the System.Web cache can be used
standalone (since 2.0; not in 1.1). Or kust keep an expiry DateTime.

Marc
 
What about the overhead on our SQL server ?

The way I see it, our applications would be making tens of calls to this
database for each and every action, times the number of users...potentially
thousands of request to retrieve the same value over and over...

Do the benefits (avoiding the need to restart the application when a
parameter is changed) justify this culprit?

Configuration parameters are by essence fairly stable values, and the need
to change them usually arises when deploying a new version of the product,
which implies a restart anyway...

Well, I am very uncomfortable with this idea, I have to admit...
 
Good morning Stephane. Welcome to Microsoft Newsgroup Support Service! My
name is Jialiang Ge [MSFT]. It's my pleasure to work with you on this issue.

I understand you concerns are: how to store&return the configuration
settings of applications without the need to restart the app pool in IIS or
windows service. As you said, neither hard-coded parameters in static
members nor the built-in configuration files (e.g. web.config) meets the
requirement because their updates need the restart of some services. We are
also concerned about the overhead of SQL server if the configurations are
stored in a DB table. Are there any other better choices?

I have three solutions for your references:
1. SQL Server + SQL Dependency Cache
2. txt or xml based setting file + file change callback
3. Windows Registry

---------------1. SQL Server + SQL Dependency Cache-----------
SqlDependency is a feature introduced by .NET 2.0. It can be used in web
applications (SqlCacheDependency) or windows applications (SqlDependency).
SQL Dependency cache stays between your business logic (the retrieval of
configurations) and SQL Server DB (2000 or 2005). The strength of SQL
Dependency is that, when the data in DB is updated, the dependency will be
notified about the change, clear the cache, and force the next retrieval of
the data to query DB. But if the data in DB is not updated, the thousands
of calls of configuration retrieval will not query DB, instead, it gets the
value directly from "cache" in the application's memory. I believe this can
solve your concern of the overhead of SQL server.

SqlDependency is briefly introduced in the MSDN:
http://msdn.microsoft.com/en-us/library/9dz445ks(VS.80).aspx
There are many other materials about it online. If you like this idea,
please tell me you SQL server version (2000 or 2005? SqlDependency is
optimized for SQL server 2005), I will give more samples and documents for
your references.

As a summary of this solution:
STRENGTH:
Good performance, good maintainability, and easy to implement.

WEAKNESS:
If your application itself does not reply on SQL server, SQL server may
look too fat solely for the storage of configuration settings.

-----2. txt or xml based setting file + file change callback ----
Instead of putting the configurations in a .net config file that requires
restart after the update, you may consider storing the values in a txt or
xml files. Our application can read/set the values with .net XML-related
classes. In order to improve the performance for thousands of queries, we
can rely on .net cache + file change callback. File change callbacks are
provided by the .NET class: FileSystemWatcher. It listens to the file
system change notification and raises events when a directory, or file in a
directory, changes.
http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx
I once wrote a small "setting" class based on FileSystemWatcher for txt
setting files at my spare time. (see the attachment with OE or windows
mail) Its code is just for your references.

As a summary of this solution:
STRENGTH:
Only reply on "local file system", thus, it's not as "fat" as SQL server.
Good maintainability, and easy to implement.

WEAKNESS:
If this solution is going to be used in web application, we may need to do
something to prevent the file from being opened directly by the clients.
You can place xml/txt data files, which cannot be requested directly via
http requests to your server, in the app_data directory.

In addition, the file access permission need to be granted to ASP.NET
application (NETWORK_SERVICE).
http://www.codeproject.com/KB/aspnet/Ahmed_Kader.aspx

---------------------3. Windows Registry ---------------------
I recommend this only for windows service or windows form applications. As
far as I know, a lot of such applications put their configurations in the
registry path:
HKLM/Software/[Company Name]/[Application Name],

There might be other better solutions other than the above three. I will do
more researches. Stephane, please let me know if you have any other
concerns, or need anything else. I will do my best to help you reach a
comfortable resolution.

Regards,
Jialiang Ge ([email protected], remove 'online.')
Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
 
Jialiang,

First of all, thanks a lot for your very detailled answer. It was very
useful and instructive.

In particular, and if I really cannot avoid it, I would tend to prefer the
first suggested approach, i.e. SQL + SQL Dependency Cache. I still think it
is a complete overkill, a fairly heavy solution to address a problem that
happens only every once in a while, but at least your solution will prevent
thousands of unecessary calls to our SQL database.

Best regards,
Stephane Varin


"Jialiang Ge [MSFT]" said:
Good morning Stephane. Welcome to Microsoft Newsgroup Support Service! My
name is Jialiang Ge [MSFT]. It's my pleasure to work with you on this issue.

I understand you concerns are: how to store&return the configuration
settings of applications without the need to restart the app pool in IIS or
windows service. As you said, neither hard-coded parameters in static
members nor the built-in configuration files (e.g. web.config) meets the
requirement because their updates need the restart of some services. We are
also concerned about the overhead of SQL server if the configurations are
stored in a DB table. Are there any other better choices?

I have three solutions for your references:
1. SQL Server + SQL Dependency Cache
2. txt or xml based setting file + file change callback
3. Windows Registry

---------------1. SQL Server + SQL Dependency Cache-----------
SqlDependency is a feature introduced by .NET 2.0. It can be used in web
applications (SqlCacheDependency) or windows applications (SqlDependency).
SQL Dependency cache stays between your business logic (the retrieval of
configurations) and SQL Server DB (2000 or 2005). The strength of SQL
Dependency is that, when the data in DB is updated, the dependency will be
notified about the change, clear the cache, and force the next retrieval of
the data to query DB. But if the data in DB is not updated, the thousands
of calls of configuration retrieval will not query DB, instead, it gets the
value directly from "cache" in the application's memory. I believe this can
solve your concern of the overhead of SQL server.

SqlDependency is briefly introduced in the MSDN:
http://msdn.microsoft.com/en-us/library/9dz445ks(VS.80).aspx
There are many other materials about it online. If you like this idea,
please tell me you SQL server version (2000 or 2005? SqlDependency is
optimized for SQL server 2005), I will give more samples and documents for
your references.

As a summary of this solution:
STRENGTH:
Good performance, good maintainability, and easy to implement.

WEAKNESS:
If your application itself does not reply on SQL server, SQL server may
look too fat solely for the storage of configuration settings.

-----2. txt or xml based setting file + file change callback ----
Instead of putting the configurations in a .net config file that requires
restart after the update, you may consider storing the values in a txt or
xml files. Our application can read/set the values with .net XML-related
classes. In order to improve the performance for thousands of queries, we
can rely on .net cache + file change callback. File change callbacks are
provided by the .NET class: FileSystemWatcher. It listens to the file
system change notification and raises events when a directory, or file in a
directory, changes.
http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx
I once wrote a small "setting" class based on FileSystemWatcher for txt
setting files at my spare time. (see the attachment with OE or windows
mail) Its code is just for your references.

As a summary of this solution:
STRENGTH:
Only reply on "local file system", thus, it's not as "fat" as SQL server.
Good maintainability, and easy to implement.

WEAKNESS:
If this solution is going to be used in web application, we may need to do
something to prevent the file from being opened directly by the clients.
You can place xml/txt data files, which cannot be requested directly via
http requests to your server, in the app_data directory.

In addition, the file access permission need to be granted to ASP.NET
application (NETWORK_SERVICE).
http://www.codeproject.com/KB/aspnet/Ahmed_Kader.aspx

---------------------3. Windows Registry ---------------------
I recommend this only for windows service or windows form applications. As
far as I know, a lot of such applications put their configurations in the
registry path:
HKLM/Software/[Company Name]/[Application Name],

There might be other better solutions other than the above three. I will do
more researches. Stephane, please let me know if you have any other
concerns, or need anything else. I will do my best to help you reach a
comfortable resolution.

Regards,
Jialiang Ge ([email protected], remove 'online.')
Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
Click to expand...
 
Forgot to tell. We are using SQL 2005 so any link to good documentation about
SQL dependency cache in SQL 2005 is of course most welcome :-)

Thank you again.

Stephane Varin

"Jialiang Ge [MSFT]" said:
Good morning Stephane. Welcome to Microsoft Newsgroup Support Service! My
name is Jialiang Ge [MSFT]. It's my pleasure to work with you on this issue.

I understand you concerns are: how to store&return the configuration
settings of applications without the need to restart the app pool in IIS or
windows service. As you said, neither hard-coded parameters in static
members nor the built-in configuration files (e.g. web.config) meets the
requirement because their updates need the restart of some services. We are
also concerned about the overhead of SQL server if the configurations are
stored in a DB table. Are there any other better choices?

I have three solutions for your references:
1. SQL Server + SQL Dependency Cache
2. txt or xml based setting file + file change callback
3. Windows Registry

---------------1. SQL Server + SQL Dependency Cache-----------
SqlDependency is a feature introduced by .NET 2.0. It can be used in web
applications (SqlCacheDependency) or windows applications (SqlDependency).
SQL Dependency cache stays between your business logic (the retrieval of
configurations) and SQL Server DB (2000 or 2005). The strength of SQL
Dependency is that, when the data in DB is updated, the dependency will be
notified about the change, clear the cache, and force the next retrieval of
the data to query DB. But if the data in DB is not updated, the thousands
of calls of configuration retrieval will not query DB, instead, it gets the
value directly from "cache" in the application's memory. I believe this can
solve your concern of the overhead of SQL server.

SqlDependency is briefly introduced in the MSDN:
http://msdn.microsoft.com/en-us/library/9dz445ks(VS.80).aspx
There are many other materials about it online. If you like this idea,
please tell me you SQL server version (2000 or 2005? SqlDependency is
optimized for SQL server 2005), I will give more samples and documents for
your references.

As a summary of this solution:
STRENGTH:
Good performance, good maintainability, and easy to implement.

WEAKNESS:
If your application itself does not reply on SQL server, SQL server may
look too fat solely for the storage of configuration settings.

-----2. txt or xml based setting file + file change callback ----
Instead of putting the configurations in a .net config file that requires
restart after the update, you may consider storing the values in a txt or
xml files. Our application can read/set the values with .net XML-related
classes. In order to improve the performance for thousands of queries, we
can rely on .net cache + file change callback. File change callbacks are
provided by the .NET class: FileSystemWatcher. It listens to the file
system change notification and raises events when a directory, or file in a
directory, changes.
http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx
I once wrote a small "setting" class based on FileSystemWatcher for txt
setting files at my spare time. (see the attachment with OE or windows
mail) Its code is just for your references.

As a summary of this solution:
STRENGTH:
Only reply on "local file system", thus, it's not as "fat" as SQL server.
Good maintainability, and easy to implement.

WEAKNESS:
If this solution is going to be used in web application, we may need to do
something to prevent the file from being opened directly by the clients.
You can place xml/txt data files, which cannot be requested directly via
http requests to your server, in the app_data directory.

In addition, the file access permission need to be granted to ASP.NET
application (NETWORK_SERVICE).
http://www.codeproject.com/KB/aspnet/Ahmed_Kader.aspx

---------------------3. Windows Registry ---------------------
I recommend this only for windows service or windows form applications. As
far as I know, a lot of such applications put their configurations in the
registry path:
HKLM/Software/[Company Name]/[Application Name],

There might be other better solutions other than the above three. I will do
more researches. Stephane, please let me know if you have any other
concerns, or need anything else. I will do my best to help you reach a
comfortable resolution.

Regards,
Jialiang Ge ([email protected], remove 'online.')
Microsoft Online Community Support

Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

==================================================
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscriptions/managednewsgroups/default.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscriptions/support/default.aspx.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.
Click to expand...
 
Hello Stephane,

Thank you for the positive feedbacks of my initial reply.

The introduction of Caching in ASP.NET with the SqlCacheDependency Class is
the MSDN article:
http://msdn.microsoft.com/en-us/library/ms178604.aspx
It depicts the main features of ASP.NET SQL cache dependency, and tells the
different implementation of the mechanism in SQL server 2000 and 2005: SQL
server 2005 has the built-in support of Query Notification, whereas, SQL
server 2000 replies on polling which is less efficient.

To use the SQL Cache Dependency in the environment of ASP.NET 2.0 + SQL
Server 2005, please follow these two steps:

*Step 1. (SQL Server) Enable "Query Notification" in SQL server 2005*
http://msdn.microsoft.com/en-us/library/ms172133.aspx
with the SQL command:

ALTER DATABASE [DB Name] SET ENABLE_BROKER;

Note1: Before the run of this command, please make sure all the SQL
connections of the DB are disconnected. Otherwise, the command will hang.
Note2: Certain versions of SQL server 2005 don't include Notification
services:
http://technet.microsoft.com/en-us/library/ms143500.aspx

*Step 2. (ASP.NET) Write code to use SqlCacheDependency*
http://msdn.microsoft.com/en-us/library/9dz445ks.aspx
This is a step-by-step demo of SqlCacheDependency in ASP.NET. If you are
interested in how to use it in Windows application, please refer to:
http://msdn.microsoft.com/en-us/library/a52dhwx7.aspx

A frequently used tip is to put the code:
SqlDependency.Start(GetConnectionString());
Into global.asax Application_Start function:

protected void Application_Start(object sender, EventArgs e)
{
SqlDependency.Start(GetConnectionString());
}

When you call this method, the runtime sets up the relationship between the
application domain and the connection string that will be used later during
the notification registration process.

For more reading of SQL Cache Dependency samples:
http://www.writebetterbits.com/2008/01/overview-of-sql-server-2005-database_
11.html
This is a good demo of SQL Cache Dependency with LINQ

By the way, you may find some articles like these online:
http://www.codeproject.com/KB/aspnet/Caching_in_ASPNET_20.aspx
http://www.codeproject.com/KB/web-cache/CachingDependencies.aspx
They are about SQL Cache Dependency for SQL server 2000, which requires the
commands:
aspnet_regsql -ed -E -d School
aspnet_regsql -et -E -d School -t Users
These are not necessary in SQL server 2005.

Please let me know if you have any other concerns, or need anything else.

Regards,
Jialiang Ge ([email protected], remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================
 
SOmething else to consider.

Greg Leake has created a "mini framework" about this configuration setting
up.

http://msdn.microsoft.com/en-us/netframework/bb499684.aspx


Focus on the configuration framework, and less on the WCF.

Basically, if you update a config setting (in the db), it pushes out all the
information.

Not sure if that's what you're after, but he did alot of work on it.
 
Hello Sloan and Stephane,

I agree that the "mini framework" in the .NET StockTrader Sample, i.e.
"Configuration Service", can be another consideration. As far as I know, the
source code of the configuration service is not published, so I cannot tell
its exact idea of the update callback. It might be using the similar
mechanism to SQL Cache Dependency. For more reading about the service,
please refer to the document:
http://download.microsoft.com/downl...-455290f83274/ConfigServiceTechnicalGuide.pdf

Regards,
Jialiang Ge ([email protected], remove 'online.')
Microsoft Online Community Support

=================================================
Delighting our customers is our #1 priority. We welcome your comments and
suggestions about how we can improve the support we provide to you. Please
feel free to let my manager know what you think of the level of service
provided. You can send feedback directly to my manager at:
(e-mail address removed).

This posting is provided "AS IS" with no warranties, and confers no rights.
=================================================
 
Greg (Leake) has a new version out. It is more friendly then the first
version.
All code is available I believe.

At least that's what he said at TechEd2008.
 
Back
Top