Best way to reduce load on DNS server??

  • Thread starter Thread starter Iain Simpson
  • Start date Start date
I

Iain Simpson

I am making use of a proxy to refresh certain areas of a webpage every
second. From the point of view of the website in question, this is not
a problem, however I feel it must put a significant load on my ISPs
DNS server, making requests every second to refresh.websitename.com
(sometimes www.websitename.com, or cache.websitename.com when grabbing
required graphics etc).

Will this be putting a load on the server that could lead to my IP
address being blocked (can they even do that)? i.e. does the host get
resolved on every request?

What is the best way to minimise the load? Use of the Windows hosts
file?

My ISP DNS server does not seem that reliable and I regularly get
"unable to resolve hostname" errors. I switched to a public DNS server
yesterday and those errors disappeared, however this morning I was no
longer able to use the public DNS server. Could they have blocked my
IP address due to excessive usage?

I understand, and agree, that I should not be using a DNS server other
than that provided by my ISP, it was simply for test purposes to prove
the root cause. I am now in the process of switching ISPs.
 
I am making use of a proxy to refresh certain areas of a webpage every
second. From the point of view of the website in question, this is not
a problem, however I feel it must put a significant load on my ISPs
DNS server, making requests every second to refresh.websitename.com
(sometimes www.websitename.com, or cache.websitename.com when grabbing
required graphics etc).

Will this be putting a load on the server that could lead to my IP
address being blocked (can they even do that)? i.e. does the host get
resolved on every request?
Click to expand...

The amount of load depends on the DNS record and zone TTL for the
hosts at websitename.com. You can find out the TTL by performing an
nslookup from the command prompt in Windows. Yes, someone could block
you from using their DNS server, with varying degrees of
effectiveness.

This doc might help:
http://www.microsoft.com/resources/...rd/proddocs/en-us/sag_DNS_und_HowDnsWorks.asp

(http://tinyurl.com/22nou)
What is the best way to minimise the load? Use of the Windows hosts
file?
Click to expand...

A Windows Hosts file might be a good way to solve the 'host not found'
error.
My ISP DNS server does not seem that reliable and I regularly get
"unable to resolve hostname" errors. I switched to a public DNS server
yesterday and those errors disappeared, however this morning I was no
longer able to use the public DNS server. Could they have blocked my
IP address due to excessive usage?
Click to expand...

You might try nslookup using that server as the default just to see if
you're blocked.

Here's a doc on nslookup if you're not familar with it:
http://support.microsoft.com/default.aspx?scid=kb;en-us;200525

Steve
 
The amount of load depends on the DNS record and zone TTL for the
hosts at websitename.com. You can find out the TTL by performing an
nslookup from the command prompt in Windows. Yes, someone could block
you from using their DNS server, with varying degrees of
effectiveness.
Click to expand...

Thanks for your reply Steve.

It looks like, for whatever reason, the public DNS server I was using
yesterday is no longer available (or at least temporarily
unavailable), as even using online ping services, it is unreachable.
A Windows Hosts file might be a good way to solve the 'host not found'
error.
Click to expand...

But will creating entries in my hosts file in any way lessen the
burden on the DNS server I specify in my router? So, if my proxy is
making a request every second to refresh.betfair.com, will making an
entry for refresh.betfair.com in my hosts file along with it's IP
address, lessen the demands on the primary DNS server I have in my
settings?
 
I am making use of a proxy to refresh certain areas of a webpage every
second. From the point of view of the website in question, this is not
a problem, however I feel it must put a significant load on my ISPs
DNS server, making requests every second to refresh.websitename.com
(sometimes www.websitename.com, or cache.websitename.com when grabbing
required graphics etc).

Will this be putting a load on the server that could lead to my IP
address being blocked (can they even do that)? i.e. does the host get
resolved on every request?
Click to expand...

No. It's cached the first time you look it up.
What is the best way to minimise the load? Use of the Windows hosts
file?
Click to expand...

That's one way, but first determine if you even have a load. I don't
know a way to do this outside of a sniffer or protocol analyzer, but
there may be other methods.
My ISP DNS server does not seem that reliable and I regularly get
"unable to resolve hostname" errors. I switched to a public DNS server
yesterday and those errors disappeared, however this morning I was no
longer able to use the public DNS server. Could they have blocked my
IP address due to excessive usage?

I understand, and agree, that I should not be using a DNS server other
than that provided by my ISP, it was simply for test purposes to prove
the root cause. I am now in the process of switching ISPs.
Click to expand...

You could resolve all of this by not using your "refresh" method. I
doubt you have a load on DNS, but it is generating traffic.

Jeff
 
Thanks for your reply Steve.

It looks like, for whatever reason, the public DNS server I was using
yesterday is no longer available (or at least temporarily
unavailable), as even using online ping services, it is unreachable.


But will creating entries in my hosts file in any way lessen the
burden on the DNS server I specify in my router? So, if my proxy is
making a request every second to refresh.betfair.com, will making an
entry for refresh.betfair.com in my hosts file along with it's IP
address, lessen the demands on the primary DNS server I have in my
settings?
Click to expand...

You always use the same address. Once you look it up the first time,
it's cached and you don't look it up again until the TTL expires or
your cache flushes. Whether you look it up in DNS or Hosts makes no
difference here.

Hosts will be queried before DNS. If the name is resolved there,
there is no DNS request made.

Jeff
 
You always use the same address. Once you look it up the first time,
it's cached and you don't look it up again until the TTL expires or
your cache flushes. Whether you look it up in DNS or Hosts makes no
difference here.

Hosts will be queried before DNS. If the name is resolved there,
there is no DNS request made.
Click to expand...

Thanks for your replies Jeff.

This makes sense on one level, but not on another.

I use a piece of software called Privoxy to refresh the relevant
section of the webpage every second. Privoxy creates a log file giving
details of each request made. Generally that stream involves requests
to refresh.betfair.com. As you can see these requests are being
successful but suddenly a request to the same domain
(refresh.betfair.com) fails with a "could not resolve hostname" error.
From what you have said, that would suggest the TTL for
refresh.betfair.com must be zero, i.e. a DNS lookup is required every
request? Otherwise how could it fail having been cached? If the TTL is
zero, would a hosts entry for refresh.betfair.com be a big help in
reducing the requests to the DNS server?

Apr 02 15:41:03 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938463000&init=0&sb=0
Apr 02 15:41:04 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938464000&init=0&sb=0
Apr 02 15:41:05 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938465000&init=0&sb=0
Apr 02 15:41:06 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938466000&init=0&sb=0
Apr 02 15:41:07 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938467000&init=0&sb=0
Apr 02 15:41:07 Privoxy(02220) Error: could not resolve hostname
refresh.betfair.com
Apr 02 15:41:08 Privoxy(02220) Request:
refresh.betfair.com/BetMgr_Refresh.aspx?ei=1693681&typ=O&iid=0&ts=1080938468000&init=0&sb=0
Apr 02 15:41:08 Privoxy(02220) Error: could not resolve hostname
refresh.betfair.com
 
You could resolve all of this by not using your "refresh" method. I
doubt you have a load on DNS, but it is generating traffic.
Click to expand...

It is indeed generating traffic, but no more than say listening to
music, a radio station, or downloading a movie over the 'net does.
That's what I pay my ISP for.

The need for the regular refresh is to gain a stream of prices, so no
different in terms of traffic than someone signed up to receive
streaming NASDAQ quotes, or whatever. Just a different way of
achieving the same kind of real-time data. I am creating more "up"
traffic than they are, but that's about it.
 
Thanks for your replies Jeff.

This makes sense on one level, but not on another.

I use a piece of software called Privoxy to refresh the relevant
section of the webpage every second. Privoxy creates a log file giving
details of each request made. Generally that stream involves requests
to refresh.betfair.com. As you can see these requests are being
successful but suddenly a request to the same domain
(refresh.betfair.com) fails with a "could not resolve hostname" error.
From what you have said, that would suggest the TTL for
refresh.betfair.com must be zero, i.e. a DNS lookup is required every
request? Otherwise how could it fail having been cached? If the TTL is
zero, would a hosts entry for refresh.betfair.com be a big help in
reducing the requests to the DNS server?
Click to expand...

Just because it's cached doesn't mean something won't prevent it from
reading the cache. Nor does it prevent another type of error that
your app reports as being unable to resolve the name.

Jeff
 
Back
Top