G
Guest
Hi all
This is one I've been working on for a few months now. I've got two IIS6 web servers running W2003. Both are member servers in a 2003 native AD. We are trying to "mirror" the content of one server to another using DFS to provide a "hot backup" site if the production one goes down. The production server is in Virginia on a firewalled DMZ. The backup server is in Boston on our internal network. Each site also has a domain controller. Both servers are talking to their respective DC's and can use Kerberos through the firewall. We have the firewall rules to allow the the following ports between the two servers: 445, 139, 135, 88, and random RPC endpoints. This is a Checkpoint firewall, so it also has a specialized RPC rule that's supposed to keep state and allow through the random high port tcp connections that RPC likes to use. You put in different COM application GUID's and it's supposed to track them. I thought I got them all, but maybe I didn't. Also SMB works fine. No problem copying files. Also, there's only one replication connection, from VA to Boston. All changes are made on the VA server, then copied to Boston
The problem is not that it doesn't replicate. It does, but not very well. It will work for a bit, then it will start building up a huge backlog. In the Ntfrs debug logs, we keep getting the following entries on the Virginia server (the server names have been changed to virginiaserver.testcompany.com and bostonserver.testcompany.com)
<FrsHashCalcString: 3076: 4832: S0: 12:52:25> Name = S-1-5-21-1957994488-746137067-839522115-4455
<SndCsMain: 3504: 883: S0: 12:52:25> ++ ERROR - EXCEPTION (000006bf) : WStatus: RPC_S_CALL_FAILED_DN
<SndCsMain: 3504: 884: S0: 12:52:25> :SR: Cmd 04764ba0, CxtG d9f50492, WS RPC_S_CALL_FAILED_DNE, To bostonserver.testcompany.com Len: (66022) [SndFail - rpc exception
<VvJoinBuildTables: 3568: 1746: S0: 12:52:25> :V: VVJOIN ABORTED; MISMATCHED JOIN GUID
<MainVvJoin: 3568: 2253: S0: 12:52:25> :V: ERROR - FrsEnumerateTable failed. JStatus: KeyDuplicat
<MainVvJoin: 3568: 2338: S1: 12:52:25> :V: Stop vvjoin for unjoining DCMA_INTRANET$\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346}\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346
<RcsForceUnjoin: 1204: 2960: S0: 12:52:25> :X: ***** UNJOINED DCMA_INTRANET$\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346}\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346} -> CONUS\VASPFQWEB2$ RemoteCx
<RcsCheckCxtionCommon: 1204: 982: S1: 12:52:25> ++ WARN - RcsUnJoinCxtion: wrong unjoin guid for 04e2194
I've done some snfiffing with Netmon, and I can see that VA is connecting to Boston and successfully makes a bunch of RPC Bind and Request calls on different endpoints like it's supposed to. But I'm no RPC expert so I may be wrong
Any help
Here's a few lines of capture for the network gurus. I've stripped out the TCP overhead for clarity
Frm Time Proto Details From T
1 22.067081 MSRPC c/o RPC Request: call 0x1A96 opnum 0x3 context 0x0 hint 0x84 virginiaserver.testcompany.com bostonserver.testcompany.com
2 22.067081 MSRPC c/o RPC Response: call 0x1A96 context 0x0 hint 0x80 cancels 0x0 bostonserver.testcompany.com virginiaserver.testcompany.com
3 22.067081 MSRPC c/o RPC Bind: UUID F5CC59B4-4264-101A-8C59-08002B2F8426 call 0 virginiaserver.testcompany.com bostonserver.testcompany.com
4 22.067081 MSRPC c/o RPC Bind Ack: call 0x1 assoc grp 0x53B2377 xmit 0x16D0 recv bostonserver.testcompany.com virginiaserver.testcompany.com
5 22.067081 MSRPC c/o RPC Alt-Cont: UUID F5CC59B4-4264-101A-8C59-08002B2F8426 call 0 virginiaserver.testcompany.com bostonserver.testcompany.com
6 22.067081 MSRPC c/o RPC Alt-Cont Rsp: call 0x1 assoc grp 0x53B2377 xmit 0x16D0 recv bostonserver.testcompany.com virginiaserver.testcompany.com
7 22.067081 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
8 22.067081 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
9 22.067081 MSRPC c/o RPC Request: call 0x12C opnum 0x0 context 0x0 hint 0x23C bostonserver.testcompany.com virginiaserver.testcompany.com
10 22.067081 MSRPC c/o RPC Response: call 0x12C context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
11 22.113956 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
12 22.113956 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
13 22.113956 MSRPC c/o RPC Request: call 0x12D opnum 0x0 context 0x0 hint 0x574 bostonserver.testcompany.com virginiaserver.testcompany.com
14 22.113956 MSRPC c/o RPC Response: call 0x12D context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
15 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
16 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
17 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
18 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
19 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
20 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
21 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
22 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
23 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
24 22.145206 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
25 22.145206 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
26 22.145206 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
27 22.160831 MSRPC c/o RPC Request: call 0x12E opnum 0x0 context 0x0 hint 0x574 bostonserver.testcompany.com virginiaserver.testcompany.com
28 22.160831 MSRPC c/o RPC Response: call 0x12E context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
29 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
30 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
31 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
32 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
33 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
34 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
35 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
36 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
37 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
38 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
39 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
40 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
41 22.192081 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
42 22.192081 MSRPC c/o RPC Request: call 0x12F opnum 0x0 context 0x0 hint 0x574 bostonserver.testcompany.com virginiaserver.testcompany.com
43 22.192081 MSRPC c/o RPC Response: call 0x12F context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
This is one I've been working on for a few months now. I've got two IIS6 web servers running W2003. Both are member servers in a 2003 native AD. We are trying to "mirror" the content of one server to another using DFS to provide a "hot backup" site if the production one goes down. The production server is in Virginia on a firewalled DMZ. The backup server is in Boston on our internal network. Each site also has a domain controller. Both servers are talking to their respective DC's and can use Kerberos through the firewall. We have the firewall rules to allow the the following ports between the two servers: 445, 139, 135, 88, and random RPC endpoints. This is a Checkpoint firewall, so it also has a specialized RPC rule that's supposed to keep state and allow through the random high port tcp connections that RPC likes to use. You put in different COM application GUID's and it's supposed to track them. I thought I got them all, but maybe I didn't. Also SMB works fine. No problem copying files. Also, there's only one replication connection, from VA to Boston. All changes are made on the VA server, then copied to Boston
The problem is not that it doesn't replicate. It does, but not very well. It will work for a bit, then it will start building up a huge backlog. In the Ntfrs debug logs, we keep getting the following entries on the Virginia server (the server names have been changed to virginiaserver.testcompany.com and bostonserver.testcompany.com)
<FrsHashCalcString: 3076: 4832: S0: 12:52:25> Name = S-1-5-21-1957994488-746137067-839522115-4455
<SndCsMain: 3504: 883: S0: 12:52:25> ++ ERROR - EXCEPTION (000006bf) : WStatus: RPC_S_CALL_FAILED_DN
<SndCsMain: 3504: 884: S0: 12:52:25> :SR: Cmd 04764ba0, CxtG d9f50492, WS RPC_S_CALL_FAILED_DNE, To bostonserver.testcompany.com Len: (66022) [SndFail - rpc exception
<VvJoinBuildTables: 3568: 1746: S0: 12:52:25> :V: VVJOIN ABORTED; MISMATCHED JOIN GUID
<MainVvJoin: 3568: 2253: S0: 12:52:25> :V: ERROR - FrsEnumerateTable failed. JStatus: KeyDuplicat
<MainVvJoin: 3568: 2338: S1: 12:52:25> :V: Stop vvjoin for unjoining DCMA_INTRANET$\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346}\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346
<RcsForceUnjoin: 1204: 2960: S0: 12:52:25> :X: ***** UNJOINED DCMA_INTRANET$\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346}\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346} -> CONUS\VASPFQWEB2$ RemoteCx
<RcsCheckCxtionCommon: 1204: 982: S1: 12:52:25> ++ WARN - RcsUnJoinCxtion: wrong unjoin guid for 04e2194
I've done some snfiffing with Netmon, and I can see that VA is connecting to Boston and successfully makes a bunch of RPC Bind and Request calls on different endpoints like it's supposed to. But I'm no RPC expert so I may be wrong
Any help
Here's a few lines of capture for the network gurus. I've stripped out the TCP overhead for clarity
Frm Time Proto Details From T
1 22.067081 MSRPC c/o RPC Request: call 0x1A96 opnum 0x3 context 0x0 hint 0x84 virginiaserver.testcompany.com bostonserver.testcompany.com
2 22.067081 MSRPC c/o RPC Response: call 0x1A96 context 0x0 hint 0x80 cancels 0x0 bostonserver.testcompany.com virginiaserver.testcompany.com
3 22.067081 MSRPC c/o RPC Bind: UUID F5CC59B4-4264-101A-8C59-08002B2F8426 call 0 virginiaserver.testcompany.com bostonserver.testcompany.com
4 22.067081 MSRPC c/o RPC Bind Ack: call 0x1 assoc grp 0x53B2377 xmit 0x16D0 recv bostonserver.testcompany.com virginiaserver.testcompany.com
5 22.067081 MSRPC c/o RPC Alt-Cont: UUID F5CC59B4-4264-101A-8C59-08002B2F8426 call 0 virginiaserver.testcompany.com bostonserver.testcompany.com
6 22.067081 MSRPC c/o RPC Alt-Cont Rsp: call 0x1 assoc grp 0x53B2377 xmit 0x16D0 recv bostonserver.testcompany.com virginiaserver.testcompany.com
7 22.067081 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
8 22.067081 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
9 22.067081 MSRPC c/o RPC Request: call 0x12C opnum 0x0 context 0x0 hint 0x23C bostonserver.testcompany.com virginiaserver.testcompany.com
10 22.067081 MSRPC c/o RPC Response: call 0x12C context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
11 22.113956 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
12 22.113956 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
13 22.113956 MSRPC c/o RPC Request: call 0x12D opnum 0x0 context 0x0 hint 0x574 bostonserver.testcompany.com virginiaserver.testcompany.com
14 22.113956 MSRPC c/o RPC Response: call 0x12D context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
15 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
16 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
17 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
18 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
19 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
20 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
21 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
22 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
23 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
24 22.145206 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
25 22.145206 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
26 22.145206 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
27 22.160831 MSRPC c/o RPC Request: call 0x12E opnum 0x0 context 0x0 hint 0x574 bostonserver.testcompany.com virginiaserver.testcompany.com
28 22.160831 MSRPC c/o RPC Response: call 0x12E context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
29 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
30 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
31 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
32 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
33 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
34 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
35 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
36 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
37 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
38 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
39 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
40 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
41 22.192081 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
42 22.192081 MSRPC c/o RPC Request: call 0x12F opnum 0x0 context 0x0 hint 0x574 bostonserver.testcompany.com virginiaserver.testcompany.com
43 22.192081 MSRPC c/o RPC Response: call 0x12F context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com