Richweb supports a variety of Cisco (tm), BSD, Juniper, and Linux based firewalls and routers as edge filtering and network service devices.
Richweb also sells and supports its own security appliance. See http://www.richweb.com/sgw for more information.
The Problem: Managing Internet access
Broadband Internet access at speeds of 3 to 10 Megabits/sec is generally available at most business locations in the US, and t1 (1.5 Mbits/sec) server is available everywhere (for a higehr price of course). Managing how this internet bandwidth is used though is more important than ever before as many companies are pushing business-critical traffic such as LAN to LAN VPNs to other offices and outsourced applications, both web and non-web based. End users that run Peer2Peer (P2P) file sharing, streaming media, and large web searches/requests like Craigslist, Ebay, and fan site message boards can easily render a corporate network unusable, unstable, or simply very slow and unresponsive for periods ranging from a few seconds to minutes or more. Content filtering devices such as WebSense, SurfControl, and Barracuda Networks Web Filter are certainly options for some organizations but these solutions tend to work best when the internet access is centralized at one common choke point, such as in front of an internet-facing firewall. What about networks that are de-centralized, though, where multiple locations of operation are spread across a WAN or VPN cloud? It certainly is less than optimal in terms of bandwidth usage and performance to backhaul internet access to a common choke point where a single appliance can manage the internet access policy. Licensing for these commercial appliances is also not cheap.
OpenDNS and proxy cache solution
OpenDNS is a service that you can sign up and test for free at http://www.opendns.com. Commercial use does come with an access charge, and there are additional benefits that come with the paid-for service. Richweb sets up an openbsd appliance on a 1u rack mount server (or desktop tower for lower budgets) and configures a DNS cache on the appliance (we refer to these appliances as an sgw box or secure gateway). The sgw box has the local DNS cache set to forward all queries to the OpenDNS servers. Additionally, squid proxy cache is installed on the sgw box. The local GPO (windows domain group policies) can be set to require that all outbound http traffic uses the proxy server on a per user, per group policy level. Once you have an account created with OpenDNS, you can automatically add the ip address of the proxy server into the managed Domain List with OpenDNS. You can then set controls over what types of content to allow via the OpenDNS web console. It acts very much like a typical web-content filter, except that the filtering is done at the DNS level, not at the http level. Since the squid proxy cache is caching both DNS responses as well as fetched http data, the cache will speed up web access and save bandwidth by usually 30 to 50% if not more. If you have multiple internet connections, such as a private t1 or vpn connection back to a head quarters link, and a broadband (FIOS or Comcast) internet connection, you can start load balancing traffic across the broadband connection by pointing the proxy server out that internet pipe. This leaves the other connection free for handling business applications, vpns, etc that will run better without being clogged with internet traffic.
Be sure to review the information in this link below for an interaction between OpenDNS and your local Email server that can cause problems: http://www.richweb.com/opendns_comcast_email
Richweb sgw boxes are similar to Cisco Pixes in that eth0 is ALWAYS the uplink or WAN or PUBLIC facing connection and eth1 is ALWAYS the internal network or a DMZ subnet.
For an incoming service (such as a port that needs to be opened from the outside world on a server) to be configured you must provide the following information so that Richweb can execute the changes. An incoming service is defined as traffic flow that originates from OUTSIDE the firewall and is destined for an internal server on the client network behind the firewall.
1. The internal, private IP address of the server
2. The protocol type (usually tcp,udp; may also be gre/pptp and/or ipsec)
3. The port number (for tcp and udp services)
Optional:
4. If you only want the service open to selected external IPs (which must be static, non-changing ips or ip blocks) you can provide this as well.
Richweb will then create a NAT translation on the firewall appliance so that traffic is forwarded to the desired internal service.
Sometimes, you will have a situation where you have a resource that needs a public IP on it (perhaps its an SSL protected appliance or web server) that must be reachable via internal and external clients on the same dns name. This can happen with non MS Exchange mail servers as well. Richweb will often push or route a static IP THRU the firewall, configure the packet inspection on the firewall, and bind the static public ip to the inside server as an alias. In this case Richweb will NOT be natting the IP on the firewall and will NOT be forwarding any ports, but will be restricting the traffic using access control lists.
ISPs that Richweb supports will often use this passthru configuration on Cisco PIXes were the PIX provides the firewalling benefit without the complexity of NAT and dual DNS (inside and public IPs).
For OUTGOING services the basic outbound NAT confguration should allow the packets to pass without any modification on the firewall unless the firewall has been locked down with a higher level of inspection or access control, which sometimes will be the case.
The basic way to test that a tcp port is open for OUTBOUND connectivity is to use the telnet program.
Example:
telnet mail.richweb.com 25
Trying 63.90.9.3...
Connected to ford.
Escape character is '^]'.
220 mail.richweb.com ESMTP Postfix
That shows that port 25 (smtp mail) is open on the outside server mail.richweb.com.
If a vendor is requesting that a "port be opened" in order to reach a service that the vendor may host, the first rule of thumb is to check that the port is tcp. In most cases it will be, and you can test for yourself that it does or does not allow a connection.
If the connection does NOT work, your telnet prompt will hang and be unresponsive. In that case try testing the connection again from a different network (such as a home broadband dsl or fios service) before assuming that the firewall is blocking OUTBOUND connections. In most cases it (the firewall) will NOT be blocking outbound traffic.
We often run into networks that are behind a firewall that blocks all ICMP.
This is not only a bad idea, it increases cost of network ownership due to increased troubleshooting costs and degraded TCP (not UDP) performance in many cases.
Remember, ICMP is designed into the internet core protocols for a specific reason: out-of-band signalling between the end notes that are talking IP AND the intermediate networks that carry the traffic. Blocking some types of ICMP traffic is a very good idea and increases network security. Blocking ALL icmp actually destabilizes your network.
A. These icmp types below should be allowed, and statefully filtered. If your firewall cannot support stateful filtering of icmp get a new firewall. Cisco ASAs and OpenBSD PF are capable of blocking dangerous icmp packets, icmp control packets that are not related to a tcp or udp session. If you are using a packet filtering router or an older firewall without stateful inspection then upgrade your firewall.
Host implementation:Mandatory.
Router implementation:Mandatory.
echo reply: type 0 (ping reply)
echo request: type 8 (ping request)
B. The ICMP types below MUST be allowed, for TCP to function properly. Blocking these types makes your network is not rfc-compliant and prone to mysterious problems that manifest themselves in strange, hard to troubleshoot ways like path mtu blackhole. If you have servers behind a firewall that blocks all icmp, and those servers set the DF bit on all outgoing tcp segments, then remote clients that are behind netwoks that have smaller MTUs, like pppoe or dsl, or some mpls WANs will not be able to transfer tcp data, or they will transfer it very slowly. To the end user this will appear as a website that is very very slow, or only some pages and images will load.
Problems with tcp window scaling and firewalls that filter these options on ip packets with tcp connection setup can also cause problems. Here is a case study:
http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a0080742d6e.shtml
From the doc:
dropped TCP connections [are] caused by some versions of PIX software not
supporting the TCP Window Scaling option. This causes it to have a much
smaller TCP window than the endpoints actually have. This causes the Cisco
PIX to drop packets that it believes are outside the TCP window, but which
really are not.
Inbound can't fragment, among other messages: icmp type 3
ttl exceeded: icmp type 11
general parameter problem: icmp type 12
Some older documentation may suggest allowing:
Inbound Sourch quench: icmp type 4
But i have found that this is not necessary really.
Basically, if you feel compelled for some strange reason to filter ICMP then at an ABSOLUTE minium you must allow path mtu discovery to work properly and that means NOT blocking icmp type 3 code 4.
Here is a simple pf rule which can do this:
macro:
icmp_types = "{ echoreq, unreach}"
rules:
pass inet proto icmp all icmp-type $icmp_types keep state
pass out on $ext_if proto { tcp, udp, icmp } all modulate state
The 2nd rule will enable stateful tracking of tcp and udp sessions, and allow icmp that is naturally part of those sessions to be passed correctly. In additon ping replies will be allow in response to inconing ping requests since that is part of the "state" that is created for you inside pf.
Here is a proper ruleset and config for a Cisco ASA firewall:
access-list inet_in extended permit icmp any any time-exceeded access-list inet_in extended permit icmp any any unreachable access-list inet_in extended permit icmp any any echo-reply access-list inet_in extended permit icmp any any echo policy-map global_policy class inspection_default inspect icmp access-group inet_in in interface outside
By default ssh binds to ALL ip addresses on a host. For application servers that run 1 or more vservers Richweb changes the sshd config on the main host to listen only on the main host ip address.
Example /etc/ssh/sshd_config entry:
ListenAddress feller.richweb.com
Where feller.richweb.com is a valid /etc/hosts entry.
When changing the ip address of an appliance or app server it is important to correct /etc/hosts and restart sshd.
Richweb will often change the default listen port as well:
Example:
Port 822
Other typical sshd settings:
TCPKeepAlive yes
X11Forwarding no
ClientAliveInterval 120
ClientAliveCountMax 3
UseDNS no
PermitEmptyPasswords no
Q. What is the firewall appliance Richweb makes - this sgw?
sgw stands for Secure GateWay; a Richweb security appliance that is based upon the OpenBSD operating system (http://openbsd.org/) which provides industry leading out of the box security and firewall feature sets. A low end sgw box is a desktop class Intel PC with a single hard disk and 1 or more network interfaces (NICs) and 512MB of RAM or more. Higher end sgw boxes are typically 1u IBM servers with hardware raid, RSA cards for remote management, and 3 Gigabit network cards and 2GB of RAM.
Richweb has a standard set of hardening techniques that are applied to the sgw boxes and a regular set of vulnerability scans and intrusion detection tests are performed against both test and live sgw instances to ensure that the sgw appliance is the most secure device you will have on your network.
sgw boxes all have a complete set of firewalling functionalty that is managed via a the /etc/pf.conf configuration file. Richweb sgw boxes are deployed as network firewalls, network packet shapers, tcp or http proxy servers, spam filtering firewalls, intrusion detection servers (running snort for example), dns caches / dns servers, and mail bastion hosts (secure smtp relay). Only the software that is needed to perform the specific job functions is loaded onto the appliance.
In many cases Richweb will install an sgw box for monitoring purposes; one or more serial ports (RS232) will be connected to Cisco router or firewall consoles or other serial devices. The sgw box is aslo very useful as a TCP proxy; it can terminate inbound TCP connections to an internal server via a backup internet connection even when the primary connection is up and traffic by default is flowing out the primary connection. This is very useful for load balancing and disaster recovery situations with smtp and rdp, where the limitations of source and destination NAT can be painful.
All sgw configurations are backed up via subversion repositories so that if a disk or raid array is lost, full functionality can be recovered by retoring the configuration files from svn.
Q. Can the sgw box do routing as well as firewalling and can it be a transparent/invisible firewall ?
Yes, it can be a full fledged router+firewall or bridge(transparent) firewall. Here is more information on the differences: http://www.richweb.com/bridging_vs_routing_firewalls
Q. What are the physical security controls in place for access to the sgw servers and software?
The cabinets which house the servers are locked; the servers are locked at the BIOS level. Only authorized (by Richweb) data center personnel have access to the servers (as directed by Richweb). Our colocation facilities have SAS70 type II compliant access control implemented. There is 2 factor authentication: ID card acquired in exchange for valid US Govt. ID as well a biometric reader that blocks unauthorized access at the man trap. Only pre-registered users that have entries in the biometric scanner database are allowed into the data center. Each user type (employee, contractor, vendor, customer) is categorized for the purposes of ID generation. All movement through the facility is monitored and tracked. Richweb sgw development and maintenance is isolated to Richweb's corporate headquarters where the senior developers (a small group of 3 to 4 analysts) have access to the code base. All source code, server configuraton files and server system files that detail setup are checked into and managed with the industry leading open source code management software: Subversion. Richweb uses Subversion over SSL protected HTTPS sessions so no source code or config files are pulled via clear text protocols at any time.
Q. How does the process of a Richweb employee gaining access to an sgwappliance work?
Richweb does not maintain or create accounts for Richweb management purposes to client sgw instances unless authorized and instructed to do so by an authorized client administrator with the client in question. This authorization should be given via the telephone. Some clients have made arrangements with Richweb whereby certain actions can be taken via emails from the client administrator but usually these directives are issued via telephone conferences or conversations. A written email will then be sent for confirmation of any action that is to be taken. This provides an audit trail for the access that was granted. Richweb logs all time spent supporting a client in our EMS project management system. The EMS system project logs will contain these notes and actions taken. Actual access to the appliance takes place via the ssh2 protocol using ssh keys (password authentication can be disabled for additional security measures). TCP Port 22 is open inbound to the firewall from Richweb Network Operations. This port allows authorized Richweb’s system administrators to access the system to perform maintenance operations. Backups also travel through this port. All communications across this port use strong encryption for SSH access (SSH = secure shell, an industry standard remote access protocol for secured hardware). SSH access to the sgw boxes is restricted at the firewall level to these 2 ip ranges. In some cases the internal (private) ip ranges of a customer's own network are allowed to ssh into the sgw appliance if a customer has 1 or more resources that are trained in the basic sgw command set, such as examining bandwidth usage.
Q. How often are updates to the sgw boxes performed?
Because there is virtually no "attack surface vector" for an sgw box setup as a firewall for example, updates are really not required as frequently as with Windows bases systems. The "attack surface vector" is kept to a minimum by virtue of having all incoming ports filtered, and all programs not needed shut off and un-installed. OpenBSD patches come out on occasion; Richweb tests and implements patches as needed once they are (A) verified not to cause any other problems and (B) relevant and needed by the sgw in question. All package updates are crypto signed and installed from openbsd sources or added via the official pkg_add utility that comes with the system.
Q. How are the sgw boxes secured against remote intrusion and rootkits?
OpenBSD uses a monolithic kernel approach; a "root kit" would require the kernel to be patched, recompiled, and the system rebooted, which would be rather obvious. The whole point of a root kit is to be able to slip a kernel loadable module undetected into a running kernel, which is not going to happen.
Q. Does the sgw appliance have any industry certifications?
OpenBSD is used in many secure network appliances. Refer to http://www.openbsd.org/ for more information. Debian Linux (used by Richweb on certain app server sgw boxes) has achieved Carrier Grade Linux status.
In this sample config the HQ LAN is protected by a cisco ASA running 8.2.2 with a LAN ip scheme of 10.100.0.0/16 and the remote side is running standard OpenBSD 4.6 with a LAN scheme of 192.168.7.0/24.
OpenBSD public IP: z.y.207.125
Cisco ASA public IP: a.b.204.181
Some notes/caveats:
1. The crypto map is named l2l (short for lan2lan).
2. The ASA has all 10.x/8 to 10.x/8 and 192.168.x/16 traffic NO-NATTed. This allows the no-nat acl to be short and simple when many spoke vpn sites are added with 192.168.x/24 or 10.x/16 or /24 subnets.
3. The OpenBSD ipsec implementation does not turn on dead peer detection it seems currently unless dynamic mode is chosen and FQDN IDs (instead of IPv4 addresses) are used as gateway IDs. This means that if you do a hard clear (clear crypto ipsec sa on the ASA or ipsecctl -F -f /etc/ipsec.conf on OpenBSD) the same hard clear may need to be done on the other side to bring the tunnel up before the key lifetime expires. This seems to happen when the tunnel is torn down on the OpenBSD side via a flush. The ASA side keeps the tunnel open and traffic is stuck without a working SA. The solution to this is to use this command alias to clear the tunnel from the OpenBSD side after changing a parameter. It will let the ASA know to tear down the existing SA if one exists, and the tunnel will come back up within a few seconds once the new SAs are built.
ipsecctl -d -f /etc/ipsec.conf;
sleep 10;
ipsecctl -F -f /etc/ipsec.conf
If you have multiple SAs (multiple tunnels) on the OpenBSD side, then use ipsecctl- sa to view the SAs and only terminate the ones that you need to restart. Same thing applies for the ASA side; you can clear crypto ipsec sa peer [ip_of_peer_to_clear].
4. You need to source your pings properly from both the ASA and the OpenBSD gateway when testing if a tunnel is up. ASA:
ping inside a.b.c.d
OpenBSD: ping -I [ipv4 of inside interface that is part of the tunnel] a.b.c.d
Example: ping -I 192.168.7.1 10.100.200.254
Cisco ASA configuration
interface GigabitEthernet0/0 nameif outside security-level 0 ip address a.b.204.181 255.255.255.252 interface GigabitEthernet0/1 nameif inside security-level 100 ip address 10.100.200.254 255.255.0.0 route outside 0.0.0.0 0.0.0.0 a.b.204.182 1 ! define the acl that will tell the ASA NOT to NAT the matching traffic: access-list nonat extended permit ip 10.0.0.0 255.0.0.0 10.0.0.0 255.0.0.0 access-list nonat extended permit ip 10.0.0.0 255.0.0.0 192.168.0.0 255.255.0.0 ! bind the acl to the no nat (nat = 0 means no nat for the ASA) config: nat (inside) 0 access-list nonat ! Define the acl which will control the traffic which will be ! encapsulated in the ipsec tunnel from the ASA to the sgw: access-list 101 extended permit ip 10.100.0.0 255.255.0.0 192.168.7.0 255.255.255.0 ! enable IKE on the outside interface of the ASA: crypto isakmp enable outside ! enable nat-awareness code in the ASA, set the keep alive for 20 sec: crypto isakmp nat-traversal 20 ! configure IKE policies on the ASA - phase 1 negotiation crypto isakmp policy 10 authentication pre-share encryption 3des hash sha group 2 lifetime 86400 ! configure the IPSec Policies - phase 2 negotiations: crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac ! tell the ASA that the inside intf is the management interface: management-access inside
Note: This will allow us to source pings from the inside interface to the remote side of the vpn tunnel. Example: ping from the inside interface ping inside 192.168.7.1
! build the tunnel group: tunnel-group z.y.207.125 type ipsec-l2l tunnel-group z.y.207.125 ipsec-attributes pre-shared-key test1234 ! install the crypto map which will scan egress traffic ! and tunnel the matching traffic based on the match keyword: crypto map l2l 10 match address 101 crypto map l2l 10 set peer z.y.207.125 crypto map l2l 10 set transform-set ESP-3DES-SHA crypto map l2l interface outside
OpenBSD IPSEC Configuration
The "main" params define the method and crypto transforms used in phase 1 of automatic keying (that is what IKE daemons do). The "auth" params define the settings used for the second phase. OpenBSD /etc/ipsec.conf
ike esp from 192.168.7.0/24 to 10.100.0.0/16 \ peer a.b.204.181 \ main auth hmac-sha1 enc 3des group modp1024 \ quick auth hmac-sha1 enc 3des group none \ psk "test1234"
/etc/pf.conf fw rules - This rule goes in section 1 - macros:
fw_main = "a.b..204.181"
these rules all go in section 5 - filter rules
## Start rules for ipsec
## ipsec rule 1: allow esp:
pass in quick on $ext_if proto 50 from { $fw_main } to $fwip keep state
## ipsec rule 2: allow ike and udp 4500 ike:
pass in quick on $ext_if proto udp from { $fw_main } to $fwip port { isakmp, ipsec-nat-t }
## ipsec rule 3: allow lan2lan tunneled traffic by default:
## Uncomment the below rule to allow all ipsec encapsulated traffic
## on the encryption interface. This will allow all lan2lan traffic to flow thru the tunnel:
pass on enc0
Upgrading phase2 negotiation to use perfect forward secrecy
1. Add this setting to the crypto map on the ASA:
crypto map l2l 10 set pfs group5
2. Change the quick auth line on the OpenBSD gw to enable DH group 5:
quick auth hmac-sha1 enc 3des group modp1536 \
There are 3 commonly used dh group settings for phase 2:
A. none
B. modp1024 = dh group 2
C. modp1536 = dh group 5
The ASA and the OpenBSD must match in order for the tunnel to come up. To look at the isakpmd logging on the OpenBSD side: tail -f /var/log/daemon You can also do ipsecclt -m to monitor interactively the keying attempts while pinging across the tunnel.
Use iperf to test your tunnel
If you have workstation with iperf installed on each side of the computer. You should test performance across the tunnel:
server side:iperf -s -m -p 5000 (let it run)
client side: iperf -c 10.100.200.10 -m -p 5000 -t 120
The server side iperf instance is running on a machine with an IP of: 10.100.200.10 which is what the client instance connects to. The -m option prints the detected maximum segment size.
Dealing with packet Fragmentation and Path MTU issues:
If you determine that you need to clamp the MSS then these commands may be useful:
ASA clamp the MSS: sysopt connection tcpmss 1340
ASA clear the df bit on packets leaving an ipsec tunnel (flowing outbound):
crypto ipsec df-bit clear-df outside
OpenBSD MSS clamp and DF bit clear:
add these 2 commands to Section 2 of /etc/pf.conf (TRAFFIC NORMALIZATION)
match in all scrub (no-df max-mss 1340)
match out all scrub (no-df max-mss 1340)
After running iperf you should have a good idea of what to set the MSS to; 1340 is pretty conservative for standard t1/t3/ethernet WANs but may be needed for ADSL or PPPoATM or PPPoE dsl connections or situations where a gre tunnel(s) is involved.
In this configuration we have an ASA5505 that connects a customer lan to the ISP. The customer has added a vendor that needs remote access to a server that is installed in a DMZ. The server should not have access back to the main customer network, but the vendor needs to have access into the server via a vpn client, and the customer will access the server via https (tcp port 443) and RDP.
interface Vlan1
description internal network
nameif inside
security-level 100
ip address 192.168.1.10 255.255.255.0
interface Vlan4
nameif vendorlan
security-level 20
ip address 192.168.240.1 255.255.255.0
!
interface Ethernet0/0
description outside interface
switchport access vlan 2
interface Ethernet0/1
description inside interface customer LAN
interface Ethernet0/4
description vendor vlan connects to vendor server 192.168.240.10
switchport access vlan 4
!
boot system disk0:/asa822-k8.bin
same-security-traffic permit intra-interface
access-list split_tunnel standard permit 192.168.240.0 255.255.255.0
access-list nonat extended permit ip 192.168.0.0 255.255.0.0 192.168.0.0 255.255.0.0
! acl entries for vendor server:
access-list inet_in extended permit tcp any host a.b.57.84 eq 3389
access-list inet_in extended permit tcp any host a.b.57.84 eq https
! vpn ip pools:
ip local pool sslvpnpool 192.168.240.240-192.168.240.247
ip local pool ipsecvpnpool 192.168.240.230-192.168.240.237
global (outside) 1 interface
nat (inside) 0 access-list nonat
nat (inside) 1 0.0.0.0 0.0.0.0
nat (vendorlan) 0 access-list nonat
access-group inet_in in interface outside
route outside 0.0.0.0 0.0.0.0 a.b.56.1 1
crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto dynamic-map outside_dyn_map 20 set pfs
crypto dynamic-map outside_dyn_map 20 set transform-set ESP-3DES-SHA
crypto dynamic-map outside_dyn_map 20 set reverse-route
crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map
crypto map outside_map interface outside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
crypto isakmp policy 65535
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
management-access inside
group-policy VendorPolicy internal
group-policy VendorPolicy attributes
banner value Unauthorized access prohibited
dns-server value 8.8.4.4
vpn-tunnel-protocol IPSec
split-tunnel-policy tunnelspecified
split-tunnel-network-list value split_tunnel
default-domain value vendorlan.customer.local
vlan none
address-pools value ipsecvpnpool
! This user is shared between IPSEC AND SSLVPN Access:
username testvpn password my.password
username testvpn attributes
password-storage enable
service-type remote-access
! IPSEC clients will connect to this ASA with Group Auth radio button checked,
! Group Name Vendor and Group Password is group.secret
tunnel-group Vendor type remote-access
tunnel-group Vendor general-attributes
address-pool ipsecvpnpool
default-group-policy VendorPolicy
tunnel-group Vendor ipsec-attributes
pre-shared-key group.secret
! SSL VPN Configs:
Setup ca authority, trustpoint and self signed cert:
crypto key generate rsa label sslvpnkeypair
crypto ca trustpoint localtrust
enrollment self
fqdn vpn.domain.local
subject-name CN=vpn.domain.local
keypair sslvpnkeypair
crypto ca enroll localtrust noconfirm
crypto ca trustpoint localtrust
crl configure
ssl trust-point localtrust outside
webvpn
enable outside
svc image disk0:/anyconnect-win-2.5.1025-k9.pkg 1
svc image disk0:/anyconnect-macosx-i386-2.5.1025-k9.pkg 2
svc enable
tunnel-group-list enable
group-policy SSLVPN internal
group-policy SSLVPN attributes
banner value Customer Vendor VPN
banner value Unauthorized access prohibited
dns-server value 8.8.4.4
vpn-tunnel-protocol svc
split-tunnel-policy tunnelspecified
split-tunnel-network-list value split_tunnel
default-domain value vendorlan.customer.local
vlan none
address-pools value sslvpnpool
tunnel-group SSLClientProfile type remote-access
tunnel-group SSLClientProfile general-attributes
default-group-policy SSLVPN
tunnel-group SSLClientProfile webvpn-attributes
group-alias SSLVPNClient enable
The split tunnel policy is also shared between both SSL and IPSEC configs.
Setup static nats:
static (vendorlan,outside) a.b.57.84 192.168.240.10 netmask 255.255.255.255
static (inside,vendorlan) 192.168.240.10 192.168.240.10 netmask 255.255.255.255
When VPN clients connect they will not be able to get internet access, they will not be able to initiate connections to 192.168.1.x customer main lan either. They will be able to connect to 192.168.240.10 vendor server only.
Inside 192.168.1.x clients will be able to make HTTPS connections to 192.168.240.10. There is an internal router (192.168.1.1) that has a static route for 192.168.240.x pointed at the ASA as well. This is not necessary for the configuration to work; it is there both to document the path to the .240 lan, as well as possibly redistribute into the IGP at a later date so a remote LAN 192.168.2.x can also make HTTPS connections to the vendor server.
To show the SSL VPN (anyconnect) vpn users that are active:
sh vpn-sessiondb svc
Log off all sessions under the username: xxx
vpn-sessiondb logoff name xxx
The ASA5505 comes with 2 SSL VPN licenses in the Base and Security Plus packages. Thats not enough licenses for use by a typical small to medium sized company, but it is enough to use for 1 or 2 admins or power users that need to be able to connect from locations where IPSEC may be blocked (hotels, airports) by the captive portals.
This tunnel uses a basic lan2lan ipsec tunnel (no gre or routing since the ASA does not have these features). We use sha for phase 1 and md5 for phase2 for a little extra speed.
The 3745 should be able to tunnel about 25 mbits/sec using IOS 12.3 with this config before cpu usage starts to climb to a point where you might want to look at a faster router.
IP Addressing:
a.b.c.194 = 3745 IOS v. 12.3 router at an enterprise LAN
x.y.z.1 = ASA 5510 firewall at a COLO
172.20.1.0/24 - COLO LAN
192.168.1.0 - Enterprise LAN
IOS router config:
! Phase 1 policy
crypto isakmp policy 12
encr 3des
hash sha
authentication pre-share
group 2
lifetime 28800
! phase 2 policy
crypto ipsec transform-set phase_2_ts esp-3des esp-md5-hmac
! pre shared key
crypto isakmp key secretpass address a.b.c.194
crypto map vpn_map 5 ipsec-isakmp
set peer x.y.z.1
set transform-set phase_2_ts
set pfs group2
match address to_colo_lan
! The no-nat acl - we do not want lan 2 lan traffic to be natted:
ip access-list extended nonatacl
deny ip 192.168.1.0 0.0.0.255 172.20.1.0 0.0.0.255
permit ip any any
! the route-map that will control natting:
route-map natmap permit 10
match ip address nonatacl
! The acl that the route-map will use:
ip access-list extended to_colo_lan
permit ip 192.168.1.0 0.0.0.255 172.20.1.0 0.0.0.255
This input acl is added to the upstream-facing interface (fast ethernet 0/0 in this case):
ip access-list extended inet_in
! permit esp and ike into our router from the ASA:
permit esp host x.y.z.1 host a.b.c.154
permit udp host x.y.z.1 host a.b.c.154 eq isakmp
! this line needed due to a 12.3 bug; ipsec lan2lan traffic should be invisible to the filter that
! was bound to the input interface.
permit ip 172.20.1.0 0.0.0.255 192.168.1.0 0.0.0.255
... other entries excluded ...
interface FastEthernet0/0
description uplink to ISP
ip address a.b.c.154 255.255.255.252
ip access-group inet_in in
ip nat outside
load-interval 30
speed 10
full-duplex
crypto map vpn_map
interface FastEthernet0/1
description inside LAN
ip address 192.168.1.1 255.255.255.0
ip nat inside
load-interval 30
duplex auto
speed auto
! ip natting is controlled by a route-map that selects which flows can be natted outbound.
! inbound nats and port translations would be added with static nat statements as needed:
ip nat inside source route-map natmap interface FastEthernet0/0 overload
ASA Config:
interface GigabitEthernet0/0
speed 100
duplex full
nameif outside
security-level 0
ip address x.y.z.1 255.255.255.252
interface GigabitEthernet0/1
speed 100
duplex full
nameif inside
security-level 100
ip address 172.20.1.1 255.255.255.0
access-list nonat extended permit ip 172.20.1.0 255.255.255.0 192.168.1.0 255.255.255.0
access-list to_ent_lan extended permit ip 172.20.1.0 255.255.255.0 192.168.1.0 255.255.255.0
! phase 1:
crypto isakmp policy 20
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 28800
! phase 2:
crypto ipsec transform-set phase_2_ts esp-3des esp-md5-hmac
! crypto map:
crypto map vpn_map 30 match address to_ent_lan
crypto map vpn_map 30 set pfs
crypto map vpn_map 30 set peer a.b.c.194
crypto map vpn_map 30 set transform-set phase_2_ts
! pre-shared secret:
tunnel-group 24.75.134.154 type ipsec-l2l
tunnel-group 24.75.134.154 ipsec-attributes
pre-shared-key secretpass
! bind the crypto map to the interface:
crypto map vpn_map interface outside
Racoon on Linux CentOS 5.5 ipsec-tools-0.6.5-14.el5_5.5
Cisco Router is 3745 running 12.3 IOS
Racoon peer is w.x.y.z; 10.0.0.0/16 network is behind this ipsec gateway.
Cisco Router is a.b.c.d; 192.168.0.0/24 network is behind this ipsec gateway.
Racoon cfg:
remote a.b.c.d
{
exchange_mode main;
my_identifier address "w.x.y.z";
peers_identifier address "a.b.c.d";
initial_contact on;
proposal_check obey;
proposal {
encryption_algorithm 3des;
authentication_method pre_shared_key;
hash_algorithm md5;
dh_group 2;
}
}
sainfo address 10.0.0.0/16 any address 192.168.0.0/24 any {
lifetime time 8 hour;
encryption_algorithm 3des;
authentication_algorithm hmac_md5;
compression_algorithm deflate;
pfs_group 2;
}
! support both sha and md5 hashes for phase 1 negotiation:
crypto isakmp policy 11
encr 3des
hash sha
authentication pre-share
group 2
lifetime 28800
!
crypto isakmp policy 12
encr 3des
hash md5
authentication pre-share
group 2
lifetime 28800
crypto isakmp key xxx address w.x.y.z
! support md5 and sha hash for phase 2:
crypto ipsec transform-set racoon_phase2 esp-3des esp-sha-hmac
crypto ipsec transform-set racoon_phase2_md5 esp-3des esp-md5-hmac
! remote racoon peer:
crypto map vpn_map 11 ipsec-isakmp
set peer w.x.y.z
set transform-set racoon_phase2 racoon_phase2_md5
set pfs group2
match address racoon_hq
! bind the crypto map to the outbound intf:
interface FastEthernet0/0
description uplink to ISP
ip address a.b.c.d 255.255.255.252
ip access-group inet_in in
ip nat outside
load-interval 30
speed 10
full-duplex
crypto map vpn_map
! acl to match networks to send across specific tunnel to austin hq:
ip access-list extended racoon_hq
permit ip 192.168.0.0 0.0.0.255 10.0.0.0 0.0.255.255
Note: we do not nat these src+dst pairs; a dmz interface that exists is not shown below was also present, and we added that to the no nat acl as well.
ip access-list extended nonatacl
deny ip host a.b.c.d any
deny ip 192.168.0.0.0 0.0.0.255 10.0.0.0 0.255.255.255
! nat everything else that egresses:
permit ip any any
! bind the nat acl to the nat routemap
route-map natmap permit 10
match ip address nonatacl
! nat outbound flows that dont have a static 1:1 nat to the
! egress interface, provided the conditions in the natmap routemap
! are met. This routemap will in turn implement the nonat conditions.
! ipsec tunnel traffic should not be natted in our case:
ip nat inside source route-map natmap interface FastEthernet0/0 overload
It should be noted that this version of IOS had a bug in that the acl was applied to the external (ISP) facing interface; the same interface that the crypto map was applied to. We had to allow the internal (10.x to 192.168.0.x) traffic on this ACL for tcp sessions across the ipsec tunnel to work. Normally this would not be the case, since the internal ipsec lan to lan traffic would flow through the external cisco interface (Fast Eth0/0) and be permitted as esp traffic (there was an acl line for esp of course). The observed behavior was the the default deny any at the end of the cisco ACL was blocking the lan to lan traffic until we added permits at the top.
The esp and udp permits specific to the racoon host src and cisco router dest are NOT needed, since the default permits for esp and udp isakmp will match too. We added them so we could capture the exact number of hits that the specific racoon gateway was creating separate from any other ipsec traffic. This is a generally useful acl technique you can use with ciscos:
(ACL Excerpt):
ip access-list extended inet_in
permit tcp any any established
permit esp host w.x.y.z host a.b.c.d
permit udp host w.x.y.z host a.b.c.d eq isakmp
permit udp host 64.20.241.76 host a.b.c.d eq non500-isakmp
! below line was needed due to IOS bug mentioned above:
permit ip 10.0.0.0 0.0.255.255 192.168.0.0 0.0.0.255
permit udp any any eq isakmp
permit udp any any eq non500-isakmp
permit esp any any
--- other management and dmz nats left out ---
deny ip any any log
The problem:
Customer has a main HQ location with a Windows Active Directory setup and one or more remote LANs that are connected to the HQ location across internet based IPSEC or OpenVPN lan2lan tunnels. The small remote LANs have 2 to 10 users, a mixture of PCs and wireless users; no domain controllers exist at the remote LANs. Users must authenticate to the AD domain across the tunnel. The question is how can DHCP+DNS be setup so that if the lan2lan tunnel is down, the users at the remote sites can still browse the internet and connected to external resources.
If the DHCP server is setup to pass out 2 DNS servers:
10.10.10.10 (IP address of the HQ AD domain controller - i.e. the DC)
8.8.8.8 (google DNS)
Then in theory computers on the remote LAN would use internal DNS and resolve AD names, and then fail over to google when the tunnel is down. This is not how it will work, however, in production. What will happen is that the resolvers on the LAN will still query the google DNS server for internal domain names (he name resolver software considers the DNS servers equal in terms of resolution capability initially). This results in potential information leakage (a security issue) as your internal resources are asking an external resource (google) for information about a network (the HQ domain) that is private, and of course google cannot answer these questions. This can cause windows logins, file shares, and printing to either break or move very slow, and become unreliable and frustrating for end users.
If you remove the google dns server and have only the internal DNS server, then AD will work better, but if the ipsec tunnel is down then no name resolution will work, and unless the users have an icon on their desktop that uses RDP to connect to a remote terminal server by ipv4 address, they will not be able to work at all (well creating host file entries would work, but this is a horrible kludge that will cause trouble down the road and extra maintenance costs).
Solution #1: Replicate DNS to the remote firewall - Richweb SGW (OpenBSD gateway)
This is good example of why an OpenBSD box makes a better 1 stop solution for remote lans than a commercial firewall that does not include an OS or DNS process that you can customize!
Legend:
10.10.10.0/24: HQ LAN Subnet
10.10.10.10: HQ DC
10.10.40.0/24: Remote LAN Subnet
10.10.40.1: Inside ipv4 addr of remote firewall
hqdomain.local AD domain name
1. Edit the named configuration file to pull the zone file:
/var/named/etc/named.conf
# make sure that the clients access list allows the networks that will query this remote dns server
# to have permission to use it. If you want to test or monitor the remote DNS server process from a
# station at the HQ location, then you will need to add the HQ location to the clients list. If your
# entire internal network (HQ LAN and all remotes) is using 10.x you could add 10.0.0.0/8; for example.
acl clients {
127.0.0.0/8;
localnets;
};
# here the forwarders are commented out, but you could use opendns forwarders here if you are
# also interested in stopping the users from accessing certain internet sites.
options {
version ""; // remove this to allow version queries
listen-on { any; };
empty-zones-enable yes;
allow-recursion { clients; };
additional-from-auth yes;
additional-from-cache yes;
## forwarders { a.b.c.d; x.y.z.2; };
};
# Add these 2 zone definitions - the transfer source is important to set as we must use our internal LAN ip and not the WAN ip or tunnel ip (OpenVPN) to connect to the HQ DC across the tunnel.
zone "hqdomain.local" {
type slave;
file "slave/hqdomain.local.txt";
transfer-source 10.10.40.1;
masters { 10.10.10.10; };
};
zone "_msdcs.hqdomain.com" {
type slave;
file "slave/_msdcs.hqdomain.local.txt";
transfer-source 10.10.40.1;
masters { 10.10.10.10; };
};
The _msdcs subzone may or may not exist. Here is an excerpt from a windowsitpro article:
http://www.windowsitpro.com/article/dns/q-what-s-the-dns-_msdcs-zone-for...
"Active Directory (AD) uses DNS as its locator service to support the various types of services that AD offers, such as Global Catalog (GC), Kerberos, and Lightweight Directory Access Protocol (LDAP). Other non-Microsoft services can be advertised in the DNS, including--but not restricted to--non-Microsoft implementations of LDAP and GC. However, sometimes clients might need to contact a Microsoft-hosted service. For that reason, each domain in DNS has an _msdcs subdomain that hosts only DNS SRV records that are registered by Microsoft-based services. The Netlogon process dynamically creates these records on each domain controller (DC). The _msdcs subdomain also includes the globally unique identifier (GUID) for all domains in the forest and a list of GC servers."
2. You need to make sure that the DC allows the remote firewall - 10.10.40.1 to perform a zone transfer request in the windows DNS manager tools.
3. Restart named on the firewall:
/opt/shared/scripts/restart_bind9.sh
Check that the zones replicated:
ls /var/named/slave/
To see what named logged - in case the zones did not transfer:
grep named /var/log/daemon
4. In the DHCP settings you would have something like this:
/etc/dhcpd.conf
subnet 10.10.40.0 netmask 255.255.255.0 {
range 0.10.40.130 10.10.10.250;
default-lease-time 14400;
max-lease-time 86400;
option routers 10.10.40.1;
option domain-name-servers 10.10.40.1;
option domain-name "hqdomain.local";
option ntp-servers 10.10.40.1;
option time-offset -18000;
option autoproxy-script "http://10.10.40.1/wpad.dat";
option netbios-node-type 8;
option netbios-name-servers 10.10.10.10;
option option-156 "FtpServer=10.10.10.11, country=1, language=1, layer2tagging=0p";
option tftp-server-name "10.10.10.11";
}
Notes:
The wpad option is used if you have a web proxy (filtering or caching-only) that protects the end users at the remote site. Even if you have a content gateway at the HQ, unless you have a very fast vpn tunnel (low latency across the tunnel) and large amounts of bandwidth at the remote and at the HQ location, its not efficient to backhaul web browsing across the vpn tunnel.
The netbios-name-servers option is only needed if you have a WINS server - most AD networks wont need this option (its for legacy SMB clients - older MS networks before AD+DNS).
The tftp server and option 156 are to support remote booting of Shoretel IP phones.
Solution #2: Use Unbound DNS server with custom forwarding rule for the internal AD domain(s)
This option works well when you cannot get the zone transfer working so that the remote firewall can replicate copies of the AD zone file(s), or you simply dont want to have to configure this on the AD side for whatever reason (policy, security, compliance, etc). We will need to use unbound, which has the ability to define custom forwarders on a per domain basis. bind9 (named) can use only an all-or-nothing approach to forwarding at this time, which will not work for our purposes. We want ONLY the AD zones forwarded across the tunnel, so that if the tunnel is down, regular inet access works as expected.
1. Install the unbound package on the openbsd server, and make sure named is stopped and not configured to start in /etc/rc.conf.local
edit /etc/rc.local to start unbound:
echo -n ' unbound'; /usr/bin/sudo -c daemon /usr/local/sbin/unbound
2. Unbound is configured via:
/var/unbound/etc/unbound.conf
interface: 10.10.40.1
# Be sure to comment out the interface automatic statement and use the ip of the inside (LAN)
# facing interface for use across ipsec tunnels. Otherwise the query will not make it across the
# ipsec tunnel and you will get a server fail result in the dns query for hqdomain.local.
# interface-automatic: yes
outgoing-interface: 192.168.1.1
port: 53
access-control: 10.0.0.0/8 allow
forward-zone:
name: "hqdomain.local."
forward-addr: 10.10.10.10
forward-zone:
name: "."
forward-addr: 208.67.220.220
forward-addr: 208.67.222.222
This will forward queries for the local AD domain across the ipsec tunnel, and cache the results, and it will forward all other dns lookups to OpenDNS servers.
You should not need any rules in pf.conf different than what you would need for standard bind9 / named dns resolution. Pass out on the external intf of the firewall the outbound dns queries, and keep state:
pass out quick proto {tcp,udp} from $fwip to any port 53
You may need to add a nat rule that will set the src address of the dns queries to be the inside ip of the firewall if unbound does not pick the inside ip itself when generating the queries.
dnstop is a useful tool to install to watch the DNS traffic to verify that it is working as intended:
You can also use tcpdump. Assuming you want to watch the inside firewall interface which is fxp1:
/usr/sbin/tcpdump -n -i fxp1 udp dst port 53
Unbound dns server can forward dns zones on a zone by zone basis. This allows you to point renote LAN clients at an unbound server running on an OpenBSD firewall that handles the tunnel for that remote office back to the HQ. unbound can route Active Directory (AD) queries so that internal DNS still works, even without a domain controller at the remote site. If the connection between the sites is lost, unbound can still handle dns resolution for internet sites if a path to the internet is still available.
The example below shows how an unbound dns server at the HQ location can provide dns service for remote lans that have only a solid state firewall and no server or DNS server on their LAN. Unbound will provide secure, seemless service for internal and external DNS requests.
Unbound config for hq dns router to handle spoke sites with no windows server or bsd firewall; only a pix/asa and pc clients.
Local pix/asa will serve as dhcp server; dns in the dhcp server is pointed to public ip of dns server/fw at hq site: x.y.hq.1
# Forward zones config:
## listen on public and private interfaces:
interface: x.y.hq.1
interface: 10.0.1.1
## access control:
access-control: 0.0.0.0/0 refuse
access-control: 127.0.0.0/8 allow
access-control: 192.168.0.0/16 allow
access-control: 10.0.0.0/8 allow
## allow hq server to query itself:
access-control: x.y.hq.1 allow
## allow remotes to query the hq dns server:
access-control: x.y.remote.1 allow
access-control: x.y.remote.2 allow
## forward public domain to richweb dns servers
## assumes richweb hosts the dns for the domain: mydom.com
forward-zone:
name: "mydom.com"
forward-addr: 63.90.9.2
forward-addr: 208.73.136.30
# forward internal domains to internal dns servers
forward-zone:
name: "mydom.local"
forward-addr: 10.0.1.252
forward-addr: 10.0.1.251
forward-zone:
name: "_kerberos.local."
forward-addr: 10.0.1.252
forward-addr: 10.0.1.251
The forwarders below can be OpenDNS, or Richweb's secure caches if you have an account with Richweb.
forward-zone:
name: "."
forward-addr: a.b.c.1
forward-addr: a.b.c.2
This config below allows the windows pcs on the local lan to resolve the wpad entry (proxy auto-detect) to the lcoal edge firewall - very useful when you have a dansguardian or squid cache on the local firewall scrubbing the internet browsing that pcs on the local LAN engage in. mydomain.local. queries are passed along to the origin server with the transparent keyword but the wpad A record is served locally by unbound.
# .local + wpad setup:
local-zone: "mydomain.local." transparent
local-data: "wpad.mydomain.local. 14400 IN A 10.0.1.1"
The local zone configs must be placed in the unbound config file right before the remote control and forwarders information. If you add the local-zone config at the bottom of the file, after forwarded zones have been listed, you will get a syntax error.
To add dns top:
pkg_add dnstop
Then start dnstop with the interface name you wish to monitor. For a firewall that is running unbound dnscache serving local PCs/servers on its inside interface and forwarding to Richweb or OpenDNS servers on its outside interface we would probably want to pick the inside interface to watch first. Assuming bge0 is outside, and bge1 is inside the command would be:
dnstop bge1
Hit the question mark and wait a second for the options screen to display. The default view is (s) DNS sources, or the sources of the queries coming in on that interface. If you have setup your inside AD servers to forward queries to the firewall inside ip address, or if you dont have an AD server but are running DHCP on the firewall and passing out the inside firewall ip address as the default dns server then you should be able to verify this by watching the sources in dnstop.
You can hit the (d) key to see where the queries are being sent though for use on a firewall with PCs and servers as DNS clients on the inside interface you might want to stop dnstop and restart it on the outside interface to watch the destinations better.
Example:
dnstop bge0
hit d key to toggle to dns destinations
Another useful command is the (2) key which breaks down the queries and summarizes at the 2nd domain level:
Query Name Count %
----------------- --------- ------
trendmicro.com 9 15.3
google.com 6 10.2
This configuration arose from a situation where a DR site was setp in a business hotel. The ISP could not provide a public routable ipv4 address for the firewall that went into the hotel suite to connect back to the corporate network. We were able to get a static natted ip. The hotel ISP firewall was fairly restrictive in terms of what ports and protcols it would allow outbound. In particular we had trouble keeping our ipsec tunnels up and running and IPSEC over UDP encap was not working properly. OpenVPN was a better choice for this setup:
Main Site: Debian Linux with OpenVPN 2.1.3, public static ip a.b.c.d
DR Site: OpenBSD 4.6 with package: OpenVPN 2.1rc15p2; static ip 192.168.1.10 natted to w.x.y.z by upstream hotel ISP firewall.
The main site is also running an existing OpenVPN instance on the standard port 1194/udp for client users so we wil create a separate instance of OpenVPN to handle the site to site tunnel.
The hotel also forces all traffic to use its http proxy or else outbound http is blocked. We will use pf rules so that browsers will work w/o needing the proxy manually set (transparent redirect):
Step 1. generate a static key, save it in the file: /etc/openvpn/static.key
openvpn --genkey --secret static.key
Copy this key to the other gateway and save it in the same location.
Step 2. Open port udp 10000 inbound on each firewall from the partner firewall.
Here are the OpenBSD pf.conf changes, port 10000 udp would be opened on the linux firewall using iptables.
Section 1 macros - we define a bypass table of ips that will not be redirected.
table <bypass_proxy> persist file "/opt/scripts/bypass_proxy.conf"
The contents of this bypass file contain the networks that will not be redirected:
172.30.0.0/16
Section 4. Nat rules:
# Exclude the ips in the bypass table from redirect on port 80 outbound:
no rdr on $int_if proto tcp from any to <bypass_proxy> port www
# redirect port 80 outbound to the bus hotel proxy:
rdr on $int_if proto tcp from {$lan} to any port www -> \
10.168.195.254 port 8080
## no nat vpn traffic:
no nat on $ext_if from 172.30.0.0/16 to 172.30.0.0/16
## nat lan traffic:
nat on $ext_if from $lan to any -> $fwip
Section 5. pass rules:
pass in quick on $ext_if proto udp from $hqfwip to $fwip port 10000 keep state
pass in quick on tun0
pass out quick on tun0
Step 3. Build the tunnel scripts:
Here is the OpenBSD side script. You should test with just running the openvpn process command by itself, without the ampersand (that backgrounds it) and without the route statements first on each side. Once you can ping the tunnel ips from each side you can procede with the static route adds.
In this example the OpenBSD side has a tunnel ip of 192.168.254.10, with the linux gateway having .9
The 172.30.40.0/24 network is behind the OpenBSD side. The linux side has several 172.30.x networks, which are listed in the setup script:
The OpenBSD box has a private static NAT ip - 192.168.1.10, so the linux side refers to this public nat ip in its tunnel configuration which is w.x.y.z. The public ip of the linux gateway is a.b.c.d. The hotel ISP gateway NATs the traffic from w.x.y.z to 192.168.1.10. The openvpn configs must not use this private ip on the OpenBSD gateway - they need to use the public ip.
The 192.168.254.9 and 10 addresses were chosen at random, they are not important other than they are used for the tunnel endpoints.
OpenBSD vpn setup script:
#!/bin/sh
echo "Current OpenVPN site to site Process:"
ps ax | grep [/]etc/openvpn/static.key
echo "Killing current openvpn process:"
ps ax | grep [/]etc/openvpn/static.key | awk {'print $1'} | xargs kill -9
sleep 3
echo "Creating new process"
/usr/local/sbin/openvpn --secret /etc/openvpn/static.key --remote a.b.c.d --dev tun0 --ifconfig 192.168.254.10 192.168.254.9 --verb 1 --port 10000 &
sleep 6
echo "Adding routes:"
route add -net 172.30.20.0/23 192.168.254.9
route add -net 172.30.36.0/24 192.168.254.9
route add -net 172.30.4.0/23 192.168.254.9
route add -net 172.30.32.0/23 192.168.254.9
echo "Current OpenVPN site to site Process:"
ps ax | grep [/]etc/openvpn/static.key
echo "Done!"
Here is the linux side vpn setup script:
There are 2 openvpn instances on the linux box, so the script is careful only to stop the process that is handling the site to site vpn.
#!/bin/bash
echo "Current OpenVPN site to site Process:"
ps ax | grep [/]etc/openvpn/static.key
echo "Killing current openvpn process:"
ps ax | grep [/]etc/openvpn/static.key | awk {'print $1'} | xargs kill -9
sleep 3
echo "Creating new process"
/usr/local/sbin/openvpn --secret /etc/openvpn/static.key --remote w.x.y.z --dev tun1 --ifconfig 192.168.254.9 192.168.254.10 --verb 1 --port 10000 &
sleep 6
echo "Adding routes:"
route add -net 172.30.40.0/24 gw 192.168.254.10
echo "Current OpenVPN site to site Process:"
ps ax | grep [/]etc/openvpn/static.key
echo "Done!"
Here is an example configuration that allows an OpenBSD firewall with 2 internet connections to use the correct connection based on the ip address that accepts the connection. Very useful for OOB (Out of Band) management when you have multiple IP connections to a network with different ip assignments from the 2 providers.
This firewall is connected to a t1 as its primary (default route facing) internet connection. In addition there is a FIOS connection that terminates in a Cisco ASA. The inside ports of the ASA and the OpenBSD box are on the same LAN. A cisco 3700 router acts as the internal router that connects to the rest of the network.
$fwinip: inside ip of the bsd fw
$int_if: inside interface of the bsd fw
172.16.8.1: ip address of the cisco router
All traffic sent to the cisco router (8.1) is defaulted out the FIOS connection and the Cisco ASA firewall. There is a static nat on the ASA that allows incoming ssh over port 2200 to hit the inside ip address of the OpenBSD firewall.
In the event that the t1 is down, we can still ssh into the OpenBSD firewall remotely via the public IP of the FIOS connection and perform troubleshooting and diagnostics on the t1 line (that router is serial consoled into the BSD box). Even though the default gateway on the OpenBSD box is pointed out the t1 provider, all ssh connections and pings (icmp echo requests) that are receieved from the NAT on the FIOS connection will have their replies (this is what the reply-to statement does) sent back out the LAN over to the router and then to the ASA via the reply-to policy routing config:
These rules should be placed at the top of the pf.conf:
pass in quick on $int_if reply-to ($int_if 172.16.8.1) proto icmp to $fwinip keep state
pass in quick on $int_if reply-to ($int_if 172.16.8.1) proto tcp to $fwinip port 2200 \
keep state
pass in on $int_if reply-to ($int_if 172.16.8.1) to $fwinip keep state
Components used:
1. OpenBSD+ pf firewall
2. Squid Web Proxy/Cache
3. DansGuardian
4. ClamAV anti-virus (phish, malware, scam detection as well)
5. OpenDNS account (*)
6. Caching DNS server (bind9 or unbound)
7. Apache web server (bundled with OS)
8. ISC DHCP server (bundled with OS)
9. Richweb Network Management and monitoring NAGIOS plugins
10. SquidGuard extra rulesets
Items marked with a (*) require a commercial subscription as per usage terms.
System Requirements:
1. 1.6GHz or better CPU
2. 768MB RAM for small office (less than 10 users), 1GB of RAM for 10-50 users, 2GB of RAM suggested for 50-150 users, 4GB of RAM for 150+ users
3. Firewall with 1 NIC for app gateway mode, 2 NICs for transparent proxy (redirect port 80)
Overview:
The proxy can be deployed as a stand alone server, or as an in-line firewall where all outbound internet access will pass through the firewall. The main difference is that in firewall mode the system can be configured to capture outbound port 80 (HTTP) such that users that are attempting to evade the proxy will have a much harder time doing so.
The proxy is able to scan all content downloaded via HTTP (web search results, redirects, web pages, etc) for malware by ClamAV and all content is also matched against both admin-controlled white and blacklists as well as keywords (intelligent filtering) and phrases. Based on the level that DansGuardian is allowed to pass as safe, virtually all adult and non-business-appropriate sites can be blocked.
OpenDNS fits into the solution as it is needed to control urls visited by HTTPS. Since HTTPS (HTTP over SSL) negotiates a secure end to end channel between the web server and web browser, it is not possible for the proxy to scan or block HTTPS-fetched content. However, all websites that are browsed over HTTPS still require a DNS lookup. By blocking at the DNS level (what OpenDNS is very good at) https:// versions of inappropriate sites can be blocked as well.
Sample Session:
1. Browser requests url, using the Dans Guardian proxy
2. Dansguardian checks local blacklists to make sure domain and url are safe
and not blocked. If you wanted to block myspace.com for example, it can be added to a local file of banned sites, which will be checked first.
3. Dansguardian connects to squid proxy and passes the url that was requested
4. Squid resolves the dns name in a url to an ip address. If Opendns is in
use, then opendns blocked-domain redirects would kick in here. squid would
fetch the bad domain or blocked domain page from the opendns web server.
Squid will be configured to use the local DNS cache on the machine. The local cache means that OpenDNS servers must be queried only when the answer is not already cached.
If you have a local MS AD domain (saee corp.local) that hosts an intranet you may find some of the solutons here to be useful in making the local AD domain resolve along with the OpenDNS forwarding and local caching.
http://richweb.com/openbsd_remote_vpn_site_dns_active_directory_configs
The local DNS cache makes the system more responsive and lessens the load on OpenDNS.
5. Squid fetches the entire block of web content requested by the url
including any sub-assets (images, css, remote ads, etc).
6. All content is passed back to dansguardian where dansguardian is going to
scan all textual content for bad phrases, etc, and block the main page (url)
requested if the naughtiness limit (score) is exceeded by what is found on the page in question.
7. All content that passes the keywords filtering is passed off to clamav for virus scanning. Any content that
fails to pass the malware checks is dropped.
8. Content is returned to the browser by Dansguardian where it is rendered by the browser. An include ad that was blocked would not block the entire webpage, just that pocket would be empty.
Because ad servers send inline javascript to the browsers which in turn fetches the ad content, all ads that have malware should still be blocked, because the browser is using the same protected mechanisms to download inline ads as it is the main web page.
Using speed tests to measure the proxy performance
The popular (and of limited use) web based speed tests are all of course not going to be able to measure any kind of network speed properly due to the multiple stages of buffering and the fact that content cant be streamed directly to the browser; each piece of content or http object has to be fetched in full by the Dansguardian engine to be scanned before it can be allowed to touch the browser. And of course if the web-based speed tester is a flash or java app or even a web app that is asking the browser or code loaded by the browser to connect on a port that is not proxied, then the content wont be even going thru the proxy.
The best way to monitor the performance of the filtering system is with systat, for system usage and either bwm (bwm-ng package) or iftop for network and port/protocol usage.
Setting Users to use the Proxy
There are several options here, you will probably use all of them:
1. Manual - this is for testing only of course or for networks with 1 or 2 stations. You wont want to manage proxy settings manually for more than 1 or 2 browsers. Set your proxy to the ip address of the firewall (inside ip) or server ip (server mode) on port 8080.
2. Active Directory Group Policy - this can be done in AD on a group by group basis, though you will need a wpad file. Of course this is an option for sites with a domain controller / local file and print login server.
3. DHCP proxy setting - this works by telling the browser / OS via DHCP the url of the wpad (Windows Proxy AutoDetect) file.
4. DNS+WPAD - proxy auto-detect when enable will work by having the browser make a request for a wpad file at http://wpad.my_ad_domain.local/wpad.dat
For more information on the DHCP and WPAD options:
http://en.wikipedia.org/wiki/Web_Proxy_Autodiscovery_Protocol
5. Transparent Proxy - this is a last resort only. Note that Transparent redirection can and will break websites that assume that there is no proxy between the client and the web server. It is suggested that one of the first 4 options above are implemented, and that transparent proxy is used ONLY to capture users that attempt to evade the policy settings of the filters.
Setting up transparent redirect is done in the pf.conf file and it again, will only break end systems (for some websites) that are NOT aware they should be using a proxy. So setting transproxy is a good idea once you have the DHCP, DNS, and WPAD options in place as it will catch end users that are either trying to evade the content filter or guests that plug-in with laptops that dont have your domain policy for example.
It is easy in pf as well to build a set of ips that are excluded from transparent proxy; servers that need to download updates that already have good local Antivirus installed and maintained are good candidates to be excluded from transparent proxy.
Logging
Dansguardian keeps all of its logs in /var/log/dansguardian/access.log
Dowmloading the log file and parsing it using a product like CyFin report is possible, and you can also use grep to search through the logs for a particular site or end user as well.
http://www.wavecrest.net/support/cyfin/reporter/
If you have a 4.5 Mbit/sec internect connection (3 bonded t1s) for example, just one user playing a full screen video can take over 75% of your available (download) bandwidth. If you have a mixture of data applications (VPN, MSTS/RDP/Citrix/Intranet) and general web use sharing the same pipe this can cause a real problem when 1 or more users fire up a video. At home users are often used to having 10 or 20 or more Mbits/sec of download speed dedicated to a single PC. They dont understand (or care) that their office connection may have considerably less available bandwidth.
I suggest having a broadband connection (FIOS, Cable, or DSL) for general web use, and a symmetric (t1, or metro ethernet) connection with a good carrier SLA for site to site VPNs, RDP, etc. The SLA for metro ethernet, t1, or bonded t1 should guarantee shorter time to repair than a entry level business class broadband service if there is a circuit failure. Its also possible in many cases to do QoS marking on vpn traffic that crosses a t1 based or metro ethernet based internet service whereas with Cable or Verizon FIOS, you cannot mark traffic and you will often be rate shaped in the cloud which can cause problems if you do not plan for this properly.
1. Small (regular size embedded) screen youtube.com video will take about 1.0Mbits/sec of bandwidth to play.
2. Full size (full screen) youtube.com video: 3.0 to 3.8 Mbits/sec.
3. Videoconference sessions typically take 1.5 to 3.0 Mbits/sec depending on the video quality.
Customer had an ASA firewall in place, and an Exchange server that was failing to deliver mail to certain locations. Getting good SMTP logs from the Exchange server can be challenging, and mail delivery was failing even after we removed the infamous inspect esmtp command from the ASA the previous admin had left enabled.
We decided to insert an OpenBSD SGW box to smart host outbound SMTP from the exchange server and provide web content filtering.
The problem was that while the ISP had provided a /29 the ISP edge router only had a single ethernet downlink and the ASA 5505 port 0 was connected to that port. The 5505 has a VLAN-capable switch built into it. So we simply plugged the SGW box uplink into a free port on the ASA and set that port for the outside VLAN (VLAN2 by default):
ASA 5505 config cchanges for port 5:
! Uplink to isp adran router:
interface Ethernet0/0
switchport access vlan 2
! Handoff to inside network:
interface Ethernet0/1
switchport access vlan 1
Handoff to public iface on bsd appliance:
interface Ethernet0/5
switchport access vlan 2
The SGW box can now be hooked up in parallel with the ASA so that it has 1 inside (LAN) interface and 1 outside (public ip) interface. This will allow the SMTP messages to be transmitted from Exchange to postfix w/o going thru the ASA firewall.
The postfix process can be tuned to help with delivery of messages to more difficult destinations:
Tweaks to improve delivery to yahoo.com and other mail providers that rate limit or block aggressive senders, this can happen if you mail a distribution list with lots of yahoo members for example:
default_process_limit = 6
local_destination_concurrency_limit = 2
default_destination_concurrency_limit = 2
initial_destination_concurrency = 2
This will imrpove logging - add this to main.cf
header_checks = regexp:/etc/postfix/header_checks
In this file: /etc/postfix/header_checks
Place these 2 lines:
/^subject:/ WARN
/^from:/ WARN
These 2 commands in main.cf will help with delivery to sites that have buggy cisco firewalls that try to inspect esmtp conversations:
smtp_pix_workaround_delay_time = 10s
smtp_pix_workaround_threshold_time = 500s
The ASA 5505 has 8 ports, but in the base license only 2 vlans can be created. The other 6 ports are simply switch ports that can be placed in the private or outside vlan. If you have an advanced license that allows for DMZs to be created then you can use the physical ports on the back of the ASA as dmzs for things like servers, guest wireless networks, etc.
The ASA 5510 model has mutiple ethernet ports as well, but these are layer3 firewall ports, not bridge/switch ports like in the ASA5505.
The ASA is NOT a router, though and while you can do things on the ASA that can make it act something like a router it is important to understand the differences between true routing and what the ASA actually does.
If you set 2 networks on the ASA to have the same security zone, you can permit traffic to pass between the interfaces with the same-security-traffic permit inter-interface command.
You can also use nat to nat traffic from one interface to another, and you can create static nats that do not actually nat, but keep state on the connections and pass the traffic w/o changing the addresses or ports.
The ASA though cannot participate in BGP routing, which makes in impossible to use in most situations where you want to talk to an upstream service provider or use with an MPLS, or GRE tunnel based WAN or value added network. You also cannot use dynamic routing protocols across the ipsec tunnels that the ASA creates, which is the most serious limitation with the ASA in my mind. BSD and/or cisco routers are much better choices for creating ipsec based tunnels across the internet because you can use:
gre + ipsec + BGP
or
gre + ipsec + OSPF/EIGRP (eigrp is cisco proprietary of course)
and get full ip routing with a stateful protocol like BGP across your tunnel. This makes failover between an MPLS cloud and a gre based backup cloud, or 2 gre tunnel paths 1 primary and 1 backup much quicker and simpler to administer than the raw ipsec methods.
The ASA does support OSPF, so it CAN learn about routes from other OSPF speaking gateways. I have found that the OSPF support is buggy in earlier 8.x releases of ASA code on the ASA5505 though. 8.2.2 seems to be pretty solid though. But just because the ASA can speak OSPF does not mean its the right choice for routing or connecting between 2 subnets and its certainly a poor choice for connecting WAN and internet based private clouds.
Even if you do need to create a security policy between 2 networks (say a research and a production lan) it still may be simpler to use a router with an ACL(s) or a layer 3 switch with ACLs between VLANs than the ASA.
Bottom line: The ASA is a solid firewall but its not a router. If you need a router and routing protocols, use a Cisco 1941 (new), 1841 (used gear). The 1941s are very comparable to the ASA5510 in terms of throughput. The 1841s are excellent low cost choices for networks that need 30 Mbits/sec or less of IP throughput and 10 to 12 Mbits/sec or less of total IPSEC throughput.
If you want a single box that combines the routing capabilities of a cisco 1941 with the firewalling capabilities of an ASA AND the layer 7 proxy capabilities (dns routing, rinetd tcp port redirect, tcp proxy, web proxy, smtp relay, etc ....) use an OpenBSD router/firewall!
This issue was seen by Richweb recently when mail from a 3rd party network was not being delivered. Initially the Mail Filter Richweb operates was suspected, as some mails (short text) would get through but longer text emails and html emails would not.
Richweb started a tcpdump (packet trace) on the firewall and observed that the mail flow would start, and then would hang during the SMTP DATA phase.
1. We found out that the 3rd party network:
A. did NOT have the infamous smtp fixup protocol Cisco PIX problem
B. was running a non-cisco firewall that blocked ALL ICMP traffic.
Filtering ALL ICMP of course is a bad idea. We referred the admin to this resource initially:
http://blogs.richweb.com/icmp_filter
C. had domains with incorrect SPF records
We asked the admin to fix the SPF records, and we whitelisted the domains in the meantime on our MailFoundry appliance.
2. We then confirmed that the mail foundry does indeed have tcp window scaling turned on (all OSes built after 2004 or 2005 or so that are trying to do better tcp bulk data transfer will have this option turned on by default, Solaris, Linux, BSDs, Windows, etc.).
3. We believe that the remote firewall in use was stripping the TCP window scaling options which was causing the issue.
What did we learn ?
Make sure this option is not set in your firewall (Cisco ASA Pixes)
tcp-options window-scale clear
http://www.cisco.com/en/US/docs/security/asa/asa70/configuration/guide/ids.html
Clears the selective-ack, timestamps, or window-scale TCP options, or drops
a range of TCP options by number. The default is to allow packets with
specified options, or to clear the options within the range, so use this
command to clear, allow, or drop them.
hostname(config-tcp-map)# tcp-options {selective-ack | timestamp |
window-scale}
{allow | clear}
For older 6.x pixes:
http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a0080742d6e.shtml
From the doc:
dropped TCP connections [are] caused by some versions of PIX software not
supporting the TCP Window Scaling option. This causes it to have a much
smaller TCP window than the endpoints actually have. This causes the Cisco
PIX to drop packets that it believes are outside the TCP window, but which
really are not.
This bug: CSCsg00748
Was resolved in pix 7.2(2)
Clear window-scale sack option in non-syn packets instead of dropping it
More information - excerpted from:
http://lwn.net/Articles/92727/
TCP window scaling and broken routers
[Posted July 7, 2004 by corbet]
Every TCP packet includes, in the header, a "window" field which specifies how much data the system which sent the packet is willing and able to receive from the other end. The window is the flow control mechanism used by TCP; it controls the maximum amount of data which can be "in flight" between two communicating systems and keeps one side from overwhelming the other with data.
In the early days of TCP, windows tended to be relatively small. The computers of that age did not have huge amounts of memory to dedicate toward buffering network data, and the available networking technology was not fast enough to make use of a larger window in any case. Modern network interfaces can handle larger packets and keep more of them in flight at any given time; they will perform better with a larger window. Some kinds of high-speed long-haul links can have very high bandwidth, but also high latency. Keeping that sort of pipe filled can require a very large window; if a sending system cannot have a large number of packets in transit at any given time, it will not be able to make use of the bandwidth available. For these reasons, good performance can often require very large windows.
The TCP window field, however, is only 16 bits wide, allowing for a maximum window size of 64KB. The TCP designers must have thought that nobody would ever need a larger window than that. But 64KB is not even close to what is needed in many situations today. The solution to this problem is called "window scaling." It is not new; window scaling was codified in RFC 1323 back in 1992. It is also not complicated: a system wanting to use window scaling sets a TCP option containing an eight-bit scale factor. All window values used by that system thereafter should be left-shifted by that scale factor; a window scale of zero, thus, implies no scaling at all, while a scale factor of five implies that window sizes should be shifted five bits, or multiplied by 32. With this scheme, a 128KB window could be expressed by setting the scale factor to five and putting 4096 in the window field.
To keep from breaking TCP on systems which do not understand window scaling, the TCP option can only be provided in the initial SYN packet which initiates the connection, and scaling can only be used if the SYN+ACK packet sent in response also contains that option. The scale factor is thus set as part of the setup handshake, and cannot be changed thereafter.
The details are still being figured out, but it would appear that some routers on the net are rewriting the window scale TCP option on SYN packets as they pass through. In particular, they seem to be setting the scale factor to zero, but leaving the option in place. The receiving side sees the option, and responds with a window scale factor of its own. At this point, the initiating system believes that its scale factor has been accepted, and scales its windows accordingly. The other end, however, believes that the scale factor is zero. The result is a misunderstanding over the real size of the receive window, with the system behind the firewall believing it to be much smaller than it really is. If the expected scale factor (and thus the discrepancy) is large, the result is, at best, very slow communication. In many cases, the small window can cause no packets to be transmitted at all, breaking TCP between the two affected systems entirely.
In the 2.6.7 kernel, the default scale factor is zero; in Linus's BitKeeper tree and the 2.6.7-mm kernels, instead, it has been increased to seven. This change has brought the broken router behavior to light; suddenly people running current kernels are finding that they cannot talk to a number of systems out there. One of the higher-profile affected sites is packages.gentoo.org. Gentoo users are, unsurprisingly, not pleased.
As a way of making things work, Stephen Hemminger has proposed a patch which adds a calculation to select the smallest scale factor which covers the largest possible window size. The result on most systems is that the scale factor gets set to two. This factor will still be corrupted by broken routers, but the resulting window size (¼ of what it should be) is still large enough to allow communication to happen.
The patch makes networking with systems behind broken routers work again, but it has been rejected anyway. The networking maintainers (and David Miller in particular) believe that the patch simply papers over a problem, and that adding hacks to the Linux network stack to accommodate broken routers is a mistake. If, instead, the situation is left as it is, pressure on the router manufacturers should get the problem fixed relatively quickly. It has been a few years, now, that Linux has a strong enough presence in the networking world that it can get away with taking this sort of position.
In the mean time, anybody running a current kernel who is having trouble connecting to a needed site can work around the problem with a command like:
echo 0 > /proc/sys/net/ipv4/tcp_default_win_scale
or by adding a line like:
net.ipv4.tcp_default_win_scale = 0
to /etc/sysctl.conf