Net Neutrality is a Scam

One of the biggest scams of the Internet is in full swing right now. You may have heard of it. It’s called “net neutrality.”

Fundamentally, net neutrality is about preventing Internet service providers (ISPs) from throttling or blocking traffic or providing paid prioritization of certain content. In addition, specific rules proposed by the FCC Chairman Tom Wheeler would allow the FCC to arbitrate peering disputes between carriers. Traditionally, carriers have connected each other’s networks with each other for a nominal cost or none at all. The idea being that the mutual benefit of using each other’s network for transit is payment enough. The proposed FCC rules, however, will turn this once amicable transaction into a litigious battleground that could result in the destabilization of the Internet’s backbone.

I recall an article from a 1997 issue of Wired magazine which predicted the collapse of the Internet would be caused by increased growth without the infrastructure to support it. That never happened, in part due to technical innovation which kept up with growth, but also because ISPs and backbone carriers were able to throttle traffic during peak times to ensure everyone could have reasonably fast and reliable internet access.

Now, almost 20 years later, we’re looking at potential regulation that will micromanage how ISPs manage and build out their networks. As a network engineer, I understand the need to throttle or simply block certain types of traffic. But unfortunately, the technical facts have gotten lost amidst the raw politicization of the net neutrality debate. I recently saw a graphic put out by the pro-net neutrality group “Battle for the Net” that shows a picture of the United States Senate and a caption that asks, “Does your state have the Internet’s worst enemy?” It then proceeds to list all the Senators that are supposedly trying to “kill Net Neutrality.” And this is the problem with the net neutrality movement. It’s purely political and devoid of any thoughtful technical or practical discussion. Organizations like Battle for the Net don’t bother to make a case for net neutrality. They assume that it is an absolute good and that being for the Internet means being for net neutrality.

The discussion has devolved from a debate into a marketing battle plagued by word games and politics. Net neutrality advocates have adopted the language that this is “a battle for the Internet” and an effort to “keep the internet open.” Apparently, by breaking decades of precedent and giving the FCC more power to control what Internet service providers do, the Internet will somehow become better. The narrative they put forth is that the big bad cable companies with their zillions of dollars are trying to make end users’ Internet experience slow and expensive, and are fighting valiant efforts to “keep the internet free” (Nevermind the fact that the cable companies gave us broadband Internet and brought us out of the dial-up era to begin with.) This David versus Goliath theme is great for stirring emotions, but it falls flat in the face of a little bit of scrutiny. Google, whose income is more than double that of Comcast, is strongly in favor of “net neutrality” regulations. So is Netflix. And Facebook.

Regardless of where you stand on net neutrality, one thing is certain: this is not about big money corporations versus the gentle folks of the Internet. It is about giant corporations duking it out for power, control, and government favor. As usual, the politics of net neutrality has turned the debate into more of a sporting event where everyone roots for his own team no matter what. But it’s actually worse than that. If you’re against net neutrality, some will perceive you as being anti-Internet or against Internet freedom. I find this both amusing and disturbing. Amusing, because the notion that giving the FCC unprecedented regulatory power over the Internet will somehow increase freedom to be absurd. And disturbing, because so many have blindly taken sides on this debate without an understanding of its implications or what it’s even about.

One such implication is privacy. How will the FCC ensure that ISPs are complying with the new regulations and not throttling or blocking certain types of traffic? The only way to know is by looking at the traffic, which can only be done with detailed logs of what an ISP’s users are doing. This goes beyond what websites you visited or how many gigabytes you downloaded. This gets down to individual connections. What IP address and port did you connect to? What protocol were you using? Certainly, these things can be logged now, and in fact probably are. But the difference is that, as of now, the FCC has no authority to demand such logs. With net neutrality regulations in place, they will, and they will also have the power to exact fines if ISPs fail to retain logs for a certain period of time. So, you will be able to BitTorrent without restriction, but Uncle Sam is probably going to know about it. Of course, this is already happening with the NSA pretty much spying on everything. But again, the difference is that instead of spying secretly, the collection of your Internet activity will be open and shameless. That may not bother you. Honestly, it doesn’t really bother me. The point is that net neutrality regulations come with some pretty long and tangled strings attached. And it’s wise to unravel them and see where they lead before throwing in your support for the wolf in sheep’s clothing.

How to Make NetApp Use the Correct Interface for iSCSI

If you’re familiar with networking you know that when a device is directly connected to two separate IP networks, traffic destined for one of those networks should egress on the interface that is directly connected to that network. For example, if your storage appliance is directly connected to the network, and you want to send a packet to a device with the IP of, traffic should egress on the interface connected to that network. Unfortunately, in the case of some NetApp filers, this does not always happen.

I ran into a peculiar issue when trying to force NetApp’s Snapmirror to replicate across a specific interface, only to be met with an ugly “Snapmirror error: cannot connect to source filer (Error: 13102)”. I confirmed with NetApp support that the Snapmirror configuration was correct for what I was trying to accomplish.

To troubleshoot, I started a packet trace on the destination filer using the command:

pktt start all -d /etc

I then kicked off the snapmirror initialization, waited for it to fail, then stopped the packet trace with

pktt stop all

Since I directed the trace files to be placed in /etc on the filer I just browsed to the hidden etc$ CIFS share on the filer and opened the traces in Wireshark. What I found was that the traffic that should have been egressing on the iSCSI VIF was actually going out on the LAN VIF. Not only that, the filer was using its iSCSI address on the LAN VIF! I’m always hesitant to label every quirk a “bug,” but this is definitely not correct behavior.

The remedy was as simple as adding a route statement similar to this:

route add inet 1

where is the iSCSI network I want to traverse to reach the Snapmirror partner, and is the gateway on my locally connected iSCSI network. The 1 specifies the cost metric for the route, which will always be 1 unless you need to add additional gateways.

To make the change permanent, simply add the route statement to the /etc/rd file on the filer.

Special thanks to NetApp’s Scott Owens for pointing me in the right direction on this.

Using IRQbalance to Improve Network Throughput in XenServer

If you are running XenServer 5.6 FP1 or later, there is a little trick you can use to improve network throughput on the host.

By default, XenServer uses the netback process to process network traffic, and each host is limited to four instances of netback, with one instance running on each of dom0’s vCPUs. When a VM starts, each of its VIFs (Virtual InterFaces) is assigned to a netback instance in a round-robin fashion. While this results in a pretty even distribution of VIFs-to-netback processes, it is extremely inefficient during times of high network load because the CPU is not being fully utilized.

For example, suppose you have four VMs on a host, with each VM having one VIF each. VM1 is assigned to netback instance 0 which is tied to vCPU0, VM2 is assigned to netback instance 1 which is tied to vCPU1, and so on. Now suppose VM1 experiences a very high network load. Netback instance 1 is tasked with handling all of VM1’s traffic, and vCPU0 is the only vCPU doing work for netback instance 1. That means the other three vCPUs are sitting idle, while vCPU0 does all the work.

You can see this phenomenon for yourself by doing a cat /proc/interrupts from dom0’s console. You’ll see something similar to this:

(The screenshot doesn’t show it, but the first column of highlighted numbers is CPU0, the second is CPU1, and so on. The numbers represent the quantity of interrupt requests.)

If you’ve ever troubleshot obscure networking configurations in the physical world, you’ve probably run into a router or firewall whose CPU was being asked to do so much that it was causing a network slowdown. Fortunately in this case, we don’t have to make any major configuration changes or buy new hardware to fix the problem.

All we need to do to increase efficiency in this scenario is to evenly distribute the VIFs’ workloads across all available CPUs. We could manually do this at the bash prompt, or we could just download and install irqbalance.

irqbalance is a linux daemon that automatically distributes interrupts across all available CPUs and cores. To install it, issue the following command at the dom0 bash prompt:

yum install irqbalance --enablerepo base

You can either restart the host or manually start the service/daemon by issuing:

service irqbalance start

Now restart your VMs and do another cat /proc/interrupts. This time you should see something like this:

That’s much better! Try this out on your test XenServer host(s) first and see if you can tell a difference. Citrix has a whitepaper titled Achieving a fair distribution of the processing of guest network traffic over available physical CPUs (that’s a mouthful) that goes into more technical detail about netback and irqbalance.

Deploying Citrix Access Gateway VPX with Web Interface 5.4 – CAG Setup with RADIUS

Deploying CAG with Web Interface 5.4 is actually very easy, there are just some “gotchas” that you have to be ready for. This is a guide to help you avoid those snags and pitfalls that commonly occur with a CAG VPX and Web Interface integration.

I recommend getting the Citrix Access Gateway VPX Getting Started Guide and HDX Remote Access Guide with Citrix Access Gateway VPX Express if you don’t already have them. The former document contains some inaccuracies but is has some useful reference info as well. The latter takes you through the fundamental setup of the CAG VPX and gets you to the web administration console, where most of the meaty configuration will take place.

There are some assumptions I’m making with this guide since it is based on my own requirements. They are:

  • The CAG VPX has two virtual NICs, external to service external users, and internal for management and communication with the XenApp servers
  • Two logon points will be configured: One that allows user authentication to take place at the web interface, and another that uses RADIUS
  • The CAG VPX will not reside in a DMZ. If your situation requires it to reside in a DMZ, setting it up is trivial once you’ve gotten everything else working.
  • You’ve already got the CAG VPX appliance imported and running, but not configured
  • You have your web interface server setup, with no websites configured.
  • You have installed the CAG VPX license on a Citrix license server

Also, a word of warning: Configuring CAG VPX with Web Interface is like playing chess. Every move you make will affect all of your successive moves. In areas where I suspect your setup might require you to deviate from this guide, I’ll offer some pointers to help you make the right move.

Let’s get started! First, follow the Getting Started guide to configure the management interface for the CAG via the VM console. If you are unsure about a setting, just take the default by hitting Enter.

Once you have your management IP assigned, you’ll need to access the web administration console by browsing to https://[IP address]/lp/adminlogonpoint . Login with the default username and password “admin”. You’ll see a nice dashboard with two dials and some nasty looking red X’s. Click on the Management tab. This will take you to the Networking portion of the System Administration menu group.

Here, enter the CAG’s hostname as an FQDN. You’ll see a list of your network interfaces (eth0, eth1, etc.) with one of them having the management IP address you assigned. To the right, there are four checkboxes labelled Internal, External, Appliance Failover, and Management.

Moving from left to right, the interface that will be used to connect to your XenApp servers should have the Internal checkbox checked.
The interface for management should already have the Management checkbox checked.
The interface that will be receiving external requests from end users should have the External checkbox checked.

If need be, your Internal and External interfaces can be the same. Make sure your DNS servers have an entry for your Internal IP!

Under the Default Gateway section, select the interface the CAG should use to route traffic for subnets to which it is not directly connected. This will probably be your External interface. Remember, the CAG has a direct connection to the subnet your XenApp servers are on, so it doesn’t need a gateway to get to those. But it does need a gateway to get back to your external users who are connecting from the Internet!

Click Save and restart the appliance using the big Restart button on the top right.

Log back into the web administration console and browse to Management > Name Server Providers. Enter your DNS servers and DNS suffixes.

Now go to Static Routes in the System Administration menu. Do you need a static route? If you plan on putting the CAG VPX into a DMZ later, go ahead and enter your static routes. Gateways you specify for static routes will take precedence over the default gateway you specified earlier. Remember, it does not hurt to add them now.

Now Browse down a few rows to Licensing and click Configure. Select the Licensing type and Remote Server for the licensing server. Enter the FQDN or IP of your Citrix licensing server, and click Save. The CAG will attempt to grab its licenses and upon a successful retrieval, it will display them as shown:

NB: If the CAG is unable to retrieve the licenses, I recommend stopping and troubleshooting until it is able to successfully pick up the licenses. You can continue your configuration, but you will not be able to test it until the license issue is resolved.

Moving right along, click on Authentication Profiles under the Access Control menu group. It’s time to add a RADIUS authentication profile! But before you do that, you have to set up a RADIUS server. I recommend reading How to Configure Radius Authentication/Authorization on Windows 2008 for Use on Citrix Access Gateway Standard Edition. (One caveat, however: don’t perform steps 13-17 in the KB article because they’re unnecessary and will cause problems.) Click Add and enter a name for the Authentication Profile. Click New and add your RADIUS server(s) and shared secret. Leave everything under Group Authorization as-is. You’re relying on the RADIUS server to check group authorization.

Now go to XenApp or XenDesktop under Applications and Desktops and enter the IP ranges of clients that can access XenApp servers via ICA and CGP. I really don’t know why there isn’t a checkbox that allows you the equivalent of a “permit ip all”, but there isn’t.

Next, click Secure Ticket Authority. These settings are arguably the most common cause of application launch issues. Select your STA servers carefully, and make sure all your XenApp servers have unique STA ID’s! If you are running Provisioning Services and streaming XenApp, read before proceeding. Once you are sure your STA IDs are unique, click New and enter the FQDN of each XenApp server that will be providing STA services. By default, the connection type is Secure, but I’m guessing your XenApp servers are not using SSL for STA traffic, so select Unsecure. Leave everything else as-is and click Add. Note the servers you selected here, because you will need them later.

At this point you can go ahead and restart the CAG VPX appliance, because it’s now time to do some work on the Web Interface (WI) side. Log into your WI server, launch the Citrix Web Interface Management console, and create a new site.

Should we do the easy one or the hard one first? Trick question. They’re both easy! We’ll set up a site to be used with RADIUS authentication. Click Create Site under the Actions pane on the right, name the website however you wish and click Next. Select “At Access Gateway” as the point of authentication and click Next. Here you’ll be greeted with an intimidating looking Authentication Service URL field. But as I said, this is easy! Just enter https://[cag-FQDN]/CitrixAuthService/AuthService.asmx and click Next, Next. After a few moments, your site will be created.

Right-click the site in the XenApp Web Sites list and select Server Farms. Configure your XenApp servers like you normally would in WI. There is nothing here unique to CAG.

Now right-click the site again and select Secure Access. Select the only item in the list and click Edit. Change Access method to Gateway direct and click Next. Next you’ll be asked for the address of the CAG. Enter the FQDN of the CAG, and optionally enable or disable session reliability. If enabled, you can request tickets from two STAs (A word on this option: When you setup a site for Citrix Receiver, this checkbox must be unchecked. This site you are creating now cannot be used for Receiver, so don’t worry about it here if you plan to use Receiver.) Click Next.

Remember I said to note the XenApp servers you entered into the CAG VPX as your STA servers? Web Interface wants to know about these servers too. Click Add and enter the URL of the first XenApp server you entered into the CAG in the following format:

I still do not know why it doesn’t just ask for the FQDN and assume the rest like the CAG does, but that’s how it is. Do this for all STA servers you entered into the CAG, and in the same order. Check, double check, and triple check the URLs! Also optionally change the “Bypass failed servers for” option to 1 minute. Click Finish.

Are we done? Almost. The CAG should be back up now, so log back into it. Click Logon Points under Access Control and click New. Now don’t be intimidated by all the settings. We only care about four things here. Enter the name of the logon point. Select this name carefully because it is what users will have to type in to connect to the CAG. If you enter “cag-logonpoint1” then users will have to go to which just looks ugly! Under Type select Basic. In the Web Interface field, enter the URL of the web interface site you created (no trailing slash). Under Authentication Profiles, select the RADIUS authentication profile you created earlier. Finally, check the “Single sign-on to web interface” check box and click Save.

Now, we are almost ready to test. But to save ourselves from a disappointing moment of temporary CAG dysfunction, click on Secure Ticket Authority again. Do you see unique STA IDs populated next to each of your XenApp servers? If not, troubleshoot until you do. If so, it’s time to test!

Browse to the logonpoint you just created. If you named your logonpoint “test1” and your CAG’s FQDN is, browse to

Do you get an SSL certificate error? Probably so. I intentionally did not cover installing a certificate because it introduces another level of complexity into the configuration. SSL Certificates are dependent on the hostname, and the hostname you use to connect to the CAG to enumerate and launch apps has to match up with the hostname on the SSL certificate. The certificate also must be signed by a trusted certificate authority. Unless you have your own certificate authority, getting a signed certificate can be a pain. Unfortunately, connecting to CAGs using untrusted SSL certs causes a lot of problems. You may encounter some of these problems or you may not. Test anyway. If you do run into problems, the good news is that the heavy lifting of configuring the CAG is done.

Now, it’s time for a moment of truth. Once you’ve gotten a login prompt, log in with an AD account that has appropriate authorization. You may need to specify the user in UPN or down-level domain format. It depends on your AD environment, but one of those should work. If you have trouble authenticating, first check the logs on the RADIUS server to make sure the denial is not occurring there. If you continue to have issues, it’s time to get acquainted with what will become your new best friend: the CAG debug log. The CAG debug log is at your service at . Watch it for any FLEXnet or STA errors.

Once you get logged in, try launching a published app. If all goes well, you should see something like…

Citrix XenApp6 0x80060016 Error In PowerShell

I ran into a little snag when executing some XenApp PowerShell commands. Certain commands like Get-XAFarm and Get-XAAdministrator would always give an “0x80060016” error. Here is an example and the fix:

PS C:\Windows\system32> Get-XAFarm
Get-XAFarm : Error reading the current administrator data (0x80060016)
At line:1 char:11
+ Get-XAFarm <<<<
+ CategoryInfo : InvalidResult: (:) [Get-XAFarm], CitrixException
+ FullyQualifiedErrorId : GetCitrixAdminType,Citrix.XenApp.Commands.GetFarmCmdlet

Typically this error code in Citrix indicates a problem with IMA. But in this case it was even simpler than that: IMA couldn’t resolve the hostname of the database server hosting the data store. Make sure that the correct DNS suffixes are being applied so IMA can find the server, and if that fails, just add it to the hosts file and try again.