Creating a Linux LVM Logical Volume on an iSCSI SAN

Recently I had an Oracle database server used by some developers that was running out of space on its data volume mounted at /u02. The volume was a simple MBR volume (think fdisk), so it couldn’t be non-destructively extended without using a third-party utility like gparted. That would have been fine, but rather than leave the volume as MBR, I decided to create a new iSCSI SAN-backed Logical Volume Manager (LVM) volume, which can be extended and resized pretty easily.

In this post, I’ll show you how to create a logical volume stored on an iSCSI SAN. Even though I did this on Red Hat Enterprise Linux 6.5 (RHEL), these steps should work on any distribution of Linux. Continue reading

Creating a Linux File Server for Windows CIFS/SMB, NFS, etc.

Recently I needed to build a multipurpose file server to host CIFS and NFS shares — CIFS for the Windows users, and NFS for VMWare to store ISOs. It needed to utilize back end storage (NetApp via iSCSI), provide Windows ACLs for the CIFS shares, and be able to authenticate against two different Active Directory domains. After careful consideration, I decided to use Red Hat Enterprise Linux 6.5 (RHEL) instead of Windows Server 2012.

Now you might be wondering, “Why on earth would you want to build a Linux file server to do all that when you can just use Windows?” There are a few reasons: Continue reading

How to Make NetApp Use the Correct Interface for iSCSI

If you’re familiar with networking you know that when a device is directly connected to two separate IP networks, traffic destined for one of those networks should egress on the interface that is directly connected to that network. For example, if your storage appliance is directly connected to the network, and you want to send a packet to a device with the IP of, traffic should egress on the interface connected to that network. Unfortunately, in the case of some NetApp filers, this does not always happen.

I ran into a peculiar issue when trying to force NetApp’s Snapmirror to replicate across a specific interface, only to be met with an ugly “Snapmirror error: cannot connect to source filer (Error: 13102)”. I confirmed with NetApp support that the Snapmirror configuration was correct for what I was trying to accomplish.

To troubleshoot, I started a packet trace on the destination filer using the command:

pktt start all -d /etc

I then kicked off the snapmirror initialization, waited for it to fail, then stopped the packet trace with

pktt stop all

Since I directed the trace files to be placed in /etc on the filer I just browsed to the hidden etc$ CIFS share on the filer and opened the traces in Wireshark. What I found was that the traffic that should have been egressing on the iSCSI VIF was actually going out on the LAN VIF. Not only that, the filer was using its iSCSI address on the LAN VIF! I’m always hesitant to label every quirk a “bug,” but this is definitely not correct behavior.

The remedy was as simple as adding a route statement similar to this:

route add inet 1

where is the iSCSI network I want to traverse to reach the Snapmirror partner, and is the gateway on my locally connected iSCSI network. The 1 specifies the cost metric for the route, which will always be 1 unless you need to add additional gateways.

To make the change permanent, simply add the route statement to the /etc/rd file on the filer.

Special thanks to NetApp’s Scott Owens for pointing me in the right direction on this.

Deduplication For Everyone

If you have a lot of duplicate data sitting around on laptops, USB drives, and external hard drives, you probably would like to clean it up and get back some of that wasted free space. But you probably don’t have time to go through and delete all the duplicates, or you’re concerned if you do you will mis-identify a duplicate and accidentally delete your only copy.

I ran into this problem recently when I came dangerously close to running out of storage space on my NAS box which was running Freenas 0.7. I purchased an external hard drive to handle the overflow, but it was just not convenient to keep the drive attached whenever I needed something. Nor was it fun searching through the NAS and the external drive whenever I was looking for a file and couldn’t remember where it was. I didn’t have time to go and manually delete duplicate data (and I knew I had a ton of it), so I started looking for a free NAS solution that incorporated deduplication (or dedup). I quickly discovered that Sun/Oracle’s ZFS filesystem had gotten dedup in late 2009, so I started doing some researching on how I could build a NAS box with OpenSolaris and the new and improved ZFS. After much searching (and struggling with getting OpenSolaris to even behave right in VMware ESXi 3.5) I found that someone else had already done the work for me.

NexentaStor is a free (for home use anyway) OpenSolaris-based storage “appliance” that has ZFS and dedup! I loaded up my old Dell PowerEdge server with some hard drives, installed it, turned on deduplication, and started dumping my data onto it. I wish I could say the process of copying data went flawlessly, but it didn’t. The appliance froze up several times, requiring a hard power off, and I’m guessing it’s because the ZFS logbias was set to latency instead of throughput and it was creating some kind of weird race condition. I don’t know. But I do know that when I changed the logbias to throughput it never froze up again. Anyway, my dedup ratio ended up being around 2.20x, which basically means that for every 2.2 GB of data I was writing, it was only taking up 1 GB of disk space. I ended up with a huge amount of free space.

But there was one problem. When creating the ZFS storage pool, I didn’t make it redundant. I just strung all my disks together to maximize the storage space, and that’s a very bad idea. If you remember nothing else about ZFS, remember this: Once you create a non-redundant ZFS storage pool, it’s non-redundant forever. I had to pull all the data off, delete the pool, and create a new one (type raidz1, the logical equivalent of raid-5). This ate up a lot of my available storage space, and I ended up having to split some of my easily replaceable data off onto a Freenas server. I also turned on gzip compression, which didn’t do much. My compression ratio is about 1.02x, but my dedup ratio is 1.74x.

Overall, I highly recommend trying NexentaStor if you absolutely must have deduplication or redundancy. ZFS is a rock-solid file system, despite Apple turning it down for OS X. I applaud the Nexenta folks for creating this product, and not just creating it, but making it very robust (you can get to a shell if you want or need to) and user-friendly at the same time . But if you just want a quick and dirty NAS solution and don’t care about dedup, I recommend Freenas.