Rahsennor wrote:I've heard rave reviews of ZFS but I'm not enough of a server buff to tell the fluff from the features. You obviously rate it highly enough to base your choice of distro on it, so should I be looking into it too? What does it do that other filesystems don't?
Not exactly. Some clarification:
I don't "rate it (ZFS) high enough", it's that I know what my needs and preferences are, and that I'm a "storage" guy (I often work with disks and storage protocols like ATA/SATA). I always recommend that people just stick with known/working/established filesystems (like FFS/UFS or ext4) as defaults. Hell, even I use those for boot/root filesystems (cold day in hell before I ever use ZFS on root!).
I generally do not recommend ZFS on FreeBSD to people because of all of the caveats involved -- particularly the ARC and how the underlying VM (virtual memory, not virtual machine!) ends up behaving with the ARC. Do not let the fact that the OS installer gives you ZFS as a choice sway your mind (when this was introduced, I actually argued against it). Some examples of caveats: you *must* do system tuning to get ZFS on FreeBSD to behave itself (particularly regarding ARC usage), snapshots are still known to cause stability problems for some people, if using compression you should use lz4 and nothing else (otherwise suffer from a system that "hitches" or "stalls"), there have been reports of problems with send/recv, and do not use dedup no matter how tempted you may be. The point is that when ZFS "issues" or "quirks" come up, they are often very severe and painful in nature, and nearly impossible for a user to figure out; an inexperienced user would have absolutely no idea what to do (and many wouldn't even think the issue/quirk may be caused by filesystem choice!). I happen to have spend many (painful) years with the thing so I've learned what to watch out for or avoid, but What ZFS brings to the table that spoils users (myself included) are easy/simple volume and filesystem management (which includes RAID-like functionality) and a checksumming filesystem; once you get used to the former, doing things the old way feels tedious and annoying.
I used to have several FreeBSD machines all over the place -- maybe 12 or 13 at the peak -- but only 2 of those used ZFS (I stuck with standard FFS/UFS everywhere else, and intentionally did not use SUJ (softupdates + journalling; SU is fine, it's the journalling part you should avoid)). These days I only have two machines, and I'm trying to get both of them moved to Linux. Only one of them uses ZFS, and it is very important that I stick with ZFS.
In comparison (since it's another filesystem that offers volume management and is new), Btrfs is a joke. I've played with it on a VM (virtual machine) a couple times and I just cannot get over how absolutely ridiculous it is. There are so many WTFs about its behaviour, particularly in administration, that I question why it even exists. ZFS's administrative model feels very natural/sane and is hard to beat. In contrast, LVM is a nightmare, often because of the PV vs. VG vs. LV terminology and overall design; it confuses even senior administrators who choose to use it (I often recommend mdadm for that exact reason); as a storage guy I'm "okay" with LVM.
I didn't "choose FreeBSD because of ZFS". I've been using FreeBSD since 2.2.7 as stated (that's roughly mid-1998). Prior to that I used Linux -- I started with 0.99pl45, and used to test things like CSLIP/SLIP drivers for kernel and userland developers, and I moved away from Linux in the mid-90s (Linux 1.3.x days) because I had grown extremely tired of how chaotic the kernel was (mainly: stock Linux required patches for major bugfixes, which came from multiple people (Alan Cox and several others), so you had patches atop patches and often did not apply cleanly, plus the Linux kernel at the time was, from a security perspective, absolutely nightmarish). But prior to Linux I was actually brought up on SunOS 4.x. Anyway, ZFS was introduced to FreeBSD in version 7.x (so roughly 2008), and I started playing with it for the purpose of wanting something RAID-like for both a server in a datacenter that did backups for the rest of the network, and for a home NAS/SAN for personal use; I had experience with ccd and vinum from my day job, and thought ZFS might make my life easier -- it both did and did not. The situation has not changed in over 10 years (though, with 9.x there was a "sweet spot" in FreeBSD ZFS behaviour that felt stable and understood; they've screwed with it in 10.x and later, so now things are precarious again).
It's funny, because ZFS on Solaris or Illumos -- where I spent time with the former for many years as part of my day job -- "just works", all because key parts of the kernel and VM were tweaked and designed with ZFS in mind.
As for ZFS on Linux: it's new material for me. I have no idea what its ARC or VM behaviour is like. But so far, playing around on a VM (virtual machine) with it, I've found several things that differ from FreeBSD and Solaris ZFS that make me a bit wary. At around the time I was looking into it, distros started adopting systemd, which threw a large wrench into my plans to migrate off FreeBSD to Linux.
As for what ZFS brings to the table? You can see Wikipedia or use a search engine and find out for yourself. Just do not forget about the caveats (at least on FreeBSD) I mentioned above; basically no one ever discusses those in blog posts, Wikipedia, etc., while if you peruse the mailing lists for the past several years you'll see exactly what I'm talking about.