This post is the second of two describing the features of the VMware vSphere Storage APIs for Array Integration (VAAI) version 2 as it works with vSphere 5. If you missed the first part, you can read it here: What Will VAAI v2 Do for You? Part 1 of 2: Block
Part 1 Review
My earlier posting covered the five features for block storage available in VAAI v2:
- Hardware-Assisted Copy
- Hardware-Assisted Zero
- Hardware-Assisted Locking
- Thin Provisioning Stun
- Thin Provisioning Block Reclamation
VAAI v2 for NFS Storage
Probably the biggest and most-anticipated aspect of VAAI v2 was the the addition of advanced features for NFS datastores. VAAI v2 includes three features for NFS:
- NFS File Cloning
- NFS Extended Stats
- NFS Reserve Space
I’ll cover each of them in detail below.
NFS File Cloning
NFS File Cloning is more or less the NFS analog of the Block VAAI feature Hardware-Assisted Copy — with one fairly significant difference.
Without VAAI, an ESXi server replicating a VM or performing a Storage vMotion operation acts the same regardless of whether it’s using VMFS or NFS: the ESXi server reads the data from storage, then writes it back to storage — essentially making the server into a “middle man” for the operation, using server resources (CPU, RAM, bandwidth) and affecting other storage traffic.
Similar to VMFS’s Hardware Assisted Copy, NFS File Cloning eliminates this middle man, however, NFS File Cloning takes things one step further.
To block storage, blocks are just blocks. It’s the ESXi server, not the array, that’s making the distinction between different blocks, deciding the placement of the files within the VMFS filesystem.
NFS storage, on the other hand, is “file-aware” if you’ll indulge my creating that phrase. This allows an NFS storage array — not just the server — to take actions at the file level within the file system.
So, when you have a vSphere 5 ESXi server with NFS File Cloning enabaled (and VMware enables the feature by default) connected to an NFS datastore on a supported array that also has the feature enabled, copy operations can happen very quickly.
NFS File Cloning allows the ESXi server to call the array’s built-in file replication facility. On some arrays this might simply create a copy of the file without having to send the data off the array. On others, this will call the array’s native snapshot solution.
On the EMC VNX (the NFS array I’m most familiar with), an NFS File Cloning call will cause the VNX to create a writeable snapshot of the VMDK file that the ESXi server wanted copied. From the server’s point of view, a complete copy of the file was created nearly instantaneously — and the only space it takes up will be any blocks within the file that are changed to be different than the original file they were copied from.
All in all, NFS File Cloning provides huge performance benefits when creating VMs from templates, and can bring with it a great deal of space efficiency.
NFS Extended Stats
Prior to vSphere 5, ESXi servers weren’t able to get as much information about what was going on deep inside an NFS filesystem — certainly far less information then they could get from a VMFS filesystem. This makes a lot of sense if you think about it — VMware created VMFS — of course they know what’s going on inside it.
What this means is that, without VAAI v2, the ESXi server only gets the most basic information about a file from storage. If a file is listed as a 100GB file, ESXi treats it like a 100GB file. All space calculations, capacity planning, anything else vSphere or any of its tools might use file information for — all will treat that file as 100GB.
Which would be exactly what you’d want it to do — if it were, in fact, taking up 100GB of space on the NFS datastore…
In reality, our hypothetical 100GB file might:
- Be, in actuality, a sparse file
- Be currently using only 24GB of its allotted 100GB of space
- Have been deduplicated by the storage system
- Have been compressed by the storage system
It is, in fact, entirely possible on modern storage devices, that this “100GB” file is, in reality, only occupying 5GB of physical space on the NFS datastore.
Without VAAI v2, vSphere will treat that 5GB as if it were 100GB, effectively wasting 95GB of potentially-usable space.
With NFS Extended Stats enabled, the ESXi server gets full insight into the inner workings of the NFS filesystem. It will know to tell the application that has up to 100GB to use, and to tell capacity planning tools that the file is 5GB.
NFS datastores, on VAAI-supported NFS arrays can become incredibly space-efficient.
NFS Reserve Space
Prior to vSphere 5, there was no way for anyone using an NFS datastore to create the NFS equivalent of an “eager-zeroed think” VMDK file.
Some of you may be asking, “so what?”
Eager-zeroed think VMDKs tend to be the go-to for VMs with high I/O performance requirements, but this just wasn’t an option for NFS.
Additionally, Microsoft Windows Server Failover Clustering (WSFC) requires the physical reservation of the entire space. So, in vSphere 4.1, WSFC was incompatible with NFS datastores.
In vSphere 5, with NFS Reserve Space enabled, when using a supported NFS array, vSphere now has the option to reserve the entire space for the VMDK files.
It’s all about VMware giving customers more options.
I hope you’ve found my write-up of the VAAI v2 features helpful. Hopefully it’s explained the tangible benefits of the various features.
If you’d like the “short version” of this information, about a week ago I posted the presentation that I gave in the EMC booth at VMworld 2011 in Copenhagen. It’s available in two versions: straight PowerPower file (with speaker notes) or a recording of the presentation (complete with the melodious tones of my voice narrating).
Me? I’m trying to second-guess what new features might be coming in VAAI v3…
A couple of things come to mind.
Firstly can you configure if the NFS Full Copy VAAI primitive creates a snapshot or a ‘full fat’ copy? I’m a Netapp user and I know that running off a snapshot long term has it’s downsides.
It’s also worth pointing on the Windows Failover clusters still only support FC so even though you can eager-thick provision NFS it doesn’t mean they’re feasible on NFS.
Maybe I’ve missed something obvious, but I’m unable to find anything in the vSphere documentation that mentions configuring how the NFS Full Copy primitive handles the copy. Since the NFS VAAI features require the installation of vendor-specific plug-ins to work, my best guess is that the answer to your question will depend on the particular vendor’s implementation, but I’ll continue digging and see what else I can turn up.
I’d been unaware of WSFC’s FC-only limitation — thanks for pointing it out. I’ll need to find a better use case example for the Reserve Space feature.
My guess is that on NetApp WAFL would perform a file level FlexClone and certainly not a full copy.
You’re partially correct.
First, VMware often refers to this feature as NFS File Cloning, rather than NFS Full Copy to make it clear what the intent is, and I should probably adopt similar naming here to avoid confusion. (Never mind, “probably”. I’ll go through and edit the post to clarify as soon as I post this comment.)
Yes, the intent is that the underlying storage would perform the file-level snapshot rather than create a full copy.
However, as of the time of this writing (I just checked), if you look in the VMware Compatibility Guides, NetApp is not listed as currently supporting the NFS File Cloning feature. I’m sure their support for it will be coming soon, but today VMware with NetApp would not do that FlexClone in this case.
(I leave searching the Compatibility Guides for all vendors that currently support any of the NFS VAAI features as an exercise for the reader, but I’ll give you a hint: at the moment it’s only one and it isn’t NetApp.)
Actually, NetApp has been doing this for quite sometime now :). They released a vCenter plugin called Virtual Storage Console. It provides many features that these “new” VAAI NFS constructs now provide. For example, I’ve been able to get visibility into both my VMFS and NFS datastores with this plug-in. I can see deduplication savings, snapshot space, etc.
This plug-in also has a feature that allows me to create space efficient clones of my virtual machines and automatically import them into a VDI broker. The really cool thing is that if I’m using NFS datastores this plug-in will use NetApp’s native File Level FlexClone to efficiently create the virtual machines.
So, NetApp does do these features, including File Level Flexclone today – you just need the NetApp Virtual Storage Console plug-in for vCenter Server – which is free. The new VAAI constructs for NFS will just make it native to ESXi.
Several storage vendors have vCenter plugins now. I think it’s great that VMware made vCenter extensible like that and awesome that vendors are taking advantage of it. Giving the VMware admin great visibility into storage — and the ability to provision and manage storage — from within vCenter is huge.
The Rapid Cloning functionality and auto-registering with your VDI broker isn’t exclusive to NetApp’s VSC — EMC’s Virtual Storage Integrator’s Fast Clone capability allows this, too.
Where you’re mistaken is saying that this is “the same” as support for native NFS VAAI File Cloning. It isn’t. Doing it via the plugin “goes around” ESXi. Don’t mis-read me — I’m not saying having the features in the plugin is a Bad Thing. Providing more options for customers and administrators is pretty much always better.
So, while NetApp may provide an alternate way to do things similar to what the NFS VAAI features do, no, NetApp does not (as of today) support the NFS VAAI feature set. Check the VMware Compatibility Guides. (Just you and me talking, I’m surprised NetApp A. didn’t have Day One support for the feature-set and B. hasn’t already added support… Given NetApp’s NFS history and strength, I’d just assumed they’d have it by now.)
Pingback: Netapp and vSphere5 - NFS is still a second class citizen | www.vExperienced.co.uk
I hear you, just think you may have misunderstood me. I realize the NFS VAAI feature set is not the ame thing as the NetApp plug-in. What I was trying to say is that the result is the same thing. Both will offload the cloning capability to the storage array to alleviate the vmkernel from having to provide that functionality.
The NFS VAAI capability has vmkernel pass the task off, while the NetApp plug-in handles it from the start – in the end it’s the same outcome.
Now, it’s great that VMware now includes this functionality, among other new cool VAAI constructs, but most storage arrays that support VAAI already have vCenter plug-ins that handle a lot of these tasks. I’m just stating that if you own a NetApp (or EMC, Dell, etc) you’ve likely installed the vCenter plug-in for your array and had the functionality that the NFS VAAI feature provides.
The NetApp VSC already provides NFS file cloning (File Level Flexclone) and NFS extended stats (VSC Monitoring). The only thing that NFS VAAI will provide that the VSC doesn’t is the NFS Reserve Space.
I can’t tell if you’re dissembling or if the distinction’s too subtle, but I’m going to take a stab at it.
Yes, the VSC provides file cloning and NFS extended stats. No, this does not give you the same thing as native NFS VAAI support of those features. Native NFS VAAI support is way better.
The VSC will only use file-level FlexClone for those VMDKs the user explicitly tells it to. Native NFS VAAI File Cloning would automatically call file-level FlexClone for any file replication. Obviously better.
The VSC will let you get the same information that native VAAI NFS Extended Stats, but only when the user explicitly goes looking for them. Native VAAI NFS Extended Stats would automatically provide that information to any program that queries vCenter and/or ESXi. Again, obviously better.
It’s great that the plug-in helps to provide extra functionality, but it isn’t, and won’t ever be, an actual replacement for the native support of the advanced storage features directly from ESXi.
LOL. I don’t think I was trying to be difficult, but I guess you just want the last word on it :-). I’m not arguing that vendor plug-ins are obviously better than the NFS VAAI constructs. I simply pointed out that these plug-ins such as the NetApp VSC obviously gave this functionality for quite sometime now. I did say that the end result of both VAAI and VSC for tasks such as vm cloning, and NFS stats is the same – in both instances the task is offloaded to the storage array, freeing up valuable cpu and network resources from the vmkernel.
I could query the NetApp from any program and receive NFS extended stats too, so that’s besides the point. I disagree that the plug-ins will never be a replacement for the native support of advanced storage features directly from ESXi. Not only do these plug-ins like the VSC supply the same functionality as the NFS VAAI constructs, but they do a lot more. Plus, the plug-ins are from the actual storage vendors themselves! I think I’d trust that the storage vendor would be able to provide me with the best integration against their own storage. Last I looked I don’t think I can right-click a datastore and initiate deduplication – as a baked in advanced storage feature directly from ESXi. As new storage features are introduced these plug-ins will always be ahead of what VMware can natively supply. VMware does a great job playing the catch up game, but they do what they do best and storage vendors do what storage vendors do best.
The Cisco 1000V is a superior product to the vSphere VDS is it not? A Checkpoint or Palo Alto virtual security appliance is going to be more robust than what vShield can offer. Vendor partners will always be able to offer better products than what vSphere can natively offer. So, I’ll agree that there may be a little more flexibility in how I’ll be able to do some things with NFS VAAI, but I can still easily do all those things and get better native storage integration with these storage plug-ins.
I wasn’t trying to get the last word (I don’t suffer from a desire for that greater than, oh, you know, everyone’s…), I just wasn’t interested in a circular discussion.
Now, however, you’ve raised a really interesting point that I hadn’t thought of, one that’s worth discussing and speculating on.
You’re right — VMware’s use of advanced storage features will always lag behind the storage vendors’ development of those new features — and storage vendors’ use of advanced VM features and VM management features will always lag behind VMware’s development of those new features.
On the one hand, providing multiple ways to accomplish the same thing benefits the customers by giving them more choice. On the other hand, too many choices can get confusing.
What can we (you, me, VMware, the storage vendors (and network and security vendors)) do to reduce and minimize that feature-lag (in all directions)? That would be an even better benefit to everyone’s customers.
NetApp is supporting VAAI for NFS (including file-level (flex)-cloning) since ONTAP 8.1 in cMode.
You’re correct. At the time I wrote the original post, only EMC VNX provided support for VAAI-NFS.
As of the time of this comment, it’s now EMC VNX, EMC VNXe, and NetApp FAS running Data ONTAP 8.1 in Cluster-Mode.
Meanwhile NetApp ONTAP 8.1.1 in 7Mode is supporting VAAI for NFS too. It uses single file cloning which is basically the reverse of dedupe. It is not only fast, but also safes tons of space while it is scaling for very large deployments.
Not according to the published VMware Compatibility Guide.
As of the time I’m typing this comment, VAAI-NAS is supported on NetApp only on ONTAP 8.1 in Cluster-Mode, not in 7-Mode.
One thing this latest discussion doesn’t touch on us the benefit of VAAI when scripting. The VSC is fine if you want to use a GUI but won’t offer any functionality to API requests. We provision most of our VMs via some in house PowerCLI scripts so VAAI would definitely be an improvement. Likewise anyone who uses Orchestrator, vCD or other third party apps etc. I like the VSC, but I don’t considerate it on a par with VAAI. It does of course offer features that VAAI doesn’t, so it’s horses for courses.
I fully agree with you. VAAI for NFS support is here on NetApp. You can use ONTAP 8.1.1 to take advantage of it.
Beside of it, you could use NetApp Powershell cmdlets (over 1000 cmdlets available today) to get the job done directly on the controller. Also, VSC is having an API if you prefer that. You’ll find the details in the documentation.
Many storage vendors provide libraries of PowerShell commandlets for use with their arrays, so NetApp’s not alone there.
I think Ed’s point is more that if you have the VAAI support, your scripts would automagically get the benefits of VAAI without having to use vendor-specific commands, commandlets, or scripts.
Thanks for pointing out the VSC API, I hadn’t really thought of using that and it might be a useful alternative. As I write this ONTAP 8.1.1 isn’t out yet although if Chuck Hollis is to be believed that might not be the case for long;
I’ve previously written an indepth post about the VAAI requirements for Netapp;
In fact 8.1.1 is available for download as an RC an will very soon be GA. I have customers who run it successfully today. The VAAI NFS Plugin can very easily deployed by VSC 4.0.
You are referring to Chuck’s blog – well I really propose that people start to focus on their technology only, rather than trying to be an expert for everyone else products, especially if they are never playing around in the lab by themselfes. His recent blog post is full of flaws and wrong assumptions. This got to an level where I as an engineer stop paying attention to it.
Pingback: Survey on Storage Integration with VMware | GeekFluent
Pingback: August’s EMC Solutions for VMware Webcasts | GeekFluent
Pingback: Isilon Integration with VAAI-NAS and VASA | GeekFluent
Pingback: Enhancement to VAAI-NAS in vSphere 5.1 | GeekFluent
Pingback: Most-Read GeekFluent Posts of 2012 | GeekFluent
Pingback: Most-Read GeekFluent Posts of 2013 | GeekFluent
Pingback: EMC/VMWare VAAI (VStoarge API for Array Integration) | Simplify Complexities
Pingback: State of the Blog Report: 2016 | GeekFluent