brutalizer
[H]ard|Gawd
- Joined
- Oct 23, 2010
- Messages
- 1,602
ZFS dedupe always had some problems and were not recommended for production. For instance, it uses gobs of RAM, 1GB RAM for every 1TB disk space. This is where the misconception comes from, that ZFS requires 1GB RAM for every 1TB disk space. This is only valid if you want to use dedupe. Also, dedupe can be slow when deleting stuff.
This summer, Oracle bought Greenbyte. Greenbyte has rewritten the ZFS dedupe engine and claims best in class performance. They can dedupe with near zero latency, they can dedupe 5,000 fat VDI clones using 210TB disk space, down to 4TB disk space. So now Solaris can boot 6,000 VMs in 5-6 minutes or so. Greenbyte ZFS dedupe will be incorporated in the next upgrade of Oracle Solaris coming this year. So, no more problems with ZFS dedupe, you can use it to full extent. Also, compression helps with dedupe too. So you should enable compression (which is very fast) and Greenbyte dedupe is also very fast, for added benefit. I think I shall dedupe and compress all my data because it is so fast. In fact, compression will increase performance because it is faster to load 1000 bytes and decompress it into 2000 bytes, than it is to read 2000 bytes.
http://www.theregister.co.uk/2013/08/27/greenbytes_latency_smash_with_flash_cache/
http://www.theregister.co.uk/2012/10/12/greenbytes_chairman/
Also, current Oracle Solaris v11.2 resilvers at full platter speed. Before, if you had fragmented zfs pool and resilvered a disk, it could take a long time, especially if it were fragmented. Now ZFS instead examines all data to be resilvered and sorts and logs them. Finally, after sorting and logging, resilvering takes place at full disk speed, i.e. 100-150MB/sec.
http://milek.blogspot.se/2014/12/zfs-raid-z-resilvering.html
Solaris 11.2 also has persistent L2ARC, i.e. if you reboot the contents of the L2ARC will be saved and reloaded, so you dont have to wait until the L2ARC is fully populated.
This summer, Oracle bought Greenbyte. Greenbyte has rewritten the ZFS dedupe engine and claims best in class performance. They can dedupe with near zero latency, they can dedupe 5,000 fat VDI clones using 210TB disk space, down to 4TB disk space. So now Solaris can boot 6,000 VMs in 5-6 minutes or so. Greenbyte ZFS dedupe will be incorporated in the next upgrade of Oracle Solaris coming this year. So, no more problems with ZFS dedupe, you can use it to full extent. Also, compression helps with dedupe too. So you should enable compression (which is very fast) and Greenbyte dedupe is also very fast, for added benefit. I think I shall dedupe and compress all my data because it is so fast. In fact, compression will increase performance because it is faster to load 1000 bytes and decompress it into 2000 bytes, than it is to read 2000 bytes.
http://www.theregister.co.uk/2013/08/27/greenbytes_latency_smash_with_flash_cache/
http://www.theregister.co.uk/2012/10/12/greenbytes_chairman/
Also, current Oracle Solaris v11.2 resilvers at full platter speed. Before, if you had fragmented zfs pool and resilvered a disk, it could take a long time, especially if it were fragmented. Now ZFS instead examines all data to be resilvered and sorts and logs them. Finally, after sorting and logging, resilvering takes place at full disk speed, i.e. 100-150MB/sec.
http://milek.blogspot.se/2014/12/zfs-raid-z-resilvering.html
Solaris 11.2 also has persistent L2ARC, i.e. if you reboot the contents of the L2ARC will be saved and reloaded, so you dont have to wait until the L2ARC is fully populated.