ZFS and streaming

N

novadude

[H]ard|Gawd
Joined
Jul 8, 2004
Messages
1,108
I've been planning on migrating my storage into the new Norco 24-bay case since I heard about it, so there is a whole new build in the planning stages.

Anyways, I'm looking for opinions from people with hands on experience with ZFS raidz2 in a media serving role. I'm planning on serving up multiple streams at once, bluray and HD tv recordings from a fairly robust system (Xeon L3426, 8GB ram, SM X8SIA-F, USAS-L8i, 8x 2tb Hitachi's per vdev, etc) but I've heard that streaming video isn't ZFS's strong suit and it can cause a bit of stuttering in the playback. I'll be hitting this server with hd recordings too while streaming, so I'd like to know if anyone has a similar setup. OS will probably be freeBSD and sharing over samba to 2-3 htpc's, one with a ceton cablecard tuner.
 
Code: 
pool: storagepool
 state: ONLINE
 scrub: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        storagepool  ONLINE       0     0     0
          raidz2     ONLINE       0     0     0
            c7t39d0  ONLINE       0     0     0
            c7t40d0  ONLINE       0     0     0
            c7t41d0  ONLINE       0     0     0
            c7t42d0  ONLINE       0     0     0
            c7t43d0  ONLINE       0     0     0
            c7t44d0  ONLINE       0     0     0
            c7t45d0  ONLINE       0     0     0
          raidz2     ONLINE       0     0     0
            c7t46d0  ONLINE       0     0     0
            c7t47d0  ONLINE       0     0     0
            c7t48d0  ONLINE       0     0     0
            c7t49d0  ONLINE       0     0     0
            c7t50d0  ONLINE       0     0     0
            c7t51d0  ONLINE       0     0     0
            c7t52d0  ONLINE       0     0     0
          raidz2     ONLINE       0     0     0
            c7t53d0  ONLINE       0     0     0
            c7t54d0  ONLINE       0     0     0
            c7t55d0  ONLINE       0     0     0
            c7t56d0  ONLINE       0     0     0
            c7t57d0  ONLINE       0     0     0
            c7t58d0  ONLINE       0     0     0
            c7t59d0  ONLINE       0     0     0

errors: No known data errors
Tau@Skynet:~#

Your network will be the bottleneck. I stream off this box to 2 HTPCs, 3 client terminals, and soon to be backup for my vmware cluster (once i get some more NIC bandwidth in the box)

Local speed test to same array;

Code: 
10GB Read

IO Summary: 47659 ops, 786.4 ops/s, (786/0 r/w) 786.4mb/s, 3957us cpu/op, 1.3ms latency

20GB Write

IO Summary: 46299 ops, 522.1 ops/s, (0/522 r/w) 522.0mb/s, 6113us cpu/op, 1.8ms latency

Single 7 disk array was still fast enough to saturate a gigE line
 
Yes, but the problem may be that ZFS is blocking reads when it is writing its journal. Not sure about that, i but i could imagine ZFS does not handle multiple streams including write streams that effectively, letting all disks seek and letting some reads have high latencies. This might not be preferred in a media stream situation.

Perhaps you should do some tests with multi-stream benchmarks or a simulated environment. I assume you would want to know the solution works properly before putting it into production.

Will the server see mixed read/write streams or only read then write; i.e. not at the same time?
 
Thanks for the reply. I'd appreciate more details on what quality video you're streaming at once and your OS, and shared via samba/cifs/nfs etc.

sub.mesa: it will see simultaneous read/writes. As for testing, I'm still planning at the moment but if I don't get the sense that this will probably work I'll be sticking with hardware raid.
 
novadude said:
Thanks for the reply. I'd appreciate more details on what quality video you're streaming at once and your OS, and shared via samba/cifs/nfs etc.

sub.mesa: it will see simultaneous read/writes. As for testing, I'm still planning at the moment but if I don't get the sense that this will probably work I'll be sticking with hardware raid.
Click to expand...

currently sharing to windows boxes with CIFS, streaming anything from 700MB dvd rips, FLAC music files, up to uncompressed 1080P bluerays (40GB files)

Just did a multistream read test 4x 10Gb files, the box is also currently streaming a blueray to an HTPC so call this a 5 stream test.

Code: 
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread4                  290ops/s 290.2mb/s      3.4ms/op     1505us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread3                  212ops/s 212.3mb/s      4.7ms/op     1549us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread2                  111ops/s 111.3mb/s      8.9ms/op     1622us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread1                  100ops/s 100.2mb/s      9.8ms/op     1569us/op-cpu

 1644: 223.647:
IO Summary:      43277 ops, 714.1 ops/s, (714/0 r/w) 714.0mb/s,   4586us cpu/op,   5.5ms latency
 1644: 223.647: Shutting down processes
 
its MB, not sure why the tool reorts it as mb/s to clairify (MegaBYTES)
 
Tau said:
currently sharing to windows boxes with CIFS, streaming anything from 700MB dvd rips, FLAC music files, up to uncompressed 1080P bluerays (40GB files)

Just did a multistream read test 4x 10Gb files, the box is also currently streaming a blueray to an HTPC so call this a 5 stream test.

Code: 
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread4                  290ops/s 290.2mb/s      3.4ms/op     1505us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread3                  212ops/s 212.3mb/s      4.7ms/op     1549us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread2                  111ops/s 111.3mb/s      8.9ms/op     1622us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqread1                  100ops/s 100.2mb/s      9.8ms/op     1569us/op-cpu

 1644: 223.647:
IO Summary:      43277 ops, 714.1 ops/s, (714/0 r/w) 714.0mb/s,   4586us cpu/op,   5.5ms latency
 1644: 223.647: Shutting down processes
Click to expand...

Sorry to ask for yet another test, but if you have time could you toss a write into the above somewhere? Thanks again, regardless.
 
Code: 
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqwrite4                 124ops/s 123.5mb/s      7.6ms/op     1894us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqwrite3                 131ops/s 130.8mb/s      7.1ms/op     1872us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqwrite2                 129ops/s 128.9mb/s      7.2ms/op     1880us/op-cpu
limit                       0ops/s   0.0mb/s      0.0ms/op        0us/op-cpu
seqwrite1                 129ops/s 129.1mb/s      7.1ms/op     1906us/op-cpu

 1718: 259.057:
IO Summary:      39511 ops, 512.3 ops/s, (0/512 r/w) 512.3mb/s,   7163us cpu/op,   7.2ms latency
 1718: 259.057: Shutting down processes

4 instance write of 4 10GB files.
 
Now I feel like even more of an ass than normal, but I was hoping for something with simultaneous reads and writes. Perhaps the write test while watching a bluray?
 
Bumping this because I know there are a number of people building/testing ZFS based media servers at the moment. Looking for testimony on how well zfs handles simultaneous reads and writes.
 
Gonna give this a bump also.
I'm also having a hard time finding more information on this. Any good links? Thanks.
 
Bumping this to give my experience so far. Playing an MKV while loading data onto the array (loading at ~110MB/s sustained while not playing) is no problem. Clicking 30 minutes ahead on the seek bar takes a few seconds, but playback is smooth.
Blurays had a little hitch once in a while when the bitrate would jump, but that is probably network congestion because I was playing them on the computer I'm loading data onto the server from. I'll have to setup a transfer from a laptop at around 60-80MB/s to see how blurays play back with headroom on the network connection.
 
You must log in or register to reply here.
Back
Top