PCIE Bifurcation

XGX13XGX

n00b
Joined
Apr 7, 2021
Messages
2
NEED HELP ASAP LOL; So i am a GPU Miner and need some help. the new 3060 gpus req at least x8 lanes, i hace a Dell T5610 server with dual xeon processors, the processors can run 80 lanes, the mother board has 4 x16 PCIe slots. What i want to do is take a x16 to x16 ribbon cable attach a x16 to 2 x8 riser then use 2 x8 to x16 to connect the gpus to the riser card enabling me to have 8 of the 3060s running on the x16 slots and then run 4 more NON 3060 cards off a x1 to x4 riser card. problem is i can not find a riser capable of doing such. I went online and found what i thought would work but it turns out it was a U2 TO 2 X8 NOT A X16 TO 2 X8 any one have an idea how i can attach a U2 to a x16 slot on the motherboard OR where i can find a x16 to dual x8 slot riser card? ive been looking and looking and cant find anything available
 
Joined
May 21, 2012
Messages
2
OK.. got my low profile x8 -> x4 + M.2 Riser ready and tested.
PCIe working as expected.. polarity inversion on clock and data as well as lane reversal.
Also my M.2 socket footprint seems to be working perfectly.

Only thing I didnt get right is the height, so one cannot directly use the low profile bracket to screw into the full height slot as intended.
(I took the specified card height for low profile cards and not the slot height for my calculations which is obviously stupid:banghead:)

Take a look. As always I have spare PCB's to sell should anyone be interested.


View attachment 48763 View attachment 48764 View attachment 48761 View attachment 48766
View attachment 48765 View attachment 48762
Are these risers available? (with correct height :))
I've got a x4 10Gbps NIC (it's a half-height card) in the second PCI-E x16 slot, and I want to add another NVME SSD to my system, and would like it to have full bandwidth.
It's a X470 K4 Gaming board with x8x4x4 and x4x4x4x4 bifurcation options in the BIOS.
 
Last edited:

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
165
Hi Guys! sorry for being unresponsive here. That always happens to me because the forum just sends one eMail for the first new post and I tend to miss it.

I have moved my store to a new location: c-payne.com

The Coolest: PCIe Bifurcation Card - x8x4xM.2 - low Profile Adapter - Ncase M1 and others
XGX13XGX: PCIe Bifurcation Card - x8x8 - 3W
inf3rno:
https://www.ruten.com.tw/item/show?22030929704581 : These are known to be non functional (check like 10-15 pages back, not sure exactly where)
https://www.asrockind.com/en-gb/PCIE-Riser-2S : This one can only work in some specific mainboards that have 2 clock outputs. Its not suitable for the normal standard PCIe Slot Pinout.
The IOI brand is decent.

3. If the motherboard has two x16 slots and supports for example x16 for the first and x8 for the second, then can I do 2x8 bifurcation for the first slot and use 3 cards with x8 connectors? Or should the motherboard manual explicitly write if this is possible?:
That will NOT work on the mainstream platforms (Ryzen, Intel Mainstream) The CPU only has 16 Lanes. So if you insert a card in the second slot, the first slot will go into x8 mode.
Its a different story on threadripper/EPYC or intel hedt/XEON

5. Is using PCIE x16 -> 4x M2 -> PCIE 4x4 a legit way to split up an x16 port or better to use x16 -> 4x4 bifurcation risers? Is it possible to split up and x16 slot with a PCIE -> M2 adapter even if the motherboard does not support bifurcation for that slot?
There is PCIe-> M.2 adapters with a Packet switch on board, but thats not the Asus one, thats just plain bifurcation and would not work if your mainboard does not support bifurcation.
They cheapest ones are around 150 bucks on aliExpress featuring an ASM2824 chip.
If going the bifurcation way: every added interconnect will add signal loss and you may end up with degraded perfowmance or a lot of AER reports/crashes.

One note about the riser.maxcloudon.com cards: Their claim to be gen4 is to be taken with a large spoon of salt. It is most likely not the case.
 

XGX13XGX

n00b
Joined
Apr 7, 2021
Messages
2
Hi Guys! sorry for being unresponsive here. That always happens to me because the forum just sends one eMail for the first new post and I tend to miss it.

I have moved my store to a new location: c-payne.com

The Coolest: PCIe Bifurcation Card - x8x4xM.2 - low Profile Adapter - Ncase M1 and others
XGX13XGX: PCIe Bifurcation Card - x8x8 - 3W
inf3rno:
https://www.ruten.com.tw/item/show?22030929704581 : These are known to be non functional (check like 10-15 pages back, not sure exactly where)
https://www.asrockind.com/en-gb/PCIE-Riser-2S : This one can only work in some specific mainboards that have 2 clock outputs. Its not suitable for the normal standard PCIe Slot Pinout.
The IOI brand is decent.

3. If the motherboard has two x16 slots and supports for example x16 for the first and x8 for the second, then can I do 2x8 bifurcation for the first slot and use 3 cards with x8 connectors? Or should the motherboard manual explicitly write if this is possible?:
That will NOT work on the mainstream platforms (Ryzen, Intel Mainstream) The CPU only has 16 Lanes. So if you insert a card in the second slot, the first slot will go into x8 mode.
Its a different story on threadripper/EPYC or intel hedt/XEON

5. Is using PCIE x16 -> 4x M2 -> PCIE 4x4 a legit way to split up an x16 port or better to use x16 -> 4x4 bifurcation risers? Is it possible to split up and x16 slot with a PCIE -> M2 adapter even if the motherboard does not support bifurcation for that slot?
There is PCIe-> M.2 adapters with a Packet switch on board, but thats not the Asus one, thats just plain bifurcation and would not work if your mainboard does not support bifurcation.
They cheapest ones are around 150 bucks on aliExpress featuring an ASM2824 chip.
If going the bifurcation way: every added interconnect will add signal loss and you may end up with degraded perfowmance or a lot of AER reports/crashes.

One note about the riser.maxcloudon.com cards: Their claim to be gen4 is to be taken with a large spoon of salt. It is most likely not the case.
i think we were speaking on ebay about the boards, and we came to the conclusion that these would not work cause the server i am using does not support bifrucation. but with that what about the plx switch chips? essentially i am gpu mining and the new cards have mining limiters on them and they REQUIRE X8 OR X16 and when using a x8 you have to have a x8 to x16 adapter to run them. the motherboard i am using can run 80 lanes with the 2 cpus, it has 4 x16 slots and 1 x16 slot wired as a x4 what i need is a splitter with a plx chip the can split the x16 in 2 or 3 but still run at either x8 or 16
 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
165
with a Packet switch (PLX) you can use any uplink, even x1. The mining limiter wont interfere as long as the Downstream GPU Link is x8.

I have some PLX boards in manufacturing currently.
 
Joined
May 21, 2012
Messages
2
C_Payne, I'm interested in a x8 -> x4x4 card.
I have a GPU in the first x16 slot, and want to have both my 10G NIC (x4) and another NVME SSD in the second x16 slot.
I'm just wondering, if such a card is available, and how much it would be.
 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
165
The Coolest

my suggestion would be a slimline based solution:
host adapter:
https://c-payne.com/products/pcie-slimsas-host-adapter-x16-to-2-8i
-or-
https://c-payne.com/products/pcie-slimsas-host-adapter-x16-to-2-8i-90deg

cable:
https://www.arrow.com/en/products/8es8-1df21-0.50/3m

device adapter:
https://c-payne.com/products/slimsas-pcie-device-adapter-8i-to-x4x4

a cheaper solution would be to use this adapter and a ribbon style cable:
https://c-payne.com/products/pcie-bifurcation-card-x4x4-3w
but i think it wont fit nicely into any build.
 

doggfather

n00b
Joined
May 9, 2021
Messages
1
Hello C_Payne - I would like to use this board:

https://www.asrockrack.com/general/productdetail.asp?Model=X570D4I-2T#Specifications

in a Sliger SM560. However it is PCIe4 - Do you have a PCIE4 X8,X8 Bifurcator that would fit this with a spacing seen here: (1 -- 2 -- FREE FOR DUAL SPACE GPU) - so basically just 1,2 ;0

https://www.sliger.com/products/cases/sm560/

I need the speed for various reasons and cant drop to 3.0 - can you build me something ? Happy to pay as i may need a few of them.... ;0 I could go commercial even so i can get you some volume. Can you IM me ?

I would also be interest in a spacing of (1 - SPACE - 2 - SPACE) I would like to use this MOBO on perhaps a larger case with 4 slots to get 2 GPU's in as well...
 
Last edited:

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
165
Hello C_Payne - I would like to use this board:

https://www.asrockrack.com/general/productdetail.asp?Model=X570D4I-2T#Specifications

in a Sliger SM560. However it is PCIe4 - Do you have a PCIE4 X8,X8 Bifurcator that would fit this with a spacing seen here: (1 -- 2 -- FREE FOR DUAL SPACE GPU) - so basically just 1,2 ;0

https://www.sliger.com/products/cases/sm560/

I need the speed for various reasons and cant drop to 3.0 - can you build me something ? Happy to pay as i may need a few of them.... ;0 I could go commercial even so i can get you some volume. Can you IM me ?

I would also be interest in a spacing of (1 - SPACE - 2 - SPACE) I would like to use this MOBO on perhaps a larger case with 4 slots to get 2 GPU's in as well...
Please drop me an email: info@c-payne.com
 

inaxeon

n00b
Joined
May 28, 2021
Messages
12
Hey folks. Inspired by C_Payne's elegant designs, I've decided to have a crack at designing my own riser.

This one is for the Casetronic T9120 to replace the manufacturer x16 riser. I will be making it available to others if it succeeds. I have a few other casetronic risers I want to do too.

It's an x8x4x4 with an x8 in place of the original slot, M.2 on the rear and an awkward mezzanine board to allow another x4 PCIe card to be installed in the left hand 3.5" drive bay. This will be connected with a short piece of this new "Germany" (whatever it is) twinax PCIe ribbon cable I'm seeing on GPU extensions.

It'll probably be a couple of months till I have time to physically build it.
 

Attachments

  • main.png
    main.png
    1.4 MB · Views: 0
  • mezz.png
    mezz.png
    80.4 KB · Views: 0

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
165
I see. Need to add some termination resistors though with that kind of outputs, or am I missing something?
 

inaxeon

n00b
Joined
May 28, 2021
Messages
12
I see. Need to add some termination resistors though with that kind of outputs, or am I missing something?
Probably not. I haven't finalised the design yet. I was also wondering about AC coupling them. More research needed.
 

inaxeon

n00b
Joined
May 28, 2021
Messages
12
Ac-coupling I would say optional, but proper termination is necessary.
You are probably right source termination with no AC coupling is fine. I have the gear to eye diagram the clock so not too worried about it.

It's the data pairs that I'm more concerned about. I don't have the kit to eye those. Just going to have to get it right, or beg to borrow it.
 

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
165
You are probably right source termination with no AC coupling is fine. I have the gear to eye diagram the clock so not too worried about it.

It's the data pairs that I'm more concerned about. I don't have the kit to eye those. Just going to have to get it right, or beg to borrow it.
Jup, best go full OCD about those, but by taking a peek at your traces you seem to know your trade.
 

inaxeon

n00b
Joined
May 28, 2021
Messages
12
Jup, best go full OCD about those, but by taking a peek at your traces you seem to know your trade.
There we go. Source termination to the rescue. Note to future self: Finish reading the datasheet for the chips on the board before layout.
 

Attachments

  • st.png
    st.png
    88.1 KB · Views: 0

inaxeon

n00b
Joined
May 28, 2021
Messages
12
Just hammered out another riser. This one is for Travla/Casetronic models C146, C159 and T1160. It's an x8 to x4x4 (that is what I need at the present time). It's is built in the form of the original Conventional PCI riser connected by ribbon cables as this was a better choice practically and for signal integrity.

I managed to get away with 2 layers for both. I'm going to build this one first because of the lower cost of the boards.
 

Attachments

  • 1u1.png
    1u1.png
    1.9 MB · Views: 0
  • 1u2.png
    1u2.png
    113.6 KB · Views: 0

inaxeon

n00b
Joined
May 28, 2021
Messages
12
The cable I'm using is designed for PCIe and is said to have the correct impedance, although I've not found a detailed spec for it. The PCB is the potential issue, but I checked in the calculator and bumping from 0.2mm of dielectric to 1.4mm doesn't make a lot of difference. It's still within spec. All of my pairs are backed by ground plane.

It's not an ideal arrangement, but it's possible to construct and for the low cost it's worth a shot.

The manufacturer x16 PCIe riser is also 2 layer, using standard 2.0mm rigid headers to bond the boards together which must be absolutely awful for sharp angle induced reflections and impedance mismatch. Pair routing / layout is also fairly poor. Despite that it works up to Gen 3.0 that I've tested.
 

inaxeon

n00b
Joined
May 28, 2021
Messages
12
I can see the benefit. It hard to run a 6/8/6 pair between pads on a slot unless they have almost no copper left on them. Am a little concerned about this is going to pan out in practice. We'll see in the coming months I guess.

You're the guy who does it right, I'm the guy who does it cheap!
 

marnold

n00b
Joined
Jun 7, 2021
Messages
3
Hi guys - I hoped you might be able to help me out, I am more or less at the end of my tether and about to give up and put my old 2080ti on ebay!

I've had a read through this thread (and google) and not quite found an answer...

Having recently managed to get a 3080 I thought I'd have a go at bifurcating and keeping my 2080ti, I'm using a X570 Phantom Gaming-ITX/TB3 with a 3700X, and one of c_payne's x8x8 cards

To get it to fit with my case/desk my initial intention was to go motherboard > x8x8 > normal riser cable to 3080, x1 usb riser cable to 2080ti, but with no initial luck

Since realising it wasn't quite a simple plug+play I've tried a whole set of combinations to see what would work. I know each part works individually by testing. Depending on the combination I use, I tend to get either just one card visible, or crashing somewhere from POST to windows logging in. Some combinations just don't work at all, some are intermittent in the way they break down

The best and most consistent response I can get right now is using the x8x8 then two normal riser cables (ones that came with a couple of my thermaltake cases). I can get windows to load when no drivers are installed, both cards appear in device manager, but I get a black screen and crash while installing drivers. After that windows will hang while loading until I go to safe mode and uninstall

In the BIOS (latest version) some of the relevant settings I've ended up enabling - Above 4G Decoding, IOMMU, setting PCIE to x8x8 (and I've tried everything from Gen1 to auto), disabling CSM

At this point I wonder if better quality riser cables would do it but I am reluctant to sink more money into what was an experiment! Hopefully that all makes sense and I haven't missed something really dumb...thanks!
 

inaxeon

n00b
Joined
May 28, 2021
Messages
12
Hi guys - I hoped you might be able to help me out, I am more or less at the end of my tether and about to give up and put my old 2080ti on ebay!

I've had a read through this thread (and google) and not quite found an answer...

Having recently managed to get a 3080 I thought I'd have a go at bifurcating and keeping my 2080ti, I'm using a X570 Phantom Gaming-ITX/TB3 with a 3700X, and one of c_payne's x8x8 cards

To get it to fit with my case/desk my initial intention was to go motherboard > x8x8 > normal riser cable to 3080, x1 usb riser cable to 2080ti, but with no initial luck

Since realising it wasn't quite a simple plug+play I've tried a whole set of combinations to see what would work. I know each part works individually by testing. Depending on the combination I use, I tend to get either just one card visible, or crashing somewhere from POST to windows logging in. Some combinations just don't work at all, some are intermittent in the way they break down

The best and most consistent response I can get right now is using the x8x8 then two normal riser cables (ones that came with a couple of my thermaltake cases). I can get windows to load when no drivers are installed, both cards appear in device manager, but I get a black screen and crash while installing drivers. After that windows will hang while loading until I go to safe mode and uninstall

In the BIOS (latest version) some of the relevant settings I've ended up enabling - Above 4G Decoding, IOMMU, setting PCIE to x8x8 (and I've tried everything from Gen1 to auto), disabling CSM

At this point I wonder if better quality riser cables would do it but I am reluctant to sink more money into what was an experiment! Hopefully that all makes sense and I haven't missed something really dumb...thanks!
PCIe design tolerances are typically System Board -> Properly designed riser -> Peripheral. If you put anything more in the mix you'll have too much signal loss/jitter/reflections and it's not likely to work.

Running PCIe over USB isn't supported by any of the PCI-SIG standards thus there are no comments that I could make about it. Yes it might work in some scenarios i.e. crypto mining where there are no other risers in the mix for example.
 
Last edited:

marnold

n00b
Joined
Jun 7, 2021
Messages
3
PCIe design tolerances are typically System Board -> Properly designed riser -> Peripheral. If you put anything more in the mix you'll have too much signal loss/jitter/reflections and it's not likely to work.

Running PCIe over USB isn't supported by any of the PCI-SIG standards thus there are no comments that I could make about it. Yes it might work in some scenarios i.e. crypto mining where there are no other risers in the mix for example.

Certainly I agree - I figure using the USB risers is a bit of a wild-west and have no real expectations of it being 100%

My real issue is I can't get things to work even with just the mobo > bifurcate > standard risers...
 

inaxeon

n00b
Joined
May 28, 2021
Messages
12
Just for good measure, here's another riser for C146, C159 and T1160. Swapping the bottom x4 slot for M.2. I'll build this one too and see how I get on with it.

1u3.png
 
Last edited:

C_Payne

Limp Gawd
Joined
Jan 6, 2017
Messages
165
this becomes especially problematic with gen4
Certainly I agree - I figure using the USB risers is a bit of a wild-west and have no real expectations of it being 100%

My real issue is I can't get things to work even with just the mobo > bifurcate > standard risers...
Even that is too much attenuation for PCIe gen4. (your 3080)
Try forcing link speed to gen3 in BIOS.
 

inaxeon

n00b
Joined
May 28, 2021
Messages
12
this becomes especially problematic with gen4

Even that is too much attenuation for PCIe gen4. (your 3080)
Try forcing link speed to gen3 in BIOS.
I notice you've been busy with redriver boards. Are those required on risers for Gen.4 ?

(EDIT: He did try Gen.1 so probably something else wrong here)
 

marnold

n00b
Joined
Jun 7, 2021
Messages
3
Yeah gen 4 plain just doesn't work which I fully expected - that's the case with just the one card also

But setting gen 1 to 3 doesn't have an impact here. I've tried 3 riser cables but they are still the only thing I can think to blame - unless there is a setting or something I've missed
Just seemed odd that the nvidia driver installation is what really breaks it

Maybe PSU also but I thought 750w ought to manage - especially when they are both idling
 

minority

n00b
Joined
Jun 22, 2021
Messages
1
Yeah, have custom BIOS for the Z97 based on their BETA firmware.

If you're got a Z97 and want to try it out let me know, I have a sneaking suspicion that they didn't actually do any hardware mods and just wanted me to get the board through them as I had mentioned that was what I was using the splitter for, lol. [...]
I've received the latest BIOS version from AmeriRack: Z87EITX(P2.50B).zip
They've been very helpful.
[...]
Hello! Been lurking here during lockdown, thinking of making an ITX build with the existing 1150 CPU that I have. Hoping for some bifurcation with one of C_Paynes boards (nice work :)).
Does anyone have the bioses that kudoz, chemist_slime and Phuncz mention (in 2016) for the ASRock Z87E/Z97E ITX boards, and can share them with me?
 
Top