Dell IDrac wizards needed

[That_Sound_Guy]

I'm at hairpulling wits end on trying to figure out an issue with specific Dell IDRACS

Ive installed dozens of these before and not had this issue.
Issue: I cannot get a Dell Idrac on a R730 or a R740 to complete a handshake exchange with Cisco switches. Layer 1 will connect, but not layer 2 - no MAC or other l2 logic.

Ive tried from server:
- reset idrac to defaults
- reset bios to defaults
- verified pins in idrac are aligned in dedicated physical port
- Upgraded idrac firmware to latest
- Upgraded BIOS to latest
- I cannot use the quad NIC interfaces as idrac as they are reserved for production usage so dedicated nic is only option
- IP info verified as correct
- Set speed/duplex to various

I've tried from switch
- Verified config/vlan
- Tried new port
- Tried alternate switch (stack switch complex)
- Set speed/duplex to various

Ive tried with cabling
- Tried connection from switch 1 port 1 to idrac
- Tried connection from switch 2 port 1 to idrac
- Tried new cables 2x
- Removed any patch panels and direct wired
- Pull a known working cable from a working idrac

These switches have 5 other IDracs connected without any trouble. However I magically have a R730 and R740 that both do the exact same thing. No L2 connectivity. Is there anything you can think of Im missing trying to figure this out? The hardest challenge I have is these servers are remote to me and they are not IT people, they are infrastructure. So progress is slow when trying to explain everything. I have 0 remote access to device.

I don't want t believe it but only thought remaining is both IDracs are bad but wow that is a stretch to believe when they fail the same way.

Thanks all.

 

[Dan_D]

Broadcom network controllers I assume?

 

[That_Sound_Guy]

Broadcom network controllers I assume?

I get the comment, burned by them before. The quad chassis NIC is a Intel x550 10G, no idea what the dedicated IDrac manufacturer is.

 

[Dan_D]

I get the comment, burned by them before. The quad chassis NIC is a Intel x550 10G, no idea what the dedicated IDrac manufacturer is.

I've actually seen a lot of issues with the Dell R730 and R740's with Broadcom NICs. I'm pretty quick to jump on that as I've seen issues with Cisco switches and certain SAN devices. Although, I can't recall seeing this with the iDRAC itself.

 

[That_Sound_Guy]

I've actually seen a lot of issues with the Dell R730 and R740's with Broadcom NICs. I'm pretty quick to jump on that as I've seen issues with Cisco switches and certain SAN devices. Although, I can't recall seeing this with the iDRAC itself.

I have far more HP DL series servers that I've really had Broadcom issues with. This IDRAC issue really has me messed up project wise. I have no real tools or triage abilities with such a limited interface, especially not being hands on with the server as well. Rebooting through BIOS to restart the IDRAC is also a giant time waste trying to test anything.

 

[vxspiritxv]

Two things I would do, first, "Dell EMC Repository Manager" since dell no longer builds the iso and puts them on their support page, ya have to get that program, and build an ISO for your machine. It will fully update everything in it, including NIC firmware.
2nd, if nothing improved, install open manage. Open manage can go into the idrac from an OS.

If memory serves, idracs default to 0.0.0.0, you have to set an IP and I personally have no idea if dhcp works on them. If an idiot is doing this (ie remotely for you) they are likely setting the lifecycle controller ip and not the idrac ip. At least I can see someone doing that on the 740. I don't think the 730 bios setup has that extra screen. It's been years since I had to set up one of those. Maybe think about getting a network KVM so you don't have to deal with idiots. That and a PDU with switch ports. Nothing like a cisco switch that requires a power cycle to update remotely. If all else fails, find a video on youtube that shows how to set it up and send it to the worker. Course the workers could be dicking you around to get more hours... had one guy attempt that on me once.

 

[pendragon1]

have you tried turning it off and back on again(both devices)? that, ^^ try the EMC/updates and confirm its pointing to the right ip.

 

[Eulogy]

iDRAC Express or Enterprise?
I know you went through and reset to default and such, but, I would double check:
Network Settings > Enable NIC -> "Enabled"
Network Settings > NIC Selection --> make sure it's "Dedicated" and not one of the LOMs.
Network Settings > NIC MTU --> Should be 1500 if reset, but, double check

By default they live at 192.168.0.120, not 0.0.0.0 as the other person said. DHCP works just fine on iDRAC if you have a DHCP server on that subnet/VLAN.

If all of that is OK, I'd plug one of the dark iDRACs into a switch port where there is a known good iDRAC. Just to rule out a whole lot of things.

Do you need the iDRAC to tag it's traffic maybe? Network > VLAN Settings > Enable VLAN & VLAN ID if so.

Unfortunately you're gonna need someone in front of the console here, and for awhile.

 

[Eulogy]

Also, if you have a local OS installed, you can interrogate the iDRAC with ipmitool. Start with

ipmitool lan print

and it'll dump most of the network settings. If any are incorrect, you can set them with ipmitool, too. So might not need a local person typing things in and rebooting the server.

 

[That_Sound_Guy]

Two things I would do, first, "Dell EMC Repository Manager" since dell no longer builds the iso and puts them on their support page, ya have to get that program, and build an ISO for your machine. It will fully update everything in it, including NIC firmware.
2nd, if nothing improved, install open manage. Open manage can go into the idrac from an OS.

If memory serves, idracs default to 0.0.0.0, you have to set an IP and I personally have no idea if dhcp works on them. If an idiot is doing this (ie remotely for you) they are likely setting the lifecycle controller ip and not the idrac ip. At least I can see someone doing that on the 740. I don't think the 730 bios setup has that extra screen. It's been years since I had to set up one of those. Maybe think about getting a network KVM so you don't have to deal with idiots. That and a PDU with switch ports. Nothing like a cisco switch that requires a power cycle to update remotely. If all else fails, find a video on youtube that shows how to set it up and send it to the worker. Course the workers could be dicking you around to get more hours... had one guy attempt that on me once.

I had to resort to onsite sending me endless cell phone screenshots of what they were doing to be sure. We were able to get one of our Dell's working in the same window, it's just these 2 being a royal bitch. 4 others in same location working as well.

I did the hard IDRAC reset to defaults (there's two of them) and I have all IP info set (verified by picture). One of the reasons I have them entering F2 BIOS to config versus F10 lifecycle at boot, lot less clutter in menus to get through to direct them. I wish I had a IP KVM, but a whole lot of security is in the way of allowing me to use one. I used one of my Dell's in my lab for walking them through the prompts. I'm thinking about trying to walk through the tech with a crossover cable and direct connect a laptop to the IDRAC port and set an IP to see if that talks.

have you tried turning it off and back on again(both devices)? that, ^^ try the EMC/updates and confirm its pointing to the right ip.

I've rebooted the Dell many times, even pulling the power and holding the power button down to drain the caps. If I rebooted the Cisco switch stack, I would have the FCC, local gubmint and legal on me within the hour. The stuff I work on is very touchy reaction wise. They don't like unapproved unnecessary actions. (Yes I know you were kidding)

iDRAC Express or Enterprise?
I know you went through and reset to default and such, but, I would double check:
Network Settings > Enable NIC -> "Enabled"
Network Settings > NIC Selection --> make sure it's "Dedicated" and not one of the LOMs.
Network Settings > NIC MTU --> Should be 1500 if reset, but, double check

By default they live at 192.168.0.120, not 0.0.0.0 as the other person said. DHCP works just fine on iDRAC if you have a DHCP server on that subnet/VLAN.

If all of that is OK, I'd plug one of the dark iDRACs into a switch port where there is a known good iDRAC. Just to rule out a whole lot of things.

Do you need the iDRAC to tag it's traffic maybe? Network > VLAN Settings > Enable VLAN & VLAN ID if so.

Unfortunately you're gonna need someone in front of the console here, and for awhile.

Enterprise with valid license
IDRAC is enabled (we save toggled it as well). NIC is dedicated, we cant use the quad LOM. MTU is 1500. IPV4 enabled, IPV6 disabled.
We don't do any DHCP, all hard coded. The IP info they have is correct, verified by picture from onsite.
That's my next step - pull a known working switch link if that fails, try crossover connecting a laptop.
VLAN tagging not required for this MGMT vlan. Its untagged on our working ones.

Not being onsite is a killer. I'm thankful I have them there to help but its so painful to IT talk to electricians/HVAC and etc skilled people.


As for the OS access to the backplane to get to IDRAC. This is tricky. For....reasons, this isnt possible in its current state. I am thinking about pulling the production drives and load spares to load a burner OS on to accomplish the local passthrough access.

Thanks for the responses all - Im open to any new ideas to solve this. I have built 40+ Dells from R720's on and never had this issue (We dont use Dell very often). Never had any similar issues with hundreds of HP ILO's either. Grr.

 

[Eulogy]

I had to resort to onsite sending me endless cell phone screenshots of what they were doing to be sure. We were able to get one of our Dell's working in the same window, it's just these 2 being a royal bitch. 4 others in same location working as well.

I did the hard IDRAC reset to defaults (there's two of them) and I have all IP info set (verified by picture). One of the reasons I have them entering F2 BIOS to config versus F10 lifecycle at boot, lot less clutter in menus to get through to direct them. I wish I had a IP KVM, but a whole lot of security is in the way of allowing me to use one. I used one of my Dell's in my lab for walking them through the prompts. I'm thinking about trying to walk through the tech with a crossover cable and direct connect a laptop to the IDRAC port and set an IP to see if that talks.

I've rebooted the Dell many times, even pulling the power and holding the power button down to drain the caps. If I rebooted the Cisco switch stack, I would have the FCC, local gubmint and legal on me within the hour. The stuff I work on is very touchy reaction wise. They don't like unapproved unnecessary actions. (Yes I know you were kidding)


Enterprise with valid license
IDRAC is enabled (we save toggled it as well). NIC is dedicated, we cant use the quad LOM. MTU is 1500. IPV4 enabled, IPV6 disabled.
We don't do any DHCP, all hard coded. The IP info they have is correct, verified by picture from onsite.
That's my next step - pull a known working switch link if that fails, try crossover connecting a laptop.
VLAN tagging not required for this MGMT vlan. Its untagged on our working ones.

Not being onsite is a killer. I'm thankful I have them there to help but its so painful to IT talk to electricians/HVAC and etc skilled people.


As for the OS access to the backplane to get to IDRAC. This is tricky. For....reasons, this isnt possible in its current state. I am thinking about pulling the production drives and load spares to load a burner OS on to accomplish the local passthrough access.

Thanks for the responses all - Im open to any new ideas to solve this. I have built 40+ Dells from R720's on and never had this issue (We dont use Dell very often). Never had any similar issues with hundreds of HP ILO's either. Grr.

My "quick" suggestion would be build an OS on a thumbdrive (or an image you can ship for them to load on the thumbdrive) with core utilities. Make it an Ubuntu Live and add ipmitool, net-tools, iputils, lldpcli (assuming this doesn't have easy WAN access).

 

[That_Sound_Guy]

Everyone likes a result to an issue ready to cause an Officespace printer beatdown:

End "fix" was to perform the BIOS and IDRAC upgrades, do a factory defaults reset of the IDRAC then pull the cable from the dedicated NIC while rebooting then reinserting it. Why that worked? How? No idea. We did the factory defaults before the firmware upgrades with no change. So my best guesses are the upgrades had a fix that applied when I reset defaults again (the jump in versions wasnt super wide) or interrupting the handshake to the switch somehow did something to restart/reset the IDRAC. Im leaning on option 1.

What a pain in the ass. Thanks for taking the time to offer suggestions and tips all.

 

[German Muscle]

sounds like the idrac was wonky, glad the fw updates fixed it. Ive never had a problem. I reset the drac when i get em, set a static and off it goes.

Two things I would do, first, "Dell EMC Repository Manager" since dell no longer builds the iso and puts them on their support page, ya have to get that program, and build an ISO for your machine. It will fully update everything in it, including NIC firmware.
2nd, if nothing improved, install open manage. Open manage can go into the idrac from an OS.

Ive been wondering how people do it. It sucks waiting for dell to update the Server Update Utility with all the latest versions. I know what im doing this weekend. tytyty