server room of doom

C

clsGrinder

n00b
Joined
Jul 15, 2005
Messages
15
I have a server room on the second floor of a building that keeps zapping motherboards, hard drives, raid cards, power supplies... you name it.

Everything is racked (25U), every component in the rack is grounded with a strap, and heavy gauge wire grounds the rack to the building (steel structure). Performed continuity check with multimeter.

Portable AC unit in server room, temp solid 21 deg Celsius.

Rel. humidity between 15% and 25%

Everything UPS'd (model of choice is CyberPower 2200 2U w/ LCD screen)

Equipment is on second floor of structure, wood sub-floor, commercial tile.


Can anyone who IT's for data centers or who has solved a recent problem please advice... i'm losing my mind.

thanks! :)
 
Gremlins or grounding issues. Do you use UPS' to condition power?

How many people have access to your server room? Lots of foot traffic and or manhandling of the gear?
 
My guess is also grounding issue or power leaking to ground (short). Check your neutral to ground voltage- should be zero, or close to it. Check ground voltage from your grounded equipment to a large metal structure not attached to ground- like a free-standing rack. Voltage should again be zero.
Check for over/undervoltage by checking voltage across AC hot (black wire or short/narrow slot on 110) and Neutral. Voltage should be ~97-125 VAC.
If anything falls outside these ranges, call an Electrician. Might be smart to call an electrician anyways (not a maintenance guy with a Voltmeter).

You may also be creating current flow accross the equipment if the equipment is not designed with a separate chassis ground, or is not intended to have the chassis grounded. Multiple grounds can actually allow voltage to flow (grounds at different electrical potential) causing your problems. This is another case of 'more is not necessarily better'.
 
I've measured instances where the computer system lifted the voltage potential of the building steel. In that case we added a isolation xmfr built a isolated grounding system and used that for the entire computer system. If we needed to connect to another system we used fiber so that we would not induce a ground loop.
 
Why would you even need ground straps from the equipment to the rack? The mounting hardware is going to give you a really good ground unless you are using non-metal mounting hardware.

You also do NOT want the ground going from the rack to the building.

You are most likely creating a short that way.

The main breaker box will have an earth ground, and all other breaker boxes will have a neutral ground.

By grounding the rack to the building, you are creating an earth ground to neutral ground short.

This is going to screw with voltages as well as give you power through ground.

Remove that ground from the rack to building and it should fix it.
 
I'd start with cyclone3d's suggestion.

Also the humidity is rather low in that room. Aim for 30-35%. Electrostatic energy builds exponentially at humidity levels under 30%.

Example... Going to 40-35% would be slightly worse. Going from 25-20% could potentially create over 10 times the ESD events.

cyclone3d said:
Why would you even need ground straps from the equipment to the rack? The mounting hardware is going to give you a really good ground unless you are using non-metal mounting hardware.

You also do NOT want the ground going from the rack to the building.

You are most likely creating a short that way.

The main breaker box will have an earth ground, and all other breaker boxes will have a neutral ground.

By grounding the rack to the building, you are creating an earth ground to neutral ground short.

This is going to screw with voltages as well as give you power through ground.

Remove that ground from the rack to building and it should fix it.
Click to expand...
 
thank-you all

perhaps some more background and an illustration

datacentre.png


Ehren8879: Server room is secured, has it's own door sensor on the building alarm system. Only management and IT has keys.

RocketTech: A commercial electrician checked the ground and power (112 v IIRC). There are no other free standing metal items in the server room for me to do your recommended check <quote> Check ground voltage from your grounded equipment to a large metal structure not attached to ground- like a free-standing rack </quote>

Mackintire: this sounds complicated

cyclone3d: the ground straps were recommended by the electrician, he observed poor ground between the rack and the wall-plate. We had issues prior to all the grounding straps... we installed the grounding staps with the hopes that it would eliminate this problem.

Mackintire: This is a dry part of the country. I am aware of offices that operate in dryer conditions than this data-centre and do not have these issues.




Thank you all for your interest! Looks like i have some multimeter homework to do this weekend!
 
clsGrinder said:
thank-you all

perhaps some more background and an illustration

datacentre.png


Ehren8879: Server room is secured, has it's own door sensor on the building alarm system. Only management and IT has keys.

RocketTech: A commercial electrician checked the ground and power (112 v IIRC). There are no other free standing metal items in the server room for me to do your recommended check <quote> Check ground voltage from your grounded equipment to a large metal structure not attached to ground- like a free-standing rack </quote>

Mackintire: this sounds complicated

cyclone3d: the ground straps were recommended by the electrician, he observed poor ground between the rack and the wall-plate. We had issues prior to all the grounding straps... we installed the grounding staps with the hopes that it would eliminate this problem.

Mackintire: This is a dry part of the country. I am aware of offices that operate in dryer conditions than this data-centre and do not have these issues.




Thank you all for your interest! Looks like i have some multimeter homework to do this weekend!
Click to expand...

Definitely get rid of that ground strap from the rack to the building.

I have rewired most of my current house due to bad wiring and did a lot of research (4+ hours in one sitting) and found out that it is very bad to have anything on a sub panel earth grounded.

It always needs to be neutral grounded or else you will get current flow through the ground.

I am pretty sure that that ground strap from the rack to the building is what is causing your problems.
 
cyclone3d said:
Why would you even need ground straps from the equipment to the rack? The mounting hardware is going to give you a really good ground unless you are using non-metal mounting hardware.

You also do NOT want the ground going from the rack to the building.

You are most likely creating a short that way.

The main breaker box will have an earth ground, and all other breaker boxes will have a neutral ground.

By grounding the rack to the building, you are creating an earth ground to neutral ground short.

This is going to screw with voltages as well as give you power through ground.

Remove that ground from the rack to building and it should fix it.
Click to expand...

Actually it is code in most locations that the RACK is tied to earth ground, this is a safety requirement. Most racks that I have worked with have a ground lug to attach a #6 green wire to just for this purpose. You will also notice that the racks will have various green/yellow wires attaching the doors and side panels to ensure the entire rack is grounded.

Also by code the ground lug can NOT have any other purpose such as mounting hardware. It must only be used for ground. Which is why the equipment has its own ground strap to the rack.
 
cyclone3d said:
Definitely get rid of that ground strap from the rack to the building.

I have rewired most of my current house due to bad wiring and did a lot of research (4+ hours in one sitting) and found out that it is very bad to have anything on a sub panel earth grounded.

It always needs to be neutral grounded or else you will get current flow through the ground.

I am pretty sure that that ground strap from the rack to the building is what is causing your problems.
Click to expand...

Your issue with sub-panel is different than grounding a metal enclosure. The earth ground is really just connected to the metal of the frame, it is there in the event that a hot wire ever come into contact with anything conductive that the current is quickly shorted to ground causing the breaker to trip. Without this, the metal structure is now HOT and ready to send current to the first thing to connect it to ground (usually the IT guy that opens the door). If everything is working correctly earth ground is really nothing different than connecting a wire to the case and to the dirt in the back yard.

The ground straps if tied to earth ground and NOT neutral are not causing any issues.
 
Telling the poster to get rid of the ground strap is very BAD. OP should have a electrician verify the setup. He will most likely find out that the ground strap is required. Ever datacenter I have worked in has large copper bus bars that every rack is ground to. This is pre electrical code.
 
15% humdity is actually on the low side. That could be an issue. I know when I had equipment at Equinix they had the humidity very low and they discovered that harddrives failure rate was related to the low humidity, the speculated that the low humidity was damaging the bearings.
 
clsGrinder said:
thank-you all

cyclone3d: the ground straps were recommended by the electrician, he observed poor ground between the rack and the wall-plate. We had issues prior to all the grounding straps... we installed the grounding staps with the hopes that it would eliminate this problem.
Click to expand...

I am not an electrician or certified tech, but this doesn't sound right to me. If you had a problem with the electrical wiring between the rack and the wall, why try to solve it by adding more wiring? Wouldn't it be better to correct the original problem? If there is something going on with the building's wiring, you could end up with a situation where other devices in the building are being grounded through your server rack, if that is the best (path of least resistance) ground.
 
m1abram said:
Actually it is code in most locations that the RACK is tied to earth ground, this is a safety requirement. Most racks that I have worked with have a ground lug to attach a #6 green wire to just for this purpose. You will also notice that the racks will have various green/yellow wires attaching the doors and side panels to ensure the entire rack is grounded.

Also by code the ground lug can NOT have any other purpose such as mounting hardware. It must only be used for ground. Which is why the equipment has its own ground strap to the rack.
Click to expand...

In that case, you either need to eliminate the ground going to the wall plug or remove the grounding straps from the equipment and have them on fully insulated mounts.

If you do not, you are going to have issues with frying equipment due to the voltage flowing back through the ground to the equipment, not to mention it is going to mess up the voltage that the equipment is getting.

ONLY IF the equipment is hooked up to the main breaker panel would it be correct for it to be earth grounded. If it is hooked up to a sub-panel which is neutral grounded, having it earth grounded is a safety hazard.

Not only does it eliminate what the ground is there for, it will cause stuff to fry because voltage will be flowing where it isn't supposed to be flowing.

Edit: Here is a link to a pdf that has to do with national electrical code.

If you look at page 18 of this pdf, it shows part of what is caused by running an earth and neutral ground together.

http://fyi.uwex.edu/mrec/files/2011/04/W4.-Biesterveld-NEC-grounding-MREC2010.pdf
 
Last edited:
cyclone3d said:
In that case, you either need to eliminate the ground going to the wall plug or remove the grounding straps from the equipment and have them on fully insulated mounts.

If you do not, you are going to have issues with frying equipment due to the voltage flowing back through the ground to the equipment, not to mention it is going to mess up the voltage that the equipment is getting.

ONLY IF the equipment is hooked up to the main breaker panel would it be correct for it to be earth grounded. If it is hooked up to a sub-panel which is neutral grounded, having it earth grounded is a safety hazard.

Not only does it eliminate what the ground is there for, it will cause stuff to fry because voltage will be flowing where it isn't supposed to be flowing.

Edit: Here is a link to a pdf that has to do with national electrical code.

If you look at page 18 of this pdf, it shows part of what is caused by running an earth and neutral ground together.

http://fyi.uwex.edu/mrec/files/2011/04/W4.-Biesterveld-NEC-grounding-MREC2010.pdf
Click to expand...

You are incorrect, even LV metal boxes have to be tied to earth ground. I think you are confusing neutral with ground.

You do NOT tie your subpanel neutral to the case since the case of the subpanel is tied to earth ground. This is completely different issue than tieing the enclosure of the equipment to earth ground. The equipment case is all tied to earth ground this is per code and described in that pdf. Note that no consumer equipment exposes neutral and does not internally tie neutral to earth ground, that would be bad. However what you may be confused with is in a sub panel you have both a neutral bus bar and the earth ground of the metal enclosure that is the panel box. On a mains panel neutral and earth are directly tied together, but on a sub panel you do not want this. Again this is not the situation with servers and racks.
 
FYI - I really hope you have your subpanel box tied to earth, but have the neutral isolated from the box itself.

This is really no different than when you wire to a metal J-box, you always tied the earth ground to the metal box but you must not tie neutral to the box.
 
Humidity is the problem here. You want 45%. I've even heard of SANs killing themselves in extremely low humidity environments. Low humidity like that is a no go for a datacenter.
 
m1abram said:
You are incorrect, even LV metal boxes have to be tied to earth ground. I think you are confusing neutral with ground.

You do NOT tie your subpanel neutral to the case since the case of the subpanel is tied to earth ground. This is completely different issue than tieing the enclosure of the equipment to earth ground. The equipment case is all tied to earth ground this is per code and described in that pdf. Note that no consumer equipment exposes neutral and does not internally tie neutral to earth ground, that would be bad. However what you may be confused with is in a sub panel you have both a neutral bus bar and the earth ground of the metal enclosure that is the panel box. On a mains panel neutral and earth are directly tied together, but on a sub panel you do not want this. Again this is not the situation with servers and racks.
Click to expand...

Maybe. Anyway, looked up some more stuff and I think I was confusing myself before.

In any case, the ground that is not getting a good connection at the wall needs to be fixed.
 
m1abram said:
FYI - I really hope you have your subpanel box tied to earth, but have the neutral isolated from the box itself.

This is really no different than when you wire to a metal J-box, you always tied the earth ground to the metal box but you must not tie neutral to the box.
Click to expand...

Got it.. Don't even have a sub-panel wired up right now because of other projects.
 
you are all wrong, all he has to do is get an isolated ground electrical socket installed.

if your equipment is grounded to earth, you also have to use an isolated ground circuit as it elimanates the possibility of ground loops.

whatever contractor was hired to install electrical service to the server room should not be allowed back on the premise. Any electrician that doesn't ask what a higher amperage plug (20a+) is going to be used for shouldn't be working on power for a server room.
 
I insist on isolated ground power plugs for all server room power outlets when I am consulted. I have not had this issue for any server room that does not have isolated ground plugs however.

I am considering leaving the wiring as is... i have observed no change with or without the grounding straps. I figure I will add humidity monitoring hardware and try getting the RH up to 35~30 levels... and be alerted remotely if the RH approaches 40 and beyond levels.
 
clsGrinder said:
I insist on isolated ground power plugs for all server room power outlets when I am consulted. I have not had this issue for any server room that does not have isolated ground plugs however.

I am considering leaving the wiring as is... i have observed no change with or without the grounding straps. I figure I will add humidity monitoring hardware and try getting the RH up to 35~30 levels... and be alerted remotely if the RH approaches 40 and beyond levels.
Click to expand...

well as it stands the setup violates NEC and is incorrect. Ground loops cause all kinds of crazy issues. They are also are usually pretty uncommon issues.

Do you have anything on any of the racks or in the same building that uses coax (cable TV, cable modem, etc) that has a 3 prong plug? If you do that can cause a ground loop and cause voltage and noise on the ground. An isolated ground outlet will also fix these issues as long as the rack's ground is also grounded to a GOOD earth ground.
 
Normal relative humidity levels are generally 40-50%. Get your levels up now. 21C/70F is also very low. Majority of equipment have upper limits of 95F. The higher your temperature, the more energy savings you will experience. 25U racks are tiny, all your servers are at the floor where cool air is anyway.
ASHRAE Data Center Cooling Guidelines
I have my data center top of rack alarm points at 85F and 60% relative humidity. I run my data center at 74F and 45% RH. I plan to implement cold aisle containment and bump my CRAC units up to 76-77F.

While not completely necessary unless stipulated by codes, it is a wonderful idea to ground all cabinets to an SRG (signal reference grid). Without them, you do have the possibility of killing people by touching rack-to-rack or other grounded source given the right conditions. Generally that grid ties back to building steel which ties back to the building's ground point. Without an SRG, you have no assurances that the potential difference between two ground points will be 0. (Grounding studies can be conducted and corrected for less than a $1k).

Sub-Panels/Remote Power Panels do not get their grounds and neutrals bonded. Ground and neutral are bonded back by the transformer. This gets rid of your ground loops, which are dangerous.

Honestly, I highly suggest you have a Power Quality study done on your facility. I regularly attend monthly seminars put on by p3-inc.com / powerqualityuniversity.com. I only name them as an example of the services they perform. I don't know where you are located, but I suggest you find something like them. None of us here are power quality experts, they are and have the correct tools to figure out what is going on correctly, the first time.
 
Temp is fine.

15% humidity is retardedly dry. 40-60%.

Get it wet, and get it wet now. If the current equipment can't do it, get better equipment.
 
mwarps said:
Temp is fine.

15% humidity is retardedly dry. 40-60%.

Get it wet, and get it wet now. If the current equipment can't do it, get better equipment.
Click to expand...
The 12 rack room I work in is normally below 15% RH (about 7 months of the year solid ). I have NEVER seen any of the issues that are being discussed in 8 years.

We actually do have a small humidifier on the CRAC, it's just not even close to high volume enough to make a difference. When it's turned off we've seen 2% RH.
 
Innocence said:
The 12 rack room I work in is normally below 15% RH (about 7 months of the year solid ). I have NEVER seen any of the issues that are being discussed in 8 years.

We actually do have a small humidifier on the CRAC, it's just not even close to high volume enough to make a difference. When it's turned off we've seen 2% RH.
Click to expand...

Check your instrumentation. You may also have appropriate, functional dissipative flooring. OP obviously does not. Science is science. If there's a static problem in air, humidity will fix it.

Also check appropriate humidity levels at professional data centers. (Hint, they're between 35 and 60 % published numbers)

Your particularly odd instance of a server room is not indicative of, and should not be taken as an example of, other server rooms.
 
You said commercial tile, but what kind specifically? Is this raised floor tile or is this ceramic tile? If it's ceramic, is it anti-static? If it's not specifically anti-static ceramic tile (or raised data center tile) then just know that ceramic tile actually builds a shit ton of static electricity. Between that and your super dry air, you may have found your problem, or atleast a large contributor to it.
 
update: humidity has been in the high 20's, issues persist. Changing out one of the stand-by UPS's and replacing it with a Double-Conversion type UPS to see.
 
okay... so there is evidence of corrosion on the PCB's of the hard drives, server motherboards and RAID card PCB's. I think I might need some other forms of environmental instrumentation to pin down what is going on. Goggling suggests some Corrosion Classification Coupons? Do I need to partner with a hazardous assessment company?
 
Call a certified electrician and have them do a head to toe inspection.

That is what you should have done instead asking here only to discover you were told to call an electrician.
 
clsGrinder said:
okay... so there is evidence of corrosion on the PCB's of the hard drives, server motherboards and RAID card PCB's. I think I might need some other forms of environmental instrumentation to pin down what is going on. Goggling suggests some Corrosion Classification Coupons? Do I need to partner with a hazardous assessment company?
Click to expand...

Your rack is grounded to two places that have different potential grounds, causing a ground loop. Your rack is passing voltage from one ground to the other.

voltage leakage from ground can cause corrosion on grounded electronics.

Call an electricion and have them put in isolated ground outlets.

I HAVE TOLD YOU THIS 3 TIMES! YOUR CURRENT ELECTRICAL SETUP VIOLATES THE NEC! THE NEC EXISTS FOR A REASON.

Ever wonder why some car's batteries corrode and other's don't? Cause of poor grounds.

Fix your ground loops and you will fix all your problems.
 
Guys ; ease down on the "call an electrician" rhetoric.

Any of you guys ever actually *had* to call an electrician to hunt down phantom ground loops ? sheeeesh , good luck with that ! ( and enjoy paying THAT bill ) :rolleyes:
....oh and I absolutely promise that 100% compliance with the NEC does NOT guarantee an absence of ground loops.

Anyways....

I'm one of Grinder's friends . I know the latest update on this.

It has exactly NOTHING to do with ground loops of any kind !!

--------

Grinder did a lot of work to discover this , so I'll let him update you properly.It's really amazing actually...

:D
 
bds1904 said:
Your rack is grounded to two places that have different potential grounds, causing a ground loop. Your rack is passing voltage from one ground to the other.

voltage leakage from ground can cause corrosion on grounded electronics.

Call an electricion and have them put in isolated ground outlets.

I HAVE TOLD YOU THIS 3 TIMES! YOUR CURRENT ELECTRICAL SETUP VIOLATES THE NEC! THE NEC EXISTS FOR A REASON.

Ever wonder why some car's batteries corrode and other's don't? Cause of poor grounds.

Fix your ground loops and you will fix all your problems.
Click to expand...

Really want to see how you can possible get a ground loop internal to the power source (i.e. battery). Corrison on a car battery terminal has absolutely nothing to do with a ground loop.
 
thx brain

this room is being analyzed by it's second master electrician. All grounds, hookups and wiring is approved. We can move on from that argument entirely.

Can anyone recommend a good incense to calm the spirits in the room? I'm thinking this is something NewEgg should start carrying :p
 
Call your counties electrical inspector and ask them how your rack should be grounded. In my area, the rack needs to be bonded via 6 gauge copper to the building ground. (Which also happens to be where the panels ground.) If you are on a sub panel, have a electrician verify the ground and neutral are seperated correctly. This alone could cause your issues. I doubt it has anything to do with the flooring or humidity as I have ran gear in 10x worse conditions (with proper power), as many others have- with no issues.


And to those screaming about NEC: Unless you are a certified electrician in his county, please knock it off. You have no idea which localized version of the NEC his state/county uses.


*Just saw your post. Two things. 1) Did he pop the panel cover and if this is a sub panel- did he check the main panel? 2) Call the inspector, I don't care what the electrician claims, it's the inspectors job to make sure they do their job right. I went through 3 electricians before I found a decent one.
 
Heh...

I'm not saying that 'Call an electrician' is a wrong answer...It's not . Asking for professional help is always a good bet....But if you are thinking 'Call some guy outta the phone book ' then this is terrible advice.

...Oh and by the way , while I'm not officially an 'electrician' , I have 20 years of experience in the field , my immediate supervisor is a master electrician 30+ years.I have some idea of what I'm talking about.

This problem is not related to any overt electrical issue.(( I know what the deal is))

---

I can't say more , other than Grinder did his homework , and his theory is sound....Grinder will update this thread when he can.

-------
I prefer 'spring mist' incense from ganesh.

:D
 
You must log in or register to reply here.
Back
Top