Here We Go Again: AI Developers Are Buying Gaming GPUs in Bulk
https://www.extremetech.com/computing/here-we-go-again-ai-developers-are-buying-gaming-gpus-in-bulk
a software engineer has revealed a bulk purchase of AMD Radeon cards for AI training.
The revelation of a sizable GPU purchase appeared on the social network formerly known as Twitter this week by George Hotz, whose profile says they are the president of an AI company. The post says AMD is a real pleasure to work with when purchasing GPUs in bulk.
AMD is negotiating and selling directly to developers instead of going through a channel partner like Amazon or Newegg. That's an interesting ripple, as in the mining craze it was never known (to our recollection, at least) if AMD and Nvidia were selling bulk orders to GPU miners. We'll never know if that happened, but it seems like it's happening now on the AI front.
One interesting aside with the above purchase of AMD GPUs is that certain AI workloads can use a lot of memory. While paying nearly $1,000 per GPU for an AMD 7900 XTX that packs 'only' 122.88 teraFLOPS of FP16 number crunching prowess might seem odd, given the RTX 4060 costs $300 and can provide around 121 teraFLOPS of FP16 compute (with sparsity), the AMD GPU provides three times as much VRAM. So, if the workloads need memory, you'd need three RTX 4060 cards or two RTX 4070 cards to get the same 24GB — plus the servers to hold all those GPUs.
(This favors the 16gb version of the 4060 ti)
https://www.tomshardware.com/news/evidence-shows-ai-driven-companies-are-buying-up-gaming-gpus