THE NVIDIA AI GPU BLACK MARKET ｜ Investigating Smuggling, Corruption, & Governments GamersNexus

Category: Technology Investigation

Tags: AIChinaGPUsNvidiasmuggling

Entities: ChinaDepartment of CommerceJensen JuanNvidiaUnited States

Transcript

00:00

[Music] If they don't have it, they come here. [Music] These CPUs are assembled in China.

So they apparently I guess they have spare

00:16

and they buy it from these warehouses which are on the upper floors of the building. They just have a carton.

There's like probably a dozen tall and about the same deep. There's a plumber.

Yep. They're I said they they're being full.

They're quite well utilized.

00:33

They one of them sent me this price sheet. Oo.

We're multiple administrations deep in a technological cold war over processing power between the United States and China. China's cyerspace administration has labeled some US graphics processing products as a security risk, seeking answers about

00:49

US government backdoors in silicon. Meanwhile, the United States has imposed heavy restrictions on exports of graphics processing units or GPUs being sold to Chinese companies by American companies.

The sale requires rarely granted licenses for each import scenario to legally export GPUs above a

01:06

certain performance level with the stated objective being to restrict progress of private and government projects, including AI development in China, while trying to maintain the US's AI leadership. The United States takes this so seriously that just this week,

01:22

the Department of Justice had two Chinese nationals arrested in California for what it alleges is the smuggling of tens of millions of dollars worth of GPUs. Because where there's prohibition, there's smuggling.

01:40

We spoke to everyone about this Nvidia AI GPU black market in China and the US. We found middlemen who connect buyers and sellers.

This is it. Cool.

Wow.

02:02

Users who can understand the demand and explain it using the most dystopian definition of wealth. How many GPUs they have.

They have much more st than the university. We're GPU leader.

[Music]

02:19

Wow. And we got them quite cheap.

What model? This is the A1 A100.

Oh, A100. These are A100.

We also found independent repair shops who simply doing their jobs salvage valuable

02:34

silicon components from dead PCBs of banned GPUs, innovatively hand modifying them to be better than and have more VRAM than Nvidia's own official products. [Music]

02:54

They should make one right now. Okay.

Hey, cool. We met multiple people who when asked the same question gave the same passcode like Chinese idiomatic expression or chu

03:11

Nvidia. Do you think Nvidia

03:27

knows? Do I try to stop?

[Music] And among others, we even spoke with a US-based Chinese national buying video guards in the US to strip them and ship the GPUs to companies in China,

03:42

violating US export control law. I haven't seen someone have a computer in a car for a long time.

These nervous laughs come from a viewer of ours while meeting in Arizona with a GPU smuggler we'll call the plug. The smuggler spoke limited English, but they

03:58

both understood one universal truth. Yeah.

Okay. Two universal truths.

The second one is money. And this battery operated Prius mounted GPU testing rig with fur mark is the least suspicious thing in his car with a spare license plate in

04:14

the trunk. We'll come back to the plug later.

Black market is normally a phrase associated with drugs or guns, but there's a new kind of black market and it's high-end AI GPUs. And this particular black market is worth billions of dollars a year.

And it's

04:29

hiding in plain sight.

04:47

This is Hong Kong. It's our first stop on this journey.

By skyscraper count, Hong Kong would be the

05:04

tallest city on Earth. The density is unbelievable.

We spent a few days here for this story, wandering markets and meeting sources. [Music] Do you feel like the ban is it targeted

05:19

at like military uses where they're trying to prevent that? I don't think from the US government's perspective there's this distinction between like Chinese academia versus Chinese military versus Chinese commercial.

I think no I don't think they they really consider there's a distinction right

05:36

yeah just total yeah meridian or band yeah so these GPUs are almost certainly moved one at a time right so it's very hard to get like a full HGX system right oh so the 90 normally comes from

05:51

Australia and Taiwan yeah and that's why we're here to learn what the demand drivers are for these GPUs how they get into China and more about the illicit side of the GPU smuggling business.

06:07

The community brought you this movie. This production is being funded by our viewers.

Rather than putting a third party ad here, we're supporting ourselves via the community so that we can produce more fully independent movies like this. We launched our own Kickstarter style fundraiser on our store linked below.

In addition to

06:22

physical products like our blind eye t-shirt, black market GPU sticker backs, blacklisted t-shirt, and our new GPU VRAMm glassware, we also have digital rewards available exclusively to backers on this page. These backer tiers are available only for one more week, featuring bonus videos, wallpaper packs,

06:39

and the highest tier even has a Q&A session with me, a tube of cane pin thermal paste, and features a canvas print of a watercolor painting painted by my mom. Actually, based on a photo I took in Hong Kong, we even hit a super stretch goal where all backers of any tier will receive digital rewards on a

06:55

USB key, including a copy of this very movie so that you can watch it anywhere, anytime, without internet, and without YouTube AdSense ads, and preserve it in case it's taken down. If you prefer a single item, we have a brand new blind eye t-shirt featuring traditional Chinese characters written right to

07:10

left, top to bottom, representing the idiomatic expression we kept hearing during this investigation. easier.

Yeah, the shirt features a cyberpunk dystopian style open eye with a camera as the iris. Our fundraisers already hit several stretch goals, and because it's brought in so much, we're going to release even more black market backer

07:28

exclusive videos that you'll get shortly for backing this project. All of these items will ship throughout September and October as we bring them in.

So, place your orders now to get them when they go out. The link is below.

Now, back to the program. This was our latest big adventure, and it began when we booked a

07:44

24-hour ticket to Hong Kong. 20 days of hotels across Hong Kong, Shenzhen, Janjjo, Huo, Donguan, Taipei, and more.

At least one of us got detained by at least one of the governments involved in this video, but we can't talk about these indecipherably singular plural instances or instance of any affformentioned detainment. Next, we talked to a lawyer about the previous

08:00

sentence, loaded into a cab that secured our luggage like this, then investigated whether or not GPS are actually smuggled with lobsters into Hong Kong. Nvidia was right.

We've uh we've checked the lobsters for sensitive electronics and aren't any in there. So,

08:16

Good move, Jensen. We found a pile of GPUs that are now export controlled and met a professor who really wants to make sure you know these are these are legally obtained legally obtained.

Came to a sudden realization that if a band GPU is assembled in China

08:32

then what's to stop them from just taking it off? That's actually very good question because um what's preventing these GPUs from falling off of the line?

Line. Exactly.

Made friends with three different cats in Hong Kong. What do you think of the temple guards, Tannon?

08:47

They're uh they're very good at their job. And said goodbye to this RGB city.

We then booked a boat to Shoko, China. I told Tannon what Shoko means.

You know what Shoko means, Tannon? What does that mean?

Snake's mouth. Sounds inviting, right?

Yeah. Awesome.

Can't wait.

09:02

We arrived in Shenzhen. Made small talk with hotel room service.

So, uh, how long have you worked at the hotel? Immediately got trapped in the hotel elevator by my new friend.

traffic jam. Oh, almost witnessed a workplace accident.

09:22

Got kicked out of a warehouse. We got kicked out.

Yes, we did. And then we instantly became fans of using highspeed rail at hundreds of miles an hour to go deep into China to meet a guy who really thinks that desoldering a GPU and rebaling it is no big deal.

Really? That's the easy part.

09:38

Oh, really? Okay.

It's a lot easier than doing this. We got a ton of information in this video about GPU smuggling, including from this guy named Five.

09:53

Mhm. After that, we went back to Shenzhen for the third time in two weeks.

Met this guy, support game natures, support Jesus. Yeah.

Yeah. I'm the he his father.

went back to Hong Kong, flew to Taiwan, got

10:08

some street food, and got out on a plane back to the US. We spoke to a lot of people for this story, to go on record and learn about the dystopian world of high-tech GPU smuggling.

We spoke to people ranging from owner operator trading companies to professors of economics building data centers.

10:33

These are very expensive items, right? I would imagine you would keep track of everything.

We even tried to talk to the US Department of Commerce, but they didn't reply and every person on the chain from the Department of State had out of office autoresponders because we coincidentally emailed them the same week about an especially problematic story that enigmatically involves name dropping. But let's keep this focused on

10:49

the hardware. Some of our sources are more confused by Nvidia than others today.

And people seem to laugh every time we ask about how many Intel GPUs they sell.

11:08

But we want to connect as many dots of the smuggling line as we can today. And that's why we shot over 12 hours of interview footage that we just spent weeks cutting down for time.

The point is finding people who know people. And each person in our lineup today has led us to at least one other person in this

11:25

video, eventually building the full pipeline of smuggler to user. Although this story isn't about drugs, it is about a different thing that billionaire executives use to get high from.

Is AI AI AI AI AI AI. But before we get to any of those interviews, we need to establish the

11:41

basics of this geopolitical mess. The story is complicated, so we'll start with defining these key facts.

Why these GPUs are banned, the new 15% license, which only applies to two GPU models, who buying and selling is legal or illegal for, and then the timeline. AI

11:56

GPUs have been in even the mainstream news constantly. China has spent decades modernizing its military with the express goal of catching up to the US.

AI enabled drones are key to achieving that goal. We still sell billions of dollars a year of semiconductors to China.

We just

12:13

cannot let them access the most sophisticated cuttingedge artificial intelligence chips. The export ban did prevent China from getting the highest end American chips, but it did not stop China from catching up in the AI race.

China's AI technology is getting a lot better.

12:28

The concerns over China's focus on AI. China's China Chinese China.

China. This story has been a complete mess to follow.

It spanned years and now two US administrations. We ended up with over 400 pages of research and maps and finding sources and trying to understand

12:45

the laws. Here's why governments care about AI GPUs.

Nvidia functionally holds a GPU monopoly in our sector of the industry which is building computers to play video games that now feels relatively innocent by comparison to AI. The company leveraged decades of gaming

13:02

domination to build a foundation for what it now focuses on, which is making the most powerful GPUs for AI in the world. Reports warn of Nvidia product use in nuclear weapons research, facial recognition technology allegedly used in Russia to suppress descent, and growing

13:17

concerns of AI facial recognition use in the US for similar deployments, alongside reports of use in international spying and in drone warfare. Nvidia finds itself in the middle of all of this.

Even though Nvidia disputes selling to some of these entities, for example, it says that it

13:34

doesn't sell GPUs to Russia, the products still find their way there. So Nvidia is making money one way or the other.

Someone is buying it maybe from someone else who bought it from someone else. And much like we're talking about today, it might be transacted through smuggling.

But Nvidia does end up

13:50

selling the device ultimately to somebody. Regardless of who they sell to, Nvidia plays a big part in this worldwide obsession with AI.

AI AI AI AI AI AI AI AI AI AI will also create AI using AI to create AI. And we think it's

14:06

playing all sides. We'll talk about that more at the end, though.

And besides, when there's a gold rush, it's better to sell the pickaxe than swin it. There's so many companies that would like to build.

They're sitting on gold mines. Gold mine.

Gold mine. It's a gold mine.

Gold mine. Gold mine.

It gives me

14:22

tremendous joy. If you just buy something from Nvidia, we're going to be rich.

Although the US doesn't talk too much about its own use of AI, it spends a lot of time talking about China's. These semiconductors are unbelievably powerful and we can't let them get into the wrong hands.

We want the most other countries using

14:40

our technology because again it's a zero- sum game. The concern of course in Washington and particularly amongst the defense department, the Pentagon is that these chips and this hardware and software ends up with the People's Liberation Army, the Chinese military and will help them accelerate in terms of innovations on that front.

The restrictions are

14:55

really on advanced semiconductors that could have um potential military applications. And the goal is really to sort of choke uh China's AI sector and and its broader AI ambitions.

The US restricts Nvidia's GPUs through export control rules that ban the sale

15:11

of certain GPUs into China. That means the sale of GPUs by American companies or companies which want to transact business in America to companies that are in China or the Chinese government itself.

Some examples of export control

15:28

GPUs include gaming GPUs like the RTX5090 and the RTX490 which are useful in AI applications mostly for their high VRAM capacity. It also includes specific data center and AI GPUs like the A100, H100, H200, and B100 with the letter

15:44

being the signifier for the architecture as well as other GPUs shown on the screen. Now, this list is in flux.

There are some new and incoming exceptions for the Nvidia H20 specifically, which has faced Schrodinger's GPU ban, depending on whether CEO Jensen Juan had a $1

16:01

million dinner at Mara Lago with Donald Trump on a given week. A timeline would help, but we have 9 years to recap, so we'll do it quickly.

Even at the end of the Obama administration in 2016, the US government was just starting to talk about AI in relation to national security. In this wired interview,

16:17

developing international norms, rules, protocols, verification mechanisms around cyber security generally and AI in particular uh is in its infancy. You got a lot of non-state actors who are the biggest

16:33

players. Part of the problem is is that identify as buying who's doing what is much more difficult.

Yeah. If you're building a bunch of ICBMs, we see them.

Uh if somebody's sitting at

16:51

a keyboard, we don't. And so, uh you know, we've begun this conversation.

Uh a lot of the conversation right now is not at the level of you know dealing with real sophisticated

17:08

AI but has more to do with essentially states establishing norms about how they use their cyber capability. Who are you more afraid of big brother and the state or the guy who's trying to empty out

17:23

your bank account? Uh, part of the reason that's so difficult is that if we're going to police this wild west, whether it's the internet or AI or any of these other areas, then by definition, the government's got to have

17:40

capabilities. If it's got capabilities, then they're subject to abuse.

And uh, at at a time when there's been a lot of mistrust built up about government, that makes it difficult. Those were the early days.

The first Trump administration started a commission advising Congress on

17:57

maintaining AI leadership, including simply banning the sale of some advanced semiconductor equipment and chips to China. Several years of back and forth, a pandemic and election and chat GBTs launched later, and the Biden administration took major action in 2022 by restricting exports to China, Hong

18:12

Kong, and Macau. Nvidia's Ampier architecture A100 and newer code name Hopper H100 GPUs and systems were all restricted.

Nvidia shed tears for $400 million worth of lost sales as a result. And it was especially sad when many of its export compliant alternatives to

18:28

these also got banned, like its newly created A800, H800, and L40s in addition to Nvidia's RTX 4090 gaming card. Nvidia said it didn't expect quote near-term meaningful impact end quote on its financials.

Nvidia responds by designing

18:44

a workaround to the previously worked around workaround leading to the H20. Then the government adds new rules for high memory bandwidth cards.

And the Biden administration tried to come up with an AI chip diffusion rule that would limit the quantity of GPUs being sold into different countries rather than only by processing power metrics because the government really didn't

19:00

know how the to measure these things and Nvidia knowing more about GPUs could tweak any dial it wanted to just barely be compliant. Then Deepseek came out, everyone panicked, stocks plummeted, and the government scrutinized the role of Nvidia's GPUs in it.

In February 2025, fiscal year 2025 results are posted and Nvidia's Singapore revenue rockets to 18% of

19:17

total revenue based on customer billing location despite shipments to Singapore being claimed to be less than 2% of fiscal year 2025 revenue, causing people to say, "Wait a minute." Unrelated, several GPU smugglers are arrested in Singapore one day after the fiscal year report was posted, causing people to say, "Ah, that makes more sense." AMD

19:35

spawns out of nowhere to say it wrote down $800 million of inventory due to export controls. Nvidia 1 ups it with a write-off of $5.5 billion.

May 2025, Trump implements wide sweeping tariffs and rescends the Biden chip diffusion rule that would have limited how many AIG GPUs Jensen could sell to other

19:51

countries. Jensen says it's just an incredible vision.

I think this is going to be a transformative idea for the next century for us. These two initiatives are completely visionary and it's going to be transformative for America.

The H20 GPU that was created to comply with the rules was still okay. Then Jensen had dinner with Trump at Mara

20:08

Lago for a million dollars. Then the H20 got banned.

Must have chosen a really bad restaurant. In July, Jensen Juan meets with Trump and is permitted to sell H20s again.

Presumably choosing a better restaurant this time. Juan goes to China.

China says Jensen's GPUs have tracking devices back doors. Nvidia says, "Nahuh." Jensen goes to Washington.

Trump tells Jensen to

20:24

Jensen. Where you stand up?

What a job. He loves him.

He loves Nvidia. He loves Lisa Sue.

And more importantly, he hates Intel CEO. Intel CEO goes to Washington.

Trump likes him now. Tim Cook materializes from the infernal plane to give Trump a 24 karat gold gift.

and gift is in scare quotes

20:39

because that's not what that's called. Then we get to this past week when Trump asked Nvidia and AMD to pay 20% to the US government for sales specifically of the H20 and AMD Instinct Mi308 sales, not all GPUs, as some erroneously reported previously.

Cousins Jensen Juan

20:54

and Lisa soon negotiate Trump down to 15%, and now they're allowed to sell two specific cards that were originally created to comply with the laws before they changed. And somehow everyone walks away a winner.

Except now China doesn't want them anyway. That about sums it up.

Except one last thing that happened as we were filming this. The Department of Commerce doesn't yet know the legality

21:11

of the deal with Tom's Hardware highlighting legal expert arguments over Article 1, Section 9 of the Constitution. That's not our area, so we'll move along from it, but just know that this is being publicly discussed still.

If you want the full details with all of the in between, and we did collect it, there will be a massive

21:28

timeline article on the website. We'll link that below when it's up within a couple days.

There was also recent news about a 15% revenue share between Nvidia and AMD with the United States government for sale of some AI GPUs. This is an old chip that China already

21:44

has and I deal with Jensen who is a great guy and Nvidia. Uh the chip that we're talking about the H20 it's uh it's an old chip.

uh China already has it in a different form, different name, but they

22:01

have it. Uh or they have a combination of two will make up for it and even then some.

But the H20 is obsolete. You know, it's one of those things, but it still has a market.

So I said, listen, I want 20% if I'm going to prove this for you, for the country, for our country, for

22:17

the US. I don't want it myself, you know, every time I say like uh like 747 I want I want Yeah.

for the Air Force. So I just wanted so when I say I want 20 I want for the country I only care about the country I don't care about myself

22:34

and he said would you make it 15 so we negotiate a little deal so he's selling a essentially old chip that Huawei has a similar chip a chip that does the same

22:49

thing and I said good if I'm going to give it to you because they have a uh you know they have a stopper what we call a stopper are not allowed to do it. A restricted is really known as a restrictive covenant.

If you were to only read the headlines, you might think this applies to all GPUs

23:04

and that the ban is over. And then you might think that a market in China or in the US's eyes an illegal market in China would cease to be so illegal.

That's not the case. I released them only from the the H20.

This new 15% revenue share would, if it's legal, and they're not sure yet,

23:21

allow Nvidia to specifically sell the NVIDIA H20 GPU to approved Chinese entities. they likely can't be on the entity list.

It would allow AMD to sell specifically the Instinct MI308 GPU to approve Chinese entities. The proposed license would not affect any other GPU

23:36

that currently does not have a license. The government hasn't made clear yet if Nvidia's partners would also be permitted to make these sales.

That means that other band GPUs, including the RTX 4090, RTX 590, H100, B100, B200, and so forth remain banned. The H20's

23:52

extremely high 96 GB memory capacity would enable large models to fit in memory and actually run, especially with multiple GPUs in a single rack, even if it's slower. That means companies can achieve performance targets by stacking GPUs, which are lower clock and core count, but higher capacity and doing

24:08

Nvidia CEO Jensen Juan's favorite thing. We were trying to solve scale up, scaling up, scale up, scale out, scaling up, scale it up, scale it out, scale up, scaled up, scale up, and scale it out.

scaled it up scaling

24:23

in that way. As for the newer Blackwell architecture GPUs, now Jensen also has Jensen's a very brilliant guy and Jensen also has a new chip, the Blackwell.

Do you know what the Blackwell is? The Blackwell is super duper advanced.

Let's not give Nvidia any ideas for new

24:40

GPU names. I wouldn't make a deal with that.

Although uh it's possible I'd make a deal uh a somewhat uh enhanced in a negative way Blackwell. In other words, take 30% to 50% off of

24:56

it. But that's the latest and the greatest in the world.

Nobody has it. They won't have it for 5 years on the Blackwell.

I think he's coming to see me again about that. But that will be a uh unenhanced version of the big one.

25:12

For now, these super duper GPUs are not licensed for sale. In short, the H20 and MI308 were compliant with the US government's original rules.

Then the rules changed during design and production. They were banned.

Then AMD and Nvidia collectively declared over $6 billion in financial impact as a result. Then the US government said, "Wait a

25:27

minute. We can help with that if you cut us in." It reminds us of our tariffs documentary.

And with this, if I see a tariff of 20 or 40% one week, 50 to 60 a day later, then I see it 120%. Then I see, well, electronics are maybe exempt and in Bloomberg and then I see on Truth

25:43

Social, no, that's not actually happening. I have no idea what that means.

So, I can't make decisions because it's not about it being a high tariff. It's about not having a clue what's going on.

The H20 most certainly is not obsolete, though. It's still very desirable in China, and with a lot of them, they become particularly potent.

Setting a

26:00

threshold for banned hardware should be objective since it can be tested. This is a graph from the Department of Commerce that visualizes the original threshold at which a computing product became automatically banned for export to parts of the Middle East and China without a granted license.

There have

26:15

been some changes since, but back when this was made, the total processing performance score on the Yaxis was used to determine cards in need of a license. The government needed a metric to calculate against, so it created its own.

Accelerators and video cards have a lot of metrics in their spec sheets,

26:31

including memory capacity, which is critical and as simple as a pass fail for certain AI and training use cases. They have hard numbers for memory bandwidth, GPU clock speed, GPU SMU count, TPC, tensor cores, ROSS that are sometimes randomly missing on Nvidia devices.

Teraflops, tops, pedops, gigaflops, gigabits, power, and more. So

26:47

then banning a product could probably be based on some sort of benchmark rather than a random metric from a spec sheet. But the US government, illustrating what an absolute cluster this situation was and now remains, decided to instead multiply one random metric from a

27:02

spreadsheet against the bit length of the operation being executed. Flops or floating point operations per second and tops or terra operations per second are calculated by the company making the spec sheet and aren't a great measure of actual performance.

These numbers are based on both marketing and whether

27:19

we're talking about FP8, half precision, floatingoint 16, single precision, floatingoint 32, double precision, FP64, integer performance or tensor performance or otherwise. So the government says, quote, the rate of MAC tops is to be calculated at its maximum value theoretically possible.

End quote. End quote.

The rate of MAC tops is

27:36

assumed to be the highest value the manufacturer claims in brochures for the integrated circuit. End quote.

So it's not based on a bunch of real world benchmarks of applications or something. The government also references Mactops as the theoretical peak of terra ops in multiply accumulate computations.

The

27:52

Biden administration used this total processing performance or TPP score in 2023 with the government later adding a performance density metric dividing the TPP by the die area in square millimeters. In other words, the government didn't want Nvidia to be able to sell more of a lower performance GPU

28:08

to make up for the loss of high performing parts with multiGPU solutions. There are a lot of reasons this doesn't capture the full picture like sparsity, APIs, different methods to calculate flops and different performance for different applications.

But the government needed a way to define a threshold. So this is what they

28:25

made. The limit was a TPP score of 4,800 exceeded by even the RTX 4090 when calculated using tensor performance.

Now if this doesn't mean anything to you, that's okay because it probably doesn't mean anything to people signing the laws either. Or maybe that's not okay, but

28:41

you get the idea. Something like a higher memory capacity, lower flops performance GPU or even series of GPUs like 30, 60, 12 GB cards might be able to get the work done more effectively if it only needs memory.

Memory wasn't factored in to the TPP calculation. That was the point of the Nvidia H20.

But

28:59

then the absurdity of the situation expanded because the government introduced an opaque memory bandwidth requirement. Per the register, quote, "Unlike with previous export controls, the Commerce Department's Bureau of Industry and Security hasn't issued specific guidance on how much IO or

29:14

memory bandwidth is too much." End quote. But they did decide that there would be a limit.

So to recap, it seems like across two administrations, the United States is creating formulas based upon numerical calculations, and then when Nvidia and AMD tweak the numbers to fit within that box, it's recconing those rules in a guess and check

29:29

process. We spoke to a lot of people in this piece and it's multilingual which made it complicated.

In order of appearance today, the discussions come from Dr. Vinci Chow, senior lecturer in the department of economics at the Chinese University of Hong Kong, responsible for building his department's machine learning servers

29:45

and sourcing the GPUs from middlemen suppliers. Dr.

Zu Ha Fu, a research assistant professor at the Chinese University of Hong Kong and an affiliated lecturer at the University of Cambridge. Zuhao specializes in both computer science and linguistics.

various retail workers at computer

30:00

markets in Hong Kong to get a ground level understanding. Joe, the creative director of product at video card manufacturer Yeston, who worked with us for a separate piece, not this one, but he showed us his factories and suppliers.

So, we've included clips here, as we relied on those tours for an understanding of how GPUs are brought

30:16

into China for assembly on PCBs. An anonymous seller who goes by the pseudonym of Silver based in Shenzhen, Paan, and managing a warehouse in Hong Kong that receives and processes smuggled GPUs.

Vincent, a resourceful fence who lives next to the warehousing and markets filled with GPUs and accelerators. He buys and sells these

30:33

devices in Shenzhen Hua and his job is to know people. In addition to Vincent's cousin, the fixers fixer.

If someone needs a component to fix a broken video card or to build a new one, they go to people like him to get integrated circuits. Mr.

5, a Billy Billbilly hardware reviewer with a specialized

30:48

focus on thermal solutions. Like us, Mr.

five has had run-ins with Nvidia that have ended in a soured relationship over disputes regarding independent reviews and editorial independence. Junga, a renowned Billy Billy uploader, basically a YouTuber in China who runs a video card repair shop in Jano.

In addition to

31:05

repairs, he regularly gets large orders from customers asking him to build them custom unofficial higher VRAM capacity Nvidia GPUs for large language model tasks. Companies in Singapore and Taiwan who act as intermediaries between Nvidia, Nvidia's partners, and companies in China.

The Singaporean and Taiwanese

31:21

companies are able to bring banned GPUs and servers into their own warehouses and resell them to Chinese companies, skirting export controls. We're unable to disclose their identities because there would be punishment from multiple governments and Nvidia.

The Plug, a US-based Chinese citizen who drives around the country buying hardware from

31:38

American end users and resells it to companies in China and Hong Kong. And finally, a special thanks to our translator Raymond Woo of the Black Soda Agency in Taiwan.

Here's the pipeline. If a GPU doesn't fall off the back of a truck in China after it was assembled, or if it isn't a QC defect that

31:54

disappears from the scrap pile, it may instead have been moved by ants to get to China. May bay ban is a linear pipeline.

Each ant in the fire line serves a specific role. It'll help to name those roles to keep everything straight.

Here's what we came up with.

32:09

The source has access to GPUs. This could be as innocent as you unknowingly selling your card on Facebook Marketplace to the plug.

The plug is responsible for acquiring from the original source and reselling the hardware to the China based distributors. In between there's a mule.

Sometimes this is the plug himself doing

32:26

a trip home. We learned that overseas students also regularly return with what are feasibly defended as personal GPUs that they bought at Best Buy and then may get resold for markup and profit.

In either case, the mule gets the GPU into the country either by shipping it without interception or by hand carrying

32:42

it. Next is the middleman, receiving the GPUs and often interfacing with or managing the warehouses that store the cards.

The middleman buys from multiple plugs, including factories, that get rid of rejects with fixable or unimportant QC defects, then sells those devices to more localized distributors. That's when

32:59

we get to the fence, who buys and sells GPUs between middlemen and warehouses to end users in China. Then we have who we're calling the fixer, except this time it's literal.

The fixer is an optional step that may involve soldering and modifying a GPU to improve it beyond its original specification and make it

33:16

more marketable for domestic AI uses. They might also just fix QC defects from factories.

Finally, we have the user. This is self-explanatory.

The user is the demand driver, and often times large enterprises want dozens or hundreds of GPUs or more, while smaller users like

33:32

the university may just want individual units or small batches. With everyone's role named, let's continue.

And now we start our journey. First stop, Hong Kong.

[Music]

33:50

Upon arriving in Hong Kong after checking out the local wildlife, one of our GPU dealer informants slipped us a price sheet for a mix of GPUs that are both export controlled and not. All right, so check this out.

This is a GPU price sheet where we uh just got to Hong

34:08

Kong and I've already been able to find a supplier who works as a middleman between smugglers. And you just pick the GPU you want.

So you can see 5090, 4090, 4090, 48 gig, 5090. Down here there's some A6000, A100, A180 gig.

That's

34:23

pretty valuable. So that's 75,000 rending B.

And so I found uh a supplier where basically the the person who provided this is local and they work with the smugglers. So they sit between the smuggler and the customer to supply

34:39

and source GPUs. Problem is I don't know what the going rate is for any of this.

And so what we're going to do is take this price sheet to our local fixer Dr. Dr.

Vinci Chow at the Chinese University of Hong Kong and see what he thinks about the prices here and if this

34:55

process is normal. So for that, we hopped in a classic Hong Kong cab and headed across Victoria Harbor over to the Chinese University of Hong Kong.

Oh, okay. S.

Okay. Thank you, sir.

35:10

Bye-bye. Bye-bye.

We found Dr. Vinci Chow through a Reuters story from a couple years ago.

Vinci works in the economics department at the university and he's responsible for having built many of the servers and machine learning systems that are in use

35:27

daily. Here's some professors.

Oh yeah, Vinci. He has a whole blog detailing his process, including the difficulty of sourcing components and the ease with fixing problems when so close to

35:44

Shenzhen, such as having custom PCIe riser cables built and basically done the same day. Dr.

Vinci Chow is the right person to start us off with this story and educate us on where to go next. We're here to learn the user's perspective because before there could be any kind of market for it, there has

36:00

to be demand. And that is what the university and organizations like it generate.

Hi. Whoa.

How you doing? Hey, nice to meet you.

Good to meet you. This is Tannon.

Hi. Nice to meet you, sir.

36:16

Right. So, uh we have several options.

Uh we can set up shop here. It's a construction site here right now.

Yeah. So, we are renovating.

An interesting anecdote is that Dr. Vinci Chow actually went to school with someone important at NVIDIA today, the

36:32

vice president of applied deep learning research. I was actually in Brooklyn with same time as Brian.

Oh, cool. 14 buddies.

That's cool. Yeah, I know him.

[Music]

36:48

Wow. We won't be able to do much talking in here because the noise.

[Music] These are these are legally obtained. These are legally obtained.

We got it. Actually very interesting

37:04

because we got these I think uh right before the ban. Uh no, right before Chad uh Chhatti was released.

Okay. Around CH several years ago.

Yes. And we got them quite cheap.

One of them was actually modified from uh SXM

37:21

before everything GPU become such a scarce resource. Yeah, there was a period of time I think right after crypto like crash that GPU was actually quite cheap.

Even A100 was actually quite cheap. We were able to get these for like

37:36

less than $10,000, right? And then check came out and everything like that.

Yeah. Even now like if you go on go on like AliExpress like you still see a lot of 2080 Tai like 2080 Ti being modified.

37:53

So for teaching we use a 3060 but because without modification in the in a gener in generation it has the highest memory capacity. Right.

Right. Uh but it's not on like it

38:08

clearly does. It probably makes sense for us to get the mics on.

Because of the way we shot this, you get to see something you don't normally get to see in our stories, which is all of the conversation leading up to formally filming. Sometimes there's more

38:24

interesting information when we're just spitballing. I guess even universities have become more secretive, right?

In the past, you probably universities have been very open, right? In terms of what like what we're doing, what we what we are capable of.

These days, I think there are second thoughts.

38:39

M some yeah some some researchers might not want to be so being so open about yeah the collateral damage is in right and uh topic like uh on the other hand like it's interesting because you know it's the it's export ban right

38:56

so it's illegal to export right to China and that'll be one of my questions for you without without without permit right but then it's completely legal on our side right There's nothing that says that you cannot buy high-end GPU,

39:12

right? So from our pers I'm from our end like uh as long as we follow all relevant procedures, right?

There's absolutely nothing illegal about for us over here to buy these GPUs, right? So that makes it for a very interesting like um

39:30

environment. Yeah.

Yeah like like like so you has because if you look at the like university like uh documents right I I I think you will be able to find like obviously these universities have been purchasing

39:46

GPS right it's completely legal over on our side right yet this is clearly not supposed to happen right from the perspective of the US government right so um for un so in I As for universities,

40:02

uh it once again it depends a lot on how committed like a particular researcher or a particular unit is when it when they when it comes to crying GPUs, right? I mean if you pay enough

40:17

supply is there, right? Maybe not if you want to build like like a super cluster, right?

Like for research, most researchers are talking about one or two GPUs, right? And if the if they can they have the funding then it's possible to obtain that.

So it just just everything is more

40:35

expensive. That makes sense.

I'm in a department of economics which is kind of interesting I guess for someone in the department of economics. You talk about GPUs but uh yeah we started uh building our own cluster during the coit.

Right. Well not just we but you specifically well me.

All right. So I I'm responsible

40:51

for that. So but it's we because the department has to fund it.

Right. Right.

So, I am the one who plans these things out, but the department funds it. I started building my own desktop computers back when I was in high school.

Cool. Right.

Uh Hong Kong is a small place,

41:06

right? Uh you know, I guess we'll visit that later on, but in Hong Kong, you can actually phys physically visit computer malls where you can just pick whatever parts you want.

And then uh during coit, then we had this idea that uh we have we probably have to try to provide remote access to computing power. The first

41:23

GPUs we have was 1080, okay, that we obtained from the computer mall. $500 each.

Back then I was thinking like that's so expensive. $500 for a GPU.

If you were to ask a local like one of the local supplier how much u let's say H200 is going to cost.

41:39

It's going to be about um 30k US. 30k US.

There's a big premium for data center GPUs, right? So even if you obtain like a aging A1 A100 that's still going to be almost 20K.

41:56

Is anything from Blackwell over here yet? No, not yet.

Not yet. Not yet.

Not yet. It's illegal for a company like Nvidia to sell them into China, right?

But not for anybody here to buy them. Yes.

I messaged some sellers on Alibaba,

42:13

right? And uh they one of them sent me this price sheet.

Woo. Tell me what what do you make of this?

Do you have any H200s in your servers? Well, we want some.

Okay, we want some. I can connect you to the supplier.

42:29

We want some, right? We want like that.

We Yeah, after tax they want 213,000. What is that in US?

Let's see. So, that is $29,700 US.

Uh, yeah, that's about I think they're right. I don't know what MSRP is or if they

42:46

even have one. Uh I think but first of all it's not like there are a lot of companies even in the states they would just like put out a MSRP and they just buy it online or something like that right but I uh I think for the few companies that actually like list a

43:01

price I think it's actually the premium is to as I said it's surprisingly low right right it's almost as much it's just about the same as their MSRP as far as I can tell do you know h how these companies like the suppliers get the GPU any idea?

43:18

Cuz I've I've read a lot, right? Like I've read that um I I came across a Chinese phrase in one of the um by do searches I did and it I think it was like mi banana or something and I the idea if I understood it

43:34

correctly was that a bunch of people kind of move one GPU. So these GPUs are almost certainly moved one at a time, right?

So, it's very hard to get like a full HGX system, right? These days, most of the vendors would recommend getting

43:52

the PCIe versions of the GPUs because it's so difficult to smuggle in a food HGX system, right? Okay.

Yeah. Um now if you want to actually exist them for example the companies that have ways of

44:08

obtaining them would recommend that uh they be in stored in a different country. Okay.

Right. So basically you can like uh acquire HX and then have it installed in like Southeast Asia.

Okay. So meaning it would actually physically somewhere else and then you remote into it.

44:23

Right. That's what they would recommend if you want that.

If you actually want it on premise, right? If you want it here, then almost every one of them would recommend that you just get the PCIe version.

Okay. So, that probably just because it's easier.

It's easier when you put in an order to buy something.

44:38

Um, is it already guaranteed here at that point or do they have to get it? Uh, depends.

Okay. Some companies have limited stock.

We're talking about probably maybe a dozen, right? So, but if you want more then you

44:54

would have you Yeah. It would probably have to be like ordered and a deposit or something.

Yeah, shipped in from somewhere else. Any idea what happens if they get caught?

I mean to your money, you know, like Well, we don't pay. Okay.

Until Okay. Actually, we have to pay

45:10

deposit, right? But then, uh I presume if they cannot deliver, you maybe get it back.

We will be able to get it back locally since uh I mean, we're not doing anything legal. we can just go to the court and suit them.

And

45:26

do the local governments even care? I mean like cuz I guess that was one of the things that was unclear to me where if it leaves the US like let's just say and and I as an example if Nvidia decides you know what we're just going to ship it straight from

45:41

Santa Clara California to Shenzhen or something you know that would be a problem right but on the receiving side do they care you know to like Hong Kong on the re on I would say the receive well I would say no that we don't pro well we probably don't care like um I don't

45:57

think the government is going to obtain these GPUs now that there's an export ban. I don't think they're going to do that.

In Hong Kong, we have like basically the government essentially funded two H800

46:13

superport, but they are both legally obtained right before the the ban. So, and since then um I there hasn't been any news that they're going to expand uh their their capacity.

So I would say the government probably not going to do it

46:29

itself. I guess the only thing they care about is if someone brings it in, they pay the import taxes.

Well, but there's no import tax in Hong Kong. We are free trade port.

Okay. You can pretty much bring whatever you want like like except like well I mean yeah things normal things like firearms like you can't do that, right?

But

46:45

otherwise, yeah, we are free trade port. Okay, that makes a lot more sense then because I think I read that Macau is the same maybe.

Free free trade port. Free trade port.

Yeah. Okay.

Interesting. But I'm pretty sure these GPUs don't go straight from Hong straight from the US

47:00

to Hong Kong, right? They would go through they probably go through some San like right is some like third party country like country somewhere you can actually legally export those GPUs to.

Singapore comes up a lot. Yeah, Singapore comes up.

Malaysia is another place that comes up.

47:15

Right. I saw have you heard of uh I saw a story from the economist actually uh where they said now there's like they they referred to it as I think data smuggling right they're talking about bringing data from countries with the export ban to

47:30

Singapore right to have it processed locally. So I think it's it's similar to what the some of these companies have off have offered us to for consideration is that uh given how hard it is to bring in a large system,

47:45

it's actually easier either to like either to fund a system in Malaysia or Singapore or you just rent capacity there, right? And then just remote access, right?

I mean, I guess if the reason why you actually have to like take physical hard

48:04

drives back and forth is due to the size of the data, it's going to be hard to transfer transfer data remotely. But uh but otherwise the idea is the same right given that you cannot bring the system in just have it somewhere you can access.

That's how you Steve. Hi, I'm Steve.

48:20

I'm good to meet you here. Yeah, good to meet you.

What department are you in? Uh I'm currently working in the linguistic department.

Okay. So my focus is still doing some programming and running on GPUs.

My background is computer science. So I

48:35

need a lot of GPUs, right? So you've already got the computer science background, but then Yeah.

Yeah. I got a computer science background in in this university.

Yeah. Do you have your own server cluster?

Like is it separate from the econ?

48:51

Yeah. Yeah.

I think they have the best one. Okay.

This is the best one in this university. So who who built the uh the one for the arts department.

Uh actually we use the university's GPU. So university have a central GPU and

49:08

everyone can use that if you buy some GPU and they can help you to manage that GPU. Okay.

So that that's how we are using but I believe they they have much more GPUs than the university GPU richer.

49:24

GPU rich. Yeah.

Yeah. What um what are the things you guys use them for normally?

Uh just training and inference uh large language models. Okay.

Uh like for me my research is about uh Lang model.

49:40

Yeah. So if you want to train a model, teach the network model to gain more knowledge, you need to fine-tune.

And before that, maybe few years ago, we just trim from scratch. But now it's not that popular, but we still need GPU to

49:57

fine-tune models. Okay.

So this is why we need that. What um what are the common uses for the models themselves?

Like do students use them for research or what's the Yeah. um if you are in the university is a meaning for research.

Okay. So you know currently because a large

50:14

bank model can do a lot of things a lot of research department will focus on using that. So a lot of research will be focused on how can language model help them for example like cleaning the data and doing some bical inference or just

50:32

uh imitate human to do some research. Okay something like this.

So you do you guys ever end up working with each other? No, we haven't.

Well, we our two faculties are in talk of co-op collaboration but uh as you said like a lot of demand really coming from uh researchers

50:51

that are using these models for things like uh processing large amount of text data. Right.

So like in the past what we would do is that uh these re uh researchers would hire junior research assistants to go through

51:06

text data manually and it's a very expensive and slow process right these days all you have to do is to feed the tax data to OM and that really drives an explosive growth in demand for these OM and because open nether neither open AI nor

51:24

and for is interested in serving Hong Kong mm which is part of China. Right.

Right. So that means that there's a lot of demand for onrem solution.

What? Um, so in linguistics, do you use it for just as an I I don't know a lot about

51:40

linguistics research, but uh well the Dwan Laure is a she's a linguistics researcher and um so I know one of the things that they do in the University of Colombia in New York is they'll scan old documents from

51:56

like the medieval ages from a thousand years ago, right? Cool.

And they try to convert it into modern English. So what what kinds of research I guess to give me a better idea.

First one h if you want to do research on

52:11

linguistic you will be very interested in what the language model what how the language model are thinking right. Okay.

So we try to explain what happened inside the language model and another uh research direction maybe is just uh use

52:26

language model to processing the data. uh for example you want to know how the children learn the language okay children learn uh bilingual or many languages so how how they do it so you can use the length model to imitate the

52:42

learning procedure so this is one of the uh possible direction so there will be a lot of things to do what um what kinds of GPUs do you guys use uh currently for our group we have uh A40 I guess Yeah. Okay.

With with 40 GB

53:00

memory. Maybe I forgot.

Sometimes we have we have a 6,000 m and uh then some uh gaming GPUs RTX series, right? Yeah.

Gaming GPU is much faster actually. Is it really?

Okay. Do you have any like

53:16

5090s yet or 4090s? I I have used the 4090 but not 590.

I think memory is the most important part for researchers because if it's running slow then you can wait for one day or two days it doesn't matter but if the

53:33

memory is small then you cannot do anything you cannot put your model inside the GPU it just doesn't run at all yeah so so I need larger memory so the speed is okay I think currently it's fine do you guys ever get bottlenecked by uh

53:50

the availability of the systems. Well, it does happen, right?

So, like as much as we have, as much GPUs as we have, we talking well, we have like 30 something like 30 top of the line GPUs. Yeah.

If we want to load say uh DCR1, right? That takes probably 60,

54:08

right? That's half our capacity, right?

And if so, if someone like one of our teachers or what would say, I need deepseat R1 for two months, right? then it would take a bit of planning because we have we would have to see if we can balance the need of other users.

It's not going to be enough

54:24

if you want state-of-the-art model given how big these state-of-the-art models are. I think all researchers would love to scale up what we do but uh the reality is uh it might not be possible to get the hardware to support

54:40

what we want to do. If if you had uh and I know you don't but if you had unlimited budget right would you be able to get as many GPUs as you want or is there still a supply constraint you know like in terms of buying let's just say somehow we obtain like

54:57

I don't know millions of dollars of funding right even if the university is providing that right there's still the problem of how of the export ban right um before the export ban as um it can be done Right. Uh the there's another

55:13

university, local university, HUSD, they obtained a full H800 super port. Yeah.

At a considerable cost, but the university was able to fund it and they managed to got managed to just get it

55:28

right before the ban. Right.

So that's like kind of a once in a lifetime opportunity opportunity right now, right? Yeah.

From today's perspective, right? and they did it and they benefit from it.

Their researchers get a full 400 GPU cluster

55:47

to work with, right? It's universally wide, centrally funded, right?

In CK, yeah, it's not some it's just not something we can do right now, right? Yeah.

I think in CK is managed by each group, right? I believe the computer department also have a very big one.

56:02

Yes. So, so here you know that's but that's our university.

It's a bit decentralized. So, yeah.

And uh part of the problem as I've mentioned is the fact that with the export ban uh departments are not necessarily willing to publicly list their cap their computing

56:18

capabil capabilities. I think the next thing we should probably check out the server.

Well in a university the working language is English and then Mandarin right. Okay.

The university's name I think. Oh that's okay.

We have to clarify because people keep people always tell

56:35

this question. the Chinese University of Hong Kong.

We teach in English. Okay.

Chinese means Chinese culture, Chinese heritage, not Chinese language. Okay.

Right. That's what a lot of the exchange

56:50

students was confused about. They thought like we teach in Chinese.

No, we teach in English. Okay.

Interesting. So, this is your cluster.

Have you seen it before? Uh in some other places.

Oh, but never his. Yeah.

Never here. Oh, never in Hong

57:05

Kong. Yeah, as I said, they they're being full.

They're quite well utilized. This is actually not part of the cluster.

This is our noncluster stuff like web servers and stuff, but uh yeah. So, these two racks are the cluster.

Did you have to do anything special for the air conditioning? So, I mean, as I said, this is kind of

57:21

like congestion, right? The thing is, uh these GPUs are very hot, right?

A single 8 GPU system can put out like 4,000 W. Oh, okay.

Right. So, it's like like a hair like like two haird dryers.

So, when you say when do you say a

57:36

single HGPU system uh how many does each rack or each blade have in it? So, like a 4U unit these days like these ones like uh this one like and the and this one these ones these are all HPU systems.

Okay. Right.

So, for you

57:55

uh is each of these all different configurations? So now our cluster was built up over like almost 5 years, right?

So that's why we we have some really old systems that come from 5 years ago and we have then we just keep adding things. That's a bit different from let's say uh some like uh the like

58:11

say super port which is purchase in one go then everything will be the same. Right.

Right. Here we just add keep adding capacity and just uh leave the old components until they gradually not work until they're not useful anymore.

Not useful anymore. Um

58:27

did you have to do special power design power in Yes, you cannot just plug these uh GPUs into a wall socket. It would blow the fuse immediately.

So we we would have we need uh well two free we have two

58:44

freephase power specially uh installed in this room. So they have these massive plugs for power.

There's one there and then there's one over here. And then you can actually see they're running flex duct work.

And you can see it shaking at the bottom of that one in the back where

59:00

it's actually shaking because it's pulling in all the exhaust from the servers. And they're piping that up.

So I think that's the return. That might be the front side.

We'll take a look. But they're basically circulating it with the flex duct work.

Crazy. And that looks like maybe more.

59:17

I think that's more AC going up to the mini split up top. I guess how many you know A100's or H100 right we we have something like like 30 A100 class GPUs and then we have a bunch of like 3090s 3060s right so in total we're

59:34

talking about I don't know 50 or 60 GPUs okay right so yeah the the top of the line GPUs are for teachers what are the what are the 60 class cards for like 30 60 is just teaching teaching it's for teaching so uh we we have we have these uh undergraduate

59:49

machine learning courses And we let students low hugging face models. These 3060s are for them.

We don't want them to occupy the A100. And 3060s have 12 GB of memory.

That's that's more than enough to load some of the smaller. The good ones have 12.

They made another

00:05

one. Right.

Right. Right.

Right. We make sure we get the 12 ones.

Right. So yeah, that's so that's sufficient for loading a small model that they can work on.

Right. Does memory bandwidth matter a lot?

It matters. So to better utilize the resources um we offer the options of for each researcher to load their own model

00:21

but we have also this centralized like our own local jetp like thing we have use open web UI and with like uh with vom as a back end so we we offer like just like this preset uh system where we can serve like germer

00:36

uh we can serve DC and the and when it comes to getting high throughput uh yeah bandwidth would matter because for the export control the government the US government does a calculation right for some amount of points like they made up a system and uh memory bandwidth is one of the

00:53

components of the formula. Yeah, they're getting smarter, right?

Because the band there has been multiple versions of the band, right? The very first band was targeting the uh Envy link, I think.

Okay, A1 the A100 to A00 N that has a little effect. It's a bit slower,

01:08

but uh it's still okay, right? And then I think they switched to targeting the the flops.

Yeah. Right.

That I think is there was a certain amount of flops you could exceed. Then I think depends on workload is I I think it didn't work as well as they might have hoped for because the H20s still still sold really

01:25

well. Well, I mean that's what's so interesting about it is the governments at least the US government hasn't really caught up to what these are, you know, like and so so they don't know how to ban it, I feel like.

So they just kind of pick a number very fast, right? Because if you go go

01:41

back like five years ago, everyone probably would think that like yeah flops is a very important metrics to consider. But turns out when it comes to loading large models, what we all care about is how much memory, how much VRAM you have, right?

You simply cannot load a model if you don't have the VRAM,

01:57

right? So now H20 becomes a very attractive option because what's the spec on the H20 again?

D. So Hract H20 has drastically lower flops, right?

But then it actually have it actually has more memory right than the original H100.

02:13

What's the capacity? I think it's 94 96 versus the 80.

Yeah, the original version 80 is 80, right? So it's actually in some sense more attractive.

So everything you guys have named so far it's all Nvidia. So are there any non Nvidia GPUs like in

02:31

the university cluster? I think currently I didn't see any nonvidia one but I think it's promising the AMD one currently pietorch also still support AMD one I think in the future it will be

02:47

another choice the instinct mi uh AMD MI GPUs uh I didn't remember yeah we what about uh Intel like the because they just started making the B60s you know is that interesting at all like So uh I mean we we always like to

03:06

explore options right more comput more more computing power. We actually expressed interest in AMD and we asked around like to the outcoming suppliers and they said the risk is too high for them right because they have to support it and uh they had I think there's one

03:23

supplier who told me they had tried to sell an instinct GPU in the past and then the and then the customer returned it because uh they couldn't figure out how to make it work and and then they realized that they don't really want to carry the risk of having a GPU that people are going to return to so they would rather just not

03:39

deal with. Do you know any AMD GPUs that also are export controlled?

I think the instinct line is also covered under the cover cover export, right? Yeah.

I mean, I guess that's also part of if you're going to go through the hassle of like of smuggling something, you better smuggle something that people actually

03:56

want, right? Yeah.

And well, is there is there any support for it? If it's if it's a smuggled device and you have a problem, what happens?

You know, right? uh like if a if a MOSFET blows up you know so um now we obtain all these under like

04:16

like under following university procedure and part of that is that uh there should be warranty right okay I mean now the the supplier is clearly going to promise you warranty right but uh they also indicate that now for the PCI version they can they

04:32

are pretty confident they can they can support it because I think they should just give then they just change it for a new one and just Yeah. move try to move that one out.

Yeah. One at a time.

Right. Right.

But if it's a HGX system, even if they manage to get it in,

04:49

they don't promise anything basically. Yeah.

Right. I mean for contractual reasons, they will promise warranty.

Right. But uh whether they can actually do it, they can't actually guarantee.

Okay. Right.

So actually for we have a legally obtained uh HGX 800 system that we got

05:05

before the ban and uh as my friend know after our initial I think 5 year support contract uh ends there's it's not possible to read there's a 4U system with eight yeah eight 800 inside.

05:20

Do you ever physically go to pick up the GPUs when you're arranging from a supplier or they just ship it here? They usually ship here.

So it's just like totally normal commerce basically, right? They just ship ship ship here by It's so interesting because I feel like when you think about like this concept of uh like the black market GPUs and um

05:40

earlier I said I I found online a story that talked about mahi banana. Oh, and uh you think of it like that, it seems like when you go to buy a GPU that's banned, it feels like you would

05:56

be going to a dark alley and handing him cash, right? And he hands you a GPU, but it's not how it works.

Yeah. No.

So, they just ship it. I think it's good that way.

It's good for personal usage. It's not quite good for a very big cluster.

06:11

Right. So, so does the if it's that sort of easy where it just ships to you, does the ban seem effective?

Like as I said, as I said, if it is effective in preventing the building of a very big

06:28

cluster, right? It's just not possible to get 100,000, okay, of these GPS like we heard about some of the other companies having, right?

Yeah. And I think I think I think there are like people from the company who recruited that idea.

Right. We don't really we don't really have that many.

I

06:44

think that Right. Right.

Why do you think the US government cares so much? Like what is it they're trying to prevent?

I I guess they just want to delay the speed of other countries of training the model. But seems uh I don't know their ban is

07:01

very weird actually, right? Yeah.

Do you feel like the ban is it targeted at like military uses where they're trying to prevent that and then it just happens to affect you guys as well or are they they targeting you think just companies in general?

07:17

I I guess it's in general every company every people will be influenced by this. M so I guess just delay the speed of research just like like my experiment

07:33

for example if I can have better GPU I can finish in one day but now two day or three days that's fine just wait a little bit you get the same I don't think from the US government's perspective there's this distinction between like

07:49

Chinese academia versus Chinese military versus Chinese commercial I think no I don't think they're they really consider There's a distinction, right? Yeah.

Just total Yeah. Very general band, right?

Is um I know Huawei has the new Ascend,

08:04

I think it's called. Is that interesting at all?

I mean, does it seem like that's promising for you guys? Uh Huawei Huawei GPU, I think they built that for many years and they also trained some language model based on that.

So currently uh seems uh no no

08:23

friends around me are using but maybe in the future they are trying to try that if the they still cannot get the Nvidia one. Do you guys need CUDA for your stuff?

Oh yeah definitely if you want to train the model you need CUDA

08:38

right? So I I feel like uh we can still get by because we we as I said we're not really needing that many GPUs.

Right. Right.

Yeah. So there's still enough supply Nvidia GPUs supply around for what we need.

Right.

08:54

Right. But uh if we want to scale up then Nvidia is just not possible right now.

So probably one way eventually like something like Huawei Ascendants would be. You said you have a an interesting

09:09

GPU that was modded, I guess. Yes, we have several GPUs like like as I said like this is one of those like a reference board.

Yeah. Uh and uh uh RTX 3090.

09:24

This is a 3090. This is a 30.

This is a 3090. I think it's very Chinese specific, right?

I I don't think you can actually They were not popular in the US. You could get them, but not many.

Yeah, because they're noisy. They're really noisy.

And the only reason to do to get these are because you want like multiple

09:41

of them. Yeah.

Packed closely together. They designed for that purpose, right?

So they have these kind of slots. One of these I think are like a normal actual PCIe version of A100.

And this one is actually a molded version, but they use the same casing.

09:58

Okay. So this one is actually molded from a SXM version.

So if you if you install it and take a look at the uh Nvidia like uh system information, this one will show up just like this show as SXM SXM, right? But they look the same.

Um

10:14

this is actually a very interesting question because as far as I understand these this is possible because they these these GPUs are assembled in China, right? These GPUs are assembled in China.

So they apparently I guess they have spare casings, right? Well, so here's a question for you.

10:32

If uh if a band GPU is assembled in China, then what's to stop them from just taking it? That's actually very good question because um as far as I know, even the H200

10:47

seems to be assembled in China. Yeah.

And I have no idea what's preventing these GPUs from falling off of the line. Line.

Exactly. So, yeah, I don't understand how it works at all.

How is the band even going to how is going to band is even working like the box of

11:04

these GPUs at least they are manufactured in China right right so I'm not sure how actually that works I mean it's yeah it's like there Nvidia is not allowed to sell it to China but they can make it here China and then the intent I guess is they ship

11:21

it somewhere else and then they can't ship it back right but it just seems like if I'm the factory it seems like there's a lot of money in So, we have some QC defects, right? Right.

So, my guess is there must be spares. I don't know.

11:37

Right. Spare spare SXM like modules, spare casings, and then somehow these spare parts just get assembled into a complete GPU and get sold.

Yeah. So, currently it seems like Yeah.

Like that this whole GPU manufacturing thing

11:53

has the flavor of that, right? the all these high-end GPUs are still manufactured in China, but they're not allowed to be sold in China.

Yeah. And somehow the US government thinks that's going to work and somehow the Chinese government also allows that to happen.

I have no idea how actually that whole

12:08

thing works, right? That's so strange cuz like the GPU silicon is made in TSMC, which is all over Taiwan mostly.

Yeah. So, they make it there, they ship it to the factory in China, I guess, and then an SMT line must populate the board.

Right. One of the GPUs we have, one of the

12:25

A100s we have that are molded mode. Um I think uh the N one of the Ning doesn't work.

So my guess the my the explan the explanation is is probably failed failed P QC right and then it just gets assembled and Right. So

12:40

and so then you have two out of three I guess NV. Yeah.

So yeah, you just have Yeah. So what?

Yeah. So it's just going to be slower.

Right. Right.

But for a bargain price Yeah. Right.

Is it cheaper than uh like a fully functioning? Yeah, like I think uh I think it Yeah,

12:56

it was cheaper. Okay, it was cheaper, but uh doesn't make a difference if you're just using one GPU, right?

So that would be a So it would be a bombing. Interesting.

Yeah. Do you think Nvidia

13:11

knows that all of this is happening? I would be surprised if they don't, right?

Yeah. I would be surprised.

I would be really surprised if they don't. I see.

Right. These are very expensive items.

Yeah. Right.

I would imagine you would keep track of everything, right? Yeah.

I'm not I don't know. Is that

13:28

I'm don't Yeah. It's hard to know how what they can do, how what what they plan to do with all these defective parts.

But I'll be very surprised that no one have ever thought of the possibility that if it's something so valuable, someone would come up with a use of

13:44

even a defective one. Effective ones.

Yeah. Yeah.

Interest. Yeah.

Yeah. Like uh the AG like the AG GPU servers like we could pull out one of them if you want if you want to see what's if it doesn't disrupt you.

That would be great. One of them and then

14:00

we have another pull out one cuz it seems interesting. So So this is one of our older ones.

this older one because um this one is not it's assem it's basically assembled

14:15

completely from like retail parts. Okay.

Right. So this whole system is around obtained around uh when the time when chachi beauty came out.

Right. Yeah.

So this is the time when this everything you can get from AliExpress right the case.

14:31

Right. So you bought all of this just on AliExpress.

Yes. Everything is from AliExpress.

everything express like CPU, RAM, this the by the part that's by far the hardest to get is the power supply. I saw you write about that in your blog,

14:46

right? Um what makes it difficult because now it's very hard to get high quality high power server grade power supply.

this like you need in order to support four H uh 4

15:02

A100 you need like uh 2000 2,000 watt power supply and you need you need these kind of server grade redundant power supply and there's just no demand in the re from a retail customers for these kind of GPU so for these kind of

15:18

power supplies this is right if you buy this case mostly they come with something like a 800 watt power supplies and we need 2,000 so that's the hardest part All right. Is this a redundant power supply?

This is a redundant power supply. Okay.

So, it's got a fail over the CPU.

15:33

What are you using for this? This is a uh Epic 776.

Epic 776 that you can get from AliExpress. No problem.

If you need to. Um and then the board is from Srock.

Yeah. Yeah.

ASRock board. As rock pack.

So, it's like a the server line of does

15:51

have like seven PCIe slot. Uh-huh.

But then in order to because we need one for we want um infinite band network. Yeah.

So we have these cables like specially made to the Oh custom made to the length we need. Okay.

Right. We need it to be this long and to

16:08

be to support the infinite band card. Is that that down there?

Yeah. This is the like So that's the infinite band card.

Right. That's a 200 that's a HDR 200 GB card.

Is is all this stuff uh pretty high priority for the university at this

16:25

point like on the I mean technology side they are they invested in the purchasing and you know I think uh I think the university is realizing how important this is right that's why we have this brainstorming session right yeah I guess university will begin to buy more GPU and try to find some

16:42

strategy to share with the whole staff members right yeah it's a good sign cool but it just start Right. It's Yeah, like uh I think we would try to catch up quickly.

Yeah, we try to catch up quickly. Very cool server room.

It's impressive.

17:00

Why is it just happily providing with their way and uh I mean I'm not encouraging working 24/7, but like customer support in China is 24/7, right? Not designed for Hong Kong.

17:15

Not designed for Hong Kong, right? They don't sell it anymore.

So Hong Kong is a free trade port I guess. Yeah.

But China mainland is not. Oh, except Hain, right?

Except Yeah. Hand and some part of Shanghai

17:31

also makes similar policy. And that just means no uh local tariffs, no import taxes.

Yeah. Right.

Right. And as very as relatively simple paperwork, I guess.

17:48

Okay. All right.

This is where we call the university mall. It's where we hold our like a like a commencement.

Cool. Yeah.

The graduation ceremony in here. Yeah.

Yeah. Wow.

This is nice. Yeah.

18:03

Holy So, that's the science park of Hong Kong. And you actually see one of the our unicorns uh a little bit.

Sense time. Sense time.

What do they do? Uh facial recognition per Yeah.

Yeah. I would say AI,

18:19

AI, AI, but uh they they they they kind of built their business on on facial recognition, CCTV stuff, right? Chinese government is a big as a big customer, right?

Do you feel like you guys are locked in at this point to Nvidia because of CUDA?

18:35

Like does I guess to rephrase it, is there a difference in what matters more to you? You know, the power of the hardware or the power of CUDA because both are very good.

So I would imagine if someone put in the enough effort so that non Nvidia

18:53

hardware runs just as well as Nvidia hardware in PyTorch. Uhhuh.

And uh then the VOM and stuff then uh would not be too hard for us to switch. But I guess that's harder to do than to say.

Yeah. Right.

Because someone has to put put in

19:08

the effort to make the models run smoothly. There has to be a good reason to switch.

Yeah. That part that part is hard.

Yeah. Only problems is if you switch to other devices, you will have a lot of bugs, right?

Some bugs you cannot fix. You need the whole community to help you.

So, this is

19:25

a very slow procedure. Definitely the CUDA and the Invidia country is the best one you can choose.

Right. After this discussion, Zhao parted ways and Vinci brought us to one of his favorite tech spots in Hong Kong, the Golden Computer Center and the outdoor

19:41

tech flea market next to it. We actually have a whole separate video looking at some of these places.

[Music]

19:58

[Music] As for this trip, we asked him if we could go find some supposedly banned GPUs like the RTX 1590 except available in public where in theory you shouldn't

20:14

be able to find it. The hope was to find a shopkeeper with some ground level knowledge or consumer level.

Everyone wants Jensen's flavor. And you do you still frequently go to the market?

Not as often as before. Yeah.

Right. So mostly because I've

20:31

mostly switched to using server components, right? And you can't you can't get these components in these markets.

Our cluster is pretty much exclusively epic, which is rare when we start building out 5 years ago. 5 years ago, there's probably still a

20:47

lot of Intel, right? Everyone is still buying Intel and we started buying AMD.

So right now in a on in a cluster uh we have the last generation Gino CPUs and I think yeah we have 256 score across two CPUs I my guess is these GPUs are probably

21:04

smuggled in from either yeah from Southeast Asia in that case unless those countries try to enforce right the ban but I feel like most of them probably wouldn't really care you know it's not it's not their ban

21:20

right it's not their banana And uh why why why cross the Chinese, right? That's a good point, too.

Yeah. Yeah.

I guess they've got two choices, right? Right.

Which is why Biden bas off America or B off China.

21:35

Right. That's why Biden had this right idea where the Biden administration had this idea where you're going to have to separate the world into different zones and some zones are trusted and some are not.

Ah, right. But then this this is scraped, right?

this. So now it's back to I guess

21:54

back to one by one by one. Yeah, back to one by one.

Right. And I I I haven't heard anything about like the countries having to prove themselves to be trustworthy anymore.

[Music]

22:09

It must be frustrating for Nvidia on some level cuz I'm sure they would they would be they would rather much that they can just sell what whatever they want to China. If they come up with a product that is in high demand that means it's useful for AI and if it's useful for AI the US

22:25

government wants to ban it. Right.

Do you think the export controls are having their desired effect like for the for the US?

22:40

Yeah. Well, yes, in terms of stopping China from building a comparable GPU cluster to US, right?

Right. So, it's really just about what you keep coming back to, which is if you want to buy thousands of them, it does make it a lot harder.

22:58

Because I guess ideally if you want to buy 10,000 GPUs, you probably buy it straight from Nvidia, you can. Not even that many.

Even if you're buying like say 400. Okay.

All right. Like let's say a super port, right?

Then the Nvidia is likely going to be

23:13

directly invoked. I do wonder, you know, it seems like if you make it hard enough to get US products, um eventually

23:28

you would think China would just dump its own resources into helping like a domestic, you So what China is already doing that China Chinese company is trying very hard to come up with competition a competitive GPU.

23:45

All right. So where are we going?

What's this place called? This is uh called this is this is a shopping mall called Dragon Center.

Watch your belongings. I did lock all the doors, right?

That's just me. I mean Okay.

I asked him

24:03

if it's safe to leave it in the car and the first time we say it see your belongings from the police. I mean they're part they're priceless too.

I mean transparent. Oh this one it's a parallel import.

24:18

Parallel import. Where is it?

Oh. Oh parallel import.

What does that mean? That means not imported by an official distributor.

So like kind of uh what it practically means is smuggling.

24:34

Smuggling. Smuggling.

Yeah. All right.

So what um what are their prices? So the cheapest one cheapest one says it's a so that's a big price difference.

Yeah. Almost 2x.

24:49

But the ROG ones are always expensive. Yeah.

But they it's not worth that much, you know, like the the price difference. I mean I guess when it supplies it's one.

Sure. Yeah, one that someone maybe brought in from another country.

I guess that's interesting too, like if they

25:05

have to account for cost of like plane tickets and stuff. So that astral is $5,223 US.

So it's the cost of even it's the cost of like almost two 5090s at home for us. It's even in stock.

The cheaper one, the

25:22

cheapest one is I think it's about 3,000 24. So yeah, 3,57 USD and that's not like that's pretty close to the average price in the US.

The average price is 3,50 right now. It's actually like the same.

I see they have D

25:40

the 5080 I guess is still okay. It's not band rate.

So yeah, like I think it's if I remember correctly, the 5090D uh wasn't that expensive. So it's probably the price has gone up, right?

25:55

Yeah, cuz I mean it's like more expensive than the cheapest denied. Yeah.

Interesting. It's possible that um it still comes with warranty.

Mhm. Like these ones probably have no warranty at all.

Oh, no. I mean 3 years Hong Kong warranty.

I think what that

26:10

means is that the store promised that they're going to help you, right? But I think these ones that 159D, they were imported when back when it's legal, right?

So they still I think they actually have official warranty. That's actually Yeah, that's a good point.

I wonder if you bought one when it was legal if you still get support.

26:27

Yeah, I think that's the case. Yeah, at least that's the case with our data center GPUs.

Okay, that makes sense. Oh, there's a D.

The D is uh banned now. Yes, they can come up with DD or D2.

Yeah, I think the price has gone up in like

26:44

now that Yeah, it's it's going to be better. Yeah, it's interesting.

You can buy a 5090 FE here. Yeah.

I mean, but it's supposed to be banned, right? Yes.

But you can just buy at retail in a in a store. As I said, it's not illegal to buy them,

27:00

right? But I guess what surprised me is how easy it is to buy them.

The right price. 36,000 HKD.

That's pretty expensive. That sounds expensive.

That's like 4,000 over $4,000.

27:15

$4,600 US. Wow.

See? Yeah.

But you can get it there. So I guess the main impact then of a band is the price goes up, right?

Yeah. Any idea how they get their 5090s?

Same way you do. Same way how the how the high-end GPUs

27:31

get moved in. Okay.

Although I think for these ones there's a chance that uh it's actually directly moved in from other countries. You think uh one by one?

One by one. One by one.

I think I I I've heard that some um students would do that.

27:47

Okay. like, you know, buy one from Best Buy, right?

Move it to move it back to a third country. No, just straight back to straight back.

Okay. Straight back.

I mean, if if it's with a premium like that, do you happen to know is Yeah. Well,

28:02

yeah. I mean, that premium is great.

They can double their money, right? Is there um do you happen to know is there any law against bringing it in for personal use in Hong Kong?

No. I mean like from the US.

I mean like if customs stops you,

28:17

you know, but I guess they would have to find it in your bag. So they'd have to search your bags.

It's pretty unlikely. Yeah.

And you would you you're relying on the custom officers knowing that this particular moto Good point. Yeah, that is a good point.

And then caring. Caring

28:33

like if Yeah. Okay.

I really don't think that's going to happen, right? I guess that's how they get in one by one then.

for uh probably 550 uh battery in China. What's that?

Uh we we only get a uh 1590 uh in stock.

28:54

So the 1590 the founders edition maybe not in stock. Yeah, we direct directly order from other countries.

What's the most popular card? Uh 90 no uh always come from Australia and uh the second one is Thailand.

29:09

Oh, so the 90 normally comes from Yeah. Australia and Taiwan.

Yeah. Can you buy bulk a lot of them at once or is it just one at a time?

Uh very tricky is that uh almost you can order from mainland China. Oh

29:24

yeah, you can buy a lot. Oh, mainland for 90.

Yeah. Okay.

Recently we got the uh app from Facebook. Yeah.

That are selling in monk. They will provide lots of uh 90.

Okay. Is it expensive?

29:42

Uh, very interesting. Uh, 90 only.

Uh, sometimes cheaper than 90D. Okay.

Wow. Do you guys ever sell like RTX 6000 or is it too?

Uh, because Rud 6000 we got ADA edition.

30:01

ADA. Yeah.

Yeah. ADA edition.

So, uh, replacement will be ADA edition. But no blackwell.

Uh, yeah. Otherwise, uh also from uh 99, right?

Okay. Interesting.

5090. Do you have to get it

30:16

from a different supplier than 80? Some models?

Yes. Some model?

No. Okay.

Because uh the kind problems uh the distributor only can send back to the original page. Got it.

Yeah. But local uh distributor uh they can do

30:34

DOA or RMA. Okay.

Yeah. because uh the US problem.

Yeah. Uh the like uh 4090 or 590, right?

If they got problems, they cannot get back to uh China.

30:51

Do you do you have any idea how uh the mainland suppliers get the 5090s? No idea.

No idea. Yeah.

Because uh too much supplier. Yeah.

they are doing trading they can uh

31:10

transfer the other regions products to Hong Kong. Okay.

Yeah. So we have no idea what way they come but we can know uh which region like Australia Taiwan Taiwan get lots of uh stock

31:27

Yeah. in in because uh like last generation like 40 90 Yeah.

the price risk very high. Yeah.

But uh I don't think that in uh 50 generation they can make lots of offers, right? Yeah.

But uh it's not seems like uh what

31:45

they thought.

32:01

It is still funny to me just like the how how much it flies in the face of US export control that you can just walk around in a public facing mall and buy a 5090. Now given that it is very

32:16

expensive, right? But it's not like you just can't buy it anymore.

So I guess if you really need it, you can get it. And I really don't think locals think much about buying the GPU GPUs other than price, right?

It's expensive. But

32:32

uh the fact that it's there's an export ban, I don't think cross the mind. They don't really consider it.

Yeah. With that, we wrapped our first full tour of Hong Kong.

The city was impressive and filled with character and culture and GPUs apparently because one

32:48

of our next stops sent a message the night we were planning to hitch a ferry to Shuko port in Shenzhen. He told me they had just received a batch of GPUs to their Hong Kong warehouse and that they were in route to their Shenzhen facility and that we could buy a GPU as soon as tomorrow as long as we wired him

33:05

money immediately. That seems like the responsible thing to do.

And because it's fully above board as a buyer and because he also texted me a photo of an RTX Pro 6000, which was very intriguing since it's Blackwell, we decided the best way to learn about buying export controlled GPUs in China would be to

33:21

just do it. So far, all the contacts we spoke with were aware that some form of factory repurposing, theft from the line, QC reject repurposing, repair, and actual byhand smuggling are involved.

But none of them knew for sure how the GPUs actually move. To get closer to the

33:38

source of the movement, we decided to wire those funds and get an address. You there, Stephen?

Yes. Stephen?

Yeah. How's it going?

Oh my god. Um Oh, you really help here?

Yeah. So, I've got $3,289 going out.

Perfect. Yeah, that'll be from the the GN account

33:55

and then it's going to and their bank and then account number is uh Swift, I guess, is what you need probably. Um.

Mhm. Okay.

Good to go. I will get this done for you.

Perfect. Okay.

Thank you. All right.

Have fun. Be safe.

Thanks. See you.

34:11

Bye. We'll see if the money's gone in 2 days.

So, that'll be that'll be our next adventure. We packed the hotel, said goodbye to the famous Hong Kong junk boat, potentially said goodbye to $3,000, and boarded a ferry to Shilo

34:28

Port. But we're going to take a quick detour in the story here because we have a source who says he has more information on how GPUs get in to China.

So before checking with our GPU middleman and our fence, we're first going to meet up with someone else.

34:49

[Music] Next up, we took a couple hour car ride to Hujo to meet with a guy known as Mr. Five.

Hujo is a city with some serious grit. [Music]

35:12

The city has grown to take some of Shenzhen's factory industry as it's been pushed out over the years with Shenzhen's expansion and as it's turned into more of a metropolis. This is a complete 180 from the finance hub that we were just in.

We've spent a good amount of time at Hujo over the last decade, mostly visiting case, painting,

35:29

tempered glass, bending, and tooling factories alongside those for heat sinks like video cards. He's a cooling hardware reviewer and has specialist knowledge in factories that make video card cooling solutions, including Nvidas.

This experience allows him more access to information about the peculiar

35:45

relationship between the factories making the cards and the companies that on paper can't sell them to the country where they're made. We instantly related to Mr.

five for his own editorial dispute with Nvidia's review sampling process where we've both felt that the company seeks to control review

36:01

direction. His username is 51972 on Billy Billy.

Even if you can't speak Chinese, you just check it out cuz he's got charts and everyone can read those. Most of this interview is in Chinese, but we've transcribed it in subtitles and there are English sections as well.

The next interviews will be a mix of

36:17

both languages. Okay.

So we are joined by five that is that is his English name. I noticed when we came to China in SGE market, I noticed a lot of 5090s.

36:52

Mhm. [Music]

37:07

Mhm.

37:38

Do you think most of the band GPUs come in one at a time? Uh, GPU.

38:16

Mhm. [Music] [Music]

38:42

[Music] [Music]

39:16

Are you willing?

39:33

MSRP.

40:04

Made in China. Fore.

40:33

It seems a little bit weird that these products are made in China. They're made here, but technically they're not allowed to be sold to here.

40:58

[Music] So essentially um you know China is the the lead manufacturing for their solutions but once you get into politics

41:13

it's just something where you don't have much control right and given the entire global political like situation yeah it's This is not a surprise anymore. So even though on on a logic standpoint doesn't really make a lot of sense, but it's all

41:29

Do you do you think Nvidia knows about this? Nvidia

41:49

like you turn a blind eye. Okay.

Yeah, there's too many. Okay.

Too much money to be made. So, they're not going to address it directly or acknowledge it, but they know what's going on.

Do you think Nvidia uh would want to

42:08

stop it, boy? No.

No. Okay.

Uh Okay, cool. viewers.

42:24

Where should they go to find his channel? [Music] This was an interesting conversation.

First of all, I learned a new idiom. Secondly, we turned it into a t-shirt.

42:41

So, if you're liking this content so far, please take this brief intermission to head over to store.gamers nexus.net and grab our blind eye t-shirt to directly support this content if you like it. Lawn form content is hard to do on YouTube, and we can do it because of your support.

We try to make some pretty

42:57

damn cool stuff to get you something useful in return as well. And if you're watching this sometime close to when we publish, we've still got the digital reward tiers available for our backer tiers for this project.

We really want to produce more stuff like this. But from our point of view, we need to earn that.

If you think we have, head over to

43:13

store.gamersex.net and grab one of our new blind eye t-shirts, black market sticker packs, GPU VRAMm glassware, or even our GPU shredder t-shirt for the paper launch. Thanks for the intermission.

Let's get back to it. Up next, Shenzhen.

43:29

Our first stop will be in Shenzhen, Paan to meet with a trading company that sits between Hong Kong and Shenzhen warehouses. They're the trading companies trading company and we're buying an RTX 1590 from them if they

43:45

exist hopefully. [Music] Shenzhen was a fishing village just 40 years ago.

Now, it's one of the most technologically advanced cities on Earth, and it's grown unbelievably fast.

44:02

Shenzhen got growth from pushing farming and fishing out and bringing factories in. Now, it's even starting to push some of the factories into the outskirts.

Shenzhen has some extreme cyberpunk vibes with its mix of technology and surveillance. It has packed tech markets in Ha Champ, including some of the

44:18

weirdest computer parts we've ever seen in our prior visits. There are entire buildings dedicated only to small phone repair shops with their own kiosks.

Another building dedicated to a mix of computer hardware, gaming, and miscellaneous components. More still, for just integrated circuits, literally

44:35

an entire building where people pretty much only sell things like capacitors and MOSFETs. And all of these places have people who know people.

That's their job. Shenzhen's technological rise comes with it an uncomfortable omnipresence of CCTV and facial

44:51

recognition, which feels more fitting today than ever before now that we know worldwide government facial recognition is a big user of AIG GPUs. We started the day by meeting up with our translator for the next 2 days, Raymond, who's helped us on factory tours for years.

Because Uber doesn't work in

45:07

China since it uses the Google Maps API and Google is blocked by the firewall, Raymond was also our man with the locally compatible apps to get us places. This became especially important now that even cash is becoming less acceptable by some cafes and restaurants.

Everyone's moved to paying

45:22

with the WeChat app. With our ride booked, we're off to see if the wired funds turn into a GPU.

So, we're going to go meet our uh

45:38

GPU supplier. These guys seem to understand that we're media and we wanted to film, but we weren't 100% sure.

So, we walked in with the cameras off and once we established that, we proceeded.

45:53

Like, would would one of them want to talk with me about just buying sign GPUs or just can't show their face? Oh, okay.

Yeah. And our company name.

Oh, okay. So, you don't want

46:09

Yeah. Okay.

So, we can talk about it, but don't show face or name, I guess. Yeah.

Okay. Face or name or company.

Okay. But talking about it, it's fine.

Talk about it is fine. Yeah.

Okay. This is the card.

I guess this is the one that

46:26

Yes. that I bought.

Okay. Can we record this?

Yes. Yes.

Let's do that. Yes.

Cool.

46:44

Um, from the Hong Kong warehouse. What's the most common GPU your customers buy?

Byebye.

47:09

Black uh GB GB GB GB GB. Okay.

how how popular like 5090 58 or

47:24

5090 5090 uh you know class cards versus the AI H100 H200

47:41

[Music] I heard the 5090D is like Nvidia stop making it.

48:06

[Music] You sent me the RTX6000. [Music]

48:25

Okay. RTX and they have it in the industry.

48:40

You basically um buy them buy the GPUs from a distributor. You mean the agent?

Yeah. Okay.

Yeah. Then everything is really Yeah.

RTX 6000 just show the shipping 196 GB, right? Yeah.

96. Yeah.

48:58

What's the price in your country? Do you need to pay the tax fee for?

Yeah.

49:15

So, I guess is is the most common user like AI companies, robotics companies? They don't really sell to any users, more like trading companies.

49:37

They don't usually get this type of inquiry like single. Yeah, we had talked about GPUs for like 40 minutes at this point.

I guess they got bored and they started asking us questions, but we'll just include it cuz it's kind of a fun culture exchange.

49:54

Yeah.

50:18

um does anyone buy AMD AI GPUs or AMD like highend mi

50:35

Oh, C U D A

50:55

Intern Be be promise.

51:18

[Music] There's no restriction. Correct.

Yeah. Mayor

51:35

they said that uh Nvidia falls into distributors I guess and then they get distributed into here. Okay.

Do you think do you think Nvidia knows about people uh buying and selling

51:50

the band GPUs in China? Like are they aware of that?

Do they try to stop it? They know, but they they're like, "Yeah,

52:06

you guys can sell it, but but but we don't want to know that." Yeah, we don't want to know like it's easy and easy. Funny.

I wonder how many people they see walk out just with a GPU.

52:22

They definitely shared more than I thought they would, so that was good. That's good.

But yeah, I it also connects the pieces really well with the professor we spoke to. Okay.

Cuz that guy, he shared like everything new. Yeah, I guess now we probably just go

52:38

basically back to uh Hua Chambe is a little bit north of that. Uh so our next stop is 10 minutes away from the hotel.

Okay, it's a pretty far ride back. Um, the next one is even more

52:54

uh unknown if they'll talk to us because you didn't buy anything from them. I didn't buy anything.

Um, and I didn't I didn't offer either and I explained the whole situation. Just

53:09

replied to every single message with okay, okay, okay, except for except for um the address. He said, "I'm not sure what to talk about, but I can try." Okay.

Which is like like maybe a good sign.

53:25

I don't know. I'm not sure.

Hard to get right until you see him. Yeah.

So, we'll try that. Um I guess we can keep the camera out in the car and then we'll just probably DD back.

53:42

I think it's going to try it. Yeah, I'll try it.

We're not 100% sure where we're going, but we'll get there. So, I think our biggest takeaway is uh these GPUs are banned for sale into China, and I just walked into an office

53:58

and bought one. So, the the efficacy of the ban is maybe questionable.

This still tracks with what Dr. Da Vinci was telling us, which is that it appears to be the difficulty comes from buying lots

54:13

and lots of them. But even the people we just bought this from, they're like, "Yeah, I mean, if you want more, just ask and we'll figure it out." So the whole concept of this is a thing you cannot own or purchase in China, uh,

54:30

it really just doesn't appear to be as strictly true or simple as we might otherwise think. From the surface of an export controller, a band, you think you just can't get it.

That's not the case. You can get it and it's not even that much more expensive than what I would

54:46

have paid in the US. In fact, their RTX 6000 Pros are the same price.

They're actually cheaper after taxes in the US if you buy it here. And that's also export controlled.

So, that's our uh that was our adventure getting a GPU. Move on to the next one.

The most

55:03

important thing we learned here was where to go next. Just like the ants moving the GPUs piece by piece, we'll have to collect our information piece by piece.

Recapping the parts of the hour-long conversation more efficiently and in one language, the company said that some cards have become more difficult to get, but they can still get

55:20

them. And more expensive or not, it is the reason that their job exists.

The fact that they texted me a photo of the brand new at the time RTX 6000 Pro Blackwell card, the very same one that we just bought for $8,500 in the US and

55:35

that was hard to get at home, shows that this company and their distributor are resourceful. Their price is $8,600 US, which is actually cheaper when factoring in US taxes and shipping.

So, the ban isn't stopping them. And the 6000 Pro is

55:51

a serious AI card with 96 GB of memory. It's banned and it is definitely in demand as well.

This company isn't used to a single GPU sale like ours today. It mostly transacts with other trading companies in high volume.

They interfaced with a Hong Kong distributor to bring the cards into Shenzhen, moving

56:08

it across one more border, another ant in the chain. That meant they could connect us with their distributor, which would be familiar, we were told, with where we could find smugglers to talk to.

But this company itself neither knew many smugglers directly nor knew many end users. And that's okay because each

56:24

link in this chain will get us one ant closer to the information we need. So, we took note of their information on the distributor and the smuggling side to use when we got back home.

Seems like a conversation to not start until after US customs. And then they helped point us toward our next stop.

Up next, Hua

56:40

Champ. We hopped in a car and drove 40 minutes to meet up with a GPU trader we found in Shenzhen.

Huh. Champ is home to the

56:55

world famous SEG market or psycha where we've found some of the strangest computer parts we've ever seen in past videos. Pa Champ has the highest concentration of technology and integrated circuits and electronics in the world with a neverending maze of multiple disconnected multi-story malls

57:12

specializing in all electronics. If your life depended on getting a complete product made in a single city block, Shenzhen Hua Chan Bay is your best bet.

Fortunately, this time we'll be with this guy. This is Vincent.

His profile picture on a messaging app is Van Gogh,

57:28

so he has a sense of humor. This guy wheels and deals with dozens of GPU trading companies near the bustling Hua Bay Plaza.

He has a reputation around here for being the smooth talker.

57:49

Our first impression was that this guy knows people and they already know him and that's what we need. Vincent generally seems to have an attitude of getting done.

But before we met him in

58:04

person, we had some concerns going into this that it would be fruitless or that no one would even be there to meet us. All right, so second location to try and talk about GPUs for this one.

No idea how this is going to go. I've been talking with a seller back and forth for a couple weeks and every single response

58:20

is is two letters. Okay.

And so I don't know what that means. Raymond is going to try and explain the concepts.

Okay. See see what I'm doing.

All right. Let's go.

58:43

[Music] [Music]

59:00

It looks like the right place is like um a lot of tech companies. Yeah, we had a lot of trouble finding our guy, but fortunately, it wasn't long before he found us.

59:20

She had two doors. Yeah.

59:41

He's pretty open about it. Okay.

Whatever he knows.

00:02

[Laughter] Is it difficult to get the GPUs? [Music]

00:20

Like there's a lot of atg. So does he like just buy and sell?

Is he a trading company I guess? Yeah, he is a trading company.

Trading company. So mostly from how do the GPUs normally get here?

uh

00:38

like what's what's the process normally for them to actually cuz highend GPU highend GPU

01:15

Sorry. Taiwan one by one.

01:40

Oh, they do it more piece by piece. Yeah.

Yeah. From Hong Kong.

Yeah.

01:55

Is there any consequence for bringing them in? foreign.

02:14

[Music] Okay.

02:30

Does China care or does Hong Kong care?

02:49

Oh, so like in China is it's not it's not like a real restriction, right? It's more from the US side, right?

Right. Yeah.

So,

03:05

[Music] Everything he says you can buy from the shops, right?

03:24

Yeah. Like they he doesn't exactly know where, but like Hong Kong is like the most closest and most common.

Yeah. That is easiest to bring it in.

What's the normal like order amount? Like how many how many cards does a company normally buy?

03:42

[Music] One or two single units like a

04:04

[Applause]

04:22

It's [Music] More speech.

04:54

[Music] [Music]

05:15

Yoch. foreign.

Okay. [Music]

05:45

years. Yeah.

Okay. Michael Jordan, does he need to make any trips to SCG to buy stuff that we could potentially go

06:00

with him for? You know what I mean?

Like to like if he has any need to go to Saiga to buy stuff, would he be interested in us following him to accomp like accompany him?

06:16

If you could find a soft way to present that, I guess. Okay.

Oh, okay. Okay.

I guess if we see like some of his favorite spots or spots he thinks are

06:31

the most interesting. Yeah.

Yeah. Okay, cool.

Yeah, that would be cool. Okay.

Is it walking distance, you know? Yeah.

Yeah. Oh, is it really?

06:56

No. [Music] Vincent led us over to various large wholesale and consumer retail markets.

07:11

His office is about 10 minutes away from SCG Market, which is his daily haunt. There's one peculiar detail in all of this.

Vincent has customers both in and outside of China, but his foreign customer base is relatively high. He mostly deals in RTX490s, which seems to

07:27

be a trend, as you'll see with our smuggler contact at the end of this. [Music] That means Vincent is buying 4090s that either never left China, but were supposed to, or were likely illegally

07:42

exported to China, which would mean he is then re-reexporting them to his foreign customers. It's like an infinite loop.

He's an interesting guy. He talked about multiple business ventures in Uganda and Kenya.

He said he went to Kenya before.

08:16

She said he had a business there before. Oh, he wanted to go there and uh start a business there.

So, he was like looking around and see if there's any opportunity.

08:36

Oh, he's like the the people couldn't afford it. Like the things that he wanted to sell was out of the the market range.

Okay. Too expensive for the locals.

Yeah.

09:02

He used to work as a shopkeeper in the mall itself and would interface with factories or anyone else who wandered in. Just talking about his business.

Yeah. Yeah.

It's pretty simple to set up a business or have start running a company like his,

09:17

right? So low cost and then uh it's like also like very scattered.

He used to own a shop in there. Oh, really?

Yeah.

09:42

[Laughter] woman. [Music] Okay.

Oh, yeah.

10:00

Okay. We can You can put it down to the side.

Did they let you do it last time? Uh, we snuck around them last time.

Yeah, we looked for them and then we went around them. Oh wow.

These days he's closed shop and is a buyer instead. He mostly takes orders

10:16

online, doesn't keep much inventory, and just walks across the street when an order comes in that day. It's basically like a concierge GPU pickup service.

10:48

Okay. Cuz you can just come here and pick up your stuff and then once you get an order and just, you know, once once you get the order online, then you pick up the stuff here and sell it.

So he doesn't have to actually put down any of his own money to buy keep stock.

11:04

Okay. This I think I've been to before.

11:30

minor. BTC minor minor.

Okay. Vincent did what we soon learned he does best, which is connect us with people.

He brought us to a friend of his who sells Bitcoin mining hardware and GPUs.

11:50

Hello. This is where we learned the

12:05

relationship. She thought he was bringing us here as customers.

So, she didn't want to share his cost with us. She said it's up to him to tell us what the price is and then she'll tell him the cost.

12:23

Okay. Yeah.

A1000. What?

A1000. Where is that?

Right here. Hello.

12:43

19,000 800. That's not bad, honestly.

Like, that's pretty pretty close to US pricing. 2,600.

The average price for 5090 right now in the US is $3,54.

12:59

It's actually like cheaper than average. But that's what the other guy was saying are like the prices is like so similar.

[Laughter]

13:27

BTC the Okay, the like with other interviews, we learned that people rarely want AMD and basically nobody wants Intel.

13:48

AMD.

14:04

Is she one of his suppliers? Do you know?

She's like he does what he does. They're very similar.

Oh, okay. He knows her, I guess.

Yeah. Okay.

cuz he started off as like doing mining

14:19

machines, so she's still doing it. We told Vincent that we were getting ready to travel 6 hours north to check out a GPU repair shop that buys small components for not only repairing broken video cards to keep them in service, especially the expensive band ones, but also modifying and building new video

14:36

cards that Nvidia doesn't make. He said he had an idea for us.

Nothing special, but we can go check it out. Yeah.

Okay. Protect peace.

Yes. Yes.

Yes. Yes.

14:54

Protect peace. Protect peace.

That's my That's my translation. I don't know if it's right.

It is. It is.

Exactly. That's like finished goods and this is like preassembled.

15:11

Okay. This is it.

Cool. Wow.

I don't know what I'm looking at. The ceiling just says New Asian.

15:28

I don't think I've ever been in here before. This is cool.

Yeah. Yeah.

[Music] Oh, MLCC's capacitors. You find those on GPUs everywhere.

SMD service mat device dip.

15:53

Wow. [Music]

16:18

Okay, cool. Okay, cool.

So he's they're relatives.

16:35

Yeah. Okay.

Cool. Well, is one kind of relative.

16:50

No, they are selling on like across different platforms like so it's more like e-commerce. Okay.

Do they sell on Alibaba also? Papa

17:11

[Music]

17:44

You know means no brother like brother. Yeah.

Shi this is like a central place for a lot of factories to come buy stuff. So like

18:00

you know all the vendor suppliers are here. So they just come here and grab this from this guy, grab this from this other store factories repair shops.

18:29

You want to try? Yeah.

Everything is sold on on

18:46

Oh.

19:16

You're good. What's he saying?

So, as long as they make money, they will like it. If they don't make money, then t something else.

19:42

So he's saying it's like Chinese are different from Chinese is like yeah it's more practical. Vincent's cousin connected a missing link for us.

People like him selling FETSS, inductors, capacitors, resistors, MLCC's and so on for repair shops can

20:00

help to supply the parts needed for upfit or modifications for GPUs. someone doing this kind of work but specializing in video cards might source additional VRAM for example.

He told us about how 5090s are easy to get because the local factories supply them matching Mr. 5's

20:16

comments earlier. Some of the sellers here told us that they're capable of sourcing devices like A800's or even A100s.

They just didn't have them on site. They have them in a more secure spot.

That's because there's a separate area for the warehousing. And fortunately, we learned where they are

20:31

from our new friends in SEG. Like every other link in this chain, we're getting closer to understanding the full story with each person who pours some tea for us.

Yeah,

20:46

[Music] he likes chatting or hanging out.

21:02

He's like they enjoy chatting with like different types of people like just to like you know learn from different culture and they enjoy they really enjoy that. So we agree but it was time to head back because tomorrow

21:17

[Music] we had a morning to kill. We wanted to find those warehouses to figure out how many GPUs there are.

This

21:34

seems like the right area. Can like breathe hot air on it and it'll help equalize it.

[Music] We only had a couple hours before our highspeed rail appointment. SCG's guards have never been

21:50

particularly friendly towards filming to begin with, but ultimately they're not police. The worst that happens is we get asked to leave.

Besides, there weren't any signs saying we couldn't be there. I I mean, I I can't read Chinese, but I assume there were

22:07

[Music] [Music]

22:33

Great. Elevator looks promising for the warehouses.

Yeah. Hey,

22:48

I'm colorful. Um, distributor.

[Music] What you were thinking? I was think [Music]

23:08

I was imagining that. I think we got our Jeep used to the left in these cartons.

If they don't have it, they come here and they buy it from these warehouses which are on the upper floors in the building. They just have the carton.

There's like probably a

23:25

dozen tall and about the same deep cartons where each one should have five or six cards in it. And then all those feed into distributed to Alibaba sellers like Vincent and then get sent out from there.

So, it's uh

23:42

it's a lot of GPUs. Not all of them are banned, but I did see a couple 5090s.

So, motherboard. Yeah, motherboard.

Yeah, I just didn't expect this. It was It was This like secretive.

23:58

Yeah. A little bit more mysterious.

No lights. No.

The perspective is amusing. From the perspective of the United States, this floor is filled with highly illegal

24:13

items. They didn't have export licenses.

From the perspective of people working here, this is just their job. It's not shady.

It's not some subterranean refuge of firearms. It's just a poorly lit warehouse like any number of other

24:28

warehouses around the world. But the perspective of the American government here would be different from people working here.

They just want to sell these things and go home. On the other hand, from the perspective of the people working here, we're the shady ones.

Walking around filming and specifically

24:44

trying to not draw attention is not normal. We're just noticing how nobody cares that we're here.

This is nobody gives a And that is called foreshadowing. It's all individual companies.

All these

25:01

all these doors are different warehouses. Should we go up to four or six?

Six. See what you got.

The feebleness electricity. Well,

25:20

there's a couple GPU cartons in there, too. It was around here that we were asked to stop recording.

So we didn't capt what they said but it was what are you doing smooth talked our way

25:37

out of it. We gave our poohaases and we called it the successful shoot.

They're like you can't record in here. It's it's not like you have to get permission to record here.

So we're like we're just documenting what we're seeing

25:54

just for fun. Travel vlog.

Yeah. Travel vlog.

Travel vlog. Yeah.

[Music] Shin

26:27

Raymond, we Um, we got kicked out. Yes, we did.

You did your best to smooth talk them. Yeah, we just told them that we're just documenting and vlogs and and walking around.

Yeah, they didn't really care. Like, they just kind of wanted to make

26:44

sure that, you know, nothing crazy was going on cuz they didn't they didn't like escort us out. They just walked away.

No, they were like, "Oh, yeah, okay. No worries." And then they walked away.

Yeah. Yeah.

They were nice about Yeah, they knew we weren't really doing

26:59

anything. We've learned a lot about GPUs and now we're back out into Shenzhen outside of Saig Market, a couple other places.

So, next up, we're going to go to uh Dunjo and meet with a repair shop who builds custom 48 GB GPUs. So, we'll do that

27:16

next. Up next, Junjo.

[Music] We regrouped with Raymond and left Shenzhen and we headed to the biggest train station any of us had ever been

27:33

in, Shenzhen Bay, which is a high-speed rail station. For the journey ahead, we had a 5-hour 50-minute extremely comfortable high-speed rail ride up to Chanjo.

[Music]

27:48

The train goes several hundred km per hour, which is good because we had to travel somewhere around 1,700 km to get to where we were going. This ended up taking about the same time as a flight would have taken with all the airports factored in.

But the reason we were doing this day trip was to visit Junga's

28:05

repair shop or Brother John's repair shop. He's famous for repair on Billy Billy.

His independent repair shop has some Rossmanlike vibes in some ways. He's got video cards all over the place, spare parts everywhere.

He saves everything that he can salvage and then rips the good components off of the

28:20

boards to reuse them for other ones. But John is an interesting guy in this chain because he's not directly part of the GPU black market.

What he does is fix video cards. He does that for consumers and for companies.

Sometimes that means people want him to modify video cards to

28:37

double the VRAM, for example. That's where it starts getting into the territory that's interesting for our movie today.

So, we're in Janjo. It's uh about 6:00 a.m.

Scorching hot. It's about 30° C already.

And just got breakfast. Now

28:53

we're going to see our repair shop guy to check out the 48 GB GPUs and see what he knows about the AI GPUs and how they get in. It's real high.

29:09

We're just walking in the road to get the car. This is how you do it here.

Or at least this is how Raymond does it here. I don't know if that's how you're supposed to do it.

Okay. All right.

So, we are at John's repair shop. He's up there right now working on GPUs.

You can

29:25

see them in the blue vest. We got a pan up there.

So, we're going to go in and work with them on uh counterfeit GPUs and 48 gig 4090s.

29:43

Hey, [Music] come on, [Music]

29:59

man. [Music]

30:17

Okay. Okay.

Okay.

30:41

Okay. [Music] Need help with that.

Okay. So uh

31:01

FE okay so there's in in a way there's like there's some 5090s but then you know the because there's not that many then you know there's not that many to repair. Okay.

Got it. Cool.

Cool.

31:31

Okay. Okay.

Okay. So, the older cards.

Yeah. Just like past warranty.

Pass warranty AMD cards. Yeah.

31:54

Okay. Okay.

So, even if like uh there's there's not many like the Intel cards and then if they want to repair it, they're unable to repair cuz they don't have the information or experience

32:10

in dealing with that. Okay.

So, there's not really any info for the Intel ones. Not any longer.

These are all finished repair ready to go. Yeah.

So, they have all this here and they're going to weigh it and then have everything packed and ready to go. Cool.

Are most of the cards that come in

32:27

local to Janto or are they from like all over China?

32:48

Okay. Yeah.

Through like his channel through Billy Billy. Billy Billy.

Cool. Very cool.

Okay. Oh, this is cool.

This is a schematic, I

33:05

guess. Oh, really?

Yeah. It's like short circuit or it's like missing circuit.

Are these while Is it hard to find the schematics, the drawings, I guess you could say? But for older models, like for the newest

33:22

ones and it's usually, you know, takes a while to C2. C2.

Is that what it is? Yeah.

Okay. Do they ever do G GPU swap?

GPU. Okay.

33:43

Okay. Really?

That's the easy part. Oh, really?

Okay. It's a lot easier than doing this.

Okay. So, this is actually one of their main fixes.

Swapping. Swapping the GPU.

That's what after after a long time usage,

33:58

right? Like, so that's one of the things that they have to Okay, cool.

We spent another 2 hours touring Junga's shop. It's pretty amazing.

We actually have a full separate video that we're going to put together and publish and it's going to take me forever to translate it. And we'll do that because

34:14

we hit our stretch goals on the fundraiser for this movie where we said we'll post a full separate breakout video just of his shop because it's so damn cool. But the real reason we're here for this specific topic today is to see how these cards are modified.

Because if there's an export ban on GPUs

34:30

going to China, then it becomes critical for China to be more self-sufficient on keeping those GPUs that they do manage to get in service. Most of the repairs are more typical things you'd expect of soldering.

There's board heaters, soldering irons, surface mount

34:46

components gathered from suppliers nearby, similar to the one that we saw in Shenzhen, and a lot of test stations. But one thing that's unique to China and especially to this shop is the ability to take an existing model video card and completely modify the skew into

35:02

something that Nvidia doesn't even make. Nvidia is intentionally restrictive with how much RAM it puts on cards.

Part of that is to upsell people to more expensive model. Actually, that's pretty much all of it.

That's basically why they do it. But at shops like this one, they double the VRAM on some of the

35:18

cards that come through. It's a fully custom skew.

So, we wanted to see if he had one lying around that he could show us. And he did one better than that.

[Music]

35:35

He should make one right now. Okay.

Did you want me to make one? If she feels like it, if it's not too inconvenient.

Yeah,

35:58

he could do he he could do the from 24 to 48 gig right now. Yeah.

Yeah. So, I think we're going to make a 48 GB 4090

36:13

right now. I think that's what's happening.

36:29

GB. This is a 24 GB 4090 that's been through a lot of repair steps.

So, if we zoom in up here, they track the repair history on the card in order. So, this is 1 2 3 4 5 six repair steps.

The board's been through a lot. It originally didn't work at all and then they had a memory

36:46

problem. They had some other problems in between here.

Uh and now it's actually working. I think that's that okay is.

And so it's functioning now. And you can see it even turns on.

So it's a 4090 uh and it's got all 24 GB detected. But because this board has been through so

37:01

much uh how do you call it like trauma or something from repairs, uh it's going to be better to harvest the expensive components and put them on a new board so they don't have to worry about if it's just going to fail again in the future from something else. So I think the next step is maybe going to be to

37:17

put the extra memory on it to go to 48 GB. So okay.

Okay, he's going to run a quick test.

37:33

Yeah. Mark, what's like the success rate of the swap for him?

37:52

Okay. Wow.

Okay. Yeah, there's still like a 1% chance.

That's really good, though. I mean, yeah, that's impressive.

Cool.

38:14

Oh, cool. Okay.

Okay. Protect it.

Oh, great. To protect from static.

Oh, ESD. Okay.

So, just protecting it from static electricity with the cover on the PCIe

38:31

slot. And now we're gonna go into the lab.

So, this is the repair lab with stacks upon stacks of GPUs. They test about 50 a day here.

Because Nvidia doesn't make a 48 gig 4090, the repair shop has to

38:48

source its own PCB. It needs something that has enough pads for all the memory modules they're adding to it.

They work with a third party supplier that builds a PCB, uses an SMT line to place all the VRM components, and has the extra wiring circuitry and pads to support 48 GB of

39:04

memory. That's what you're looking at here.

It's a custom board just for this. Cool.

So, I think they're going to disassemble it now. They're just showing us the PCB.

Uh, it's made custom. He said you just buy it from shops.

They all have it. And

39:20

it's he was saying they're very common here. So first is disassembly.

Just going to take apart the old card. So what did he say?

Sometimes people buy

39:36

like 10 or 20 at a time or something. Uh he gets a 10 or 20 orders at a time.

And then that was like um like numbers or model. Yeah.

for AI. Yeah.

39:52

Running like langu models or right whatever they're doing. Does he think does Nvidia care does he think Nvidia cares that people cuz this isn't an official skew?

Not that he should give, you know, he has no reason

40:08

to care, but yeah. Uh

40:26

They can't do anything anyway. And it's after market, I guess.

You can It's kind of like sweeping up your car, right? [Music] Sometimes Nvidia blocks stuff.

Okay. like they'll do something in the the firmware of the BIOS to specifically

40:44

try and like block a modification. How long does this step normally take?

[Music]

41:02

So, I'm going to heat it up right now to 260°C. This is really similar to what we saw with um EVGA.

And uh just blasting the GPU with heat. Takes about 5 minutes and then he'll be able to pull the steaming GPU off.

41:23

All right. So now he's pulling the GPU.

It's going to happen really fast. He puts it on just a metal stand.

41:54

Soy memory. So now they're using a board heater.

You can see that there's a temperature reading on the front. uh currently at 133 Celsius.

It's going to heat the board and then he's gonna use the hot air station which

42:09

uh is up and right underneath a benchtop power supply. So that hot air station will be used to heat pinpointed memory modules and remove them without risking damage to other components on the board.

And so this is what Lewis Rossman uh was

42:26

talking about in our video about tariffs previously. the thing he sells similar.

So, they keep the original memory modules. So, he's going to pull these and keep them.

And then they also have new modules they bought sounds like just from Tao basically is retailer here. And they're going to add those to the board so they can they can double the memory

42:43

capacity. Cool.

I'm quiet. These are quieter.

43:02

Wow. So, the board heater is at 320° and the hot air station's at 396° C.

43:22

This part is done. Okay.

So, right now they have this uh template for the solder balls. So, this guy is currently he's cleaning it looks like with mostly rubbing alcohol.

That's a cool dispenser. Uh so, he's got wipes it

43:39

down. He's going to pour solder balls into the uh the template and then use that to position it on top of the memory modules.

They do eight at a time and that'll allow them to bake the new solder balls onto the memory modules which they can then attach to the PCB.

43:57

So you can see it's actually sized for the memory modules. Looks like now he's uh he's already applied and he's

44:12

just wetting the solder balls.

44:27

Okay. So, now he's using a wick.

So, this is a solder wick. Uh, it's just a piece of copper that basically pulls off the excess solder.

So, he's running that over the surface. What do uh 48 GB cards normally cost?

44:44

Like what do they normally sell for? 20,000 20,000 plus 2,800 US.

45:01

That's not bad. Actually, the soldering ball.

Yeah.

45:25

Okay. So, it's kind of like a what we call it.

It's kind of have everything kind of filter through, right?

45:42

Okay. Okay.

So, they they're going to add the heat now, right? That's what he said.

Okay.

46:00

Okay. Oh, okay.

Cuz he needs to do another one, right? Yeah, he's going to do another one cuz he could only do eight at a time.

At a time based on his Okay. tool.

Yeah.

46:26

Now he's adding heat. So, the hot air station is currently running at 390 plus or minus degrees Celsius.

He's directly applying heat to the surface to where the solder balls are. And then he's got the memory modules

46:42

themselves heated from underneath by a hot plate. So, the hot plates plugged in.

That's running at 195° C. And now he's just holding it down while he presses it into the hot plate.

47:00

Is it hard to get uh like 4090s or 5090s for him?

47:20

customers bring it. He doesn't have to buy he doesn't actually buy just repair.

Got it. It's just so crazy to me how like there's a couple shops in the US that that can do repairs,

47:38

but they don't really do like this level of repair. They Okay.

But I I don't know of a single shop in the US that does like the memory modding. Yeah.

Cuz labor cost is a huge issue,

47:53

right? For the US for him it's like it's worth it for the customer is worth it.

Yeah. Yeah.

Cuz it doesn't like this the labor cost or the service cost is not as high.

48:16

Oh, okay. Okay.

Okay. So, next stage at this point, they've um

48:33

reballed all of the memory and the GPU. And now he's cleaning the surface of it to prep it to mount to a new PCB.

We saw the new PCB earlier where it's it's a brand new custom PCB they just ordered

48:50

from Tao, which is the retailer. It's adding flux now.

Lewis Rosman says you can never have enough flux. And so that's going to help it

49:06

when he melts the solder to apply it to the board. Another custom.

So, there's a custom template just like for the memory. He's going to apply that to the surface.

And now he's got a tray to catch the

49:22

excess. And he's going to just Yeah.

basically like sifting for gold or something and sand. So, he's sifting it.

wipe off the extra

49:40

and then carefully remove the template while leaving them all in place and they should be held in place by the flux. So, this is a jig to hold the GPU in place.

I guess the question is, is that to protect the GPU from the heat?

50:18

I guess. Are they putting the stuff on the card now?

This is a new one.

50:53

Okay, so now we are the GPU is on there. So we're at the final stage.

This whole process is like about 2 hours I guess. And now we're doing the the opposite of what happened earlier in the day.

So now instead of removing the GPU, he's

51:09

adding the GPU. So, he's very carefully positioning the GPU to precisely align the PGA with the solder balls.

51:28

And now he's left the room. So, I guess Tannon's going to take over from here.

Interesting. So, it looks like he's starting to reach steady state on the temperature.

a couple hundred degrees. Ton of

51:44

ton of bubbling going on at the edges, but uh they're just baking the GPU back on. So, at this point in the process, the PCB has been swapped.

The PCB already had all the VRM components and the IO populated 12vt high power. And

51:59

now they've added new memory to the front and the back. One's got reused memory, one's got brand new, and then the GPU has been pulled and swapped and is now being remounted.

And they're telling us this is a 99% chance success.

52:16

All right, this is what Kenpin called the nervous part. Okay, so he's going to cool it off with the fan.

52:36

Do they reuse the same cooler that they took off the old board or do they need a new cooler now? Okay,

52:54

that's the cool snapping. Moment of truth.

Moment of truth. Yeah.

So, he's just going to hold the heat sink on and see if it powers on. Cool.

Okay. So, now he's testing to see if it actually powers on after all that.

It's got a green LED with displays up

53:11

debug up top. And he's just holding a heat sink in place cuz all he needs is power to see if it spits out display.

We got display up top.

53:27

So, it is functional. I was just a question of everything detects.

I said, "No, we're lucky." He's like, "Usually it doesn't turn on the first time in a while." They know we have a train to catch. Yeah.

Does he know do the customers who buy

53:44

these do they mostly put them in servers? Yeah.

Like if the customer buys more like multiple units, then it's probably like, you know, server. Yeah.

Yeah. That's why the blower fan would be better for servers cuz they pack them in

54:00

so tight. I was just asking if they have different choices for the cooler and then they're like there's only one they only they don't have a choice basically.

54:17

Okay. So next step, one of the last steps in the process since it booted and worked.

Now they are pulling the protective film from the thermal pads. There's a PTM or

54:33

actually is that that might be a paste actually pre-applied. Looks like it was just silk screened on there.

[Music]

54:59

[Music] Looks like it works. So, within 2 hours, start to finish, we have a working 4090 48 GB.

At the beginning of this process, the card was working, but it had been through so many repairs that it was very likely it'd break again at some point from something they couldn't really

55:15

predict. First of all, from a waste standpoint, it's just way better to figure out a way to save that card because the PCB is kind of the least valuable thing in all of that and is also the least precious of the resources.

Being able to save the silicon that's in the memory and in the actual GPU itself, that has a lot of

55:32

value, but especially has value in China where they just don't have as much supply. Even though the supply looks pretty damn good from what we saw, it's still intentionally restricted.

So purely from the perspective of the people living there, not only is this better than what Nvidia shipped to begin with, and really not that far off in

55:49

price when it was at its most scalped, it's also just a good way to kind of keep the silicon in circulation even as things like MOSFETs, capacitors, or PCBs die. The silicon tends to be pretty resilient and it doesn't really die that often on GPUs.

So, what this tells us is

56:04

that this is maybe an alternative method to getting more GPUs into supply. If they take the broken boards and then put functional silicon back into circulation, it's a certain level of ingenuity, and there aren't many places that are set up to handle it.

Brother John's shop is impressive. We're excited

56:21

for the whole separate video on that. After finishing at the repair shop, we rushed over to the highspeed rail.

Spent 6 hours traveling back to Shenzhen Bay, adding another thousand mi or so to our travel log here. And then we took a car over to visit this guy.

He calls himself the Chinese Eminem.

56:38

Eminem and China. So me.

This is Joe. He works as a creative director of product at the video card manufacturer.

Yes. Most famous for their uh waifu video cards.

You know the ones. Yeah.

Yeah. I'm he his father.

56:56

She's father. She's the father.

Yes. She's his father.

Yeah, we have husband. Anything else you want to talk about on camera?

I think we got it all. Game natur.

57:12

[Laughter] Yeah, I also have a same nickname. Which one?

Yes. of yes.

What does that mean in English? Uh yes.

Uh yes. Tough guy.

Tough guy. All for I will put your head

57:31

on the ground. Joe and Yes were not officially contributing to the black market side of the research here just to be clear to their own partners.

Instead, they showed us around their factory. This will be a separate upload later with an actual factory tour with them showing how the waifuss are made.

But in this particular

57:48

video, there were a few interesting things we learned that we thought fit here pretty well. The big takeaway from the factories is that they receive their GPU supply, of course, from their partners.

That would be AMD, Intel, or Nvidia, and the memory is often packaged with it. But for the most part, everything else that they use, unless

58:04

they're buying reference PCBs, all that comes from whatever sources they want to find for their supply. We were also informed that there's very little oversight in terms of the management of the rejects at these places.

They can file for refunds if, for example, a chip is bad. But if they screw up a board in

58:21

their own process, like the SMT lines miscalibrated and optical inspection doesn't catch the error and they place a MOSFET on backwards and blow something up or whatever, then it's going to be on them to deal with the defect. Now, this type of thing basically never happens.

All of this stuff is highly automated. The point of pointing it out though is

58:37

that for at least the gaming class cards, which the 5090 more or less is one of, there's enough volume and Nvidia's hands off enough that it would be pretty easy to make these disappear. The biggest reason for this is that there are partners for the 5090s.

It's not like a card that only Nvidia makes,

58:52

such as the RTX Pro 6000 and the Blackwell edition. And since there are partners, Nvidia is only tracking when they sell them the GPUs.

They don't necessarily keep tabs on where those get distributed once they're done being manufactured. So, for this reason, if a factory wants to just sell it

59:08

domestically in China, and they think they can get away with it without being added to the entity list, which it's very likely would kill their business with Nvidia, then they can go ahead and sell them domestically. That might be how some of these trading companies get a hold of them.

It's an easier way to make them disappear than through channels that are more publicly visible

59:25

from Nvidia and government. Now, Yestin doesn't make 5090s, but the processes they taught us about illustrate how that could happen.

As for the server grade solutions, there are manufacturers for those that are pretty tightly controlled. Basically, the silicon is made in Taiwan.

It's shipped to wherever

59:41

it's going to be assembled. Almost all of the other components are sourced in China, and as a result, it's easiest to keep the whole supply chain there, including the assembly.

And the assembly involves the same stuff you're seeing for the lower-end cards and the AMD solutions we're looking at from in this factory line. They use surface mount

59:57

technology or SMT lines with conveyored inflow of PCBs, reels upon reels of components that are placed smallest to largest with the most valuable typically at the very end. That'd be the GPU.

And then manual placement for heat sinks. There is often manual assembly even for

00:12

Nvidia's high-end stuff. Although we can't show you that here, we have seen it being made.

That's in different factories. And at the end of the process, it's possible they find a defect.

Again, Dr. Vinci Chow had one where he has a broken link on his board.

There's a good chance that stayed in China and never left. If it was reported

00:28

as a defect to Nvidia, then it may have effectively just been written off and considered trash. But after seeing how a highly skilled group like Brother John's Repair Shop a second ago can just repurpose all those components that are still good, you could see how there would be value in figuring out a way to

00:45

make that board disappear, even if it's supposed to be in the trash. This comes back to Nvidia's turning a blind eye because they'd eventually notice it from the serial number popping up somewhere, but maybe they choose not to.

And besides, once someone has it, especially in China, there's not a ton they can do

01:01

anyway. That's a look inside the manufacturing process, which we felt was an important piece to this story.

Let's move on to the next stop. My favorite place to go in the world, Taipei, Taiwan.

Most of our stay in Taiwan was for other

01:17

stories. However, we did accidentally unturn one stone with something interesting under it.

We can't show any B-roll for this because we do protect sources, so we're just showing some other footage instead. A B2B company in Taiwan noted to us that it's commissioned by Chinese companies to import servers for pre-esting and

01:33

pre-assembly and setup steps. When we asked what the GPUs were that the servers ran, having just done all the rest of this, we realized that all of the hardware is export controlled.

Once the B2B agency completes its testing, it ships the system out to the original buyer. Basically, they act as an

01:49

intermediary to buy the machine, mark it up, and rehip it to the original purchaser, maybe with actual testing. We also spoke with a company that conducts business in Singapore.

The company informed us that they are also aware of similar passroughs for so-called testing or data services. Our last link in this

02:05

chain is actually the first, the smugglers, bring it full circle, and the smugglers themselves also put us back in the United States. We didn't think we'd be able to find someone doing the dirty work here, but in the final hours, a viewer had a lead.

This led to about a

02:21

12 or 16 hour production delay as we were wrapping the project, but as far as we're aware, this is the first content piece that actually features someone doing the highest risk dirty work. Everyone else we spoke to is in China.

They're safe from US retaliation. But large-scale smugglers get arrested, find

02:37

millions of dollars, and can spend years in prison if they were really serious. A viewer contacted us to say that he'd connected with a traveling GPU buyer.

The story went that this guy drives around buying specifically RTX490s. He doesn't care about anything else, including 5090s because those don't sell

02:53

as well to China. 4090s are ideal, he told us, because they can be modified into cost-effective 48 GB models like we saw at Brother John's shop.

The smuggler was extremely open with us in text messages written only in Chinese, but wasn't open to us flying out to him. He

03:08

did allow us to share this video of his car, though. The plug, as we call him, has an ATX test bench, motherboard, simple downdraft cooler, and a gigantic battery in the trunk of his Prius.

He also has at least one spare license plate in the trunk in addition to the one on the back of his car. We're not

03:24

sure why, but better not to ask. People like The Plug post on Facebook Marketplace and other common online reseller forums looking to buy GPUs just like anyone else in your city would.

The Plug offers $2,000 flat per 4090. Doesn't matter which model it is.

It's not a bad price. He told us that he then

03:40

finds ways to get them into China. He recently was scammed out of $5,500 though of payments owed by a Hong Kong buyer and that buyer resells in Hong Kong.

He's been considering hand carrying the devices and luggage instead. Margin is slim.

He makes just under $300 US per GPU profit, but that's

03:58

before counting gas, potential hotels as he travels, and time. Our understanding is that some in his shoes will strip the GPU cooler off and ship just the PCB back to China, increasing their margin.

Coolers are available in abundance where they're made, so it can be cheaper to do this than pay for the weight and the

04:13

size. He sometimes buys entire computers just to take the 4090s out as that's what the bounty's on.

He then sells the remaining system back to whoever will buy it. He told us that 5090 prices in China are falling so fast due to over supply, ironically, that he'd lose money reselling it to China and would do

04:29

better flipping it to Americans. The plug isn't rich.

She seems to be doing okay, but it's not the type of wealth you might expect for taking such risk. We can't overstate though how important it was to get this piece of the story as this allowed us to fully complete the chain.

And a huge thanks to our

04:44

anonymous viewer who set that up. Many of the sources we met in Asia during this trip told us they simply don't know how the GPUs actually get in.

And now we know. People doing this on a small scale, like the plug, are unlikely to be caught.

But operations transacting millions of dollars worth of GPUs would

05:00

have a harder time getting them out undetected. And that brings us to the end of our travels, but not the end of this piece.

We have two more entities to look at and that's Nvidia and the US government. The legality is simple.

This is our understanding of it. There's no restriction on purchasers only on

05:16

sellers the US has control over. Even Americans buying GPUs in China are not violating any laws.

It is not restricted. So, we'll start with selling.

The simplest answer to who the sale of GPUs is restricted for is anybody who doesn't have a specific

05:32

export license and who would be governed by US law and Nvidia can't just bypass it by shipping from a different country instead. It has to do with the sale, not with the shipment location.

That includes entities buying from US companies like European companies who would be under guidance from both Nvidia

05:49

and if they do business with the US, its government. It's also illegal for anybody in the United States, citizen or not, to sell these GPUs to China, Hong Kong, Macau, or companies in those locations if the seller does not have a reexport license.

It is not illegal for

06:06

a person of any nationality in America to sell a GPU to any non-restricted entity. As for buying, there's absolutely nothing illegal about for us over here to buy these GPUs, but it's also not illegal to sell them in China for a Chinese company.

The

06:22

Chinese government doesn't generally enforce American export laws. Other nations cooperating with the United States might, such as recent arrests in Singapore.

But once the GPU is in China, the people in possession of it likely don't care. Buyer or seller, or someone

06:38

who does both, like GPU dealer Vincent. The only control over Chinese companies

06:54

that the US has is the entity list, which would hurt their business prospects with American companies, but only if they care about that. Chinese GPU middleman Sutan Halei in Tianin landed on the entity list for transacting banned GPUs and being found out.

Sutang Halei purchases GPUs from

07:11

anyone who can get them into China and then they bid on domestic projects like data center buildouts. The entity list was used to restrict for example Deepcool previously resulting in their American partners ceasing business with them for fear of frozen assets, audits or collateral bans.

So this hurts

07:27

companies like Deepcool that want to operate in the US and it even shut down their California office. But for Sutan Halei, they mostly want to do business with other Chinese companies and so it have limited impact unless they wanted to expand to do business with Americans.

Beyond the entity list, the US really

07:42

has no control over what happens inside Chinese borders. That means the only point at which a GPU could feasibly be intercepted and a person arrested would be operating in the US or in transit to intermediary countries such as Singapore, which may have their own export laws.

The people buying and

07:59

selling them within China are not breaking any of their own government's laws, though. And so, we come back to Nvidia.

At every turn, it really looks like Nvidia is playing all sides. If there's enough money to be made, anybody is Nvidia's friend.

We think the story

08:14

of AI GPUs has become a story of corruption between governments and the wealthiest company on Earth. For example, on April 30th, Amazonbacked AI startup Anthropic called on the US government to increase export control restrictions to China.

As part of this blog post, Anthropic said the government

08:30

needs to improve its export enforcement to reduce smuggling. The company cited examples of chips being smuggled with quote prosthetic baby bumps end quote and quote live lobsters end quote.

This upset Nvidia obviously because Nvidia doesn't like restrictions to making

08:46

money. So Nvidia shot back, seemingly taking a page out of Trump's playbook.

Nvidia essentially called this fake news. Quote, "American firms should focus on innovation and rise to the challenge rather than tell tall tales that large, heavy, and sensitive electronics are somehow smuggled in

09:02

scare quotes bumps or scare quotes alongside live lobsters end quote." Except Anthropic isn't telling tall tales. It's right.

A 2022 video that we uncovered showed a security check at Duhai, a port in Guangong. It's not far

09:18

from Shenzhen, wherein a woman with a prosthetic baby bump was shown to have been carrying CPUs and iPhones instead. The report made it to customs.gov.cn, stating that she arrived from Macau, a common go-between, similar to Hong Kong.

This story got international attention

09:34

in technical media, and we reject the possibility that Nvidia wasn't aware of it. We found the official Chinese government posting about smuggling from Macau.

Imports to most of China are taxed and so tax evasion coupled with smuggling will increase margin on the electronics rather than sharing it with

09:51

one of the two governments. If you're already breaking a US law, it seems some just go for a hat-tick and increase the profits.

As for the lobsters, that's real, too. We unearthed this official Hong Kong customs website detailing a white van busted driving on the Juhai

10:06

Macau bridge filled with 280 kg of undeclared live lobsters and yes 70 smuggled GPUs complete with photo evidence. The Hong Kong customs itself calls this so or smuggling and notes a maximum sentence of 7 years.

Nvidia's

10:24

response to call these tall tales then is not only defensive but serves to gaslight and grossly mislead we think and is tantamount to lying for sake of downplaying reality for its own benefit. But then this is a common Nvidia tactic including its dishonest approach to

10:40

reviews that we've already detailed in our opinions and its deceptive and we think false advertising of the RTX 5070 as being equivalent to an RTX 4090 which is provably and comically false. Another instance of Nvidia's fake news defense was following a July 24th publication by

10:56

the Financial Times, reporting that more than a billion dollars of Nvidia AI chips had been smuggled into China. In response, Nvidia, whose blind eye is turning an awful lot lately, downplayed the issue, stating, quote, "Trying to cobble together data centers from smuggled products is a losing

11:12

proposition, both technically and economically. Data centers require service and support, which we provide only to authorized Nvidia products." End quote.

The last part sounds like something a company selling support would say. And the first part doesn't really match Nvidia's whole the more you buy the more you save thing.

It's only a

11:30

winning proposition by all of their prior years of statements. If your only option is a useless insufficient data center or a cobbled together sufficient data center, then a cobbled together one is still a winning proposition by comparison.

It's weird for them to pretend that this isn't worth doing.

11:45

It's worth like lots of money. That same day, whitehouse.gov gov posted an article titled wide a claim for President Trump's visionary AI action plan.

Near the top of the post, it highlighted a sickopantic quote from Jensen Juan which read quote America's unique advantage that no country could

12:01

possibly have is President Trump. End quote.

And we would call that sickopantic if it's about any president. On August 5th, Nvidia got another opportunity to talk.

The US Department of Justice announced that it had arrested two people in California for smuggling quote tens of millions of dollars worth of sensitive microchips

12:17

used in artificial intelligence applications and quote to China. The court documents say the shipments included the Nvidia H100 and the Nvidia RTX490.

Rather than admit smuggling exists, Nvidia downplayed the situation again and stated this time, quote, "This case demonstrates that smuggling is a

12:34

non-starter." End quote. except that anyone who made tens of millions of dollars before getting caught had a pretty good start and so did their customers.

This is a statement from Nvidia that seemingly aims to downplay and deflect to reduce lawmaker attention on its monopoly. Nvidia also

12:50

said, quote, "We primarily sell products to well-known partners, including OEMs, who help us ensure that all sales comply with the US export control rules." Nvidia noted that, quote, "Any diverted products would have no service support or updates." End quote. Again, this is not fully true.

Our own sources in this

13:06

story noted that although something like an HDX system would be hard to service, a standalone PCIe GPU could be parted out and covered under a separate warranty, even in China. Meanwhile, the US began scrutinizing Nvidia's technology for getting into China, whether or not the company itself was

13:21

directly involved. In a bipartisan report called Deepseek Unmasked, a government committee called Deepseek a quote threat to national security end quote and said Deepseek had used Nvidia's technology.

Quote, "AI model appears to be powered by advanced chips provided by American semiconductor giant

13:37

Nvidia and reportedly utilizes tens of thousands of chips that are currently restricted from export to the People's Republic of China or PRC. NVIDIA designed and manufactured many of these chips to create the most sophisticated possible chip while skirting US export controls.

This has allowed these chips

13:53

to be exported to China as the US government develops stricter restrictions. Since March 2024, it is estimated that Nvidia has produced over 1 million chips for the Chinese market.

End quote. The government also examined Nvidia's significant revenue growth in Singapore compared to China based on SEC

14:09

filings, particularly in years featuring restrictions. The government questioned, quote, whether PRC customers are arranging for the diversion of sensitive chips that are reportedly sold through Singapore, end quote.

This was stated because revenue from Singapore had grown from almost nothing since 2020 or 2021.

14:26

Nvidia has defended its sales to Singapore by stating that quote customers use Singapore to centralize invoicing while our products are almost always shipped elsewhere. End quote.

According to Nvidia, shipments destined to Singapore were only 2% of the company's total revenue in 2025. But we

14:43

also know that Singapore has made numerous arrests relating to GPU smuggling. So there appears to be some reason for the concerns whether or not Nvidia itself wants to turn a blind eye to it.

Despite the allegations, Nvidia downplayed any smuggling of AI chips. In

14:58

this video uploaded two months ago from what appears to be either a potato or a VHS tape, Jensen Juan speaks on smuggling. Governments understand that diversion is not allowed.

And there's no evidence of any AI uh chip diversion. Except that there is evidence of it, not

15:15

only in this very movie, but in readily available reports online for years now. Hang continued, "Our data center GPUs are massive.

These are massive systems. The Grace Blackwell system is nearly two tons.

And so you're not going to be uh sh you're not going

15:31

to be putting that in your pocket or your backpack anytime soon. And so these systems are fairly easy to keep track of.

And um and but the important thing is that the countries and the companies that we sell to recognize that diversion is not allowed and everybody would like

15:47

to continue to buy convenient technology. And so, uh, they're they're very well, uh, they monitor themselves very carefully and and, uh, they're quite careful about that.

This one's interesting. He's right in that it is much harder to smuggle Grace Blackwell or Hopper HGX class systems

16:03

completely. Dr.

Vinci Chowo's statements align with this. It's very hard to get like a full HGX system, right?

But it still happens. At least one of Nvidia's GPU and server customers in another country told us that they facilitate intermediary transmission to

16:19

China and in fact showed us the server racks on site in their facilities. We weren't allowed to film them, so this footage is from Computex instead, but we saw them.

A separate representative told us that document forgery through third party countries can also disguise such trans shipments. One middleman told us

16:35

on camera earlier that an Nvidia distributor gets parts into China, a downchain factory, and not the one we're showing right now, in case Nvidia has its famed journal of retaliation out, told us that Nvidia's QC rejects sometimes end up repurposed and kept in China, salvaging the GPU and the VRAM

16:52

and scrapping the rest. Dr.

Vinci Chow told us that one of his own devices had a defective link on it, contributing to this statement because he acquired said device. It just happens to have a defect.

And he also said this. Do you think Nvidia

17:08

knows that all of this is happening? I would be surprised if they don't.

Right. Yeah.

I would be surprised like I would be really surprised if they don't. I see.

Right. These are very expensive items.

Yeah. Right.

I would imagine you would keep track of everything. Right.

Yeah. I'm not I don't know.

Is that

17:24

I'm don't Yeah. It's hard to know how what they can do, how what what they plan to do with all these uh defective parts.

But I'll be very surprised that no one have ever thought of the possibility that if it's something so valuable, someone would come up with a use of

17:40

even a defective one defective ones. Yeah.

So in one set of statements, Nvidia says that smuggling doesn't really happen because the export controls work and keep partners in line. But in another statement, Jensen Juan called the US export controls a quote failure end quote talking out of both sides of his

17:57

mouth. We think he spoke of competing Chinese GPU brands posing a threat to Nvidia.

Quote, "The local companies are very, very talented and very determined, and the export control gave them the spirit, the energy, and the government support to accelerate their development. I think all in all, the export control

18:13

was a failure." End quote. But the stakes and the dollar signs for Nvidia had increased.

Jensen said Nvidia's market share in China had dropped from 95% to 50%. And in Nvidia's May quarterly earnings before the age 20 exemption and revenue share, Jensen Juan

18:29

said the company's data center business in China was done. Quote, however, the $50 billion China market is effectively closed to US industry.

The H20 export ban ended our hopper data center business in China. End quote.

Or as Jensen says, the China market is worth

18:45

one Boeing. Uh the Chinese market in a couple years is probably about $50 billion.

Uh the the market we've left behind is utterly gigantic. $50 billion just so you so you have a feeling for that number.

$50 billion is like Boeing.

19:00

Probably not the best example, but okay. Let's look back at the timeline one more time for these export controls on Nvidia.

This time we'll focus on highlighting Nvidia's side of things, which is its relentless appetite for global domination. The October 7th, 2022 Biden administration export controls had

19:16

a stated goal of quote protecting US national security and foreign policy interests end quote by implementing new export controls restricting China's ability to build high-end semiconductors including for the development of supercomputers. There was an included goal of staving off the potential for China to develop quote nuclear weapons

19:31

and other military technologies. End quote.

Blocking the H100 and A100 led to Nvidia creating an export compliant A800 at about 70% of the speed of an A100 for the Chinese market. A year later, on October 17th, 2023, the US Department of Commerce updated its export compliance

19:47

and restricted Nvidia's A800 chip as well, alongside the newer China targeted H800. Weeks later, on December 6th, 2023, Nvidia told reporters in Singapore that it would be working on another new chip that would comply with the US's new restrictions.

On July 13, 2025, which

20:03

was after Donald Trump won the 2024 presidential election, but a week before he took office, the outgoing Biden administration tightened export controls by introducing national chip caps for many countries except for 18 allies. That's the AI diffusion rule we spoke of

20:18

earlier, and it would have gone into effect in May. Nvidia, who had remained relatively quiet about the regulations up until this point, criticized the restriction and made an attempt to appeal to the president-elect, stating, quote, "It makes no sense for the Biden White House to control everyday data center computers and technology that's

20:34

already in gaming PCs worldwide, disguised as an anti-China move. The extreme country cap policy will affect mainstream computers in countries around the world, doing nothing to promote national security, but rather pushing the world to alternative technologies.

AI is mainstream computing, ubiquitous

20:50

and essential as electricity. This lastminute Biden administration policy would be a legacy that will be criticized by US industry and the global community.

End quote. Seems like Nvidia tried to also set up an appeal to the president-elect, stating this quote, "We would encourage President Biden to not

21:07

preempt incoming President Trump by enacting a policy that will only harm the US economy, set America back, and play into the hands of US adversaries." End quote. This is where in the timeline we'd hit the million-dollar dinner and then the ensuing ban unban of the H20 chip, but we've already covered that.

Later that month on April 30th, Juan

21:24

said this of Trump. Without the president's leadership, his policies, his support, and very importantly his strong encouragement, and I mean a strong encouragement I

21:39

frankly frankly uh manufacturing United States uh wouldn't have accelerated to this pace. A month after the Mara Lago estate dinner on May 7th, the US Department of Commerce confirmed that it will not implement the AI diffusion rule that Nvidia campaigned against and that was created under the Biden administration.

The rule was supposed to

21:56

go into effect a week later on May 15th. This does not unban GPUs like the H100, 59, EB100, and so on.

Following the termination of what was supposed to be a rule to address national security implications, the Department of Commerce, now under President Trump, stated, quote, "The Biden AI rule is

22:12

overly complex, overly bureaucratic, and would sty American innovation. We'll be replacing it with a much simpler rule that unleashes American innovation and ensures American AI dominance." End quote.

Nvidia predictably celebrated this statement. The company, which has been begging to sell to China while also

22:28

praising Taiwan's importance, now took an America first posture, collecting countries like Pokemon. Quote, "We welcome the administration's leadership and new direction on AI policy.

With the AI diffusion rule revoked, America will have a once- in a generation opportunity to lead the next industrial revolution

22:43

and create high-paying US jobs, build new US supply infrastructure, and alleviate the trade deficit." End quote. Job creation promises coming from this company in particular are interesting, but they are playing all sides here consistently.

Later that month on May 28th, Juan spoke with Mad Money host Jim

23:00

Kramer. He stated, quote, when he rescended the AI diffusion rule, um it was a visionary move.

It was a bold move and he recognizes that there's an AI race and we're not alone and he wants America to win. June 23rd via

23:16

Reuters, an official of the US State Department, which didn't reply to Gamers Nexus except the department did send us three out of office autoresponders, warned of Deepseek military and intelligence operations and warned of the use of shell companies, they called them, in Southeast Asia to circumvent export restrictions. The report

23:32

mentioned that Deepseek had quote large volumes and quote, of high-end H100 ships, which are banned in China. Nvidia didn't like that and responded to Reuters and stated, quote, "We do not support parties that have violated US export controls or are on the US entity lists." Also saying, quote, "With the

23:47

current export controls, we are effectively out of the China data center market, which is now served only by competitors such as Huawei." Turning a blind eye to the situation, Nvidia added, quote, "Our review indicates that Deepseek used lawfully acquired H800 products, not H100." End quote. 3 days

24:03

later on June 26th, the information reported that Deepseek's next AI model had been delayed due to a shortage of NVIDIA AI GPUs in China. This directly contradicts Juan's comments that export controls, according to him, do not work.

July 4th, Bloomberg reported that the

24:19

Department of Commerce, which also did not reply to Gamers Nexus's emails, was preparing new export controls on Malaysia and Thailand to reduce chip smuggling. Interestingly, Singapore, which now comprises a significant portion of Nvidia's revenue, was not on that list despite being a known

24:35

smuggling passrough. July 10th, Bloomberg reported that Juan and Trump were scheduled to meet again ahead of the CEO's planned trip to China.

Days later, on July 14th, Nvidia confirmed that it will resume sales of H20 chips to China with Juan stating, quote, "The US government has assured Nvidia that

24:51

licenses will be granted. Nvidia hopes to start delivery soon." End quote.

On the 15th, a CNBC business news anchor said this. He really just pulled off something that few tech cos have managed.

He played both Washington and Beijing and he won. Jensen has stayed disciplined and

25:07

diplomatic with a clear message and that is Nvidia's dominance serves America's interest. On the 28th, the Financial Times reported that the US Commerce Department was not going to make tough moves to tighten export controls to China.

This is in spite of several congressional members warning the administration not

25:23

to loosen the US export controls for AIG GPUs. Several national security experts also voiced concern by sending a letter to the US Commerce Department which read, quote, "We write to express our deep concern over the recent decision to resume exports of Nvidia's H20 chips to China.

As policy makers and

25:38

professionals with a background in national security policy, we believe this move represents a strategic misstep that endangers the United States economic and military edge in artificial intelligence." End quote. On August 11th, we got the comments about the US potentially allowing Blackwell chips to be sold to China as well for 30 to 40 or

25:56

50% of a revenue share. And that brings us to today.

Nvidia might have started off as a much more humble company. But it has become a savvy political player in a game that is seemingly pay to win.

That seems only fitting for a gaming company to be particularly good at

26:12

payto-in games. There's no one better equipped to play that game than actually the most valuable company by market cap in the world.

Now at $4 trillion, led by a man whose net worth is estimated at $148 billion. Nvidia knows when to bite its ton and how to effectively appeal to

26:28

the ego of politicians of all parties and all countries. We think Nvidia is playing all sides.

We think it's greedy, manipulative, and carefully employs propaganda, such as its use of this fake news playbook for news which was literally reality. But we don't think

26:43

Nvidia has a particular set of beliefs beyond just making more money. We think Nvidia will sell anybody out to make a buck.

Nvidia is in the big leagues now. It's not like the 1080 Ti.

inside of one month reportedly paying $1 million to a sitting president after which followed the unlock of $5.5 billion of lost H20

27:01

revenue followed next by a 15% split of that unlock going to the US government is what raises these new questions of Nvidia's integrity but we've kind of already talked about that over the last year now as for the actual black market side of things and smuggling that was a cool

27:17

story this was like crazy exciting to work on I loved actually being out in the field and meeting all these people like we we set up some of it in advance, but a lot of people we met on the ground there, we were able to like actually get to the bottom of something. It's pretty cool.

We've learned that common methods

27:32

include factory so-called QC defects or sometimes real ones where they repurpose hardware that might otherwise have been written off by especially Nvidia as just trash. But people with the right skills can reclaim it.

We learned about the other method of hand carrying items in like with students or actual smugglers

27:48

on the ground in the US facilitating the transfer of the GPUs. And of course, we learned about suppliers through thirdparty countries among other methods.

So, we did answer the question that we had going into this, which was all about how these cards get in. Again, we loved working on this story and

28:05

meeting all of these unique people. Each person played a key role in helping us find the next person and allowing us to complete the first public on record start to finish cataloging of a smuggling pipeline for high-end silicon.

Our sincere thanks to everybody who made

28:20

this possible and appeared on camera with us, including our viewers who funded it. And a big thanks to my team working on this project.

Vitali, Andrew, and Mike spent about a month training Tannon how to use a camera cuz we needed a camera operator for this trip, and he's a writer. The tally helmed overall

28:36

production, organization of the files, and cutting of the 12 hours of footage initially. Andrew led the artistic efforts, worked on the new products, did the great scene transitions you saw and general graphics.

Tannon was on the camera and worked with me in Asia for three weeks on this story and during a

28:52

lot of travel. Tim on the team handled the bulk editing of theformational sections and the timeline for this video.

And Ben compiled hundreds of pages of research and worked directly with me throughout the process to figure out what the hell this story was going to be. A thanks to Ryan for doing the

29:09

black market logo as well and all the other contributions in between like from Aaron Pauly and Matthew Kitihifi on some of the music contributions. Thanks for watching.

Subscribe for more. We have more of these in the works.

You can go to store.gamesac.net to support this directly. Thank you.