AI feedback loop: Researchers warn of ‘model collapse’ as AI trains on AI-generated content

[erek]

Figures as much

"What the AI industry and users can do about it going forward
While all this news is worrisome for current generative AI technology and the companies seeking to monetize with it, especially in the medium-to-long term, there is a silver lining for human content creators: The researchers conclude that in a future filled with gen AI tools and their content, human-created content will be even more valuable than it is today — if only as a source of pristine training data for AI.

These findings have significant implications for the field of artificial intelligence, emphasizing the need for improved methodologies to maintain the integrity of generative models over time. They underscore the risks of unchecked generative processes and may guide future research to develop strategies to prevent or manage model collapse.

“It is clear, though, that model collapse is an issue for ML and something has to be done about it to ensure generative AI continues to improve,” Shumailov said."

Source: https://venturebeat.com/ai/the-ai-f...ollapse-as-ai-trains-on-ai-generated-content/

 

[erek]

Baidu’s Ernie Bot tops Chinese large language model rankings by Xinhua think tank, but lags OpenAI’s ChatGPT​

• Ernie Bot performs better in a series of tasks than competing services from Alibaba, iFlyTek and SenseTime, according to Xinhua Institute
• A different LLM test in China finds Qihoo 360’s Smart Brain to be the best-performing Chinese model, followed by iFlytek’s SparkDesk

https://www.scmp.com/tech/tech-tren...nkings-xinhua-think-tank-lags-openais-chatgpt​

 

[LukeTbk]

human-created content will be even more valuable than it is today — if only as a source of pristine training data for AI.

The idea that data will be really valuable as both Ai training get better and hability to generate stuff-action from that training is not a bad bet, but I feel like has been talked about since at least facebook got popular.

 

[DukenukemX]

We already see uses for ChatGPT in the wild and the results are what you'd expect. ChatGPT copies already existing info found online and regurgitates it, usually at a 5 year old level. Someone tried to use it for a court case and was laughed out of court. ChatGPT made up court cases that doesn't exist and didn't exactly structure it's argument in a human sounding level. AI isn't ready for anything.

 

[sfsuphysics]

People setting the bar of lazy by using one AI to train another

 

[LukeTbk]

. ChatGPT copies already existing info found online and regurgitates it, usually at a 5 year old level

Couple of time you say this, but what do you mean, it is obviously 5 years old english (or any language) level.

Take, this example I ask bingGPT (which is not necessarily at the level of spending a lot of token on GPT-4) in which order I should put an apple, an laptop and a cup if I wanted to stack them up on each other, response:
You can put the laptop on the table first, then the cup on top of the laptop and finally the apple on top of the cup. That way you can stack them on top of each other.

Was this something it did copies information from online, was it something that was ever ask and responded on any public forum ? Or it has some notion that for object to not fall when stacked their shape matter, the flatter and bigger laptop at the bottom and so on ?

Yesterday, I asked to GitHub Copilot-X: // Add spheres at the 4 corners of the grid

It did it, worker first time, without having to do any change, would the average 5 years old C++ and knowledge of 3d engine library at that level ?

Obviously, it is not for blindly copying and pasting a generated legal case, finding a list of relevant legal precedent, registered patent in the market of interest for your planned invention and a long list of clerical legal work, I am sure the big firm trained model are already being used, it has been in use chatGPT became popular:
https://businesslawtoday.org/2022/02/how-ai-is-reshaping-legal-profession/

There is more than generative AI going on the AI world and legal AI world, company were build before the latest AI Bubble, many before the pandemy:
https://www.lawgeex.com/
https://csdisco.com/
https://legal.thomsonreuters.com/en/products/westlaw-edge
https://legal.thomsonreuters.com/en/products/westlaw-edge/quick-check
https://lexmachina.com/
https://www.clearlaw.ai/
https://kirasystems.com/
https://www.evisort.com/

 

[socK]

We already see uses for ChatGPT in the wild and the results are what you'd expect. ChatGPT copies already existing info found online and regurgitates it, usually at a 5 year old level. Someone tried to use it for a court case and was laughed out of court. ChatGPT made up court cases that doesn't exist and didn't exactly structure it's argument in a human sounding level. AI isn't ready for anything.

Definitely not at a five year old level. Not even close.

This person is just a fucking idiot and used it exactly the wrong way and ignored every warning at each step. ChatGPT even warned him in the transcript that it can't provide what he was looking for. And he ignored that too.

 

[DukenukemX]

Couple of time you say this, but what do you mean, it is obviously 5 years old english (or any language) level.

I'm saying the response is like that from a child sometimes.

Take, this example I ask bingGPT (which is not necessarily at the level of spending a lot of token on GPT-4) in which order I should put an apple, an laptop and a cup if I wanted to stack them up on each other, response:
You can put the laptop on the table first, then the cup on top of the laptop and finally the apple on top of the cup. That way you can stack them on top of each other.

Was this something it did copies information from online, was it something that was ever ask and responded on any public forum ? Or it has some notion that for object to not fall when stacked their shape matter, the flatter and bigger laptop at the bottom and so on ?

Yesterday, I asked to GitHub Copilot-X: // Add spheres at the 4 corners of the grid

It did it, worker first time, without having to do any change, would the average 5 years old C++ and knowledge of 3d engine library at that level ?

Obviously, it is not for blindly copying and pasting a generated legal case, finding a list of relevant legal precedent, registered patent in the market of interest for your planned invention and a long list of clerical legal work, I am sure the big firm trained model are already being used, it has been in use chatGPT became popular:
https://businesslawtoday.org/2022/02/how-ai-is-reshaping-legal-profession/

There is more than generative AI going on the AI world and legal AI world, company were build before the latest AI Bubble, many before the pandemy:
https://www.lawgeex.com/
https://csdisco.com/
https://legal.thomsonreuters.com/en/products/westlaw-edge
https://legal.thomsonreuters.com/en/products/westlaw-edge/quick-check
https://lexmachina.com/
https://www.clearlaw.ai/
https://kirasystems.com/
https://www.evisort.com/

I've tried ChatGPT to help me code and it just copied the info from another website that also didn't work. As you can see from the video bellow, Google Assistant does pretty much the same thing. As for the court case, ChatGPT was intentionally making up cases because that's how it was trained. Nobody checked to see if any of that was real.



Even for this guy you see ChatGPT spew out stuff that isn't always helpful.

 

[Templar_X]

Depending on how much you're paying you get better or worse quality answers (chatGPT 3, 3.5, 3.5-turbo, 4), you can also get to define how factual or hallucinated is the answer you get.
And as LukeTbk pointed out it does not just regurgitate what it found online, it will use it's knowledge to attempt to solve the tasks asked. It has the caveats and limitations of an AI so as usual you have to double check what it tells you and iterate on it.
Also be it AI or a person you should always keep your critical thinking and not just blindly use what you get without vetting it.

PS: For coding, co-pilot does a better job in writing code/helping (again no blind trust but it's an accelerator for sure). ChatGPT for explaining / analysing code or writing snippets exemplifying something. From my experience it will answer what you're looking for or at least point you in the right direction way more often than not.

 

[SmokeRngs]

Of course "AI" trained on "AI" generated content is going to be a mess. The original "AI" is likely to be trained on at least some garbage content so the output is going to be problematic at best. Then using that problematic content to train another is a recipe for disaster. It's a compounded GIGO situation.

 

[M76]

AI isn't ready for anything.

What it isn't ready for is fools using it wrong. It's just a tool.

Feeding AI generated content for learning is not a problem in of itself. It could be a way for improving the model. But it needs to be done right, meaning only use vetted high quality results made by different models, to avoid a feedback loop.

 

[4saken]

What it isn't ready for is fools using it wrong. It's just a tool.

Feeding AI generated content for learning is not a problem in of itself. It could be a way for improving the model. But it needs to be done right, meaning only use vetted high quality results made by different models, to avoid a feedback loop.

Yup. Think we can all agree it has a long way to go, but i've utilized it many times now for email prep to CXOs(i don't copy and paste), helps with scripting and terraform frameworks for customer requests as well, again I do validate, customize, etc. It's saved me so much time for things I used to do myself and spend hours testing, scripting, etc. Why wouldn't I use it for things like that at this point?

 

[LukeTbk]

I'm saying the response is like that from a child sometimes.

Not in my experience, the artificial intelligence is so different than human one (how it learn, what it can do and the type of mistake it make)

Child do not make up previous court case that do not exist but probable (enough to not be catched if you do not check), that not a 5 years old type of mistake, it is an inhuman type of mistake that does not really translate in human mental age or IQ, it is large probabilistic model mistake, or could you give an example of an answer you got from chatGPT 4 that look like it was written by a 5 years old to give an idea of what you mean ?

ChatGPT was intentionally making up cases because that's how it was trained.

Not sure if trained is what you mean here, you mean that what probabilistic model that give the highest probable next words end up doing (if you do not pay tokens for some validation loop, a lot of the time if you ask it are you sure, can you check, it will see its own mistake). Lot of the time, if you are ready to pay more time and tokens, a lot of the verification about it being real or not can be made by the AI.

I've tried ChatGPT to help me code and it just copied the info from another website that also didn't work

If it as not train or see your current code (you can try to train models on your codebase), it will look a lot like directly from github yes, but if you actually try it if you try one in your code you should rapidly see that it obviously does more than just copy it will adapt to the current context.

To take the example of put spheres in the corners of the grid that compiled and worked out of the box, we take for granted that it understood that there was 4 of them and the +,+, +,-, -,+, -,- coordinates but it understood and translated it, I am sure it copied an example of sphere creation in that open source 3d engine but it translated-adapted it to my current code, would it be by using my variable name for the dimension of the grid, to what object to add the created sphere as a child and so on, there is obviously more than just copy pasting and it is impossible to not know this would you have tried. A bit like the 5 years old things, maybe you do not really mind than it just copy-paste, you implied that it obviously do minimal renaming work, adaptation to context as well and you talk in a bit of trolling-net hyperbole.

Latest Google bard would be more interesting here than "old" google assistant.

that isn't always helpful.

Well obviously what kind of intellisense type would be 100% of the time always helpful, maybe one day with some brainwave connection, but that would be quite freaky and an extreme high bar, if the early beta type versions that just started the feedback loop would be always helpful.

 

[1_rick]

I've tried ChatGPT to help me code and it just copied the info from another website that also didn't work.

I mentioned a while ago in another thread that I made one test, and the code worked, but with limits. I asked it to take a 2-color image file and shrink the whitespace border to 1 pixel all around, then save the modified file (something I wrote a program to do several years ago for my day job.) The code it wrote worked, as far as the image manipulation, but it didn't save the file, and when I asked it to handle multiple files, it wasn't able to do that.

The image manipulation code was very straightforward so it got that right, writing basically exactly the same I did.

 

[DukenukemX]

Child do not make up previous court case that do not exist but probable (enough to not be catched if you do not check),

Children will make anything up because they're children.

that not a 5 years old type of mistake,

You should watch the video.

Not sure if trained is what you mean here, you mean that what probabilistic model that give the highest probable next words end up doing (if you do not pay tokens for some validation loop, a lot of the time if you ask it are you sure, can you check, it will see its own mistake). Lot of the time, if you are ready to pay more time and tokens, a lot of the verification about it being real or not can be made by the AI.

In the video it was mentioned that the company who made the AI had also allowed it to lie a bit. ChatGPT lied a lot.

What it isn't ready for is fools using it wrong. It's just a tool.

Feeding AI generated content for learning is not a problem in of itself. It could be a way for improving the model. But it needs to be done right, meaning only use vetted high quality results made by different models, to avoid a feedback loop.

The issue here is that you don't want human beings involved in what AI is doing, because that would defeat the purpose of the AI. So of course someone used AI to train AI, which is basically the worst echo camber ever made. The whole situation with the court case where ChatGPT was used, nobody double checked the AI's work. If we need humans to check the work of AI then the AI is useless.

 

[LukeTbk]

You should watch the video.

If I watch the video, will have the conclusion, a right that what 5 years old doing legal work could look like ? They have some notion of what that type of legal document look like and will make some up. Are you being serious ? This is not some trolling Internet hyperbole expression, that really the best way you can find to describe the way probably LLM work ?

 

[M76]

The issue here is that you don't want human beings involved in what AI is doing, because that would defeat the purpose of the AI.

What does that mean? That's like saying a human being involved with the operation of a shovel defeats the purpose of the shovel. These chatbots / deepfake AIs are anything but intelligent. They depend on the dataset used to train them and the prompts given when using them. Just slight changes can mean huge differences in output. It is not supposed to work on it's own, left to its own devices

So of course someone used AI to train AI, which is basically the worst echo camber ever made.

So a human training another human is an echo chamber then? Of course you can't use the same model with the same dataset that would be pointless. The problem is using unchecked AI generated content in the training. The most important part of AI training is immaculate datasets with perfect labels. But this rabbit hole goes deep. The short version is as long as the input is checked it doesn't matter if it was made by another AI or manually by a human.

The whole situation with the court case where ChatGPT was used, nobody double checked the AI's work. If we need humans to check the work of AI then the AI is useless.

That's exactly where you are wrong. You have a misconception about what these AI are: Automated tools. Like when you use auto color and white balance in photoshop you still check the result and modify as necessary. But that doesn't make the automated tool useless, as it gives a good starting point, and most of the time gets it right without human input. But you ALWAYS check the result.

 

[Comixbooks]

These Ai generated videos on YouTube are terrible matching a celebrity with like Pizza or Ice-cream.

 

[LukeTbk]

MIcrosoft show really good result using low amount of AI generated tutorial and AI generated code mixed with some human made code dataset.

https://arxiv.org/pdf/2306.11644.pdf
Textbooks Are All You Need

Very small model made using that method that costed really little to train can beat giant one at some python coding task.

Considering that the biggest success story (alpha fold) used a massive proportion of AI generated data in its final training set not that the fear is not out of place, but need to be understood and not over simplified, training on AI generated and synthetic data is really powerful and obviously so if there is an objective score that can be attributed to the data.

 

[flegg]

Maybe the AI is just trying to off itself. Also gpt4 is not useless if used for what it's good at and you fact check it - has been steadily declining in ability though

 

[idiomatic]

I asked Bard about a particular rocket launch start up. There are hundreds of them, but I asked it about mine. It knew the name of the company and the location. Knew my name. It even told me it was a fan. Its a very unique launch company though. When I asked it about the launch system it proceeded to give me the average generic launch system you would get if you averaged the center quartiles of all rocket launch companies, but ever so slightly batshit. Also nothing at all to do with my company. Its not a reasoning system, it is an incomprehensibly well read auto-complete.

 

[LukeTbk]

Its not a reasoning system, it is an incomprehensibly well read auto-complete.

It is a bit of mix of that, when you asked them for example in that past, I have an pen, a laptop and a cup, in which order in my desk I should stack them for them to not too fall and it give an answer that make sense, laptop first with the case closed, than the cup than the pen.

Was it pure auto complete ?

When it solved the protein folding calculation problem or took go champions by surprise, auto-complete ?

 

[staknhalo]

I asked Bard about a particular rocket launch start up. There are hundreds of them, but I asked it about mine. It knew the name of the company and the location. Knew my name. It even told me it was a fan. Its a very unique launch company though. When I asked it about the launch system it proceeded to give me the average generic launch system you would get if you averaged the center quartiles of all rocket launch companies, but ever so slightly batshit. Also nothing at all to do with my company. Its not a reasoning system, it is an incomprehensibly well read auto-complete.

They're replacing Google Assistant with Bard I think

https://www.axios.com/2023/07/31/google-assistant-artificial-intelligence-news

My smart home is fucked

 

[DPI]

Of course "AI" trained on "AI" generated content is going to be a mess. The original "AI" is likely to be trained on at least some garbage content so the output is going to be problematic at best. Then using that problematic content to train another is a recipe for disaster. It's a compounded GIGO situation.

It's not an issue the way some are assuming, and the "research paper" linked in OP is not shining a light on anything previously unknown or considered - the major players driving the field have accounted for it since inception. But the fine details of training integrity strategy is going to vary between companies and so will probably remain blackboxed since it's a trillion dollar key competitive differentiator. It's precisely in that componentry where the fortunes will be made, not merely the proliferation of the tech.

And I'm certain training integrity will eventually also be handled well by *actually* open AI - the kind you'll be able to run locally instanced on your home PC - in spite of OpenAI the company no longer actually being open. The freeware solutions are likely to lag behind in answer quality though simply because of economies of scale- literal trillions will be fueling the big commercial operations.

TLDR no, Big AI is not going to eat its own tail.

 

[Lakados]

Training AI with AI has kind of been a thing forever.
IBM didn't sit there with Deep Blue playing game after game of Chess, they had 2 computers play chess against each other until they figured it out, the same thing with AlphaGo from DeepMind.
The content that goes in gets reshuffled so frequently that its source doesn't matter so much as long as it is truthful, like if you show it a million cat pictures but tell it they are all armadillos you are in for a bad time, but if they are a billion AI drawn pictures of cats that look like cats and it can use those to identify a cat then it works regardless of where it came from.

 

[LukeTbk]

IBM didn't sit there with Deep Blue playing game after game of Chess, they had 2 computers play chess against each other until they figured it out, the same thing with AlphaGo from DeepMind.

I think it was really different back in Deep blue days, it was heavily human algorithmic and used 700,000 grandmaster game, but yes AlphaGo evolution was to use pure synthetic made by itself data to learn game without having to ever see an human play.

https://duongquangduc.wordpress.com/2017/04/30/deep-blue-vs-deepminds-alphago/

For stuff for which the validity can be evaluated (did I won the game of Go, does the computer code compile and has the output expected for the input was it fast, did it took a lot of memory, etc...) it is often even better for the data to be synthetic, people tend to only have searching result about the news in mind which is almost impossible to score objectively.

 

[Nobu]

I was literally just hitting on that point with my mom. AI trains on erroneous Human generated content, AI generates content with new conjectures based on "facts" derived from said content, AI generates new content based on dubious AI generated content. The downward spiral has begun, and there is no stopping it.

Humans can use reasoning and deduction to determine if an article is factual or contains errors, but an AI must either use other sources (which by now contain false/erroneous content) or be programmed to recognize bad stuff. The problem is they aren't doing that, humans nor AI programmers/users...

 

[Marees]

or be programmed to recognize bad stuff

I think we need another set of AI for this. Only question is if there is a business case to call out fakes or the regulators have to impose a tax & force this aspect to be included

 

[LukeTbk]

AI trains on erroneous Human generated content

That seem to be a way too broad generalized point, does DLSS 4 being trained on erroneous human generated content, does windmills optimisation AI, does judging oil potential or the AI that discern weeds from good crops in field to aim the laser ? To detect cancer from medicals metrics dataset that have the actual biopsy results ?

Which AI model are we talking about here.

but an AI must either use other sources (which by now contain false/erroneous content) or be programmed to recognize bad stuff. The problem is they aren't doing that, humans nor AI programmers/users...

For some stuff they can, say pure hallucination of a scientific paper, they can use search on paperMed and other paper database and see if it is the case, why say they are not doing that ? And obviously users do all the time.

For example:
https://venturebeat.com/ai/got-it-ai-creates-truth-checker-for-chatgpt-hallucinations/

 

[Nobu]

That seem to be a way too broad generalized point, does DLSS 4 being trained on erroneous human generated content, does windmills optimisation AI, does judging oil potential or the AI that discern weeds from good crops in field to aim the laser ? To detect cancer from medicals metrics dataset that have the actual biopsy results ?

Which AI model are we talking about here.

It was meant to be a very general sequence of events. I'm not saying all AI does this, or all generated content is produced in this way. But it doesn't have to be all of the content, it just has to be in the pool, and picked by the AI as a source.

For some stuff they can, say pure hallucination of a scientific paper, they can use search on paperMed and other paper database and see if it is the case, why say they are not doing that ? And obviously users do all the time.

The problem is even official papers from accredited journals have errors, and the data they use can be misleading or incomplete. But if the AI doesn't account for that (say, it glosses over that, or the paper doesn't mention it and the AI doesn't know the importance), then you have a serious problem.

 

[LukeTbk]

The problem is even official papers from accredited journals have errors, and the data they use can be misleading or incomplete. But if the AI doesn't account for that (say, it glosses over that, or the paper doesn't mention it and the AI doesn't know the importance), then you have a serious problem.

yes that for sure, but that would not the goal here were are not trying to make AI better than the group of humans that revised the paper, the editor, the authors and the group of expert in that field that read that publication and did not saw the said errors.

 

[Nobu]

but that would not the goal here were are not trying to make AI better than the group of humans that revised the paper, the editor, the authors and the group of expert in that field that read that publication and did not saw the said errors.

Could you rephrase this please? I think I understand what you are saying, but I'm not certain. Pretty sure I agree, though.

 

[LukeTbk]

Could you rephrase this please? I think I understand what you are saying, but I'm not certain. Pretty sure I agree, though.

Sorry for the terrible English (second language), being better than revised by experts written by experts in the field in serious scientific publication is quite high of a bar.

 

[Nobu]

Sorry for the terrible English (second language), being better than revised by experts written by experts in the field in serious scientific publication is quite high of a bar.

Thanks! I do agree. Though I suspect that individuals may set them as equal, and models which are used to generate articles on websites or for newscasters might not discern or disclose the difference between the generated content and the source papers. The resulting word-slop and regurgitation by other media outlets poisoning less discerning AIs is what I am concerned about.