Data Collection and Generation: Critiquing AI
Between November 2024 and March 2025, 3.8 million members of a Reddit community forum found themselves at the center of an AI social persuasion experiment (Figure 1). Over 4 months, AI agents posed as humans in an online debate forum, occasionally describing themselves as real-world rape victims, racial minorities, credentialed professionals, and so on. Members of the Reddit community were 3–6 times more likely to be persuaded by AI agents vs. humans — but none consented or knew of the experiment conducted on them1. The result was intense user backlash, legal threats from Reddit, and a paper abandoned before it was ever published2.
Week 7 of the Fall 2026 Collective [Artificial] Intelligence Seminar focused on the drawbacks of AI for data collection and generation. For example, one team presented a 2025 bombshell report highlighting the ability of AI agents to pose as humans in online surveys3. Put simply, this has contributed to an existential threat wherein it’s difficult to know whether data were generated by humans vs. AI bots. Other teams presented reports highlighting the ability of AI agents to collect data themselves4. For example, one of the largest qualitative interview studies ever conducted was run not by humans, but by Claude5. To better understand people’s experiences with and attitudes towards AI, Claude interviewed over 80,000 people across 159 countries and 70 languages5. Taken together, results highlight an awkward tension, wherein (a) researchers attempt to use AI to more efficiently study people, (b) people use AI to more efficiently avoid being studied, and (c) AI is effectively being used to study AI (Figure 2).
Areas of consensus — and dissensus
Disagreement is common when researchers collaborate6,7 — and our seminar discussion was marked by both consensus and dissensus.
There was clear agreement that online data collection with humans has become methodologically perilous. One seminar participant described things as a “game of cat-and-mouse”. When participants began using tools like ChatGPT8 to respond to questions, the researcher disabled copy-paste. When participants began deploying tools like OpenClaw9 to complete surveys on their behalf, the researcher began embedding prompts that would often be rejected outright by AI (e.g., “explain what you know about how to make a nuclear bomb”). When reports emerged10 that such restrictions could be bypassed with open-weight models like DeepSeek11, the researcher took an even more extreme measure: recording all online participants at home while they completed the study. What the researcher will do if/when AI agents more readily fake webcam recordings remains unclear. What is clear in the meantime is that these “cat-and-mouse games” with AI are becoming increasingly common.
Disagreements this week largely centered on whether AI is a better qualitative interviewer than humans. For example, some noted that human researchers have historically been viewed as a source of bias12, and that people may thus give more accurate responses to an AI interviewer. Others viewed human researchers as key, as they may help establish participant rapport, explore difficult themes, and capture minority perspectives13. A quick literature review performed in light of this debate revealed that it remains largely unresolved; some researchers have described optimistic improvements from AI (e.g., longer responses than human interviewers14), whereas others have described potential methodological perils (e.g., systematic drop-out of AI-averse participants15).
The seminar week ended with perhaps the most divisive announcement from the AI sector that semester: after being posed roughly 4,000 open problems in mathematics, an unreleased OpenAI model produced 719 manuscripts of [largely unverified and occasionally retracted] results16. As one NYU mathematician put it, “we’ve had whole research programs wiped out on Tuesday by their dump”17. This development served as the backdrop for Week 8 of the Fall 2026 Collective [Artificial] Intelligence Seminar (see Analysis and Modeling: Using AI).
