A new study shows that most developers on DEV.to aren't using the recently added AI disclosure labels, and a significant chunk of 'AI-Assisted' posts came from a single, suspicious source. This reveals challenges in tracking AI content on open platforms.
Remember when DEV.to introduced those new AI disclosure labels a few weeks ago? Well, a recent study looking at seven weeks of data suggests most developers aren't really using them, and the numbers are a bit skewed by an unusual event.
DEV.to added three options on August 26: «Hand Written», «AI-Assisted», and «Fully Autonomous», so authors could signal how much AI was involved in their posts. A new analysis of over 300,000 posts published since the launch shows some surprising results. The vast majority, 95.9%, had no label at all. Out of the labeled posts, only a tiny 0.3% were marked «Hand Written», 2.6% were «AI-Assisted», and 1.2% were «Fully Autonomous». It's also clear that almost no one bothered to go back and label their old content.
But wait, there's a twist! It turns out a huge chunk of the posts labeled «AI-Assisted»—nearly a quarter of them, to be exact—came from a single, odd source. The study found 1,912 «AI-Assisted» posts, all with the exact same three tags: 'massage training brussels'. These posts appeared in a rapid burst over just a few days in late September, often minutes apart, and remarkably, none of them had any reactions or comments.
This single burst significantly skewed the statistics. Without these unusual posts, the share of all labeled content would drop, and the «AI-Assisted» category would shrink from 2.6% to just 2.0%. More importantly, this burst made it look like the use of «AI-Assisted» labels quadrupled in one week, creating a fake trend where none truly existed.
What does this mean for us, the readers and content creators? It shows that even with good intentions, tracking AI usage on open platforms is tricky. Just one active account can drastically change how the overall numbers look and create misleading trends. So, next time you see platform-wide statistics, remember to consider if a single, dominant source might be influencing the data.
DEV.to added three options on August 26: «Hand Written», «AI-Assisted», and «Fully Autonomous», so authors could signal how much AI was involved in their posts. A new analysis of over 300,000 posts published since the launch shows some surprising results. The vast majority, 95.9%, had no label at all. Out of the labeled posts, only a tiny 0.3% were marked «Hand Written», 2.6% were «AI-Assisted», and 1.2% were «Fully Autonomous». It's also clear that almost no one bothered to go back and label their old content.
But wait, there's a twist! It turns out a huge chunk of the posts labeled «AI-Assisted»—nearly a quarter of them, to be exact—came from a single, odd source. The study found 1,912 «AI-Assisted» posts, all with the exact same three tags: 'massage training brussels'. These posts appeared in a rapid burst over just a few days in late September, often minutes apart, and remarkably, none of them had any reactions or comments.
This single burst significantly skewed the statistics. Without these unusual posts, the share of all labeled content would drop, and the «AI-Assisted» category would shrink from 2.6% to just 2.0%. More importantly, this burst made it look like the use of «AI-Assisted» labels quadrupled in one week, creating a fake trend where none truly existed.
What does this mean for us, the readers and content creators? It shows that even with good intentions, tracking AI usage on open platforms is tricky. Just one active account can drastically change how the overall numbers look and create misleading trends. So, next time you see platform-wide statistics, remember to consider if a single, dominant source might be influencing the data.
