Decoding Diffusion: From Grammatisation to the Diffusionisation of Knowledge
This transformation manifests across multiple domains, from creative industries to manufacturing, suggesting a broader reorganisation of labour and cultural production through algorithmic means. The political economic implications are significant, as diffusion models enable new forms of value extraction through the capture and commodification of human creative and cognitive capacities. These developments demand critique to understand how diffusionisation alters technical systems together with conditions of possibility of knowledge production in contemporary society (Berry 2014).
This article examines the technical foundations and social implications of diffusionisation. I argue that this process creates what can be conceptualised as diffusionism: a distinctive computational paradigm characterised by the continuous probabilistic transformation of knowledge through algorithmic systems. Where diffusionisation describes the technical processes of dissolution and reconstitution through diffusion models, diffusionism represents the broader social logic and epistemological conditions that emerge when these processes become key to knowledge production. This distinction is important for understanding how technical changes in knowledge processing connect to broader transformations in social relations under algorithmic capitalism. This has implications for labour, creativity and what we understand knowledge to be. Through analysis of concrete examples from robotics and creative industries, I demonstrate how diffusionisation intensifies and extends processes of algorithmic capture while transforming how knowledge is generated and reproduced. Under conditions of diffusionisation, knowledge production becomes subject to what can be understood as vector representation and latent space manipulation (Rombach et al. 2022). These mathematical abstractions allow artificial intelligence systems to blend, morph and generate new cultural forms through probability distributions rather than deterministic rules or simple reproduction.[1]
Understanding the relationship between diffusionisation and diffusionism is key to understand their distinct but interrelated roles in computational societies. Diffusionisation represents the technical processes through which knowledge and cultural forms are probabilistically transformed through computational diffusion processes. This includes the specific mathematical operations of adding and removing noise, the manipulation of latent spaces, and the generation of synthetic variations through machine learning systems. Diffusionism, in contrast, constitutes the broader social logic and epistemological conditions that emerge when these technical processes become important to knowledge production and circulation in society. Where diffusionisation describes concrete technical operations within artificial intelligence systems, diffusionism captures how these operations restructure social relations, labour processes and cultural production under algorithmic capitalism. This distinction is useful as it allows us to analyse both the specific technical mechanisms through which knowledge is transformed and their broader social implications. Diffusionism thus represents the aggregate of technical processes and rather a qualitatively new paradigm of knowledge production that emerges when diffusion models become central to how society generates, circulates and validates knowledge. This involves new forms of labour organisation, novel regimes of accumulation and transformations in how meaning and cultural forms operate under computational conditions.
Diffusion models represent a distinctive approach to computation that differs from traditional algorithmic programming. Where classical software development requires explicitly programmed rules and transformations, diffusion models employ neural networks trained through iterative noise addition and removal processes (see Dhariwal and Nichol 2021). The underlying computation still occurs through classical digital logic, but the way information is processed marks a significant departure. Rather than applying predetermined transformations, these models learn to traverse high dimensional latent spaces that encode probabilistic relationships within training data. This creates a form of computation that operates through learned statistical patterns rather than explicit rules. The parallel with physical diffusion processes inspired their development, in fact they mathematically simulate the gradual transformation of noise into structured information, analogous to how particles diffuse through space (Ho, Jain and Abbeel 2020). This computational architecture alters how knowledge production occurs. Instead of processing knowledge forms through predefined algorithms, diffusion models navigate probability distributions that encode the patterns and possibilities present in their training data. This represents a qualitative transformation in how computation intersects with knowledge production.
Vector representation and latent space manipulation constitute the technical foundations of diffusion processes in artificial intelligence systems. Vector representation transforms knowledge artefacts such as images, text, or behaviours into high dimensional numerical arrays (vectors) that can be computationally processed. These vectors capture patterns and features of the original content in an abstract mathematical form. The latent space refers to a compressed dimensional representation where similar knowledge forms are mapped close together, creating a continuous field of possibilities between discrete examples. Through manipulation of these vector representations within latent space, diffusion models can traverse the learned patterns to generate new variations. This process operates through iterative denoising, where random noise is gradually removed according to learned patterns, enabling the controlled generation of novel cultural forms that maintain coherent relationships to the training data. These technical processes alter how computation works and transform our entire understanding of cultural production and reception.
Where previous digital technologies maintained relatively stable distinctions between moments of creation and consumption, diffusion models collapse these boundaries through their continuous processes of probabilistic transformation. This technical architecture suggests the need to reconceptualise traditional frameworks for understanding how cultural forms circulate and generate meaning. The dissolution and reconstitution of cultural forms through latent space manipulation challenges established models of cultural transmission and reception that presupposed discrete moments of encoding and decoding. In particular, the technical operation of diffusion models problematises influential frameworks from cultural studies that sought to theorise how meaning circulates through media systems.
For example, Stuart Hall's (1973) influential encoding/decoding model theorised how meaning was encoded into cultural texts by producers and subsequently decoded by audiences, with the possibility for dominant, negotiated or oppositional readings. This framework presupposed discrete moments of production and reception, with cultural forms maintaining relative stability between these poles. However, under conditions of diffusionisation, this linear model becomes increasingly inadequate. Hall's encoding/decoding model, was outlined in his 1973 paper Encoding and Decoding in the Television Discourse, proposed that mass communication should be understood as a complex circuit of meaning production and reception. He argued that media messages are encoded with particular meanings by producers according to dominant ideological frameworks, but these messages must then be decoded by audiences who may interpret them differently based on their own social position and cultural context. Hall identified three potential positions from which decoding could occur: the dominant hegemonic position (accepting the preferred reading), a negotiated position (partially accepting but modifying the preferred reading), or an oppositional position (rejecting the preferred reading entirely). This model challenged linear transmission models of communication and recognising audience agency in meaning-making processes. It became important for cultural studies approaches to media analysis and audience reception research. However, as this article argues, the discrete moments of encoding and decoding that Hall identified become problematically blurred under conditions of diffusionisation where cultural forms exist in states of constant potential transformation.
While Hall's model provided insights into how meaning circulated under conditions of broadcast media, the emergence of diffusion models requires us to rethink these processes. The shift from discrete moments of encoding and decoding to continuous probabilistic transformation suggests a technical change but a qualitative transformation in how cultural forms operate. This transformation becomes particularly significant when we consider how diffusion models restructure possibility of stable meaning upon which traditional models of cultural circulation depended. The political implications of this shift are, as the technical architecture of diffusion models potentially forecloses certain forms of oppositional consciousness while creating new possibilities for algorithmic contestation.
Diffusionisation disrupts the encoding/decoding paradigm by dissolving the boundaries between production and reception into a continuous field of probabilistic transformation. Rather than encoding and decoding, we confront what could be understood as perpetual cycles of dissolution and reconstitution. Cultural forms no longer maintain stability between production and reception but exist in a constant state of potential transformation through latent space manipulation. The 'preferred reading' that Hall identified becomes replaced by probability distributions of possible variations, while the notion of oppositional readings gives way to prompt engineering that steers generative processes in unexpected directions. This suggests the need for a new theoretical framework that can account for how meaning operates under conditions of algorithmic generation and transformation. Where Hall's model traced the circulation of relatively stable cultural forms, we must now theorise how meaning persists or mutates through processes of probabilistic dissolution and reconstitution.[2]
The concept of porosity provides a framework for understanding how diffusion models create new forms of technical interpenetration. The technical process of diffusion itself operates through a form of computational porosity that is to its generative capacities. Just as Walter Benjamin and Asja Lācis identified how Naples' architecture created porous boundaries between public and private spheres, diffusion models generate forms of technical porosity. When diffusion models add noise to transform images or other cultural forms into pure mathematical abstractions, they create a state of absolute porosity where distinct forms become completely permeable to one another. This noisy latent space enables new forms of technical interpenetration, as previously discrete cultural forms dissolve into probability distributions that can flow into and through each other. The subsequent denoising process maintains this porosity while gradually reconstructing coherent forms, allowing features and patterns from different sources to merge and recombine in novel ways.[3]
This technical porosity is particularly visible in the latent spaces of neural networks, where cultural forms exist in states of perpetual potential transformation. The high-dimensional vector spaces that these systems construct create zones of radical porosity where the boundaries between different cultural forms become completely permeable. When generating new images or behaviours, the system traverses these porous latent spaces, allowing features from multiple sources to interpenetrate and combine. The gradual process of denoising maintains this porosity while reconstructing coherent forms, creating outputs that embed multiple sources within new synthetic wholes (Rombach et al. 2022).[4]
The transformation of knowledge production through diffusionisation is particularly visible in recent developments at the Toyota Research Institute, where diffusion models are being deployed to generate robotic behaviours. Rather than simply programming robots with prescribed movements, these systems use probabilistic diffusion processes to learn complex manipulations from human demonstrations. This represents a shift in how technical knowledge is constituted and transmitted under computational conditions. As the Economist explains,
Traditionally, robots are programmed with reams of computer code. This can be produced manually or created by remotely moving the robot’s arms and hands to replicate the actions required... Yet even 100 demonstrations are not enough to cover every eventuality, which is where diffusion comes in... For image generation, this involves adding random “noise” to a picture until it becomes unrecognisable and then reversing the process to learn the steps involved in generating a new, realistic image... For robot training, the AI uses the actions it has been taught to randomly generate potential new movements, which are then refined into useful actions that can deal with new environments (Economist 2024: 69-70).
Where traditional robotics required explicit programming of each movement and action, diffusion models allow robots to learn behavioural patterns through what could be understood as computational dissolution and reconstitution. The robot's actions are first encoded into abstract vector spaces, then probabilistically transformed through iterative noise addition and removal processes. This creates a form of technical knowledge that exists between human demonstration and algorithmic generation. Again, quoting the Economist,
Having got diffusion to work for a variety of different tasks, the researchers are now trying to bring hundreds of such tasks together into what they call a large behaviour model (LBM). This will be analogous to a large language model (LLM), which is used to power ai services like Chatgpt. Instead of generating answers to questions based on information which an LLM has been trained on, an LBM contains sets of behaviours which can be used to generate new behaviours (Economist 2024: 70).
The emergence of AI image generation systems such as Midjourney and Stable Diffusion exemplifies this shift in knowledge encoding and decoding (Rombach et al. 2022). For example, where illustrators and graphic designers previously relied on human skill and training mediated through digital tools, their work increasingly involves curating AI generated outputs through prompt engineering. This shift from direct creation to algorithmic curation represents more than automation, in fact it transforms creative practice. For example, in an interview with Tom May in 2024, Andy Taylor, chief creative officer at creative agency Trouble Maker explains that they've "been using AI... to bring to life personalised, reactive campaigns that adapt to multiple situations and markets... when we worked on a campaign centred around a global sports tournament, AI allowed us to create content that could instantly react to every possible fixture and result. Even down to changing the ethnic makeup of the crowd to reflect the local market" (May 2024). Similarly, Camm Rowland, chief creative officer at Kepler, outlines the extent to which AI is being utilised in agencies stating that "currently, Midjourney stands out for static imagery, while we anticipate Open AI's Sora to significantly impact our video production. For voice synthesis, Elevenlabs leads in quality, and we use both Gigapixel and Magnific for upscaling. We also employ Zapier for creating automations" (May 2024).
The political economy of diffusionisation becomes apparent in how it reshapes labour processes. Traditional programming labour is replaced by demonstration and prompt engineering, while the core generative work occurs through proprietary computational processes. This represents a new form of capital accumulation through the capture and commodification of behavioural and knowledge patterns via diffusion models. The logic of diffusionisation extends beyond creative industries to potentially transform sectors previously thought immune to algorithmic automation. In manufacturing, traditional assembly line programming could be replaced by demonstration based learning, where robots acquire complex manipulation skills through probabilistic pattern recognition. In healthcare, diagnostic processes might shift from rule based expert systems to diffusion models trained on vast archives of medical data and practitioner behaviour. Even professional services like law and accounting face potential transformation as diffusion models learn to generate legal documents or financial analyses from historical examples. The media and news industries also face huge challenges from generative artificial intelligence using these models, indeed "the trend towards disintermediation of news by tech platforms will continue" (House of Lords 2024: 32). This suggests a broader reorganisation of labour under algorithmic capitalism, where human expertise across industries is increasingly captured, dissolved and reconstituted through proprietary diffusion systems. This diffusional logic represents an intensification of cognitive capitalism, as formerly complex human skills and knowledge practices become subject to probabilistic dissolution and algorithmic reconstitution.
Diffusionisation operates as a form of labour extractivism that appropriates and commodifies human creative capacities in novel ways. Unlike traditional forms of digital labour exploitation which capture discrete outputs of human work, diffusion models extract the underlying patterns and potentialities of creative labour itself. The training data used to build these models represents countless hours of human knowledge and physical work, from which the systems learn to generate unlimited variations without compensation or attribution to the original labour. This extractive logic operates at two levels: first through the initial appropriation of training data derived from human knowledge production, and secondly through the ongoing capture of prompt engineering labour required to refine and curate the model outputs. The result is a new regime of accumulation where capital controls the products of creative labour and the generative capacity itself. This represents an intensification of what Marx identified as the real subsumption of labour under capital, where process of cultural production becomes internal to computational capital.[5] Through diffusionisation, labour is automated but transformed into a probabilistic resource that can be endlessly recombined and exploited through proprietary algorithmic systems.
Diffusionisation signals a qualitative shift in how knowledge is constituted under computational conditions. This suggests implications for how cultural forms are transmitted and reproduced in an era of algorithmic generation. The implications extend beyond individual practices to organisation of knowledge production. Professional identities built around traditional creative skills face potential deskilling as the core activity shifts from manual creation to prompt crafting and output selection. This recalls earlier transformations under industrial capitalism but operates at the level of epistemic and cognitive labour rather than physical production. Through the technical processes of noise addition and removal, these systems create new forms of computational porosity that enable the manipulation of knowledge.
Yet understanding diffusionisation requires more than technical or economic analysis alone. It demands new theoretical frameworks capable of grasping how these systems reshape conditions of possibility for knowledge production and cultural experience. It might be thought that diffusion models represent an extension of existing digital technologies rather than a qualitative break, or that their impact on knowledge production has been overstated. However, such arguments fail to grasp how diffusion transforms the relationship between knowledge and its reproduction. Where earlier digital technologies maintained discrete moments of production and reception, diffusion models create continuous fields of probabilistic transformation that alter ontology of knowledge.[6]
Others might contend that the labour extractivism identified here could be addressed through better regulation or alternative ownership models. While such reforms are important, they do not address how diffusion models reshape the relationship between human creative capacity and technical reproduction. The conversion of knowledge into probability distributions that can be endlessly recombined represents more than a new form of exploitation; it signals a transformation in how knowledge itself is constituted under algorithmic conditions.
As an example, regulatory frameworks, such as the European Union's AI Act 2024, attempt to address the challenges posed by artificial intelligence through traditional governance mechanisms. The Act proposes a risk based approach to AI regulation, requiring 'high risk' systems to meet requirements for transparency and human oversight. However, such legislation misunderstands how diffusionisation operates. By focusing on discrete algorithmic systems and their outputs, these regulatory approaches fail to grasp how diffusion models transform conditions under which meaning and knowledge are produced. The Act's emphasis on transparency and explainability presumes stable computational processes that can be understood through traditional analytical frameworks. Yet diffusionisation creates conditions where the boundaries between algorithmic and human agency become increasingly porous, making traditional notions of transparency inadequate. While the Act attempts to regulate artificial intelligence as a technical system, it cannot address how diffusionisation restructures social relations and knowledge production at a key level. The focus on technical compliance and risk management obscures the deeper political economic transformations enabled by diffusion models. This suggests the need for new regulatory paradigms that can address technical systems and the broader social implications of probabilistic knowledge production under algorithmic capitalism.[7][8]
The processes involved in computations diffusionisation and the paradigm of diffusionism offered here represents an initial attempt to theorise these changes, drawing on critical theory to illuminate both the constraints and possibilities created by these new technical systems. The task ahead is to develop both theoretical tools and practical interventions that can help reshape these systems toward more democratic and emancipatory ends. This requires moving beyond simplistic narratives of automation or technological determinism to understand how diffusion models alter technical processes and the nature of knowledge production and social relations under algorithmic capitalism.[9] Only through such critical engagement can we begin to imagine how these powerful new technical capacities might be redirected toward new digital forms of life rather than serving the logic of capital accumulation.
Blogpost by David M. Berry
Notes
[1] Adorno's concept of the shudder (Schauer), developed in Aesthetic Theory, describes a primordial aesthetic experience where the subject encounters its own natural being through art. For Adorno, this aesthetic shudder represents "a memento of the liquidation of the I, which, shaken, perceives its own limitedness and finitude." This experience (Erfahrung) stands in opposition to "the weakening of the I that the culture industry manipulates" (Adorno 2013: 333). Where Adorno identified the shudder in aesthetic experience as a moment where consciousness is manifested through "the utmost tension", diffusion models systematise this process technically, transforming cultural forms into pure mathematical noise before reconstituting them. However, unlike the aesthetic shudder which Adorno saw as potentially emancipatory, this technical dissolution serves algorithmic capital's drive to transform cultural forms into manipulable probability distributions. The systematised dissolution in diffusion becomes not a moment of potential freedom but rather an industrial process of cultural liquidation and reconstitution. As Adorno argues, "consciousness without shudder is reified consciousness. That shudder in which subjectivity stirs without yet being subjectivity is the act of being touched by the other" (Adorno 2013: 437). In diffusion models, this potential for being "touched by the other" is replaced by algorithmic processes that dissolve and reconstitute cultural forms according to computational logics rather than human experience.
[2] This disruption of the encoding/decoding paradigm signals more than a technical transformation; it represents a reconfiguration of the possibilities for democratic politics and resistance. Where Hall's model identified spaces for oppositional readings and counter hegemonic practices, diffusionisation threatens to subsume these critical capacities within algorithmic processes. The dissolution of stable cultural forms into probability distributions does change how meaning circulates, but potentially undermines conditions for political consciousness and collective action. The political stakes become clear when we consider how diffusionisation affects the formation of political subjects. Traditional models of political consciousness presumed relatively stable cultural forms through which subjects could recognise their conditions and develop critical perspectives. When cultural forms exist in perpetual probabilistic transformation, the cognitive mapping required for political consciousness becomes increasingly difficult. This suggests that new forms of political organisation and resistance must develop that can operate within conditions of computational flux rather than stability.
[3] Benjamin and Lācis's concept of porosity, developed in their 1925 essay Naples, identified how architectural forms enabled the interpenetration of public and private spaces in Neapolitan life (Benjamin and Lācis 1986). This architectural porosity finds its contemporary parallel in how computational systems create fluid boundaries between knowledge objects. Benjamin and Lācis's concept of porosity gains new technical specificity when applied to diffusion processes. Where they identified architectural porosity in Naples' physical spaces, diffusion models create mathematical spaces of absolute porosity through their noise-addition processes. This technical porosity operates not through stone and concrete but through the transformation of knowledge into pure mathematical noise, enabling forms of interpenetration and recombination. The latent spaces of neural networks represent perhaps the most radical manifestation of computational porosity, where discrete forms dissolve into probability distributions that can flow freely into one another.
[4] Adorno theorises how nature and history interpenetrate dialectically in his 1932 essay The Idea of Natural History (Adorno 1984). This dialectical understanding helps illuminate how diffusion processes create spaces where technical and social forces become mutually constitutive. The transformation of knowledge into mathematical noise and back represents precisely the kind of dialectical interpenetration that Adorno identified, though now operating through computational rather than historical processes. This suggests how diffusion models exemplify new forms of technical-social porosity that require careful dialectical analysis to understand their full implications.
[5] The labour theory of value, developed by classical political economists and significantly elaborated by Marx, provides a more appropriate theoretical framework for understanding value creation under diffusionisation than marginal utility theory. While marginalist approaches focus on subjective consumer preferences and market equilibrium, they cannot adequately account for how diffusion models extract and commodify accumulated human labour. The value embedded in these systems derives not from marginal utility but from the vast quantities of human creative and cognitive labour captured in training data, combined with the ongoing labour of prompt engineering and curation. Indeed, it could be argued that diffusion models problematise marginalist concepts of scarcity and diminishing returns, as they enable unlimited generation of variations from captured labour with low additional input costs. This aligns more closely with Marx's analysis of how fixed capital (in this case, the diffusion model) absorbs and mobilises human labour power. The labour theory of value helps us understand how diffusionisation intensifies capital's appropriation of labour by transforming discrete instances of human work into infinitely reproducible probabilistic patterns.
[6] The emergence of diffusionisation raises challenges for intellectual property regimes, particularly copyright law which was designed around notions of original authorship and discrete creative works. When diffusion models transform existing works into probability distributions that can generate endless variations, traditional concepts of originality and substantial similarity become increasingly problematic. Copyright law traditionally protects 'original works of authorship', but diffusion models problematise both originality and authorship by creating a technical continuum between existing works and their algorithmic variations. This suggests that copyright, conceived in an era of mechanical reproduction, cannot adequately address how cultural forms are now probabilistically dissolved and reconstituted. The extractive logic of diffusion models further undermines copyright's economic function of incentivising creation, as these systems can generate unlimited variations from captured training data without compensating original creators. Some jurisdictions have attempted to address this through broad licensing or opt-out approaches to the computational use of copyrighted works, but such reforms fail to grasp how diffusionisation transforms the ontological conditions that copyright law presupposes. This points toward a deeper crisis in intellectual property law under algorithmic capitalism, where traditional categories of creative labour and cultural ownership become increasingly inadequate to regulate new forms of computational knowledge production (see House of Lords 2024: 35-41). The likely result is an intensification of corporate control over cultural production as companies leverage their ownership of diffusion models and training data to circumvent traditional copyright restrictions while simultaneously using intellectual property law to protect their algorithmic systems.
[7] The concept of explainability becomes particularly problematic under conditions of diffusionism. Traditional approaches to algorithmic explainability presume a stable computational process that can be rendered intelligible through technical description or visualisation. However, diffusion models operate through probabilistic transformations that resist conventional forms of explanation. When knowledge exists in a state of perpetual potential transformation through latent space manipulation, notion of explaining an algorithmic decision becomes conceptually inadequate. This recalls Adorno's critique of identity thinking, as attempts to explain diffusion processes through traditional technical descriptions inevitably reify what are actually dynamic probabilistic relationships. The EU AI Act's requirement for explainable AI systems thus confronts a contradiction: the more sophisticated diffusion models become at generating and transforming knowledge, the less amenable they are to traditional forms of explanation. This suggests that rather than seeking technical solutions to explainability, we require new theoretical frameworks that can account for how meaning and knowledge operate under conditions of computational flux. I argue that the emphasis on individual algorithmic decisions obscures how diffusionism transforms the broader conditions of knowledge production. Any adequate approach to explanation must therefore address technical processes and the social totality within which diffusion models operate. This points toward the need for what we could understand as infrastructural or social forms of explanation that can account for how diffusionism reshapes technical systems and entire patterns of cultural production and interpretation (Berry 2021).
[8] The problem of regulating diffusionisation requires moving beyond traditional metaphors of algorithmic "black boxes" that can be opened and inspected. Instead, a more appropriate metaphor might be that of a watershed or river system, where cultural forms flow and transform through computational processes in ways that cannot be frozen or discretely analysed. Just as water management requires understanding entire ecosystems rather than attempting to control individual water molecules, regulating diffusionisation demands attention to the infrastructural conditions and flows of cultural production rather than individual algorithmic outputs. Indeed, the European Water Framework Directive (WFD) focuses on maintaining "good ecological status" across entire river basins, rather than measuring discrete pollutants and offers an instructive model (see WFD 2024). This metaphor suggests possible ways of thinking about new regulatory paradigms, For example, we might consider (1) infrastructure regulation that focuses on controlling the conditions of possibility for diffusion systems rather than their specific outputs. This could include requirements for maintaining public diffusion infrastructure, similar to water utilities; (2) cultural flow management that addresses how knowledge circulates through computational systems. Rather than attempting to regulate individual instances of generation, this would focus on maintaining national patterns of cultural reproduction and transmission; (3) commons based governance frameworks that treat certain forms of cultural production as public goods requiring collective stewardship. This could include requirements for maintaining cultural diversity and preventing monopolistic capture of generative infrastructures and technologies, e.g. by funding open source alternatives; (4) democratic oversight mechanisms that focus on systemic effects rather than individual companies technology. Similar to environmental impact assessments, this would require evaluation of diffusion impact assessments to determine how diffusion systems affect broader cultural and media ecosystems. However, the water management metaphor has significant limitations for understanding diffusionisation. Unlike water systems, which follow relatively stable physical laws, diffusion models actively learn and transform through their operation. The metaphor also potentially naturalises what are actually social relations of production and circulation. While the watershed model offers useful insights for thinking about systemic regulation, I argue that we must remain attentive to how diffusionisation operates through specifically computational forms of power and control that resist natural analogies.
[9] It is important to note that current diffusion models are prone to hallucinations and confabulations, wherein the systems can generate outputs that are incoherent, factually incorrect or depart significantly from the training data. While techniques such as prompt engineering and output filtering can mitigate these issues somewhat, they remain an inherent risk in probabilistic generative models. The propensity for hallucination stems from the way diffusion models work as they traverse mathematical spaces of pure possibility rather than discrete symbolic representations. The training process, based on iterative denoising, allows these models to generate plausible outputs by progressively refining random noise according to learned statistical patterns. However, this process can also lead the model to "imagine" patterns or correlations that do not actually exist in the training data. Clearly, the model itself cannot distinguish between real and confabulated outputs as both are simply points in its latent possibility space. The risks are particularly acute when models are prompted to generate outputs on topics with limited or skewed representation in the training data. The implications of these hallucinations can become severe as diffusion models are applied to higher-stakes domains such as medicine, law or governance. Confabulated medical advice, legal opinions or policy proposals could have serious consequences if taken as legitimate. Even in creative fields, hallucinations could enable the mass production of computational misinformation and disinformation. Hallucinations also raise epistemological and political questions. As diffusion models blur the line between fact and confabulation, the epistemic foundations of rational discourse and critique may also be called into question.
Bibliography
Comments
Post a Comment