Legal and operational realities

Every generative imagery product becomes a content moderation problem at scale. Users will try to generate things you do not want them to generate, sexualized content, content involving minors, content depicting real people, copyrighted characters, deepfakes, hate symbols, violence, and so on. The technical mitigations are: input filtering (block prompts containing problematic terms), output filtering (run generated images through a safety classifier and block ones that fail), rate limiting (slow down users who are clearly probing for failures), account suspension (kick users who repeatedly violate policies), and proactive monitoring (flag anomalous usage patterns).

None of these technical mitigations is sufficient on its own. Even the best safety classifiers have false negatives, they miss things they should catch, and false positives that frustrate legitimate users. The operational reality is that you need a human moderation team or contracted moderation service to handle the cases the automated systems miss. For a small startup, this can be a co-founder spending a few hours a week reviewing flagged content. For a larger company, it is a real team of moderators with the associated psychological costs of looking at the worst things humans try to make AI generate.

The legal exposure varies by jurisdiction. In the US, Section 230Section 230The United States law that shields online platforms from liability for content their users post, whose application to AI-generated output is unsettled. protections still apply to user-generated content on platforms, but the landscape is shifting and there are open questions about whether AI generation counts as 'user-generated' or as something the platform itself produced. In the EU, the AI Act creates explicit obligations for generative AI providers around content moderation and watermarkingWatermarkingHiding a detectable signal inside generated media so its origin can later be identified. For audio and AI media it remains an arms race rather than a guarantee.. In the UK, the Online Safety Act adds further requirements. Any product serving users globally has to comply with the strictest applicable regime, which usually means following EU rules everywhere.

Models are trained on copyrighted images, and there are several active lawsuits about whether this constitutes infringement. Stability AI, Midjourney, and Runway have all been sued by groups of artists and stock photo companies. The legal status of training on copyrighted data is unsettled and probably will be for years. The pragmatic stance for an operator is: assume there is some legal risk, do not amplify it by enabling obvious infringement (do not let users promptPromptThe text description you provide to a model to specify what you want it to generate. for 'in the exact style of [named living artist]' without expecting blowback), and consider commercial indemnificationIndemnificationA vendor's promise to cover a customer's legal costs if the product triggers a lawsuit, offered by some licensed-data model providers as protection against copyright claims on generated output. clauses in your customer contracts so that the legal risk is contained.

Adobe Firefly has built its commercial positioning around being trained only on licensed dataLicensed dataTraining data used with the permission of and usually payment to its rights holders, the basis for models marketed as commercially clean., which lets Adobe offer customers full IP indemnification. This is a competitive advantage for B2B customers who care about legal exposure, even though Firefly's outputs are generally lower quality than the frontier alternatives. If you are serving enterprise customers who write large checks, the legal certainty matters more than the raw model quality.

Every model you depend on will eventually be deprecated. The model labModel labA company that trains and ships the frontier generation models themselves, such as Black Forest Labs, Runway, or Google. will release a new version, retire the old one, and your product will need to migrate. Some migrations are easy (FLUX 1.0 to FLUX 1.1 was mostly drop-in). Some are painful (SD 1.5 to SDXL required retraining most LoRAs and significantly changing promptPromptThe text description you provide to a model to specify what you want it to generate. conventions). One was disastrous (the OpenAI Sora shutdown left every product depending on Sora with nothing). The operational defense is: do not depend on any single model for more than 50 percent of your generations, monitor the model labs' release patterns, and have a migration plan written before you need it.