What happens when Microsoft Defender flags your software?

Independent software development often occupies an ambiguous space between hobby and industry. My own scientific simulation package, used in hundreds of peer-reviewed publications worldwide, is firmly in that grey area. It is closed source because I may eventually commercialise it, yet free to download, and maintained by one person. It has a global user base but none of the commercial infrastructure that usually accompanies software with that reach. For years, this hybrid identity was an advantage. Until, suddenly, it wasn’t.

For over a decade, my Windows distribution method was deliberately simple: an unsigned executable inside a ZIP file served over HTTPS. This avoided the administrative and financial burdens of code-signing certificates, which would have cost more each year than the entire project’s hosting budget. Users would encounter the usual ‘unknown publisher’ prompts from Microsoft's SmartScreen, but aside from that, Defender left the software alone. The system built up a quiet reputation over the years. Nothing triggered deeper scrutiny.

That stability encouraged complacency. I believed I understood how Windows handled unsigned binaries. I did not appreciate how finely tuned Microsoft’s modern reputation system has become — or how sensitive it is to shifts in developer behaviour.

The self-signing experiment

As I started improving usability, documentation and general polish to grow the user base, I decided to self sign the executable. My intention was purely cosmetic: to make the software appear more professional, attach my name to the publisher field and publish cryptographic hashes to reassure new users. So I generated a keypair, signed the executables and DLLs, repackaged everything, and uploaded it to my website. Within minutes, the entire ecosystem collapsed. When I downloaded the new build through Edge, I was greeted not by the familiar SmartScreen interstitial but a red banner declaring the file contained a virus. It wasn’t just the new executable, everything I uploaded afterward — even older releases that had been completely safe for years — were immediately blocked. The domain itself appeared to be marked as suspicious, and users started reporting that my software was blocked as far away as Japan.

Only after some panicked investigation did I piece together what had happened. Windows Defender had silently uploaded the new binary’s hash and behavioural metadata to Microsoft’s cloud analysis pipeline. From Microsoft’s viewpoint, a previously benign site had suddenly begun serving self-signed executables with no historical reputation, delivered in unusual ZIP structures and updated repeatedly within minutes. My attempts to reverse the problem only made matters worse — the more I fiddled, the more suspicious the domain appeared. 

The result was a complete systemic block. Even file versions that had foundational reputation and had been widely used were instantly flagged. The automated system responded exactly as designed — what I saw as harmless experimentation, Microsoft quite reasonably interpreted as the behaviour of someone attempting to evade malware detection.

Slow recovery

Recovery happened gradually and only once I stopped trying to ‘fix’ the situation. After about an hour a months old version that had been downloaded extensively in the past stopped generating a virus warning on download. The explanation: it already had SmartScreen reputation, whereas any new version appeared unknown and suspicious. SmartScreen reputation is built up over time as users successfully download and run a file without incident. Once that trust profile is reset — for example, by changing the hash — the system treats the file as completely unknown.

At the same time, Microsoft’s backend likely executed one of my binaries in its cloud sandbox, concluded it was harmless and slowly withdrew the global ban. But to allow the reputation mechanisms to stabilise, I left the website untouched for more than six weeks. Only after that quiet period did I upload a new build, unsigned this time, and it was accepted without incident.

The experience was an uncomfortable reminder that a self-signed certificate offers no trust at all in the public-key infrastructure (PKI) world because you’re acting as your own certificate authority. Windows treats this as less trustworthy than an unsigned file. To Microsoft’s systems, a self-signed certificate is not a sign of authenticity — it is a hallmark of malware.

Understanding the sandboxing pipeline

One unexpected upside of the ordeal was a deeper understanding of how Microsoft’s security infrastructure works. Reputation is not simply a matter of whether a file is signed. It is influenced by real-world usage, distribution stability and behavioural patterns. 

For you

Be part of something bigger, join BCS, The Chartered Institute for IT.

Each file is identified not by its filename but by its cryptographic hash — a unique fingerprint of its contents. SmartScreen builds reputation per hash, meaning that even tiny changes to the executable generate a completely new identity with no historical trust attached. Windows often uploads newly encountered binaries from users’ PCs to their cloud sandboxes. The cloud sandbox often executes only the installer, but depending on the structure of the bundle, it may also run the main program.

In both cases, it observes only the earliest behaviour. Copying large files, touching the network or performing actions that resemble unpacking or system modification can all look suspicious. Tools such as py2exe, which generate executables with lots of empty space, can therefore appear anomalous even when benign.

This realisation prompted a redesign of my own startup sequence. Previously, the application copied a substantial set of scientific data files on launch, which is neither dangerous nor unusual for research software, but which can look concerning in a sandbox. I therefore moved these operations behind user-initiated actions so the sandbox sees a much quieter and more predictable program. Network calls were similarly delayed. The idea is simple: when Microsoft’s backend examines your executable, show it the most boring behaviour you can.

A surprisingly balanced system

Despite the initial panic, I came away with a certain respect for Microsoft’s approach. They could have followed Apple’s model: forbidden unsigned applications outright and forced developers into a controlled app-store ecosystem, something that imposes significant overhead on small or unfunded projects.

Instead, they have constructed a layered security apparatus that tries to strike a balance between safety and flexibility. It is strict but not closed. It can make mistakes, but it is also capable of self-correction. Most importantly, it does not punish independent developers who lack the financial or organisational capacity to maintain full code-signing infrastructure.

The experience was a stark reminder that modern security systems evaluate software not only by its contents but by its context — its distribution patterns, its reputation and its observed behaviour. For independent developers, these factors are now as much a part of the release process as compilers and test suites. Understanding them is no longer optional. For independent developers, these systems can feel opaque and unpredictable, yet they ultimately exist to protect users.

Understanding how your software appears to an automated security pipeline is now part of the craft of shipping code in 2025. It’s one more piece of infrastructure we all inherit, whether we have a corporation behind us or are working alone.