Antivirus Development Guide.

Summary
I’ll cover here the guidelines for a basic antivirus coding, for Windows and in C/C++. One can found here the pointers to design an antivirus engine, or simply learn how most of them are built.

Protection
For a good protection, an Antivirus must have at least one driver, to be able to run code in kernel and overall have access to kernel APIs. Starting with Vista, Microsoft understood that the Antivirus industry needed keys to enter the kernel and activate filters in strategic places, such as file system, registry and network. Don’t be stunned if building an antivirus for pre-Vista systems can be a real pain, because it was not designed for this.
• However, on Pre-Vista systems, Antivirus companies used to use rootkit-like features to guard the doors (even if it was not recommended at all by Microsoft) and be able to protect your system. They used what we call “Hooks” (API detours for filtering purpose).
• On Vista+, Microsoft provided APIs to insert our low level driver between userland calls and kernel APIs. That way, it’s easy to register an antivirus product into the kernel. More, that kind registration based system allows us to dispatch our system security into layers, where several products with different aims can cohabit. This was not the case for hooks, as the implementation was totally product dependant.

Process
The first thing to protect the user from, is the launching of malicious processes. This is the basic thing. Antivirus should register a PsSetCreateProcessNotifyRoutineEx callback. By doing this, on each process creation, and before the main thread starts to run (and cause malicious things) the antivirus callback is notified and receives all the necessary information.
It receives the process name, the file object, the PID, and so. As the process is pending, the driver can tell its service to analyse the process’s memory for anything malicious. It it founds something, the driver will simply set CreationStatus to FALSE and return.

Thread
In the same idea than for processes, threads can be a way for malicious things to cause damages. For example, one can inject some code into a legit process, and start a remote thread on that code inside the process’s context.
That way, a legit process can do malicious things.
We can filter new threads with the PsSetCreateThreadNotifyRoutine callback. Each time a thread is created, the antivirus is notified with the TID and the PID. Thus, it’s able to look into the thread’s start address code, analyse it and either stop the thread or resume it.

Images
The third dynamic threat is about images that can be loaded into memory. An image is a PE file, either a EXE, a DLL or SYS file. To be notified of loaded images, simply register PsSetLoadImageNotifyRoutine. That callback allows us to be notified when the image is loaded into virtual memory, even it’s never executed. We can then detect when a process attempts to load a DLL, to load a driver, or to fire a new process.
The callback gets information about the full image path (useful for static analysis), and the more important in my opinion, the Image base address (for in-memory analysis). If the image is malicious the antivirus can use little tricks to avoid the execution, like parsing the in-memory image and go to the entrypoint, then call the assembly opcode “ret” to nullify it.

Filesystem
Once every dynamic thing is secured, an antivirus should be able to notify user for malicious things on-the-fly, not only when they are about to start. An antivirus should be able to scan files when user opens a folder, an archive, or when it’s downloaded on the disk. More, an antivirus should be able to protect himself, by forbidding any program to delete its files.
The way to do all of this, is to install a driver into the file system, and more specifically a minifilter of a legacy filter (old way). Here we will talk about minifilter.
A minifilter is a specific kind of driver, able to register callbacks on every Read/Write operation made on the file system (IRP major functions). An IRP (Interrupt Request Paquet) is an object used to describe a Read/Write operation on the disk, which is transmitted along with the driver stack. The minifilter will simply be inserted into that stack, and receive that IRP to decide what to do with it (allow/deny operation).
For a little example of minifilter, please check that useful link or that one. The microsoft guidelines are here.
You’ll find also 2 examples of the WDK documentation here and here.
A basic minifilter callback look like this. There are 2 kinds of callback, Pre operation and Post operation, which are able to filter before of after the query. Here’s a preOperation pseudo code:

Network
To guard the doors of the whole internet traffic which can be huge on certain systems (servers, huge bandwidth users) without being slowed down by the context switching that takes place in userland, it’s totally not recommended to install a firewall that have no underlying driver, except for some web browser filters that can be enough for http traffic, but that will not protect against malware communication in/out.
In order to have a correct implementation of firewall, one should code a NDIS, TDI or another method for low level IP filtering driver. NDIS/TDI is a bit tricky to do, and would require lot of knowledge (more than other filters in my opinion).
Anyway, here’s some pointers to start coding such a driver, the microsoft guidelines, and old codeproject tutorial (but still good to read), an example of NDIS firewall, and an example of TDI firewall. Here’s also a good writing about NDIS firewall bypass trick, and a little explanation about the network driver stack,

Analysis Engine
The analysis engine is one of the most important part, it’s responsible for analysing file/memory samples coming from the drivers. If must be fast (even with a huge database), and should be able to handle most of the file types (Self-extracted executables, Archives – RAR, ZIP, Packed files – UPX, … ) and thus should have many modules to do this:
• Unpacker : That module must be able to detect and unpack most of the known packers (UPX, Armadillo, ASPack, …)
• Signature engine: The database of an antivirus contains millions of signatures, and the engine should be able to fast search for them into a sample. Thus, a very powerful algorithm should be part of it. Some examples : AhoCorasick, RabinKarp, string matching algorithms.
• Sandbox : That module is not necessary, but would be a plus to be able to run samples into a limited memory, with no effect on the system. That could help to unpack samples packed with unknown packers, and help the heuristic engine (see after) to detect API calls that could be considered as suspicious or malicious. Some good sandbox here.
• Heuristic engine : As said above, the heuristic engine does not search for signatures, but rather look for suspicious behaviour (ie. sample that opens a connexion on the website hxxp://malware_besite.com). That can be done by static analysis, or through a sandbox.

Self-protection
The self protection is very important for an antivirus, to avoid being defeated by a malware and continue to protect the user. This is why an antivirus should be able to guard its own installation and keep persistence at reboot.
There are several place to protect: Files, Registry keys, Processes/Threads, Memory.
• File protection is implemented into the minifilter, with particular rules on the files of the antivirus (No access in deletion, renaming, moving, writing).
• Registry protection is made into the registry filter, with access denied for registry keys of the driver and the service.
• The drivers threads are protected, because it’s quite impossible to unload kernel module without crashing the system
• To be able to protect the service, which is a userland process, 2 solutions:

1. The easiest would be to add rules for failures in the service manager, and set every failure rule to “restart service”. That way, when the service is not stopped by service manager, it restarts. Of course, the service should not be able to accept commands until the system is not restarting (or stopping).
2. The second method, which is more generic, would be to set callbacks on process handles with ObRegisterCallbacks.
   By setting the ObjectType to PsProcessType and the Operation to OB_OPERATION_HANDLE_CREATE, you receive a Pre and Post operation callback, and you are able to return ACCESS_DENIED into ReturnStatus if the process handle queried has GrantedAccess which have process terminate rights(or process write rights, or anything that can lead to a crash/kill), and of course if the process is one the antivirus needs to guard (its service for example).
   Of course, one also needs to guard Duplicate handle and the PsThreadType to avoid any termination method that requires to grab a handle on the process or a thread.

End