Introduction

Git is a distributed version control system that helps track changes in source code during software development. This article will cover the fundamental concepts and commands that every developer should know.

Basic Concepts

Repository

A Git repository is a storage location where your project files and their complete history are stored.

Commit

A commit is a snapshot of your project at a specific point in time. Each commit has a unique identifier (hash) and contains information about what changed.

Branch

A branch is a separate line of development. The default branch is usually called main or master.

Basic Git usage

Initial Configuration

If this is the first time you are using Git as a specific user, you should configure it with your details first.

• Set user credentials:

  git config --global user.name "Your name"
  git config --global user.email your_name@yourdomain.com
  

  • The global user credentials are saved in the ~/.gitconfig file. You can open it and see the current status of configuration.

• You can also configure credentials per repository, by using:

  git config user.name "Your name"
  git config user.email your_name@yourdomain.com
  

  • These local settings are saved in the .git/config file, in the local directory.

• To use the “git send-email” command (which allows sending patches to a mailing list for review) your email server and account details will need to be configured in the ~/.gitconfig file as well. Use these settings for the Daynix mail server (Obviously, replace the “from” and “smtpuser” fields with your own data):

  [sendemail]
          from = Your Name <your_name@yourdomain.com>
          smtpserver = smtp.gmail.com
          smtpuser = your_name@yourdomain.com
          smtpencryption = tls
          chainreplyto = false
          smtpserverport = 587
  

Getting remote repositories, and creating new ones

Clone repository (get it locally)

To clone a repository into a local directory do:

git clone <repo URL> [optional local directory]

This operation sometimes called “checkout” in other source control systems.

Create repository

In the required directory do:

git init

The above command will create a new empty repository. You can also use “–bare” option to create repository without a file tree (can be useful if this is a central repository that no one is actively working on but only pushes to and pulling from).

Next, assuming that we are working with GitHub the stages are:

• Create the remote repository on GitHub:

  curl -u '<USER>' https://api.github.com/user/repos -d '{"name":"<REPOSITORY>"}'
  

  • In the above command, ONLY <USER> and <REPOSITORY> should be changed to the appropriate values (your username on GitHub and the desired name for the new repository). The “user/repos” part should be kept as-is, and used literally.
  • The output of this command will print a lot of useful information on the newly created repository. It may be useful to keep.

• Next, define the remote “origin” for the new local repository, in the newly created one on GitHub:

  git remote add origin https://github.com/<USER>/<REPOSITORY>.git
  

  • Again, <USER> and <REPOSITORY> should be changed here to the same values that were used above.

  • If you had a typo above, or just want to use a different remote origin, use the command below to remove the origin and start over:

    git remote rm origin
    

Important commands to see the current status

• git status : Shows the files that were modified since the last commit, and the files that are currently untracked by Git. Very useful to get the current status of things
• git log : Shows the history of the commits, their hash numbers (very useful), and their commit messages.

Committing

1. Before beginning to work on a specific issue, create a branch for it, and switch to it. These two actions can be achieved using a single command:

   git checkout -b newFeature
   

   …Where newFeature is the name of the new branch you’re creating. If you forgot to perform this before making your changes, you can do this later. But, it will be best to create the new branch before any changes are made.

2. Edit the files and make the desired changes to the code.
3. Perform “git status”, “git diff” or “git diff <file>”, in order to see the changes from the master branch (in all files or in a specific one).
4. To stage the desired changes to a commit, use:
   1. “git add -i” or “git add -i <file>”. This will let you interactively select the wanted changes for committing.
      1. Press “5” (patch) then the number of file to look at (if there are several) and then press Enter.
      2. Blocks of changed code will be presented to you. To include the whole block press “y”, to skip the whole block press “n”, to split it into smaller blocks (if, e.g., it is not continuous, and only one part is required) press “s”, or, to manually edit the changes, press “e”.
      3. After finishing going over a file, select the next file to look at, or quit (“7”).
      4. If you would like to include a previously untraccked file, just type “a” in the prompt, and choose the number of the file.
   2. Or “git add -u” to add all changes in tracked files (be careful - new files will not be added!).
   3. Or “git add .” to add all changes and untracked files in local directory (be very careful, as this can add intermediate files. Should be used for initial commits).
5. Run “git diff --cached” to see only the changes that will be included in the next commit.

   • In case there were mistakes in staging (running “add”) you can always fix it by:

     git reset <file>
     

     This will unstage the file, and you will need to add it (or parts of it) again. You can also run this command without “<file>”, to unstage all changes.

6. Perform the commit:

   git commit -s
   

   In the editor that will appear, write the commit title (starting with “PROJECT NAME:”) then, if the title is not informative enough, make a one-line space and underneath it write a more detailed description. Save and quit.

7. Repeat steps 4–6 as many times as necessary, to create commits for all the introduced changes. It is a good practice to make the commits as small and specific as possible.

Formatting patches and sending them for review

How to determine the subject prefix of a patch: The subject prefix of a patch is determined by the mailing list the patch will be sent to, for example:

• When a patch is send to a mailing list related to one repository only (like qemu-devel) subject should be prefixed with [PATCH]

  git config format.subjectprefix "PATCH"
  

1. To format the patches use the command:

   git format-patch master -n -o <desired output directory>
   

   (If this is the second (or any other number) iteration of patches after review use):

   git format-patch master -n -o <desired output directory> -v<iteration number>
   

   This command will format the patches (as differences from the “master” branch), and output them as “.patch” files into the specified output directory.

   • Other useful options for this command include:
     • ”--cover-letter”, which will enable you to write a cover letter explaining the patches.
     • ”--subject-prefix=<Desired-Subject-Prefix>”, which will replace the standard subject prefix from the default, which is “[PATCH]”.
     • It is also possible to replace subject prefix for all patches generated from a specific working copy by running "git config format.subjectprefix <Desired-Subject-Prefix>"
     • When renaming files use –minimal or -M in order to make patch size minimal and readable.
     • Use -v option to append version number to [PATCH] prefix. Example -v 2 will create “[PATCH v2]” prefix.

Important note for GMail usage

https://support.google.com/accounts/answer/6010255 - users should allow the usage of “less secure apps” in order to be able to send emails with send-patch

Editing commits

After peer review, or if you just found an error in a commit, editing the commits may be needed. TBD

Resolving merge conflict while rebasing

Pull Requests

TBD

Force push a Pull Request

TBD

Pushing changes to Master

After your patches were peer-reviewed and approved, and no changes are needed to be performed on the commits, you can push them to the production branch (Master) on the remote server.

[PLEASE VERIFY AGAIN THAT ALL THE COMMITS ARE CORRECT AND FINAL!!!]

There are several ways to push changes:

The simplest one assumes that no one was changing the remote repository while you were working on your feature, and you did not modify your local “master” branch while working on the branch dedicated to your new feature. Basically, it means that you are the only one working on the project, and you make one change at a time:

[[You are the only one working on the remote repository, and you KNOW it did not change while you were working on your feature]]

TBD

If other people may have changed the remote repository, but you did not change the local “master” branch since starting to work on your separate branch, use the following method:

[[Your LOCAL “master” branch didn’t change since you started to work on your feature]]

TBD

The longest method does not assume anything. IF IN ANY DOUBT, USE ONLY THE FOLLOWING METHOD:

[[You are not sure of anything - the longest, but the most reliable way]]

TBD

Advanced How-To’s

How to revert commit(s) in GIT

Let’s say you have the branch history like:

HashA -> HashB -> HashC ... -> HashPreLast -> HashLast

Then you can use:

• to revert HashLast commit either:
  • git revert HEAD
  • git revert HashPreLast..HEAD
• to revert HashPreLast commit:
  • git revert ^HEAD
• to revert HashB commit:
  • git revert HashA..HashB
• to revert HashB to HashLast commits:
  • git revert HashA..HEAD

More information can be found here

Checking out pull request from GitHub

Sometimes you would like to checkout pull request from Github to do additional tests. In order to do it follow: https://help.github.com/articles/checking-out-pull-requests-locally/

Fetch pull request by ID and add new branch name:

git fetch origin pull/ID/head:BRANCHNAME

Check out the branch:

 git checkout BRANCHNAME

Bisection with GIT (git bisect)

Let’s say you have a branch history with regression, you know commit that worked and commit that doesn’t work, but there are commits between those and you do not know which one introduced a problem. For cases like that GIT provides the bisect mode (git bisect) that allows to find a change that introduced a bug by binary search.

git bisect command is described in great details at http://git-scm.com/docs/git-bisect

It worth noting that GIT bisect mode may be used for finding both regressions, i.e. commits that broke something (in a natural way) and commits that fixed something. In the latter case good and bad commits concepts have exactly opposite meaning, i.e. good - commit that doesn’t work, bad - commit that works.

Resource management is a solved problem in Ruby… or is it?

In C language, you need to call free() whenever an allocated memory is no longer used. If you fail to call free(), you will get out of memory. If you try to use memory already freed, it results in a hard-to-debug bug and security vulnerabilities. And the code gets cluttered with so frequent free() calls.

But such free() calls are unnecessary on Ruby. Ruby has a garbage collector so the memory associated with an object that is no longer referenced by anybody will be freed automatically. Resource management is a solved problem in Ruby… or is it?

Unfortunately, the garbage collector only works with resources managed by Ruby. A notable example of resources not managed by Ruby is an external process created with spawn. In this post, we discuss why the garbage collector is not a one-size-fits-for-all approach, and how to safely manage resources without the garbage collector by illustrating a new resource management mechanism I’m proposing for our Ruby product, AutoHCK.

What is AutoHCK?

AutoHCK is a tool to automate Windows driver testing. AutoHCK runs the Windows Hardware Lab Kit (Windows HLK), a testing framework for Windows drivers, on QEMU virtual machines. It is particularly useful when you want to test with para-virtualized devices implemented in QEMU or when you want to run Windows HLK on a Linux host.

AutoHCK is responsible to prepare files required by test runs, to manage virtual machines, and to collect test results. These features are implemented in Ruby.

Problem with garbage collector

As described earlier, resource management is not a solved problem even for Ruby, which comes with a garbage collector.

The problem with a garbage collector is that it is highly asynchronous. Its asynchronous nature benefits when managing pure-Ruby objects. For example, consider of freeing a long linked list of Ruby objects:

S = Struct.new(:next)
l = 10000000.times.reduce(S.new) { |sum| S.new(sum) }

If you naively try to free something like l on a platform that lacks garbage collector (e.g. C++ and Rust), it will freeze the process for a while as it will need 10 million objects at once, but Ruby can do that when it is idle and avoid freezing.

However, it relies on the fact that there is plenty of resources available; nobody cares even if the garbage collector increases the memory usage by a few megabytes. It is untrue for scarce resources. Such an example in AutoHCK is access to a disk image. AutoHCK sometimes performs a sequential run of two QEMU processes using a disk image; the earlier one sets up a virtual machine and the later one does the actual work. It will corrupt the disk image if the later reads it while the earlier is still writing. This means only one virtual machine can use a disk image at a time so access to the image is very scarce. In such a case, synchronous resource management is mandatory, and a garbage collector is a poor option for that.

Alternatives

As shown above, the garbage collector is not a silver bullet. This section presents alternative options for resource management.

Reference counting

The idea behind reference counting is simple; remember how many references are out there and release the resource when the count reaches zero. The resource deallocation timing in a reference counting system is more predictable than in the garbage collector as the deallocation happens exactly when the references are deleted.

The obvious downside is that a stale reference may keep a resource occupied. In the earlier example of a sequential run of two QEMU processes, if the earlier process keeps the stale reference to the disk image, the later process will never have access to the image, which makes AutoHCK hang. In such a case, you will want to mark the stale reference invalid instead of waiting forever till it disappears.

Resource scope

The resource scope is an inherent concept in many programming languages. In such languages, a variable becomes available when the execution enters a scope, and it will be released when the execution leaves the scope.

The resource scope mechanism works neatly when the execution does not involve persistent states. For example, if a system works like a mathematical function that simply computes the output from the input, obviously intermediate values computed in the system can be released after the execution.

The mechanism will not work well for applications like daemon. For example, consider a key-value storage that accepts set and get operations. A pair of a key and a value needs to persist after exiting the scope of the set operation. In other words, the pair of a key and a value needs to escape the scope.

Introducing AutoHCK::ResourceScope

So how can we complement the garbage collector for resource management in AutoHCK? Determining the best method for resource management requires an understanding of the workload. In the case of AutoHCK, there are two important facts:

• AutoHCK is not a daemon but more function-like. It consumes a Windows image and test configurations, outputs the test results, and terminates.

• AutoHCK has essential resources that require mutually-exclusive access: disk images. Disk images are needed for virtual machines and virtual machines are needed for… literally everything AutoHCK does.

They make a clear case for introducing a resource management mechanism using scope.

The interface

Below is an example usage of AutoHCK::ResourceScope:

ResourceScope.open do |scope|
  a = File.new('a.txt')
  scope << a
  a.read
end

This code opens a file, reads it, and closes it. ResourceScope can be applied to any resources that can be freed by calling its close method.

For this simple example, you may just give a block to File.open to get the identical result, but the strength of AutoHCK::ResourceScope is that it is composable.

The composability relies on the fact that it can be nested.

ResourceScope.open do |a|
  file = ResourceScope.open do |b|
    path_file = File.new('path.txt')
    b << path_file
    File.new(path_file.read)
  end

  a << file
  file.read
end

This way, you can have function-like structures nested. Moreover, ResourceScope has a special method named transaction:

ResourceScope.open do |a|
  do_something_with a.transaction do |b|
    file = File.new('a.txt')
    b << file
    raise 'unexpected header' if file.readline != 'magic'
    file.rewind
    file
  end
end

Resources attached to a transaction scope will be freed if an exception occurs. Otherwise, they will escape from the transaction scope to the original scope. It is known as move semantics in C++ and Rust. This makes sure resources will not leak due to exceptions during resource allocation.

Finally, a scope object can be passed to methods.

class C
  def new(scope)
    @file = File.new('a.txt')
    scope << @file
  end

  def read
    @file.read
  end
end

ResourceScope.open do |scope|
  c = C.new(scope)
  c.read
end

A scope parameter of a method forces the programmer to consider what scope is appropriate to manage the allocated resource.

Nested scopes, transactions, and scope parameters make it possible to compose a complex procedure from simple ones involving resource allocation and usage.

Interrupts

A major pitfall when managing resources without the garbage collector in Ruby is interrupt handling. In Ruby, an interrupt can happen anywhere in the code so a resource will leak if an interrupt happens after it is allocated and before it is freed.

Thread.handle_interrupt allows to mask interrupts but its use is tedious. AutoHCK::ResourceScope.open automatically use the method and make it simple.

Below is the actual implementation of AutoHCK::ResourceScope.open:

def self.open(resources = [])
  scope = new(resources)
  Thread.handle_interrupt(AutoHCKInterrupt => :never) do
    Thread.handle_interrupt(AutoHCKInterrupt => :on_blocking) do
      yield scope
    end
  ensure
    resources.reverse_each(&:close)
  end
end

Thread.handle_interrupt(AutoHCKInterrupt => :never) ensures an interrupt will not occur when freeing resources.

Thread.handle_interrupt(AutoHCKInterrupt => :on_blocking) ensures an interrupt will not occur after a resource is allocated and before it is assigned to the scope.

AutoHCK::ResourceScope.open use a specialized exception class (AutoHCKInterrupt) so that third-party code that creates a child thread and interrupt it works. A typical example is timeout. As timeout works by creating a thread and interrupting it after a specified period, masking generic interrupts prevents it from functioning.

This also means AutoHCK::ResourceScope cannot be modified in a thread that can have an interrupt other than AutoHCKInterrupt.

Conclusion

We discussed the resource scope as an alternative to the garbage collector, and how to implement it in Ruby. There is no single universal way to manage resources. For example, Chromium, a large program written in C++, uses all of a garbage collector, reference counting and resource scope. Sometimes you may find something missing in the programming platform and you may need to implement it by yourself. In such a case, carefully design your own solution and also watch out for platform-specific pitfalls.

The first post roughly described there are several development steps:

1. Writing a boilerplate and adding basic features
2. Adding QTest
3. Adding advanced features and libvirt support
4. Running more tests and increasing feature coverage
5. Debugging
6. Contributing to the upstream

This post discusses 5 and 6. Though this post is written with new device emulator development in mind, the debugging techniques shown here apply to other kinds of QEMU development as well.

Debugging

Debugging is the most interesting (and tiresome) part of the device emulator development.

The first step of debugging is to make a bug reproduction case. Usually, you will start debugging by creating a minimal reproduction case, but it is often impractical for low-level software like QEMU as the higher-level software is just too big; you cannot trim Linux down. The higher-level software may even lack source code (c.f., Windows). In such a situation, you can’t do anything but use the real workload as a reproduction case. That also means there can be bugs anywhere: guest user-space program, guest kernel, QEMU, or even compiler.

But don’t worry. It is always possible to figure out a bug no matter how the reproduction case is complex. You just need right tools, knowledge, methodology, and patience.

Tools

First, let me introduce useful tools. You can’t (or don’t want to) debug everything just with printf(), but you don’t need dozens of tools either. The most important thing is to choose tools according to the situation.

Hand-written scripts

The first thing to do, even before you try to figure out the bug, is to make the iterations fast. Write some scripts to automate repetitive tasks, especially the program launch. Scripts I used during development are available at: https://github.com/akihikodaki/q

You can choose whatever script language. Below is possible options:

Shell script

Shell is available everywhere so it may be handy. Unfortunately, it has various quirks and compatibility issues.

C

C is actually not that bad. Basically, you just need to call system() everywhere and omit error handling to write it “script-like”.

The downside is that it requires compiling and is bad at error handling.

Ruby

An interesting characteristic of Ruby is that it exposes various Unix functions. It can also call C functions without any external library so you can have unlimited access to the underlying system.

While Ruby is a good option for system scripting as described above, it is not as popular as Python or JavaScript. For example, Visual Studio Code comes with mature support for Python and JavaScript, but Microsoft does not directly provide support for Ruby.

Node.js

Node.js employs JavaScript language. Modern JavaScript is comfortable to write, and its rich documentation on MDN and formal specification are helpful if you are not familiar with the language.

Node.js’ asynchronous nature is also attractive for scripting. For example, if you need to buildProject() and copyFiles() (assuming both of them return Promise) in parallel, you can just write as follows:

await Promise.all([buildProject(), copyFiles()])

Google’s zx makes it even easier to write scripts.

However, Node.js is somewhat more restrictive than Ruby so you cannot easily interact with file descriptors, for example.

git-bisect

The man page says:

git-bisect - Use binary search to find the commit that introduced a bug

In principle, it’s so simple and very effective to debug regressions, but sometimes the bug you are debugging may involve several regressions introduced in different commits. Naively performing binary search may not be sufficient in such a case so you need to keep considering what is happening during git-bisect.

Sanitizers

Sanitizers detect common bugs in C code at runtime. AddressSanitizer and UndefinedBehaviorSanitizer can be enabled by passing the --enable-sanitizers flag to the configure script. I recommend keeping them enabled.

Note that the entire QEMU code base is not free of addressability issues and undefined behaviors. It’s better to keep your workload minimal to avoid noise from existing issues. There were several noisy bugs affecting my minimal workload so I fixed them to keep the console clean¹²³⁴.

printf()

It may sound stupid but, in fact, printf() is a great tool for debugging. It works everywhere, and it has all of the expressiveness of the programming language. Want the state of the program when the loop count is 42? Just write if (i == 42). Need to parse the bytes in an array? Write a parsing function and call it. It’s so simple and effective.

A downside is that such code can be often left after debugging. You may annotate the logging code with a debug flag to pollute the console, which mitigates the problem but does not solve it; it’ll still end up with a lot of logs you cannot really digest when the debug flag is set.

The rule of thumb for printf() debugging is to check in the code beforehand. You can then just git checkout . (or safer: git stash) to discard all of debug code at once. Another useful trick is to add your name to debug logs. A person’s name is often very distinguishable from the other code; you can easily find it with grep. And if you forgot to delete it, others will ask you: “Hey, there is your name in the code, did you forget to remove it?”

Another problem with printf() debugging is that it needs recompilation. There may be tools for live patching, but live patching has inherently confusing as the state of the old code persists. Don’t stick to printf(), and switch to debugging using GDB/LLDB when appropriate.

tracepoints

QEMU has tracing infrastructure. It is basically printf() with external tooling support, but it is still useful even if you just use the default “log” trace backend, which outputs traces to standard error as it allows filtering traces.

For example, you can get all traces from igb (except e1000x-common code) by adding the following to the command line:

-trace 'igb_*'

The name of tracepoints for igb is listed in hw/net/trace-events. Tracepoints for other devices are listed in corresponding trace-events files.

The output of this pattern will be huge as it enables all tracepoints igb has so you can first skim it and narrow down the code. If you find something related to IRQ suspicious, you can write:

-trace 'igb_irq_*'

…and keep narrowing it down. It is also possible to enable/disable tracepoints at runtime using the QEMU monitor, which will be described later.

To add a new trace point, you can add a new line to the trace-events file and call the corresponding trace function. For example,

igb_set_vfmailbox(uint32_t vf_num, uint32_t val) "VFMailbox[%d]: 0x%x"

It is easy to filter tracepoints, but it is still important not to add too many tracepoints. The documentation gives some hints for adding new trace events:

1. Trace state changes in the code. Interesting points in the code usually involve a state change like starting, stopping, allocating, freeing. State changes are good trace events because they can be used to understand the execution of the system.
2. Trace guest operations. Guest I/O accesses like reading device registers are good trace events because they can be used to understand guest interactions.
3. Use correlator fields so the context of an individual line of trace output can be understood. For example, trace the pointer returned by malloc and used as an argument to free. This way mallocs and frees can be matched up. Trace events with no context are not very useful.
4. Name trace events after their function. If there are multiple trace events in one function, append a unique distinguisher at the end of the name.

In other words, you should think carefully if you add a trace event that does not fit well with this description. For example, it usually does not make sense to add tracepoints for individual if statements or functions. If there are well-designed trace events to trace state changes and guest operations with meaningful correlator fields, you should be able to infer which if statements or functions will be evaluated.

GDB and LLDB

GDB and LLDB are debuggers for C programs. They both provide features you will expect; it allows you to pause the execution at some point of code (breakpoint) or to pause the execution when it accesses some region of memory (watchpoint) without recompilation. GDB is widely used so you can expect good support from debugged programs. LLDB is a more modern alternative and (in my experience) has superior reliability. You can pick either of them or use both of them, depending on the situation.

As QEMU debugging involves software at various levels, you need to attach GDB to some or even all of them, depending on the nature of the bug. In particular, there are three levels you may be interested in:

• QEMU
• Guest kernel
• Guest userspace

QEMU

Attaching GDB or LLDB to QEMU is no different from attaching it to other userspace programs; simply put, you just type:

gdb build/qemu-system-aarch64

QEMU also has some GDB scripts that tweak it for debugging QEMU so it’s better to use them. To use them, load the .gdbinit file in the QEMU source tree. GDB has a feature to load .gdbinit automatically, but usually, it’s disabled for the security reason so type the following to load it manually:

source .gdbinit

Guest kernel

Kernels often implement remote debugging protocols; one of our blog posts covers such a protocol of Windows. Linux also comes with a similar feature called kgdb. However, they are sometimes unreliable when e.g., interrupts happen.

The remote debugging feature of QEMU is more reliable as it operates in a more lower-level. This feature is documented well, but, in short, start QEMU with option -s added to the QEMU command line, start gdb (with no argument), and type:

target remote :1234

Now GDB is attached to the guest and you can freely place breakpoints and watchpoints.

However, this comes with no debug information so it’s not practically usable in this state. To use debug information from a guest kernel, you need to take some procedure specific to it. Linux comes with a good documentation describing the procedure to debug it with QEMU so I recommend reading it. A common pitfall is that KASLR confuses the symbolizer, which is described in the documentation. I also had to disable in-kernel pointer authentication (CONFIG_ARM64_PTR_AUTH_KERNEL) for my Arm64 environment.

It may be still necessary to interrupt the guest execution without knowing the address to place a breakpoint. For example, Windows does support debugging the kernel with QEMU, but it requires another Windows machine and so troublesome to set up. It doesn’t come with source code so you cannot know the exact address to place a breakpoint anyway. A useful technique in such a case is to pause the execution from the QEMU device code.

First, add the following includes to the head of the source file:

#include "exec/gdbstub.h"
#include "hw/core/cpu.h"
#include "sysemu/runstate.h"

Now you can add the following to the point where you want to pause the guest:

gdb_set_stop_cpu(current_cpu);
vm_stop(RUN_STATE_PAUSED);

The first line switches the CPU GDB interacts with. The second line actually stops the VM. You can add conditions with if or do whatever since it’s C.

filter-dump

Sometimes there may be debug tools specialized for a particular feature. filter-dump is such a tool included in QEMU for networking and allows to dump network traffic in QEMU. It’s usually superior to dumpcap running on a tap device attached to QEMU netdev as it runs at the very end of QEMU; it can see corrupted packets sent from QEMU that will be filtered out by the tap device, for example.

This tool was not working with packets with virtio-net headers attached so I wrote a fix. Such investment for a specialized tool is not essential but pays off well.

QEMU monitor

QEMU monitor allows you to interact with a live QEMU session. You can change the configuration of QEMU by changing the command line, but it requires frequent restart so it’s recommended to learn how to change the configuration at runtime with the QEMU monitor. Sometimes live configuration change is inevitable; you may need to add and remove filter-dump to capture only a particular part of network traffic, for example.

Snapshotting and hotplugging are only available with interactive access with QEMU. You should occasionally test them as they are error-prone features.

Case study

This section describes actual bug hunting during the igb development. The cause of the bugs are not limited to QEMU, but include guest kernel and compiler.

DMA reentry bug

After sending igb patches to the upstream, several crash reports generated with fuzzer came up:

Heap-use-after-free in e1000e_receive_internal (#1543) · Issues · QEMU / QEMU · GitLab

This crash happened with e1000e, but I still got the report as I did lots of cleanup before copying the e1000e code for igb. It is also likely that a bug affecting e1000e affects igb as well.

Locally reproducing the crash resulted in the following logs:

==17147==ERROR: AddressSanitizer: heap-use-after-free on address 0xfffec4259798 at pc 0xaaabab108f88 bp 0xffffe3a4a780 sp 0xffffe3a4a790
READ of size 8 at 0xfffec4259798 thread T0
    #0 0xaaabab108f84 in e1000e_write_packet_to_guest ../hw/net/e1000e_core.c:1651
    #1 0xaaabab108f84 in e1000e_receive_internal ../hw/net/e1000e_core.c:1776
    #2 0xaaabab0cb978 in net_tx_pkt_send_custom ../hw/net/net_tx_pkt.c:823
    #3 0xaaabab0fe608 in e1000e_tx_pkt_send ../hw/net/e1000e_core.c:682
    #4 0xaaabab0fe608 in e1000e_process_tx_desc ../hw/net/e1000e_core.c:762
    #5 0xaaabab0fe608 in e1000e_start_xmit ../hw/net/e1000e_core.c:953
    #6 0xaaabab102694 in e1000e_set_tctl ../hw/net/e1000e_core.c:2496
    #7 0xaaabab10f918 in e1000e_core_write ../hw/net/e1000e_core.c:3349
    #8 0xaaababc913bc in memory_region_write_accessor ../softmmu/memory.c:493
(snip)

0xfffec4259798 is located 24 bytes inside of 208-byte region [0xfffec4259780,0xfffec4259850)
freed by thread T0 here:
    #0 0xfffecfc9d7cc in __interceptor_free /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52
    #1 0xaaabab0cc3d0 in net_rx_pkt_iovec_realloc ../hw/net/net_rx_pkt.c:76
    #2 0xaaabab0cc3d0 in net_rx_pkt_pull_data ../hw/net/net_rx_pkt.c:99
    #3 0xaaabab0cf33c in net_rx_pkt_attach_iovec_ex ../hw/net/net_rx_pkt.c:153
    #4 0xaaabab102e70 in e1000e_receive_internal ../hw/net/e1000e_core.c:1764
    #5 0xaaabab0cb978 in net_tx_pkt_send_custom ../hw/net/net_tx_pkt.c:823
    #6 0xaaabab0fe608 in e1000e_tx_pkt_send ../hw/net/e1000e_core.c:682
    #7 0xaaabab0fe608 in e1000e_process_tx_desc ../hw/net/e1000e_core.c:762
    #8 0xaaabab0fe608 in e1000e_start_xmit ../hw/net/e1000e_core.c:953
    #9 0xaaabab102694 in e1000e_set_tctl ../hw/net/e1000e_core.c:2496
    #10 0xaaabab10f918 in e1000e_core_write ../hw/net/e1000e_core.c:3349
    #11 0xaaababc913bc in memory_region_write_accessor ../softmmu/memory.c:493
(snip)
    #21 0xaaabab0e5e5c in pci_dma_write /home/alarm/q/var/qemu/include/hw/pci/pci_device.h:271
    #22 0xaaabab0e5e5c in e1000e_write_to_rx_buffers ../hw/net/e1000e_core.c:1474
    #23 0xaaabab105218 in e1000e_write_packet_to_guest ../hw/net/e1000e_core.c:1646
    #24 0xaaabab105218 in e1000e_receive_internal ../hw/net/e1000e_core.c:1776
    #25 0xaaabab0cb978 in net_tx_pkt_send_custom ../hw/net/net_tx_pkt.c:823
    #26 0xaaabab0fe608 in e1000e_tx_pkt_send ../hw/net/e1000e_core.c:682
    #27 0xaaabab0fe608 in e1000e_process_tx_desc ../hw/net/e1000e_core.c:762
    #28 0xaaabab0fe608 in e1000e_start_xmit ../hw/net/e1000e_core.c:953
    #29 0xaaabab102694 in e1000e_set_tctl ../hw/net/e1000e_core.c:2496
    #30 0xaaabab10f918 in e1000e_core_write ../hw/net/e1000e_core.c:3349
    #31 0xaaababc913bc in memory_region_write_accessor ../softmmu/memory.c:493
(snip)

previously allocated by thread T0 here:
    #0 0xfffecfc9ed24 in __interceptor_malloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0xfffece700ea4 in g_malloc (/usr/lib/libglib-2.0.so.0+0x70ea4)
    #2 0xaaabab0cc3d8 in net_rx_pkt_iovec_realloc ../hw/net/net_rx_pkt.c:77
    #3 0xaaabab0cc3d8 in net_rx_pkt_pull_data ../hw/net/net_rx_pkt.c:99
    #4 0xaaabab0cf33c in net_rx_pkt_attach_iovec_ex ../hw/net/net_rx_pkt.c:153
    #5 0xaaabab102e70 in e1000e_receive_internal ../hw/net/e1000e_core.c:1764
    #6 0xaaabab0cb978 in net_tx_pkt_send_custom ../hw/net/net_tx_pkt.c:823
    #7 0xaaabab0fe608 in e1000e_tx_pkt_send ../hw/net/e1000e_core.c:682
    #8 0xaaabab0fe608 in e1000e_process_tx_desc ../hw/net/e1000e_core.c:762
    #9 0xaaabab0fe608 in e1000e_start_xmit ../hw/net/e1000e_core.c:953
    #10 0xaaabab102694 in e1000e_set_tctl ../hw/net/e1000e_core.c:2496
    #11 0xaaabab10f918 in e1000e_core_write ../hw/net/e1000e_core.c:3349
    #12 0xaaababc913bc in memory_region_write_accessor ../softmmu/memory.c:493
(snip)

I omitted most stack trace entries of common QEMU code for brevity.

Looking at the logs, it seems use-after-free happened at line 1651 of hw/net/e1000e_core.c. The content of the line is:

if (iov_ofs == iov->iov_len) {

This means iov was a dangling pointer when AddressSanitizer got triggered. iov is assigned very early in the e1000e_write_packet_to_guest function and the function appears in the stack trace of free so it’s likely that the iov was valid at the beginning of the function but it got corrupted while the function progresses. This leads to two possible explanations for the bug:

1. It was a mistake to retrieve iov early and to keep it in the function.
2. It was a mistake to free iov in the function.

To prove explanation 2, I looked at the stack trace of free. The stack trace shows it tried to write the packet with DMA. In common sense, such DMA should write RAM, but the stack trace says it triggered another packet transmission. This sounds very wrong. In a nutshell, the following is what happened here:

1. The address of the MMIO register to trigger transmission was specified as the address of the received packet buffer.
2. Packet transmission was triggered.
3. The packet was looped back, and written to the address of the received packet buffer, which in reality points to the MMIO register specified in 1.
4. Another packet transmission and loopback happened, corrupting the first loopback.
5. The second loopback finishes, and the first loopback resumes.
6. The first loopback sees the corrupted state and AddressSanitizer triggers.

The first condition is very unlikely to happen in the real world, but a fuzzer and malicious actors it tries to mimic can create such an abnormal situation.

I could devise a fix with this explanation; such a fix should add a flag which indicates an MMIO access already happened and reject another MMIO access if the flag is set. The easiest way is to add such a flag to e1000e, but considering the entire QEMU program, it is likely that other devices need the same flag so I decided to add it to the common memory subsystem.

Eventually, I finished my fix and sent the upstream to find out there is already a fix for this kind of situation.

Lessons learned

• Sanitizers and fuzzers do help.
• It is sometimes better to fix the common code instead of fixing the device-specific code.
• Don’t forget to look for patches and issue reports for a common code bug ;)

Linux bug

I first tried Linux Test Project with e1000e instead of igb. That led to a kernel warning and device reset:

[  194.808645] NETDEV WATCHDOG: enp0s4 (e1000e): transmit queue 0 timed out
[  194.809321] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x6f4/0x7f0
[  194.810092] Modules linked in:
[  194.810388] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 6.3.0-rc6+ #24
[  194.810987] Hardware name: QEMU KVM Virtual Machine, BIOS edk2-stable202302-for-qemu 03/01/2023
[  194.811805] pstate: 61400005 (nZCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[  194.812461] pc : dev_watchdog+0x6f4/0x7f0
[  194.812845] lr : dev_watchdog+0x6f4/0x7f0
[  194.813233] sp : ffff800013227ad0
[  194.813552] x29: ffff800013227ad0 x28: ffff0000ef528510 x27: ffff80000e7cc000
[  194.814237] x26: 0000000000000003 x25: 0000000000000000 x24: 0000000000000000
[  194.814916] x23: ffff80000e7cc520 x22: ffff80000e7afac0 x21: ffff0000ef528420
[  194.815602] x20: ffff0000ef528000 x19: 1ffff00002644f6a x18: 0000000000000000
[  194.816284] x17: 1ffff00001f49ab0 x16: 0000000000000000 x15: 00000000a0f72ce0
[  194.818185] x14: 0000000000000000 x13: 0000000000000001 x12: ffff700002644ef3
[  194.820795] x11: 1ffff00002644ef2 x10: ffff700002644ef2 x9 : ffff80000831e908
[  194.823443] x8 : ffff800013227797 x7 : 00008ffffd9bb10e x6 : 0000000000000001
[  194.826174] x5 : ffff800013227790 x4 : 1fffe000184e560a x3 : dfff800000000000
[  194.828648] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000c272b040
[  194.831117] Call trace:
[  194.831960]  dev_watchdog+0x6f4/0x7f0
[  194.833231]  call_timer_fn+0x1f4/0x860
[  194.834378]  __run_timers+0x73c/0xa50
[  194.835201]  run_timer_softirq+0x54/0xb8
[  194.836098]  __do_softirq+0x304/0xe4c
[  194.836986]  ____do_softirq+0x14/0x28
[  194.837879]  call_on_irq_stack+0x24/0x30
[  194.838782]  do_softirq_own_stack+0x20/0x30
[  194.839736]  __irq_exit_rcu+0x23c/0x5e8
[  194.840615]  irq_exit_rcu+0x18/0x88
[  194.841421]  el1_interrupt+0x34/0x50
[  194.842314]  el1h_64_irq_handler+0x14/0x20
[  194.843321]  el1h_64_irq+0x68/0x70
[  194.844117]  cpuidle_idle_call+0x254/0x358
[  194.844856]  do_idle+0x124/0x1e0
[  194.845460]  cpu_startup_entry+0x2c/0x38
[  194.846174]  secondary_start_kernel+0x12c/0x168
[  194.846976]  __secondary_switched+0xa4/0xa8
[  194.847741] irq event stamp: 447143
[  194.848385] hardirqs last  enabled at (447142): [<ffff80000b28db18>] el1_interrupt+0x40/0x50
[  194.849731] hardirqs last disabled at (447143): [<ffff80000b28d4e0>] el1_dbg+0x20/0x90
[  194.850987] softirqs last  enabled at (446880): [<ffff800008011128>] __do_softirq+0x8f0/0xe4c
[  194.852350] softirqs last disabled at (447119): [<ffff80000801b184>] ____do_softirq+0x14/0x28

The call trace originates from a watchdog timer and does not tell much. The error message implies something with the transmission is wrong, but it is not enough to identify the bug.

I first minimized the reproduction case because repetitively running LTP takes too long. The test that triggered the warning was a test to make parallel HTTP requests. In the end, I figured out that this bug can be reproduced by running Apache on e1000e and requesting a 2-GiB file twice in parallel.

As I wrote earlier, it originates from a watchdog timer, which triggers a bit after a problematic behavior occurs. This means it is unlikely that you can understand the issue by looking at the last few QEMU traces.

It is necessary to inspect the kernel to identify the true origin of the warning. I attached GDB to the guest and put the breakpoint at the line that emits the warning. Below is the snippet from the kernel:

for (i = 0; i < dev->num_tx_queues; i++) {
	struct netdev_queue *txq;

	txq = netdev_get_tx_queue(dev, i);
	trans_start = READ_ONCE(txq->trans_start);
	if (netif_xmit_stopped(txq) &&
	    time_after(jiffies, (trans_start +
				 dev->watchdog_timeo))) {
		some_queue_timedout = 1;
		atomic_long_inc(&txq->trans_timeout);
		break;
	}
}

if (unlikely(some_queue_timedout)) {
	trace_net_dev_xmit_timeout(dev, i);
	WARN_ONCE(1, KERN_INFO "NETDEV WATCHDOG: %s (%s): transmit queue %u timed out\n",
	       dev->name, netdev_drivername(dev), i);
	netif_freeze_queues(dev);
	dev->netdev_ops->ndo_tx_timeout(dev, i);
	netif_unfreeze_queues(dev);
}

When the breakpoint hit, I typed: p *dev. This spams the console with all of the information related to the network device. Skimming the output, I found the value of the name member is enp0s4, which is the name of the e1000e device and confirms it is a problem with e1000e.

Investigating further requires a more educated methodology than just skimming p *dev. Below is the definition of netdev_get_tx_queue.

static inline
struct netdev_queue *netdev_get_tx_queue(const struct net_device *dev,
					 unsigned int index)
{
	return &dev->_tx[index];
}

p dev->num_tx_queues tells it’s 1, so dev->_tx[0] should be the queue that timed out. Then I looked at netif_xmit_stopped:

static inline bool netif_xmit_stopped(const struct netdev_queue *dev_queue)
{
	return dev_queue->state & QUEUE_STATE_ANY_XOFF;
}

where QUEUE_STATE_ANY_XOFF is defined as QUEUE_STATE_DRV_XOFF | QUEUE_STATE_STACK_XOFF. p dev->_tx[0].state showed QUEUE_STATE_DRV_XOFF was set so certainly this transmit queue is stopped.

Searching the kernel code base for QUEUE_STATE_DRV_XOFF, it seemed that the bit is cleared with netif_tx_wake_queue and netif_tx_start_queue. Looking for these symbols in drivers/net/ethernet/intel/e1000e led to the e1000_clean_tx_irq function, which means the guest is not receiving tx IRQ after the queue is stopped. I looked for kernel code that requests transmission since tx IRQ will be generated only after that, finding such code in the e1000_xmit_frame function.

netdev_sent_queue(netdev, skb->len);
e1000_tx_queue(tx_ring, tx_flags, count);
/* Make sure there is space in the ring for the next send. */
e1000_maybe_stop_tx(tx_ring,
		    (MAX_SKB_FRAGS *
		     DIV_ROUND_UP(PAGE_SIZE,
				  adapter->tx_fifo_limit) + 2));

if (!netdev_xmit_more() ||
    netif_xmit_stopped(netdev_get_tx_queue(netdev, 0))) {
	if (adapter->flags2 & FLAG2_PCIM2PCI_ARBITER_WA)
		e1000e_update_tdt_wa(tx_ring,
				     tx_ring->next_to_use);
	else
		writel(tx_ring->next_to_use, tx_ring->tail);
}

It seemed the kernel requests transmission by writing the first TDT register, which is pointed to by tx_ring->tail. If QEMU works properly, the last call for the e1000e_set_tdt function in the device emulation code before the reset happens should be matched with a call for the e1000e_set_interrupt_cause function with E1000_ICR_TXQ0 as its argument. To confirm this, I added printf calls to print when e1000e_set_tdt is called or e1000e_set_interrupt_cause is called with E1000_ICR_TXQ0, and enabled e1000e_core_ctrl_sw_reset trace.

Surprisingly, e1000e_set_interrupt_cause was always happening after the last e1000e_set_tdt call, meaning QEMU emulates the device properly. This made me question the kernel driver implementation. There was an earlier call of the e1000_maybe_stop_tx kernel function, which stops the Tx queue, in e1000_xmit_frame, but it was not followed with TDT register write:

int count = 0;

/* snip */

/* reserve a descriptor for the offload context */
if ((mss) || (skb->ip_summed == CHECKSUM_PARTIAL))
	count++;
count++;

count += DIV_ROUND_UP(len, adapter->tx_fifo_limit);

nr_frags = skb_shinfo(skb)->nr_frags;
for (f = 0; f < nr_frags; f++)
	count += DIV_ROUND_UP(skb_frag_size(&skb_shinfo(skb)->frags[f]),
			      adapter->tx_fifo_limit);

if (adapter->hw.mac.tx_pkt_filtering)
	e1000_transfer_dhcp_info(adapter, skb);

/* need: count + 2 desc gap to keep tail from touching
 * head, otherwise try next time
 */
if (e1000_maybe_stop_tx(tx_ring, count + 2))
	return NETDEV_TX_BUSY;

I added writel(tx_ring->next_to_use, tx_ring->tail); before the return statement and it worked.

So this was a kernel bug. Before submitting it as a fix, I tried to understand the intention of e1000_xmit_frame. Below is the summary of this function:

1. Check if there is sufficient space to put the current packet in the tx ring. Otherwise, return NETDEV_TX_BUSY.
2. Queue the current packet to the tx ring.
3. Stop the tx queue if there is no sufficient space for the next packet in tx ring.
4. Trigger transmission if there are no more packets or the tx is stopped.

And what I observed is that the check in step 1 was failing. In such a case, the previous e1000_xmit_frame should have stopped the tx queue in step 3 triggered transmission in step 4. In fact, Documentation/networking/netdevices.rst states the check in step 1 should never fail:

• NETDEV_TX_BUSY Cannot transmit packet, try later Usually a bug, means queue start/stop flow control is broken in the driver. Note: the driver must NOT put the skb in its DMA ring.

This implies it is a bug that the tx queue was not stopped in step 3 so I inspected the condition to stop the tx queue in step 3. e1000_maybe_stop_tx stops the queue if the tx queue has space specified with the size argument. In this case, the value is: (MAX_SKB_FRAGS * DIV_ROUND_UP(PAGE_SIZE, adapter->tx_fifo_limit) + 2)

This looks so complex and suspicious. I followed the computation of the size argument of the earlier e1000_maybe_stop_tx call triggering NETDEV_TX_BUSY in step 1 to find its maximum value. There are two increments of count at the beginning so that counts two. DIV_ROUND_UP(len, adapter->tx_fifo_limit) will be MAX_SKB_FRAGS at maximum. The later iterations on fragments will be summed into MAX_SKB_FRAGS * DIV_ROUND_UP(PAGE_SIZE, adapter->tx_fifo_limit) at maximum. In the end, 2 is added to count. Summarizing this, the maximum value will be:

2 + MAX_SKB_FRAGS + MAX_SKB_FRAGS * DIV_ROUND_UP(PAGE_SIZE, adapter->tx_fifo_limit) + 2
= (MAX_SKB_FRAGS + 1) * DIV_ROUND_UP(PAGE_SIZE, adapter->tx_fifo_limit) + 4

This is greater than the value used in step 4. A patch to update the value fixed the problem so I submitted the upstream and it’s now merged⁵. This bug was present since the introduction of the e1000e driver in 2007 and 15-year-old when it was fixed. It is surprising considering the affected workload is a plain common HTTP server.

Lessons learned

• Don’t naively trust the kernel even if the code is mature. (I also hit several other kernel bugs⁶⁷⁸⁹.)
• The kernel source visibility matters.
• Sometimes it’s necessary to track the bug both from the host and guest side.

GCC bug

We covered a kernel bug so the next thing to cover is obviously (?) a compiler bug. This compiler bug was found when running the Internet Protocol (IP) Pipeline application of DPDK under DPDK Test Suite. A DPDK program has a somewhat unconventional architecture that polls the hardware with an infinite loop, which confused GCC.

It was quite easy to locate the bug except it was something that looked impossible. It was clear that the DPDK application hung since the QEMU trace and the output of the application stopped. That usually implies QEMU did something wrong and the application got confused by that and froze. However, I could not find any bug looking at the source code of the application or adding lots of printf() and attaching GDB. It seemed the request for the worker thread is silently ignored so I looked at the implementation of the request queue to find no bug.

After observing this impossible behavior for an hour, eventually, I decided to trim down the infinite loop of the worker thread. There were several conditions to process requests and function calls so I removed each of them and replaced the request processing function with simple puts(). Below is what I got in the end:

#include <stdio.h>

int g;
int h;

int main(void)
{
	for (int i = 0; ; i++) {
		for (int j = 0; j < g; j++);

		if (i & 1) {
			if (h)
				continue;

			if (g)
				puts("a");

			g = 1;
		}
	}
}

This code should indefinitely emit a, but it did nothing. It was indeed a compiler bug.

To identify the buggy optimization pass I replaced the -O3 compiler flag with individual optimization flags listed in the documentation and performed binary search and derived the minimal reproducing combination of flags: -O1 -ftree-pre -ftree-partial-pre

The bug was reported to the upstream and promptly fixed¹⁰. Meanwhile, as it takes a long for a fix to be available on a distribution, I added -fno-tree-partial-pre to the compiler flag to disable the faulty optimization.

Lessons learned

• Something that looks impossible happens sometimes. (I hit yet another seemingly-impossible bug during igb development¹¹. Although it was not the fault of the compiler, it also teaches the same lesson.)
• The usual methodologies like minimizing the reproduction case or binary search do apply to unusual cases.

Contributing to the upstream

With all known bugs fixed, it’s time to submit the changes to the upstream. QEMU has extensive documentation for patch submission.

This process is a repetition of getting reviews and submitting a new version. New device emulation code is often large as in the case of igb, and such code takes many rounds of reviews; it took 9 versions before igb gets merged¹². As igb and e1000e have generic virtual-device fuzzing enabled, I also received reports of bugs found with the fuzzer and had to fix each of them.

Patience you must have, my young Padawan. - Master Yoda

The merge of the new device code marks the end of the initial development, but it is not the end of the entire development process. Introducing a new device makes you its maintainer so you will get patches and bug reports for it. Community involvement keeps the code being improved.

Conclusion

This series covered what new device emulation code for QEMU looks like by depicting the introduction of igb, which was a 6-month journey¹³. It is complex and time-consuming, but tractable if you break it down into several steps and divide-and-conquer. The new device code will benefit the entire QEMU community and downstream once you finish the process and get it merged.

In the previous post we discussed capturing a guest Windows crash dump using the vmcoreinfo device, the virtio-win driver called FwCfg, and the QEMU command called dump-guest-memory -w. In contrast to that method, this post considers a method to create a Complete Memory Dump of the 64-bit Windows guest, running inside a QEMU/KVM virtual machine, which doesn’t require any actions inside the guest. We will discuss the elf2dmp tool which can convert QEMU ELF dump obtained with dump-guest-memory to WinDbg-readable DMP format.

Preparation

Obtaining elf2dmp

The elf2dmp tool is a standalone executable, but its source code is a part of the QEMU project and resides in contrib/elf2dmp. For building along with QEMU, elf2dmp requires tools to be enabled in configure and depends on libcurl.

Also, elf2dmp can be obtained through package manager. For example, qemu-tools package contains elf2dmp on Fedora.

Capturing ELF dump

QEMU has a dump-guest-memory command to dump guest memory into an ELF file. In contrast to dump-guest-memory -w, no complex work will be done other than writing a snapshot of physical memory and register contexts to the ELF file.

Converting ELF to DMP

Run the following command to convert ELF dump file to DMP dump file:

elf2dmp memory.elf memory.dmp

The example output for VM with Windows Server 2022 and 4 CPUs:

4 CPU states has been found
CPU #0 CR3 is 0x000000012b109002
CPU #0 IDT is at 0xfffff8007d545000
CPU #0 IDT[0] -> 0xfffff80081e88100
Searching kernel downwards from 0xfffff80081e88000...
Data directory entry #0: RVA = 0x0000c000
Data directory entry #0: RVA = 0x00008000
Data directory entry #0: RVA = 0x00108000
Data directory entry #0: RVA = 0x00024000
Data directory entry #0: RVA = 0x0005b000
Data directory entry #0: RVA = 0x00065000
Data directory entry #0: RVA = 0x00007000
Data directory entry #0: RVA = 0x000190b0
Data directory entry #0: RVA = 0x0000e000
Data directory entry #0: RVA = 0x00022000
Data directory entry #0: RVA = 0x00063000
Data directory entry #0: RVA = 0x00012000
Data directory entry #0: RVA = 0x000356e0
Data directory entry #0: RVA = 0x0013a000
KernBase = 0xfffff80081400000, signature is 'MZ'
Data directory entry #6: RVA = 0x000154b8
CodeView signature is 'RSDS'
PDB name is 'ntkrnlmp.pdb', 'ntkrnlmp.pdb' expected
PDB URL is https://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/adc00fa5fc34456ba16e2687457240991/ntkrnlmp.pdb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11.5M  100 11.5M    0     0  1046k      0  0:00:11  0:00:11 --:--:-- 1674k
KdDebuggerDataBlock: 0x0000000000c00000(24:'.data') + 0x00000a30 = 0x000c00a30
KdDebuggerDataBlock = 0xfffff80082000a30
KdVersionBlock: 0x0000000000c00000(24:'.data') + 0x00015508 = 0x000c15508
KdVersionBlock = 0xfffff80082015508
KiWaitNever: 0x0000000000d0f000(25:'ALMOSTRO') + 0x00001bd0 = 0x000d10bd0
KiWaitNever = 0xfffff80082110bd0
KiWaitAlways: 0x0000000000d0f000(25:'ALMOSTRO') + 0x00001e38 = 0x000d10e38
KiWaitAlways = 0xfffff80082110e38
KdpDataBlockEncoded: 0x0000000000c00000(24:'.data') + 0x0010b148 = 0x000d0b148
KdpDataBlockEncoded = 0xfffff8008210b148
[KiWaitNever] = 0x1a5b030053af49fd
[KiWaitAlways] = 0x0029d7b15ae2d898
Decoding KDBG header...
Owner tag is 'KDBG'
Decoding KdDebuggerDataBlock...
Filling context for CPU #0...
Filling context for CPU #1...
Filling context for CPU #2...
Filling context for CPU #3...
Writing header to file...
Writing block #0/13 to file...
Writing block #1/13 to file...
Writing block #2/13 to file...
Writing block #3/13 to file...
Writing block #4/13 to file...
Writing block #5/13 to file...
Writing block #6/13 to file...
Writing block #7/13 to file...
Writing block #8/13 to file...
Writing block #9/13 to file...
Writing block #10/13 to file...
Writing block #11/13 to file...
Writing block #12/13 to file...

Since elf2dmp downloads the PDB file from Microsoft public symbol server, the network access is required. Also, both read and write accesses to the ELF dump file is required, although the ELF file is guaranteed to remain unchanged.

After the conversion, the DMP file can be opened in WinDbg.

Typical issues

QEMU generates ELF dump with 0400 rights, but elf2dmp requires read and write accesses, so the file rights should be modified to give these permissions to the user. Otherwise elf2dmp will say something like that:

Failed to map ELF dump file 'memory.elf'
Failed to initialize QEMU ELF dump

Internals

As described in the previous posts in this series, Complete Memory Dump consists of a header and a snapshot of the physical memory of the system (which contains, among other things, the register contexts for all CPUs). The physical memory and register contexts are taken from the ELF dump file and saved to the DMP file, just like dump-guest-memory -w does, so the main challenge for elf2dmp is to fill the dump header correctly and decrypt the KdDebuggerDataBlock structure.

ELF

ELF dump file produced by QEMU with dump-guest-memory consists of a PT_NOTE with registers for each virtual CPU and several PT_LOAD with physical memory snapshots:

ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              CORE (Core file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          0 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         14
  Size of section headers:           0 (bytes)
  Number of section headers:         0
  Section header string table index: 0

There are no sections in this file.

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  NOTE           0x0000000000000350 0x0000000000000000 0x0000000000000000
                 0x0000000000000cc0 0x0000000000000cc0         0x0
  LOAD           0x0000000000001010 0x0000000000000000 0x0000000000000000
                 0x0000000000018000 0x0000000000018000         0x0
  LOAD           0x0000000000019010 0x0000000000018000 0x0000000000018000
                 0x0000000000001000 0x0000000000001000         0x0
  LOAD           0x000000000001a010 0x0000000000019000 0x0000000000019000
                 0x0000000000001000 0x0000000000001000         0x0
  LOAD           0x000000000001b010 0x000000000001a000 0x000000000001a000
                 0x0000000000001000 0x0000000000001000         0x0
  LOAD           0x000000000001c010 0x000000000001b000 0x000000000001b000
                 0x0000000000001000 0x0000000000001000         0x0
  LOAD           0x000000000001d010 0x000000000001c000 0x000000000001c000
                 0x0000000000084000 0x0000000000084000         0x0
  LOAD           0x00000000000a1010 0x00000000000a0000 0x00000000000a0000
                 0x0000000000010000 0x0000000000010000         0x0
  LOAD           0x00000000000b1010 0x00000000000c0000 0x00000000000c0000
                 0x0000000000004000 0x0000000000004000         0x0
  LOAD           0x00000000000b5010 0x00000000000c4000 0x00000000000c4000
                 0x000000000001c000 0x000000000001c000         0x0
  LOAD           0x00000000000d1010 0x00000000000e0000 0x00000000000e0000
                 0x0000000000020000 0x0000000000020000         0x0
  LOAD           0x00000000000f1010 0x0000000000100000 0x0000000000100000
                 0x000000007ff00000 0x000000007ff00000         0x0
  LOAD           0x000000007fff1010 0x00000000c0000000 0x00000000c0000000
                 0x0000000001000000 0x0000000001000000         0x0
  LOAD           0x0000000080ff1010 0x0000000100000000 0x0000000100000000
                 0x0000000080000000 0x0000000080000000         0x0

The following QEMUCPUState structure is available for each virtual x86 CPU:

typedef struct QEMUCPUSegment {
    uint32_t selector;
    uint32_t limit;
    uint32_t flags;
    uint32_t pad;
    uint64_t base;
} QEMUCPUSegment;
        
typedef struct QEMUCPUState {
    uint32_t version;
    uint32_t size;
    uint64_t rax, rbx, rcx, rdx, rsi, rdi, rsp, rbp;
    uint64_t r8, r9, r10, r11, r12, r13, r14, r15;
    uint64_t rip, rflags;
    QEMUCPUSegment cs, ds, es, fs, gs, ss;
    QEMUCPUSegment ldt, tr, gdt, idt;
    uint64_t cr[5];
    uint64_t kernel_gs_base;
} QEMUCPUState;

Processing of QEMU ELF dump file is implemented in qemu_elf.c.

DirectoryTableBase

One of the header fields is DirectoryTableBase (DTB). DTB contains the physical address of the root of the virtual-to-physical page address translation. In the method based on vmcoreinfo device and guest driver the field was filled by the guest OS, but elf2dmp had to calculate this value based on data from the ELF file. DTB is also required to access guest kernel virtual memory when searching for KdDebuggerDataBlock and other header fields. In elf2dmp addrspace.c is responsible for handling the physical and virtual address spaces.

The elf2dmp finds DTB value for several tries. The correctness of the value is checked by accessing 0xfffff78000000000 in kernel virtual address space. This address belongs to SharedUserData. There are 3 attempts in total:

1. Set DTB to CPU#0 CR3 taken from ELF.
2. Set DTB to CR3 for each CPU context from ELF with a negative GS base. Negative GS base means the CPU is in Windows kernel context.
3. Set temporary DTB to CPU#0 CR3 from ELF. Set DTB to a value from [IA32_KERNEL_GS_BASE+0x7000] by using temporary DTB to access virtual memory, where IA32_KERNEL_GS_BASE is an MSR value from ELF.

After finding the DTB, elf2dmp can access both the user and kernel virtual address space. Now elf2dmp needs to find where in virtual memory the Windows kernel PE image is located.

Kernel base

Windows kernel image is represented by ntoskrnl.exe. This PE image exports all the debugging symbols required by elf2dmp, such as nt!KdDebuggerDataBlock and nt!KdVersionBlock. To be able to resolve the debug symbols, elf2dmp needs to find the address where the kernel image is loaded.

To find the kernel base address, elf2dmp relies on the fact that there are interrupt handlers somewhere inside the Windows kernel image. Because the IDT base register value is stored in the ELF dump, elf2dmp takes the first address from the interrupt descriptor table. Now elf2dmp can scan the virtual memory downwards starting at that address using the following kernel image properties:

• Any PE image starts with the signature MZ.
• The kernel image is aligned to the page boundary
• The kernel export table lists the image name as ntoskrnl.exe

Once elf2dmp has access to the loaded kernel image, it needs to find debugging symbols for it.

ntkrnlmp.pdb

Debugging symbols are stored in PDB format. The PDB file for the Windows kernel (ntkrnlmp.pdb) and some other system modules can be downloaded from Microsoft public symbol store. WinDbg also downloads PDBs from this location. To do this, elf2dmp must form a download UR which looks like https://msdl.microsoft.com/download/symbols/ntkrnlmp.pdb/adc00fa5fc34456ba16e2687457240991/ntkrnlmp.pdb. The link contains a PDB name and a PDB hash. The PDB hash is calculated from a GUID from the debug directory of the kernel PE image. A routine named pe_get_pdb_symstore_hash does this in elf2dmp.

When the URL is generated, elf2dmp downloads the PDB file. This is described in download.c.

Now elf2dmp can parse the PDB file and resolve the required debugging symbols with the pdb_find_public_v3_symbol function from pdb.c.

KdDebuggerDataBlock

The importance of the KdDebuggerData block structure was described in the previous post of the series. To find this structure in the virual address space, elf2dmp resolves the symbol nt!KdDebuggerDataBlock. Recall that when Debug Mode is disabled, this structure is encrypted. The encryption function is parameterized by the symbols nt!KdpDataBlockEncoded, nt!KiWaitAlways and nt!KiWaitNever and looks like this Here ror is right circular bit shift and bswap is 64-bit byte swap. Obviously, the function is easily reversible. Since elf2dmp now has access to all the debug symbols, knowing these parameters it can decrypt the structure.

KdVersionBlock

The dump contains fields such as MinorVersion, MajorVersion, MachineType and SecondaryDataState. Their copies are available in a structure called KdVersionBlock, the address of which can also be resolved.

Dump file writing

When DirectoryTableBase, KdDebuggerDataBlock and KdVersionBlock are available, elf2dmp can completely fill the dump header. The tool always sets the BugcheckCode to 0x161 (LIVE_SYSTEM_DUMP). After the physical memory blocks are copied from the ELF file to the DMP file, the Complete Memory Dump is ready.

Conclusion

We have discussed the usage and internals of elf2dmp tool which can help to capture the Complete Memory Dump of the 64-bit Windows guest running under QEMU, without any actions inside the guest. This is a useful tool when the debug dump is needed, but the vmcoreinfo device is not connected or virtio-win drivers are not installed in the guest.

Microsoft Validation OS is a lightweight, fast, and customizable Windows 11-based operating system that you can use on the factory floor to diagnose, mitigate and repair hardware defects during Windows device manufacturing. Validation OS boots into a Command Line environment to increase reliability on the factory floor and supports running Win32 apps, smoothing the transition from early hardware bring-up to retail OS and apps development.

Image customization in general

Microsoft provides enough documentation to customize image and add necessary software:

• Mount and customize a Validation OS image
• Launching an app or command when Validation OS starts

The important point for all steps is select proper WIM image. From Microsoft note:

• Use index:1 if you’ll be applying your image to a device’s hard drive.
• Use index:2 if you’ll be using the image to boot from a USB drive.

In case when we prepare image for VM boot from ISO we should use /index:2 in the DISM command.

What should be done for SSH access:

1. Generate SSH key pair
2. Copy SSH public key into Validation OS as a authorized key
3. Add OpenSSH server to autostart
4. Export new image

Image preparation

Microsoft provide non-bootable Validation OS ISO. This image should be used only as a base.

1. Download Validation OS from the link above and extract is to some folder (C:\validation_os)
2. Start the Deployment and Imaging Tools Environment as an administrator.
3. Run copype to create a working copy of the Windows PE files.

   copype amd64 C:\WinPE_amd64
   

4. Copy Validation OS wim and replace the WinPE boot.wim

   xcopy C:\validation_os\ValidationOS.wim C:\WinPE_amd64\media\sources\boot.wim
   

5. Create a folder where you’ll mount your image

   md c:\mount
   

6. Use DISM to mount the image

   DISM /Mount-Image /imagefile:"C:\WinPE_amd64\media\sources\boot.wim" /index:2 /MountDir:"C:\mount"
   

7. Generate SSH key pair

   md c:\validation_ssh
   ssh-keygen -t ed25519 -C "your_email@example.com" -f c:\validation_ssh\id_ed25519
   

8. Configure SSH server and add SSH public key as a authorized key

   copy C:\mount\Windows\System32\OpenSSH\sshd_config_default C:\mount\ProgramData\ssh\sshd_config
   type c:\validation_ssh\id_ed25519.pub > C:\mount\ProgramData\ssh\administrators_authorized_keys
   

9. Create and configure the SSH startup script C:\mount\sshd.bat with the following content

   rem "Generate SSH host keys"
   cmd /c "ssh-keygen -A"
   rem "Start SSH server"
   cmd /k "c:\Windows\System32\OpenSSH\sshd.exe"
   

10. Add script to startup via registry

    reg load HKLM\Image_SOFTWARE C:\mount\windows\system32\config\software
    reg add "HKLM\Image_SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v Shell /t REG_SZ /F /D "cmd /k c:\sshd.bat"
    reg unload HKLM\Image_SOFTWARE
    

11. Unmount image and apply changes

    DISM /Unmount-Image /MountDir:"C:\mount" /Commit
    

12. Generate bootable iso

    MakeWinPEMedia /ISO C:\WinPE_amd64 C:\ValidationOS_SSH.iso
    

Usage of created ISO

1. Boot any VM with ethernet adapter from ValidationOS_SSH.iso
2. Use the following SSH command to connect

   ssh -i c:\validation_ssh\id_ed25519 -o IdentitiesOnly=yes Administrator@<VM_IP>
   

Conclusion

In conclusion, configuring the OpenSSH server on Microsoft Validation OS opens up a world of possibilities for secure remote access within this specialized testing and validation environment. By following the steps outlined in this article, you can establish a robust infrastructure that ensures the confidentiality, integrity, and authenticity of your remote connections. With the OpenSSH server in place, you can confidently administer and manage Microsoft Validation OS, transfer files securely, and streamline your workflows. Embracing the power of OpenSSH within the Microsoft Validation OS environment not only enhances your productivity but also reinforces your commitment to maintaining a strong security posture.

1. Writing a boilerplate and adding basic features
2. Adding QTest
3. Adding advanced features and libvirt support
4. Running more tests and increasing feature coverage
5. Debugging
6. Contributing to the upstream

This post discusses 3 and 4. The goal is to make the device implementation, which was introduced in the previous post, usable for practical purposes.

Adding advanced features and libvirt support

Determining test workloads

The previous post discussed bringing up the first device implementation and letting the guest initialize the device. In the case of igb, it is not very difficult to bring features implemented for e1000e, the predecessor in Intel’s NIC product line, to igb as these devices are very similar, and igb development started by forking e1000e’s code. However, this initial state of the device emulation is not really usable; you would expect the new device will bring features not present in the existing devices.

Therefore, the next step of the device development will be to implement such advanced features. As described in the first post, the most notable features of igb are VMDq and SR-IOV. VMDq implements L2 switching among (nested) VMs, and SR-IOV allows to attach some tx/rx queues as a virtual function to a VM. The first thing to do when implementing a new feature is to fabricate a workload that will exercise it. To exercise VMDq and SR-IOV features, we have to perform the following operations on a Linux guest:

• Enabling SR-IOV virtual functions
• Sending and receiving packets with different combinations of peers:
  • The physical function and an external host
  • The physical function and a virtual function
  • A virtual function and an external host
  • A virtual function and the physical function
  • A virtual function and another virtual function

The physical function is the PCI function always available. Testing the various combinations of peers is important to ensure the L2 switching capability VMDq provides correctly functions.

Once the workloads were determined, you can test it and implement missing features and fix bugs for them, just like the basic features are tested and fixed. This process is incremental so it is important to prevent regressions and to ensure forward progress. It is better to continuously run qtest and simple workloads like ping and curl. Commit changes to Git frequently so that you can isolate regressions by e.g., running git bisect.

Note that it is not necessary to complete this step before sending the new device implementation to the upstream. It is actually often better to send the patches upstream as soon as you get confidence in the implementation of the basic functionalities. A small patch that implements the basics is easier to review than a long patch that includes many advanced features. The upstream reviewers can also discuss the design of the implementation early and you can correct it before writing more changes if you send patches early. It is even possible to send patches before you thoroughly test them as RFC.

For igb, I sent the first version of patches when VMDq and SR-IOV features become usable on Linux. The other target platforms, Windows and DPDK followed later.

libvirt support

libvirt is a very popular frontend for QEMU so we need it to have the support for the new device to some extent at least. The details of the code change vary by the type of the device, but the principle same with QEMU applies: find a similar device, copy the code for it, and extend it if necessary.

libvirt often requires few changes for a new device. The libvirt change for igb is just a copy of e1000e support, and you can see it at: https://gitlab.com/akihiko.odaki/libvirt/-/commit/f0e85eed7398c35b77a775d8178edc23b757ae6d

Running more tests and increasing feature coverage

As you add more and more features, it gets harder to continuously test all of the implemented features. Automated testing is essential to keep various features working.

It is not common that there are tests for the emulated device as hardware vendors do not release tests they presumably created during the hardware development. It is theoretically possible to develop a complete unit testing suite that covers all the features, but in practice, it consumes too much time.

A more practical way to introduce automated testing is to borrow existing test suites for drivers and platforms. As they target more high-level constructs, sometimes they do not suit well for device testing. For example, such a test suite may include tests for a pure software construct, which costs time and contributes nothing to device development. Such a test suite neither tests the device functionalities the platform does not use. Nevertheless, high-level tests are pragmatically very useful as they can ensure the success of the ultimate objective: to get the device to work for the target platform.

Usually, there are test suites specialized for a device type and an operating system so you have to choose appropriate ones. Such test suites for NICs and popular platforms will be presented below:

ethtool

ethtool is a Linux utility for controlling network devices. The kernel exposes an ioctl that triggers device tests when requested by ethtool. The coverage of the tests is fairly limited but it is so easy to use that it is automated for igb using Avocado testing framework. The automating script is available at: https://gitlab.com/qemu-project/qemu/-/blob/v8.0.0/tests/avocado/netdev-ethtool.py

Linux Test Project (LTP)

Its README describes itself as follows:

The LTP testsuite contains a collection of tools for testing the Linux kernel and related features.

As the README states, LTP is not specific to networking and contains various tests for Linux. The network tests have their own README.

The main advantage of LTP is that it contains various kinds of tests from ping to applications like HTTP, FTP, telnet, etc. The downside is that its network tests are somewhat unmaintained, and require manual setup of network applications.

I have sent patches to fix the tests for a modern system, and wrote some scripts to manage VMs. These scripts are particularly helpful as they allow us to easily reproduce the model system for manual testing/debugging and to automatically run the LTP. The automated scripts test all of the combinations of peers described above.

As Linux was the first target platform, the LTP was the first test suite igb passes. However, later changes for the other platforms sometimes regressed the Linux support and automated LTP runs greatly helped to spot such regressions.

Windows Hardware Lab Kit (HLK)

The Windows Hardware Lab Kit (Windows HLK) is a test framework used to test hardware devices and drivers for Windows 11, Windows 10 and all versions of Windows Server starting with Windows Server 2016.

Windows Hardware Lab Kit | Microsoft Learn

Windows HLK includes comprehensive tests for network devices and exercises more advanced features like VMDq and SR-IOV. We maintain AutoHCK, a tool for automating HLK runs on QEMU so it is quite easy to set up VMs using it. Windows HLK spotted several bugs which were missed by the LTP in the case of igb.

The downside of Windows HLK, or rather Windows platform is that it is closed-source. When I first tried igb with Windows, its driver just hung and repeated resets without any helpful information. You can only see traces from QEMU and the disassembly of the driver. The next post will discuss debugging with this closed-source platform, but you will want to get the device to work with an open-source platform before testing with Windows.

DPDK Test Suite (DTS)

DPDK is not an operating system like Linux and Windows, but an application framework for low-level packet processing. DPDK implements its own driver to implement the interaction with the hardware in the userspace and to eliminate context switches with the kernel. The DPDK driver needs to be tested as it employs low-level hardware access as operating system drivers do.

The nice thing with DPDK is that it can be debugged as a normal user-space application. It can also use more minor device features, and DPDK Test Suite covers details of such features. However, DPDK is not for a user-facing application like curl so DTS cannot ensure the usability of the device emulation in common situations.

Conclusion

This post discussed how to add advanced features and libvirt support, and to test them. This part of device development takes a significantly long time so you need some strategy. You have to choose features and tests appropriate for the current development stage and sometimes send (RFC) patches to get design reviews.

The development process is not always smooth, and you may have difficulty getting a new feature to function, or the device implementation may suffer from regressions. Therefore the next post will mostly be spent on one thing: debugging.

Writing a boilerplate and adding basic features

QEMU provides infrastructure common for different devices. The first step of device emulator development is to write a boilerplate to utilize it. Then, you can implement some basic features like MMIO register accesses.

There are three possible options when implementing a new device.

1. Extending the existing code for a similar device.
2. Copying and rewriting the existing code for a similar device.
3. Writing from scratch.

The first option is the easiest if the new device has small differences from an existing device or it is a strict superset of one, but in the case of igb, there are so many differences from the predecessor, e1000e, that we had to choose option 2.

Even if there is no device that is particularly similar to the device you are going to implement and you are going with option 3, it is still a good idea to look for a device recently added that uses the same type of bus (e.g., PCIe) or implements the same category of feature (e.g., NIC). This ensures that the new device implementation follows the convention already established in the QEMU code base.

The initial goal of the development is to get the device to work. Concretely, this implies:

• The code can be built.
• The operating system can see and initialize the device.

If you choose option 2, the first thing to do after copying the code is to rename C identifiers to make the copied code buildable. Once renaming the identifiers is done, rename the device type name.

In the copied code, the type name is defined with the following line:

#define TYPE_IGB "e1000e"

We can simply replace it with igb.

#define TYPE_IGB "igb"

Before doing any more changes, we ensure that we can build the code and it functions as a drop-in replacement for e1000e. First, write a simple command line to run QEMU with e1000e.

build/qemu-system-aarch64 -M virt -device e1000e,netdev=netdev -netdev user,id=netdev...

Make sure this command line works, and then replace e1000e with igb.

build/qemu-system-aarch64 -M virt -device igb,netdev=netdev -netdev user,id=netdev...

Now the new device should be working 🎉

…but the operating system sees e1000e instead of igb because the actual code is just simply copied from e1000e. Before making it function as igb, let’s check in this first version of the new device into Git so that you can always go back to the buildable state. Implementing a device is complicated and it is likely to have regressions during development; frequently committing to Git may save hours in such a situation.

We can then change the identifiers exposed to the guest to make it function as a new device. We have the following lines in the class initialization function of the copied code.

    c->realize = igb_pci_realize;
    c->exit = igb_pci_uninit;
    c->vendor_id = PCI_VENDOR_ID_INTEL;
    c->device_id = E1000_DEV_ID_82574L;
    c->revision = 0;
    c->romfile = "efi-e1000e.rom";
    c->class_id = PCI_CLASS_NETWORK_ETHERNET;

    rc->phases.hold = igb_qdev_reset_hold;

    dc->desc = "Intel 82574L GbE Controller";
    dc->vmsd = &igb_vmstate;

Replace the PCI identifiers and other fields.

    c->realize = igb_pci_realize;
    c->exit = igb_pci_uninit;
    c->vendor_id = PCI_VENDOR_ID_INTEL;
    c->device_id = E1000_DEV_ID_82576;
    c->revision = 1;
    c->class_id = PCI_CLASS_NETWORK_ETHERNET;

    rc->phases.hold = igb_qdev_reset_hold;

    dc->desc = "Intel 82576 Gigabit Ethernet Controller";
    dc->vmsd = &igb_vmstate;

Now the guest thinks the device is igb, but the actual implementation is e1000e so the guest cannot properly initialize it. In theory, you can implement all features igb has according to the datasheet and get the working code, but it’s unrealistic considering that the datasheet has even 960 pages. A practical approach is to read the device driver code and implement features accessed during the initialization.

Linux is a good target for the initial bring-up. Its drivers are open-source, and, combined with QEMU, it can provide interactive debugging experience with gdb which is comparable with usual user-space development.

Once you get confidence that you implemented everything necessary for the initialization, run the guest and debug it until it works. We’ll cover the details of debugging technique in a future post, but basically, normal debugging tools like GDB work well with QEMU. QEMU also has a tracing feature which greatly helps debugging.

-trace e1000* -trace e1000e* -trace igb*

Once it works, check in the code again.

Fortunately, this step was already done by Gal Hammer and Marcel Apfelbaum so I had working code when I started working on igb. Nevertheless, I performed this step by myself again to understand the difference between igb and e1000e, and to backport the recent changes made for e1000e to igb, which was already diverged from e1000e.

Adding QTest

QTest is a device emulation testing framework and provides low-level access to the emulated device such as MMIO access. While QTest does not run a real guest and is not suitable for integration testing, it can cheaply test specific aspects of the device.

A recommended usage of QTest is to write tests for the basic features. As QTest is ran by the CI, it can effectively prevent fatal regressions after the code is merged into the upstream. Of course, you can run it during development too: QTest is so fast that you can run the test cases for a device always after compiling.

It is also possible to use the framework in a manner more like unit testing and write tests for most behaviors. Some argue that low-level software components are not suited for unit testing and while I believe otherwise, it is also true that unit testing incurs significant cost. For igb, I deemed the gain from the unit testing to be limited compared to the cost.

The process of writing QTest is not so different from writing the device emulation itself: copy the existing code, rename identifiers, and tweak it for the new device.

Conclusion

With these steps finished, now we have:

• Buildable code
• Running Linux guest
• Basic QTest

These are the foundations of the further development. New features will be added and more testing will be done in the future, but that cannot be done without these essential components.

The next post will discuss implementing advanced features and libvirt support to make the device practically usable.

This is a series of four posts and presents challenges and their resolutions by describing the process of the development of igb, a device implementation we recently contributed to the project. While igb emulates a PCI Express (PCIe) network interface card (NIC) and this series will illustrate some aspects of PCIe and NIC, many parts of it should be generally applicable for new device emulator development.

* By the way, for igb development, I used a M2 MacBook Air, which contains Apple Silicon and runs Asahi Linux, a community port of Linux for this system. While there are some pitfalls, it has decent KVM support and provides a nice environment for QEMU development.

QEMU device emulator development in a nutshell

Briefly speaking, developing a new QEMU device emulator involves the following steps:

1. Writing a boilerplate and adding basic features
2. Adding QTest
3. Adding advanced features and libvirt support
4. Running more tests and increasing feature coverage
5. Debugging
6. Contributing to the upstream

Writing a boilerplate is necessary to integrate the new device into QEMU. After completing the boilerplate, basic features need to be added so that the guest can access more advanced features.

qtest is QEMU’s testing framework and useful to ensure there is no regression in essential features.

With basic features covered, more advanced but non-essential features can be added. libvirt is a popular frontend of QEMU, and it is important for usability to make the device and its features accessible from libvirt.

During the development, tests will reveal regressions or features which are missing but necessary for a workload. It is often unclear what is making tests fail and a significant debugging effort will be required.

Finally, the code will be sent upstream. By upstreaming, you can ensure that the code will be updated as the other parts of the code base change, and others can appreciate the contributed code.

This series details each step and provides insight into how a device emulator can be developed.

The next post will describe steps 1 and 2. The third post will describe steps 3 and 4. The last post will be about steps 5 and 6.

What is igb?

Before describing the device development process in depth, first, let me introduce the hardware which will be emulated by igb.

igb is actually the name of a family of Intel’s NIC. In particular, QEMU’s igb implementation emulates 82576, launched in 2008.

QEMU already has emulation code for several Intel NICs. The following device implementations are present in the current QEMU (from oldest hardware to newest one):

• i82596
• eepro100 for 8255x
• e1000 for 8254x
• e1000e for 82574L

igb succeeds e1000e in Intel’s product line. As the name suggests, e1000 and newer devices have gigabit bandwidth.

While paravirtualization is common these days, igb emulates actual hardware. In most cases, a para-virtualized device is superior to emulated hardware, but there are a few advantages emulated hardware has:

• Widely available driver. In particular, Windows bundles the driver for igb and works with it out-of-box.
• Feature-richness. Particularly features which make little sense for software implementation may be absent in a para-virtualized device, but having such features emulated may be required for testing purposes.

The latter is the major motivation for introducing igb. igb has so many features added since e1000e so the later part of this section lists particularly notable features of igb.

VMDq

VMDq is a feature to assign queues to different virtual machines. (nested virtual machine in this case as the underlying system with igb is emulated/virtualized). A network interface typically has a “queue” for each of tx and rx, and the software puts a packet to transmit to the Tx queue and retrieves the received packet from the rx queue.

e1000e has two queues for each of tx and rx to avoid contention of queues on a multi-processor system. For tx, two processors can submit packets to transmit at the same time by using different queues. For rx, e1000e can assign packets from different peers to different queues; this feature is called RSS (receive side scaling). Each queue can be processed in parallel by two processors, improving the throughput.

igb has 16 queues for each of tx and rx. These queues can be used for effective use of multiple processors as in e1000e, but igb also allows to use the queues for efficient isolation of virtual machines. This isolation is achieved by hardware-level L2 switching.

In a conventional system, the hypervisor emulates a NIC for each VMs and performs L2 switching at the software level to route packets to the appropriate VM. For example, the diagram below illustrates the case when VMs are receiving packets from an external host:

In this diagram, packets are first stored in queues determined with RSS. The hypervisor acquires these stored packets and performs L2 switching to route them to appropriate VMs. This switching imposes a substantial load on the host.

With VMDq enabled, igb automatically performs this switching and assigns packets to different queues accordingly, reducing the load of the host:

This L2 switching is achieved with the packet filtering capability igb has. e1000e can filter packets by the destination MAC address, and igb extends this feature further by allowing the driver to configure different filters for queues.

SR-IOV

VMDq is more effective when combined with igb’s SR-IOV feature. SR-IOV is an extension of PCIe to allow a device to export several virtual functions which looks like different devices for the operating system. Virtual functions can be directly assigned to VMs (by using VFIO in the case of Linux), which removes the hypervisor from the traffic path between the VM and hardware:

The context switch between the hypervisor and VM incurs significant overhead so bypassing the hypervisor can result in a great performance gain.

Summary

This article introduced QEMU device emulator development in general and igb, QEMU’s new NIC emulation. The next post will discuss the first step of a new device emulator development: writing a boilerplate and adding basic features. The goal of this step is to get the emulated device to work (even though the functionality may be limited) and to create the foundation for further development.

Preparation

This method is applicable for all Windows versions from Windows Server 2012 to, at least, Windows Server 2022 (in other words, any of the versions supported by Microsoft as of 2023) for both 32-bit and 64-bit platforms. It requires two things:

• vmcoreinfo device on the QEMU side
• FwCfg driver on the guest side

vmcoreinfo

If QEMU is run from the command line, add -device vmcoreinfo to argument list. In case of libvirt, add <vmcoreinfo state='on'/> to the XML config under <features></features>.

virtio-win

Install FwCfg driver from virtio-win to the guest Windows. Note: There is also an old driver which doesn’t support 32-bit platform, but supports deprecated Windows kernel versions starting from 6.1 (Windows 7 and Windows Server 2008 R2).

Capturing crash or live dump

When either the system is still alive, or a BSoD occurs and the system does not reboot (this can be achieved using PVPanic device or unchecking “Automatically restart” option), the following command can be executed in QEMU monitor:

dump-guest-memory -w memory.dmp

The Complete Memory Dump will be written to memory.dmp file. Then it can be opened in WinDbg like any other dump.

Typical issues

These are messages typically reported by dump-guest-memory -w:

• win-dump: invalid vmcoreinfo note size: most likely, guest driver is not running yet (too early stage of Windows kernel initiatization) or not present at all, so the dump can’t be captured
• win-dump: number of QEMU CPUs is bigger than NumberProcessors (%u) in guest Windows: mostly likely, a desktop version of Windows is running and limits number of CPUs, but the dump can still be captured

Internals

This section is dedicated to internals of the method discussed above.

Anatomy of the Complete Memory Dump

The structure of the Complete Memory Dump consists of several large parts:

• A one-page (4 KiB) header on a 32-bit system and 2 pages (8 KiB) on a 64-bit system.
• Snapshots (so-called “runs”) of contiguous regions of physical memory (Run #0 - Run #N).

Simplified scheme of the Complete Memory Dump

The number of runs, start addresses, and lengths are stored in the header. The dumpchk.exe utility, which comes with the WinDbg debugger, is able to show information about the physical memory stored in the dump file. Usually there is more than one region due to PCI holes, because some of the physical address space is used for communication with peripherals and is not suitable for data storage. Therefore, not all of the physical address space is stored in the dump file. On 64-bit platform, each physical memory region is described in the dump header with the _PHYSICAL_MEMORY_RUN64 structure, which stores the start and the length of the region:

struct _PHYSICAL_MEMORY_RUN64 {
    ULONG64 BasePage;
    ULONG64 PageCount;
}

The list of physical memory regions is displayed by the dumpchk.exe

In addition to information about physical memory regions, the dump header contains other fields required for debugger operation, including:

• BugcheckData - error code and 4 parameters that describe the reason of the crash
• RequiredDumpSpace - total dump size in bytes
• DirectoryTableBase - the physical address of the root of the virtual-to-physical address translation for the debugger
• PsLoadedModuleList - the virtual address of the list of loaded executable modules
• PfnDatabase - virtual address of page frame number database
• MinorVersion, MajorVersion - two fields that together determine the version of Windows kernel
• KdDebuggerDataBlock - virtual address of Windows kernel structure, which stores information required for debugger (see below)

KdDebuggerDataBlock

KdDebuggerDataBlock contains the addresses of another important kernel data structures and offsets within them, for example:

• KernBase - virtual address of the Windows kernel image loaded into memory (ntoskrnl.exe) which is required for WinDbg to download PDB symbols
• KiProcessorBlock is a pointer to an array of pointers to PRCBs (processor control block) where Windows stores the data on each processor in use processor used
• OffsetPrcbContext - the offset of the structure inside PRCB, where Windows saves register context on crash
• OwnerTag - KdDebuggerDataBlock signature - ASCII characters "KDBG".

The following fields contained in KdDebuggerDataBlock have their analogs in the dump header:

• KiBugcheckData
• MmPfnDatabase
• PsLoadedModuleList

The structure of the KdDDebuggerDataBlock contents is described in _KDDEBUGGER_DATA64 structure in wdbgexts.h, which comes with WinDbg. With a new version of Windows, new fields are only added to end of this structure, and already existing fields remain at the same offsets, this can be relied on regardless of the version of Windows.

It is easy to see that header fields such as RequiredDumpSpace can be calculated and filled based on data from QEMU, but fields like KdDebuggerDataBlock and PsLoadedModuleList are known only to the guest kernel.

Unfortunately, practice shows that KdDebuggerDataBlock in modern versions of Windows may be encrypted by the system at boot time and be inaccessible during operation, and decrypted only during a crash. So we need a way to decrypt it, but let’s come back to this problem a bit later.

KeInitializeCrashDumpHeader

KeInitializeCrashDumpHeader is a partially documented function that driver calls to get the dump header. According to the documentation, the function returns a header that will be correct for the lifetime of the system, although it has the following limitations:

• If the amount of physical memory is changed, the header has to be retrieved again.
• The header received in this way does not contain data about the occurred exception (BugcheckData).

Also, according to the documentation, starting with Windows 8 the DirectoryTableBase address in the resulting header always corresponds to the system context, but for earlier versions the context will be the same as the context of the current process. It can also be user context, which may then prevent the debugger from accessing the system structures using the virtual addresses.

Furthermore, after analyzing of some number of such dumps, some more inconsistencies were found:

• RequiredDumpSpace is not filled
• Context structure is empty
• PfnDatabase field has wrong value

The RequiredDumpSpace value must be filled properly othewise WinDbg will not work properly. The filling of the Context structure does not affect the result, but the similar structures in the memory do. They are required by WinDbg to display register and call stack values. Some header values are duplicated in KdDebuggerDataBlock, so it can be used to restore them. The ways of restoring the header fields are gathered in the corresponding table:

fw_cfg and vmcoreinfo

After the header is received, it must be passed to the host. The vmcoreinfo device is used to do this. The vmcoreinfo device is an add-on to the fw_cfg device.

fw_cfg is a virtual device provided by QEMU that helps guest software to communicate with the host. From the guest point of view, it is a device with I/O ports 0x510-0x51B. It provides access to an array of entries, which are simply blocks of arbitrary data and a string key associated with them.

vmcoreinfo is a virtual device from QEMU accessed via a fw_cfg entry named "etc/vmcoreinfo". The device was originally designed to transfer some Linux kernel data when creating guest Linux dumps.

To read from any fw_cfg entry, the driver does the following:

1. Sends the number of the desired entry to port 0x510
2. Reads the data from port 0x511 byte-by-byte

To write data to vmcoreinfo, we first need to find the entry by its name ("etc/vmcoreinfo") and check that data transfer from the guest to the host is possible:

1. Read 4-byte entry with number 0x19 to get total number of entries
2. Read entries until the desired name is found
3. Read 8 bytes from 0x514-0x51B and check against "QEMU CFG" - this means that fw_cfg supports DMA-like writing interface

In case of VMCoreInfo a packed structure of the following type is passed:

  struct FWCfgVMCoreInfo {
      uint16_t host_format;  /* formats host supports */
      uint16_t guest_format; /* format guest supplies */
      uint32_t size;         /* size of vmcoreinfo region */
      uint64_t paddr;        /* physical address of vmcoreinfo region */
  } QEMU_PACKED;

Data transfer to host is done through a DMA-like interface:

1. Write physical address of FWCfgDmaAccess structure containing the physical address of the data to be written, size, entry number and control bits to 0x514-0x51B
2. Check control bits, if they are all equal to 0, then the data has been successfully transferred

After this, QEMU will have access to the data at paddr address. QEMU interprets them as an ELF Note section, which consists of a name (in this case "VMCOREINFO") and content - the dump header. If the structure of the section is correct, it becomes available to the QEMU dumping subsystem.

All the driver logic described above is implemented here.

Register context

In the debug dump analysis process, the register values are important in and of themselves, moreover, their values are necessary for the correct reconstruction of the call stack.

In the wdm.h header file, which is delivered as part of the WDK, there is a definition of a structure called CONTEXT. After comparing this definition, the register values from QEMU and Context field from the saved Windows dump, it is clear that Context field contains an implementation of this structure. But as on modern systems there is more than one processor, register contexts of all processors can’t fit into dump header (there is only space for one CONTEXT instance).

An address space of Windows kernel contains per-CPU PRCB structures with ContextFrame fields. Each of these fields stores an address of a context frame from a corresponding CPU. If these structures are filled with zeroes, WinDbg cannot recover the context. In addition, the context structure contains a field of flags, one of which indicates that the context is 64-bit, so when the structure is zeroed, the debugger displays a message that only the 32-bit context is available. Once the context structures are filled in, these messages do not occur. So WinDbg definetely takes the contents of registers from these structures.

Thus, in order for WinDbg to retrieve the actual register values, they must be stored in the corresponding structures inside the memory dump. This procedure is implemeted in patch_and_save_context function from dump/win_dump.c.

KiBugcheckData

BugcheckData is automatically saved at BSoD and this data can be simply copied to the header. But it turns out that when creating a live system dump, it is not enough to write BugcheckData to the header. In order for the debugger to use this data, it must also be saved in KdDebuggerDataBlock->KiBugcheckData. This logic is implemented in patch_bugcheck_data routine from dump/win_dump.c.

Small Memory Dump

Along with a full memory dump, there is a Small Memory Dump. For example, Windows creates a dump in this format after a BSoD at default settings. Such dump contains the following data:

• BugcheckData structure
• EPROCESS structure with an information about the faulty process
• ETHREAD structure with an information about the faulty thread
• PRCB structure for the CPU on which the error occured
• List of loaded modules
• KdDebuggerDataBlock structure

Unlike the Complete Memory Dump, KdDebuggerDataBlock is stored in the Small Memory Dump at the offset written in its header, so the address translation is not required. Thus, if we have a small dump, we can use it to create a full memory dump.

KeCapturePersistentThreadState

KeCapturePersistentThreadState is an undocumented Windows kernel function. The driver can use it to retrieve a Small Memory Dump. KdDebuggerDataBlock inside such dump will be decrypted. The guest driver can pass to the hypervisor not only the address of the original KdDDebuggerDataBlock (which can be encrypted), obtained with KeCapturePersistentThreadState, but also the address of its decrypted version, which will be stored in the driver’s memory.

In order for the hypervisor to take advantage of this feature, it must check the signature of the KdDebuggerDataBlock whose address lies in its usual place in the header and, if the signature does not match, use the KdDebuggerDataBlock whose address is passed by the driver through one of the unused header fields, such as BugcheckParameter1, since this field has a null value anyway and must be filled in by the hypervisor. This is what QEMU does in check_kdbg from dump/win_dump.c.

Connection between dump elements at the stage of their loading from the guest memory

Overall host-side algorithm

1. Pause the VM
2. Synchronize state with KVM
3. Take the header from the vmcoreinfo device
4. Calculate RequiredDumpSpace field value as the sum of the sizes of continuous physical memory regions described in the header
5. Use DirectoryTableBase field value from the header as CR3 register value when further accessing the guest OS virtual address space from QEMU
6. Take KdDebuggerDataBlock structure address from the header
7. Substitute PfnDatabase field value with the KdDebuggerDataBlock->MmPfnDatabase value
8. Fill BugcheckData structure (error code and parameters)
   • In the case of the BSoD, take the content from KdDebuggerDataBlock->KiBugcheckData
   • In the case of the live system dump, write 0x161 (LIVE_SYSTEM_DUMP) code and zero error parameters to KdDebuggerDataBlock->KiBugcheckData
9. Write the register context to KdDebuggerDataBlock->KiProcessorBlock[i]->ContextFrame for each i in a range of CPU numbers, based on the registers from QEMU
10. Write header to the file
11. Write regions of the guest physical memory to the file
12. Unpause the VM

Conclusion

We have discussed the usage and internals of dump-guest-memory -w command which is a useful tool for anyone looking to debug their Windows guests in the QEMU/KVM environment. The next post will be devoted to creating a dump with literally no action on the guest side.