Friday 11 May 2018

[Up to First Evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD

Note: This post has gotten rather long. For summary of everything after the first evaluation, check the new post here.


Up to May 10:

I did a lot of stuff before deciding to publish a progress blog. I will try to list as much here as I remember:
  • Set up FreeBSD in a VirtualBox VM.
  • Checkout the FreeBSD HEAD and build from source.
  • Checkout OpenBSD's repo.
  • Read through OpenBSD's Xen implementation and try to make sense of the code.
  • Learn about Grant Tables and Grant References.
  • Read about what DMA is and how the bus_dma(9) interface works.
  • Try to understand how OpenBSD uses the bus_dma(9) interface in its Xen driver.
  • Listen to a talk by Mike Belopuhov, the person who wrote the Xen drivers for OpenBSD, to learn more about the implementation
  • Listen to a talk by Stefan Sperling about developing device drivers for OpenBSD to get a better idea of how drivers work in OpenBSD, and thus getting some idea of how the Xen driver will work in the system.
  • Subscribed to some mailing lists which might come in useful for my project:
    • freebsd-hackers
    • freebsd-virtualization
    • xen-freebsd
    • openbsd-tech
    • xen-devel (xenproject.org)
  • Tried to understand where bus_dma(9) will fit in FreeBSD's implementation (not much success. I'll ask my mentors a few questions about it).
  • Created a wiki page about my project.

May 11:

  • Created this blog and wrote my progress so far.
  • Time to finish setting up my working environment. Talked to my mentors about it. The problem was how to move code from my Ubuntu working machine to my FreeBSD testing machine. Edward suggested sshfs, Akshay suggested scp. I'll try both
  • Currently, my VM is building world. It is taking a long time.
  • In the meantime, I'll learn about ssh and scp and see how I can set them up. Setting up a bridged network from my VM to host looks like a good idea. If this does not work, I'll try doing it over the internet. Not sure how either will work though, I need to wait for the world to finish building.
  • Read about how to use SSH and how SSH works.

May 12:

  • Read up about how to configure the SSH Daemon.
  • After a lot of Googling, finally found out how to connect my host to my VM via SSH. Turns out using a bridged network on my Wi-Fi interface was the answer. I spent a lot of time trying to forward my port 22 to my VM over a NAT network, but that didn't work.
  • Finally able to SSH into my VM.
  • Tried out scp, it works.
  • Got git-scp to work. Now I can transfer changed code from my working machine to my testing VM.
  • git-scp has a weird bug where it stages my changes after sending them over to the remote machine. Need to look into that.

May 14:

  • Changed from password auth to RSA key auth in SSH.
  • Enable the serial console in the FreeBSD VM to get kernel logs. The serial port can be configured in VirtualBox settings to output to a file.
  • Time to install Xen and get a DomU running.
  • Strangely, the Xen package is missing from the FreeBSD package repository. Maybe I can build from source.
  • SVN dropping the connection during checkout. Need to check my internet connection.
  • Couldn't get SVN to work. So used a hack. The packages missing were there in FreeBSD 11's repository. Downloaded them from there and installed. Xen is now installed.
  • The system does not boot. It is stuck in a loop. It goes from boot screen (select whether to boot multi user, single user, etc) and reboots to there again. Maybe my hack was not a good idea.
  • The system went into a boot loop because of a typo in /boot/loader.conf. Fixed it through a Live CD.
  • The kernel panics and does not boot. Xen writes an error message saying Intel processor family 6 processor 60 not supported. It looks like Xen might not boot in a VM.
  • Wrote to freebsd-xen mailing list about the problem. Let's see what they say.

May 15:

  • So it turns out we can't boot Xen with FreeBSD as Dom0 in VirtualBox. Unless I can find another hypervisor that can run this, I will have to boot FreeBSD on my old laptop.
  • Spent a lot of time trying to fix the boot loop in the VM to little success. I am now dropping the idea to use a VM for testing.

May 16-17:

I spent a LOT of time trying to get Xen to boot on my other laptop to no success. These two days have been very frustrating and tiring. I feel like I'm wasting so much time just trying to get Xen to boot. I got Xen to boot on Linux in a few minutes. Getting it to boot on FreeBSD has been a massive pain so far.
  • Time to install FreeBSD on my old laptop. I want to keep my Windows so I'll try to install FreeBSD on a different partition. Then I can restore Windows bootloader when I'm done with the project.
  • Had to drop the idea of keeping Windows. FreeBSD complains about the disk being MBR not GPT and it can not install in GPT.
  • Format the disk and install FreeBSD. Set up the environment, install some useful packages and get my network card working.
  • Time to install Xen.
  • Xen installed. On reboot it says don't know how to load module.
  • Googled around a bit. Turns out I installed FreeBSD in UEFI mode and Xen can't boot in UEFI.
  • Here is where I hit my biggest roadblock. I can't figure out how I can install in Legacy mode.
  • After hours of reading through forums and mailing lists, I finally figure out why I can't install FreeBSD in legacy mode. There in a bug in my laptop's BIOS where it does not detect USBs with GPT in Legacy mode.
  • My working laptop does though, so I use it to install FreeBSD on MBR-formatted USB.
  • The USB boots in my old laptop.
  • At this point a hack comes into my mind. I boot a Live CD from another USB drive and use dd to copy the contents of the USB disk with FreeBSD to the hard disk of the laptop.
  • Works! Now I can boot FreeBSD in Legacy mode on my old laptop.
  • Let's install Xen. Ugh... Doesn't boot.
  • Ask on the freebsd-xen mailing list. Not much help.
  • So finally I decide to try it on my new laptop. Create a USB, insert it and boot FreeBSD.
  • Installed Xen, it goes further than last time but then gets stuck on an empty screen. Ask on the freebsd-xen mailing list.

May 18:

  • Akshay's reply puts me on the right track. After a bit of Googling, I fix that blank screen.
  • Ugh... STILL DOES NOT BOOT. I'm getting really frustrated now.
  • Fiddled with the Xen command line, no change.
  • Wrote to the freebsd-xen mailing list about it.
  • After a bit more research, I find that my old laptop's processor does support VT-d but there is no option to enable it in the BIOS; just an option for VT-x. So no way I'm using the old one for testing. My new laptop though, does support VT-d. I will use it for testing. But right now, I can't get it to boot. No response from the mailing list yet.
  • There is a weekend coming, so I might not get a response till Monday. I'll spend the Saturday reading through some grant table code. I feel like I spent too much time on trying to set up my testing environment. Need to compensate for that.

May 19:

  • Akshay suggested I should try booting from my hard disk drive (HDD) rather than my USB.
  • Started looking in using GRUB with a mix of BIOS and UEFI OSes because I don't want to lose my Windows and Ubuntu installations. Not much success, but I did find one StackExchange answer that answers exactly my problem. I need to create a partition and label it with bios_grub.
  • Created the partition and installed FreeBSD. Does not boot.
  • Hm, probably something wrong with my GRUB config. Open a Linux Live CD to install grub. Can't install because GRUB does not recognize FreeBSD. Also, the FreeBSD partitions do not mount, and I need to save the config in /boot/grub/.
  • Oh! I never realized GRUB also had a package for FreeBSD. Time to try it.
  • Chrooted into my installed system using a FreeBSD Live CD. pkg needs to be bootstrapped, and it fails.
  • A hack comes in my mind. I modify the /etc/fstab of my Live CD to point root to an invalid location. The root mount at boot will ask me to specify the device of the root filesystem, and I can give my HDD location there. So basically I use the Live CD for bootstrapping and then mount my HDD system.
  • In the HDD, install grub. Works. Then used grub-mkconfig to create the config file. Reboot. Does not boot.
  • Again used the hack to get in my HDD installation. This time, I hand-write my grub.cfg that I found on the FreeBSD forums. Reboot. We get into grub!
  • But FreeBSD panics on boot. Googled some more config files for FreeBSD and noticed that some people are specifying kfreebsd /boot/loader, but my config is specifying /boot/kernel/kernel. Fix it.
  • Works! I can now boot FreeBSD from my HDD.
  • Installed Xen and guess what, it doesn't boot. Same panic message as when booting from USB. All I did to get GRUB to work was an exercise in futility.
  • Roger replied. He suggested I try dom0_max_vcpus=1. Also asked me to give some info for debugging.
  • FINALLY XEN BOOTS! YAY!

May 21-22:

I don't have much to report for these two days. It has been a lot of reading, and there isn't much material to write here.
  • I still have not identified where the bus_dma(9) wrappers should go in FreeBSD's Xen implementation. I am much more familiar with the OpenBSD implementation than I am with the FreeBSD one. I need to ask my mentors about that, but I feel I should do a bit more homework before asking. I'll draft an email by tomorrow evening.
  • Listened to Mike Belopuhov's talk on OpenBSD's Xen implementation again. I understand it much better now than I did before.
  • He mentions the Xen Netfront drivers need to use the grant tables, and he explains how the bus_dma(9) can be used for that. Started reading through FreeBSD's Netfront driver. It's huge.
  • Started reading OpenBSD's Netfront driver instead. It seems much simpler, in part thanks to Mike's talk.
  • Every time I try to understand FreeBSD's implementation I just feel lost. Need to fix that. I do have a little better understanding of the OpenBSD implementation, mostly because it's simpler and I have spent more time understanding it.
  • At this point, I feel like I understand how grant tables work, and how they use bus_dma(9), but I do not have a grasp of the bigger picture; about how all these systems combine together. Will ask my mentors/Roger about this.

May 23:

  • More reading. Read the OpenBSD Xen Netfront driver (sys/dev/pv/if_xnf.c) to understand how it uses the bus_dma(9) wrappers. It calls bus_dmamap_load() and then uses the ds_addr from the dma_seg_t which contains the grant reference of the page. That grant reference was loaded by the custom wrapper method xen_bus_dmamap_load()
  • Looked at the FreeBSD man page of bus_dma(9) interface. It is quite different from OpenBSD's interface. I did not expect them to differ much. It seems like implementing the wrappers is more work than I thought.
  • Reading the man page, I do think how I might achieve the same effect as OpenBSD. OpenBSD's interface seems more intuitive.
  • Rather than creating a custom bus_dma_tag I need to use the callback functions to call the xen-specific methods and update the grant table from there.

May 24-25:


  • Decisions to make. Emailed Edward and Akshay about the potential design choice we have to make. Haven't heard from them yet.
  • Dug deeper into FreeBSD's bus_dma(9) interface. I realize there are more things to take care of than I thought. The different interface OpenBSD and FreeBSD's bus_dma API exposes is posing some problems.
  • Wrote the xen_bus_dmamap_load(). More problems present themselves. It turns out I have to use the callback method. This creates the problem of returning error codes in case allocating a grant reference fails.
  • Maybe I can allocate the grant reference when we create the bus_dma_tag with frame number 0 and then simply update the frame number in the callback. This would mean we won't have to deal with returning error codes from the callback method.
  • Pushed a commit of a rough implementation of xen_bus_dmamap_load(). Check it here.
  • Unrelated: I noticed there was some code in gnttab.h that was wrapped in a #if 0. Asked Roger about it, he says it is left over from when it was imported from Linux. Submitted a patch to delete it. Roger reviewed and accepted it. Check it here.
  • Pushed a commit of a rough implementation of xen_bus_dma_tag_create(). Check it here.
  • Pushed another commit updating xen_bus_dmamap_load(). Check it here.

May 26:

Today's commits:
  • Still haven't heard from my mentors. In the meantime, a rough implementation of all the main functions is done. I still have not done bus_dmamap_create/destroy() and bus_dmamap_load_mbuf. The dma map creation/destruction methods are pretty straight forward. I'll probably call the main bus dma methods and return straight away. No grant table related stuff needs to be done there.
  • The mbuf variant of bus_dmamap_load has to be done, but this should be pretty similar to the non-mbuf variant xen_bus_dmamap_load.
  • This leaves us the review of design choices and code review. I'll wait to hear back from my mentors. There isn't much I can do right now.

May 28-29:

Commits:
  • Asked my mentors for review of my implementation. Haven't heard back anything yet. I have not received replies to my emails for a few days now. Maybe they are busy at work. There isn't much I can do right now until I get some reviews.
  • Fixed some minor problems.
  • Fixed the issue of not being able to call the client driver's callback function. Used a struct containing the function pointer, client's callback argument and xen's callback argument. Pass this to the xen-specific callback we give to bus_dmamap_load() and call the callback from there.
  • Compiled the code to catch syntax errors. Fixed the errors found.

May 30:

Commits:

May 31 - June 2:

  • I was on holiday these 3 days.

June 4-5:

Commits:
Summary:
  • Asked Roger Pau Monne to review my implementation.
  • He pointed out a bunch of code style problems. Fixed them.
  • He also pointed out a potential security flaw where every domain would have access to physical frame 0. Will fix it in a follow-up commit.
  • So it turns out I misunderstood what exactly I had to do. I built the Xen-specific bus_dma(9) handlers on top of the existing bus_dma(9) interface. Instead, I was supposed to build another implementation of the bus_dma(9) interface specific to Xen, kind of like how busdma_dmar is implemented.
  • I'll have to rewrite some code, and add some other handlers to create the alternative bus_dma(9) implementation. I will start on it tomorrow.

June 6:

Commits:
  • Need to refactor the functions I wrote to conform to the bus_dma implementation headers exactly.
  • Introduced the bus_dma_tag_xen. It will contain the information needed for the xen-specific dma 
  • Added more fields to xen_callback_arg. These will be used for grant allocation in the callback.
  • Converted map load, map unload, tag create, tag destroy.

June 7-8:

Commits:


Summary:
  • A lot of reading. Looked at busdma_dmar.c and busdma_machdep.c and busdma_bounce.c to see how other variations of the bus_dma(9) interface are implemented.
  • I'm confused. There are so many things to factor in.
  • Let's take it step by step. Wrote tag creatiion and destruction. Now time to start with map creation and destruction. Looks simple enough. The different types of load (load_phys, load_buffer, load_ma) are the problem. I need to figure out how to handle each.
  • Finished writing the new implementation. Need to compile it and go through it again before asking for a review from Roger.

June 11:

Summary:
  • Code review with Roger Pau Monne. He did point out some small mistakes, but nothing major.
  • Came up with a plan of what to do when a grant allocation fails. Earlier, the errors were silent. Roger suggests we handle the errors during map_load_*(), and not during map_complete(). Sounds like a good idea to me.
  • Discussed the issue of the parent tag allocation. I was creating a tag descended from the parent tag, which was then saved and later used for the dma operations. Roger says that is not necessary. The parent can not be destroyed as long as there is a child for it, so no need to create another tag. Just save the parent directly.
  • First GSoC evaluations open today. It will be 9:30 PM in my time zone (IST) so I'll probably fill them tomorrow.

June 12:

Commits:
  • Filled the GSoC First Evaluation.
  • Made the changes that Roger suggested.
  • The grant references are now allocated in tag create, so that there can be no failures in map_complete. Refactored the code accordingly.
  • Defined a new malloc type M_BUSDMA_XEN and used it to allocate the refs array, instead of using M_DEVBUF.
  • Earlier, map_complete did not check for the error code and made the allocation anyway. Now, it checks the code and does not map the references if there has been an error.

June 13:

Summary:
  • Asked Roger for this thoughts about yesterday's changes. He pointed out one problem. I am allocating the grant references in tag create. But what I didn't know was that a single tag can be used to create multiple maps. So the grant references should be allocated for each map, not each tag.
  • This means a major refactor. I was not using a Xen-specific dma map before. But now I need it because it has to hold the grant references array and other associated data.
  • All the hooks that use a dma map need to be refactored.
  • Done with most of the changes. Will finish up tomorrow.

June 14:

Commits:
  • Looking at busdma_bounce.c, I see that a map does not necessarily use all the segments. Dive deeper into the implementation to see if there is any way to find out how many segments it does use, so we can allocate as few grant references as we can.
  • Further investigation leads me to suspect there is a bug in the busdma interface. Wrote a long email to freebsd-hackers@freebsd.org about it. Check it out here.
  • So I did find out a way to get the number of segments a map uses. It looks kind of like a hack to me. Not sure whether I should use it. I'll ask Roger.
  • Finished up the changes.

Note: This post has gotten rather long. For summary of everything after the first evaluation, check the new post here.

at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)

[After second evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD

[After second evaluation] Import the Xen grant-table bus_dma(9) handlers from OpenBSD The links to the commits are prob...