Short term
- Optimize cursor movement.
Currently virtual monitor using VNC. But for mouse movement, there is little difference. typically, VNC client will send mouse move message to server, and client will draw cursor. But for virtual monitor, Mouse movement was triggered by server, in current implementation, every Mouse moving, we need sent two frame pictures to client. e.g. move cursor from point A to point B. we need restore background of A, and draw cursor on B. And mouse sampling frequency make it even worse. e.g. you move cursor from A to B, actually, we may get N MOUSE_MOVE event.
The better idea is, when cursor shape was changed, Send new shape to client. when cursor was moved, then send cursor coordinate instead of pixels data, client will draw cursor locally.
Yes current VNC protocol support this feature. but it controlled by client. e.g. if you are using vncviewer, you can specify ViewOnly=1 UseLocalCursor=0. But use may not specify this argument when they are connect to virtual monitor. I want server side can force this behavior.
- Optimize Window movement.
Currently, if you click and hold left mouse button, then drag a window from position A to B. the whole window pixel data will be sent. for a 800×600 with 32bit true color window, 800*600*4 = 1,920,000. almost 2M byte, and don’t including the any protocol header, e.g TCP/UDP header, IP header. although we are using JPEG compress, but reduce the data size is very important. especially for mobile device which only has wifi. a better way show as following picture, which only need send the dotted line over network.
- GUI
- bug fix
Long term
Once basic function works, Then we can focus on driver side, and USB, there are lots of work challenge, also lots of fun.
Windows WDDM Driver
We can’t port virtualbox’s WDDM guest addition driver to host directly, just like what i did for XPDM driver. Based on MSND If multiple graphics adapters are present in a system, all of them must use the same WDDM driver. So WDDM driver is not easy.
1. WDDM Hook
We can do a WDDM Hook, if request for real hardware, then just pass it to under line real driver. otherwise handle by our virtual driver. Anshul Makkar did some research on this approach. But didn’t handle any GPU command. VirtualBox Guest addition WDDM driver is using chromium to handle GPU command. and probably can use WDDM Hook + chromium to implement this.
- Advantage 1: We can reuse lot’s of VirtualBox’s code.
- Advantage 2: Chromium can using real GPU handle 3D command efficiently. If request send to VirtualMonitor, we forward it to Chromium service, Chromium will send request to real GPU.
- Drawback: WDDM Hook is not support by microsoft, it is hard to make sure WDDM Hook Driver stable, and require some reverse researching.
2. Virtual output connector
usually a real GPU has multipule output connector. e.g. for my thinkpad T430, there are following connector avalible:
- LVDS1 connected 1600×900+0+0 (normal left inverted right x axis y axis) 310mm x 174mm
- VGA1 disconnected (normal left inverted right x axis y axis)
- HDMI1 disconnected (normal left inverted right x axis y axis)
- DP1 disconnected (normal left inverted right x axis y axis)
- HDMI2 disconnected (normal left inverted right x axis y axis)
- HDMI3 disconnected (normal left inverted right x axis y axis)
- DP2 disconnected (normal left inverted right x axis y axis)
- DP3 disconnected (normal left inverted right x axis y axis)
if you connect a external monitor to VGA1, then it will generate an interrupt, and GPU’s interrupt handler will be called, and it will read EDID through IIC bus, then driver will get enough information. e.g. the verdon of the connected monitor, and the supported resolution. probally, we can simulate this process. Then GPU driver handle all Command, and We just dump out the frame buffer.
- Advantage: All command handled by real GPU.
- Drawback: No public way.
Linux Driver
1. Re use VirtualBox Guest addition driver, and it support lot’s of Linux version, e.g. Fedora/Ubuntu.
2. Using the ihadzic’s Virtual CRTC approach. his code are here And his proposal are here He implement this for AMD/ATI Radeon GPU. For different GPU, it requires different implementation.
A better way is implement a uniform interface on DRM, DRM query under layer GPU driver if it is support Virtual CRTC feature. if it is, then the driver need regist some callback to drm layer during driver initialization.