Audio and Video Communications with QuantumGate
Back in March I mentioned that I was working on an example extender for QuantumGate that would demonstrate how to do video and audio communications, comparable to video and audio calls via WhatsApp and Skype. One of the reasons for that was to provide yet another example to show interested developers how to use and build on top of QuantumGate. And another very important reason was so that I could have an extender in my test suite that I could use to test QuantumGate’s performance with real-time communications.
I’m happy to report that I’ve made some nice progress with the extender (which I’ve called ‘AVExtender’) and right now it could even be used to do video and audio calls over the Internet. It’s still very basic and unoptimized however, and I’ve integrated it into the main TestApp that I use to test most of QuantumGate’s functionality. By the way, if you want more information about what QuantumGate is, then click here for an introductory post.
Testing QuantumGate and the AVExtender with two calls to myself on the same and another PC on the network
Based on the work and the tests that I’ve done so far I’ve identified a few areas where I can improve QuantumGate. In fact I’ve already added one important feature to QuantumGate based on the tests. And that is that extenders, when sending messages, can now specify an additional parameter for each message indicating what priority should be used for that message. This gives extenders that rely on real-time communications the ability to have priority processing of the messages that they send. So if there’s a file transfer in progress in the same or another extender, then messages containing audio and video communications data can be sent with higher priority to make sure there’s as little latency as possible. Latency is especially noticeable and problematic when it comes to transmission of audio.
I’ve also added various configuration options in the AVExtender that can be changed in real-time to perform various tests. For example, I can independently enable or disable compression of audio and video data and increase or decrease video resolution to see how much stress QuantumGate is able to handle. Sending uncompressed audio and video data really stresses QuantumGate’s internal data processing logic and any inefficiencies and bottlenecks become so much more noticeable. In this case we’re talking, for example, sending multiple uncompressed video frames of more than 1MB per second. You can then start to hear gaps between the audio frames and it becomes a challenge to minimize this as much as possible and optimize the overall performance. I plan to keep pushing it to get as much real-time throughput as possible with very large quantities of data. That way I’ll be sure QuantumGate will be able to handle the normal workloads without issues.
When it comes to the audio and video compression used by AVExtender, I’ve chosen to use the built-in Media Foundation codecs that come with Windows. Specifically the H.264 and ACC codecs for video and audio respectively. I thought about using the VP8/9 codec for video and Opus for audio, but they don’t come with Windows, and I would have to use other libraries which means adding two additional dependencies to the QuantumGate project. This would make it a bit more tedious for people who want to compile and try out QuantumGate from the source code on GitHub. They would first have to go download and build two additional dependencies, apart from the ones QuantumGate already requires, before doing that. And since this is a test extender I don’t care as much about using the best compression quality available, which are provided by VP8/9 and Opus. The only problem with my choice is that the H.264 codec from Microsoft that comes with Windows is not suitable for real-time communications because it initially buffers a lot of input before starting to provide the compressed output, and this results in about a 3 second delay in the video stream. So the video and the audio aren’t synchronized when using AVExtender. But again, since this is just a test extender it’s not that big of a problem. I just don’t want people to think that the video latency is caused by QuantumGate itself.
It was also interesting to see how the AVExtender interacted with other functionalities in QuantumGate such as the available Security Levels. At the higher security settings QuantumGate adds extra random data to messages (‘padding’) and sends noise messages to confuse eavesdroppers. This requires additional processing capacity and bandwidth depending on the settings used (which are customizable by the user), and real-time communications using the AVExtender was flawless under normal conditions. Below you can see a graph of the bandwidth being used during one test call that I made with transmission of video and audio. On the right side a higher security level was used and you can clearly see that more bandwidth was being used by QuantumGate where it sent additional random data.
At the higher security levels the encryption keys being used are also updated more frequently and this is one of the cool features of QuantumGate. While this real-time communication is happening, QuantumGate essentially does a full handshake in the background during regular intervals, updating the encryption keys to guarantee forward secrecy. And it happens seamlessly without interrupting other communications. New encryption keys are used as they become available. You can actually see one such case in the above screenshot if you look at the last two yellow lines in the terminal.
There’s lots more work to do and I haven’t even tested the AVExtender over relayed connections yet. I’ve worked on the AVExtender mostly in my spare time and there are large periods of time in between when I need to focus on other projects. But it keeps me excited thinking about the improvements I can make to QuantumGate in the near future. So excited in fact, that I think about it in my sleep, which means difficulty in getting a good rest as you can no doubt see from the dark areas surrounding my eyes in the above screenshot.
I think I’ll be pushing these updates to the master branch on GitHub by January next year when I’ve done most of the high priority work on QuantumGate on my to-do list. Then I’ll also be updating the documentation and adding AVExtender to the other examples on GitHub. Right now the above mentioned updates are already available in the ‘dev’ (development) branch that gets updated often, sometimes daily. That branch is not always in sync with the QuantumGate documentation on GitHub, but you’re welcome to try it out if you want to experiment.
Comments
There are 0 responses. Follow any responses to this post through its comments RSS feed. You can leave a response, or trackback from your own site.