Bridging Asterisk RTP streams with OVS

I’m at the AstriCon conference this week, which is a conference built around the Asterisk open source project.  I worked on the Asterisk project for about 7 years before joining Red Hat to hack on cloud infrastructure.  I also helped write a book about it.  While I’m not working on Asterisk directly anymore, I still find it a very interesting project.  The community is full of great people.  Another reason I still pay attention is that communications infrastructure in general is an incredibly important use case for cloud infrastructure. The telco world is going through a rapid transformation with SDN and NFV.

I did a keynote at AstriCon last year about open cloud infrastructure and its importance to Asterisk and communications infrastructure more broadly.  This year I did a talk more focused on networking and how some of the SDN trends apply to this project.  One of the things this conference has started doing is have a session called “dangerous demos”.  The idea is for people to come up on stage and attempt a short (3-5 minute) live demo.  They give awards for various categories, including the most amusing case of a demo crashing and burning, as is often the case with live demos, especially using conference wifi.  Sounds fun, doesn’t it?  I thought so.

Last Friday I set off to see what kind of demo I could whip up in an afternoon.  Here’s what I came up with.

Asterisk Call Bridging

Before getting to the demo, it’s important to have some background on how Asterisk and some related technologies work.  Asterisk supports many different communications technologies.  It supports many different methods of traditional telephone network (PSTN) connectivity.  It also supports several Voice over IP (VoIP) protocols.  Any connection to the system via any of these technologies is represented as an Asterisk channel.

[A Single Call Leg, Represented by a Single Channel]

In some cases, there is only one channel.  This is when Asterisk itself is the endpoint of the call.  Some traditional examples would be something like voicemail or a system that implements an IVR such as an automated system to make payments on account.

It’s also common to have two channels bridged together.  Imagine two phones on a call talking to each other.

[Two Call Legs Represented by Two Channels]

Architecturally, there are some layers involved here.  There is channel technology abstraction so that two channels using different technologies can still be bridged together.

[Channel Technology and Abstract Channel Layers]

This is an incredibly powerful part of Asterisk’s architecture.  It lets you bridge new technologies like WebRTC to traditional telephony protocols.  However, bridging media streams through the abstract channel layer is not the most efficient way to do it if the two channels bridged together are actually the same technology.  So, Asterisk also has a concept of “native bridging”.  This lets channel technology implementations implement more efficient ways of bridging.

SIP is the most commonly used VoIP protocol.  SIP is actually just a signaling (control) protocol.  The actual media streams are independent streams using the RTP protocol.  In some cases, the media streams can be sent directly between endpoints, but not always.  Asterisk sometimes has to transcode the media streams between two different codecs.  Another common case is that the streams may be fully compatible, but the system is used to put all streams through a controlled point (or set of points) at the edge of a company’s network. This use case is sometimes referred to as a Session Border Controller (SBC).

An RTP stream is a good example of a painful scenario for packet processing performance.  It’s a stream of small packets.  A typical RTP stream would be 50 UDP packets per second in each direction.  Each packet would hold 20 milliseconds of audio.  This can be different.  You can increase packet sizes, but it comes at the cost of increasing latency into the call. 20 ms of audio using G.711 is 160 bytes of audio payload. There are several other codecs that may increase or decrease the audio payload. For example, 20 ms using G.729 would be only 20 bytes of audio payload. Every packet also includes ethernet, IP, UDP, and RTP headers.

When two of these RTP streams are bridged in Asterisk, there is a thread handling the call that’s polling on two UDP sockets.  When a packet comes in on one socket, it’s processed if necessary and then written out to the other socket.

You can find a somewhat dated chapter that I wrote several years ago about Asterisk in the book “Architecture of Open Source Applications”. I re-used some of the diagrams from that chapter for this post.

The Demo

This demo is targeted at the case of Asterisk bridging two RTP streams that are fully compatible (same codec, same payload sizes, among other things).  During my talk about “SDN and Asterisk” yesterday, I talked about several things. One thing I talked about is how the Linux networking datapath is becoming more programmable and I talked about Open vSwitch (OVS) as a specific example of that.

My demo consists of two VMs on my laptop (asterisk1 and asterisk2).  They both have a single vCPU and 1 GB of RAM.

asterisk1 serves as both endpoints of calls passing through asterisk2, so asterisk2 is doing bridging of compatible RTP streams.  Both ends of the call on asterisk1 are executing the Milliwatt() application, which just generates a tone. Each call looks like this:


I also customized the networking configuration on asterisk2. Instead of just having eth0, I have an OVS bridge named breth0 and eth0 is attached to that bridge.

[rbryant@asterisk2 ~]$ sudo ovs-vsctl show
    Bridge "breth0"
        fail_mode: standalone
        Port "eth0"
            Interface "eth0"
        Port "breth0"
            Interface "breth0"
                type: internal
    ovs_version: "2.4.0"

[rbryant@asterisk2 ~]$ ip addr list breth0
4: breth0@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether 52:54:00:31:cf:ce brd ff:ff:ff:ff:ff:ff
    inet brd scope global dynamic breth0
       valid_lft 2258sec preferred_lft 2258sec
    inet6 fe80::5054:ff:fe31:cfce/64 scope link 
       valid_lft forever preferred_lft forever

With this setup in place, I generated 100 calls, which means both asterisk1 and asterisk2 have 200 active channels.  On asterisk1:

asterisk1*CLI> core show channels
Channel              Location             State   Application(Data)             
SIP/asterisk2-000000 555@public:2         Up      Milliwatt()                   
SIP/asterisk2-000000 555@public:2         Up      Milliwatt()                   
SIP/asterisk2-000000 555@public:2         Up      Milliwatt()                   
200 active channels
200 active calls
200 calls processed

and on asterisk2:

asterisk2*CLI> core show channels
Channel              Location             State   Application(Data)             
SIP/asterisk1-000000 (None)               Up      AppDial((Outgoing Line))      
SIP/asterisk1-000000 555@public:1         Up      Dial(SIP/555@asterisk1)       
SIP/asterisk1-000000 (None)               Up      AppDial((Outgoing Line))   
SIP/asterisk1-000000 555@public:1         Up      Dial(SIP/555@asterisk1)          
200 active channels
100 active calls
100 calls processed

Why 200 channels? It’s a nice round number. It also generates enough load on asterisk2 for the demo without making my laptop melt.

I mentioned earlier that in this case, Asterisk does the bridging of two RTP streams in a thread that’s polling on two UDP sockets, reading packets from one, doing any necessary processing, and then writing it back out to the other socket. In this scenario, Asterisk is using roughly 25% of the vCPU on asterisk2.

What if in the simple forwarding case we could push this forwarding down into the kernel?

To pull this off, first I needed to know about all of the RTP streams active on asterisk2. I actually need to know about pairs of RTP streams. When a packet arrives on one stream, I need to know what other stream it’s associated with for sending it back out. Asterisk honestly does not make it very easy to get this information. You can get it using the CHANNEL() function. I probably could have written an AMI script to get the info I needed. I’m not sure if I could have done it with ARI.  All of that sounded like too much work for my Friday afternoon hack.  The easiest way for me was to write a custom Asterisk C module that provided a CLI command to dump all of the info I wanted.  Here’s the relevant code minus all of the module and CLI command boilerplate code:

	struct ast_channel *chan;
	struct ast_channel_iterator *chan_iter;
	chan_iter = ast_channel_iterator_all_new();
	for (; (chan = ast_channel_iterator_next(chan_iter)); ast_channel_unref(chan)) {
		char src[1024] = "";
		char dest[1024] = "";
		char src2[1024] = "";
		char dest2[1024] = "";
		struct ast_channel *chan2;
		ast_func_read(chan, "CHANNEL(rtpsource)", src, sizeof(src));
		ast_func_read(chan, "CHANNEL(rtpdest)", dest, sizeof(dest));
		chan2 = ast_bridged_channel(chan);
		ast_func_read(chan2, "CHANNEL(rtpsource)", src2, sizeof(src2));
		ast_func_read(chan2, "CHANNEL(rtpdest)", dest2, sizeof(dest2));
		ast_cli(a->fd, "%s %s %s %s\n", src, dest, src2, dest2);

This code is a terrible hack that you’d never use on anything but this controlled environment, but it got me the info I wanted quickly.  The output looks something like this:

asterisk2*CLI> rtpstreams

Now that we have the info we need about RTP stream pairs, we want to program the OVS bridge to do the RTP forwarding for us. We do that using the OpenFlow protocol. In this case, we’ll use the ovs-ofctl command line utility to create and delete flows as needed.

I don’t intend to go into any great detail about OpenFlow or how OVS works, but I think a really high level overview of flows is needed to be able to understand what happens next. OpenFlow lets you define a multi-stage packet processing pipeline. Each stage is a table. Processing starts in table 0. Processing may continue in other tables based on what actions are executed. Each flow in a table has a priority. The flow that gets executed in a table is the one with the highest priority that matches the packet. If multiple flows at the same priority match, which one gets executed is undefined.

What we want are flows that match an incoming RTP stream. In this demo we create flows with the following match conditions: the packet arrived on eth0, it’s a UDP packet, and the UDP destination port number is N. When a packet matches one of our flows, we execute these actions: change the source and destination MAC addresses, change the source and destination IP addresses, change the source and destination UDP port numbers, and send the packet back out where it came from (eth0).

An example command to install a flow like this would be:

sudo ovs-ofctl -O OpenFlow13 add-flow breth0 priority=100,in_port=1,udp,udp_dst=10758,actions=mod_dl_src:52:54:00:31:cf:ce,mod_dl_dst:52:54:00:88:75:61,mod_nw_src:,mod_nw_dst:,mod_tp_src:14508,mod_tp_dst:10060,in_port

Of course, typing up 200 of those would be pretty tiring, so I just scripted it. Here is a simple Python script to generate all of the flows we need:

#!/usr/bin/env python

import os
import subprocess

asterisk1_mac = '52:54:00:88:75:61'
asterisk2_mac = '52:54:00:31:cf:ce'
asterisk1_ip = ''
asterisk2_ip = ''

output = subprocess.check_output(['sudo', 'asterisk', '-rx', 'rtpstreams'])
pairs = []
for l in output.splitlines():
    parts = l.split()
    if parts[0] == 'Setting':
        pair = ((parts[0].split(':')[1], parts[1].split(':')[1]),
                (parts[2].split(':')[1], parts[3].split(':')[1]))
        print "Failed to parse parts: %s" % parts
    reverse_pair = (pair[1], pair[0])
    if reverse_pair not in pairs:

for p in pairs:
    os.system('sudo ovs-ofctl -O OpenFlow13 add-flow breth0 '
            % (p[0][0],
               asterisk2_mac, asterisk1_mac,
               asterisk2_ip, asterisk1_ip,
               p[1][0], p[1][1]))
    os.system('sudo ovs-ofctl -O OpenFlow13 add-flow breth0 '
            % (p[1][0],
               asterisk2_mac, asterisk1_mac,
               asterisk2_ip, asterisk1_ip,
               p[0][0], p[0][1]))

After running the above script, we can view the flows on breth0 using the following command:

[rbryant@asterisk2 ~]$ sudo ovs-ofctl -O OpenFlow13 dump-flows breth0 | grep table | cut -f4- -d' '
table=0, n_packets=591, n_bytes=126474, priority=100,udp,in_port=1,tp_dst=12164 actions=set_field:52:54:00:31:cf:ce->eth_src,set_field:52:54:00:88:75:61->eth_dst,set_field:>ip_src,set_field:>ip_dst,set_field:18364->udp_src,set_field:19818->udp_dst,IN_PORT
table=0, n_packets=588, n_bytes=125832, priority=100,udp,in_port=1,tp_dst=18364 actions=set_field:52:54:00:31:cf:ce->eth_src,set_field:52:54:00:88:75:61->eth_dst,set_field:>ip_src,set_field:>ip_dst,set_field:12164->udp_src,set_field:10322->udp_dst,IN_PORT
table=0, n_packets=588, n_bytes=125832, priority=100,udp,in_port=1,tp_dst=10364 actions=set_field:52:54:00:31:cf:ce->eth_src,set_field:52:54:00:88:75:61->eth_dst,set_field:>ip_src,set_field:>ip_dst,set_field:10110->udp_src,set_field:17640->udp_dst,IN_PORT

We can see in the n_packets field of each flow that packets are matching all of our flows for forwarding RTP streams.

Here’s what’s really cool about this. After these flows are configured, Asterisk takes up less than 1% of the vCPU and the vCPU is 96-97% idle.

If we want to clear all of these flows and let RTP go back through Asterisk in userspace, we can run this script:


for n in $(sudo ovs-ofctl -O OpenFlow13 dump-flows breth0 | grep "priority=100" | cut -f7 -d' ') ; do
    sudo ovs-ofctl -O OpenFlow13 del-flows --strict breth0 $n

At this point, the CPU usage jumps back up to where it was before.

Future Work

This was just the result of an afternoon hack.  My primary goal was just to spur some interest in exploring how cool things happening in the SDN space could provide new ways of doing things.

If someone wanted to explore doing this in Asterisk more seriously, you could write some code in Asterisk that could speak OpenFlow to the local OVS bridge to create and delete flows as needed.  You could also imagine the possibility of speaking OpenFlow to a top-of-rack switch to push the forwarding out of the host completely, yet still through a controlled point in your network.

Another major caveat in this demo is that OVS and OpenFlow don’t know what RTP is. There’s no way (that I know of) to do any sort of validation on the packets before forwarding them along.  If one end started sending garbage, this setup would happily forward it along.  It’s up to you how much that matters.  RTP devices are supposed to be built for the possibility of media streaming directly between endpoints, and in that case, there’s nothing in the middle doing any checking of things.

If you were at AstriCon, thank you for coming to my talk and/or demo.  To everyone, I hope you found this interesting and that it inspires you to go off and learn more about this cool technology!

Taking On New Challenges

I began working on the Asterisk project in 2004.  My work on Asterisk has led to an exciting career in open source software engineering.  At the end of July 2011, I will be leaving Digium to take on some new challenges.  Specifically, I will be joining the Cloud Infrastructure team at Red Hat as a Principal Software Engineer where I will be working on projects related to clustering, high availability, and systems management.  Additionally, I will be moving back to Charleston, SC to be closer to my family.

While I will no longer be working with Asterisk full time, I still plan to participate in the open source community.  I am excited to watch both Asterisk and Asterisk SCF continue to evolve and grow.  The engineering team at Digium, as well as the global Asterisk development community are as strong as they have ever been and will continue to accomplish big things.

I have met many great people from all over the world in my time with Asterisk.  Thank you all for making the past seven years so memorable.

Best Regards,

Russell Bryant


Related Posts:

Debugging the Asterisk Dialplan with Verbose()

Leif Madsen and I are working on a new book, the Asterisk Cookbook. One of the recipes that I am working on this morning is a method of adding debug statements into the Asterisk dialplan. I came up with a GoSub() routine that can log messages based on log level settings that are global, per-device, or per-channel. Here’s a preview. I hope you find it useful!

Channel logging GoSub() routine.

  • ARG1 – Log level.
  • ARG2 – The log message.

Channel logging using this routine will be sent to the Asterisk console at verbose level 0, meaning that they will show up when you want them to regardless of the current “core set verbose” setting. This routine uses a different method, values in AstDB, to control what messages show up.

AstDB entries:

  • Family: ChanLog/ Key: all
    • If the log level is less than or equal to this value the message will be printed.
  • Family: ChanLog/ Key: channels/
    • This routine will also check for a channel specific debug setting. It will actually check for both the full channel name as well as just the part of the channel name before ‘-‘. This allows setting a debug level for all calls from a particular device. For example, a SIP channel may be “SIP/myphone-0011223344”. This routine will check:
      • Family: ChanLog/ Key: channels/SIP/myphone
      • Family: ChanLog/ Key: channels/SIP/myphone-0011223344

Example Dialplan Usage:

exten => 7201,1,GoSub(chanlog,s,1(1,${CHANNEL} has called ${EXTEN}))

Example of enabling debugging for a device from the Asterisk CLI:

*CLI> database put ChanLog SIP/myphone 3 

chanlog routine implementation:


exten => s,1,GotoIf($[${DB_EXISTS(ChanLog/all)} = 0]?checkchan1)
    same => n,GotoIf($[${ARG1}  n(checkchan1),Set(KEY=ChanLog/channel/${CHANNEL})
    same => n,GotoIf($[${DB_EXISTS(${KEY})} = 0]?checkchan2)
    same => n,GotoIf($[${ARG1}  n(checkchan2),Set(KEY=ChanLog/channel/${CUT(CHANNEL,-,1)})
    same => n,GotoIf($[${DB_EXISTS(${KEY})} = 0]?return)
    same => n,GotoIf($[${ARG1}  n(return),Return() ; Return without logging
    same => n(log),Verbose(0,${ARG2})
    same => n,Return()

Asterisk 1.10 Update

I just posted an update on the development of Asterisk 1.10 to the asterisk-dev mailing list. Here is the content:


Shortly after the release of Asterisk 1.8, we had a developer meeting
and discussed some of the projects that people would like to see in
Asterisk 1.10 [1]. We discussed the schedule there a bit, as well. Now
that Asterisk 1.8 has settled down and we are well into the development
cycle for Asterisk 1.10, it is a good time to revisit the plans for the
next release.

At Digium, the biggest thing we have been working on for 1.10 so far is
replacing the media infrastructure in Asterisk. Most of the critical
and invasive plumbing work is done and has been merged into trunk. Next
we’re looking at building up some features on top of that, such as
adding more codecs, enhancing ConfBridge() to support additional
sampling rates (HD conferencing), adding features that exist in
MeetMe() but not ConfBridge(), and enhancing codec negotiation.

Of course, many others have been working on new developments as well. I
would encourage you to respond if you’d like to provide an update on
some new things that you’re working on.

We would like to release Asterisk 1.10 roughly a year after Asterisk
1.8. This will be a standard release, not LTS [2]. To have the release
out in the October time frame, we need to branch off 1.10 (feature
freeze) at the end of June. At that point we will begin the beta and RC
process. If you’re working on new development projects that you would
like to get into Asterisk 1.10, please keep this timeline in mind.

As always, comments and questions are welcome.



Russell Bryant

Asterisk: The Definitive Guide – Call for Technical Review

We are finally wrapping up our book, “Asterisk: The Definitive Guide”. The contents have been available on a web site for a while, but now that the content is complete, we’re looking for a lot of review over the next couple of weeks. We would really appreciate your input!  Here is a post from Leif on the asterisk-doc mailing list from earlier today:

Hey all!

We’re getting VERY close to having the first draft of the next Asterisk book, Asterisk: The Definitive Guide ready to be sent off to production. We’re very close to meeting our target dates, but our review timeline is very tight. Only about 2 weeks!

Each morning we’re continuing to work on the book, taking in your comments, reviewing chapters, testing dialplan and installation steps, and all that good stuff.

However, we’ve been looking at this book since May 2010 and our eyes are starting to get glazed 🙂 We’d love for the community to have a look at the book and offer some constructive criticism.

It’s far too late to take requests for things to cover. What we have is what we’re going to get in for this edition. After we finish this book though we plan on continuing to update it, so there will be a chance to take suggestions again soon.

For now, head on over to and check out the book (updated this morning). There are a couple of bugs in the OFPS software which are causing comments to not be available after chapter 8, but we’re hoping to have those resolved by Friday. However, we do have this fancy mailing list that we can use.

Update: This issue has been resolved

Russell, Jim and myself will be monitoring this list for comments, and we’ll try and get all of them satisfied before publication. If there is a particular area we’re covering that you’re an expert in, we’d love to have you focus on that chapter. You can email me back directly for more information on what we might be looking for in that type of situation.

We do have editors to help with grammar and spelling, but pointing anything out is certainly useful. The best use of your time though is testing the dialplan snippets, the installation instructions for both Ubuntu and CentOS (we’re covering two Linux distributions this time around, which increases the testing load significantly), and making sure anything we’re explaining is concise, relates to what we’re talking about, and makes sense. The goal is to build an Asterisk system from scratch, so following through our dialplan via the chapters to make sure it all continues to build on itself would be ideal.

Additionally, if you see any sections which say, “see chapter XXX for more information” that are not links, please let us know, as those are meant to be placeholders until the chapters existed and we could link back to them. Now that all chapters are created, we should be linking to the appropriate locations. If you’re reading a section and notice a good spot to reference another part of the book (for example, lets say we’re talking about database functionality in one of the other chapters, and there is an appropriate spot in the Database Integration chapter to link to), then let us know!

Thanks for your interest! Books should be shipping sometime between March and April. Pre-orders are available now at And yes, we’ll be releasing under a Creative Commons license like the last two books, so you’ll have access to the book at any time online.

This book has been pretty much written from the ground up, and is well over 600 pages of content. It’s been a lot of work, but we hope you like it!

Russell, Jim and Leif.

Open Source Telephony @ FOSDEM 2011

FOSDEM (Free and Open Source software Developers European Meeting) is an amazing conference held each year in Brussels, Belgium. I have been lucky enough to attend in 2009 and 2010. Both times I was very impressed with the number of attendees and the quality of the talks.

For FOSDEM 2011, I am proud to take part in the conference by organizing a day of talks on open source telephony. The talks will take place on Sunday, February 6th. Without any further delay, here are the talks that are scheduled for the open source telephony dev room at FOSDEM 2011.

  • “Introduction to Asterisk Developement”, Russell Bryant
  • “Digital PSTN Connectivity with Asterisk”, Jakub Klausa
  • “Mobicents 2.0, The Open Source Java Communication Platform”, Jean Deruelle
  • “Scaling location services in large SIP networks with Kamailio”, Henning Westerholt, Marius Zbihlei
  • “Unifying SIP and Web worlds with Lua (Kamailio)”, Daniel-Constantin Mierla
  • “XiVO IPBX OpenHardware”, Xavier Carcelle
  • “Unified Communications – Future (Yate and YateClient)”, Diana Cionoiu
  • “Asterisk SCF (Scalable Communications Framework)”, Kevin P. Fleming
  • “Developing rich VoIP applications with SIPSIMPLE SDK”, Saúl Ibarra Corretgé
  • “SIP Communicator: Building a Multi-Protocol Multi-OS Communications Client”, Emil Ivov

I hope to see you there!

Automated Testing Update


A while back, I posted a message about an effort to improve automated testing in the Asterisk project. I wanted to give an update on how that project has progressed for those that have not been following along very closely.

We started using Bamboo as a continuous integration tool, which you can find running at Note that some of the pass/fail statistics on there are a bit skewed, as the Bamboo server was just rebuilt and things were failing as everything was put back together.

A lot of really good automated test cases have been developed, and more are constantly being added. There are currently 85 test cases that run against Asterisk trunk after every change to the code. While some tests are small in scope, many of them cover significant call scenarios, such as various methods of doing transfers and call parking.

I apologize for the previous flood of Bamboo emails to the -dev list. 🙂 I now have a new mailing list created for those that would like to subscribe to those messages.

Additionally, one of the latest updates to our Bamboo setup is automated testing code coverage analysis. It will tell us exactly what code ran as a result of our automated test cases. It provides a good metric to start using to help identify areas of Asterisk that are in need of more test cases. You can find the code coverage reports for the latest builds of Asterisk trunk and 1.8 on Linux in the artifacts tab when viewing the details of a build.

I’m proud of the progress we have made so far and am excited to continue aggressive development of automated test cases for Asterisk. The tests we have are already catching problems on a regular basis. The resulting quality improvements make the job of the development team easier, as well as result in a better experience for end users.

If you’re looking for a way to contribute to Asterisk and you are more comfortable writing scripts instead of C code, then the external test suite is a great way to get involved and help out.

Thank you all for your continued support of Asterisk!

Best Regards,

Russell Bryant
Digium, Inc. | Engineering Manager, Open Source Software
445 Jan Davis Drive NW – Huntsville, AL 35806 – USA
jabber: -=- skype: russell-bryant -=- -=-