Setting up a virtual VyOS router

I’ve had many routers over the years — Linksys WRT54GL, Linux and iptables, ASUS RT-N66U, Ubiquiti USG, OPNsense, Ubiquiti EdgeRouter, and MikroTik CCR1009. At least those are the ones I can remember.

But I’ve been fascinated by VyOS ever since I first heard about it, even more so after using the EdgeRouter CLI. As both VyOS and EdgeOS are forks of Vyatta.

Fascinated and intimidated is probably more accurate — the lack of a GUI doesn’t give you a lot of help… But at the same time, it’s a great learning experience — you have to know and understand what you are doing 🙂

So I set out to replace my internet facing router — with a virtualized VyOS instance 🙂

Getting started

It all started with this tweet:

The plan changed a bit along the way, but we’ll get to that.

Hardware

My initial plan was to use a Lenovo Thinkstation E32 as the hypervisor, but with an Intel Core i7-4770 @ 3.40Ghz CPU and 16GB RAM, it seemed a bit overkill. So I used a HP EliteDesk 800 G2 SFF instead, Intel Core i5-6600 CPU @ 3.30GHz CPU and 8GB RAM.

The Chelsio T520-CR was recommended for pfSense, by ServeTheHome, and I figured it had a higher chance of working with the ISP SFP module than the Intel X520-DA2 cards I’m using in my other servers.

I did try a Ubiquiti UF-SM-1G-S SFP module in an Intel X520-DA2 card, and the port just turned off. This is apparently some protection mechanism against “unsupported modules” — there are ways to unlock the card, but I didn’t spend any time testing it.

When I first tried the Chelsio T520-CR NIC, I found that it runs incredibly hot 🔥 So I ordered a Noctua NF-A4x10 FLX 40mm fan and mounted it to the heat-sink.

To secure it properly; I removed the heat-sink and cleaned off the old thermal paste.

Then I drilled two holes in the heat-sink, aligned with the screw holes on the fan.

I applied new thermal paste, mounted the heat-sink to the card, and secured the fan with two strip ties 🧑‍🏭

Making room in the rack

I needed to make room in the rack for the hypervisor. I wanted it at the top of the rack, close to the patch panel and switches, and I just like having the router at the top — it makes hierarchical sense to me 🙂

To get the 3U that I needed, the ATS and PDU had to be moved down 1U. The PDU suddenly died — so that kind of worked out 😛

OS

I was a bit ambiguous to the OS; I initially planned to use Ubuntu with KVM, Libvirt, and Cockpit. Then I considered running VyOS bare-metal, as it would make the NICs easier to configure.

Someone on Twitter pointed out that I could just pass the NICs through to the VM — so the virtualization plan was back on 🙂

I had some problems passing the Chelsio card through with Ubuntu, KVM, and Libvirt. It complained about IOMMU group:

group 1 is not viable, Please ensure all devices within the iommu_group are bound to their vfio bus driver.

I didn’t spend too long debugging the problem, instead I switched to Proxmox — which I have wanted to try for a while. And in Proxmox it just worked 🙂

Testing

So, obviously I did several tests in the section above. But I feel this is where the real testing started 🙂

Just a quick recap of that I am testing with:

• HP EliteDesk 800 G2
• Chelsio T520-CR NIC
• Tsuhan THMPRS-3511-10A SC SFP
  • Tx 1310 nm / Rx 1550 nm
  • From ISP, Altibox, came with VMG8825-B60C home central
• Proxmox

WAN connectivity

A few things made it difficult to test and to get opportunities to test:

• I have to take the internet down, quite unpopular in the household
• Many things needs to be set correctly to get a WAN connection, making it difficult to debug
  • The SFP module has to work with the Chelsio card
  • I need to spoof the MAC address of the NIC
  • Internet is available on VLAN 102

I was able to pass the Chelsio card through to VyOS, but I was having big problems with the SFP module. A message in dmsg indicated that the module was not recognized:

unknown port module inserted, forcing TWINAX

I also got an error in VyOS:

could not set flowcontrol for eth1

And when I tried to enable autoneg with ethtool, I got a netlink error. I’ve been thinking about this; and this might not be a problem. It could simply be that the SFP module doesn’t support auto negotiation, and that speed and duplex must be set manually.

I spent three evenings trying to get the SFP module working directly in the Chelsio card, passed through to VyOS. I think I got a DHCPv4 response once, but I was unable to replicate that result… 😕

So, to make things easier to test — I ordered a media converter; TP-Link MC220L, SFP to 1000Base-T. This way I can eliminate potential SFP problems and focus on the rest.

The media converter doesn’t have any configuration, it’s completely transparent and just converts whatever network there are on the SFP module — into Ethernet. I tested it with my EdgeRouter, and after reconfiguring the WAN interface — I had internet 🙂

I passed the onboard 1 Gbit Ethernet NIC on the hypervisor through to the VM, and changed the MAC address from VyOS, this is the interface configuration:

 ethernet eth1 {
     description WAN
     mac xx:xx:xx:xx:xx:xx
     vif 102 {
         address dhcp
     }
 }

And with that — I was able to get WAN up and running on the VyOS 👍

Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             192.168.1.2/24                    u/u  LAN
eth1             -                                 u/u  WAN
eth1.102         my.wan.ip.adr/21                  u/u
lo               127.0.0.1/8                       u/u
                 ::1/128

I’ll revisit the SFP problem at a later stage, trying more with the ISP SFP module, or getting another module with compatible wavelengths.

Routing speeds

With the Chelsio T520-CR paravirtualized as a bridged interface — I got around 8 Gbits/s between two hardware hosts on separate subnets. During the test iptraf3, being single-threaded, was maxing out one core.

Implementation

For now, I’m doing a bare minimum implementation of VyOS. My goal is simply to get it into the network infrastructure.

I’m only doing IPV4, and forwarding all traffic on the WAN interface to my EdgeRouter. Then I can work on moving the networks from the EdgeRouter to VyOS, one network at the time — without interrupting services (too much).

I’ll also revisit IPv6 and router advertisement in a future post.

Proxmox

VM specs:

• Memory: 2 GiB
• Processors: 2 cores
• BIOS: SeaBIOS
• SCSI Controller: VirtIO SCSI
• Hard disk: 8 G
• Network LAN: VirtIO, Chelsio T520-CR bridged
• Network WAN: Passthrough, Intel I219-LM

vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
update-grub

VyOS

I mounted the VyOS 1.4 rolling release ISO in Proxmox, booted into VyOS, and installed:

install image

After a reboot, I started configuring WAN and LAN:

set interfaces ethernet eth0 address '10.0.0.1/29'
set interfaces ethernet eth0 description 'LAN'

set interfaces ethernet eth1 description 'WAN'
set interfaces ethernet eth1 mac '98:xx:xx:xx:xx:xx'
set interfaces ethernet eth1 vif 102 address 'dhcp'

Enabled SSH management:

set service ssh port '22'

Enabled source NAT masquerade on the WAN interface:

set nat source rule 100 outbound-interface 'eth1.102'
set nat source rule 100 source address '10.0.0.0/29'
set nat source rule 100 translation address 'masquerade'

Defined a DHCP server for the LAN subnet:

set service dhcp-server shared-network-name TRNST subnet 10.0.0.0/29 default-router '10.0.0.1'
set service dhcp-server shared-network-name TRNST subnet 10.0.0.0/29 lease '86400'
set service dhcp-server shared-network-name TRNST subnet 10.0.0.0/29 name-server '1.1.1.1'
set service dhcp-server shared-network-name TRNST subnet 10.0.0.0/29 range 0 start '10.0.0.2'
set service dhcp-server shared-network-name TRNST subnet 10.0.0.0/29 range 0 stop '10.0.0.6'

Added a firewall for traffic from the outside, to VyOS:

set firewall name OUTSIDE-LOCAL default-action 'drop'
set firewall name OUTSIDE-LOCAL rule 10 action 'accept'
set firewall name OUTSIDE-LOCAL rule 10 state established 'enable'
set firewall name OUTSIDE-LOCAL rule 10 state related 'enable'
set firewall name OUTSIDE-LOCAL rule 20 action 'accept'
set firewall name OUTSIDE-LOCAL rule 20 icmp type-name 'echo-request'
set firewall name OUTSIDE-LOCAL rule 20 protocol 'icmp'
set firewall name OUTSIDE-LOCAL rule 20 state new 'enable'

set interfaces ethernet eth1 vif 102 firewall local name 'OUTSIDE-LOCAL'

I’m not adding any firewall rules for traffic to the LAN, instead I am forwarding all inbound traffic on the WAN interface to my Ubiquiti EdgeRouter:

set nat destination rule 10 inbound-interface 'eth1.102'
set nat destination rule 10 translation address '10.0.0.2'

To be able to download new system images; I’m setting a system DNS server:

set system name-server '1.1.1.1'

And now to commit and save:

commit && save

Conclusions

It took me a while to get a virtual VyOS into my network stack, and I’m not done yet — I still need to migrate all networks over from the EdgeRouter.

Having the VyOS currently just forward everything to the EdgeRouter, means that the EdgeRouter configuration can stay pretty much untouched for now. And I can migrate one network at the time.

I still have a few issues I’d like to solve; like dropping the media converter and plugging the ISP fiber straight into the Proxmox server. And I need to get IPv6 up and running.

I’m invested in using and learning more about VyOS — there will be more posts in the future 🖖

Resources

• VyOS on blog.kroy.io
• VyOS system optimization
• VyOS forum