Building Home Networks Like a Pro – 2 – The Router: Initial setup

The first thing you are going to improve on your network is your router, since in a small business network as well as in an home network, it will be on of the central components offering basic and sometimes quite advanced network services.

As a bastion host, don’t hope you will be using it as a NAS, it is a very bad security decision. NAS devices are not designed to withstand attacks, while a router and firewall devices is designed to do exactly that. Security and file sharing do not mix well.

As a home network, you will be making decision to improve the performance of entertainment services and online games, probably, while shaping down traffic from file sharing and such. Somehow, as said before, it is a mirror world from business applications.

Don’t expect this to be as cheap as a COTS device, but also don’t expect this first project will break the bank.

Choosing the software for our Router

As a firewall-router distribution, I’ve chosen for you PFSense as the operating system for our router. Or you can use OPNSense, a current spinoff of the project.
Nothing against any of them, PFSense is on the market since a long time. I have quite a long history of projects using it under my belt, OPNSense is very promising, very modern and I will give it a try ASAP, I really love it from the first tests I carried out. PFSense on the other end is something i used for a long time in production environments, so my personal experience with it is way higher. Most of this tutorial would have been similar even with OPNSense, so please feel free to give it a spin.

The Hardware

Netgate, the current owner of the project makes some fine hardware devices for PFSense routers, with the software preinstalled.
You COULD do that, sure, but where would be the fun!?

We will be building our firewall-router from very nice embedded hardware. There are many devices and boards on the market that can be used with PFSense, even an old pc you have tossed in the attic. Just make sure you have a decent number of supported network cards, like 3 or 4.

I’ve built my home device on a PCENGINES board. There are some kits on a fine website, VARIA STORE, that are totally ok with PFSense based routers. By the way: if you are rich or paranoid or both, you can buy two and make a very resilient redundant setup with minimal effort, but this is beyond the scope of this tutorial now.

The router part list

• the board (like an APU3C4 with 4gb of ram)
• an SSD, a 16GB mSATA is ok. PCEngines do support booting from SD cards, you could install PFSense there, but don’t use SD cards as the root drive, really, don’t do that.
• a power supply
• a chassis
• a temporary usb drive to store the install image
• (usually very optional) an LTE/4G router or modem with ethernet port, or go for the internal add-on board for APU3C4, but I’ve never personally tested it.
• (very NECESSARY) a serial to usb adapter with an RS232 female connector (or an adapter)

WHAT? Serial? Yes, what’s the point of an embedded system if you have a monitor connector? 😀

Now just assemble the hardware. Beware that some kits have heat-sinking plates that will be using the chassis as the final heat sinking elements. Don’t forget to use them/remove protective sleaves from them, since this will be a fanless device.

When it is assembled, just download the appropriate PFSense boot image.

Installing the software for our Router

You will need an AMD64 memstick/ISO image. NanoBSD images, even if they seem to imply that they are made for this kind of hardware, are now deprecated. Don’t use them.

To download an appropriate image for the described setup use the following options:

Setup 101

Just flash it to the usb using WIN32Imager or DD or something like that, just follow this official guide.

Once it is flashed, plug it into the USB port on your soon-to-be new router.

Connect the RS232 between your computer and new router with the adapter and fire up your minicom. Putty or other serial terminal client of choice. Set the COM port to 115200bps, 8-n-1.

Plug the power supply to the device, you will see a text bootloader. Quickly send the appropriate key to choose the boot device and choose the usb. If no boot screen is shown, please remember that some APU boards have a default serial speed of 38400bps set in the bios/coreboot, so change your serial settings to that. As soon as the bootloader loads up the PFSense kernel the speed will be changed again to 115200bps, so please correct that again to interact with the serial console.

The installation will take place almost automatically, just accept the default for everything.

Shutdown, remove power, remove the usb and serial adapter and connect to the middle ethernet port with your pc, DHCP client enabled.

At some point you will hear a jingle from the pc speaker. The DHCP should be assigning you an IP address.

Now, just fire up a browser and connect to https://192.168.1.1 and login with admin/pfsense.

Adding uplinks

I will guide you adding uplinks first in this kind of setup, to let you make sure they work. That if something breaks down the line while defining VLANs and such, well, you will know you broke it, not your ISP. I know it is quite the reversal of what it usually done professionally, but in a non-pro environment this can save you a lot of pain.

Basic PFSense router options

When you first login to your new PFSense appliance, you are greeted with a wizard.

You will see that you already have two network interfaces configured, a LAN and a WAN. The WAN is configured to get its address from DHCP, while the LAN has DHCP server enabled. If you plug the operator’s CPE to the first ethernet on the left, you will probably be surfing the web.

Let’s give the interfaces proper names now.

Let’s add some more WANs

Add the next network port available (should be em2) like in the slideshow below.

Repeat until you run out of uplink services or ports on the appliance, excluding the LAN port, obviously.

Now go to advanced in the system menu. Choose the Miscellaneous tab.

What the “use sticky connections” does is simple: it loads balancing works by sending connections in round robin between active uplinks. If an inside host is selected for going out with its first connection thru a certain gateway, with this on it is guaranteed to go out with that gateway for all the subsequent connections, for at least 3600 seconds (or what you set on the parameter, expressed in seconds). Why? Because from the contacted host perspective you would appear constantly switching IP address. This will probably invalidate you http sessions frequently, login you out from services and such. This is at least if the link doesn’t go down, some lines below.

What controls that is this:

What happens, if you choose to enable this, is that all states will be reset when a gateway is marked is down. This can quickly make the computers resume connectivity when a gateway goes down, but it will reset all states even for those who were on the still functioning uplink. The choice is yours, but in my opinion enable it only if your connections are very stable and you require a very quick switchover between failing connections. Normally keep it off, the states will timeout on their own and new connections will go thru the right gateway.

Now that you know how the load balancing works, think about this: if you have two connections but they are not equal in terms of bandwidth, the connections will go evenly thru both. Can this be changed? Sure. In every gateway there is a parameter, the weight. Well hidden in the advanced settings. Click on Show Advanced.

Weight is how much priority the connection has when used in a routing group where it has another connection with the same TIER-N number. It will make more sense soon, but think about it in this terms. If your VDSL is 3 times “bigger” than your WIMAX uplink and you will balance between the two, set the VDSL to 3 and the WIMAX to 1. This has no effect in failover groups.

Tweaking the DNS settings of our Router

Now it is time to tweak the final setting to the DNS settings.

Always remember the diagnostic protocol when a problem arises and DNS servers could be involved.

1. It is not DNS.
2. There’s no way it’s DNS!
3. It was DNS.

So please be careful with those settings.

First things first, go to System, General Setup. Add some DNS you like, two for uplink you have and assign two of them to every outgoing connection and take note of which went where, but mix them up, so two DNS servers from the same operator do not end up on the same uplink.

Save it and proceed. Now go to Services, DNS Resolver. Configure it this way:

Now the final tweak into the gateways. Remember when I told you to note down which DNS went where? Now add one of the DNS addresses to each corresponding gateway as a monitor IP.

We do this because the default monitor IP is the default gateway, but it can be working even if the line is down, because it usually is a local CPE. The monitor IP is important because it is how the firewall decides to include or exclude an uplink from the specific gateway group.

Gateway groups

Finally, we are almost done!

Let’s define appropriate gateway usage policies.

Go into System, Routing, Gateway groups tab.

It is time to define how we want to use our gateways.
Every group is a potential usage sequence of the gateways, defined as a list of outgoing uplinks in priority order, called TIER-N, where the smallest N has priority.

What happens if you define a group with two connections on the same TIER? If it is the smallest N-TIER that has a connection alive, traffic is sent in a round robin fashion out of those links. The sticky connections flag you enabled before ensures that the same host always goes out with the same IP for at least 3600 seconds. Do not disable sticky connections. It can be done only if your provider allows you somehow to retain your IP address across multiple links via a NAT on their side for example, but this is a very rare occurrence even in business scenarios, I personally never saw that in a home scenario.

I usually define at least 3 groups (1 for priority to wan1, 1 for priority to wan2, 1 to balance between the two):

We defined 3 scenarios in the picture above. What will happen if we assign traffic to each gateway group is this:

• FTTC_PREF = “Try FTTC, if down go to WIMAX, else go to 4G”
• WIMAX_PREF = “Try WIMAX, if down go to FTTC, else go to 4G”
• BALANCE = “Try to balance between FTTC and WIMAX, if one is down use the remaining, if both down go to 4G”

Gateway groups balance: never use it!

Last thing to do is that: which gateway group shall we use as default. NEVER use BALANCE. Unpredictably weird stuff will happen. Just choose on of the other two. Always use a group that gives a strict TIER priority. When you made up your mind, set the correct group here. Disable IPV6 for now, as always.

Firewall Rules

Now it is finally time to balance connections and packets. Go to firewall rules and select the ADMIN tab.

For many interface you will see you have many rules already in place. This is good, they are there to protect you from mistakes and evil guys.

On the admin interface you will see there are “antilockout” rules in place. You can’t remove them from the firewall rule list interface, you have to go to advanced settings to remove them, but don’t do that. At least for some time until you are confident with pfsense. If you remove them and lock yourself out while thinkering with the firewall ruleset, be prepared to use the serial cable again.

Create a rule using the up facing arrow, this will position the rule on top of the others.

You should end up with something like this when saving, confirming and reloading.

Now check the firewall, NAT, outbound tab. It should be set to fully Automatic. Leave it that way for now. It won’t stay long that way, but for now just leave it as it is.

Bonus. Click on the PFSense logo on the top left corner to return to the dashboard and add this widget to monitor the gateways.

Many widget are there tempting you. Do not add too many of them. The hardware you are on is limited and it is best used for network routing, load balancing and firewalling. Some of them are really resource-intensive.

This is it. Your devices should be balanced 3:1 between your FTTC and your WIMAX, or whatever you connected to your brand new PFSense home made appliance.

The reasons for having multiple uplinks to the router at home

As you can see, I assumed you have more than one internet connection at home while many businesses still don’t have a redundant internet connection.

Am I crazy? I will explain why I’m not. Sometimes a redundant internet connection at home is even more desirable than at the office.

Why you should consider having it

Dedicated business connectivity usually has business-oriented SLAs for restoring a faulty link, link quality and traffic priority is way higher than those of home internet connections. I personally saw MOLO17’s main uplink failing two times totalling a whopping 3 minutes downtime in one year. Sure a critical event would take more to recovery, so a second uplink is always desirable in a business setting. But anyway, the quality of service and uptime are excellent even with only one link.

Sure, if you get your commercial operator’s generic offerings for “business”, things usually get worse, but never as bad as with home user-targeted connections. You should always let a professional design your network and buy a “naked” dedicated internet uplink from a pro-grade ISP. By the way, drop a line to our sales team, if you want us to help your business with all this.

Anyway, comparing all this with my home connections, my home suffers something like 30 minutes downtime per connection, per month.

Two connections also means more bandwidth available

Having two connections can also really help you with bandwidth if proper load balancing is done. As I probably mentioned before, I live in an old big house that was built by my ancestors and we always lived there throughout the generations. We divided it in different apartments for each floor, so that we can share resources while maintaining our privacy, but the day my parents discovered Netflix was a hard hit on the total bandwidth.

Stay away from critical events

Also, speaking of critical events and downtime. One day a car crashed against the mini DSLAM that terminates our VDSL line. It took a week to have a new mini-DSLAM in place. During that week, Netflix was sometimes sluggish in our home, but we still had more than decent connectivity, because the second WAN is a WIMAX antenna, sitting safely on the roof, three floors away from the crash. Should it go down, the 4G/LTE connectivity will kick in as last resort.

What about planning to live in a smart home?

With many services, like IoT, building automation, even security systems, as well as network streaming rapidly taking the place of cable/satellite TV, it is nice to have a backup uplink. In normal conditions you will still use it to gain some additional bandwidth for Netflix and such. The Internet is quickly replacing other services, or to be more precise, wrapping them as the default delivery medium, so I don’t feel very strange wanting to ensure that connectivity is almost never lost even (or especially?) at home.

Choosing the second (and third?) uplink

If you are still reading, you either want to know how deep my madness goes or you are starting to believe that you need a second uplink for your house. 🙂

Choosing your second uplink is very simple:

• 1. Pick different technologies for the physical medium;
• 2. Pick different operators (really different, if possibile, with a different uplink path to the internet).

For example, in my case, the main connection was a wired one that existed for ages. I was running my first 9600bps modem on that same cable. Time passed and we upgraded the line to an ISDN, then to ADSL, then to VDSL, FTTC. We are in a rural area, so there is no FTTH at the moment.

A radio connectivity can be a smart decision

When I started to think to upgrade the connectivity with a second line, I went with a radio provider, never thinking in getting a second xDSL line, because… Well, the car event made it abundantly clear. Most operators share the same cabinets or at least are one near another because they share the same underground tubing. If something bad happens to one uplink, you’ll want the other one to be as unrelated to that event as possible and as far away from it as possible.

Even if you see in your street two cabinets from two providers one far away from the other, don’t assume that they are unrelated. The day an excavator down the road will break the connection for both of them, you will find out what a web of delicate connections unfolds from those small cabinets under our feet. A brief moment of enthusiastic awe for human engineering will quickly be overwhelmed by the sadness of being the lucky paying customer for two offline connections. It’s the true story from a friend who didn’t listen to me.

So basically just get a wired and a wireless connection.

What if there’s no way to get a wired connection to our router?

But what if you are so unlucky that no wired internet is available at your house, or its quality is simply not worth it?
(For non-Italians reading this: yes, this still happens an awful lot here).

In that case I suggest you to get a wireless unmetered uplink, like a WIMAX/HIPERLAN and 4G/LTE ethernet modem with, usually, a metered contract. In that case you will probably be using the metered line as a pure failover line, not as part of a load balancing group. If you are lucky enough to be able to get an unmetered 4G uplink, then you could also use it for balance, just keep in mind that latency is often an issue with 4G/LTE.

In general if you don’t plan to balance because you don’t feel your bandwidth is low, but you only want a failover connection, a good LTE modem or router is generally ok.

Should I consider to put a third uplink?

Using, like myself even a third line as an absolute failover is somewhat an overkill, I agree with you on this. But at some point I will explain you how I’ve done it, and you will see that it is a nice to have additional feature. Just a little spoiler: I have a mobile router that always carry with me. When at home and I dock it in its “home made cradle”, it acts as a third failover, while when I’m outside it is my main “road warrior” connectivity. But I will cover this in a future post!

Conclusion

Just make sure to follow this blog for the rest of the PFSense tutorial. Next time we will deal with the internal networking.

This tutorial is part of a Series.