Re-vamped my home network again

Joined
Dec 7, 2012
Messages
4,149
I have quite a bit of Aruba Instant On switches and Access Points. 1x 1930 24-port 195W, 2x 1930 8-port POE, 1x 1930 8-port NON-POE and 1x 1830 8-port POE powered desktop. Then an AP22 and an AP12.

Mid-summer last year, with HPE (Hewlett-Packard Enterprise) trying to acquire Juniper Networks, the DOJ is requiring them to divest the Aruba Instant On line. It is kinda up in the air as to who will acquire the line, and what might happen to it. With that coming down the line, and me having some weird controller issues as of late (trying to commit changes on the main 1930 that would error out and leaving certain ports wrongly configured), I wanted to sell off this infrastructure stack while it still holds some value.

I did try managing the AIO switches locally, but was not pleased with how slow they were to manage locally.

A friend mentioned he had two Cisco Catalyst 3850s that were recently decommissioned that were mine if I picked them up... OKKK! So I picked up a 12-port 10G SFP+ Catalyst WS-C3850-12XS and a 48-port 1GbE POE Catalyst WS-3850-48P. I have them stacked with StackWise 480 cables and stacked power. Each switch has one PSU.

Replaced the Attic 1930 8-port with a Catalyst C3560 WS-C3560CX-8PC-S uplinked by 2x OS2 SMF with 1GB LR transceivers. Replaced the Aruba Instant On AP22 with an Aruba (Enterprise) AP535 running in Instant mode with ArubaOS 8.

I have another 1930 8-port in the shed that will need to be swapped out. I have to find it, but I think I have an ancient 8-port 10/100 2960 in my attic that could serve this shed well as it's uplinked by a Ubiquiti 2.4GHz NanoBeam and truly only serves to uplink 4 POE cameras and my shed iMac. Usual throughput in this shed is 70-80Mbps, which is more than fine.

My desk switch will be replaced with another C3560CX.

Routing wise, I have been wanting to get rid of my Ubiquiti EdgeRouter X SFP for some time in preparation for Spectrum's eventual high-split symmetrical Gigabit service and enact some QOS. That was replaced with a Lenovo ThinkCentre M920q Tiny (9th Gen i5, 16GB of RAM and 512GB NVMe) with a 10GB Intel X710-DA2 SFP+ network card running VyOS. Very similar in configuration to the EdgeRouter being so close to Vyatta. WAN in comes via SMF to one interface, uplinked to the 10G 3850 via MMF. Also bought another Intel X710-DA2 FlexLOM NIC to replace a 4x1GB FlexLOM card in the HP DL360 G9 and uplinked my Hyper-V host via 2x10G over MMF with both interfaces in a SET team.

VyOS is setup as zone based, all LAN/trusted networks routed at the 3850 stack, DMZ, Camera and WAN zones routed at the VyOS router, with static routes on the 3850 core and in the VyOS router to tell where each network lives. Also implemented a transit network so traffic does not go over the Management VLAN.

This is truly my first time diving into Cisco switches. Every organization I have worked at has always had Cisco switches, but their management has never been my responsibility. Configuring them is not hard and pretty easy to pickup. What I like is how instant they are. I can shut, no shut a port in seconds versus waiting on the Aruba Instant On cloud controller to take minutes to re-sync the changes. Just a prime example of SMB/SoHo vs Enterprise class equipment.

I can definitely tell the Aruba AP-535 having 4x4 radios and higher bandwidth channels over the AP22 2x2 is easily performing better. I'd usually be able to pull 400-500Mbps off of the AP22 with the amount of wireless clients. The AP-535 will easily hit 800Mbps+.

So I'm very pleased, this will serve me for quite some time. I will say the placebo effect is in full swing as the network feels "snappier". Speedtests on Apple TV which has a direct Cat6 run back to the 48-port 3850, and a Gigabit NIC seem to "saturate" the line speed of my Spectrum Gigabit service instantly. Where as, during previous Speedtests on the AIO equipment, would take a little time to get up to ISP line rate and almost "buffer".

This stack will eventually all go into a new 42U basement rack when we build our new house in a few years. Also... cheap. The AP-535s are $50-70 used. Enjoy some pics.

1774804754842.webp


1774804772604.webp


1774804782333.webp


1774804788067.webp
 
Last edited:
I followed most of it but wondered why his home network was so complex?

I only pay for 500 Mbps service. Have (4) Deco wifi7 access points 3 of which are connected via 16 port unmanaged gigabit switch. Some sort of TP-Link wireless extender 800’ to the shop where the 4th Deco is located. Every room is wired CAT6. Have a PoE Deco outside AP to support Eufy cameras at the gate.
 
I followed most of it but wondered why his home network was so complex?

I only pay for 500 Mbps service. Have (4) Deco wifi7 access points 3 of which are connected via 16 port unmanaged gigabit switch. Some sort of TP-Link wireless extender 800’ to the shop where the 4th Deco is located. Every room is wired CAT6. Have a PoE Deco outside AP to support Eufy cameras at the gate.
It’s really not that complex. A single mode fiber run goes thru the floor of the 1st floor network closet through the basement up the wall to the 2nd floor walk in attic. Single mode fiber was chosen cause I wanted future proof.

The 48 port will really come alive in the new house running 5 APs, probably 15 cameras, office drops and drops per every tv. Not to mention garage drops, and a couple fiber runs to a cold storage pole barn (with cameras) and driveway cameras.
 
Nice setup!

My rack's a mess. Here's the less messy part...

cKivsl4.jpg
Thanks, I like those cables. What are those? I want some attractive cables when I rack in the “permanent” 42u of the new house. I’ll most likely do a 24p patch panel above and below the 48p switch for cleanliness.
 
Nice setup! I'm familiar with that Cisco kit, as that's what I've generally deployed at work, along with some Meraki. Any particular reason for going VyOS over PFSense or OPNSense?
 
Nice setup! I'm familiar with that Cisco kit, as that's what I've generally deployed at work, along with some Meraki. Any particular reason for going VyOS over PFSense or OPNSense?
Thank you. About a month back I spun up a VM on my DL360G9 and gave it two NICs, temporarily, one as WAN, one trunked back into the SET team feeding the core switch. Was testing out something to get off of my EdgeRouter X SFP.

I tried OPNSense and PFSense but within 10 minutes I really didn’t care for the layout, I’m sure it works great but I will be honest I didn’t give it a chance. I had known about VyOS and tried it out and loved how it was so close to Vyatta/EdgeRouter CLI. Spun it up on the VM, demoed it for a month and then decided to move it to its dedicated machine as a router.

Thought I may try again someday in the future. What’s your thoughts? You like one over the other? I guess I’m missing IDS with VyOS, but I do heavily filter DNS traffic via Pihole blocklists and then Cloudflare 1.1.1.2 malware for families as upstream.
 
Thank you. About a month back I spun up a VM on my DL360G9 and gave it two NICs, temporarily, one as WAN, one trunked back into the SET team feeding the core switch. Was testing out something to get off of my EdgeRouter X SFP.

I tried OPNSense and PFSense but within 10 minutes I really didn’t care for the layout, I’m sure it works great but I will be honest I didn’t give it a chance. I had known about VyOS and tried it out and loved how it was so close to Vyatta/EdgeRouter CLI. Spun it up on the VM, demoed it for a month and then decided to move it to its dedicated machine as a router.

Thought I may try again someday in the future. What’s your thoughts? You like one over the other? I guess I’m missing IDS with VyOS, but I do heavily filter DNS traffic via Pihole blocklists and then Cloudflare 1.1.1.2 malware for families as upstream.
I've been very impressed with OPNSense and is what I would use if I wasn't using Unifi. PFSense became a bit too "commercial" for a lot of people, and then the website fiasco soured many more, but it's a solid product.

I actually moved away from PiHole, as I had so many performance issues with V6 on baremetal. This (the performance problem) is an ongoing known issue, so I'm currently on Adguard Home, which supports encrypted DNS out of the gate (unlike PiHole), and is considerably faster. I was originally running it on OpenBSD, but a recent hardware swap had me decide to try it on Rocky Linux 10 instead, so that's what it is currently running on, just an HP mini w/i5 CPU. I may actually move it back to BSD though, since the integrated NIC (Intel PRO1000) is depreciated in Rocky, which is a bit weird... :oops:
 
I feel you on Instant On… I went from Ubiquiti everything to Instant On everything back to Ubiquiti everything lol.

There was like a 5 year period where UniFi was trash and Instant On was awesome and slowly its going back the other way.
 
I've been very impressed with OPNSense and is what I would use if I wasn't using Unifi. PFSense became a bit too "commercial" for a lot of people, and then the website fiasco soured many more, but it's a solid product.

I actually moved away from PiHole, as I had so many performance issues with V6 on baremetal. This (the performance problem) is an ongoing known issue, so I'm currently on Adguard Home, which supports encrypted DNS out of the gate (unlike PiHole), and is considerably faster. I was originally running it on OpenBSD, but a recent hardware swap had me decide to try it on Rocky Linux 10 instead, so that's what it is currently running on, just an HP mini w/i5 CPU. I may actually move it back to BSD though, since the integrated NIC (Intel PRO1000) is depreciated in Rocky, which is a bit weird... :oops:
Thanks for the mention of Adguard. I have not played with it before but will check it out. You had me at encrypted DNS. I run Pihole on a Docker container on a Debian VM and can’t say I’ve noticed any performance issues but like you mentioned maybe I haven’t cause of not bare metal.

I’ll at some point have to check out OPNsense again.
 
I feel you on Instant On… I went from Ubiquiti everything to Instant On everything back to Ubiquiti everything lol.

There was like a 5 year period where UniFi was trash and Instant On was awesome and slowly its going back the other way.
Yeah it has really irked me as of late. Trying to change port profiles on my 1930 doing some moving around and it won’t commit… just errors out. I’m like you need to leave lol.

I was courting a UniFi purchase but my wants had me in the multi thousand budget and I didn’t wanna spend that — I was being cheap.
 
Thanks for the mention of Adguard. I have not played with it before but will check it out. You had me at encrypted DNS. I run Pihole on a Docker container on a Debian VM and can’t say I’ve noticed any performance issues but like you mentioned maybe I haven’t cause of not bare metal.

I’ll at some point have to check out OPNsense again.
If you are interested in the issue (it happens in VM's as well), here's the thread on github:
https://github.com/pi-hole/FTL/issues/2687
 
If you are interested in the issue (it happens in VM's as well), here's the thread on github:
https://github.com/pi-hole/FTL/issues/2687
Just spun up AdGuard Home in another container, configured it, and shut down Pi-hole. I’m really pleased with it—the visibility is excellent, and performance seems better for queries as well.

For anyone curious, here’s a bit more detail on how I have things structured:

I have Management, Transit, Server/VM, Trusted-Wired, and Trusted-Wireless VLANs, all of which are routed at the core (stacked 3850s) with SVIs living on the core. These are essentially part of the LAN zone; however, only the Management and Transit VLANs have interfaces that live on the router assigned to the LAN zone. These networks generally allow east-to-west traffic.

Then I have DMZ-Wired, DMZ-Wireless (guest Wi-Fi), and Camera VLANs, all of which are routed at the VyOS router. The two DMZ VLANs are part of the DMZ zone, and the Camera VLAN is part of the CAM zone.

All traffic between the core and the VyOS router traverses the Transit VLAN. Static routes are configured on both the core and the router for any non-local networks. The CAM zone is heavily firewalled, allowing only specific outbound traffic (Reolink UUID and related services) for the external app to function.

For DNS on the LAN networks, I use Windows domain controllers, and they forward upstream to Pi-hole (now AdGuard). My wife and I are heavily invested in centralized storage—documents, records, music, photos, etc.—so a file server with centralized authentication made sense a long time ago. The environment is pretty low-maintenance, with everything virtualized and Windows Updates handled via GPOs configured long ago. It's easy for her to go to the S:\ drive and save or open whatever we have. Also being with an IT feller for over 20 years has her very well knowing that the PC should never be trusted as something that'll never fail.

For the DMZ and CAM networks (which have their gateways on the VyOS router), I also assign an additional interface per network (.253, I've always liked high IPs as gateways -- .254) via DHCP for DNS. NAT rules translate that .253 address to the internal Pi-hole/AdGuard instance IP, with corresponding firewall rules allowing DNS traffic. Also have Client Isolation enacted for that Guest SSID on the Aruba APs.

This design allows guest and camera traffic to be filtered without exposing internal IPs—clients just see what appears to be a local DNS server within their subnet. The idea is to keep the experience as “pedestrian” as possible and not invite unnecessary attention. Realistically, I don’t expect anything—most guests on the Guest WiFi would be people I know—but with kids’ friends, extended family, and larger friend gatherings, and the "do you have WiFi" question, I prefer not to leave the door open. Along those same lines, I use 192.168.x.x addressing for DMZ networks to keep things looking typical. Maybe that’s just the security mindset overthinking the human factor a bit.

Everything works very well to be honest. I'm big on "build it to let it run and run well". I do not have the desire to babysit something especially at home. Now I would be the first to agree that with this explanation and the above pics... this may look like a TON, but it truly runs itself. In the next couple of years, we intend to build a new house and a robust home infrastructure will be a cornerstone of the new build. I will run CAT 6A everywhere.

The plan includes 2 APs in the house, 1 in the garage, 2 outdoor APs, 1 AP in the pole barn (fiber uplink from the 10G 3850), hardwired Apple TVs, multiple drops in the office (plus spares), drops in the shop/garage, a full camera system, and likely a fiber run to the end of the driveway for additional cameras. It’ll definitely have plenty of use, and most ports will likely be populated or uplinked. One of the small C3560CX's will live in the pole barn for that subset of cameras and AP, another 3560 will live at my desk and the relic 2009 date code 10/100 3560 (but hey 1G SFP so fiber uplink!) will most likely life inside of a weather proof outdoor box inside of a decorative brick/stone light column running cameras at the end of the driveway.

Given that, it made sense to build this out now with a future-state mindset so that when we move, I can essentially lift-and-shift, cross-connect everything, and be up and running quickly.
 
Back
Top Bottom