The Lighthouse and the Pendulum

…or “How We designed and brought to life the electrical brains, nerves and muscles of our new Headquarters while the clock was ticking“

A quick video message from the protagonists

The story so far…

It was the late afternoon of a relatively calm day at MOLO17, when Mr. Angeli called me.

The Assessment

“I will send you my position, can you get there now?”

“Sure Daniele, can you give me some more information about the task I’m doing there? Shall I bring diagnostic tools or…?”

“No, just come here, it’s just a quick assessment”

“I’m on my way.”

As I got there, I quickly identified my CEO’s car and I parked near it. As I left the car with my tool-ridden backpack (I know how fast a “quick assessment” can turn into a major multi-disciplinary security assessment with some pentest-like recognizance) I saw Mr. Angeli waving at me while standing in front of a beautiful but very neglected piece of architecture. 

The new Headquarters

“Hi Daniele, what is this place?”

“With all the probability, you are looking at our new headquarters”

“Wow!”

My expression included both the awe for the beauty of the place and the worry about all the marks the time left on the building. 

A jump into the ’80s

We entered our new headquarters. It was like passing through a wormhole directly connected to the mid ’80s. Everything, dust and dirt aside, was frozen in time. The fake plastic plants did in fact contribute to that feeling with their perfect appearance despite the years of abandon.

There were no electricity, no heating implants, no air-conditioning and no internet connection.

Most of the stuff inside was seemingly coming right out of a MacGyver or Knight Rider episode, both for the yellowish tone every piece of plastic had acquired with age and for many old-but-very-high-tech-for-that-time technological artifacts, like the glass doors with key card readers on them and the IBM-branded network sockets mounted into brown connection boxes.

“I guess you want me to estimate and plan the new IT infrastructure, is that right?”

“Not only that. You said some of your past experiences were in building automation. What I would like here is an intelligent building with building automation, easy access at every hour, day and night by employees and all the technology you can imagine to reduce the hassle of managing it. And the most common tasks should be doable on a mobile with specific apps. You think you can do that?”

“That’s for sure Daniele, no problem.”

I always get too excited with new and unusual projects. Let alone being part of making MOLO17’s new “home”. This usually makes me forgetting about asking about trivial stuff like deadlines and such…

State of the Art

I opened one of the network sockets. Very old and worn CAT-5 cable. It was surely cutting-edge at that time. In what was and would be the server room, one wall was basically made of ISDN NT1 boxes, a clear sign that intelligent life has existed inside the building after the ’80s, but not for long after that. Various other boxes were riveted on the walls. No trace of a rack or anything, only a bunch of cables coming out of the wall, probably cut away from a patch panel. 

The Basement

In the underground basement, there was a big door, labeled “research and development”. After lock-picking the aforementioned door due to the missing handle, we found the remnants of a small network rack, with a whopping-10mbps network hub inside (yes, hub), with coax and AUI uplink ports. On the walls more ISDN NT1 boxes, along with many fuse / breaker boxes for the electrical wiring.

Electrical implant needs to be replaced. Entirely.

Electrical breaker boxes are not my main field of knowledge, but the panels around the building were obviously way too old to be used today according to current regulations. 

“Well, Daniele, this is going to be a lot of work, both for designing the system and to configure it, let alone we are going to need electricians and such to make new electrical and network wiring from scratch.”

“We need to move the headquarters here in 25 days from now. At least basic services must be up and running by that day”.

“So we are going to need a huge workforce or a miracle. Probably both. But I’m confident we are going get at least one of the two somehow. And probably we’ll also manage to have some fun in the process”

Last time I took part in a team behind a building automation project made from scratch on a new building, it took at least 20 days just to get to a point where all subsystems where designed, mapped on paper and every contractor was aware of what was to be done on their side. The whole thing was powered up the first time almost six months later.

Planning the Miracle

We were in a hurry, apparently, to leave the old headquarters. 

On the way to the soon-to-be-former headquarters, I started making up my mind on how to make said miracle happen and how to control the possible damages if such miracle would be, even partially, unavailable.

By the time I parked the car, this is what was clear:

1. All designing work was to be done ASAP
2. We had no guarantee that the contractors were aware of every building automation system in the world, so selected technologies would have to be chosen between those that are generally easy to explain to electrical wiring-aware personnel, and at the same time all the technology to implement had to be something at least two of us would be perfectly confident using and configuring. 
3. A clear overview of the final system was mandatory, but it had to be broken down to independent modules so that if something wouldn’t be ready in 25 days, well, it’s bummer but it wouldn’t be a total “blocker”.
4. Power supply, “Life support” and network were the top priorities.
5. The more I was thinking about the process of moving physical servers between the two buildings the more it started to look like The Recipe for The Disaster, because of the switchover time between two different fiber uplinks, the operation of dragging tons of metal around in an hurry, the possible security problems leaving the servers unattended during the construction and renovation works, let alone possible damages in transit to hardware and data, ip changes and such. What a better day to finally complete the process of migrating all the production systems to the cloud and to create our HQ datacenter even before installing a single rack unit inside it? Only test systems would remain local. Being test systems, the final users wouldn’t have experienced any downtime, no sensitive data is being moved on the roads and if a server breaks or arrives to destination late, it won’t be a disaster.

I firmly believe that the last proposition saved us all at a certain point. More on that later. 

Requirements and Materials

I started building the network diagram.

“Daniele, are you really sure that we are going to go with wifi as the main connectivity?”

“Yes, we must be able to use our computer around the new HQ without any kind of tether”

“So basically you want the cables to exist mainly in the datacenter and for specialized tasks. I’ll try to find appropriate products in respect to that, but please be prepared: the density of wifi clients in the open space is going to be, well, high. This means using specialized equipment for high density”

“No problem with that”

Wireless-first approach

So we decided to build system around the idea of “wireless first” as client access mean of connection, and because of that we opted for the use of Ubiquiti Network’s UniFi SHD APs everywhere, due to the implementation that favors high density of clients. The added benefit of using UniFi equipment is that we can now manage most of the network settings directly from a mobile phone, PoE switches included.

The wifi controller for UniFi also doubled as a small DVR for the cameras of the same brand, with the same added benefit of viewing the footage directly on the phone or other devices that support the app. So we decided to play along with the convergence strongly suggested by the brand and install cameras from them and we are not disappointed so far.

Firewall

For the firewall, we went with our classic choice, Watchguard, due to our experience with the brand and the dependability it showed through the years / tried and tested compatibility in our scenarios. 

Access control

“I think we should go with NFC cards for physical access, so everybody can access the structure at every hour necessary as you wanted”

“Ok Marco, but I don’t quite like the idea of cards. It looks old. Isn’t there something that I cannot forget at home?”

“Well fingerprint scanners, but you know, after the GDPR, biometry on workers is a dangerous thing…”

“Something that uses phones maybe? I mean, nobody forgets the phone at home”

“Never had to do with such a thing but I believe somebody already made such a product, let me check”

“…and for the doorbells?”

“Those will be internal numbers of the PBX that will call a specific ring group or queue during the day and another during closing hours, so you can answer the door even if nobody is there”

“Can we make the door intercom speak and explain that so the caller won’t be confused?”

“I believe so”

Cutting-edge Intercoms

Basically we ended up with doorbells / intercoms from 2N, with bluetooth and relay modules, that the PBX uses to play music and speak while you are waiting to be connected to a local or remote operator after ringing the doorbell. The operator can activate the relays to open gates and doors while looking at you via the integrated camera and/or the other cameras around the building. The bluetooth module is used to recognize the users, that we can invalidate at will from a database, with time period based permissions (for example external service personnel can only access the structure at certain times, while full time employees can access it as they want).

Access gate

We also integrated every access gate with HTTP APIs with the building automation functions, so we can use the gate as a controller accessory inside time or condition-based scenes. For example opening the gate at 08.30 during working days for our workforce to find it already open in the peak hours, and closing it back at 9.15 when most of the workforce is already inside, both for ease of use and power/wear saving on the electric motor. 

Turn-on the Lights

“Daniele, are you still of the idea of a full fledged building automation?”

“Maybe, but we can always implement it later”

“I beg to differ, if you want to control every LED light in the building with IoT / building automation, it is now or never. I know that the building must be completely rewired even in the electrical parts, so. This is the perfect moment: the wiring for building automation is kinda different from what they are going to do now, so you would end up rewiring everything again in the future”

“I see… Did you already found a supplier for the building automation parts?”

“Yes, I sent you a document with the part list some minutes ago…”

Daniele checked the list- “Ok then. Order what’s needed. What technology are we using?”

“Z-Wave, basically every relay is a node of a radio mesh, with a “special” node that is the controller that will be put in the server room”

“Marco, please make sure that nobody has to get up from their desk just to turn on the light in their room ok?”

“Ok… that’s a bit weird but I think we can manage…”

“Also, I don’t really want to see those ugly IR remotes for the air conditioning. Come up with something else.

“Ok… I think I just saw something that you might like… What about a big red button that controls everything in the room? You press it one time for the light, two times for the air conditioning, three times for something else maybe… You can have up to five presses sequences per button”

“Is it wireless?”

“Sure, we already agreed on the “wireless first” principle!”

A box of about thirty Z-Wave relays, a Fibaro z-wave controller, some IR blasters and a bunch of motion-light-temperature sensors and door opening sensors, along with The Buttons, quickly arrived at the soon-to-be-former HQ, just in time to reach the electricians at the new HQs. I jumped in the car to bring the stuff to them.

Bad news and the avoided failure

As I got there, some bad news were waiting.

“We measured the routes from the gate to the server room and some other places that are on the diagram here” – the installer shows to me my diagram with his notes – “the existing tubing is very, very long from here to there, I believe way too long for network cables”. 

“Yes, 150 meters are way too much for a PoE cable”. 

I opened my laptop and quickly updated the network diagram to add a second network rack in the basement and showed to the installer. This seemed to work.

“Daniele, we need to order another PoE switch and another rack with accessories, but just a small wall-mounted one in this case”

“Why is that?” 

“We just found out that the routes from the server room to the most distant nodes are way more distant than what we imagined from the outside, the tubing takes some weird routes from the server room to the gate.

This is the problem with old buildings, you didn’t design the tubing for the wiring, but you have to design the wiring and then adapt it to the tubing.

Another bad news broke in.

“It’s a long way to the fiber if you wanna rock and roll”

“There is no internet connection in the area, to route the fiber connection to the new HQ, the internet provider needs to procede with a very long excavation work, but the permits from the Municipality require time to be granted. Probably the fiber won’t be there in time“

“That could be a big problem. I will come up with something, don’t worry Daniele.”

No fiber… no party! But there’s plan B

And this is where “condition 5” of the above guidelines I gave myself probably saved us all: no fiber means no public ip. No public ip means production systems offline, if they were already being moved to the new HQ instead of the cloud. Since we migrated everything to the cloud, even the ERPs that run on Windows, converting them to terminal servers, along with our PBX and our main storage migrated to Google Drive, we just needed an uplink to our brand new concentrator in front of our private cloud, while final clients-faced services were all already there long before moving.

As soon as the main network rack came to existence, I fetched one of our spare mobile data SIM card, an LTE/4G USB adapter and a couple of high-gain LTE antennas and plugged everything to the Firebox. Then I just created an application control ruleset that only allowed critical services through the USB device, and there we went online, albeit slowly, way before the official opening.

Good News everyone!

The project was starting to take shape in the physical realm. As the electricians installed a group of relays, I was following them adopting the devices in the integrator. All in all everything was smooth, with only some minor mishaps, like when a too high load was plugged into the relays, effectively melting the solenoid inside them in a cloud of gray smoke.

Network cables were all in place and the APs started to appear around the HQ, broadcasting their SSIDs. Finally we were online, at least for essential services.

Controlling temperatures

The HVAC system was installed just after that and with the help of the IR blasters, it was integrated into the building automation controller, to allow scheduling and reactions to outside events.

Since Tomorrow.

We have always been on schedule, albeit on the edge of it, thanks to careful planning, inside help from many volunteers from the MOLO17’s staff and some fantastic contractors.

We are still implementing new technology and fine tuning the existing automations, for example since a couple of days ago the whole system is HomeKit-enabled, with a custom gateway made from opensource software, custom plugins for it, an embedded APU2 board and our sweat and blood.

That is one of the serious advantages of undertaking such an endeavour with in-house resources in an IT company. You will get it exactly the way you wanted it. I’ve seen tons of home and building automation projects go terribly wrong because of that: someone sells you the building automation package, the contractor installs it, and that is what you get, frozen in time. Without any chance of improvement or update over the years, to the point that COTS IoT devices will often surpass it in features by the time the contractor has finished installing it.

Instead, here we will improve it, day by day.
This building is and will be alive and evolving with us, with cutting edge technology, day by day.