We recently underwent a re-engineering of our Internet facing infrastructure to increase resilience and redundancy. We utilized a boutique network consulting firm that has designed the infrastructure of such giants as facebook and myspace.
Our Internet facing infrastructure is pretty standard: redundant firewalls, load balancers, and network switches. With this new network design, we can undergo multiple failures (for example one firewall, a load balancer and a network switch) without dropping, in most cases, a single packet.
This is really neat stuff which I won’t dive into. The interesting point is that since the infrastructure is so redundant, we now need to work extra hard to realize that a componant has failed. When the entire set of network devices are so self-healing, it is actually pretty hard to determine that one has failed. What an unexpected problem for an MVaaS provider!
As the Brits would say, “touch wood“.