XMission’s troubles started Sunday evening when an ethernet card servicing longtime UTOPIA customers on a Juniper ERX-700 began to fail. It would run for about 20 minutes, then reboot, disconnecting everyone involved. This wouldn’t be so bad if it didn’t take a good 10 minutes to return to service. For the last few years this Juniper ERX-700 has been the bane of our existence. So much that we spent a considerable capital investment in buying a new and powerful Juniper MX240 as a replacement. The MX240 is a much more stable, powerful, and mature system than the ERX-700. Nevertheless, in spite of our repeated reminders for customers to spend some time migrating their service to the new MX240 in a controlled manner, most people are busy, and as long as their Internet is working, it is understood that changing a technical aspect of it is low on their priority list. Unfortunately that came to a point on Sunday night.
Being able to transparently migrate services to upgraded hardware and systems is always the ideal, and I made the call Sunday night to try and do this. Unfortunately, we ran into some inconsistencies between how the old hardware handled connections and how the new hardware handled connections. This left about 10% of our UTOPIA customers offline. The change also had a routing side-effect which hit part of our networking fabric and took out our entire DSL service. Some DSL connections restored immediately, others required some hand holding.
Moreover, the inconsistencies between old and new Juniper hardware was an enormous puzzle. One that Juniper engineers failed to see a way out of for us. It wasn’t until early Tuesday morning when inspiration struck and we found a workaround to fix the majority of the connections.
On behalf of XMission, and personally as well, I apologize for the service interruption that many of our customers experienced on Monday. Since I started XMission, the motto has always been, “The service I would want as a customer.” Yesterday, our system services fell short, and I am sorry for that.