In recent years, XMission has greatly expanded our focus on business products, including colocation, advanced web hosting (with our Stackable cloud product), hosted email (with Zimbra), and business telephony. As those products matured, it was only a matter of time before we saw the need to perform this audit, especially as enterprise clients started to look more seriously at XMission as a vendor. While we are a privately held company, and therefore have no Sarbanes Oxley compliancy concerns ourselves, we recognized that compliance sensitive companies often require SSAE 16 certification, which include publicly-traded enterprises, financial firms, and healthcare organizations.

Ultimately, I think that all of the work we did in preparation for the audit brought about many positive changes. While we have done many things related to IT security for years not only due to our own commitment to best practices but also to maintain PCI compliancy, we took this opportunity to review and refine our policies and procedures. We performed a new risk assessment and found better ways to mitigate, if not outright prevent, a few more potential issues. We had some productive conversations and better organized our documentation. All in all, the process brought renewed rigor and focus to things that warrant close scrutiny on an annual basis.

In order to complete the audit, XMission management developed rigorous internal control objectives to support first-class data center, hosting and networking management services. You can think of internal controls as the processes by which an organization manages its people and systems. It is how a business conducts business, day to day. These controls should be closely aligned with an entity’s goals and objectives. When an outside auditor comes in, they first review the organization’s control objectives to determine if they appear to be reasonable and then secondly test their processes and see if the entity reliably meets those objectives. Professing best practices isn’t enough; the proof is in the pudding. And since we chose a type 2 audit, we were required to prove the operating effectiveness of our internal controls throughout the audit period. Abiding by new requirements under SSAE 16, the report also contains a written assertion from management regarding the systems and a services auditor’s opinion letter.

An SSAE 16 audit report provides a framework for a service organization to have an outside entity examine their internal controls, which can then be provided to its enterprise clients. Therefore, an SSAE 16 certification assures new and existing customers that XMission’s policies and procedures are sound and that their critical Internet services and data are secure. Colocation and Stackable customers can request a copy of our audit report, which should make it easier for them to pass their own SSAE 16 audit. If anything is missing which could help them with their own audit, or better set them at ease regarding the products they purchase from XMission, we gladly welcome such requests. We have already started to evaluate how we will expand the scope in next year’s audit.

Our audit was conducted by CPA firm A-lign, who specializes in these audits for IT firms. SSAE 16 replaces the previous industry standard, SAS 70 and brings the US closer to the international standard, ISAE 3402.

For more information, please visit xmission.com/SSAE16.

Grant Sperry

XMission VP of Operations

Of course, XMission has always provided local support. All of our support technicians are based in Salt Lake City, Utah although some do work from home these days, for at least part of the week.

Systematizing and Standardizing Using Puppet

A particularly useful tool for us is Puppet, which we have setup to automatically perform certain tasks. This not only saves us time but ensures standardization of systems, including installation and configuration management. When similar systems (e.g., web servers) are identical, maintenance and management become much easier.

Monitoring and Notifications Via Monit and Nagios

Both Monit and Nagios provide not only excellent monitoring and notification solutions but they can also be configured to automatically restart services when they notice something has gone down. Nagios also provides a particularly helpful dashboard to easily appraise the health of our systems at any moment in time and generates statistical information regarding uptime.

Utilization Aggregation and Graphing with Cacti

While system and network uptime is very important, uptime alone is not good enough since a service could be running but at a degraded level of performance. Cacti allows us to see exactly how many resources are being used at any given moment as well as over time. This also helps us track trends and potential bottlenecks and plan for future growth.

Virtualization using Xen and Virtuozzo

Virtualization allows us to abstract out the hardware layer and run multiple servers on one piece of hardware. This not only allows us to more efficiently utilize hardware, thus reducing power consumption, but virtualization tools now include the ability to clone server images, back them up, and quickly migrate a virtualized server from one computer to another, which could be in a facility anywhere else in the world. In a word, magic.

XMission uses Xen for many of our servers, including many of our traditional web hosting Linux servers. We run Virtuozzo on our next generation web hosting solution “Stackable” (http://stackable.com) to provide virtualized server instances (that we call “containers”) on top of racks full of blade servers with Linux and the Apache web server in various configurations. While Stackable could therefore be considered a “cloud computing” solution, especially since customers are able to instantaneously adjust resources both vertically and horizontally, the innovative product most closely resembles managed hosting solutions.

Conclusion

I’ve only touched upon a handful of the many solutions we’re currently using. The days of dedicating a piece of hardware to a single task is a thing of the past, as is manually installing and configuring servers. As computing becomes more powerful and complex, better tools and software emerge to help us provide ever better products and services.

Product Links

Puppet: http://www.puppetlabs.com/

Monit: http://mmonit.com/monit/

Nagios: http://www.nagios.org/

Cacti: http://www.cacti.net/

Xen: http://www.xen.org/

Virtuozzo: http://www.parallels.com/products/pvc45/

XMission Links

Our Nagios page: http://stats.xmission.com/nagios/

Realtime networking statistics: http://stats.xmission.com/routers/public/

The idea is that within the technology, education, community and business tracks, there will be plenty to talk about.

• HOWTOs will help others get started
• HOWTOs will help families use open source together
• HOWTOs will provide a stepping stone along the road to technology mastery
• HOWTOs will let those that are masters get together and make the products better.

The conference boasts an impressive list of speakers covering an extensive range of topics:

http://2008.utosc.com/speaker/list/

Two of XMission’s employees will be speaking at the conference and many XMission staffers will be attending so we hope to see you there.

While XMission buys 100% wind power for our data center, we want to do better than simply offset our carbon footprint. We want to improve efficiencies. Not only is that always good for business, it is forward thinking.

With these concerns in mind, what is the solution? Of course, there are many things that can and should be done to help remedy these challenges. Last year, we built a return air plenum in our ceiling to better extract heat from the hot rows (we already had hot and cold rows, like most data centers). This has helped up to a point but we are next going to use blanking panels to better isolate the hot and cold rows. This not only helps get the chilled air to the computers but it minimizes bleed over into the hot rows; air conditioners can more efficiently cool 90+ degree air than 80 degree air. The Liebert systems we have in our facility can cool air down about 30 degrees but they can only make the air so cold.

We’ll see how much the blanking panels help. The next step is to fully isolate the hot and cold rows using plexiglas above the cabinets and at the end of the rows. This includes putting plexiglas doors at the end of each row, so it is a somewhat extreme measure but it may well be worth doing.

After that? Our current facility is almost full but we are evaluating expansion. If we build our own, we might go bleeding edge green, which includes using huge fans to extract the heat entirely out of the building and large evaporative coolers to help with the cooling. Mechanical engineers are seeing some success with these methods, dramatically increasing air flow to manage the heat problems. Another alternative, water cooling, is also being explored but only can work where all gear supports water cooling systems directly to the CPU. That might be a solution in the future but for now it doesn’t make sense in colocation facilities where customers have diverse systems which often can change.

Stay tuned while we adapt and look towards the future.