Most modern hacking attacks use this approach so it is essential to assume that you are no longer protecting your network from a hacker who is outside your firewall.  You must now concentrate on protecting your information assets from a hacker who is inside your network.  By far the most important aspect of this effort is to limit a hacker’s ability to move from their beach head within your network to the machines which house your sensitive or valuable information assets.  With the rise of hacktivism (politically or ideologically motivated hacking) and socialized hacktivism (which is coordinated using social media) it is also possible that one or more of your network users will be aware of and may even be sympathetic to or participate in the attack.

A strong border firewall is an essential element of any security posture and having a strong firewall is nearly infinitely better than not having one.  Once your border firewall is up, however, it does little more than to establish a border to your defendable network enclave.  A border firewall is simply a barrier that a hacker must get past to own your network and your information.  The border firewall, like everything else on your network, must be updated regularly and quickly to remain effective.

There are several things that you must do including deploying strong internal firewalls to separate your sensitive and valuable information assets from your network operational zone, patching network devices, workstations, and servers as quickly as possible, and testing continuously to ensure that your patched systems are in fact secure.

A good way to ensure that you understand the extent of your organizations vulnerability from both an external and an internal perspective is to employ an on-demand penetration testing tool. This is particularly important for companies which hold trade or national secrets, handle large amounts of money, or hold large amounts of personal information (defense contractors, financial institutions, law firms, etc.).   We have recommended and used  Core Security’s IMPACT Professional with great results.  Tools such as IMPACT Professional allow your risk management and operations teams to penetration test systems during Certification and Accreditation or following patches, installs, or restores and to test those systems again whenever necessary.

Impact’s big brother, CORE INSIGHT Enterprise, automates and schedules the penetration testing effort using the CORE IMPACT Pro engine.  Risk Management professionals provide basic network information, define what test are to be run, and define campaign parameters and INSIGHT Enterprise automatically tests your systems for open, exploitable vulnerabilities and reports when they are found.  Your Risk Managers can also define a goal, such as your most critical systems or those holding your most sensitive or valuable information assets and INSIGHT Enterprise will crawl your network looking for ways to reach and exploit those goals.   INSIGHT Enterprise will produce network maps showing attack vectors that it has found that give it access to those defined goals so that you can close those vectors.  INSIGHT Enterprise will then repeat those campaigns on the schedule that you define looking for new open exploitable vulnerabilities and vectors to reach its defined goals.

Both products take advantage of more than 2,500 exploits which can test more than 10,000 unique targets, with new exploits added automatically as they are discovered often at the rate of between 30 and 50 per month.  The highly sensitive results of these penetration tests stay within your organization, providing the highest level of confidentiality while providing insight to areas of increased risk.

In this third and final entry in this series, I’ll discuss how I configured the networking and storage options for my vSphere environment, and how to get a VM to run inside another VM. (If you haven’t read Parts 1 and 2 yet, just scroll down this page till you find them!)

Networking

As you may recall if you’ve read Parts 1 and 2 of this series, my two ESXi 4.1 servers are running as VMs under VMware Workstation 7.1.3.

VMware Workstation allows us to create 10 virtual NICs per ESXi server.  I chose to create 5 separate networks (port groups) for each ESXi server, with each network having 2 NICs per ESXi server. My port groups are named as follows:

• Management Network
• Storage Network
• vMotion Network
• Fault Tolerance Network
• VM Network

To accomplish this without using VLAN tagging, I first went into VMware Workstation’s Virtual Network Editor, which is found under the Edit menu. This is how I configured the VMnets:

For this set up, I am using VMnets 1, 3, 4, 5 and 6.

Yes, I skipped VMnet2 as it kept thinking it was a Microsoft Automatic Private IP Addressing (APIPA) address. That is, ipconfig reported that it had an address in 169.254.0.0, even though I clearly set it otherwise as you can see from the screen cap. So rather than pounding my head on the desk to figure out why, I just moved on to the next available VMnet! Fortunately, none of the other VMnets had this problem.

All of these VMnets are set as Host-only networks, which means that anything on those networks can talk to anything else on the same VMnet, as well as my host laptop computer.

Note: If for whatever reason you want to reach your management network from another computer, or if you want to allow your guest VMs under ESXi to be able to reach the Internet, then select ‘Bridged’ for the VMnets you will be using for your management and/or VM networks. The Bridged option will allow you to manually or automatically (using the DHCP option) assign an IP address from the ‘real’ network that your physical laptop is currently connected to.

Next, I edited the settings of each ESXi VM in VMware Workstation and set the 10 interfaces as shown below.

Now, using the vSphere Client,  I went into each ESXi server’s Networking Configuration section and created vSwitches for each of these 5 networks. I added two vNICs to each vSwitch so that I can play with the NIC Teaming options. Here’s an example of how I configured one of my ESXi servers:

Storage

Now that networking was set up, it was time to get some iSCSI storage going.

I had previously configured my OpenFiler VM with 6 volumes and iSCSI targets. (Refer to the link I mentioned in Part 2 for tips on setting up OpenFiler.) Here’s what my volumes look like inside OpenFiler:

At this point, you should change the IP address of OpenFiler so that it resides in the same VMnet that you put your Storage Network port group in. In my setup, this is VMnet3 and I gave OpenFiler an IP address of 192.168.143.2. You also need to go into the VMware Workstation settings for the OpenFiler VM and change the Network Adapter so that it resides in the correct VMnet:

Once OpenFiler’s ready to go, then go back into your vSphere Client and go to the Storage Adapters section under the Configuration tab. If you haven’t already, enable your iSCSI Software Adapter via the Properties link. Then, do a Dynamic Discovery using the IP address of your OpenFiler system. Within a couple of seconds after rescanning you should get a list of all of the iSCSI targets you configured.

If you’re not getting this far then review your OpenFiler configuration, especially your iSCSI target Network ACL settings (they should all be set to ‘Allow’) and your Network Access Configuration under the System menu.

If you think you’ve done everything right, but still can’t discover the OpenFiler iSCSI targets, then as a last resort log into the OpenFiler console as root and rename /etc/initiators.deny to /etc/initiators.deny.bak and then rescan. This is technically a no-no as getting rid of the initiators.deny file basically allows anyone else on the same network as OpenFiler to install their own initiator and discover and use your iSCSI targets. However, since in our case this is a private host-only network, there really is no risk in doing this.

You can now proceed to the Storage section under Configuration. Here you’ll click Add Storage so that you can create a new VMFS datastore on each OpenFiler storage volume.

Creating a VM

Finally! We’re now ready to create our first VM inside a VM, because our ESXi servers are themselves VMs under VMware Workstation, don’t cha know? But you already knew that because you’ve been following along diligently since Part 1… right?

What shall our first VM in a VM be? How about good old CentOS Linux 5.5? I just happen to have the ISO file downloaded and ready to go.

First, I had to make my CentOS ISO file visible to one of my ESXi servers. This is easily accomplished by editing the settings of our ESXi virtual server in VMware Workstation, and then adjusting the CD/DVD drive so that it points to our CentOS ISO.

Make sure the ‘Connected’ box is checked!

Next, go to your vSphere Client, right-click one of the ESXi servers or your HA/DRS cluster and select New Virtual Machine. When you get to the part of the wizard that prompts for a datastore, select one of our new shiny OpenFiler iSCSI datastores.

One of the next steps is to select a Guest Operating System. vSphere does not have a ‘CentOS’ option when selecting the operating system in the new VM wizard. But, CentOS by any other name is Red Hat! So I selected Red Hat Enterprise Linux 5 (32-bit).

It’s important to note at this point that while you could create a 64-bit VM inside your virtualized ESXi server, you will not be able to run it. But, you can create and run any supported 32-bit OS.

The reasons for this have to do with the hardware-assisted Virtualization Technology (VT-x) present on the Intel CPU, or the AMD-V Technology if you’re using AMD chips. I’m talking about your physical hardware here. These CPU features allow you to run your 64-bit ESXi servers as VMs, but unfortunately these features are not passed down to the ESXi virtual CPUs. So our ESXi servers see a 64-bit CPU (which is really virtual), but without the hardware-assisted virtualization features. Ergo, we cannot run a 64-bit VM inside our virtualized ESXi servers.

If you did try to run a 64-bit VM inside your virtual ESXi server, you’ll see an unfriendly message like this:

So until VMware decides to virtualize the Intel VT-x and AMD VT features, you’ll  just have to stick to 32-bit VMs inside your virtual ESXi servers!

Here’s the Guest Operating System selection for our (32-bit!) CentOS 5.5 VM:

Now edit this new VM’s properties and change the CD/DVD drive such that it uses the host device. This will allow the VM to see the CentOS ISO file that we previously referenced from the ESXi settings window inside WMware Workstation. (Getting confused yet?)

Make sure the CD/DVD is set to ‘Connect at power on’.

To ensure that your guest boots from the virtual CD/DVD drive, you can hit the Options tab and select the check box under Force BIOS Setup, as shown:

Now when you power on this guest, the PhoenixBIOS Setup Utility will run automatically. From here you can go to the Boot menu and move the CD-ROM Drive to the top of the list as shown:

Hit F10 to save and exit.

If all goes well the CentOS installation program should start, and you can proceed through the installation via the Console tab.

Performance

So now I have running my two ESXi 4.1 servers, vCenter Server 4.1 and the OpenFiler network storage appliance, all under VMware Workstation 7.1.3 on a single laptop.

Obviously, when you’re running this kind of setup you want to avoid unnecessary resource hogs. So close all unneeded applications (Outlook, Word, IE, Messenger, etc.) including any un-required background applications that appear on your taskbar next to the system clock. Kill the Windows Search service if your system is struggling.

My laptop has 4 CPU cores and 8 GB RAM. Let’s take a look at the Performance tab in Task Manager to see how my laptop is holding up (the CentOS guest was installing when I took this screen cap):

As you can see, all CPU cores are active and memory usage is pretty close to max. So you really don’t want to try this with less than 8 GB of physical memory.

A Last Look

Here’ a screen capture showing my VMware Workstation client after all is said and done:

And here is what my vSphere Client looks like:

What’s Next?

The next is up to you! You can install more guest VMs, experiment with vMotion, DRS and HA, or created a distributed virtual switch in vCenter. Or anything else that you feel like testing, learning about or experimenting with!

If you made it all the way through this blog series successfully, congratulations! You now have your very own vSphere-in-a-box that you can take with you wherever you go, and impress your techy friends and your girlfriend with. (Ok, maybe not your girlfriend.)

I hope you found these blogs entertaining and useful. Have fun in great wide world of virtualization!

So if you haven’t read Part 1 none of this is likely to make much sense! So get on down there and read it first! This second part discusses putting the various components together to create a functional vSphere -in-a-box environment.

Putting it all Together

1.  ESXi 4.1

Install your first ESXi server as a VM in VMware Workstation. Choose the ESXi 4.1 ISO file when prompted for an installer disc image. Workstation will automatically detect the ISO file as ‘ESX Server 4’. (Yes, you will be running a VMware hypervisor inside another VMware hypervisor! Pretty neat, huh?)

Make sure that you give this server at least 2 processors, with 1 core per processor. (Or, if you prefer, 1 processor with 2 cores per processor.) The result is the same: you will have 2 processor cores on the ESXi server. (Your physical machine needs at least this many cores, as discussed in Part 1.) Give this VM 4 GB of memory. For now, use Host-Only Networking. Select an LSI Logic SCSI Controller.

The VM’s disk should be SCSI. To save on physical disk space, do not allocate all of the VM’s disk space now (i.e., use thin provisioning).

Before you finish the new VM wizard and start the ESXi installation, customize the hardware in order to add some more network interfaces. By default, there will be only one, which is not so useful. A great thing about virtualizing our ESXi servers is that we can add a total of 10 virtual interfaces per server, without spending any money on real interfaces. This allows us to follow VMware best practices in terms of having separate management, storage and vMotion networks, and play around with different networking combinations. Again, for now, set these to be the default Host-Only (VMnet1) network.

After you finish installing your first ESXi server, give the management interface a static IP address in the ‘Host-Only’ IP address range provided by VMware Workstation and test Web browser access to it. Download and install the vSphere Client on your laptop and use it to attempt to connect to the ESXi server. If everything looks good, repeat this process for your second ESXi server.

2.  vCenter Server 4.1

Next, install your 64-bit Windows platform (Windows XP Pro SP2, Windows Server 2003 SP1 or Windows Server 2008). VMware Workstation uses ‘Easy Install’ when it detects Windows from your ISO file. This allows you to install Windows very quickly and mostly unattended. Give this Windows VM 3 GB of RAM. Once Windows is installed, give it a static IP address (again, use an IP address from the VMware Workstation Host-Only VMnet1 network) and then change this VM’s DVD drive to point to the vCenter 4.1 ISO. Run the vCenter autoinstall program and follow the installation prompts. I just used the built-in SQL Server Express 2005, which is fine for a test environment. When finished, launch the vSphere Client, connect to your new vCenter server and add your two ESXi servers to it.

vCenter will run license-free for 60 days, with all features unlocked. If you already have some VMware licenses purchased, you may want to hold off on installing them until your 60 days are up. I say this because unless you’ve purchased the Enterprise Plus edition of vSphere, as soon as you apply your licenses you will also cancel any extra vSphere/vCenter features that you would have gotten for free for 60 days otherwise! So stick with the free trial license until it expires.

3.  OpenFiler

Finally I installed OpenFiler. Choose ‘Other Linux 2.6.x kernel’ when picking the operating system type in Workstation. I gave my OpenFiler VM 512 MB of memory.

Configuring OpenFiler is perhaps the trickiest and most error-prone bit and could take a whole blog post on its own.  Fortunately, someone has already written one! Follow this link for detailed instructions on how one person set up OpenFiler for a personal vSphere lab: http://www.techhead.co.uk/how-to-configure-openfiler-v23-iscsi-storage-for-use-with-vmware-esx. This link mentions an older version of vSphere, but the information is still valid.

This guy is using physical hardware for his servers, but the fundamental steps are the same. (I’ll admit it took me a couple of tries!) Once OpenFiler is properly configured, you can then use vCenter to discover the LUNs you’ve created, and then create VMFSs on them. Remember to enable the software iSCSI Initiator on each ESXi host!

For now, assign a static IP address from the VMware Workstation VMnet1 network. We’ll change this later.

Now we have all of the pieces of our home lab installed with a basic configuration.

In Part 3 of this series, I’ll go over configuring networking and storage on our ESXi servers and OpenFiler network appliance, and I’ll show you how to run VMs on our virtualized ESXi servers. This means that we will be running VMs inside of VMs !

If you’re ready for that, then swallow the red pill and see how deep the rabbit hole goes…

See you in Part 3!

Why?

Why would you want to do this? As anyone who is striving towards their VCP (VMware Certified Professional) status knows, hands-on experience is essential. Most of us are not lucky (or wealthy) enough have access to a fully equipped vSphere-powered lab at work or home that can be modified, reconfigured and brought down at will. We need a private sandbox to test and learn about vSphere technology, without fear of bringing down lab, or worse, a production environment. If you are an existing VCP, then a sandbox environment is equally as important in order to keep your skills up to date.

There are numerous other blogs on the Web that discuss doing this type of thing using one or more dedicated physical servers. You can save a lot of money by building these machines yourself, termed “whiteboxes”, by buying individual components on eBay or other Web sites and putting them together to create your own home-based lab. This approach costs money and time to acquire and build the hardware, but may feel more “realistic” since you will be working with real hardware as you would in the real-world. Another cost to consider if choosing the hardware approach is the cost of electricity that one or more servers will add to your monthly power bill! (And for us married guys, we have to think about what our significant others will have to say about us turning the basement into a datacenter…)

If you’d like to try the hardware approach, one online source is http://ultimatewhitebox.com. A quick Google search will reveal several others.

This is not the approach I used.

Instead, I built my personal vSphere lab consisting of two VMware ESXi 4.1 servers, one vCenter 4.1 Server, and an iSCSI storage array… all on my Windows 7 laptop. This is way cheaper. Plus, it makes your vSphere lab portable, which is great if you’d like to show your setup to someone or just play with vSphere while on the road. I’ve seen this topic discussed to varying degrees on other Web sites, but I have not come across a thorough blog entry or Web page dedicated to running the latest incarnation of vSphere (4.1 at the time of writing) under VMware Workstation 7. Therefore, here’s my somewhat-thorough attempt at boiling down the essential components and steps required to build your very own portable vSphere-in-a-box.

The Goods

Here’s what you need:

1.  A Laptop (or Desktop) Computer

This is where you need to invest some cash if you don’t already have a somewhat higher-end laptop or desktop. (If you want your lab to be mobile, a laptop is the way to go.) My laptop is a Dell Latitude E6510, 64-bit Windows 7 system with 8 GB RAM and a 500 GB hard drive. My CPU is an Intel Core i5, which has 4 cores.

You must have a physical CPU with native virtualization support (Intel VT-x or AMD-V). This CPU should have 4 cores, but you may get by with 2.

You can make this setup work on a 32-bit physical platform, but it will not perform as well. I’d say that 8 GB of RAM is an absolute minimum if you want to run all parts of your vSphere lab at the same time (otherwise, what’s the point?) If you can get 16 GB, all the better.

2.  VMware Workstation 7 or Higher or VMware Server 2.0 or Higher

In my case, I used Workstation 7.1.3. This is where you will run your ESXi servers, your vCenter server and your iSCSI storage appliance… all as VMs.

VMware Workstation is not free. However, if you are already a VCP you should have received a license for Workstation 7 with your VCP certificate.

If you are not a VCP yet, you can look forward to getting your free copy of Workstation after you pass your exam. In the meantime, you can try it for 30 days for free, and then buy it if you like it. Here is where you can download a trial copy: https://www.vmware.com/tryvmware/?p=vmware-workstation&lp=1

Another option is to use VMware Server 2.0 or higher, which is completely free. You can obtain VMware Server here: http://www.vmware.com/products/server/

Note however that I haven’t tested this setup under VMware Server.

3.  VMware ESXi 4.1

This of course is the VMware hypervisor platform, the core component of any vSphere implementation. In this setup, I installed two ESXi servers as VMs running under VMware Workstation. ESXi 4.1 can be downloaded for free from VMware right here: http://www.vmware.com/products/vsphere-hypervisor/index.html.

Note that while the ESX platform still exists, go with ESXi unless you have a specific need for ESX. VMware has announced that the ESX platform will no longer be supported in future versions, and ESXi is the future and the way to go.

4.  VMware vCenter 4.1

You can run your ESXi servers without vCenter, but in the real world if you have more than one or two ESX servers deployed, you’ll need vCenter along with the great features that it offers, such as vMotion, DRS and HA. Besides, if you are building a lab to prepare for a VCP examination, then you must know how to work with vCenter! Once you add vCenter to your ESXi environment, you now have a vSphere infrastructure!

Note: vCenter is not free. However, you can download a fully-functional 60-day trial right here: http://www.vmware.com/products/vsphere/overview.html

5.  64-bit Windows Platform (XP Pro SP2, 2003 SP1 or 2008)

This is our OS for vCenter 4.1.

vCenter 4.1 must run on a 64-bit Windows platform. This can either be Windows XP Pro SP2, Windows Server 2003 SP1 or Windows Server 2008. Just make sure that the flavour of Windows you select is 64-bit. If you accidently install a 32-bit OS, then great sadness and frustration will ensure as you realize your mistake and have to start over on this step.

The good news is that we can run this server as a virtual machine under VMware Workstation 7. The not-so-good (and obvious) news is that Windows is not free! (I guess that’s good if you work Microsoft.) If the organization you work for has an appropriate Windows license they can assign to you, then you’re set. The kinda-good news is that you can actually download a fully functional trial copy of Windows Server 2008 R2 directly from Microsoft: http://www.microsoft.com/windowsserver2008/en/us/trial-software.aspx

Keep in mind that this will be a trial copy, but it will last you a lengthy 180 days.

The official VMware ESX 4.1 and vCenter Server 4.1 installation best practices can be found here: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1022101

6.  OpenFiler

Ah, OpenFiler. This is going to be your iSCSI SAN-in-a-box (or, SAN-in-a-VM in our case).

OpenFiler is an open-source storage appliance. This is going to be your shared storage between your ESXi servers. Having shared storage will allow you do those nifty vCenter operations such as vMotion, Storage vMotion, DRS (Distributed Resource Scheduler) and HA (High Availability).

OpenFiler, being an open-source product, is completely free. Here’s where you get it: http://www.openfiler.com/

There are of course other free storage options out there, such as FreeNAS, that you can try. At some point I’ll try some of these alternatives and let you know what I think.

That sums up the major pieces you’ll need for your home VMware lab.

In Part 2 of this blog series, I’ll discuss putting these components together to create a working vSphere-in-a-box environment. Stay tuned!

The world of applications has changed, and a modern Application Performance Management (APM) solution needs to be designed for today’s distributed and complex environments. It should enable IT operations and infrastructure professionals to locate and resolve problems without necessarily calling in the development team for assistance. It must be extremely intuitive and far-ranging in its capability, and it should be able to speak the language of business—rather than the language of developers.

In particular, an APM solution must be selected with the five following things in mind.

1. How Simple is it to Install and Use?

Many APM tools are difficult to install, use, and maintain. But within the APM space, there are standard practices available, such as the ability to auto-discover an application’s architecture and quickly alter the mapping of the application’s topology when agile release cycles introduce code changes.  Your APM solution should leverage those best practices and make application management extremely easy for you.

For example, an APM solution should not require more than a series of simple steps in order to be installed.  It should be up, running, and instrumenting the distributed application within hours or even minutes.

2. Can I get the Visibility I need With No Compromise?

A typical modern IT environment consists of a multitude of components and services, all attempting to communicate together in order to perform complex business transactions. And problems or bottlenecks can appear anywhere. If an APM tool can’t visualize potential problems between the services as well as drill deeply into the code and data access calls, it will have blind spots – and you’ll have a hard time diagnosing the root cause of problems. But here’s the hard part – APM must provide this deep visibility into the performance of your production application without introducing its own performance impact. Unfortunately, this is where most APM solutions struggle.

3. How Intelligent is it?

Many web applications can be considered dynamic.  The daily or hour load patterns change and there are both expected and unexpected peaks in demand. The only thing that’s constant is change.

In the past, Operations teams have been limited to setting static performance thresholds so the APM tool knows when to trigger an alert. But this doesn’t work well in dynamic environments and can lead to being flooded by a series of false alerts.  Using static thresholds are akin to setting your car’s cruise control at 100km/hr and not having any ability to override it.

An APM tool that leverages best practices should be able to set those thresholds for you.  This is often called dynamic baselining. This means being able to set baselines for your application by discovering how the application’s performance may vary over specified operating periods.  It observes periodic variations, accounts for them, sets baselines accordingly, and only triggers alerts when it senses those baselines being violated.  A tool that sets dynamic baselines will be highly accurate and eliminate false alarms.

4. Will it be Able to Understand my Business Transactions?

The ability for an APM tool to focus on business transactions is important.  It allows the tool to create a common language between developers and IT operations by representing the transaction, rather than a snippet of code.

A business transaction represents a “user generated” action.  For example, a user might add a book to the check-out cart, or a hiring manager might pull up an online resume.  The APM tool needs to be able to make these actions highly visible to the IT operations team.  This is an essential part of the simplicity and usability of the APM tool: the ability to talk in the language of business.

5.  How does it work in Cloud Environments?

Many businesses are eyeing the cloud as the next important initiative for their critical business applications.  They understand the need for business agility—the need to dramatically ramp up capacity without pouring tons of budget into their physical data center.  In addition, they plan to release new services that are capacity-intensive, and which require the ability to provision hundreds or even thousands of cloud nodes quickly.

An APM solution must be able to understand the volatile environment of the cloud and scale accordingly, being adept enough to monitor nodes as they spin up and spin down. Legacy APM tools will be hopelessly lost in such an ever-shifting terrain; modern APM solutions will be able to monitor thousands of nodes and enable a company to leverage the power of the cloud.

Summary

Managing the performance of your application requires a strategic approach, particularly in a space crowded with multiple application performance solutions boasting very similar messages. To assist your decision, leverage these 6 questions to make sure you get all the info you need to ensure the ongoing health and reliability of your revenue-critical applications.

The presenter was Adrian Cockcroft , the chief cloud architect for Netflix.  He explained how they use the cloud and how they identify and resolve performance problems. He also explained how he uses AppDynamics as his preferred APM solution to monitor his cloud apps. Click here to watch the recording.

Below are some takeaways from the session.

Why did Netflix migrate from a physical data center environment to a cloud environment?

Adrian points out the need for “business agility”—the ability to quickly build and release new products (e.g.  iPhone/iPad movie streaming) without having to dramatically ramp up expensive capacity in their physical data center.  Some new services are capacity intensive, and their ability to provision 100s or 1000s of cloud nodes has reduced their time-to-market with new movies and new products.

Netflix is also experiencing tremendous business growth, with 40% growth Y/Y member growth.  Thus, they also have a need for more capacity to serve this higher demand.  Adrian stated that some of the demand spikes were hard to predict—thus, the need for elastic capacity.

The #2 reason he states is to avoid “undifferentiated heavy lifting.” By using cloud capacity, they no longer have to do the things in the data center that don’t differentiate Netflix from its competitors.  They can focus all of their time and passion on innovation and differentiation.

Note – He doesn’t cite cost-savings as the #1 or #2 reason.

What is different about managing applications in a physical data center vs. a cloud environment?

Quick answer: Everything. Adrian made a pretty bold statement–“Datacenter oriented tools don’t work” in the cloud environment.

There are “more things to manage” by a factor of 10: Whereas the physical data center may have had 40-50 megaservers in the past, the cloud nodes are made up of 1000s of commodity, low-cost servers.

Thus, an individual server means less. Managing application performance and availability by the health of servers (CPU utilization, memory utilization) is no longer a reliable proxy for application health.

There’s also the issue of Dynamic versus Static. No longer is the same set of megaservers serving traffic each and every day.  Cloud servers are easily replaced and 100s of instances can be added or dropped in a minute.  Thus, any concept of management that relied on a static set of servers, connections, agents, etc. is severely outdated.

Adrian also talked about reinventing the Agile Release Process. When new capabilities are ready to be released, you no longer need to update/patch the existing servers.  You now have the option to put the new release binaries on 100s of new cloud instances, send traffic to them, verify that they are performing well, and then take down the 100s of nodes with the old release.

And finally, relationships change. Amazon becomes Netflix’s IT Operations/Infrastructure department and the relationship of App Dev & Architecture for the new cloud apps is with Amazon.

How do APM solutions need to be architected to work in the Amazon Cloud?

Two situations in particular must be handled elegantly in this highly distributed and dynamic environment:

1) The APM solution must be able to monitor 100s or 1000s of cloud nodes from a single management server to provide end-to-end transaction performance metrics and tracing.  If the APM solution can only scale to 20 nodes per management server, you will need multiple consoles and you won’t have a single pane of glass.

2) The APM solution must be able to handle 100s of nodes being provisioned and de-provisioned in real-time.  The performance monitoring, metrics, transaction tracing, service dependency modeling, and deep diagnostics all need to work in this extremely dynamic environment.  Legacy APM solutions that don’t dynamically adapt to infrastructure changes will become useless quickly.

Netflix’s path into the cloud is one that any company stands to learn a lot from. They were one of the first to move their critical applications into the cloud—but many other companies are sure to follow.

The Internet did not Break – Phew!

“The Internet did not break,” said Donn Lee, a senior network engineer at Facebook, to Computerworld near the end of World IPv6 Day. “As we expected, and as we’d hoped for, it was completely a non-event by technical standards … I talked to folks who have call centers and they said they had totally unchanged volume for any normal day. We have not noticed any difference in user tickets or stats that we track on folks using the site.”

It seems that only a few minor glitches were detected.  Others issues were anticipated and the subject of ongoing development.  Many dual-stack clients suffer when a choice of IPv4 and IPv6 paths is available.  Some will prefer an IPv6 path even if it’s slower, experimental, or doesn’t work.  Failing over quickly from IPv6 to IPv4, or from IPv4 to IPv6, can improve connection times.

IPv6 is here to Stay

Native IPv6 web traffic bumped up significantly in relative terms during World IPv6 Day, however it remained just a small fraction of total Internet traffic.  Most traffic growth was in IPv6 tunneling protocols, like free 6in4 tunnels, or Teredo, included with Windows Vista and Windows 7.

Key sites like Facebook, YouTube and Yahoo, and content delivery networks Akamai and Limelight – proclaimed World IPv6 Day a resounding success, and said they would continue support for IPv6 on key sites for developers.

IPv6 Adoption may be slow but it will happen

In the weeks since World IPv6 Day, we’ve come to share the view that “It’s going to be a long hard slog to IPv6-Land.“  Expect pressure to support IPv6 to build over time, from different perspectives.

You will need to support IPv6 within the 3-5 year lifecycle of anything you buy today.

• Make support for IPv6 a mandatory requirement for all new equipment purchases.
• Start validating claims of IPv6 support, and test what support is present on existing deployed equipment.
• Equipment vendors, expect RFPs to get more specific on IPv6 support, and backed up with real testing. Empowered can help you expand your IPv6 functionality, conformance and performance testing.

At least three key considerations may accelerate your timeline to support IPv6:

• Do you have a mandate to support IPv6?
  The US Government is the clearest example of where a mandate to support IPv6 has been imposed by the executive suite.  There may come a day when an IPv6 mandate comes down in your shop.  Being prepared is better.
• Is geographic expansion and connectivity part of your business plan?
  In some parts of the world, especially Asia, publicly-addressable IPv4 addresses are in short supply, if you can get them at all.  By contrast, IPv6 addresses are easy to get, there’s broader adoption of IPv6, and your business partners and customers are more likely to drive you to support IPv6.
• Do you need more (reachable) address space?
  Supporting the growth of virtualization, cloud computing – not to mention smartphones, tablets and other mobile devices – will put pressure on your address space.  There are limits to how much Network Address Translation (NAT) and private network addressing approaches can help.  Moreover, certain applications – VoIP, peer-to-peer, gaming, and many types of servers – must jump through hoops to work with NAT.  Let’s set aside the point that some new applications – RFID, sensors, Smart Grids, 4G/LTE – will scale the number of connected devices so much that IPv6 will be required from Day 1.

Don’t Wait, Get Started Now

Here’s some pragmatic advice for our customers and friends:

Start thinking about supporting IPv6 on external-facing websites and services.
Beyond obtaining IPv6 connectivity from your ISP, there are other considerations that will affect your longer-term migration to support IPv6, both externally and internally:

• Review how you configure and track IP Addresses.  Managing IPv4 addresses manually with spreadsheets was possible, but adding IPv6 to the mix makes it impractical, at best.
• Ensure your DNS server architecture supports both IPv4 (A Records) and IPv6 (AAAA Records), and that DCHPv6 servers provide other configuration information, like DNS and Time server addresses, in addition to IPv6 addressing.
• There’s a strong business case for solutions to automate DNS, DHCP and IP Address Management, that support other initiatives like VoIP, redundancy, and more. These tools can be the foundation of an effective IPv6 migration strategy.

Start implementing some basic IPv6-aware Security.
Just because you haven’t deployed IPv6 doesn’t mean it’s not running on your network. Newer Operating Systems, and devices, are IPv6-capable, if not IPv6-enabled by default. Here’s a couple items to consider, for starters:

• Turning off IPv6 on every client is not a practical alternative.  Configuration control at the desktop is never absolute.  Mistakes can create significant vulnerabilities.
• Configure your firewalls to block protocol 41 (IPv6 encapsulated with IPv4) and more generally, defend against IPv6 tunneling methods.
• Configure network devices to filter rogue RA and rogue DHCPv6 server traffic, per RFC 6104 and 6105.  Network configuration management tools can help automate applying and managing such changes.

Make IPv6 part of your network planning cycle

In the end, IPv6 should become part of your network planning cycle … Very soon.

It probably won’t be forcing its way onto most people’s networks in 2011 and this is good news because you can start now, start slowly, get prepared, manage risks and be ready.  Being ready is always a good thing!

And of course, Empowered Networks is here to help.  We have the professional services, testing and management solutions you need to start, or accelerate, your IPv6 planning, testing and migration activities.  If you’d like to discuss your needs with one of our skilled engineers, contact us to set up a time to chat – no sales pitch, just a discussion.

RIM knew that lightning fast 802.11n Wi-Fi connectivity at Blackberry World 2011 was essential to showcasing the power and agility of their new PlayBook.  For many, it was the first chance to see the PlayBook up close and personal.  Additionally, everyone – 5000+ attendees, media, registration, event staging and more – had some kind of mobile device that needed tethering to Wi-Fi and the Internet … if not two or more devices.

The event space for BlackBerry World was enormous – the Orlando World Center Marriott Resort & Convention Centre has over 450,000 square feet of event space on a single floor – more than 10 acres of space.  At last year’s event, Wi-Fi was a challenge, and a bottleneck.  Some 700 concurrent users got lack-lustre performance from a “typical” Wi-Fi network using 2-radio access points.

Event organizers at RIM turned to Empowered and Xirrus to deliver a high performance 802.11n Wi-Fi network, with support for all key Wi-Fi standards (802.11 a/b/g/n) and devices.  Empowered’s team, led by Matt Rose, performed an Active Site Survey of the facilities in advance of the event, and planned coverage, capacity and channel management to support a minimum of 2000 concurrent users – RIM’s stated requirements.

Onsite in Orlando, the design goals changed – the organizers now demanded support for 10,000 concurrent Wi-Fi users – a five-fold increase!  Undaunted, the team was able to stage, deploy, commission and install the Wi-Fi network, and to connect it to the backend network supporting the show in just 2 days.

A key enabler was that only 40 devices – mostly Xirrus XN8 High Performance Wi-Fi Arrays, deployed on portable, rapid deployment kits – were required to provide a dense blanket of lightning-fast Wi-Fi connectivity.

Once deployed, the team turned to managing the network and service performance, and optimizing coverage and capacity to satisfy changing loads and conditions with the ebb and flow of pre-show briefings for BlackBerry Alliance Partners, RIM investors and media.

And the Wi-Fi network delivered.  Live PlayBook demos conducted by RIM co-CEOs Jim Balsillie and Mike Lazaridis were flawless, with HD Video streamed over the Wi-Fi.  Any challenge that RIM threw our way was handled with ease.  The only puzzling part was some heavy Wi-Fi traffic we detected after hours Monday evening.

Then, at the Tuesday morning keynote address and kickoff, the proverbial other shoe dropped.  Mike Lazaridis announced that every attendee – some 5000 guests, media and partners – would receive a free PlayBook.  Our team was taken by surprise, as much as anyone.

Now the new design goals for the Wi-Fi made sense.  So did the mysterious Wi-Fi traffic after hours Monday – RIM staff were pre-loading the first of the new PlayBooks.  Since activating a PlayBook triggers a software update approaching 290MB, activating some 5000+ PlayBooks was a huge concern for the event organizers, who were sworn to secrecy and couldn’t tell us in advance.

That concern was easily addressed.  The Empowered/Xirrus team quickly deployed 5 additional Wi-Fi Arrays to the lobby area where the new PlayBooks were distributed, and tuned the Wi-Fi network to optimize download performance, and support the new capacity demands.

High Performance Wi-Fi saved the day, and all went off without a hitch.

Kudos to Matt Rose and his team for taking these unheard of demands in stride, and kudos to the Xirrus product line that truly lived up to its reputation for over-the-top performance.

I suspect that this has a lot to do with the organic way that virtualization has grown within organizations over the past few years, growing from a mere mention on CIOs annual priorities list published by Gartner 4 years ago, to claiming the number 1 spot in this past year’s list.  A good deal of this meteoric rise has to do with the initial economic benefits of virtualization.  The typical IT organization has implemented virtualization initially as a better mousetrap for traditional application consolidation -  the practice of hosting more than one application on a common physical platform.  With virtualization, organizations get application consolidation without the conflict and complication of administration associated with traditional methods.  Consolidation by any method of course allows organizations to leverage their physical investment and achieve operational efficiencies in facilities and HVAC.

Then of course there is the way in which organizations have approached adoption of virtualization, with many testing the waters first in their non-production environments before venturing into production.  As we all know, non-production environments seldom have the same level of operational scrutiny and oversight and their production counterparts, allowing virtualization to often-times occur quickly with little regard to application performance, and of course without those pesky application owners in the business looking over your shoulder.

Contributing to these factors is the fear, uncertainty, and doubt placed on IT organizations by the businesses who want to ensure that their applications run without performance impacts. A widely held perception is that applications that run purely on physical infrastructure outperform virtualized applications that by nature share physical infrastructure.  This same perception has also widely contributed to the over-provisioning of both physical and virtual infrastructure, which moves in total opposition to what an organization is trying to achieve in the first place with virtualization; that is the efficient maximal use of an organizations physical compute assets.

Having said this, it is not difficult to come to the conclusion that much of the success to be had in the P2V conversion process is related to how well application performance can be measured to provide assurance back to the application owners and users that their applications are performing within the available compute resources.  Establishing an application performance baseline will help you to understand how the application and it’s supporting components should behave under ideal conditions, and will provide key insight down the road when optimizing the application for compute resources and performance.

This last point is especially relevant as most organizations are very immature when it comes to capacity planning, using very simplistic methods such as spreadsheets to do capacity planning, and in most cases simply referring to the vendor’s recommendations without directly measuring or verifying actual capacity.  Being able to compare an application’s current performance to an ideal baseline is quite effective when rightsizing compute resources as any impacts to performance as a result of the rightsizing effort will be immediately apparent and can be adjusted accordingly.

So where does a good P2V initiative begin?  Let’s take a look at the people, process, and tools that can play a key part:

People

As with any other IT initiative, it is important to first identify the key stakeholders for what you’re trying to accomplish.  In the case of P2V, this is obviously the application owners themselves, but also may include business stakeholders, and IT operations.  The goal here is twofold:

1. Communication of the strategic importance of your virtualization initiative in order for the organization to achieve greater operational and economic efficiencies with compute resources; and

2. Enlist stakeholder feedback that can be used to establish key metrics to indicate progress and success of the virtual conversion of the application in question.  Sharing your P2V process with your stakeholders will help to instill confidence that there is a measurable plan in place that will ensure success of the initiative.

Process

Almost all organizations can benefit from initially documenting the process that they intend to use during P2V.  It can be simple or complex, depending on the requirements of the business, regulatory and operational considerations, etc..  Documenting the P2V flow will help you to think through the steps required to achieve success.  It is important to document specific entry and exit criteria for each step of your process, along with the specific stated objectives of each phase.  Once you have done this, you may have something similar to the following:

Of course your process might look different, but the important part is to be able to have it documented in order to be able to effectively establish when you have achieved success and reached the end of your P2V process.

Tools

One of the most frequently overlooked parts of the P2V process is the importance of managing applications as a system instead of a simple (often spreadsheet-based) inventory.  What I mean by this is the ability of an organization to map the component parts of applications (i.e. Web servers, application servers, supporting databases, etc.) and their dependencies, and be able to measure the results of transactional performance between the component parts that make up the application.  There are many tools available to an organization to accomplish this available from the big 5 software management vendors, but it’s important to ensure that the tool in question can also interface with the CloudOS API (i.e. BlueStripe’s Factfinder that uses the vSphere API).  With the user’s experience being related to the sum of the performance of all of the application components, if you are unable to measure transactional performance between various components, such as the time it takes for a database query to execute and return results to the application, you’re constantly going to get stuck with hearing users and application owners whine that “virtualization makes the application slow”.  The truth is that the application components themselves seldom impact application performance and most often times it is a combination of factors that can contribute to the overall perception that an application is under performing.  Because of the dynamic nature of many modern CloudOS’ such as vSphere, it is even more important that the discovery and dependency mapping of applications is an automated process lest the process become too time consuming or inaccurate to be of any practical use.

Summary

Given the right amount of planning, the physical to virtual conversion process need not be a daunting task.  Should you be having difficulty and need a helping hand, Empowered offers several products and services to assist organizations in accelerating their P2V process to help get you past your virtual stall and achieve your organizations goal of getting to the cloud that much quicker.

Now let’s move ahead to today’s data center, where companies are moving to virtualization. In our new virtual environment, suddenly we can create new Virtual Machines, Virtual Networks, and Virtual Storage in a fraction of the time it used to take us to do the equivalent in the physical world.  We can now move these around with the click of a mouse, or in a fully automated environment, create and then close down that virtual system even more quickly.  In our new reality, all we need to know is that resources are available (CPU, disk, memory) and voila, we are up and running.  At first glance, life sure got easier.

However, what people often miss is the impact of virtualization on our oh-so-critical management systems. Some people may feel that all that ITSM stuff goes away, but I have seen over and over again how not only is it more important than it ever was, but in fact virtualization has now become the ITSM Gap Amplifier.

Why?

There are inherent controls in a traditional environment – controls that are tied to the  more cumbersome process of acquiring new hardware and the requirement to attach resources to physical devices. For example, it used to take weeks and often months to provision a new service.  Machines would be ordered and configured, other components attached, and the very act of taking this much time, actually touching a machine, and testing out all of the dependencies before introducing it, introduced an element of control.

In the new virtual world, configuring and deploying systems got easier and faster.  Suddenly there is often no need to purchase any new hardware, almost any existing server can be used, a VM can be spun up in minutes – so easy, dare we say too easy?  At the same time, dependencies don’t go away, and with the ability to move VMs from machine to machine with a mouse click, and potentially many VMs moving at the same time, the probability that something will go wrong goes up, often way up. Its simple math – more activity means more risk.  Virtualization makes things move faster.  With the ability to accelerate delivery of services by an order of magnitude almost overnight, you can see how the risk increases exponentially.

Secondly this ability to move quickly and easily and with less expense, means you inevitably end up managing more environments than ever before. The hypervisor still needs occasional patching and has device drivers that may need to change as the underlying hardware changes. So the very things that we love about virtualization, the ease of deployment and the speed at which we can meet the needs of our business users are also the very things that make managing this environment more difficult.  More activity without processes that can adapt to this new world order means that the gaps that cause problems get bigger too.  More activity without strong management tools to provide visibility also contributes to bigger gaps.

That’s why we call virtualization the ITSM Gap Amplifier.  The very things that are valued in a virtualized environment actually bring an extra layer of complexity to your IT operations, requiring an extra layer of knowledge, an extra layer of monitoring tools, management tools and most of all some process changes so that controlled model called your management strategy can operate seamlessly.

So our takeaway is this – before you get too far down the path of virtualization, find out where you stand relative to possible gaps in your virtualized environment.  Making changes now will stop virtualization from becoming a Gap Amplifier.

Mike Crabtree