Jeremy's Life

6/15/24

June, 2024

It has been a few years since my last post. I'd like to make a few notes here today.

Today, I learned how to RDP into my Entra joined laptop with an Entra account, by checking the highlighted checkbox below (documentation).

Earlier this month I learned that the Firefox web browser ignores PDF printing restrictions, so that's a way to get around the issue if you think you need to print such a file.

10/1/17

Port security with 802.1x using HP ProCurve (Aruba) switches and Windows NPS

So I'm learning about 802.1x authentication as a means of securing access to the LAN of a client who wants to be PCI compliant (and one part of PCI compliance is securing publicly accessible network jacks).

The environment consists of:

Windows 10 domain-joined PCs (and some not domain-joined)
VoIP phones
Receipt printers
Letter printers
Barcode printers
Credit card readers
Windows Server running NPS for RADIUS authentication

That said, these screenshots and notes will all come out of my home lab. Here's a link to my syntax source.

Installed NPS on a test server and told it that my test switch would be a RADIUS client. Created a wildcard "Connection Request Policy" to accept the MAC address of a test Windows PC:

Told the switch what RADIUS server to use:

radius-server host 192.168.100.5 key "xxxxxx"

Configured ports 3 and 4 on the switch to require RADIUS authentication - and if RADIUS authentication doesn't happen for any reason, to drop a connected device onto VLAN 3:

aaa authentication port-access eap-radius

aaa port-access authenticator 3-4 unauth-vid 3
aaa port-access authenticator 3-4 client-limit 2 #allows up to two devices per port
aaa port-access authenticator active

The switch offers both chap-radius and eap-radius...these are simply frameworks for transporting authentication to the server...for chap-radius to work, the user/computer password in Active Directory needs to be stored with reversible encryption (not the default), so eap-radius seems preferable.

Next, I started the "Wired Autoconfig" service on the test Windows PC (you can do this via group policy in a production environment)...this service offers credentials to the switch for authentication by RADIUS...I learned (and you can see this in a Wireshark packet capture!) that when a computer boots up it'll try to authenticate with its computer account...and then after a user logs in, it re-authenticates with the logged-in user account. So cool, huh?

So far, no certificate was needed on the NPS server. However the Windows PC complained on its NIC that "Authentication Failed" (but it was still given the "authorized" VLAN due to the Connection Request Policy that I'd created).

So, how to fix this message? Created a Network Policy with an authentication method of PEAP:

Now the NIC was happy:

...and bear in mind that this is without any client-side certificates. I like that. Keeps it simple.

Next: get my VoIP phone working. However, when I plugged it into the switch, the switch never said a thing to the RADIUS server, instead it just dropped the phone into my unauthenticated VLAN (3).

The solution is to enable MAC authentication in the switch as a fall-back mechanism for when devices don't do 802.1x authentication. (Syntax source).

The first step of this was to disable something I'd done earlier:

no aaa port-access authenticator 3-4 unauth-vid

...then replace it with this:

aaa port-access mac-based 3-4 unauth-vid 3

This is because if 802.1x authentication fails on switch ports 3 or 4, then MAC address authentication will be tried...and if that fails, then the device will be dumped into VLAN 3 (my DMZ).

Let's support up to 2 devices per-port on ports 3 or 4 (e.g. a phone with a computer behind it):

aaa port-access mac-based 3-4 addr-limit 2

Now let's enable MAC address authentication for ports 3 and 4:

aaa port-access mac-based 3-4

Now we'll view status:

A VoIP phone was on port 4...disconnected / reconnected the phone's patch cable....it got a new IP from VLAN 3 and rebooted.

This was good. Now, how do I authorize it in NPS?

I'm using a wildcard MAC address here, and I got the format of the MAC address from a Wireshark packet capture:

So at this point I have a PC that authenticates via its computer account and a phone that authenticates via its MAC address. Yay!

8/5/17

A Powershell note

How to copy the Desktop, Documents, and IE Favorites folders from an old terminal server to a new, when usernames were slightly different between the systems.

-------------------------------------------------------------

Import-Csv mylist.txt | ForEach-Object {

robocopy \\rwterminal\c$\users\$($_.Old)\Desktop c:\users\$($_.New)\Desktop /E /DCOPY:DAT
robocopy \\rwterminal\c$\users\$($_.Old)\Favorites c:\users\$($_.New)\Favorites /E /DCOPY:DAT
robocopy \\rwterminal\c$\users\$($_.Old)\Documents c:\users\$($_.New)\Documents /E /DCOPY:DAT

}

-------------------------------------------------------------

mylist.txt contained usernames like this:

New,Old
christopher,christopherb

sally,sallyh

4/6/17

Yes, Windows Server Backup on Server 2016 will flush Exchange 2010 log files in a Hyper-V guest

The title says it all.

4/4/17

One way to audit activity on a print server

When managing a print server with a large number of printers, its nice to know which ones are being used and which are idle/abandoned.

To audit printer activity on a print server:

1. Enable the appropriate event log -> right-click on it and choose “Enable”:

2. Wait a while for events to accumulate in the log.

3. Now, run a Powershell script:

$MyFilter = @{LogName = "Microsoft-Windows-PrintService/Operational";ID=307; StartTime = (Get-Date).AddHours(-24)}

$Events = Get-WinEvent -FilterHashTable $MyFilter

[System.Collections.ArrayList]$PrintJobs = @()

ForEach ($IndividualEvent in $Events)

{[void]$PrintJobs.Add([regex]::match($IndividualEvent.Message,'(?<=printed on ).*?(?= through port)').Groups[0].Value)}

"Used printers with number of print jobs:"

$PrintJobs | Group-object | Select Name, Count | Sort Name

$UsedPrinters = $PrintJobs | Group-Object | Select Name -Unique | Sort Name

$AllPrinters = Get-Printer | Select Name | Sort Name

"."

"Idle printers:"

Compare-Object $AllPrinters $UsedPrinters -Property "Name" -PassThru | Select Name

…and then you get a nifty little report that shows which shared printers (on this server) were used and which were idle during the audited period of time! 😊

12/10/16

Missing the "attributes" tab on user and computer objects in AD Users & Computers

If you're in Active Directory Users & Computers, you've enabled "Advanced Features" from the View menu and you're still unable to see the "Attributes" tab, here's the fix:

-Open ADSIedit and connect to the Configuration
-expand (click on the +) CN=configuration, DC=, DC=local
-expand CD=DisplaySpecifiers
-click on CN=your language. The language # can be found on http://support.microsoft.com/kb/324097 (for example US English is 409, so CN=409)
-in the right hand window locate CN=User-Display right click on it and choose properties.
-Locate AdminPropertyPages, highlight it and click “edit” and add the line 11,{c7436f12-a27f-4cab-aaca-2bd27ed1b773}
-in the right hand window locate CN=Computer-Display right click on it and choose properties.
-Locate AdminPropertyPages, highlight it and click “edit” and add the line 12,{c7436f12-a27f-4cab-aaca-2bd27ed1b773}
-in the right hand window locate CN= Default-Display right click on it and choose properties.
-Locate AdminPropertyPages, highlight it and click “edit” and add the line 4,{c7436f12-a27f-4cab-aaca-2bd27ed1b773}

Source: https://www.experts-exchange.com/questions/27727599/Windows-SBS-2011-Missing-Attributes-tab-in-Active-Directory.html

11/13/16

Reinstalling DFS after upgrading Server 2008 R2 to 2012 R2

This weekend I used Disk2VHD to P2V a file server running Server 2008 R2. Afterward, I wanted to install the Hyper-V Integration Components...which required SP1 for the OS...which required a reinstall of the OS...did all that...then upgraded to Server 2012 R2...and found that DFS shares were inaccessible. These are my notes from troubleshooting.

1. Well, you're not supposed to re-install the OS of a DFS server (reference).

2. So I deleted the DFS namespaces and uninstalled DFS.

3. I couldn't re-install DFS. Server Manager said "The request to add or remove features on the specified server failed. The operation cannot be completed, because the server that you specified requires a restart."

4. Side note: Server Manager also said "Online- Cannot get role and feature data".

4a. Event Viewer contained the details - it seems that several old event log entries were corrupt or something and Server Manager couldn't enumerate them.

4b. So following instructions here, I backed up and then deleted the corresponding registry keys under HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Channels.

5. That made Server Manager happy, but I still couldn't install DFS. So started reading the C:\Windows\Logs\CBS\CBS.log file...and found this little gem:

"Error...EventAITrace:Channel Microsoft-Windows-DFSN-ServerFilter/Analytic is declared by an existing provider DFSN-Filter{{564af62f-e581-4998-9991-c4bc18178815}}"

...an orphaned registry key still needed to be deleted...after that, DFS installed successfully.

6. After creating a DFS namespace, I was puzzled as to why it wasn't visible in \\domain.local...the fix was to add the domain controllers to the list of namespace servers:

7. Lastly, needed to search through all the group policy settings that my predecessor had setup and find where he had mapped network drive letters to the DFS shares so I could update them...fortunately, there's a handy GPO search function.

10/21/16

DHCP and secure dynamic for DNS

Consider this. You support a network where:

DNS is Active-Directory integrated and requires secure dynamic updates
IPAM has been implemented to track IP address utilization
DHCP is set to update DNS records automatically

However, the DHCP server never updates DNS records. Domain members appear in DNS, but not the smart phones seen in the DHCP lease table. This limits the info available in IPAM.

The solution is to give the DHCP server a specific username/password to use when registering DNS records. Further, you want this dedicated user account to be a member of the DnsUpdateProxy group so that if the DHCP server creates a record, the workstation can come along later and update that record itself (I ran into this once in a DirectAccess manage-out scenario).

https://technet.microsoft.com/en-us/library/dd334715(v=ws.10).aspx

9/27/16

Configuring Web Application Proxy in Server 2012 R2 for on-premise Exchange 2010 and RDS Gateway using a single public IP address

Several clients run an on-premise Exchange server and would benefit from having a RDS Gateway to simplify their firewall rules and improve their security posture. It also comes in handy when the CEO wants to remote into her office PC from her phone.

1. If you install the AD FS role on a domain controller, follow the steps here first: https://support.microsoft.com/en-us/kb/2832204 to grant the "Log on as a service" right to "NT SERVICE\ALL SERVICES"

2. Follow instructions at http://blog.ittoby.com/2014/04/web-application-proxy-server-in-2012-r2.html

3. Add "Authenticated Users" to the "Pre-Windows 2000 Compatible Access" group in Active Directory (reference).

4. Configure the Web Application Proxy to support *all* Exchange services: http://searchexchange.techtarget.com/tip/How-to-configure-Active-Directory-to-publish-Exchange-to-the-Internet

5. Point your firewall's NAT policy to the WAP server instead of the Exchange server.

6. Allow end-users to change expired passwords by running this on the AD FS server (source):

Enable-AdfsEndpoint "/adfs/portal/updatepassword/"

Set-AdfsEndpoint "/adfs/portal/updatepassword/" -Proxy:$true

Restart-Service AdfsSrv -Force

7. If you support Outlook 2011 clients on a Mac, you'll need to add a wildcard certificate binding per https://blogs.technet.microsoft.com/applicationproxyblog/2014/06/19/how-to-support-non-sni-capable-clients-with-web-application-proxy-and-ad-fs-2012-r2/ because Outlook 2011 doesn't support the Server Name Indicator (SNI) extension of TLS SSL.

Note: the command below seems to need to be run from a CMD prompt, not Powershell.

Example:
netsh http add sslcert ipport=0.0.0.0:443 certhash=2ff79b325a0c4aa4eb5cb04b1330ff78750a1639 appid={f955c070-e044-456c-ac00-e9e4275b3f04}

Update after configuring ADFS in Server 2016 at a different place:

1) I was getting tons and tons of errors in the Application Event log - ID 28005 - "An exception occurred while enqueueing a message in the target queue.  Error:15404, State: 19. Could not obtain information about Windows NT group/user XXX\XXX, error code 0x5.

This was resolved by right-clicking the OU which contains my ADFS service account and delegating the "read all attributes" permission to "Authenticated Users".  (reference)

2) Windows Authentication wasn't working.  This was resolved by deleting the internal CNAME DNS record for adfs.domain.com and replacing it with an A record. (reference)

3) An invaluable list of ADFS customizations for Server 2016 (including how to enable changing of expired passwords)

How to deal with an expired application certificate:

So after you add a new application certificate to both the target host (e.g. Exchange server) and your Web Application Proxy server, you need to either remove and re-add the associated application in the Web Application Proxy GUI, or else use Powershell to update the associated certificate.  Here's an example of the Powershell syntax:

Get-WebApplication{ProxyApplication Autodiscover | Set-WebApplicationProxyApplication -ExternalCertificateThumbprint 50F089C9DF06EC5EC48C2110BDC3AE28BAA73543

4/20/16

Cleaning up a mish-mash of domain controllers

After seizing all the FSMO roles on a single domain controller and deleting its replication partners in AD Sites & Services, I tried to join a PC to the domain and got this error:

After verifying that the DNS server settings were correct, rebooting, and verifying that the SYSVOL share was present on the domain controller, I ran dcdiag on the domain controller and saw this:

The DNS console was unusable:

The DNS event log was littered with red ink (so to speak):

So I followed instructions from https://support.microsoft.com/en-us/kb/2751452 and it worked! Yay!

These were the instructions:

Stop the KDC service.
Run the following command with elevated rights: netdom resetpwd /server: /userd: /passwordd:*
It will prompt for the password of the Domain Admin account that you used, enter that.
Once the command executes, reboot the server.
DNS zones should load now.

10/10/15

Windows 10 - initial research

Where do I start?!
Gartner's "Ten Things You Need to Know About Windows 10 for a Successful PC Deployment" for IT managers or executives (PDF).

The New User Experience with Windows 10 from Microsoft Ignite in May of 2015 (1 hour video).

Minasi's Guide to Managing Windows 10: New Windows, New Tools (a hilarious 1 hour video covering a lot of ground for IT Pros)

If you have a large amount of unstructured time in your life, you may enjoy perusing the entire collection of Windows 10 videos from the May, 2015 Microsoft Ignite conference.

You've upgraded to 10, hard drive crashed, and are now reinstalling. Where's the product key?!
According to this report, when Windows 7 or 8 is upgraded to 10, your new product key is kept in the Windows Store account associated with your Windows account, so if a complete reinstall is needed, you won't be prompted for a product key.

What's in it for the end user?
Coming from Windows 7, this is a downgrade. The UI is ugly and WindowsBlinds isn't available yet. Also, it has bugs...for example, as of 10/10/2015 the "Enable shortcut underlines" accessibility setting in Control Panel doesn't stay "On" when you try to change it from "Off".

Coming from Windows 8.1, this is an upgrade. You get the "Start" menu back.

Microsoft's decision to abandon the UI options in Windows 7 (transparency, detailed and colorful icons) will result in continued loss of market share to Apple in the consumer space.

What's in it for the IT manager?

Not much...yet. Windows 10 is laying a foundation for future growth and as such, it's not very pretty at the moment. You could say that it's basically Windows 8.1 with a better engine, a Start Menu, and better integration with Microsoft's $16,000,000,000 Azure cloud investment. Someday (I hope!) Windows 10 will be really nice. Microsoft's vision of the future is that Azure + Windows 10 will offer the most secure and manageable tools for your company's digital assets.

How do you deploy this to 50 computers in a small business?

Here's a TechNet write-up explaining how to use the Microsoft Deployment Toolkit for that.

Alternatively, you could just download the ISO, mount and share it on your file server, then walk to each desktop and run an in-place upgrade that way.

9/24/15

EMC 2010: The attempt to connect to using "Kerberos" authentication failed.

Last weekend I couldn't launch the EMC on an SBS 2011 server:

The solution? The MSExchangePowerShellAppPool wasn't running. Thank you, TechNet blog.

Invalid DNS server prevents RDP connections

A headless Hyper-V core server in my lab wouldn't accept RDP connections because it couldn't authenticate the incoming username due to an invalid DNS configuration on its NIC.

Connecting a monitor and keyboard would've been too easy

A remote Powershell session (authenticated as the server's local administrator) worked...

Enter-PSSession -ComputerName COMPUTER -Credential USER

...but sconfig wouldn't run.

Fear not, you can see your entire network configuration by running netsh dump. To set the primary DNS server to 1.1.1.1 on my Server 2012 R2 host with a network interface called "vEthernet (External_Internal)":

netsh interface ip set dnsservers "vEthernet (External_Internal)" static 1.1.1.1 primary

9/12/15

When your domain controller has been offline for > 60 days

When trying to demote a domain controller which had been powered off for several months, I ran into this error:

Also, the new domain controller that I'd just spun up couldn't access the NETLOGON share.

DCDIAG had this warning:

The DFS replication event log had this warning, with instructions that don't work:

Thankfully, this blog post explained what to do (all on one line):

wmic.exe /namespace:\\root\microsoftdfs path DfsrMachineConfig set MaxOfflineTimeInDays=200

4/18/15

Auditing changed / deleted files on Windows 2008 R2, 2012, or 2012 R2

What

This is the story of using Powershell via Scheduled Task to audit files that are remotely modified, deleted, renamed, or moved on a file server running Microsoft Windows Server 2008 R2, 2012, or 2012 R2. It's been tested via Windows 7 and 2012 R2.

Why

Auditing (metaphorically) positions your finger closer to the pulse of a file server, helps assist users who've misplaced files, and serves up the answer when a manager asks "Who deleted it?"

There are commercial auditing solutions (Lepide, NetWrix). That said, this project helped me:

Learn about Powershell
Build awareness of security auditing on Windows servers
Fill my evenings for 4 months after putting the baby to bed

4/16/15

Windows ADK

Today I was so frustrated because I couldn't get the Windows ADK to make a catalog of a Windows 8.1 install.wim - it said "Catalog creation failed to complete. This 64-bit version of Windows SIM can only create catalogs for 64-bit Windows images."

The solution was to install the Windows ADK on Windows 8.1 (I'd been trying it on 2012 R2 and 7).

1/24/15

Upgrading a PowerEdge T410 from 2008 R2 to Hyper-V Server 2012 R2

This was similar to an upgrade last August (convert two RAID 1 arrays into one RAID 10 array).

Here's what I learned:

#1: Always use the /R:1 switch with ROBOCOPY.  Failure to do so today cost me 1.25 hours.

#2: This server has neither iDrac Express nor iDrac Enterprise, it also lacks the Lifecycle Controller and Unified BIOS of newer Dell servers. So, downloaded the bootable Systems Build and Update Utility (SBUU)...and got stuck...so downloaded the 10GB Dell Server Update Utility and ran it under Windows 2008 R2 to apply firmware updates for the RAID controller, HDs, NICs, etc.

#3: On Hyper-V Server 2012 R2, to see what devices are missing a driver, use the free Portlock Device Manager.

#4: PNPUTIL -i -a was useless. In Hyper-V Server 2012 R2, if drivers aren't available through Windows Update, forget about it.

#5: You can install the Dell Server Manager 7.4 on Hyper-V Server 2012 R2.
       Installation: msiexec /i SysMgmtx64.msi
       Web access: https://hostname:1311/OMSALogin

#6: The Dell Server Administrator (GUI and CLI) did not show current values for the RAID rebuilding process.  Restarting the "DSM SA Data Manager" service in Windows refreshed it.  omreport storage vdisk is the command to show RAID rebuild progress.

--------------------------------------------------------------------------------------------

Lastly, wanted to convert my original dynamic VHDs to fixed VHDXs for better performance thru reduced fragmentation (link, link).

First, tried New-VHD -Path c:\file.vhdx -Fixed -SourceDisk 3 -SizeBytes 50GB, but that doesn't work. Here's what does:

Mount-VHD -Path original.vhd -Readonly (the host OS acts like it's an attached drive now)
Get-Disk (to identify what number is assigned to the VHD mentioned above)
New-VHD -Path new.vhdx -Fixed -SourceDisk 3 (duplicates the original into a fixed VHDX)
Dismount-VHD original.vhd

Optionally reduce the size of the VHDX file:

Mount-VHD new.vhdx
DISKPART
LIST VOLUME (to see which volume number to select)
SELECT VOLUME 6 (for example)
SHRINK QUERYMAX (to see how much empty space there is (reference))
SHRINK DESIRED=50000 (sets the partition size to 50GB in this example)
EXIT

Dismount-VHD new.vhdx

Resize-VHD new.vhdx -ToMinimumSize

Replace the old VHD with the shiny new VHDX:

Get-VMHardDiskDrive -VMName SBS (optional)

Remove-VMHardDiskDrive -VMName SBS -ControllerLocation 0 -ControllerNumber 0 -ControllerType IDE
Add-VMHardDiskDrive -VMName SBS -ControllerLocation 0 -ControllerNumber 0 -ControllerType IDE -Path new.vhdx
Start-VM SBS

So, you ask, did I achieve the goal of improved disk I/O? Yes, but don't die from laughter when you see the numbers - these are SATA disks on a Perc H200 RAID controller (write cache is disabled).

For what it's worth, moving from RAID 1 to RAID 10 nearly doubled R/W performance of the Hyper-V host. The graphs below are from inside a Hyper-V guest.

To benchmark disk performance, I used DiskSpd with a sample set of parameters from the download page: diskspd.exe -b8K -d30 -h -L -o2 -t4 -r -w30 -c50M c:\io.dat

...and I just discovered Crystal DiskMark! Post-upgrade numbers:

12/14/14

Notes about setting up AD FS for Hosted Exchange in Office 365

I've been learning about Active Directory Federation for Office 365's Hosted Exchange. Here are some links and notes about the process that I don't want to forget.

Where to sign-up: Exchange Online account ($4/month/mailbox).

Instructions on how to configure DirSync to sync a single OU.

The excellent tutorial that I followed to setup AD FS w/ O365.

A discussion of the 3 sign-in models for Office 365.

A discussion of whether to use SQL or WID for ADFS authentication to O365.

Should you use O365? Yes!

How to test AD FS functionality: https://adfs.trimideas.com/adfs/ls/idpinitiatedsignon.htm

Introduction to Azure multi-factor authentication for Office 365 (it's free!).

The AD FS Proxy appears to fetch configuration information from the AD FS farm once per minute.

This guy says the Web Application Proxy can route requests to different servers via hostname!

Here's how to configure extranet lockout with AD FS 3.0. You can only configure this on the primary ADFS server...and I think it pushes the configuration out to the web proxy, but haven't validated that thought.

Here's how to customize the logo on the AD FS sign-in page.

12/7/14

How to prevent rogue DHCP servers with DHCP snooping in an HP Procurve switch

This is cool. I'm using an HP Procurve 2530 switch, running firmware YA.15.16. These switches, by the way, come with a lifetime next-business-day warranty.

To learn how to configure this feature, I went to HP's support lookup tool, typed in J9777A, looked up the manuals, went into the general reference area, then opened the HP Switch Software Access Security Guide for YA/YB.15.16.

Here are the commands to run:
dhcp-snooping (this enables the feature)
dhcp-snooping trust 8 (this permits a DHCP server to operate on port 8)
dhcp-snooping vlan 1 (this enables the feature on the default VLAN of 1)

Here's a screenshot pre-DHCP-snooping (two DHCP servers respond to the request):

Here's a screenshot with DHCP replies permitted only on port 8 (server is 192.168.100.254):

Here's a screenshot with DHCP replies permitted only on port 1 (server is 192.168.100.110):

The switch also keeps track of what IP address(es) and MAC address(es) are connected to each untrusted port - show dhcp-snooping binding. If you see a bunch of IP addresses on a port, that could indicate someone has added a downstream switch.

It appears (per this post) that if you have multiple HP switches, you'd want to use no dhcp-snooping option 82.

11/30/14

Storage Spaces - an update

Today I noticed that the StorageSpaces event log is full of dire notifications:

So I tried to move my virtual machines to another host (so I could safely work on the StorageSpace), but was told "Virtual machine migration operation failed at migration source. Failed to establish a connection with host. No credentials are available in the security package" even though I'd long-since configured constrained delegation with these commands from AidanFinn:

$HostName = "host1"
$HostFQDN = "$HostName.demo.internal"
Get-ADComputer host2| Set-ADObject -Add @{"msDS-AllowedToDelegateTo"="Microsoft Virtual System Migration Service/$HostFQDN", "Microsoft Virtual System Migration Service/$HostName", "cifs/$HostFQDN", "cifs/$HostName"}

Restarting the NETLOGON service on the source Hyper-V server fixed this.

With my virtual machines moved off, I wanted to remove (from the pool) the physical disk that StorageSpaces reported an I/O error on (to run Seagate diagnostics on it)...but because a virtual disk which used that physical disk was in a degraded state, the Server Manager wouldn't let me do that.

In a production environment, you might just pull out the suspected bad drive and put in a new drive, but here I really want to run the Seagate diagnostics on the drive while it was still in the computer case (you could say I'm lazy)...so I deleted the virtual disk, removed the physical drive from the storage space, launched the Seagate diagnostic tool (it was able to see the drive) and started the "Fix All - Long" test (link) which took 3.5 hours and reported the drive as good, so I added the physical disk back into the Storage Space and recreated the virtual disk and volume on it.

Upon trying to move virtual machines back to this host, I got the same error message as above! In fact, I couldn't even RDP into the VM host by hostname...although doing so by IP address worked. The solution to both problems was to configure the NIC with an online DNS server.

In the process of re-learning Storage Spaces a bit, I found this nice overview of the technology and this in-depth explanation of how to replace a failed disk.

OK, so life is back to normal. In the future, I want an email alert when Storage Spaces writes an error or warning into its Windows event log...here are the scripts:

c:\email.ps1
Function Send-Email($EmailSubject, $EmailBody)
{
$Username = "jeremy@comcast.net"
$Password = ConvertTo-SecureString 'MyPassword' -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential $Username, $Password

Send-MailMessage -from VM-HOST1<jeremy@comcast.net> -to jeremy@mydomain.com -subject $EmailSubject -body $EmailBody -smtpserver smtp.comcast.net -port 587 -usessl -Credential $cred
}

c:\monitor-storagespaces.ps1
#Import the function above
. "C:\Email.ps1"

$MyFilter = @{LogName='Microsoft-Windows-StorageSpaces-Driver/Operational';Level=1,2,3;StartTime=(Get-Date).AddMinutes(-5)}

Get-WinEvent -FilterHashTable $MyFilter | ForEach-Object {

    $EmailSubject = "Storage Spaces " + $_.LevelDisplayName + " (" + $_.ID + ")"
    $EmailBody = $_.Message
    Send-Email $EmailSubject $EmailBody
    }

c:\create-task.ps1
$T = New-ScheduledTaskTrigger -Once -At (Get-Date).AddMinutes(5)
$T.RepetitionInterval = (New-TimeSpan -Minutes 5)
$T.RepetitionDuration = (New-TimeSpan -Days 3650)

$A = New-ScheduledTaskAction -execute "Powershell.exe" -argument "-nologo -noprofile -noninteractive -ExecutionPolicy Bypass -File C:\Monitor-StorageSpaces.ps1"
Register-ScheduledTask -TaskName "Monitor Storage Spaces (setup by Jeremy)" -Trigger $T -Action $A -User "NT AUTHORITY\SYSTEM" -RunLevel 1

11/28/14

Misc notes

Beginning on Wednesday night and continuing today, I’ve been working on my home lab setup. Here are things I’ve been learning.

– You can’t set a “default domain” for email addresses in Office 365 if you’re using AD FS. Instead, you’re supposed to define a user’s primary email address in the on-prem Active Directory and let Dirsync send it up to the cloud.

– So I set the correct email address for my account in Active Directory, went to bed, woke up the next morning and wondered “Why hasn’t it been updated in O365?”! Well, for one, the “Forefront Identity Manager Synchronization Service” was stopped on the box that has DirSync installed…so I started it…and then wondered why the status screen had so many “stopped-extension-dll-exception” lines…that was because the username was incorrect for the O365 account that DirSync was trying to use…fixed that…then I *thought* I was executing some manual syncs, but actually wasn’t – here’s the correct sequence of syncs when DirSync is working properly:

– The AD FS sign-on page wasn’t loading after I’d been tinkering with Failover Clustering…found that the AD FS service had been set to manual! Fixed that.

– Glanced at a current sFlow report from the switch and thought “Why is my laptop sending so much UDP traffic to 10.10.10.22?”! It turns out that Windows enables SNMP on TCP/IP printer ports so it can see if they’re online or offline…however, you then wind up with this chatty behavior…so I turned off SNMP on all my TCP/IP printer ports.

– Spent a bunch of time tinkering with IGMP multicast in both the NLB cluster for ADFS and inside the HP switch…found it to be temperamental and essentially worthless when I tested it by pulling the plug on a node. Switched back to basic multicast on a dedicated VLAN…which works great.

– Placed the wireless network on its own VLAN and isolated it with Sonicwall firewall rules…added a rule to allow my laptop’s IP address to pass through to the LAN…created a DHCP reservation for the MAC address of my laptop.

– My laptop’s clock (a domain workstation) had drifted several minutes behind, so I logged into a domain controller and followed instructions to run (syntax of w32tm):

w32tm /config /manualpeerlist:0.us.pool.ntp.org /syncfromflags:MANUAL
w32tm /config /update
w32tm /resync

…this worked fine on the domain controller, but when I ran w32tm /resync on my laptop, there was no change to the clock…why?! Well, I had assumed that I knew which server is the PDC on my network, but I assumed wrong…by going to ADUC…right-click on the domain at the top, click Operations Masters…I realized that my *other* domain controller is the PDC, and that’s what domain workstations are going to check for their time…so I corrected its time with the commands above and then ran w32tm /resync on my laptop successfully.

11/21/2014 -

Last Wednesday night Dustin or Kalen told me about “NetFlow” which is a traffic reporting tool baked into Cisco hardware. HP invented a similar traffic reporting tool called “sFlow” which doesn’t report every single packet: its advocates describe this as a more “scalable” approach, its detractors state that this reduces precision. Anyway, I have a shiny new HP switch on my hands and am eager to learn this sFlow stuff!

This blog post introduces sFlow by comparing it to jelly beans. This sflow.org essay describes the math behind sampling. On the switch, I need to configure a “sampling rate” and a “polling interval”. The sampling rate refers to the switch telling you what 1 packet out of every X packets is; the polling interval is (maybe?) how often that data is pushed to a collector server.

To configure sFlow on my Procurve switch, I found a PDF file called “Traffic monitoring on ProCurve switches with sFlow and InMon Traffic Sentinel”.

sFlow is more efficient than SNMP. Here’s a guide to sampling rates.

SSH’d to the switch’s IP with Putty and logged in with “admin” and no password. The HP CLI includes tab-completion and instantly shows command options when you append a ? to any valid command.

First, wanted to enable sFlow:

enable
config
sflow 1 destination 192.168.x.x
sflow 1 sampling all 50 (the lowest allowed value in packets)
sflow 1 polling all 20 (the lowest allowed value in seconds)

…then set the date/time:

time 11/21/2014 17:28
time timezone -480 (for Pacific Standard)

Lastly, saved the running configuration to the startup configuration:

write memory

To view all the sFlow data coming out of the switch, you need software. In an enterprise, I think I’d like to try “Traffic Sentinel”. Tonight, I tried out Plixer’s Scrutinizer, but found it too resource intensive and slow. By contrast, the free Java-based sFlowTrend by inMon works well. One way of viewing the traffic on your switch(s) is with a “network circle” and that made me wonder why my computer was connecting to interesting hostnames ending in 1e100.net. Here’s why.

11/20/2014 -

This evening I received a Sonicwall TZ 105 ($193). Created a MySonicwall.com account which let me immediately register the device, but it took several minutes before I could login to the web portal to download firmware. It appears that I’m licensed to download firmware updates for only 90 days. Applied latest firmware to the Sonicwall (5.9.6).

Applied latest firmware to the HP Procurve switch (YA.15.16), downloaded from here.

8/25/2014 -

Recent projects at work:

Migrated a metal-roofing company’s email to Office 365 (10 users) and virtualized their servers onto new hardware. Installed Veeam with High-Rely drives.
Migrated a concrete washout company’s email to Office 365 (28 users).
Installed a Bluesocket wireless access point for a winery (it was plagued by so many delays that I bought flowers for the manager as an apology).
Moved the servers of a company that monitors vibrations for the Alaskan Way Viaduct.
Troubleshot dropped VoIP calls for an Alaskan seafood company.
Audited Microsoft licensing compliance at a flooring company.
Upgraded Asterisk to fix poor call quality for a property management client.
Setup an automatically-deployed SSL VPN for a satellite company.

10/11/14

Exploring Microsoft's "Azure" - setting up an IPsec VPN.

I need to explore Microsoft's Azure cloud services. Azure offers on-demand SSTP VPN access and/or always-on IPsec VPN access for $27/month. I want the latter. At my work, we'd use a Sonicwall router for the IPsec VPN, but I'd like to test this out on a lower budget. The Cisco RV110 ($67) would work, but then I realized it could be done for free with the pfSense open source firewall software.

Step one was to buy a USB-to-Ethernet adapter that has drivers for Hyper-V Server 2012 R2 ($12). The setup.exe ran just fine on the Hyper-V server. Created a virtual "external" Hyper-V switch, not shared with the management OS, single-root SR-IOV enabled, for the WAN. Used "Legacy NICs" for compatibility purposes...they top out at 100Mbps, which is fine for my 15/5Mbps cable link.

The pfSense's setup wizard is case sensitive if you enable MAC address spoofing and type in a MAC (it wants lower case). If you go this route, also enable MAC spoofing on the Hyper-V NIC.

The pfSense WAN interface had a hard time obtaining a DHCP lease from the cable modem. Eventually I rebooted the cable modem (again) and then traffic started flowing.

Enabling DHCP on the LAN interface unexpectedly disabled access to the web GUI (couldn't even ping the LAN interface), so rebooted the pfSense and it still didn't work. Needed to choose option 8 at the console (shell access) and typed ifconfig de1 down...then ifconfig de1 up...that fixed it...then I was able to access the web GUI and saw the WAN interface had no IP, so clicked the "Renew" button and then it got its DHCP lease OK from the cable modem.

OK, now I had reached the point where I'd have been 5 minutes after bringing home a router from the store!

To setup the IPsec VPN link, followed this outstanding tutorial. However, I didn't follow it exactly - I enabled dynamic routing in the Azure gateway...and that's not compatible with pfSense...so deleted the gateway and recreated it with static routing per the tutorial...connected OK now. Lastly, edited the firewall rules in pfSense to allow all traffic in/out of the VPN.

I'd spun up a domain controller in Azure and now assigned a static IP address to it....which promptly disabled my access because you're not allowed to do that in Azure - every virtual machine must have a DHCP lease in Azure. Oops!

There's no console access to Azure virtual machines as of this writing (October, 2014), so if you can't RDP into your virtual machine, here are your options (I used option 1):

1. Delete the VM, retaining the VHD(s). Recreate it with the original VHD(s).
2. Use Powershell to download the VHD, boot it in a local Hyper-V session, fix it, upload it.

I've learned what the DNS section of each virtual network is for - because your virtual machines must use DHCP assigned addresses...and you don't have access to the Azure DHCP server, this is where you specify the DNS server(s) that you want the Azure DHCP server to issue to your virtual machines. You can type in any DNS server that you want.