IIS Log parser

Excellent blog detailing Log Parser Studio
https://blogs.msdn.microsoft.com/friis/2014/02/06/how-to-analyse-iis-logs-using-logparser-logparser-studio/

Tool>>
https://gallery.technet.microsoft.com/Log-Parser-Studio-cd458765

https://stackoverflow.com/questions/11296698/understanding-iis-7-log-files

Install .NET Core on Linux (Ubunut) and deploy ASP.NET Core RESTful service

Once ASP.NET Core rest service is compiled, below are the steps to setup Linux (Ubuntu) and deploy .net core service code.

Note: My setup pulls published code from S3 bucket on AWS. I provided links to sample code and conf files for your reference in the comments

Here is the bash script to install .NET Core on Ubuntu and setup .net sample rest service.

#!/bin/bash

#! Install dotnetcore

sudo sh -c 'echo "deb [arch=amd64] https://apt-mo.trafficmanager.net/repos/dotnet-release/ xenial main" > /etc/apt/sources.list.d/dotnetdev.list'

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 417A0893

sudo apt-get update -y

sudo apt-get install dotnet-dev-1.0.1 –y   #!Finished installation of dotnetcore

sudo apt install awscli –y #! Install aws cli command tool

sudo apt-get install unzip –y #! Install unzip library

sudo apt-get install supervisor –y #! Install supervisor service which starts service and monitors

cd /home/ubuntu/

mkdir code

cd code/

aws s3 cp s3://dotnetonlinux/publish.zip publish.zip #! Download code from s3 onto server

cd /var/

mkdir dotnetcore

cd dotnetcore/

cd /home/ubuntu/code/

unzip publish.zip -d /var/dotnetcore/ #! Unzip code to var/dotnetcore folder

#!Run this manually.

#!This requires sudo su permissions...

service supervisor stop #! Start supervisor service

cd /etc/supervisor/conf.d

aws s3 cp s3://dotnetonlinux/dotnetcore.conf dotnetcore.conf #! Download conf file from s3. 

Automating Cloudeberry Drive installation in AWS

Note: Cloudberry technical support personal are fantastic and they are very prompt in answering questions. A 5* customer support.

As we use Phoenix server’s pattern, our instance are teared down either with new installations or when auto scaling (scale down policy) is kicked in.

One of the challenges we ran into is with automating Cloudberry Drive(cbd) setup. Cloudberry Drive, we use to map and sync s3 bucket to a drive on the server (windows 2012R2) which gives access to files that are then processed by Talend jobs.

As the job processing server is a single node setup and in an effort to build fault tolerant system (infrastructure), we decided to automate the process of setting up Talend box with Cloudberry drive setup using Cloudformation and scripting (Powershell).

I ran into challenges in automating cloudberry drive though there is a neat documentation on commands with detailed technical documentation on CloudBerry site.  

There’s few guidelines on activation and release of cbd which became a challenging in automating cbd setup:

1. License is per server
2. License cannot be reused.
3. To port license to new server, it needs to be released from previous server before re-registering on new server.

This forced me to think on different strategies to implement a solution and here is what I’ve done.

Created cbd_unregister script and attach it to Windows shutdown event. It turned out to be a challenging to automate attaching unregister script to windows shutdown event as there is not easy way to do it without getting cluttered with windows registry.

Created a .reg file and called it in register_cbd script. Other option is to use ps1 commands to add/update registry settings. I see .reg file is simple enough. cbd_unregister.ps1 script will be called when windows instance is terminated (aws).

Created register_cbd.ps1 which installs vc++ redistributable component, installs cbd and configures drive mappings and s3 account.

Setup Cloudformation userdata with powershell command to point to register_cbd.ps1.

Voila!!!

There is another catch, that the server name should be same. As it’s a one node setup, Now the process includes a step to update windows name on AWS. Note that the instance needs to be rebooted, for the first time, every time a new instance is created to make new hostname effective and then activate cbd license.

Cloudberry tech support is very helpful in answering my every email which helped me to draft implementation solution for out automation process.

Simple script to unregister cbd.exe

function unregister_cloudberrydrive_on_system_shutdown{

    #stop cloudberry service...

    Stop-Service "CloudBerry Drive Service"

    try{

        #change directory (set location path)

        #cd "C:\Program Files\CloudBerryLab\CloudBerry Drive"

        Set-Location -Path "C:\Program Files\CloudBerryLab\CloudBerry Drive"

        .\cbd.exe /silent /releaselicense

        return $true

    }

    catch{

        $ErrorMessage = $_.Exception.Message

        $FailedItem = $_.Exception.ItemName

        Write-Host "Error unregistering CloudBerry Drive"

        Write-Host $FailedItem

        return $false

    }

} 

Rename AWS ec2 instance >>

Rename-Computer -NewName $newServerName -Restart -Force 

install/setup cbd.exe

function setup_cloudberry_for_talend{

     param(

        [Parameter(Mandatory=$false]

        [string]$account_display_name="S3",

        [Parameter(Mandatory=$true]

        [string]$accesskey,

        [Parameter(Mandatory=$true]

        [string]$secretkey,

        [Parameter(Mandatory=$true]

        [string]$cbd_license_key,

        [Parameter(Mandatory=$true]

        [string]$cbd_registered_email,

        [Parameter(Mandatory=$false]

        [string]$driveLetter="F",

        [Parameter(Mandatory=$false]

        [string]$accountName="S3",

        [Parameter(Mandatory=$true]

        [string]$containerWithfolder,

        [Parameter(Mandatory=$false]

        [string]$volumeLabel="File-Import"

        )

    #check if VC++ redist is intalled, if not go ahead and install

    $isVCinstalled = Get-WmiObject Win32_Product  -Filter "Name LIKE '%Visual C++%'"

    #install vc++ redistributable components for Cloudberry to be installable.

    if($isVCinstalled -eq $null){

        write-host "installing VC++ Redist......"

        c:\deploy\vcredist_x64.exe /q

        Write-Host ".....Done installing VC++ Redist."

        #pause for few seconds (may not require, just giving it a breathing period)

        Start-Sleep -Seconds 5

    }

    #check if VC++ redist is intalled, if not exit out with an error

    $vcInstallation= Get-WmiObject Win32_Product  -Filter "Name LIKE '%Visual C++%'"

    if($vcInstallation -eq $null){

        #return with an error?

        Write-Host "<<<<< Something wrong, cannot find VC++ Redist. >>>>>"

        #Exit out with an error message.  

 return "VC++ installation failed. Aborting cbd setup" 

    }

    else{

        Write-Host "Installed " $vcInstallation.Name ", Version:" $vcInstallation.Version

    }

    #install cloudberry software

    c:\deploy\CloudBerryDriveSetup_v2.3.4.1.exe /S

    Start-Sleep -Seconds 10

    #change directory (set location path)

    Set-Location -Path "C:\Program Files\CloudBerryLab\CloudBerry Drive"

    #first activate the license before moving further

    .\cbd.exe /activatelicense -k $cbd_license_key -e $cbd_registered_email

    #register S3 account by calling cloud berry drive exe (cbd.exe)

    .\cbd.exe addAccount -d $account_display_name -st S3 -ac $accesskey -sk $secretkey -ssl yes 

    #add drive

    .\cbd.exe addDrive -d $driveLetter -an $accountName -path $containerWithfolder -label $volumeLabel

   

    #mount drive

    .\cbd.exe mountDrive $driveLetter

    #finally start service

    .\cbd.exe /startservice

    Write-Host "...Done Start CloudBerry service."

   

    #start tray so mounted drive is visible and accessible...

    .\CloudBerryDriveTray.exe

    #now run .reg file to register windows shutdown event to run unregister_cloudberrydrive_on_system_shutdown.ps1 scrit

    #regedit /s Update-registry-to-add-local-policy.reg

    reg import Update-registry-to-add-local-policy.reg

}

Update-registry-to-add-local-policy.reg file >>>

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown\0]

"GPO-ID"="LocalGPO"

"SOM-ID"="Local"

"FileSysPath"="C:\\WINDOWS\\System32\\GroupPolicy\\Machine"

"DisplayName"="Local Group Policy"

"GPOName"="Local Group Policy"

"PSScriptOrder"=dword:00000002

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown\0\0]

"Script"="unregister_cloudberrydrive_on_system_shutdown.ps1"

"Parameters"=""

"IsPowershell"=dword:00000001

"ExecTime"=hex(b):00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Startup]

There is a master ps1 script in which above scripts are called. Master script is set to UserData in Cloudformation template allowing it to be executed on stack update.

Setup is such that all the needed software, scripts are on S3 and is downloaded on to the server as configured in Cloudformation template.

AWS => Continuous Integration & Continuous delivery/deployment

TeamCity + PowerShell => AWS Cloud

PhoenixServers Pattern: Avoiding Snowflakes, we tend towards Immutable servers which is logical conclusion of PhoenixServers pattern where our continuous deployment process will tear down existing servers and rebuild new cloud server from latest base image and deploy code/configuration onto new servers.

Our Philosophy: Build code one-time and should be environment agonistic. By ‘Environment‘, I mean Dev, Stage, UAT, Production. Compiled code should be runnable across all the environments as long as OS and any other dependent software is same across the environments this includes but not limited to IIS, JBoss, Database, and any 3^rd party software like Reddis etc.,

Code doesn’t change by environment, it is just that each environment has separate database, configuration or cache server to name a few.

Software & Tools: TeamCity, PowerShell, AWS PowerShell, AWS CLI, TFS, SVN, & Windows 2012

Software Development Platform: .net, java, Postgresql, Talend, Spark

1. What is the end package that is being deployed? 
• .exe, .dll, .jar, .ear, .zip, .config, .txt, Etc.,
2. What additional steps involved during deployment?
• Update config/properties files.
• Change end system configuration like iis settings or jboss configuration.
• Setup schedule tasks, cron jobs, triggers etc.,
3. What additional post deployment steps?
• Post production deployment smoke test.

Continuous Integration & Continuous Deployment to lower environments

  

Continuous Integration

1. Developers check-in code to TFS/Git
• TeamCity listens to source control server change check-ins
2. TeamCity fetch changes onto build server
3. TeamCity compiles code and generates repositories (nupkg/jar/ear/zip)
4. TeamCity runs unit tests and selenium tests
5. TeamCity records build/tests success or failure
6. TeamCity sends out success failure group notifications
   
   
   
7. On Successful build, TeamCity connects to AWS S3 and pushes latest build output (nupkg/jar/ear/zip/config) and deployment scripts/configs to S3
8. TC calls AWS stack creation with CloudFormation template as the input
9. On Successful AWS Stack call, TeamCity executes DB change deploy*
10. AWS Cloudformation receives CloudFormation template and executes on AWS Cloud within VPC (virtual private cloud) by creating AWS EC2 instances. AWS Cloudformation also downloads all the code from S3 which was uploaded to in step 7.
11. EC2 instance
• For .net deployment, will initialize AWS PowerShell and run ps1 scripts which installs required software like iis etc., and deploys  code. This is done by running ps1 scripts on the server which is get called from userdata section of CloudFormation template.
• For java deployment, will initialize AWS CLI and run commands which installs required software like jboss etc., and deploys Hub code. This is done from userdata section within CF template
• PS/CLI script take responsibility of attaching pre-configured security groups to new EC2 instances and EC2 to Elastic Load Balancers.
12. Upon successful completion of code deployment, EC2 instances will send out email notifications utilizing preconfigured AWS SNS.

*DB Change Deploy: New framework has been developed for database change deployments.

• During development, db change scripts are checked into TFS to ‘In Progress’ folder.
• Once change is reviewed, developer then checks-in to TFS ‘Completed’ folder.
• Every domain model now includes a table ‘deploy_log’.
• TFS ‘Completed’ folder is monitored by TeamCity.
• During .net & Java code build process, a step DB-Deploy, is introduced within TeamCity build process which gets latest changes from TFS onto build server.
• Once code build is successful, DB-Deploy step will connect to data base and query ‘deploy_log’.
• ‘Completed’ folder scripts are compared with ‘deploy_log’ table and missing new scripts are executed.
• Newly ran scripts are then inserted into ‘deploy_log’ table keeping it upto date of all the scripts that are ran.
• This entire db-deploy process is by environment and by domain model.

Continuous Delivery to Production

For Production deployments, step 8 is manual and is triggered from TeamCity, by Configuration management group once a CR is signed-off by QA and Product management team in UAT.

For Production deployments, step 9 is handled by DBA group and is outside of Continuous Deployment process.

Continuous Delivery of .net & java code to production are one-click deployments from TeamCity.

Stop/kill windows service - Automate deployments

We had issues in automating deployments, because some of our windows services took 2 or 3 attempts to stop them and sometimes they stuck in "stopping" status.

Our services are built for to use ESB mass transit and our understanding is that MT, constantly pinging queues for any messages and not giving a break to service to shutdown. We are still investigating the problem though.

So, our interim solution is to try stopping the service, if it don't, then kill the process.

Here is a powershell script to stop and kill if it the process is still hanging out there.

#$pServiceName - service name e.g., BloggService
#$pProcessName process name e.g., for BloggService process name is: BloggServiceProcess

  Param($pServiceName,$pProcessName)
 
  write-host "Executing script to Stop service:" $pServiceName
 
 #check if service is installed
 If (Get-Service -Name $pServiceName -ea SilentlyContinue) 
 {#service exists on this box, then stop it
  
  #get computer name for logging
  $computername = gc env:computername
  
  #Stop Service
     $arrService = Get-Service -Name $pServiceName
     if ($arrService.Status -eq "Running"){
      #stopping service
   Write-Host "Stopping service" $pServiceName "..."
   Stop-Service -Name $pServiceName 
     
      #if process is still active/running, then kill it
      if (Get-Process $pProcessName -ea SilentlyContinue)
      {
    $host.UI.WriteLine()
    Write-Host "Unable to stop" $pProcessName "process for service " $pServiceName", killing process..."
       $arrProcess = Get-Process $pProcessName -ea SilentlyContinue
    $arrProcess.kill() 
    $host.UI.WriteLine()
    Write-Host $pProcessName "process killed"
      }
 
   $host.UI.WriteLine()
   Write-Host $pServiceName "Service Stopped."
   $host.UI.WriteLine()
  }
  else
  {
   $host.UI.WriteLine()
   Write-Host $pServiceName "service is not running on " $computername
   $host.UI.WriteLine()
  }
 }
 else
 {
  $host.UI.WriteLine()
  Write-Host $pServiceName "is not installed on"  $computername
  $host.UI.WriteLine()
 } 
 

XmlPreprocess - switching configs across environments:Automate deployments

XMLPreProcess from sourceforge http://xmlpreprocess.sourceforge.net/

I love it, it helped us to gracefully handle config changes across different environments (Dev, Test, PreProd, Prod) during our deployments.

Especially we had issues to handle http bindings in web.configs across environments as our dev/test doesn't equipped with ssl certifications, and only PreProduction and Production are.

e.g.,

and post build event executes xmlpreprocess command by passing ${securitMode} value and result:

Its easy to use and fun.