Automating Cloudeberry Drive installation in AWS

Note: Cloudberry technical support personal are fantastic and they are very prompt in answering questions. A 5* customer support.

As we use Phoenix server’s pattern, our instance are teared down either with new installations or when auto scaling (scale down policy) is kicked in.

One of the challenges we ran into is with automating Cloudberry Drive(cbd) setup. Cloudberry Drive, we use to map and sync s3 bucket to a drive on the server (windows 2012R2) which gives access to files that are then processed by Talend jobs.

As the job processing server is a single node setup and in an effort to build fault tolerant system (infrastructure), we decided to automate the process of setting up Talend box with Cloudberry drive setup using Cloudformation and scripting (Powershell).

I ran into challenges in automating cloudberry drive though there is a neat documentation on commands with detailed technical documentation on CloudBerry site.  

There’s few guidelines on activation and release of cbd which became a challenging in automating cbd setup:

1. License is per server
2. License cannot be reused.
3. To port license to new server, it needs to be released from previous server before re-registering on new server.

This forced me to think on different strategies to implement a solution and here is what I’ve done.

Created cbd_unregister script and attach it to Windows shutdown event. It turned out to be a challenging to automate attaching unregister script to windows shutdown event as there is not easy way to do it without getting cluttered with windows registry.

Created a .reg file and called it in register_cbd script. Other option is to use ps1 commands to add/update registry settings. I see .reg file is simple enough. cbd_unregister.ps1 script will be called when windows instance is terminated (aws).

Created register_cbd.ps1 which installs vc++ redistributable component, installs cbd and configures drive mappings and s3 account.

Setup Cloudformation userdata with powershell command to point to register_cbd.ps1.

Voila!!!

There is another catch, that the server name should be same. As it’s a one node setup, Now the process includes a step to update windows name on AWS. Note that the instance needs to be rebooted, for the first time, every time a new instance is created to make new hostname effective and then activate cbd license.

Cloudberry tech support is very helpful in answering my every email which helped me to draft implementation solution for out automation process.

Simple script to unregister cbd.exe

function unregister_cloudberrydrive_on_system_shutdown{

    #stop cloudberry service...

    Stop-Service "CloudBerry Drive Service"

    try{

        #change directory (set location path)

        #cd "C:\Program Files\CloudBerryLab\CloudBerry Drive"

        Set-Location -Path "C:\Program Files\CloudBerryLab\CloudBerry Drive"

        .\cbd.exe /silent /releaselicense

        return $true

    }

    catch{

        $ErrorMessage = $_.Exception.Message

        $FailedItem = $_.Exception.ItemName

        Write-Host "Error unregistering CloudBerry Drive"

        Write-Host $FailedItem

        return $false

    }

} 

Rename AWS ec2 instance >>

Rename-Computer -NewName $newServerName -Restart -Force 

install/setup cbd.exe

function setup_cloudberry_for_talend{

     param(

        [Parameter(Mandatory=$false]

        [string]$account_display_name="S3",

        [Parameter(Mandatory=$true]

        [string]$accesskey,

        [Parameter(Mandatory=$true]

        [string]$secretkey,

        [Parameter(Mandatory=$true]

        [string]$cbd_license_key,

        [Parameter(Mandatory=$true]

        [string]$cbd_registered_email,

        [Parameter(Mandatory=$false]

        [string]$driveLetter="F",

        [Parameter(Mandatory=$false]

        [string]$accountName="S3",

        [Parameter(Mandatory=$true]

        [string]$containerWithfolder,

        [Parameter(Mandatory=$false]

        [string]$volumeLabel="File-Import"

        )

    #check if VC++ redist is intalled, if not go ahead and install

    $isVCinstalled = Get-WmiObject Win32_Product  -Filter "Name LIKE '%Visual C++%'"

    #install vc++ redistributable components for Cloudberry to be installable.

    if($isVCinstalled -eq $null){

        write-host "installing VC++ Redist......"

        c:\deploy\vcredist_x64.exe /q

        Write-Host ".....Done installing VC++ Redist."

        #pause for few seconds (may not require, just giving it a breathing period)

        Start-Sleep -Seconds 5

    }

    #check if VC++ redist is intalled, if not exit out with an error

    $vcInstallation= Get-WmiObject Win32_Product  -Filter "Name LIKE '%Visual C++%'"

    if($vcInstallation -eq $null){

        #return with an error?

        Write-Host "<<<<< Something wrong, cannot find VC++ Redist. >>>>>"

        #Exit out with an error message.  

 return "VC++ installation failed. Aborting cbd setup" 

    }

    else{

        Write-Host "Installed " $vcInstallation.Name ", Version:" $vcInstallation.Version

    }

    #install cloudberry software

    c:\deploy\CloudBerryDriveSetup_v2.3.4.1.exe /S

    Start-Sleep -Seconds 10

    #change directory (set location path)

    Set-Location -Path "C:\Program Files\CloudBerryLab\CloudBerry Drive"

    #first activate the license before moving further

    .\cbd.exe /activatelicense -k $cbd_license_key -e $cbd_registered_email

    #register S3 account by calling cloud berry drive exe (cbd.exe)

    .\cbd.exe addAccount -d $account_display_name -st S3 -ac $accesskey -sk $secretkey -ssl yes 

    #add drive

    .\cbd.exe addDrive -d $driveLetter -an $accountName -path $containerWithfolder -label $volumeLabel

   

    #mount drive

    .\cbd.exe mountDrive $driveLetter

    #finally start service

    .\cbd.exe /startservice

    Write-Host "...Done Start CloudBerry service."

   

    #start tray so mounted drive is visible and accessible...

    .\CloudBerryDriveTray.exe

    #now run .reg file to register windows shutdown event to run unregister_cloudberrydrive_on_system_shutdown.ps1 scrit

    #regedit /s Update-registry-to-add-local-policy.reg

    reg import Update-registry-to-add-local-policy.reg

}

Update-registry-to-add-local-policy.reg file >>>

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown\0]

"GPO-ID"="LocalGPO"

"SOM-ID"="Local"

"FileSysPath"="C:\\WINDOWS\\System32\\GroupPolicy\\Machine"

"DisplayName"="Local Group Policy"

"GPOName"="Local Group Policy"

"PSScriptOrder"=dword:00000002

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown\0\0]

"Script"="unregister_cloudberrydrive_on_system_shutdown.ps1"

"Parameters"=""

"IsPowershell"=dword:00000001

"ExecTime"=hex(b):00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Startup]

There is a master ps1 script in which above scripts are called. Master script is set to UserData in Cloudformation template allowing it to be executed on stack update.

Setup is such that all the needed software, scripts are on S3 and is downloaded on to the server as configured in Cloudformation template.