Note: Cloudberry technical support personal are fantastic and they are very prompt in answering questions. A 5* customer support.
As we use Phoenix server’s pattern, our instance are teared down either with new installations or when auto scaling (scale down policy) is kicked in.
As we use Phoenix server’s pattern, our instance are teared down either with new installations or when auto scaling (scale down policy) is kicked in.
One of the challenges we ran into is with automating Cloudberry Drive(cbd) setup. Cloudberry Drive, we use to map and sync s3 bucket to a drive on the server (windows
2012R2) which gives access to files that are then processed by Talend jobs.
As the job processing server is a single node setup and in
an effort to build fault tolerant system (infrastructure), we decided to
automate the process of setting up Talend box with Cloudberry drive setup using
Cloudformation and scripting (Powershell).
I ran into challenges in automating cloudberry drive though
there is a neat documentation on commands with detailed technical documentation on CloudBerry site.
There’s few guidelines on activation and release of cbd which
became a challenging in automating cbd setup:
- License is per server
- License cannot be reused.
- To port license to new server, it needs to be released from previous server before re-registering on new server.
This forced me to think on different strategies to
implement a solution and here is what I’ve done.
Created cbd_unregister script and attach it to Windows
shutdown event. It turned out to be a challenging to automate attaching unregister
script to windows shutdown event as there is not easy way to do it without
getting cluttered with windows registry.
Created a .reg file and called it in register_cbd
script. Other option is to use ps1 commands to add/update registry settings. I
see .reg file is simple enough. cbd_unregister.ps1 script will be called
when windows instance is terminated (aws).
Created register_cbd.ps1 which installs vc++
redistributable component, installs cbd and configures drive mappings and s3
account.
Setup Cloudformation userdata with powershell command to
point to register_cbd.ps1.
Voila!!!
There is another
catch, that the server name should be same. As it’s a one node setup, Now the process
includes a step to update windows name on AWS. Note that the instance needs to
be rebooted, for the first time, every time a new instance is created to make
new hostname effective and then activate cbd license.
Cloudberry tech support is very helpful in answering my every email which helped me to draft implementation solution for out automation process.
Simple script to unregister cbd.exe
function unregister_cloudberrydrive_on_system_shutdown{
#stop
cloudberry service...
Stop-Service
"CloudBerry Drive Service"
try{
#change
directory (set location path)
#cd
"C:\Program Files\CloudBerryLab\CloudBerry Drive"
Set-Location
-Path "C:\Program
Files\CloudBerryLab\CloudBerry Drive"
.\cbd.exe
/silent /releaselicense
return
$true
}
catch{
$ErrorMessage
= $_.Exception.Message
$FailedItem
= $_.Exception.ItemName
Write-Host
"Error unregistering CloudBerry Drive"
Write-Host
$FailedItem
return
$false
}
}
Rename AWS ec2 instance >>
Rename-Computer -NewName $newServerName
-Restart -Force
install/setup cbd.exe
function setup_cloudberry_for_talend{
param(
[Parameter(Mandatory=$false]
[string]$account_display_name="S3",
[Parameter(Mandatory=$true]
[string]$accesskey,
[Parameter(Mandatory=$true]
[string]$secretkey,
[Parameter(Mandatory=$true]
[string]$cbd_license_key,
[Parameter(Mandatory=$true]
[string]$cbd_registered_email,
[Parameter(Mandatory=$false]
[string]$driveLetter="F",
[Parameter(Mandatory=$false]
[string]$accountName="S3",
[Parameter(Mandatory=$true]
[string]$containerWithfolder,
[Parameter(Mandatory=$false]
[string]$volumeLabel="File-Import"
)
#check if
VC++ redist is intalled, if not go ahead and install
$isVCinstalled
= Get-WmiObject
Win32_Product -Filter "Name LIKE '%Visual C++%'"
#install vc++
redistributable components for Cloudberry to be installable.
if($isVCinstalled -eq
$null){
write-host
"installing VC++ Redist......"
c:\deploy\vcredist_x64.exe
/q
Write-Host
".....Done installing VC++ Redist."
#pause for few seconds (may not require, just giving it
a breathing period)
Start-Sleep
-Seconds 5
}
#check if
VC++ redist is intalled, if not exit out with an error
$vcInstallation= Get-WmiObject Win32_Product
-Filter "Name
LIKE '%Visual C++%'"
if($vcInstallation -eq
$null){
#return
with an error?
Write-Host
"<<<<< Something wrong, cannot
find VC++ Redist. >>>>>"
#Exit out with an error message.
return
"VC++ installation failed. Aborting cbd
setup"
}
else{
Write-Host
"Installed " $vcInstallation.Name
", Version:" $vcInstallation.Version
}
#install
cloudberry software
c:\deploy\CloudBerryDriveSetup_v2.3.4.1.exe
/S
Start-Sleep
-Seconds 10
#change
directory (set location path)
Set-Location
-Path "C:\Program
Files\CloudBerryLab\CloudBerry Drive"
#first
activate the license before moving further
.\cbd.exe /activatelicense -k
$cbd_license_key -e
$cbd_registered_email
#register S3
account by calling cloud berry drive exe (cbd.exe)
.\cbd.exe addAccount -d $account_display_name -st
S3 -ac $accesskey -sk $secretkey -ssl yes
#add drive
.\cbd.exe addDrive -d $driveLetter -an
$accountName -path
$containerWithfolder -label $volumeLabel
#mount drive
.\cbd.exe mountDrive $driveLetter
#finally
start service
.\cbd.exe /startservice
Write-Host "...Done Start CloudBerry service."
#start tray
so mounted drive is visible and accessible...
.\CloudBerryDriveTray.exe
#now run .reg
file to register windows shutdown event to run unregister_cloudberrydrive_on_system_shutdown.ps1 scrit
#regedit /s
Update-registry-to-add-local-policy.reg
reg import Update-registry-to-add-local-policy.reg
}
Update-registry-to-add-local-policy.reg file >>>
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts]
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown]
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown\0]
"GPO-ID"="LocalGPO"
"SOM-ID"="Local"
"FileSysPath"="C:\\WINDOWS\\System32\\GroupPolicy\\Machine"
"DisplayName"="Local Group Policy"
"GPOName"="Local Group Policy"
"PSScriptOrder"=dword:00000002
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Shutdown\0\0]
"Script"="unregister_cloudberrydrive_on_system_shutdown.ps1"
"Parameters"=""
"IsPowershell"=dword:00000001
"ExecTime"=hex(b):00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\Scripts\Startup]
There is a master ps1 script in which above scripts are called. Master script is set to UserData in Cloudformation template allowing it to be executed on stack update.
Setup is such that all the needed software, scripts are on S3 and is downloaded on to the server as configured in Cloudformation template.
