Unix - Shell Script - Connect to SFTP from script

Description:

This post cover various ways to connect to SFTP from shell script. Passing the password in the command prompt is easier and secure as user is explicitly typing it down when prompted. But in shell script, password needs to be passed automatically and securely.

Approach-01: using sshpass command

Resource-01 covers detail about the command and how to install in various version on unix.  

SFTP_USER=<user>
SSHPASS=<password>

TARGET_SERVER=<host>
sshpass -e sftp -oBatchMode=no -b - $SFTP_USER@$TARGET_SERVER <<EOF
    ls -lrt
    cd <target path>
    put <source path/file name>
    bye
EOF

Approach-02: using lftp command

SFTP_USER=<user>

LFTP_PASSWORD=<PASSWORD>

TARGET_SERVER=<host>

lftp --env-password sftp://$SFTP_USER@$TARGET_SERVER <<EOF
    ls -lrt
    cd <target path>
    put <source path/file name>
    bye
EOF

Resources:

1. Detail about 'sshpass' command. How to install the command, various ways to pass password to connect to SFTP etc. 
   
   https://www.cyberciti.biz/faq/noninteractive-shell-script-ssh-password-provider/

Selenium - Learning

Description:
Intention of this post is to track high level concepts, components needed to understand Selenium-2 or Web driver. This can be considered as hand out for quick reference. Examples or demo of using it in real code will be covered in subsequent posts.

What is Selenium-2 or Web Driver:


Development stacks:

1. Java - 1.8
2. Eclipse IDE - neon
3. Windows - 2012 R2

How to setup:

1. Download Selenium web driver. I used Selenium 3.8.1 for Java. This will download a zip file which has 'client-combined-3.8.1-sources.jar' and 'client-combined-3.8.1.jar' jars at root level. Copy these two jars along with all the jars from lib folder. Check Link-1 below.
2. Download the browser driver intent to use for testing. In my case, I have downloaded web driver for IE, chrome and firefox as my application has to support these browser and phantomJS to run testing on headless browser to support deployment/testing automation. Check Link-2 below.  

High level overview:

1. WebDriver commands 
1. Fetching a page
1. Different ways 
1. driver.get("http://www.google.com");
2. driver.navigate().to("http://www.google.com");
2. Activity on browser 
1. Closes the current browser associated with the driver. driver.close();
2. Quit the driver and close all associated windows. driver.quit();
3. Refresh the current page. driver.refresh();
3. Locate UI element named as WebElement. 'Find Element' and Find Elements' method available in webdrvier or WebElement which takes locator or query object called 'By'
1. Different 'By' strategies are 
1. By ID
2. By Name
3. By Class
4. By tag name
5. By name
6. By link text
7. By xpath
8. Using javascript
9. and more
4. User interactions - check Link-5 for examples
1. Textbox 
2. RadioButton
3. CheckBox
4. Drop down item selection
5. Keyboard action
6. Drag and drop
7. Mouse action
8. Multi selection
9. Find all links
5. Moving between windows and frames
1. Examples 
1. driver.switchTo().window("windowName");
2. driver.switchTo().frame("frameName");
6. Popup dialogs
1. Examples
1. Alert alert = driver.switchTo().alert();
7. Navigation: History and Location
1. Examples
1. driver.navigate().to("http://www.google.com");
2. driver.navigate().forward();
3. driver.navigate().back();
8. Manage cookies. Add/delete/list all cookies
2. Synchronization
1. Old fashioned java way using 'Thread.sleep(xxx);' This will halt the execution for specified milliseconds.
2. Implicit wait - the web driver will wait for specified time and will not try to find the element before the time is elapsed. After the time is crossed, web driver will try to find the element, execution will continue on success, throws exception on failure. This wait works at global level for all web element search. 
1. Example:
1. WebDriver driver = new FirefoxDriver();
   driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
3. Fluent wait - defines maximum amount of time to wait before a condition take place as well as the frequency to check the condition
1. Example:
1. Wait wait = new FluentWait(driver).withTimeout(60, SECONDS).pollingEvery(10, SECONDS).ignoring(NoSuchElementException.class);
   WebElement dynamicelement = wait.until(new Function <webdriver,webElement>() {
   public WebElement apply(WebDriver driver) { return driver.findElement(By.id("dynamicelement")); } });
4. Explicit wait - set a certain condition to occur before move to next step. The condition will be periodically checked (by default every 500ms) until the specified timeout is reached which is defined during explicit wait creation. Common use is to perform an action based on element is visible. WebDriverWait extends FluentWait.
1. Example:
1. WebDriver driver = new FirefoxDriver(); driver.get("http://somedomain/url_that_delays_loading"); WebElement myDynamicElement = (new WebDriverWait(driver, 10)) .until(ExpectedConditions.presenceOfElementLocated(By.id("myDynamicElement")));

Links:

1. Selenium download page - http://www.seleniumhq.org/download/
2. Download web driver - check 'Third Party Drivers, Bindings, and Plugins' section of Link-1
3. http://www.seleniumhq.org/docs/01_introducing_selenium.jsp
4. https://www.tutorialspoint.com/selenium/
5. https://www.tutorialspoint.com/selenium/selenium_user_interactions.htm

Unix - Shell Script - Find files and execute a command on matches

Description:

Often time we need to perform additional tasks on file/folder found in a certain path. Instead of fetching the list and looping through the items, it's easier to combine the search and additional task in to one command, more like chaining commands. "find" unix command allows executing additional command to be executed on matched items. This article provides few examples to demonstrate how to do so with explanation.

Example-01: Delete all the log files older than 14 days

find . -maxdepth 1 -type f -mtime +14 -exec rm -fr {} \;

Explanation:

• Scan current folder path for only files (find .)
• Skip scanning sub folder(s) (-maxdepth 1)
• Consider only files, exclude directory (-type f)
• Match files which are older than 14 days (-mtime +14)
• Perform delete on matched items (-exec). -exec allows to execute a command on items
• Forcefully delete any match found without any confirmation message (rm -fr)
• Use {} (curly braces) as a placeholder for each file that the find command locates
• Need to add command terminator (\;). This is a must if -exec attribute is used in find command

Example-02: Copy all the matching files 

find . -maxdepth 1 -type f -exec cp {} ~/ \;

Explanation:

• Copy all matched items in home directory (-exec cp {} ~/). 
• Other parts are explained in example-01

Example-03: Change permission of matched files

find . -maxdepth 1 -type f -exec chmod 744 {} \;

Explanation:

• Change the file permission to 744 (-exec chmod 744). 
• Other parts are explained in example-01

Example-04: Execute multiple commands. Change permission and move all files to a folder 

find . -maxdepth 1 -type f -exec chmod 744 {} \; -exec mv {} ~/test/ \;

Explanation:

• Chains multiple commands using (-exec) multiple times. Each command must be ended with (\;)
• First command changes the permission and second command moves the files to test folder under home path.
• Other parts are explained in example-01

References:

1. Detail about 'find' command - https://kb.iu.edu/d/admm