CLI Guidelines and fast Support

Posted on Sun 09 June 2024 in Technology

AWS - Amazon Web Services

AWSCLI V2 via Curl (Linux Command Line)

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Please look for this page to more info

AWSCLI V2 Official Installation Page

Working with EC2 Instances

Launching EC2 Instance via aws cli command --> Very handy

aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type t2.micro --key-name MyKeyPair --security-group-ids sg-903004f8 --subnet-id subnet-6e7f829e

Terminating Instances

aws ec2 terminate-instances --instance-ids $(aws ec2 describe-instances --filters  "Name=instance-state-name,Values=pending,running,stopped,stopping" --query "Reservations[].Instances[].[InstanceId]" --output text | tr '\n' ' '

Terminating all Instances at once per InstanceID

INSTANCE_ID=$(aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | .InstanceId')

aws ec2 terminate-instances --instance-ids $INSTANCE_ID

Killing instances aws-cli Examples

Describing Instances

Discovering the Private/Public Ip

aws ec2 describe-instances --filters Name=tag:Name,Values=$1 --query 'Reservations[].Instances[].PrivateIpAddress' --output text
aws ec2 describe-instances --filter "Name=tag-key,Values=Name" "Name=tag-value,Values=*$name_tag*" "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*][Tags[?Key=='Name'].Value[],NetworkInterfaces[0].PrivateIpAddresses[0].PrivateIpAddress]" --output text
aws ec2 describe-instances --filters Name=tag:Name,Values=$1 --query 'Reservations[].Instances[].PublicIpAddress' --output text

Working with ECS Clusters

aws ecs list-clusters

aws ecs list-container-instances --cluster dash-operacoes-frontend-EcsCluster-hml

Working with CloudFront Distributions

Retrieving Distribution ID

distribution_ids=$(aws cloudfront list-distributions --query "DistributionList.Items[*].Id" --output text)

Invalidating Distribution ID

aws cloudfront create-invalidation --distribution-id $distribution_id --paths "/*"

Working with AutoScalingGroup

Grep ASG per ResourceID

aws autoscaling describe-auto-scaling-groups | grep ResourceId

Auto Scalling Group - Change Desire Capacity

Scaling down ASG

aws autoscaling set-desired-capacity \
    --auto-scaling-group-name Produto-AutoScalingGroup-$Env \
    --desired-capacity 0 \
    --no-honor-cooldown

Scaling UP ASG

aws autoscaling set-desired-capacity \
    --auto-scaling-group-name Produto-AutoScalingGroup-$Env \
    --desired-capacity 1 \
    --no-honor-cooldown

RDS - Stop DB Instance

aws rds stop-db-instance --db-instance-identifier coredb

Working with AWS Credentials

aws configure list
aws configure credentials
export AWS_PROFILE=profile
aws configure set region us-west-2 --profile profile

You might want to check AWS Vault 99 Designs GitHub page

Working with Lambdas

for i in `aws --region sa-east-1 lambda list-functions | jq --raw-output '.Functions[] | .FunctionArn'` ; do echo $i ;aws --region sa-east-1 lambda list-tags --resource "$i" | grep  ManagedBy  ; done | wc -l

S3 Storage

   aws s3 cp test.txt s3://mybucket/test2.txt

   aws s3 cp s3://mybucket/test2.txt $HOME

   aws s3 ls | grep bucket

   aws s3 rb s3://bucket-name --force  

   aws s3 sync . s3://bucket/subfolder/

Describing Security Groups

aws ec2 describe-security-groups | jq -M -r -f filter.jq | grep -v "INBOUND" | grep -v "0.0.0.0" | grep -v "80" | grep -v "443

Obs: you need the filter.jq to parsing the output

IaC CloudFormation

aws cloudformation create-stack --template-body file://single-instance.yml --stack-name
k8-master --parameters ParameterKey=KeyName,ParameterValue=tutorial ParameterKey=InstanceType,
ParameterValue=t2.micro

aws cloudformation list-exports

aws cloudformation delete-stack --stack-name jenkins-cluster-ecs-negociacao  --region us-east-1

aws cloudformation describe-stack-events --stack-name NegociacaoServiceStack --region us-east-1

DYNAMODB

aws dynamodb create-table --cli-input-json file://security-manager.json

CODECOMMIT

aws codecommit list-repositories --region us-east-1 | grep pontos

aws codecommit delete-repository --repository-name "dash-operacoes-front" --region sa-east-1

aws codecommit create-repository --repository-name "td-posicao-data-stream" --region us-east-1

aws codecommit get-repository --repository-name "pontos-pi-infra" --region us-east-1

FOR LOOP - Managing repositories with AWSCLI

for i in `cat repos.txt`; do aws codecommit create-repository --repository-name "$i" --region us-east-1; sleep 1; clear; done
for i in `cat repos.txt`; do aws codecommit delete-repository --repository-name "$i" --region us-east-1; sleep 1; clear; done
for i in `cat repos.txt`; do aws codecommit delete-repository --repository-name "$i" --region sa-east-1; sleep 1; clear; done
for i in `cat repos.txt`; do aws codecommit list-repositories "$i" --region us-east-1; sleep 1; clear; done

KUBERNETES

kubectl

Kubernetes Install Tools

Kubectl on MACOS

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

export KUBECONFIG=$KUBECONFIG:~/.kube/config-k8s-ci
echo 'export KUBECONFIG=$KUBECONFIG:~/.kube/config-k8s-ci' >> ~/.bashrc
echo 'export KUBECONFIG=$KUBECONFIG:~/.kube/config-k8s-ci' >> ~/.zshrc

echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc

AWSCLI Updating EKS Context

aws eks --region us-east-1 update-kubeconfig --name k8s-ci --profile greenbrasil

Printf

printf $(kubectl get secret --namespace default jenkins-cloudbees-jenkins-distribution -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo

ECR Registry Tag Mutability

Very nice reading regarding scanning and tagging your images pushed to ECR aws registry repository!

AWS Image Tag Mutability

Kubernetes Complete Guide

Very easy going reading about Kubernetes and its concepts!

Check it out:

Kubernetes Definite Guide

CKA, CKD, CKS Kubernetes Certification

Linux Foundation CKA, CKD, CKS

LINUX

SSH StrictHostKeyChecking

EC2INSTANCE="`aws ec2 describe-instances --filters Name=tag:Name,Values=Negociacao-AutoScalingGroup-prd --query 'Reservations[].Instances[].PublicIpAddress' --output text`"
ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=10 -i $HOME/google-drive-somospi/AWS/key-pair/key.pem ubuntu@$EC2INSTANCE
AWK + tr -d
VARPUBLIC=`aws ec2 describe-instances --instance-ids $EC2INSTANCE  | grep PublicIpAddress | awk ' { print $2 }'` && echo $VARPUBLIC | tr -d '",'
echo "ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=10 -i /home/bernardo/GoogleDrive/AWS/key-pair/key.pem ubuntu@"$VARPUBLIC""
echo "ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=10 -i /home/bernardo/GoogleDrive/AWS/key-pair/key.pem ubuntu@"$VARPUBLIC"" > ssh.sh

chmod +x ssh.sh

./ssh.sh

Get the Internet ISP IP via cli

wget -qO- http://ipecho.net/plain | xargs echo

SYSTEMD

SYSTEMCTL (Ubuntu/RedHat/CentOS) usage

To enable or disable services automatically on CentOS or RedHat Flavors.

systemctl disable httpd
systemctl enable docker
systemctl status kubelet
systemctl grep | grep running
systemctl list-unit-files | grep enabled

Control Commands

hostnamectl

Static hostname: snake09
  Icon name: computer-vm
    Chassis: vm
 Machine ID: 0d8a36db78064458a56e691770e08ab5
    Boot ID: 958d267a172a42a3ac515c941dee5a57
    Virtualization: oracle
    Operating System: Ubuntu 18.04.2 LTS
     Kernel: Linux 4.15.0-54-generic
      Architecture: x86-64

timedatectl

Local time: Tue 2019-07-16 18:12:38 UTC
    Universal time: Tue 2019-07-16 18:12:38 UTC
   RTC time: Tue 2019-07-16 18:12:37
  Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
systemd-timesyncd.service active: yes
   RTC in local TZ: no

localctl
System Locale: LANG=C.UTF-8
VC Keymap: n/a
      X11 Layout: us
X11 Model: pc105

Kill other user session

Verify how many users are logged in with the command "w" (who)
Sample answer: 
USER     TTY      FROM      LOGIN@   IDLE   JCPU   PCPU WHAT
vagrant  pts/0    192.168.0.20     19:55    0.00s  0.27s  0.00s w
vagrant  pts/1    192.168.0.20     20:17    2.00s  0.05s  0.05s -zsh

ps -ef | grep pts/1
Once you have the PID just kill -9 $PID! that's it

or you can try with AWK command
ps aux | grep vagrant | awk '{print $2}' | xargs sudo kill -9

Find below fewer examples about it:

User Management Session

Link to change the timezone* (CentOS)

Linux Academy Changing the timezone in your system

GIT Command line

Creating remote repo through command line

git init
git add .
git commit -m "first commit"
git remote add origin https://github.com/Eddie-Uncle/Test1.git
git push -u origin master

Merging branches

$ git init
$ echo "something" > new_file.txt
$ git add . 
$ git checkout -b master
$ git commit –m "First commit message"
$ git checkout -b new_branch
$ echo "appending something" >> new_file.txt
$ git add . 
$ git commit -m "merge"
$ git checkout master
$ Switched to branch 'master'
$ git merge new_branch

Delete git — Locally and remote

git checkout master 
git checkout -b edsonnewbranch 
git branch -d Test_Branch # (Delete locally desired branch)
git branch -D Test_Branch # (Delete locally force option)

git push origin --delete eddie-branch # (git push origin --delete branch to be deleted) (Delete remote branch)

git pull / push commands

Pulling code from Remote repo to local repo:

git pull origin master || git pull origin edsonnewbranch
git pull --all || (all branches)


Pushing code from Local repo to remote repo:

git push origin master || git push origin edsonnewbranch

Git clone

Default clone
git clone git@github.com:Eddie-Uncle/linux-automation.git

Specific branch
git clone --branch ubuntu git@github.com:Eddie-Uncle/linux-automation.git

Git set Upstream

git branch --set-upstream-to=origin/master master

git operations

git log -p
git status
git show

git credentials and user settings

git config --global user.name Eddie-Uncle
git config --global user.email bernardsp@gmail.com
git config --global core.editor vim
git config --global merge.tool vimdiff
git config --list
git config --global --edit

Save credentials - Store Git Credentials

git config --global credential.helper store
$ git push http://example.com/repo.git
Username: type your username
Password: type your password

SELINUX APACHE

semanage port -l | grep http
semanage port -a -t http_port_t -p tcp 80
sestatus

Working Ip

ip addr show eth0 | grep -oP '(?<=inet\s)\d+(\.\d+){3}'

Working with Native CENTOS 7 Firewall

Using iptables behind the scenes!

(CENTOS)
sudo firewall-cmd --zone=public --add-service=http --permanent
sudo firewall-cmd --zone-public --add-port=8080/tcp --permanent
sudo firewall-cmd --zone=public --list-services
rpm qa | grep firewalld

UFW (Ubuntu)

sudo ufw status verbose
sudo ufw allow from 192.168.0.0/24 to any port 8080

Files Handling

LSOF usage and examples

echo "OpenSockets="$(/usr/bin/lsof | grep -i weblogic | wc -l) > results.txt

sudo sh -c "echo '192.168.0.22' > /etc/hosts

/usr/sbin/lsof | grep -i weblogic

/usr/sbin/lsof | grep -i weblogic | grep -i deleted
and then kill stuck pid's!

/usr/sbin/lsof  | grep weblogic | awk '{ print $3 }' | sort -u

lsof -u weblogic | wc -l

/usr/sbin/lsof | grep weblogic | awk '{ print $2 }' | sort -u

[root@ares ~]# cat /proc/sys/fs/file-max
65536

Kill -9 expert usage

sudo kill -9 $(ps -ef | grep -i -E "openvpn" | grep -v grep | awk '{print $2}')

AWK and CUT usage

ps -ef | grep java | awk ‘{ print $2 }’ | sort -u | xargs kill -9
ps - ef | grep java | cut -d " " -f2 | xargs kill -9
ps -ef | grep weblogic | awk '{ print $2 }' | sort -u | xargs kill -9
kill -9 $(ps auwwwx | grep -i -E "jboss|weblogic|websphere" | grep -v grep | awk '{ print $2 }')

SED

sed -i 's/10.13.0.87/10.13.0.193/g' *.xml
sed -i s/8080/8080/g' *.xml
sed -i 's/server01:10110/server02:60101/g' appconfig.properties

var=value_of_the_var

remove 
sed '/myhost=.*/d' -i $var

sed '/exec $JAVA.*/d' -i $var

#replace at the line 28 the empty space fo the value of myhost variable
sed -e '28 s/^/myhost=$(hostname -f)\n\n/' -i $var
sed -e '8 s/^/JAVA_OPTS="$JAVA_OPTS -javaagent:\/opt\/sysmgmt\/AppDynamics\/Java_Agent\/ver4.4.3\/javaagent.jar"\n/' -i $var

#replace after searched term found

sed '/Launching /a \exec $JAVA -classpath "$classpath" -Xms$min_mem -Xmx$max_mem' -i $var

SED ON MACOS - FreeBSD

sed -i'' -e 's/sa-east-1/us-east-1/g' *.yaml

RSYNC - Examples

Copy files or folders from source to destination (incremental copy)
rsync -auvzh --progress OneDrive/* /tmp/OneDrive

Delete files at destination if it is not present in source (--delete)
rsync -avzh --delete --progress OneDrive/ /tmp/OneDrive

More examples online:

Rsync examples at Linuxtechi

GREP and PGREP usage

grep <search> <file> | awk '{print $1}' >> <new_file_output>.txt

pgrep -u hybris java
pgrep -u apigee java | wc -l 
usage: pgrep -u <user> <language Process>

example: grep -ir Carrefour *.sh | awk '{print $1}' >> search.txt

FOR LOOP USAGE

for x in `cat name_file`;do echo $x;done
for x in `cat name_file`;do ssh -q $x;done
for i in `cat file.txt` ; do ping -c1 $i 2>&1 | tee >> ping - output.txt; done

(Creating serveral files at a glance)
for i in {1..9}; do touch kubn40${i}.sh; done

usage: for i in {range..}; do command filename$${i}.txt; done
for i in {1..9}; do cat kubn410.sh >>  kubn40${i}.sh; done

DU Usage

du -sk * | sort -n
du -sh /logs/* | sort -nr | grep G

FIND Usage

find . -size +50M
find / -type f -name "index.php"
find / -type f -name "*.log" -size +1M
find / -type f -name "*.log" -size +1M | xargs rm -rf {} \;
find . -iname "server.xml"

Quick GREP and Tailing Logs (examples)

tail -f *.out | grep "Socket read timed out" | wc -l
tail -f *.out | grep "Socket read timed out" --color
tail -f *.out | grep "error" --color

cat bwbpelsp02_01-yyyy-MM-dd-HH-mm.log | grep "Socket read timedout" | wc -l

tail -f *.out | egrep -i 'ERR|ORA-|OVERL|SUSPEN'
tail -f *.out | egrep -i 'STUCK'

THREAD COUNT BY USER

ps -uweblogic -lf | wc -l
ps -uapache -lf | wc -l
ps -unginx -lf | wc -l

CHOWN and CHMOD smart usage

chown apigee: logs/ -R || chown apigee:apigee -R logs/
chmod +x shellscript.sh || chmod 755 shellscript.sh

Verify and count a specific java class

grep com.yourcompany.product.exception.ServiceException log.out | wc -l 
result = 11466

DMESG (old fashion)

[root@bwsubacat04 sitecontent]# dmesg | grep file-max
VFS: file-max limit 65536 reached
VFS: file-max limit 65536 reached

ENCODE LINUX

echo $LANG

LIMITS - SO LINUX (CENTOS 6)

LIMITS
echo '###WEBLOGIC###' >> /etc/security/limits.conf
echo 'weblogic  soft nproc 20000' >> /etc/security/limits.conf
echo 'weblogic  hard nproc 20000' >> /etc/security/limits.conf
echo 'weblogic  soft nofile 65536' >> /etc/security/limits.conf
echo 'weblogic  hard nofile 65536' >> /etc/security/limits.conf
echo 'weblogic  soft memlock 16384000' >> /etc/security/limits.conf
echo 'weblogic  hard memlock 16384000' >> /etc/security/limits.conf

IFCONFIG MTU

ifconfig ib0 | grep MTU; ifconfig ib1 | grep MTU; ifconfig bond0 | grep MTU

CHECK - LDAP USER

getent passwd | grep <user>

Processors Statistics

mpstat
sar 1 10
top -u 
sar -d
vmstat 1 10
iostat

Basic example sudores file (there are better practices, in this case these users are allowed to excecute as root without password - General overview)

SUDOERS
weblogic ALL=(ALL) NOPASSWD: /bin/vi /etc/hosts
fmwadmin ALL=(ALL) NOPASSWD: /bin/vi /etc/hosts

VI (vim) Hacks

.vimrc

edit .vimrc under $HOME/ folder or just create a new one, then include these lines below:

set number
syntax on
set tabstop=2
set autoindent
color desert

Undo vim

press the keyboard "ESC + u (undo)

Reverse Columns within a file

:%!awk '{print $2, $1}' --> or vice-versa

Saving without exit

After modify the file desidered, type
:w <the name of the new file> and then hit the enter button

CRONTAB

crontab -l (Exhibits the current crontab created)

crontab -e (Create a crontab under running user)

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

Any questions, please click link below:

Tee usage (similar to echo)

tee /etc/yum.repos.d/docker.repo <<-EOF

Obtain PID specific application

ps aux | awk -v app='Apache' '$0 ~ app { print $1 }’

Working with Aliases

alias
alias vi='vim'
alias cp='cp -i'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias mv='mv -i'
alias rm='rm -i'
alias k='kubectl'
alias python='/usr/sbin/python3'

USING CURL & WGET

Using Post method with cuRL

curl -i -X POST -H "Content-Type:application/xml" -d '<movementRequestJson><sku>3213219</sku><seller>13933305000106</seller><totalQuantity>1</totalQuantity><leadTime>0</leadTime><subInventory>9801</subInventory><warehouse>98</warehouse><stockType>FISICO</stockType></movementRequestJson>' http://company-stock-v2.elasticbeanstalk.com/stock

Downloading Files

curl -O https://download.virtualbox.org/virtualbox/5.2.30/VirtualBox-5.2.30-130521-OSX.dmg

Web Examples for CURL usage

<a href="http://www.codingpedia.org/ama/how-to-test-a-rest-api-from-command-line-with-curl/" target="_blank">Curl and examples GET POST All methods</a>

Using WGET - This case downloading JDK 1.8u51 from Oracle

wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.tar.gz"

Java download examples:

StackOverflow wget java downloads

Installing JAVA Runtime - OpenJDK using apt install

apt install default-jre
apt install openjdk-11-jre-headless
apt install openjdk-8-jre-headless
apt install openjdk-9-jre-headless

Backup using shell script (this example weblogic domain)

Backup I - Sample

#!/bin/bash -e

set -euf -o pipefail

DATA=`date +%Y-%m-%d-%H.%M`
tar -zcvf  /repo/bpel/backup/BPEL/backup-config-"$DATA".tar.gz /domains/BPEL/config
tar -zcvf  /repo/bpel/backup/BPEL/backup-security-"$DATA".tar.gz /domains/BPEL/servers/bwBPEL_Admin/security
tar -zcvf  /repo/bpel/backup/BPEL/backup-ldap-"$DATA".tar.gz /domains/BPEL/servers/bwBPEL_Admin/data/ldap
clear
echo "Backup successfully..."
exit

Backup II - Sample - excluding tarballs

#!/bin/bash -e

set -euf -o pipefail

echo "Starting Backup"
DATE=`date +%Y-%m-%d-%H.%M.$$`
echo "Starting Backup, except *.tar extensions"
tar -zcvf blog-"${DATE}".tar.gz --exclude='*.tar' .
if [ "$?" -eq "0" ]
then
echo "Backup Successfully..."
exit 0
else
echo "Backup has failed, please check"
exit 1
fi

Working with Certificates

Import certificate - Using Java keytool

keytool -import -alias mundipagg -file CA_b2w_cert_test.cer -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -trustcacerts

Listing Certificates

./keytool -list -v -keystore $JAVA_HOME/jre/lib/security/cacerts

Example:
/usr/java/jre1.7.0_80/bin/keytool -list -v  -alias "arizonacert" -keystore /usr/java/jre1.7.0_80/lib/security/cacerts

Delete certificates on keystore!

keytool -delete -alias server1.domain.com -keystore server_keystore.jks

DOWNLOAD CERTIFICATE CHAIN AND INPUT INTO A FILE

echo "" | openssl s_client -connect smartwalletstaging.mundipaggone.com:443 -showcerts 2>/dev/null | openssl x509 -out certificado_do_site.cer

rdesktop

rdesktop -u ebsantos -d ecom 100.68.2.40

**Markdown Guide"

Markdown Guide

Quit Telnet stuck session

To exit the telnet session, type CTRL + ]