CLI Guidelines and fast Support
Posted on Sun 09 June 2024 in Technology
AWS - Amazon Web Services
AWSCLI V2 via Curl (Linux Command Line)
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
Please look for this page to more info
AWSCLI V2 Official Installation Page
Working with EC2 Instances
Launching EC2 Instance via aws cli command --> Very handy
aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type t2.micro --key-name MyKeyPair --security-group-ids sg-903004f8 --subnet-id subnet-6e7f829e
Terminating Instances
aws ec2 terminate-instances --instance-ids $(aws ec2 describe-instances --filters "Name=instance-state-name,Values=pending,running,stopped,stopping" --query "Reservations[].Instances[].[InstanceId]" --output text | tr '\n' ' '
Terminating all Instances at once per InstanceID
INSTANCE_ID=$(aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | .InstanceId')
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
Killing instances aws-cli Examples
Describing Instances
Discovering the Private/Public Ip
aws ec2 describe-instances --filters Name=tag:Name,Values=$1 --query 'Reservations[].Instances[].PrivateIpAddress' --output text
aws ec2 describe-instances --filter "Name=tag-key,Values=Name" "Name=tag-value,Values=*$name_tag*" "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*][Tags[?Key=='Name'].Value[],NetworkInterfaces[0].PrivateIpAddresses[0].PrivateIpAddress]" --output text
aws ec2 describe-instances --filters Name=tag:Name,Values=$1 --query 'Reservations[].Instances[].PublicIpAddress' --output text
Working with ECS Clusters
aws ecs list-clusters
aws ecs list-container-instances --cluster dash-operacoes-frontend-EcsCluster-hml
Working with CloudFront Distributions
Retrieving Distribution ID
distribution_ids=$(aws cloudfront list-distributions --query "DistributionList.Items[*].Id" --output text)
Invalidating Distribution ID
aws cloudfront create-invalidation --distribution-id $distribution_id --paths "/*"
Working with AutoScalingGroup
Grep ASG per ResourceID
aws autoscaling describe-auto-scaling-groups | grep ResourceId
Auto Scalling Group - Change Desire Capacity
Scaling down ASG
aws autoscaling set-desired-capacity \
--auto-scaling-group-name Produto-AutoScalingGroup-$Env \
--desired-capacity 0 \
--no-honor-cooldown
Scaling UP ASG
aws autoscaling set-desired-capacity \
--auto-scaling-group-name Produto-AutoScalingGroup-$Env \
--desired-capacity 1 \
--no-honor-cooldown
RDS - Stop DB Instance
aws rds stop-db-instance --db-instance-identifier coredb
Working with AWS Credentials
aws configure list
aws configure credentials
export AWS_PROFILE=profile
aws configure set region us-west-2 --profile profile
You might want to check AWS Vault 99 Designs GitHub page
Working with Lambdas
for i in `aws --region sa-east-1 lambda list-functions | jq --raw-output '.Functions[] | .FunctionArn'` ; do echo $i ;aws --region sa-east-1 lambda list-tags --resource "$i" | grep ManagedBy ; done | wc -l
S3 Storage
aws s3 cp test.txt s3://mybucket/test2.txt
aws s3 cp s3://mybucket/test2.txt $HOME
aws s3 ls | grep bucket
aws s3 rb s3://bucket-name --force
aws s3 sync . s3://bucket/subfolder/
Describing Security Groups
aws ec2 describe-security-groups | jq -M -r -f filter.jq | grep -v "INBOUND" | grep -v "0.0.0.0" | grep -v "80" | grep -v "443
Obs: you need the filter.jq to parsing the output
IaC CloudFormation
aws cloudformation create-stack --template-body file://single-instance.yml --stack-name
k8-master --parameters ParameterKey=KeyName,ParameterValue=tutorial ParameterKey=InstanceType,
ParameterValue=t2.micro
aws cloudformation list-exports
aws cloudformation delete-stack --stack-name jenkins-cluster-ecs-negociacao --region us-east-1
aws cloudformation describe-stack-events --stack-name NegociacaoServiceStack --region us-east-1
DYNAMODB
aws dynamodb create-table --cli-input-json file://security-manager.json
CODECOMMIT
aws codecommit list-repositories --region us-east-1 | grep pontos
aws codecommit delete-repository --repository-name "dash-operacoes-front" --region sa-east-1
aws codecommit create-repository --repository-name "td-posicao-data-stream" --region us-east-1
aws codecommit get-repository --repository-name "pontos-pi-infra" --region us-east-1
FOR LOOP - Managing repositories with AWSCLI
for i in `cat repos.txt`; do aws codecommit create-repository --repository-name "$i" --region us-east-1; sleep 1; clear; done
for i in `cat repos.txt`; do aws codecommit delete-repository --repository-name "$i" --region us-east-1; sleep 1; clear; done
for i in `cat repos.txt`; do aws codecommit delete-repository --repository-name "$i" --region sa-east-1; sleep 1; clear; done
for i in `cat repos.txt`; do aws codecommit list-repositories "$i" --region us-east-1; sleep 1; clear; done
KUBERNETES
kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
export KUBECONFIG=$KUBECONFIG:~/.kube/config-k8s-ci
echo 'export KUBECONFIG=$KUBECONFIG:~/.kube/config-k8s-ci' >> ~/.bashrc
echo 'export KUBECONFIG=$KUBECONFIG:~/.kube/config-k8s-ci' >> ~/.zshrc
echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc
AWSCLI Updating EKS Context
aws eks --region us-east-1 update-kubeconfig --name k8s-ci --profile greenbrasil
Printf
printf $(kubectl get secret --namespace default jenkins-cloudbees-jenkins-distribution -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
ECR Registry Tag Mutability
Very nice reading regarding scanning and tagging your images pushed to ECR aws registry repository!
Kubernetes Complete Guide
Very easy going reading about Kubernetes and its concepts!
Check it out:
CKA, CKD, CKS Kubernetes Certification
Linux Foundation CKA, CKD, CKS
LINUX
SSH StrictHostKeyChecking
EC2INSTANCE="`aws ec2 describe-instances --filters Name=tag:Name,Values=Negociacao-AutoScalingGroup-prd --query 'Reservations[].Instances[].PublicIpAddress' --output text`"
ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=10 -i $HOME/google-drive-somospi/AWS/key-pair/key.pem ubuntu@$EC2INSTANCE
AWK + tr -d
VARPUBLIC=`aws ec2 describe-instances --instance-ids $EC2INSTANCE | grep PublicIpAddress | awk ' { print $2 }'` && echo $VARPUBLIC | tr -d '",'
echo "ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=10 -i /home/bernardo/GoogleDrive/AWS/key-pair/key.pem ubuntu@"$VARPUBLIC""
echo "ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=10 -i /home/bernardo/GoogleDrive/AWS/key-pair/key.pem ubuntu@"$VARPUBLIC"" > ssh.sh
chmod +x ssh.sh
./ssh.sh
Get the Internet ISP IP via cli
wget -qO- http://ipecho.net/plain | xargs echo
SYSTEMD
SYSTEMCTL (Ubuntu/RedHat/CentOS) usage
To enable or disable services automatically on CentOS or RedHat Flavors.
systemctl disable httpd
systemctl enable docker
systemctl status kubelet
systemctl grep | grep running
systemctl list-unit-files | grep enabled
Control Commands
hostnamectl
Static hostname: snake09
Icon name: computer-vm
Chassis: vm
Machine ID: 0d8a36db78064458a56e691770e08ab5
Boot ID: 958d267a172a42a3ac515c941dee5a57
Virtualization: oracle
Operating System: Ubuntu 18.04.2 LTS
Kernel: Linux 4.15.0-54-generic
Architecture: x86-64
timedatectl
Local time: Tue 2019-07-16 18:12:38 UTC
Universal time: Tue 2019-07-16 18:12:38 UTC
RTC time: Tue 2019-07-16 18:12:37
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
systemd-timesyncd.service active: yes
RTC in local TZ: no
localctl
System Locale: LANG=C.UTF-8
VC Keymap: n/a
X11 Layout: us
X11 Model: pc105
Kill other user session
Verify how many users are logged in with the command "w" (who)
Sample answer:
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
vagrant pts/0 192.168.0.20 19:55 0.00s 0.27s 0.00s w
vagrant pts/1 192.168.0.20 20:17 2.00s 0.05s 0.05s -zsh
ps -ef | grep pts/1
Once you have the PID just kill -9 $PID! that's it
or you can try with AWK command
ps aux | grep vagrant | awk '{print $2}' | xargs sudo kill -9
Find below fewer examples about it:
Link to change the timezone* (CentOS)
Linux Academy Changing the timezone in your system
GIT Command line
Creating remote repo through command line
git init
git add .
git commit -m "first commit"
git remote add origin https://github.com/Eddie-Uncle/Test1.git
git push -u origin master
Merging branches
$ git init
$ echo "something" > new_file.txt
$ git add .
$ git checkout -b master
$ git commit –m "First commit message"
$ git checkout -b new_branch
$ echo "appending something" >> new_file.txt
$ git add .
$ git commit -m "merge"
$ git checkout master
$ Switched to branch 'master'
$ git merge new_branch
Delete git — Locally and remote
git checkout master
git checkout -b edsonnewbranch
git branch -d Test_Branch # (Delete locally desired branch)
git branch -D Test_Branch # (Delete locally force option)
git push origin --delete eddie-branch # (git push origin --delete branch to be deleted) (Delete remote branch)
git pull / push commands
Pulling code from Remote repo to local repo:
git pull origin master || git pull origin edsonnewbranch
git pull --all || (all branches)
Pushing code from Local repo to remote repo:
git push origin master || git push origin edsonnewbranch
Git clone
Default clone
git clone git@github.com:Eddie-Uncle/linux-automation.git
Specific branch
git clone --branch ubuntu git@github.com:Eddie-Uncle/linux-automation.git
Git set Upstream
git branch --set-upstream-to=origin/master master
git operations
git log -p
git status
git show
git credentials and user settings
git config --global user.name Eddie-Uncle
git config --global user.email bernardsp@gmail.com
git config --global core.editor vim
git config --global merge.tool vimdiff
git config --list
git config --global --edit
Save credentials - Store Git Credentials
git config --global credential.helper store
$ git push http://example.com/repo.git
Username: type your username
Password: type your password
SELINUX APACHE
semanage port -l | grep http
semanage port -a -t http_port_t -p tcp 80
sestatus
Working Ip
ip addr show eth0 | grep -oP '(?<=inet\s)\d+(\.\d+){3}'
Working with Native CENTOS 7 Firewall
Using iptables behind the scenes!
(CENTOS)
sudo firewall-cmd --zone=public --add-service=http --permanent
sudo firewall-cmd --zone-public --add-port=8080/tcp --permanent
sudo firewall-cmd --zone=public --list-services
rpm qa | grep firewalld
UFW (Ubuntu)
sudo ufw status verbose
sudo ufw allow from 192.168.0.0/24 to any port 8080
Files Handling
LSOF usage and examples
echo "OpenSockets="$(/usr/bin/lsof | grep -i weblogic | wc -l) > results.txt
sudo sh -c "echo '192.168.0.22' > /etc/hosts”
/usr/sbin/lsof | grep -i weblogic
/usr/sbin/lsof | grep -i weblogic | grep -i deleted
and then kill stuck pid's!
/usr/sbin/lsof | grep weblogic | awk '{ print $3 }' | sort -u
lsof -u weblogic | wc -l
/usr/sbin/lsof | grep weblogic | awk '{ print $2 }' | sort -u
[root@ares ~]# cat /proc/sys/fs/file-max
65536
Kill -9 expert usage
sudo kill -9 $(ps -ef | grep -i -E "openvpn" | grep -v grep | awk '{print $2}')
AWK and CUT usage
ps -ef | grep java | awk ‘{ print $2 }’ | sort -u | xargs kill -9
ps - ef | grep java | cut -d " " -f2 | xargs kill -9
ps -ef | grep weblogic | awk '{ print $2 }' | sort -u | xargs kill -9
kill -9 $(ps auwwwx | grep -i -E "jboss|weblogic|websphere" | grep -v grep | awk '{ print $2 }')
SED
sed -i 's/10.13.0.87/10.13.0.193/g' *.xml
sed -i ’s/8080/8080/g' *.xml
sed -i 's/server01:10110/server02:60101/g' appconfig.properties
var=value_of_the_var
remove
sed '/myhost=.*/d' -i $var
sed '/exec $JAVA.*/d' -i $var
#replace at the line 28 the empty space fo the value of myhost variable
sed -e '28 s/^/myhost=$(hostname -f)\n\n/' -i $var
sed -e '8 s/^/JAVA_OPTS="$JAVA_OPTS -javaagent:\/opt\/sysmgmt\/AppDynamics\/Java_Agent\/ver4.4.3\/javaagent.jar"\n/' -i $var
#replace after searched term found
sed '/Launching /a \exec $JAVA -classpath "$classpath" -Xms$min_mem -Xmx$max_mem' -i $var
SED ON MACOS - FreeBSD
sed -i'' -e 's/sa-east-1/us-east-1/g' *.yaml
RSYNC - Examples
Copy files or folders from source to destination (incremental copy)
rsync -auvzh --progress OneDrive/* /tmp/OneDrive
Delete files at destination if it is not present in source (--delete)
rsync -avzh --delete --progress OneDrive/ /tmp/OneDrive
More examples online:
GREP and PGREP usage
grep <search> <file> | awk '{print $1}' >> <new_file_output>.txt
pgrep -u hybris java
pgrep -u apigee java | wc -l
usage: pgrep -u <user> <language Process>
example: grep -ir Carrefour *.sh | awk '{print $1}' >> search.txt
FOR LOOP USAGE
for x in `cat name_file`;do echo $x;done
for x in `cat name_file`;do ssh -q $x;done
for i in `cat file.txt` ; do ping -c1 $i 2>&1 | tee >> ping - output.txt; done
(Creating serveral files at a glance)
for i in {1..9}; do touch kubn40${i}.sh; done
usage: for i in {range..}; do command filename$${i}.txt; done
for i in {1..9}; do cat kubn410.sh >> kubn40${i}.sh; done
DU Usage
du -sk * | sort -n
du -sh /logs/* | sort -nr | grep G
FIND Usage
find . -size +50M
find / -type f -name "index.php"
find / -type f -name "*.log" -size +1M
find / -type f -name "*.log" -size +1M | xargs rm -rf {} \;
find . -iname "server.xml"
Quick GREP and Tailing Logs (examples)
tail -f *.out | grep "Socket read timed out" | wc -l
tail -f *.out | grep "Socket read timed out" --color
tail -f *.out | grep "error" --color
cat bwbpelsp02_01-yyyy-MM-dd-HH-mm.log | grep "Socket read timedout" | wc -l
tail -f *.out | egrep -i 'ERR|ORA-|OVERL|SUSPEN'
tail -f *.out | egrep -i 'STUCK'
THREAD COUNT BY USER
ps -uweblogic -lf | wc -l
ps -uapache -lf | wc -l
ps -unginx -lf | wc -l
CHOWN and CHMOD smart usage
chown apigee: logs/ -R || chown apigee:apigee -R logs/
chmod +x shellscript.sh || chmod 755 shellscript.sh
Verify and count a specific java class
grep com.yourcompany.product.exception.ServiceException log.out | wc -l
result = 11466
DMESG (old fashion)
[root@bwsubacat04 sitecontent]# dmesg | grep file-max
VFS: file-max limit 65536 reached
VFS: file-max limit 65536 reached
ENCODE LINUX
echo $LANG
LIMITS - SO LINUX (CENTOS 6)
LIMITS
echo '###WEBLOGIC###' >> /etc/security/limits.conf
echo 'weblogic soft nproc 20000' >> /etc/security/limits.conf
echo 'weblogic hard nproc 20000' >> /etc/security/limits.conf
echo 'weblogic soft nofile 65536' >> /etc/security/limits.conf
echo 'weblogic hard nofile 65536' >> /etc/security/limits.conf
echo 'weblogic soft memlock 16384000' >> /etc/security/limits.conf
echo 'weblogic hard memlock 16384000' >> /etc/security/limits.conf
IFCONFIG MTU
ifconfig ib0 | grep MTU; ifconfig ib1 | grep MTU; ifconfig bond0 | grep MTU
CHECK - LDAP USER
getent passwd | grep <user>
Processors Statistics
mpstat
sar 1 10
top -u
sar -d
vmstat 1 10
iostat
Basic example sudores file (there are better practices, in this case these users are allowed to excecute as root without password - General overview)
SUDOERS
weblogic ALL=(ALL) NOPASSWD: /bin/vi /etc/hosts
fmwadmin ALL=(ALL) NOPASSWD: /bin/vi /etc/hosts
VI (vim) Hacks
.vimrc
edit .vimrc under $HOME/ folder or just create a new one, then include these lines below:
set number
syntax on
set tabstop=2
set autoindent
color desert
Undo vim
press the keyboard "ESC + u (undo)
Reverse Columns within a file
:%!awk '{print $2, $1}' --> or vice-versa
Saving without exit
After modify the file desidered, type
:w <the name of the new file> and then hit the enter button
CRONTAB
crontab -l (Exhibits the current crontab created)
crontab -e (Create a crontab under running user)
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
Any questions, please click link below:
Tee usage (similar to echo)
tee /etc/yum.repos.d/docker.repo <<-EOF
Obtain PID specific application
ps aux | awk -v app='Apache' '$0 ~ app { print $1 }’
Working with Aliases
alias
alias vi='vim'
alias cp='cp -i'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias mv='mv -i'
alias rm='rm -i'
alias k='kubectl'
alias python='/usr/sbin/python3'
USING CURL & WGET
Using Post method with cuRL
curl -i -X POST -H "Content-Type:application/xml" -d '<movementRequestJson><sku>3213219</sku><seller>13933305000106</seller><totalQuantity>1</totalQuantity><leadTime>0</leadTime><subInventory>9801</subInventory><warehouse>98</warehouse><stockType>FISICO</stockType></movementRequestJson>' http://company-stock-v2.elasticbeanstalk.com/stock
Downloading Files
curl -O https://download.virtualbox.org/virtualbox/5.2.30/VirtualBox-5.2.30-130521-OSX.dmg
Web Examples for CURL usage
<a href="http://www.codingpedia.org/ama/how-to-test-a-rest-api-from-command-line-with-curl/" target="_blank">Curl and examples GET POST All methods</a>
Using WGET - This case downloading JDK 1.8u51 from Oracle
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u151-b12/e758a0de34e24606bca991d704f6dcbf/jdk-8u151-linux-x64.tar.gz"
Java download examples:
StackOverflow wget java downloads
Installing JAVA Runtime - OpenJDK using apt install
apt install default-jre
apt install openjdk-11-jre-headless
apt install openjdk-8-jre-headless
apt install openjdk-9-jre-headless
Backup using shell script (this example weblogic domain)
Backup I - Sample
#!/bin/bash -e
set -euf -o pipefail
DATA=`date +%Y-%m-%d-%H.%M`
tar -zcvf /repo/bpel/backup/BPEL/backup-config-"$DATA".tar.gz /domains/BPEL/config
tar -zcvf /repo/bpel/backup/BPEL/backup-security-"$DATA".tar.gz /domains/BPEL/servers/bwBPEL_Admin/security
tar -zcvf /repo/bpel/backup/BPEL/backup-ldap-"$DATA".tar.gz /domains/BPEL/servers/bwBPEL_Admin/data/ldap
clear
echo "Backup successfully..."
exit
Backup II - Sample - excluding tarballs
#!/bin/bash -e
set -euf -o pipefail
echo "Starting Backup"
DATE=`date +%Y-%m-%d-%H.%M.$$`
echo "Starting Backup, except *.tar extensions"
tar -zcvf blog-"${DATE}".tar.gz --exclude='*.tar' .
if [ "$?" -eq "0" ]
then
echo "Backup Successfully..."
exit 0
else
echo "Backup has failed, please check"
exit 1
fi
Working with Certificates
Import certificate - Using Java keytool
keytool -import -alias mundipagg -file CA_b2w_cert_test.cer -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -trustcacerts
Listing Certificates
./keytool -list -v -keystore $JAVA_HOME/jre/lib/security/cacerts
Example:
/usr/java/jre1.7.0_80/bin/keytool -list -v -alias "arizonacert" -keystore /usr/java/jre1.7.0_80/lib/security/cacerts
Delete certificates on keystore!
keytool -delete -alias server1.domain.com -keystore server_keystore.jks
DOWNLOAD CERTIFICATE CHAIN AND INPUT INTO A FILE
echo "" | openssl s_client -connect smartwalletstaging.mundipaggone.com:443 -showcerts 2>/dev/null | openssl x509 -out certificado_do_site.cer
rdesktop
rdesktop -u ebsantos -d ecom 100.68.2.40
**Markdown Guide"
Quit Telnet stuck session
To exit the telnet session, type CTRL + ]