otrdiena, 2024. gada 27. augusts

Hard questions of server monitoring via Zabbix

How to properly check of host and some items availability. 

In this article will be covered
  • Availability monitoring by DNS name 
  • EXE monitoring
  • HDD monitoring
  • Forecasting function Timeleft

Did remember old soviet cartoon about python, elephant, monkey and parrot? Python wanted to know, how long he is. Monkey said python is 4/4 of python, but python disagree. Then they asked to parrot and parrot said  python is 38,25 parrots long. Python was happy and now if I don't know what units to measure something I put in parrots or ppg.
Availability monitoring is required not from Zabbix server , but from  host1-> host2:port
In 2012 I ordered exe file to programmer , that exe just tried to ping port from host,  by example SQL server available from web server. Nowadays you can just use Zabbix Net.tcp.service, just don’t forget units PPG. It appears net.tcp.service is available since version 2. Some people don't know that net.tcp.service can do remote ip and port checks too. In most cases it is enough, but actually it shows if port is open from particular host. You can add powershell check in Zabbix agent and get similar result. Add 
UserParameter=CheckDnsAndPort,powershell.exe -NoProfile -ExecutionPolicy bypass Test-NetConnection PYTHON-WEB.CFLA.GOV.LV -Port 443 -InformationLevel Quiet
Parameter Quiet at the end returns only one word "True" or "False"


https://www.zabbix.com/documentation/4.0/en/manual/config/items/itemtypes/simple_checks  It appears that net.tcp.service function can do DNS checks. I find it out just because I was preparing to presentation and double check all that I wrote., dont know when this feature appears but it works so all my previos job, sophsticated scripts, clever powershell scripts goes to trash. In documentation there is no reference to it, so maybe Zabbix guys don't know or forget. See the manual  net.tcp.service[service,<ip>,<port>]   https://www.zabbix.com/documentation/6.0/en/manual/appendix/items/supported_by_platform  There are pointed <ip>, I just tried DNS for curiosity and it worked! As it is not documented checking by dns not sure if it disappears one day as it appeared.

Net.Tcp.Service.perf[tcp,python.yourdomain.com,443]

Zabbix do what? EXE file monitoring

Last year after Zabbix conference I met old  colleague and told  I am monitoring a RAM usage by EXE from Zabbix and he said «What?» like Eminem so I figured out lot of admins don't know all capabilites of Zabbix.  12 years ago it was hard job to enumerate all processes by procmon, add custom config, now it is just standard template.
Don't forget to Add Custom multiplier to get proper bytes.


True or False- Microsoft SQL eats all RAM? As you can see on picture there are 32 GB of RAM but Microsoft SQL does not pretend on more than 22 GB of RAM. He keeps what he get it but does not take more.


HDD monitoring



Some admins thinks they can install out of the box template and it will work fine. NO! If we show in dashboard only space usage we could get a wrong impression about volume space, as you can see there are alert Disk space is low >90% red line. Zabbix use dynamic (delta) view but if we dive deeper

to monitor together total space and used space we see, red line D disk is filling very slowly so we can leave it Admin overreacted by buying 300GB expensive DATACENTER storage In Azure it will cost You 1470$ in AWS 1008$. Check for Yourselves. https://azure.microsoft.com/en-us/pricing/details/managed-disks/  and https://aws.amazon.com/ebs/pricing/ In next saection I will explain how I calculated that my admin overreacted and disk will be full in 7 years.

Predictive functions

There are 2 predictive functions in Zabbix- Timeleft and Forecast. I am using only Timeleft.
Timeleft function dont always work fine, to get accurate data You need to keep data of disk usage for 6 month or more.  I am using only 6 days of data because default templates out of the box, keeps only 7 days of data so to get more accurate data You have to change it. 


You can add some alert and get noted like this. 

Conclusion

To better monitor server availability
  • Use DNS name check from remote host to port instead of IP from Zabbix server
  • Use Timeleft instead of Disk almost full 
  • Use EXE file monitoring instead of service state


There are article and formulas that make my headache stronger. 
https://www.zabbix.com/documentation/5.0/assets/en/manual/config/triggers/prediction_docs.pdf
You don't have to use formula I just made 2 templates You can use to work with timeleft and dns check.  Here You find Windows Template to monitor EXE, DNS and TimeLeft and Linux template to check Timeleft for / /home and /root volumes.
https://github.com/giorsgeks/ZebbixTemplates/tree/main


otrdiena, 2021. gada 5. oktobris

Dell PowerEdge R510 legacy offline firmware update upgrade and ESXi install

This is my personal notes on Dell PowerEdge R510 firmware upgrade, when DRAC is not an option.
Catalog update is not working anymore as they are out of support, so R510 is likely not added to catalog.
You need 3 disks
LiveUpdate disk
SBU disk
https://downloads.dell.com/secure/FOLDER02670866M/1/sbuu_2.4_1294_A01.iso?authkey=E7dWOBNFc44tYlgQGdZy+wXasroZI9iUJIyhAdL7cTcprGIoMA6BVHhSwjQDW8zR&__gda__=1527074517_4e43366bc9da032f87b1e95560f8eae7&uid=d50b5da9-4dc0-4a29-2254-9e7cfa15ef67
1. SUU disk
https://downloads.dell.com/FOLDER04240219M/1/SUU-WIN32_17.04.200.145.ISO?uid=9c02226f-92e9-41b3-1a9d-aae3d9e57ee7&fn=SUU-WIN32_17.04.200.145.ISO
Extract content of SUU repository to USB flash drive
2. Run computer with SBU disk, Point as repository added flash disk
Check which drivers you need
Make prinstcreen or smthing elseOpenManage Disk
Go to
http://www.dell.com/support/home/us/en/19/drivers/driversdetails?driverId=C31J4
Run LiveCD
From LiveCD open fladh doisk in terminal, run drivers with command sh ./drivername.bin
Thats it
Dont forget SATA disk firmware must be upgraded before formatting RAID partition.


To add inner DVD you have to open BIOS and enable latest SATA device.

service NetworkManager stop
ifconfig em2 192.168.1.2 netmask 255.255.255.0
route add default gw 192.168.1.1
service em2 NetworkManager start
curl -s http://linux.dell.com/repo/hardware/dsu/bootstrap.cgi | bash
./dsu_Install.sh


Upgrade BIOS 1.9.0 to 1.13.0
PERC H700 nevar palaist kamēr nav pabeidzies initialize

http://linux.dell.com/repo/hardware/latest/
Appears that latest working VMware are ESXi 5.5.

mount
lasty will be usb drive

cd to usb drive

execute sh ./nameofdriver.bin

http://www.dell.com/support/home/us/en/19/drivers/driversdetails?driverId=V38WK
 | Seagate STS3600057SS


sudo mount -t ntfs -o rw,auto,defaults,exec /dev/disk/by-label/Data /mnt/Data

right click run in terminal sudo sh ./driver.bin

got licence
https://my.vmware.com/en/group/vmware/evalcenter?p=free-esxi5
got VMware client
http://vsphereclient.vmware.com/vsphereclient/4/0/3/2/3/6/5/VMware-viclient-all-5.5.0-4032365.exe
Due to suspend of support, links to VSphere client is not wqorking anymore. Hovewer all vmware clients ar available.
http://www.vmwarearena.com/download-link-for-all-versions-of-vmware-vsphere-client/


SUU GUI fails to open in Red Hat Enterprise Linux 7.2 operating system
If SUU GUI, is unable to launch in Red Hat Enterprise Linux 7.2 (64–bit) operating system perform the following -

Run the command in the terminal:
#echo -e "Section \"Device\"\nIdentifier \"Device0\"\nOption \"SWCursor\" \"True\"\nEndSection" > /etc/X11/xorg.conf.d/xorg.conf
On completion of the command, ensure that the contents are:
#cat /etc/X11/xorg.conf.d/xorg.conf
Section "Device"
Identifier "Edifice"
Option "SWCursor" "True"
EndSection
Type the command, #systemctl restart gdm.service to restart the GDM service.
The user will be logged off after running the command.

ceturtdiena, 2020. gada 6. augusts

Ubuntu 20 how to increase swapfile

After installing server, I increased RAM, but swap file remains the same, so I have to increase itrmanually. I increased RAM from $ to 8 GB so I have to increase swap file accordingly. In Ubuntu 20.04 swapfile out of the box is called swap.img not swapfile, so the commands will be as follows: 
  • type sudo su so you don't have to type sudo before each command
ubuntu@grayflow:~$ sudo su
[sudo] password for ubuntu:
  • check swap size and name
root@grayflow:/home/ubuntu# swapon --show
NAME      TYPE SIZE USED PRIO
/swap.img file   4G 2.1G   -2
  • turn off swap
root@grayflow:/home/ubuntu# swapoff -a
  • increase swap file
root@grayflow:/home/ubuntu# dd if=/dev/zero of=/swap.img bs=1M count=8192
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 131.193 s, 65.5 MB/s
  • turn on swapping
root@grayflow:/home/ubuntu# mkswap /swap.img
Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
no label, UUID=a1dfcfc3-577d-4ed2-b706-f7c8eeeec69f
root@grayflow:/home/ubuntu# swapon /swap.img
root@grayflow:/home/ubuntu# swapon -s
Filename                                Type            Size    Used    Priority
/swap.img                               file            8388604 780     -2

trešdiena, 2019. gada 10. aprīlis

How to install calculator and photoviewer if Windows store is disabled

If Windows store is disabled in AD, Windows photoviewer and calculator are not available for users. To install photoviewer and calculator. It works when Calculator and photoviewer are installed only for local users and need to be Registered for AD users.
From User context
Get-AppxPackage -User $env:UserName *photos* | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register "$($_.InstallLocation)\\AppXManifest.xml"} 
From Computer context
Get-AppxPackage -AllUsers *windowscalculator* | Foreach {Add-AppxPackage -DisableDevelopmentMode -Register "$($_.InstallLocation)\\AppXManifest.xml"}
Can be done via Group policy.

otrdiena, 2019. gada 12. marts

Powershell script to Get all AD users in unicode UTF8 format



Script is mention to run on Domain Controller, otherwise you need to enable PowerShell and Import Active Directory Modules.
c:\temp directory must be created before running script
At first you need to read aut all properties to determine, which you want to see in final list.

Get-ADUser -identity giors -properties *|Export-CSV c:\tmp\giors.csv 

where giors is my username. I need to determine, which user accounts is expired so I add LastLogonDate and PasswordLastSet
Simple Export-CSV export in plain ANSI text, to convert to UTF8 add -Encoding UTF8

Get-ADUser -Filter * -SearchBase "DC=domain,DC=com" -properties *|select CanonicalName,SamAccountName,Name,LastLogonDate,LockedOut,PasswordLastSet,whenChanged|Export-CSV -Encoding UTF8 c:\tmp\allusersinUTF.
csv


ceturtdiena, 2018. gada 12. jūlijs

ESXI 5.5 backup and restore with ghettoVCB script

ESXI backup/restore with ghettoVCB script

Valid upon version ESXi 5.5
Before make sure ssh is enabled on your VMware host. SSH and ESXi shell must be enabled.
It would be good idea to add restrictions in VMware firewall to access ssh only from admin computers. ESXi remote shell must be used to create access key, or you have to enter password every time you connect. 

Backup

Backup Plan is based on document  https://communities.vmware.com/docs/DOC-8760. Download source from Github to your local computer.
https://github.com/lamw/ghettoVCB/downloads Unzip and download to esxi.
Conf file show count of backups, leave 50% of disk space free for restoration purposes
/ghettoVCB-master/ghettoVCB.conf.
VM_BACKUP_ROTATION_COUNT=1
Write in crontab, when copies will be created. In my example backup take place every wednesday at 6:16 pm. Add to  /var/spool/cron/crontabs/root.
18  18    *   *   3  /ghettoVCB-master/ghettoVCB.sh -a -g /ghettoVCB-master/ghettoVCB.conf
~
~ # kill -HUP $(cat /var/run/crond.pid)
~ # /usr/lib/vmware/busybox/bin/busybox crond
After esxi restart cronjob will be erased so it must be added on boot. Add to /etc/rc.local.d/local.sh.
/bin/kill $(cat /var/run/crond.pid)
/bin/echo "18  18    *   *   4  /ghettoVCB-master/ghettoVCB.sh -a -g /ghettoVCB-master/ghettoVCB.conf" >> /var/spool/cron/crontabs/root                                                   
/usr/lib/vmware/busybox/bin/busybox crond

Restore

/ghettoVCB-master/ghettoVCB-restore.sh -c /ghettoVCB-master/vms_to_restore
In file /ghettoVCB-master/vms_to_restore put a location of backup and where to restore.
"/vmfs/volumes/5b05449c-c4081b22-b538-842b2b780912/Backup/Graylog/Graylog-2018-07-12_18-18-02;/vmfs/volumes/5b05449c-c4081b22-b538-842b2b780912;1"
Restoration will not occur if folder with same directory already exists.

trešdiena, 2018. gada 16. maijs

Get File Hash on Windows PowerShell

My instructions are for Windows 10, but I am sure it will work on every Windows wirh powershell.
It could be useful for security updates and suspicious executables.
Open PowerShell Windows key->Type PowerShell
Move to the folder where file are placed
cd c:\folder
where your file are located
Get-FileHash .\filename.extension -Algorithm SHA256
or 
Get-FileHash .\filename.extension -Algorithm SHA1
Lets say wou want to check Gimp downloaded in c:\temp
Get-FileHash -Path c:\temp\gimp-2.10.0-setup-2.exe -Algorithm SHA256

Thats it.

ceturtdiena, 2018. gada 1. februāris

Add SSL to Apache site

Most instructions are for usual http sites but now it is almost standart to use ssl on your site, especially if you require client login. My description is for simple web site located in lan where you require port 443 and ssl certificate.
mkdir keys
 cd keys

Create response file

touch csr_details.txt
nano csr_details.txt

trešdiena, 2017. gada 20. decembris

How to extend linux vm drive and partition and recreate swap file without gparted and without gui

Preface.

 HDD which is actually bunch of files in vm, emulating HDD are extended via VMwareVCenter. I need to increase my primary partition in Ubuntu Linux VMware virtual machine. In other words  "/" must be extended.  In this example parted and fdisk which are preinstalled in all modern linux systems will be used. 

How to extend root partition /

First we need to extend device. In my situation there are problem- after device /dev/sda1 are /dev/sda2 so I need firstly to move /dev/sda2 to the end of HDD partition.
Fdisk shows I can increase /dev/sda2 but cannot /dev/sda1
root@Log:/# fdisk -l
Disk /dev/sda: 450 GiB, 483183820800 bytes, 943718400 sectors
Device     Boot     Start       End   Sectors   Size Id Type
/dev/sda1  *         2048 615234375 615232328 293.4G 83 Linux
/dev/sda2       617187328 629145599  11958272   5.7G 82 Linux swap / Solaris
As you can see there are 943 mil sectors but only 629 occupied.
So I first need to move swap to the end of disk.

Move swap partition

All commands must be typed as root user, in ubuntu type sudo su or type sudo before each command. Before deleting swap partition, stop all active applications like database and web servers. After moving swap partition must be at the end of disk /dev/sda
Type fdisk /dev/sda and then type
p to print info about partitions
d to delete swap partition
2 number of partition
n to create new partition
p to make partition primary
2 number of partition
calculate sectors, partition take, subtract from the last sector and provide to numbers First sector of partition and last sector. Some spreadsheet could help.