This is a guide to setting up a Linux machine using Ansible or a manual process
This post machine setup (especially the work setup) assumes you have a memory stick with the correct configuration files and jar files in the correct directories, and once the initial machine o/s is installed you copy this to the correct locations on the machine BEFORE you run the Ansible code.
You should setup you OWN stick so you can setup multiple hosts easily with the things you need, the Ansible setup described below assumes specific items in specific locations, adjust for your own needs.
Ansible will handle setting up the host machine to include vpn, access to Jenkins etc, proxy settings in Browser, system proxy settings etc, openvpn proxy, look at /etc/resolve.conf /etc/hosts etc.
You MUST install with full disk encryption, not just home directory encryption, so ensure you select the appropriate options for this.
So your disk will look something like the following, note the boot efi partitions do not have to be encrypted, but the entire rest of the disk is, including the swap disk.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 732M 0 part /boot
└─sda3 8:3 0 464.6G 0 part
└─sda3_crypt 253:0 0 464.5G 0 crypt
├─vgmint-root 253:1 0 463.6G 0 lvm /run/timeshift/backup
└─vgmint-swap_1 253:2 0 980M 0 lvm [SWAP]Burn image of required install onto USB, boot off the USB, start the o/s and run the Install script.
Apply any updates immediately.
Check System Reports, apply any driver or other updates now.
Now it is time to configure the machine.
The default image will have been created by the disk os setup, but there are a number of dirs and files which should be put in place to create our build. We should copy the files from the memory stick (or another machine) and put these into place to get the expected files in the correct location.
As a minimum one should copy from the memory stick or other machine the following, all from top level to the destination the appropriate level, which will later allow syncdrives.sh to be used as a daily fast backup:
other directories that are recommended are:
Now these files are in place we can configure the machine.
Go to the setup dir
cd ~/work/home/shellTo make it easy to backup stuff, the syncdrives.ini file will need to have an entry base on the name of the machine, add this now, see Daily Backup.
At this point, if you applied any updates, which ends with a recommendation to reboot first, now is the time to reboot.
You can download and install the whole set of required apps, but there is also a set of Ansible instructions to install:
You will need python pip to install Ansible. On Windows now this is easily installed, simply type python on the command line which will take you to the Microsoft apps store at the Python3 installer page. Linux is easy too, look at the next section.
To read THIS page, use a markdown reader, I like ReText best, works on Linux, Mac and Windows. On Windows just type pip install ReText, for Linux we do install ReText as part of the machine setup, but you can do it before using apt or yum. You can then run ReText with python -m ReText. Failing that Markdown Document Viewer is free and good.
Please note that the install yamls are based around a Debian/Ubuntu Linux release so use apt rather than yum, but the principles are the same for yum, you can easily change if needed.
Optionally, Flatpak is used where available for main and also additional packages. To use the flatpak_setup_ansible.yml you might want to install
ansible-galaxy collection install community.generalInstall Ansible, you may need to install python3-pip first
# optional
pip --version
pip3 --version
sudo apt install python3-pip
# required
sudo apt install ansibleI have used Cygwin, Mobaxterm, git bash and others but now prefer WSL. Install WSL @ Microsoft or Install WSL2 @ OMGUbuntu and from there you can run the Ansible install for WSL which has it’s own files (ending _wsl.yml).
Summary, in an administrator PowerShell
wsl --installYou must reboot once this has completed to complete the Ubuntu install. Then start a UBUNTU shell, or run wsl from the command line (or PowerShell) and follow the Linux commands above.
Chocolatey enables you to install software onto a windows platform from the the command line. To install this software you must start Windows PowerShell as an administrator (right click on application icon and choose ‘Run as an administrator’) and run the command
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))For more details and in case you have issues please see link chocolatey
Go to the setup dir, expected path is provided below, but can be wherever you put it
cd ~/work/home/shell/machine_setupYou run a playbook to execute Ansible instructions, as shown below, the path to the resources is relative to the machine_step dir, so although the base can be anywhere, the structure must be the same.
A selection of productivity apps and settings to enhance the machine.
ansible-playbook -vv --ask-become-pass machine_setup_ansible.ymlor for Windows Subsystem for Linux (WSL), run in a WSL Windows/Linux shell
ansible-playbook -vv --ask-become-pass machine_setup_ansible_wsl.ymlIf the above FAILS and it is the very first time you are running on a new release, it might be that the dependencies have changed or possibly the version requested is not available. It IS worth trying just to re-run the above, as the script will try to resolve dependencies, and if that is the issue a re-run will fix/work.
Addtional tags supported:
To install the lsetup scripts and config files, you need to have the scripts and configs in the correct location, these can be run with
ansible-playbook -vv --ask-become-pass machine_setup_ansible.yml --tags personalThere are some additional personalisation to do for Look-and-Feel via Themes and putting specific icons on the desktop, these can be run with
ansible-playbook -vv --ask-become-pass machine_setup_ansible.yml --tags desktopIf you are making a server which you wish to be able to wakeup via a network call, apply the wakeonlan feature (as long as your machine supports this feature).
sudo ethtool eth0Look for Supports Wake-on: g , as long as there is at least a g in this line, the machine supports wake on lan. Then you can install this feature as shown below.
The below example is for a machine with a primary network interface of enp2s0.
ethtool -s enp2s0 wol gNOTE: This needs some manual configuration to identify the network device to use and change one of the files, see below. Note the script defaults to eth0, but this might not be the right interface for your machine.
ansible-playbook -vv --ask-become-pass machine_setup_ansible.yml --tags wakeonlanIn the file /usr/bin/wakewol, you should check/change (using sudo) the network device is actually the one you use, this you can check via either
ip add | grep BROADCAST | grep UP | grep -v DOWN | cut -d: -f 2or
ifconfig | grep -B 2 192.168 | head -2 | tail -1 | cut -d: -f1You can then use any wakeonlan service to wakeup the machine using the MAC address, e.g. on your client install via
apt install wakeonlanPutting the machine to sleep automatically is also useful to save power, use a cron job for this, you can do it on the command line as your own user for suspend but need to be root for hibernate, for cron set as root. Below I am creating a job to put the server to sleep at 1 am at night.
$ sudo crontab -u root -e
0 1 * * * systemctl suspendA selection of apps I use for work.
ansible-playbook -vv --ask-become-pass work_setup_ansible.ymlAddtional tags supported:
or in a WSL Windows/Linux shell
ansible-playbook -vv --ask-become-pass work_setup_ansible_wsl.ymlA variety of lightweight apps to enhance productivity.
ansible-playbook -vv --ask-become-pass additional_apps_setup_ansible.ymlThe following tags are supported:
A variety of useful audio/visual and game applications.
ansible-playbook -vv --ask-become-pass entertainment_apps_setup_ansible.yml --tags see_belowThe following tags are supported:
Use these with the --tags option.
For Plex setup, remember that when adding folders (disk directories) to a library, you will need read and execute permissions for users otherwise Plex cannot see the directories or files therein, I use the following from the directory above the one I want to change:
sudo find movies -exec chmod 755 {} \;A variety of apps from the Windows o/s which for reasons of productivity or habit I like to use even on Linux.
Install Wine the usual way or the wine_apps Ansible setup will handle this for you.
IMPORTANT - one MUST run winecfg BEFORE you attempt any Wine installations, we try to do this as the FIRST action of the Ansible script once Wine is installed, which the script also tries to do.
ALSO IMPORTANT - the Windows application setup for winecfg is interactive, requiring user input, so expect to click Next or something like that at some point.
ansible-playbook -vv --ask-become-pass wine_apps_setup_ansible.ymlAddtional tags supported:
To install addtional apps for a Web Server, you need to use tags to explicitly state what to install.
The --skip-tags vnc (or other skips) are not required as all components under webserver being set to never and thus explicitly need to be named to be actioned.
The following tags are supported:
Firstly, so you can get onto your server, we setup VNC for a graphical remote login and Security options.
ansible-playbook -vv --ask-become-pass wserver_setup_ansible.yml --tags vnc
ansible-playbook -vv --ask-become-pass wserver_setup_ansible.yml --tags ufwAs part of the VNC setup it will ask you one time for the password to use for VNC, this will be displayed in plain text so you can verify it, do not lose it.
As part of ufw the firewall ufw and also fail2ban are both installed. UFW is configured to allow entry into the Nginx server for http(s) as well as samba traffic and localhost ports.
I find VNC is sometimes unstable and crashes, so have a script which restarts it if it is down, this script runs in a loop, or can be called from cron, example below to run the script from its default location on my box, every 10 minutes, as root,
$ sudo crontab -u root -e
0/10 * * * * /home/theodore/startvnc.shor you can start it running in a loop using the -L parameter, do as root.
For NGINX to be installed on the system.
ansible-playbook -vv --ask-become-pass wserver_setup_ansible.yml --tags nginxFor PHP to work, you will need to configure /etc/nginx/sites-enabled/default and uncomment the FPM version of fastapi with the correct path and version of php. Test the config with nginx -t, and run http://localhost on the server or http://the.server.ip.address from any box.
To setup and use the Apache in a Docker container
ansible-playbook -vv --ask-become-pass wserver_setup_ansible.yml --tags docker
ansible-playbook -vv --ask-become-pass wserver_setup_ansible.yml --tags apacheSome typical commands you can execute follow.
— First time, creates container etc
sudo docker run -d --name apache-server -p 80:80 httpd
sudo docker stop 2aaf5b75eb9c470e4bae0b3ad569f98525d86191a7bce780c6907d2c2f200164
sudo docker start 2aaf5b75eb9c470e4bae0b3ad569f98525d86191a7bce780c6907d2c2f200164
sudo docker container ls— Short name works too
sudo docker stop 2aaf5b75eb9c
sudo docker start 2aaf5b75eb9c— Check persistent data (created during Ansible stage)
sudo docker volume inspect apache-data
[
{
"CreatedAt": "2022-06-22T10:52:06+01:00",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/apache-data/_data",
"Name": "apache-data",
"Options": {},
"Scope": "local"
}
]The Mountpoint is where you can place your index.html and all other site resources as required.
— Redeploy
sudo docker rm 2aaf5b75eb9c
sudo docker run -d --name apache-server -p 80:80 -v apache-data:/usr/local/apache2/htdocs httpd— Container name works too
sudo docker container stop apache-server
sudo docker container start apache-server— See the logs
sudo docker logs apache-serverLets make the machine look like how we like it.
Personally I like the Mint-X and Mint-Y look and feel, but Preferences ==> Appearance allows you to select a different look and feel, further configuration is possible. For now we can install the theme from /home/theodore/work/home/shell/machine_setup/dot_themes, which goes into /home/theodore/.themes , this should have been does as part of the machine_setup, so just open the Install option and select the theme.
Add to menu bar the Workspace Switcher.
Most configuration is handled by Ansible, but some has not been automated.
If you already have connections configured, there is a file you can probably find thus:
find ~/.sqldeveloper/ -name connections.xml
find ~/.sqldeveloper/ -name connections.jsonPut your latest existing copy at this location (or add) to configure your DBs, we have a copy of our xml and json which you can copy to the destination.
something like (depending on the version)
cp ~/work/home/shell/machine_setup/dot_sqldeveloper/connections.json ~/.sqldeveloper/system20.4.1.407.0006/o.jdeveloper.db.connection/connections.jsonIf you already have connections configured, there is a file you can probably find thus:
find ~ -name data-sources.jsonPut your latest existing copy at this location (or add) to configure your DBs, we have a copy of ours which you can copy to the destination. Default seems to be ~/.local/share/DBeaverData/workspace6/General/.dbeaver/data-sources.json
something like
cp ~/work/home/shell/machine_setup/dot_dbeaver/data-sources.json ~/.local/share/DBeaverData/workspace6/General/.dbeaver/data-sources.json Add to toolbar:
Set the startup pages:
As an alternative to Ansible, you can install all required software by hand, typical lists are provided in the following sections.
All the following are best installed via the Package/Software Manager:
sudo apt install jqchoco install jqsudo apt install gitsudo install awsclisudo apt install resolvconfAll the following are best installed via the a direct download, or are already on the memory stick / disk image:
winecfg first)The remainder of the setup info is related to work and project setup, see work_setup.md.html
To check the sites are up and running we should set a cron job for the check script as follows
$ sudo crontab -u root -e
0 */2 * * * /var/www/html/test/checksites.shThe tool syncdrives.sh provides a wrapper for rsync, with configuration on which drives and directories you want backed up, and which directories within a set you also want to ignore.
This tool can copy to or from a source machine or backup device, allowing reliable, controlled and fast synchronisation between remote machines via the use of external movable storage such as a USB device.
There are two level of entries, Machine entries which defines how a specific machines are laid out, and Drive Set entries which defines drive structures and layouts. A combination of these provides flexibility on what level of backup you want to do.
Example
[DESKTOP-NFCPL74]
CHECKLINKS=false
DRIVE1=/drives/c
DRIVE2=/drives/d
DRIVE3="/home/$USERNAME/storengo"
DRIVE4="/home/$USERNAME/work"
[theodore-Latitude-E5450]
CHECKLINKS=true
DRIVE1=/home/$USERNAME
DRIVE2=/home/$USERNAME/edrive
DRIVE3=/media/$USERNAME/6ABC6A2CBC69F34D
DRIVE4="/home/$USERNAME/work"
[PegasusM19]
CHECKLINKS=true
DRIVE1=/home/$USERNAME
DRIVE2="/home/$USERNAME/edrive"
DRIVE3=""
DRIVE4="/home/$USERNAME/work"
[DIRSET_1]
DIRSET="work/ddrive"
EXCLUDEDIR=""
EXCLUDEDIRSET=""
[DIRSET_3]
DIRSET="work/ddrive work/home work/work"
EXCLUDEDIR="tdcs/workspace"
EXCLUDEDIRSET="--exclude tdcs/workspace"
For more examples look at syncdrives.ini
Typical usage.
This is a Dry run (-a is not present), to see the changes on dirset 3 (-3), and will also delete things on the slave drive if not present on the master (the -d), the master drive is always specified, here the local is the master
sh syncdrives.sh -3 -d localSame as above, but the -a makes it actually do the work
sh syncdrives.sh -3 -d -a localYou then ship the drive to the remote machine and run the following on the machine
sh syncdrives.sh -3 -d -a remote