How to fix Windows 10 boot after partition resize and change in number of partitions?

While inside Linux, if you use GParted to modify partitions (e.g., resize/insert/delete partitions), your existing Windows might not be able to boot. This tutorial help you recover Windows boot record for GPT disks.

Below are the steps:

1. Make sure your partitions are indexed in sequence, if you have inserted/deleted partitions, you will end up with `/dev/sda[1245]` without 3 or `/dev/nvme0n1p[1345]` without 2, use Linux command `gdisk` to re-index all the partitions on the GPT disk. Somehow, Windows bootloader does not boot if partitions are not in order.

2. Boot into any Windows installation CD and run the repair, usually this will not work (but if it works, then a good stop). Enter the command prompt in advanced settings.

3. Make sure both Windows main drive (C:) and EFI partition is assigned a drive letter. If not, run `diskpart`:

list vol
list disk
select disk #X
select partition #Y
assign letter=H
exit


4. Enter EFI partition H: and delete (or move to backup) existing EFI boot record for Windows:

rmdir /s EFI\Microsoft EFI\Boot

5. Rebuild EFI boot record

bcdboot C:\Windows /s H: /f UEFI

6. (optional) If you just want to rebuild BCD entry (without rebuild the entire EFI boot entry), delete (or move to backup) the existing BCD file in EFI/Microsoft/Boot and run:

bootrec /rebuildbcd


How to clone an fully installed Linux system to another computer, or do live offline system upgrade?

A. Full-system Clone

When managing Linux server cluster, very often we need to clone a fully-installed Linux system to other computers/server so that we do not need to re-install all required packages and libraries, and do not need to reconfigure some packages such as inputrc, vimrc, bashrc, etc. Here, I will describe two common methods:

1. The recommended way is to use MX-Linux's mx-snapshot

This is a great utility. It can create a Linux-rescue ISO image that is a large bootable live image containing all packages/libraries/configurations, optionally containing home folder contents. At the same time, you can use the Live system to install onto as many harddisks as possible.

Remeber to select "Preserve /home (ext4)" option if you want to keep existing user folders.

2. Manually copy over all folders and setup grub. When copying over all file, you need to preserve file permissions, thus use either "cp -rfPp" or "rsync -avlP", or "tar --numeric-owner -czf"

mount /dev/sda3 /mnt (root partition)
mount /dev/sda2 /mnt/boot (required if separate boot partition)
mount /dev/sda1 /mnt/mnt/efi
mount --bind /dev /mnt/dev
mount --bind /dev/pts /mnt/dev/pts
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys

grub-mkdevicemap
grub-install --efi-directory=/mnt/efi /dev/sda
update-grub
update-initramfs -u -k all 

However, the SUID/SGID/sticky bit will be reset when copying over directories or extracting archives (even if the preserve-permission option is set), so you need to manually redo setting them afterwards. The following list of commands need to have SUID bit set: passwd, su, sudo, ping*, chsh, mount, umount, fusermount, etc.

B. Live Offline System Upgrade

For offline live upgrade (while all other users are still using the system), Method A1 will introduce a very long down-time (typically a few hours, depending on the size of your fully-installed system), as the installation requires booting into the live system while other users cannot access. So we typically use Method A2, for which the only downtime is the server reboot. The steps are as follows:

1. In a running Linux system, create two new folders under /, e.g., /full-backup /full-upgrade.

2. Copy over the entire new-OS root system folders (i.e., /etc, /bin, /sbin, /usr, /opt, /root, /var, /lib, /lib64, /boot, etc., except /home, /dev, /sys, /mnt, /proc, /run, etc.) from USB storage to /full-upgrade using Method A2 ("cp -rfPp" or "rsync -avlP", or "tar --numeric-owner -czf"). This typically takes a few hours.

3. Copy over statically-linked busybox to the root folder (the version of busybox must not require ld-linux-x86-64.so interpreter). Typically, this file should already be prepared somewhere inside /full-upgrade.

root@my-laptop:~# file /usr/bin/busybox
/usr/bin/busybox: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=36c64fc4707a00db11657009501f026401385933, for GNU/Linux 3.2.0, stripped

4. Copy over credential/configuration files from / to /full-upgrade, e.g.,

    /etc/passwd, /etc/groups, /etc/shadow, (do NOT directly copy over, only copy over real user entries because you need to keep the user IDs for service users such as gdm/sshd/_apt/openvpn/lp/etc., otherwise, these services will not function properly)
    /etc/network/interfaces, /etc/NetworkManager/*, /etc/openvpn, /etc/audit, /etc/motd, /etc/logrotate.*
    /etc/fstab, /etc/exports, /etc/hostname, /root/.ssh, etc.

5. Move all old-version root system folders from / to /full-backup and move all new-version root system folders from /full-upgrade to / , i.e.,

cd /full-upgrade; for f in *; do /busybox mv -v /$f /full-backup/; /busybox mv -v $f /;done

From this point onwards, all newly launched programs will use new-version libraries and packages, while existing running programs will continue to use old libraries and packages. Since existing processes might open configs/files/folders or spawn new processes, all of which will be of new-version, there might be some conflicts/errors/failures because the running services and kernel are still of old-version before system reboot. However, the time period will be short because you only need to do the following.

6. Setup the boot-loader for the new-version root system using steps in Method A2.

• unmount the old EFI partition, (if previously your EFI is mounted at `/boot/efi`, the "/busybox mv" command will move its mount-point to `/full-backup/boot/efi`, so `umount /full-backup/boot/efi`)
• mount the EFI partition to /boot/efi (`mount /dev/sda1 /boot/efi`)
  
• delete unused EFI boot images in /boot/efi/EFI (some bios will remember the previously booted EFI image, since OS has changed, the old EFI boot image might not work, so typically just run `rm -rf /boot/efi/EFI/*`, or move them to some backup location.)
• Install the new grub EFI boot image:
  

grub-mkdevicemap
grub-install --efi-directory=/boot/efi --root-directory=/ /dev/sda
update-grub
update-initramfs -u -k all 

7. Delete /busybox (for security purposes)

8. Reboot the server into the new-version system.

9. Double-check the auto-start status for system services such as nfs-kernel-server, auditd, rsyslog, clamav, openvpn, etc., which can be different on different computers.

Typically, to minimize the system down time, Steps 5-8 should be done in one go, preferably at the end of the day when most people has left office. If ML/CUDA training happens during the night, Steps 5-8 should be done before lunch, so that after lunch, users can restart all their programs.

How to bring a stopped process into a tmux session with console display?

Very often, we run some long-waiting command with tons of console output out of tmux, then realize that we should move it into a tmux session so that we can remotely log in and monitor its progress. But then the process can hardly be terminated and re-run, and Ctrl+Z and `fg` can only resume it in the same console.

To do so:

1. Press Ctrl+Z to stop the process

2. launch a new tmux session or attach an existing tmux session

3. run `reptyr <PID>` inside tmux session with process ID of the stopped process

How to tmux an X11 GUI application so that it can persist through session detach and client disconnection?

The solution is to use xpra in addition to tmux.

Firstly, add the following 4 lines to your $HOME/.profile

alias xp_start='xpra start :100  --start-child=xterm --start-via-proxy=no --opengl=yes'
alias xp_list='xpra list'
alias xp_stop='xpra stop :100'
alias xp_attach='xpra attach :100'

 

To run an X11 app that persist through sessions:

0. SSH (with X11 tunneling, i.e., -X or -Y) into your server containing <your-x11-app>

1. create a new virtual xpra session, run xp_start

2. enter any tmux session or create a new tmux session, run `tmux a` or `tmux`

3. run the X11 app in tmux session with DISPLAY set to 100, run `DISPLAY=:100 <your-x11-app>`

4. inside tmux, attach the xpra session, run xp_attach. This will display the X11 app on your current screen. You can detach the xpra session by Ctrl+C. Detaching the tmux session or SSH disconnection will auto detach the xpra session as well.

Working Principle:

Xpra works by creating a virtual display (with number 100 in this example), then running <your-x11-app> on this virtual display. Since this is a virtual display, all apps running inside it will not be killed due to disconnection or session detach (unless you manually stop the display by xp_stop). When you attach this display :100, all x11-apps running inside this display will be shown on your screen, and they will persist through sessions.

Multiplex the Analog PIN on ESP8266 Without Any Additional Components

Unlike ESP32, ESP8266 has only one A0 analog input port, so is it possible to read analog input from multiple resistive sensor probes? The answer is YES. There are several solutions that can be found online. For example, @breagan22 has provided a workable solution by adding a diode to each sensor probe; you may also use input multiplexer chips such as ADS1015. However, those methods require additional electronic components. In this tutorial, we claim that under the condition that all sensor probes are resistive in nature and are not too far away from the common range of 10KΩ-300KΩ, it is possible to use a more elegant solution without using any additional electronic components (Method A), not even the voltage-divider resistor that is connected in series to each sensor probe.

The key idea of component-less multiplexing is to make use of INPUT_PULLUP to provide voltage divider resistance (so that you do not need additional voltage divider resistors in series with each sensor probes) and use INPUT pin-mode (that has very high impedance >1MΩ) to isolate unselected sensors.

Since INPUT_PULLUP has a predefined fixed resistance of 30kΩ-100kΩ, all sensor probes have to share this same voltage divider resistance. This gives rise to Method A which requires (N+2) ports (including A0) for N sensors in total, as shown below:

In the application scenario where all sensors need to use different voltage divider resistances or the required voltage divider resistance is too far away from the common range of 10KΩ-300KΩ provided by INPUT_PULLUP, we have to provide our own voltage divider resistors. This gives rise to Method B which requires (N*2+1) ports (including A0) for N sensors in total, as shown below.

Here are the components we used:

• Any ESP8266 development board, here, we use a WEMOS D1-mini
• A computer with Arduino IDE installed and a USB cable connected to the ESP8266
• A breadboard with electrical wires and resistors package (optional for Method A)
• Sensor probe 1, a thermistor
• Sensor probe 2, a photo-resistor
• Sensor probe 3, a variable resistor
• Multi-meter (optional)

Method A

Step 1: Initialize All GPIO Ports As INPUT

pinMode(GPIO0, INPUT);
pinMode(GPIO4, INPUT);
pinMode(GPIO5, INPUT);
pinMode(GPIO14, INPUT);

GPIO input port has very high impedance, the current is in the micro-amphere range. By setting all these ports to INPUT, this effectively isolates all sensor probes

Step 2: Set Common PULLUP to All Components

pinMode(GPIO14, INPUT_PULLUP);

According to ESP8266 specification, INPUT_PULLUP has internal resistance between 30K-100K. Therefore, this effectively connects the common terminal of all sensor probes to VCC via a resistor of 30K-100K. However, since the other terminal of all sensor probes are connected to INPUT, no current flows through sensor probes, none of the sensors is activated yet.

Step 3: Reading a Particular Sensor - Method 1

pinMode(GPIO5, OUTPUT_OPEN_DRAIN);
delay(500);
int value = analogRead(A0);
pinMode(GPIO5, INPUT);

Firstly, we need to open drain on the 2nd terminal of the target sensor probe, this effectively pull that pin to ground, allowing current to flow through the target sensor. After waiting for some time for the voltage to stabilize, we can read voltage from the A0 pin. At the end, remember to disable the sensor by setting the pin mode back to INPUT.

Step 4: Reading a Particular Sensor - Method 2

pinMode(GPIO5, OUTPUT);
digitalWrite(GPIO5, LOW);
delay(500);
int value = analogRead(A0);
pinMode(GPIO5, INPUT);

The other way to allow current to flow through the target sensor is to write digital LOW to the other pin. The difference is that the internal resistance between that 2nd pin and ground is lower in the case of OUTPUT_OPEN_DRAIN. In practice, you can choose between Method 1 and 2 depending on your sensor probe's resistance characteristics. If your sensor probes have very small resistance variation, you should use OUTPUT_OPEN_DRAIN to increase the current so that the voltage variation is more obvious.

 

Method B

Step 1: Initialize All GPIO Ports As INPUT

pinMode(GPIO0, INPUT);
pinMode(GPIO4, INPUT);
pinMode(GPIO5, INPUT);
pinMode(GPIO12, INPUT);
pinMode(GPIO13, INPUT);
pinMode(GPIO14, INPUT);

Using the same principle in Method A, we set all GPIO pins to INPUT pin mode to have very high impedance, the current is in the micro-amphere range. This effectively isolates all sensor probes.

Step 2: Power up the intended sensor

pinMode(GPIO14, OUTPUT);
digitalWrite(GPIO14, HIGH);
pinMode(GPIO5, OUTPUT_OPEN_DRAIN);

Here we use digitalWrite(HIGH) and OUTPUT_OPEN_DRAIN to power up the intended sensor and use INPUT to isolate all other sensors. Similar to Method A, we can use digitalWrite(LOW) to ground the other pin with a slightly higher internal resistance:

pinMode(GPIO14, OUTPUT);
digitalWrite(GPIO14, HIGH);
pinMode(GPIO5, OUTPUT);
digitalWrite(GPIO5, LOW);

Step 3: Reading a Particular Sensor

delay(500);
int value = analogRead(A0);
pinMode(GPIO5, INPUT);
pinMode(GPIO14, INPUT);

Similarly, we sleep for some time to wait for the voltage to stabilize, then we can read voltage from the A0 pin. At the end, remember to disable the sensor by setting the two pins back to INPUT pin mode.

 

 

System Admin: how to create a multiple-operating-system bootable and runnable harddisk/thumbdrive?

Expert computer system administrators often wants to install multiple operating systems (such as Windows 7/8/10/11/etc, and different version/flavors of Ubuntu, MX-Linux, Redhat, CentoOS, etc) and try running live systems directly without installation (for testing hardware compatibility), all using one single harddisk. For this purpose, the right tool to use is Ventoy and MX-Linux.

Ventoy allows directly copying over OS-installation ISO images onto the harddisk and booting into any one ISO image. You can select which ISO image to boot into during reboot. At the same time, the harddisk can be used to store other user data.

MX-Linux allows snap-shoting a fully-installed version of Ubuntu/Debian with all packages and configurations, so you do not need to install all useful softwares and packages again and again on each computer. Moreover, the the snap-shot ISO image can run in live mode without installation, so you can test the hardware compatibility of a fully installed OS.

 

Tips for Ventoy:

- specify a folder for storing boot ISO images, so that other folders (that are used for storing data) will not be scanned during Ventoy boot select: create a JSON file at /ventoy/ventoy.json , put the following configuration info in the JSON file


{
    "control": [
        { "VTOY_DEFAULT_SEARCH_ROOT": "/ISO" }
    ]
}

Tips for MX-Linux:

- 

Tutorial: Enter/Exit Batocera from Raspbian OS on Raspberry Pi 4

The Batocera OS is highly optimized for games, as a result, it has left out all other capabilities a normal operating system has. This makes life a bit difficult for users who wants both playing games and doing computing stuffs, because you need to install a separate operating system, need to buy and flash a separate microSD/SSD/USD-HDD, and reboot in order to switch between operating systems.

In this tutorial, we describe how to directly run Batocera's EmulationStation from inside Raspbian OS on Raspberry Pi 4, the same principle can be applied to PC or other systems (such as Ubuntu) as well.

This is an advance Linux tutorial, you are assumed to be familiar with basic Linux principles, file systems, Linux kernels, files and commands.

The steps are as follows:
1. Download and extract all Batocera files.
i). You have already downloaded the Batocera image for RPi (either the official image from Batocera Official Download , or pre-made game pack images from Arcade Punks, etc.) This is typically an XXX.img.gz or XXX.img file. If it is gzipped, extract it first, gunzip XXX.img.gz
ii). mount the image as a loop device, losetup -vfP XXX.img
iii). mount the two partitions /boot (the boot partition BATOCERA) and /rootfs (the main data partition SHARE containing all roms/save-games/game-previews/etc.)
mkdir -p /mnt/batocera-boot /mnt/batocera-rootfs
mount /dev/loop0p1 /mnt/batocera-boot
mount /dev/loop0p2 /mnt/batocera-rootfs
iv). copy over all folders in these two partitions, you can use any folder other than /opt (this step is optional since you can access files on mounted loop device images directly, however it will be super slow when accessing files)
cp -rfpP /mnt/batocera-boot /opt/batocera-boot
cp -rfpP /mnt/batocera-rootfs /opt/batocera-rootfs

2. Mount the /opt/batocera-boot/boot/batocera SquashFS (~850MB) root file-system which contains emulationstation as well as all its dependencies. For space-speed efficiency, Batocera uses ZSTD-compressed Squash File System for its main root file-system, however, the stock Raspbian kernel does not support SquashFS. Therefore, you can either:

2a. re-compile and install the Raspbian kernel with SquashFS (ZSTD format) enabled, refer to the RPi4 stock kernel guide, make sure you backup the /boot partition before you install.

2b. install squashfs-tools and extract manually:
apt install squashfs-tools
unsquashfs -f -d /opt/batocera-squashfs /opt/batocera-boot/boot/batocera

3. Create the following shell script on your desktop and make it executable:

#!/bin/bash

MOUNT_POINT=192.168.50.2:/nfs/batocera
DISPLAY_MANAGER=lightdm

# run as root if not
if [ "`whoami`" != root ]; then
    sudo "$0"
    exit 0
fi

# install tmux if not
if [ ! "`which tmux`" ]; then
    apt-get install -y tmux
fi


set -e -x -o pipefail

mount_if_not () {
    if [ $# -lt 2 ]; then
        echo "Usage: mount_if_not source target [options]"
        exit 1
    fi
    if ! mountpoint -q "$2"; then
        mkdir -p "$2"
        if [ `ls "$1"/ 2>/dev/null | wc -l` -ge 1 ]; then
            mount --bind "$1" "$2"
        else
            mount "${@:3}" "$1" "$2"
        fi
    fi
}

# 1. mount Batocera /boot
mount_if_not $MOUNT_POINT-boot /batocera/boot

# 2. mount Batocera root filesystem (/batocera/rootfs) as an overlay (upper:/bool/boot/overlay, lower:/bool/boot/batocera)
# 2a. create an overlay on memory
mount_if_not tmpfs /batocera/overlay_root -t tmpfs -o size=256M
for d in base overlay work saved; do
    mkdir -p /batocera/overlay_root/$d
done
# 2b. copy out overlay files into upper
mount_if_not /batocera/boot/boot/overlay /batocera/overlay_root/saved
cp -pr /batocera/overlay_root/saved/* /batocera/overlay_root/overlay/
umount /batocera/overlay_root/saved
# 2c. mount batocera squashfs onto lower
if [ `ls $MOUNT_POINT-squashfs/ 2>/dev/null | wc -l` -ge 5 ]; then
    mount_if_not $MOUNT_POINT-squashfs /batocera/overlay_root/base
else
    mount_if_not /batocera/boot/boot/batocera /batocera/overlay_root/base
fi
# 2d. mount the overlay filesystem
mount_if_not overlay /batocera/chroot -t overlay -o rw,lowerdir=/batocera/overlay_root/base,upperdir=/batocera/overlay_root/overlay,workdir=/batocera/overlay_root/work

# 3. mount Batocera data partition onto /userdata
mount_if_not $MOUNT_POINT-rootfs /batocera/chroot/userdata

# 4. bind batocera/boot and batocera/overlay folder
mount_if_not /batocera/boot /batocera/chroot/boot --bind
mount_if_not /batocera/overlay_root /batocera/chroot/overlay --bind

# 5. mount bind system runtime directories
for p in sys proc dev run var tmp; do
    mount_if_not /$p /batocera/chroot/$p --bind
done


# Prepare shutdown signal for returning to Raspbian
if [ ! -p /batocera/chroot/signal.fifo ]; then
    mkfifo /batocera/chroot/signal.fifo
fi
echo -e "#/bin/bash\necho exit>/signal.fifo" >/batocera/chroot/sbin/shutdown
chmod +x /batocera/chroot/sbin/shutdown

# Switch into Batocera, run commands in tmux so as to survive logging out the current session
if [ ! "`tmux ls | grep switch_to_bato`" ]; then
    if [ ! -s /etc/rc.local ]; then
        echo -e "#!/bin/sh -e\nexit 0" >/etc/rc.local
        chmod +x /etc/rc.local
    fi
    if [ ! "`grep switch_to_bato /etc/rc.local`" ]; then
        sed -i "s:^exit:tmux new-session -s switch_to_bato -d -x 240 -y 60\nexit:g" /etc/rc.local
        reboot
    fi
    echo "tmux daemon has not started during boot, please reboot"
    exit 0
fi
tmux send-keys -t switch_to_bato.0 -l "service $DISPLAY_MANAGER stop; chroot /batocera/chroot/ /etc/init.d/S31emulationstation start;read </batocera/chroot/signal.fifo; service lightdm start"
tmux send-keys Enter

The above script should be run as root, or it will sudo itself. It will install tmux if not yet installed and add a line into /etc/rc.local to launch tmux server during boot if not added. It will reboot the first time it adds the tmux line into /etc/rc.local. The reason why I use tmux is because you are running this script inside the current display manager (lightdm), so if you stop the display manager, you will get logged out, all processes including this script itself will be killed before it can chroot and launch Batocera's emulationstation.

In summary, the overall underlying principle is very simple: mount all Batocera's file systems as it does (in BATOCERA/boot/initrd.gz's /init), chroot into its root folder, and start its emulationstation by /etc/init.d/S31emulationstation start. In addition, since Batocera's emulationstation acts a standalone display manager, you need to stop your current display manager (there are lightdm, gdm, xdm, sddm, etc., Raspbian uses lightdm) before entering Batocera and restart your current display manager after exiting Batocera.