How to properly build a usable Linux kernel with Nvidia driver?

MX-Linux is known as the best Linux Distro for some years because it has a brilliant feature called MX-snapshot. To me, MX-Linux is essentially Ubuntu+OS_Snapshot. However, one major defect of MX-Linux is that its kernel has disabled ACPI so that after installing Nvidia driver, CUDA does not support GPU acceleration for PyTorch/Tensorflow/etc.

To overcome this drawback, you can either install a Ubuntu kernel by try-and-error or compile a kernel that support ACPI.

To compile a kernel step-by-step:

1. Download a stable or long-term version Linux kernel from https://www.kernel.org/
2. Make sure that essential building packages are installed
   git fakeroot build-essential ncurses-dev xz-utils libssl-dev lz4 bc
   
3. Extract the kernel archive and cd into that folder
4. Copy current kernel config (in /boot/config-$(uname -r)) into .config
5. Revise the configuration, there are 3 interfaces:
   make menuconfig (this is in console mode)
   make gconfig (this requires GTK, suitable for gnome-desktop)
   make xconfig (this requires Qt5, suitable for KDE-desktop, can be installed by `apt install qt5-default`)
6. In the configuration, for every kernel module there are 3 options:
   y: the corresponding binary file is linked with vmlinux
   n: do not build
   m: although it will not link with vmlinux, it will be compiled and you can use modprobe or insmod to manually load the .ko kernel driver on demand
   
7. Compile the kernel, `make -j 4`
8. Install kernel modules striping unneeded symbols,
   make INSTALL_MOD_STRIP=1 modules_install -j 4
9. To shrink the generated kernel image size, change `MODULES=most` into `MODULES=dep` in /etc/initramfs-tools/initramfs.conf 
   
10. Install kernel image into /boot, `make install -j 4`
11. Now the main folder will be of huge size (>30GB), you need to cleanup the folder
    make clean
    find /lib/modules/<kernel_version>/ -iname "*.ko" -exec strip --strip-unneeded {} \;
    

After that,  you can now rebuild and install Nvidia driver by either:
- rebooting into the new kernel and run:
    dkms install nvidia/460.67
OR
- without rebooting, directly run:
    dkms install nvidia/460.67 -k <kernel-version>

  12. If you manually upgraded Nvidia driver, you also need to update initramfs, otherwise, your `/boot/initrd.img` will still contain the old-version driver.

    update-initramfs -c -k $(uname -r)

How to control which device event can wake up your computer from standby?

Level 1 method:

Look at /proc/acpi/wakeup:

Device  S-state   Status   Sysfs node
PEG0      S4    *disabled  pci:0000:00:01.0
PEGP      S4    *disabled  pci:0000:01:00.0
PEG1      S4    *disabled
PEGP      S4    *disabled
PEG2      S4    *disabled
PEGP      S4    *disabled
RP01      S4    *disabled  pci:0000:00:1c.0
PXSX      S4    *disabled  pci:0000:3b:00.0
RP02      S4    *disabled
PXSX      S4    *disabled
RP03      S4    *disabled
PXSX      S4    *disabled
RP04      S4    *disabled
PXSX      S4    *disabled
RP05      S4    *disabled  pci:0000:00:1c.4
PXSX      S4    *disabled  pci:0000:3c:00.0
                *disabled  platform:rtsx_pci_sdmmc.0
                *disabled  platform:rtsx_pci_ms.0
RP06      S4    *disabled
PXSX      S4    *disabled
RP07      S4    *disabled

This device file shows devices that are allowed to wake up the system from standby. You can toggle each item by (e.g.):

echo PEG0 >> /proc/acpi/wakeup

For the device code name:

• PS2K: PS/2 keyboard
• PS2M: PS/2 mouse
• PWRB or PBTN: Power button
• LID: Laptop lid
• RP0x or EXPx: PCIE slot #x (aka PCI Express Root Port #x)
• EHCx or USBx: USB 2.0 (EHCI) chip
• XHC: USB 3.0 (XHCI) chip
• PEGx: PCI Express for Graphics slot #x
• GLAN: Gigabit Ethernet

More can be found in ACPI specification.

Level 2 method (if Level 1 method does not work):

If the device does not appear in /proc/acpi/wakeup, you can search for device files named wakeup in /sys/devices folder recursively. You can enable/disable wakeup by (e.g.):

echo enabled > /sys/devices/platform/i8042/serio0/power/wakeup

Since ACPI devices are organized in a tree structure, on certain computers, it might happen that even if you enabled the leaf node, the parent node is still disabled, so it still does not wakeup. On one laptop, after such configuration, pressing the power button does not wake up the system; while holding down the power button for 10 seconds wakes up the system from standby.

How to pass a filled C/C++ structure from Python to C/C++?

Python has an excellent design of being compatible when interfacing with C/C++ libraries.

In this tutorial, we show an example of passing a filled C/C++ structure from Python 3 to C/C++ dynamic library function (in .so). The advantage of doing things in this way is that you do not need to include Python.h because ctypes is built-in library of Python.

C/C++ source code file test.cpp :

#include <iostream>
#include <cstdlib>
#include <inttypes.h>

#pragma pack(push, 1)
struct c_struct{
   int32_t v1_int32;
   int64_t v2_int64;
   float v3_float32;
   double v4_float64;
   char v5_char;
   char *v6_char_p;
   int *v7_ptr_int32;
   float *v8_ptr_float32;
   float v9_arr_float32[2];
};
#pragma pack(pop)

extern "C" int cfunc_ptr(c_struct *obj) {
   printf("c_struct:\n");
   printf("v1_int32 = %" PRId32 "\n", obj->v1_int32);
   printf("v2_int64 = %" PRId64 "\n", obj->v2_int64);
   printf("v3_float32 = %.15g\n", obj->v3_float32);
   printf("v4_float64 = %.15g\n", obj->v4_float64);
   printf("v5_char = %c\n", obj->v5_char);
   printf("v6_char_p = %s\n", obj->v6_char_p);

   printf("v7_ptr_int32 =");
   for(int x=0; x<10; ++x) printf(" %d", obj->v7_ptr_int32[x]);
   printf("\n");

   printf("v8_ptr_float32 =");
   for(int x=0; x<10; ++x) printf(" %f", obj->v8_ptr_float32[x]);
   printf("\n");

   printf("v9_arr_float32 = [%.6f, %.6f]\n", obj->v9_arr_float32[0], obj->v9_arr_float32[1]);
   return 1234;
}

extern "C" float cfunc_cst(c_struct obj) {
   cfunc_ptr(&obj);
   return 1.234f;
}

int main(int argc, const char *argv[]){
   char str[] = "Hello1\0Hello2";
   printf("Hello world\n");
   int vi[10];
   float v[10];
   for(int x=0; x<10; ++x){ v[x] = x+0.5f; vi[x] = x+1;}
   struct c_struct ca1 = {123456789, (int64_t)123456789123456789, 3.14159265358979, 3.14159265358979,
       'Q', &str[0], &vi[0], &v[0], {1.2f, 3.4f}};
   printf("cfunc_ca1() returns %f\n\n", cfunc_cst(ca1));
   printf("cfunc_ptr() returns %d\n", cfunc_ptr(&ca1));
   return 0;
}

Run the following command line to compile and generate the dynamic library test.so:

g++ -shared -o test.so test.cpp

Python source code which calls the C/C++ function with filled C structure c_struct :

import ctypes

class c_struct(ctypes.Structure):
   _pack_ = 1
   _fields_ = [('v1_int32', ctypes.c_int32),
               ('v2_int64', ctypes.c_int64),
               ('v3_float32', ctypes.c_float),
               ('v4_float64', ctypes.c_double),
               ('v5_char', ctypes.c_char),
               ('v6_char_p', ctypes.c_char_p),
               ('v7_ptr_int32', ctypes.c_void_p),
               ('v8_ptr_float32', ctypes.c_void_p),
               ('v9_arr_float32', ctypes.c_float*2),
               ]

if __name__ == '__main__':
   import numpy as np

   arr_int32 = np.arange(10, dtype=np.int32) + 1
   arr_float32 = np.arange(10, dtype=np.float32) + 0.5
   test_lib = ctypes.CDLL('test.so')
   obj = c_struct(123456789, 123456789123456789, 3.14159265358979, 3.14159265358979, b'Q', b'Hello1',
                  arr_int32.ctypes.data, arr_float32.ctypes.data, (ctypes.c_float*2)(1.2, 3.4))

   test_lib.cfunc_cst.restype = ctypes.c_float
   ret = test_lib.cfunc_cst(obj)
   print('cfunc_cst() returns %f\n' % ret)

   ret = test_lib.cfunc_ptr(ctypes.pointer(obj))
   print('cfunc_ptr() returns %d' % ret)

Take note that:

1. A C/C++ pointer is a memory address (i.e., an integer), whether it is interpreted as pointing to integer array, float array, or etc., is all defined by user. So the interpretation does not matter, we just use void pointer in general.
2. The default C/C++ function return type is always int when interpreted by Python, for non-integer function return type, you need to set function.restype explicitly.
3. Numpy arrays support direct access by C/C++ functions, use array.ctypes.data (which is of void pointer type)
4. The above Python code demonstrates both passing C structure by instance and by pointer.
5. You must use {extern "C"} to define/declare every function to be exported.
6. This method does not need to include Python.h, neither need to install python-dev
7. For struct packing, C++ uses "#pragma pack(1)", Python uses "_pack_=1"
   

How to transfer install a legacy copy of Linux/Ubuntu onto a UEFI computer in legacy mode?

Nowadays, Ubuntu installation is no longer a problem. However, sometimes we might want to:

• transfer a fully-functional copy of Ubuntu (i.e., with all software packages such as Wine, MS-Office, WeChat/QQ/WhatsApp installed) onto another computer
• the fully-function copy of Ubuntu is in legacy mode, but the target computer requires booting in UEFI mode
• during installation, the computer is booted in legacy mode (by default, installing UEFI Ubuntu requires booting Ubuntu install CD/thumb-drive/etc in UEFI mode)

This can be easily done for legacy-boot Ubuntu, but for UEFI, it is slightly more complicated.
The steps are as follows:

1. Resize existing partitions and create new partitions for the new Ubuntu OS.
2. Boot into {a copy of Ubuntu system} or {Ubuntu installer} in either legacy or UEFI mode.
3. Copy the pre-installed Ubuntu system folders recursively onto the target partition (e.g., /target) (use "cp -rfa", exclude /dev, /proc, /sys, /mnt, /run, but create these folders on the target partition and use the mount command to bind them, also bind /dev/pts)
4. Mount the EFI partition to /target/boot/efi
5. chroot into /target, "sudo chroot /target", inside the fake root environment:
• install grub-efi, "sudo apt-get install grub-efi"
• link up EFI partition, "grub-install --target=x86_64-efi /dev/sdX", ignore any errors
6. copy grubx64.efi (check /target/boot/EFI/ubuntu folder) into /target/boot/efi/EFI/BOOT/bootx64.efi and /target/boot/efi/EFI/ubuntu/grubx64.efi
7. copy grub.cfg (check /target/boot/grub/grub.cfg) into /target/boot/efi/EFI/ubuntu/grub.cfg
8. edit /target/etc/fstab and /target/boot/efi/EFI/ubuntu/grub.cfg to make sure all partitions are correct

(If Secure Boot is active, it gets more complex; you must copy shimx64.efi to EFI/BOOT/bootx64.efi and copy grubx64.efi to EFI/BOOT/grubx64.efi.)

How to use Intel GPU for desktop display and dedicate Nvidia GPU (nvidia-415 and above) for CUDA in Ubuntu?

Nvidia has updated their graphics driver, so in the latest nvidia-415, my previous solution (2017.9 "How to use Intel GPU for desktop display and dedicate Nvidia GPU for CUDA in Ubuntu?") no longer works.

Here is the updated solution to deal with the new driver:
METHOD A:
1. You still select Nvidia by "sudo prime-select nvidia"
2. add the following in /etc/X11/xorg.conf

Section "Device"
    Identifier      "intel"
    Driver          "intel"
    BusId           "PCI:0:2:0"
EndSection

Section "Screen"
    Identifier      "intel"
    Device          "intel"
EndSection

where the BusId is the PCI address of your integrated graphics card (run lspci).


METHOD B:
1. To use Intel GPU to display desktop environment, you first need to switch to Intel GPU using "sudo prime-select intel" and reboot.

2a. Every time after entering desktop, you need to run the following command to load the kernel driver:
sudo modprobe ipmi_msghandler
sudo insmod /lib/modules/$(uname -r)/updates/dkms/nvidia.ko

Take note that as Nvidia keeps updating their driver:
- the filename and location of nvidia.ko might change, you need to know these every time you install the Nvidia driver.
- its dependency kernel module, ipmi_msghandler, might also change. You can use modinfo to check Nvidia kernel driver's dependency

2b. Alternatively (instead of 2a), you can simply run the following command to load Nvidia driver:
sudo prime-select nvidia
sudo modprobe nvidia
sudo prime-select intel

How to install busybox onto an Android emulator in order to access more linux commands

I have created the following script which can install busybox silently and headlessly via command line. It works on both 32-bit and 64-bit Android emulators.

Firstly, you should download busybox binary executable from https://busybox.net/downloads/binaries/ and copy it into your $HOME/bin folder.

Next, make a script called install-busybox.sh in $HOME/bin with the following content:

if [ $# == 0 ]; then
    devs="`adb devices | awk '{if(NF>0)print $0}' | sed -n '2,$p'`"
    N=`echo $devs | wc -l`
    if [ $N -gt 1 ]; then
        echo "Usage: $0 device-serial" >&2
        adb devices
        exit 1
    fi
    set -- `echo "$devs" | awk '{print $1}'`
fi

set -x

adb -s $1 root
adb -s $1 shell setenforce 0
adb -s $1 shell mount -o remount,rw /
adb -s $1 push $HOME/bin/busybox /sbin

# on 64-bit system, it fails
if [ $? != 0 ]; then
    adb -s $1 shell mount -o rw,exec,remount /mnt
    adb -s $1 shell cp -rfp '/sbin' /mnt/
    adb -s $1 shell mount /mnt/sbin /sbin
    adb -s $1 push $HOME/bin/busybox /sbin
fi

adb -s $1 shell /sbin/busybox --install -s /sbin
res=$?

set +x
if [ $res == 0 ]; then
    echo Successful
else
    echo Failed
fi

So far, I have tested it on API 26-28, but it might not work on all Android images.

Ubuntu HDMI extended monitor low resolution solved

It seems that recent versions of Ubuntu (16.04) has a bug in handling extended HDMI monitors. The HDMI monitor has up to 1980x1080 resolution, my laptop has up to 1366x768 resolution. Both screens have native 16:9 ratio.
However, in "mirrored display" mode, I can only select 1024x768 and 800x600 resolution as shown in the image below:

In such mode, my laptop screen has two blank strip on the left and right side. On the HDMI monitor, it shows a squashed screen (1024x768 resolution stretched to 16:9 full screen).

The solution is to use xrandr to list all display modes and add each mode into every monitor, the shell script is as follows:

displays=(`xrandr | grep '^[^ ]' | awk '{print $1}'`)

xrandr | grep "^ " | awk '{print $1}' | grep [0-9]x[0-9] | sort | uniq \
        | while read mode; do
        echo "Adding $mode ..."
        for display in ${displays[*]}; do
                xrandr --addmode $display $mode
        done
done

You just need to run this script before plugging in the HDMI cable.