Blobless boot with RockPro64

This is a guide for booting RockPro64 computer (https://www.pine64.org/rockpro64/) without using any proprietary blobs. RockPro64 is based on Rockchip’s rk3399 SoC, so if you have some other rk3399 board, you might still find this guide useful.

I’m using Gentoo GNU/Linux in this guide but steps should be quite similar on other distributions.

Overview of boot sequence

Before we proceed with detailed instructions, let us briefly describe how rk3399 boots.

rk3399 chip has two types of internal memory that is inside the chip itself:

  • 32 KiB BootROM which is read only.
  • 200 KiB SRAM (Static RAM).

When rk3399 is powered on, CPU loads BootROM code into SRAM (at this stage main system RAM is not yet initialized). BootROM is a fairly small program baked into hardware that is responsible for loading initial bootloader. It supports booting from SPI, eMMC, SD and supports downloading next bootloader over USB OTG interface. Since it is quite small and baked onto the chip itself, for the purposes of this guide we will consider it as hardware, not software.

So, BootROM loads U-Boot TPL into SRAM. Since SRAM is quite small, we cannot load the full U-Boot bootloader into it, so only a small part called TPL is loaded. Its main job is to initialize main system RAM (RockPro64 has up to 4 GiB of LPDDR4 based RAM).

Then U-Boot TPL hands control back to BootROM which then loads a slightly bigger part of U-Boot called U-Boot SPL. At this stage SPL loads ATF (Arm Trusted Firmware: https://github.com/ARM-software/arm-trusted-firmware) and U-Boot Proper into memory. Then ATF starts and finally runs U-Boot.

U-Boot can boot payloads from a variety of sources including, eMMC, SD, USB as well as do network boot. Also, U-Boot has support for UEFI booting specification, so it can boot EFI binaries located on ESP partitions. Even though U-Boot can load Linux kernel directly, I personally find it more convenient to first load Grub2 and then load Linux kernel.

Compiling required components

Toolchain

First of all you need to install required toolchain. If you are following this guide on RockPro64 itself or other ARM64 system, you can just use gcc. If you are not on ARM64, you can use crossdev (https://wiki.gentoo.org/wiki/Cross_build_environment) to install ARM64 cross-compiler (other distributions often ship cross compiler binaries too, e.g. on Debian GNU/Linux you can use https://packages.debian.org/sid/gcc-aarch64-linux-gnu).

In addition to ARM64 compiler, you also need ARM32 cross-compiler:

crossdev --target arm-none-eabi -s4

At the moment on my system I have cross-arm-none-eabi/gcc-8.3.0-r1 with USE="cxx graphite jit multilib pgo".

Arm Trusted Firmware

As mentioned before, source code for ATF can be downloaded from https://github.com/ARM-software/arm-trusted-firmware.

Building ATF is quite easy but you might want to first remove some blobs (alternatively, removal of blobs is available in my git fork https://git.stikonas.eu/andrius/arm-trusted-firmware)

find . -name '*.bin' -exec rm -rf {} \;
make PLAT=rk3399

This should produce build/rk3399/release/bl31/bl31.elf. Now, we can make the path to bl31.elf binary known to U-Boot build process

export BL31=path/to/arm-trusted-firmware/build/rk3399/release/bl31/bl31.elf

U-Boot

U-Boot only gained support for training LPDDR4 memory in v2019.10 which is not released yet, so for now we will get U-Boot from git:

git clone https://gitlab.denx.de/u-boot/u-boot.git/

Let’s use default configuration. I also tried tweaking configuration a bit to enable HDMI display inside U-Boot but so far I was not successful.

cd u-boot
make rockpro64-rk3399_defconfig
make

This should produce idbloader.img which contains U-Boot TPL and SPL and u-boot.itb which contains U-Boot Proper. We can install those with the following script

#!/bin/sh -e

cd u-boot
sudo dd if=idbloader.img of=/dev/mmcblk1 seek=64
sudo dd if=u-boot.itb of=/dev/mmcblk1 seek=16384

Optionally you might want to create two 4 MiB partitions that start at sectors 64 and 16384 and use those with dd without seek. Before running commands above, make sure that your eMMC or SD card is represented by /dev/mmcblk1 block device. I have only tested booting from eMMC and did not try SD card myself.

At this stage you can use any of the boot methods supported by U-Boot. In this guide I’ll be using UEFI boot to load GNU GRUB.

GNU GRUB

Let’s get GRUB2 from package manager:

emerge sys-boot/grub

On my eMMC card I have created EFI System Partition (which should be FAT32 formatted). Then add moutpoint /boot/efi to /etc/fstab. GRUB should then be installed to ESP:

#!/bin/sh

# mount /boot/efi # you might needs this if /boot/efi is not mounted
grub-install /dev/mmcblk1 --removable
grub-mkconfig -o /boot/grub/grub.cfg

The script above should have created /boot/efi/EFI/BOOT/BOOTAA64.EFI

Compiling kernel

In this guide I’ll use mainline kernel with a few patches. Support for RockPro64 is not yet complete in the mainline kernel, so it is essential to use the latest kernel. At the time of writing this is 5.3.

Configuring kernel is out of scope for this blog post, there are other guides online. You can use my configuration from https://stikonas.eu/files/gentoo-sources/. Copy config file into your kernel source directory and rename it to .config. However, for UEFI boot as described in this blog, you need to enable CONFIG_EFI_STUB=y in your kernel config.

Let’s download kernel sources. First of al, we’ll apply a few patches. Download the patches from https://stikonas.eu/files/gentoo-sources/ and put them to /etc/portage/patches/gentoo-sources/. Note that the last patch 0003-arm64-dts-rockchip-Add-PWM-fan-for-RockPro64.patch will be included in Linux 5.4. The other two patches fix booting from eMMC (cards sold by pine64 seem to need it, other cards, e.g. from Hardkernel seem to work fine) and terrible IPv6 performance (it seems to have been fixed in recent kernels).

emerge gentoo-sources

To achieve fully blobless boot we can deblob the kernel:

cd /usr/src/linux # assuming that is where you unpacked your kernel
wget https://linux-libre.fsfla.org/pub/linux-libre/releases/5.3-gnu/deblob-5.3
wget https://linux-libre.fsfla.org/pub/linux-libre/releases/5.3-gnu/deblob-check
wget https://linux-libre.fsfla.org/pub/linux-libre/releases/5.3-gnu/deblob-main
chmod +x deblob-5.3 deblob-check
./deblob-5.3

To compile the kernel, simply run

make -j$(nproc)

Then kernel can be installed with

make zinstall
make modules_install
make dtbs_install

The last command will install device tree files to /boot/dtbs/kernel_version/rockchip. In particular this directory should contain rk3399-rockpro64.dtb. Copy this file to ESP partition, so that it is available to U-Boot when it is loading Grub.

Copying this dtb file is in principle optional. If it is missing, then kernel will simply use dtb from U-Boot which is mostly good enough, but kernel usually has slightly more up to date device tree file. At the moment I was not able to get HDMI working if I skip this step. This is possibly related to my failure of getting screen to work in U-Boot itself

kernel_version=5.3.0-gentoo-gnu
mkdir -p /boot/efi/dtb/rockchip
cp /boot/dtbs/${kernel_version}/rockchip/rk3399-rockpro64.dtb /boot/efi/dtb/rockchip

Don’t forget to generate initramfs, for example you can use dracut

dracut --xz -H /boot/initramfs-${kernel_version}.img $kernel_version
grub-mkconfig -o /boot/grub/grub.cfg

At this stage your can reboot and if everything goes fine, you’ll hopefully boot into fully free system.

If you grab latest mesa package with panfrost driver you can even use accelerated KDE Plasma desktop or play some 3D games.

Configuring fan with fancontrol

The last patch we applied to the kernel exposes fan interface to the kernel. At the moment this makes fan spin at full speed. You can control it with e.g. fancontrol from sys-apps/lm-sensors package. I use the following /etc/fancontrol configuration file:

INTERVAL=10
DEVPATH=hwmon0=devices/platform/pwm-fan
DEVNAME=hwmon0=pwmfan
FCTEMPS=hwmon0/device/pwm1=../thermal/thermal_zone0/temp
MINTEMP=hwmon0/device/pwm1=35
MAXTEMP=hwmon0/device/pwm1=60
MINSTART=hwmon0/device/pwm1=100
MINSTOP=hwmon0/device/pwm1=70

fancontrol can be started with systemctl enable fancontrol; systemctl start fancontrol

Caveats

Unfortunately, on my system booting is not reliable yet. I’m not yet sure why. Sometimes U-Boot Proper fails to initialize eMMC card (typing boot until it succeeds usually fixes the problem), occasionally boot fails in other U-Boot stages.

Update: This seems to be specific to some eMMC cards. Maybe related to command queue support (see kernel patch that I applied to my kernel that was also disabling command queueing on eMMC cards).

Booting from SPI

U-Boot also supports booting from on-board SPI flash. Version 2020.01 can boot U-Boot TPL and U-Boot SPL stages from SPI flash. In order to also boot U-Boot Proper, you need to patch boot order and use adjust U-Boot config (see provided defconfig in the following link)

https://github.com/u-boot/u-boot/commit/7db83e4dcaf2c987f6545ecdbe0eba35a05b9273

You can flash these stages with the following script:

#!/bin/sh -e
cd u-boot
image_name="spi_idbloader.img"
combined_name="spi_combined.img"

tools/mkimage -n rk3399 -T rkspi -d tpl/u-boot-tpl.bin:spl/u-boot-spl.bin "${image_name}"
padsize=$((0x60000 - 1))

image_size=$(wc -c < "${image_name}")
[ $image_size -le $padsize ] || exit 1
dd if=/dev/zero of=${image_name} conv=notrunc bs=1 count=1 seek=${padsize}
cat ${image_name} u-boot.itb > "${combined_name}"

dd if="${combined_name}" of=/dev/mtdblock0

Binaries

If you prefer to run binaries but do not want to compile them yourself, you can get them from https://stikonas.eu/files/gentoo-sources/u-boot/

Become a patron Donate using Liberapay Bitcoin: bc1qe2dfqjwgse5v6cl6rhtk352ru90t0hnve45f2c

28 comments on “Blobless boot with RockPro64

    • There is more than one way. One of the easier ways is to start with pre-existing image of some other distro, unpack it to storage media. Mount rootfs, remove everything except kernel (so keep /boot and /lib/modules) and copy in arm64 stage3 and follow Gentoo installation handbook from there.

      If you don’t want to rely on some other image to start, you need to compile bootloader (U-boot) and kernel as I’ve done in this post (I also used Grub here which is optional and you might want to skip it for first boot to avoid writing grub configuration file manually). Write those onto boot media (emmc or sdcard, since you can’t easily write to spi chip before you booted). Note that you’ll have to partition your boot media which you don’t have to do if you use first method.

      At the moment I don’t have any step-by-step instructions since I don’t have a spare rockpro64 to test with. In general, once you have bootloader and kernel working, there isn’t really any difference from any other GNU/Linux or Gentoo system, so you need to do the same installation steps.

      There is another option you can take and try to boot some LiveUSB on rockpro64 and then do installation from there. But before you can do that, you still need to get U-Boot to some media from which rockpro64 can boot (so SPI, eMMC or SDdcard). And then boot USB and do basically the same installation procedure as on AMD64 but using ARM64 stage.

      Reply
  • Alistair says:

    Hi,
    Thanks for your work and for sharing it.
    I’m going to install a bunch of RockPro64.
    One of the requirement is ZERO blob.
    You wrote this article eight months ago.
    Has anything changed ?
    Can I use your github repo to get a blobless ATF directly ?
    Thanks

    Reply
    • Well, you can go for newer versions of everything now. Other than that not really. You can use my github repo to get blobless ATF (from deblob branch, it’s just one commit removing blobs).

      Reply
  • Jinaria says:

    Can I set a DRAM frequency in u-boot? my board has DDR3 933Mhz memory chip but my current u-boot set it to 800Mhz.


    DDR Version 1.09 20171117
    In
    soft reset
    SRX
    Channel 0: LPDDR3, 800MHz
    CS = 0
    MR0=0x58
    MR1=0x58
    MR2=0x58
    MR3=0x58
    MR4=0x2
    MR5=0x1
    MR6=0x5
    MR7=0x0
    MR8=0x5F
    MR9=0x5F
    MR10=0x5F
    MR11=0x5F
    MR12=0x5F
    MR13=0x5F
    MR14=0x5F
    MR15=0x5F
    MR16=0x5F
    CS = 1
    MR0=0x58
    MR1=0x58
    MR2=0x58
    MR3=0x58
    MR4=0x2
    MR5=0x1
    MR6=0x5
    MR7=0x0
    MR8=0x5F
    MR9=0x5F
    MR10=0x5F
    MR11=0x5F
    MR12=0x5F
    MR13=0x5F
    MR14=0x5F
    MR15=0x5F
    MR16=0x5F
    Bus Width=32 Col=11 Bank=8 Row=15/15 CS=2 Die Bus-Width=16 Size=4096MB
    no stride
    ch 0 ddrconfig = 0x202, ddrsize = 0x4040
    pmugrf_os_reg[2] = 0x1000CCA1, stride = 0x17
    OUT
    Boot1: 2017-06-09, version: 1.09
    CPUId = 0x0
    ChipType = 0x10, 1902
    SdmmcInit=2 0
    BootCapSize=100000
    UserCapSize=59640MB
    FwPartOffset=2000 , 100000
    SdmmcInit=0 20
    StorageInit ok = 66689
    LoadTrustBL
    No find bl30.bin
    Load uboot, ReadLba = 2000

    Reply
  • joe bloggs says:

    Hi,
    Sorry if this is something obvious, I’m fairly new to building stuff like this.
    I get this error whenever I try to compile arm TF (using make PLAT=rk3399, after deblobbing)
    gcc: error: unrecognized command-line option ‘-mstrict-align’; did you mean ‘-Wstrict-aliasing’?
    I am running Debian (buster I think, its all up to date), and the bit that confuses me is that I get this error when trying to compile on my rockpro64 as well, so I don’t think its a cross compilation issue (unless its trying to compile to arm64 when it needs to be arm32?) When I get a chance I will try again with Ubuntu as that seems to be known to work, but would still be nice to know if there’s some basic error I’m making.
    Thanks!

    Reply
      • joe bloggs says:

        yeah just double checked and I have gcc-aarch64-linux-gnu and gcc-arm-none-eabi installed and updated, but still not working and producing the same error message. Im not in a chroot and am running it as root (maybe not best practice, idk), so not sure if thats whats wrong?
        Anyway, I know its a bit off topic, but thanks for putting together guides like this and responding to questions. Linux can be a real pain sometimes but people like you make it a lot more bearable to deal with. Thanks!

        Reply
    • joe bloggs says:

      Just going to leave this here in case it helps anyone. Switched to Ubuntu on the rockpro64 (ayufans image, bionic), and it seems to work perfectly for compiling ATF now, provided that gcc-arm-none-eabi (arm32 compiler) is installed, trying it without fails (no surprise there). Never found out what the issue with Debian was, since the compilers seemed to be a version that should have worked, but since it failed on the rockpro64 and a separate computer (intel 64bit cpu) with the same error message, both running Debian, I assume Debian has something to do with it somehow.

      Reply
      • Got the same error in Ubuntu. The problem is solved for me with
        export CROSS_COMPILE=aarch64-linux-gnu-
        Which makes sense if you think about it

        otherwise, in Ubuntu the required toolchains are

        sudo apt install gcc-arm-none-eabi gcc-aarch64-linux-gnu

        Reply
        • On a non arm Ubuntu, if /usr/lib/grub/arm64-efi/ is missing a cross compiled grub is necessary

          * git clone https://git.savannah.gnu.org/git/grub.git
          * ./configure –target=aarch64-linux-gnu –with-platform=efi –localedir=/usr/share/locale
          * make
          { //There should be a cleaner way
          * sudo mkdir /usr/local/lib/grub
          * sudo ln -s /home/user/Build/grub/grub-core arm64-efi
          }
          * sudo ./grub-something -O arm64-efi -o /path/To/BOOTAA64.EFI

          Reply
  • I built u-boot according to above instructions and flashed the SPI using your script from “Booting from SPI” section.

    Got the message “No space left on device”.

    Isn’t the 1310k “spi_combined.img” file truncated when copied onto “/dev/mtdblock0” which is 384k in size?

    The board still boots fine though!

    Reply
    • Why do you say `mtdblock0` is 384k in size? It’s 16 MB in size. I think you have a normal file in /dev/mtdblock0 instead of block device node. I would check that and delete it.

      Reply
      • I installed gentoo starting from ayufan’s latest stable image. The kernel included in that image shows:

        /dev/mtdblock0, size 384k
        /dev/mtdblock1, size 3.6M
        /dev/mtdblock2, size 32k
        /dev/mtdblock3, size 12M

        I booted a Manjaro SD card which shows /dev/mtd0, size 16M.
        After dd if=spi_combined.img of=/dev/mtd0 the board is bricked.
        It is not booting even after bridging pins 23 and 25.

        Well, that’s life, one more thing goes to rubbish.

        Reply
        • Jinaria says:

          You can unbrick almost any rockchip based device using OTG port and a pc, I unbricked many rk3399 set top boxes. you need factory image and rockchip software.

          Reply
  • Hi Andrius,

    I’m using Gentoo on my desktop for a long time. Well versed with GPT, UEFI, GRUB2.

    Since you seem to be using Gentoo too, I was wondering if you can clarify for me a few things for me, mainly about partioning.:

    Just got my RockPro64 and I’d like to install Gentoo on eMMC (NVME or SSD will come later when I get my hands dirty).

    On a bare bones machine I usually create an EFI partition and a root partition, then I boot SystemRescueCD on USB and use that to start Gentoo installation from stage3.

    For the RockPro64 I installed Manjaro on SD card and can use that as my starting point for Gentoo installation on eMMC.

    1. Will GPT partioning scheme work?

    2. Is this layout all right?
    – EFI partition, 2 GB, fat32 (with GRUB2)
    – root partition, 32 GB, ext4 or btrfs (for Gentoo system)
    – data partittion, 90 GB, ext4 or btrfs (for /home)

    3. Will the board be able to boot the above eMMC without me touching the U-Boot or SPI that came with it?

    Thank you,
    Eugen

    Reply
    • 1. Yes, GPT partitioning scheme works as it is supported by U-Boot.

      2. This would work, although I don’t know why you need 2 GB for EFI partittion. Grub.efi is about 300K large. I think my EFI partition is 112 MiB on RockPro64 and even that is too much.
      Root and data partitions are alright (especially if you use ext4). Personally, I would use a single btrfs with subvolumes for / and /home. That way your free space is not partitioned, but subvolumes still allow some separation.

      3. No. The board does not come with U-Boot on SPI or eMMC. It comes completely empty. You need to install U-boot to either SPI (and full SPI boot needs not yet released U-Boot v2020.07) or eMMC.

      Reply
      • I will reduce the EFI partition to a few MiB, then apply the info in this blog to install U-boot on eMMC.
        I may also reduce the size of / and home to make room for a future install of FreeBSD.
        Really appreciate your answer!

        Reply
  • Visitor says:

    I think one can remove one more blob with
    rm plat/rockchip/rk3399/drivers/dp/hdcp.bin
    just before building ATF with
    make PLAT=rk3399
    and HDMI still works. Maybe some DRMed content won’t be displayed, I haven’t tried, but that’d be a feature, wouldn’t it?
    In reality I didn’t try it exactly so. I additionaly removed code from ATF for other hardware (just because I didn’t want to browse it looking for blobs). But that paranoia should not be needed.

    Reply
  • Philipp says:

    Hi,

    I’m trying to follow your guide inside a Gentoo chroot on ayufan’s Ubuntu build on the Rock Pro 64.
    When building the ATF I always get the following error:

    ld.bfd: build/rk3399/release/bl31/bl31.elf section `.pmusram' will not fit in region `PMUSRAM'
    ld.bfd: region `PMUSRAM' overflowed by 104 bytes

    When I change the “-O3” in plat/rockchip/rk3399/drivers/m0/Makefile to “-Os” it gets a litte smaller:

    ld.bfd: build/rk3399/release/bl31/bl31.elf section `.pmusram' will not fit in region `PMUSRAM'
    ld.bfd: region `PMUSRAM' overflowed by 96 bytes

    That happens with both gcc 9.2.0-r2 and 8.3.0-r2.

    Do you have an idea / suggestion how to fix this?

    Thanks!

    Reply
    • Hmm, at some point I had this error too. Unfortunately, I don’t remember when exactly it went away.
      My cross-arm-none-eabi/gcc has the following USE flags:
      cxx graphite jit lto multilib pgo

      and sys-devel/gcc has
      cxx d fortran go graphite lto nls nptl objc objc++ objc-gc openmp pch pgo pie sanitize ssp vtv

      I’m on gcc 9.2.0-r2 at the moment.

      What about ubuntu toolchain? Can it compile ATF?

      Reply
      • Philipp says:

        Stupid me…I haven’t tried the Ubuntu toolchain. Yes it does work with Ubuntu’s compilers. For reference what I did was:

        # apt-get install gcc-arm-none-eabi

        The system (aarch64) compiler is:
        gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)
        The embedded compiler (arm-none-eabi-gcc) is:
        gcc version 6.3.1 20170620 (15:6.3.1+svn253039-1build1)

        Thanks!

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Time limit is exhausted. Please reload CAPTCHA.