Skip to content

Latest commit

 

History

History
638 lines (441 loc) · 45.7 KB

ExternalDrives.md

File metadata and controls

638 lines (441 loc) · 45.7 KB

How To Connect a USB Drive to Raspberry Pi?

Table of Contents

Background and Objectives:

Following is a procedure to mount a USB Drive on a Raspberry Pi. USB drives are typically small, portable data storage devices typically used for data transfer. USB drives are block storage devices which implies that they will be formatted IAW a particular file system. This recipe utilizes one of the so-called FAT file systems, chosen because of its compatibility with virtually all modern operating systems. Note however that while the FAT file system is ubiquitous, it does not compare particularly well performance-wise against other, more modern file systems.

Rather than simply listing the steps in rote fashion, this recipe includes some background and context. This is done primarily for my benefit - as a learning exercise. I hope this approach will be useful to others as well, an alternative to the typical "copy and paste tutorial" that invites copying command lines from a blog into a terminal window without thinking about what they mean. But if you don't care about the explanations, ignore them; just follow the steps inside the code blocks.

Why would I want to connect an external drive?

  1. The Raspberry Pi has 4 USB ports, and
  2. external drives can be useful for all sorts of things:
    • local backup of RPi files, or a 'disk image' of the entire SD card
    • file sharing with your Mac, PC or another RPi
    • reduce wear on your SD card
  3. USB drives are cheap

But all of these instructions! Why is this so complicated? On my Mac, I plug the drive in, and it just works. I can read from it, and write to it immediately!

That's a good question. Unfortunately, the answer may not be straightforward, and will not satisfy all parties. The answer comes down to "cultural differences" between the "Unix way", and the "Mac way" of getting things done on a computer. The process for mounting a drive in Unix/Linux/*nix systems is a cultural artifact, and it highlights the differences between two different philosophies for interacting with the user of the system. As a cultural practice, perhaps there is not one right way to do something, and the best way depends on your cultural orientation. For example, I recently learned that as a dinner guest in France, bringing a bottle of wine may be considered offensive. Who knew?

And I have been sadly remiss for some time by suggesting that mounting external drives in macOS is "straightforward" - it's actually anything but "straightforward"!

If that doesn't satisfy you, try this: It's self-evident that the drive-mounting process for many *nix systems (including RPi OS) is more complicated, and therefore time- & labor-intensive than it is for a Mac (or PC). Compared against the simpler mounting process for a Mac, we are hard-pressed to identify any real, compelling advantage that would induce a rational, unbiased person of average intelligence or better to use the more complicated process when given the choice. It comes down to this: If things are "better" in some way as a result of taking a more challenging path to our objective, the rational person may be inclined to take on that extra effort. But if there are no gains or advantages to "pay for" that extra investment of time and effort then one might question that approach. Cultural sensitivity demands awareness and a non-judgmental attitude. Be aware that there are people from the *nix culture who feel quite strongly that their way is the correct way, and will argue their point vehemently. That said, a more user-centric approach seems to be gaining some traction in the *nix culture; there are now Linux "desktop systems" that mount external drives when plugged into the system - same as with the Mac. But many, including RPi OS 'Lite', still require a "manual" mount process.

But none of this changes one simple fact: If you want to experiment with a Raspberry Pi, you'll need to adapt to this "Unix culture". And here's a carrot: learning is good, and it will clearly improve your skills in your Mac (maybe even PC) environment.

Getting to the job at hand (finally)

Since I deploy my RPi's in headless mode, this recipe reflects my choices. For example, I like to use the exFAT, or exfat file system on USB thumb drives connected to the RPi. I've chosen exFAT for the simple reasons that: a) it's supported by Linux, MacOS and Windows, and b) it doesn't have the limits on file size that FAT & FAT32 do (see exFAT details). If you want to use another file system, @wjglenn has written a good article on the "How-To Geek" website reviewing the tradeoffs between the most widely-used file systems. He recommends, with sound rationale, using FAT32; but you're free to choose whatever suits you.

1. Determine what drives are currently connected to the RPi

Before we plug our external drive into the RPi, let's check to learn what drives are already connected; we know there's (probably) at least one. Having a list of connected drives will provide a reliable baseline for comparison in the next step.

$ sudo fdisk --list

Your output may resemble mine (trimmed for brevity) below; you might see 16 "RAM Disks" (discussed below) listed in the output:

Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
  
Disk /dev/ram1: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

...  ad nauseum for '/dev/ram2' through '/dev/ram15'

If your RPi has an SD card, the listing continues as shown below, and it will include at least one more device named /dev/mmcblk0.

We should cover some semantics: A device name refers to the entire disk; in this case /dev/mmcblk0 is the entire SD card. Device names are usually cryptic abbreviations such as: /dev/sda, /dev/sdb, or in this case /dev/mmcblk0. The /dev identifies it as a device, and is followed by a name. The "mmc" part of the device name refers to "multi media card". As we shall see shortly, another common type of device is named "sd", which refers to "SCSI driver" - not Secure Digital. Oddly perhaps, sd device names are also used for USB drives; this, owing perhaps to culturally-biased decisions, or perhaps this explanation is more accurate. Following is the fdisk report on the SD card used in my RPi:

Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xbb8517b1

And immediately following in this same listing, you'll likely also see the two partitions of the SD card (*p1 and *p2): (We'll come back to this concept of a partition shortly...)

Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1       8192    93802    85611 41.8M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      98304 31116287 31017984 14.8G 83 Linux

If you have other Devices or Disks connected, they will also be listed in the fdisk --list output.

We've now seen the output fdisk --list produces. We shall not use it again here as fdisk is primarily a tool for formatting and partitioning block devices, and that's not what we're after at this point. As we've seen, fdisk produces a lot of output that we don't need now, but it's instructive to see what it does.

Next, compare the output of fdisk to that of the lsblk tool; lsblk gives us what we need for the task of mounting an external drive for the RPi. lsblk excludes RAM Disks as they are a special class (contrived actually) of block devices. There are numerous optional arguments for lsblk (man lsblk is your friend); we'll use the --fs (file system) option because its output is beautiful :)

pi@raspberrypi3b:~ $ lsblk --fs

Which yields the following:

NAME        FSTYPE FSVER LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINT
mmcblk0
├─mmcblk0p1 vfat   FAT32 boot       19E2-67CF                             200.9M    20% /boot
└─mmcblk0p2 ext4   1.0   rootfs     97ca6ca8-5cb1-413f-84d0-569efd4e2c0f   25.8G     7% /

This output is indeed beautiful... a nice, concise presentation in "tree" format with headings! Here we see again the SD card device (mmcblk0), and its two partitions: p2 ( root, / ) and p1 ( /boot ). We also note that /boot is reported as formatted in vfat (a variant on FAT) under the heading FSTYPE, whereas root ( / ) is formatted in ext4. Having established our baseline, we'll move on to the next step.

2. Plug the USB drive in the RPi, partition and format it

I'll use a SanDisk Cruzer 8GB USB drive for the balance of this recipe. Plug this drive in to the RPi, and then run lsblk --fs again at the RPi command line:

$ lsblk --fs
NAME        FSTYPE FSVER LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
mmcblk0
├─mmcblk0p1 vfat   FAT32 boot   19E2-67CF                             200.9M    20% /boot
└─mmcblk0p2 ext4   1.0   rootfs 97ca6ca8-5cb1-413f-84d0-569efd4e2c0f   25.8G     7% /

Your output may (will likely) vary from that shown above, unless you happen to have plugged an identical USB drive into your RPi. Note in the lsblk output above the MOUNTPOINT column is empty for sda, and that sda does not have a partition; e.g. sda1. The absence of a partition was deliberate; i.e. existing partitions were removed for the purposes of this example. The absence of a MOUNTPOINT for sda simply reflects the fact that this drive is not yet mounted.

Let's digress briefly to review file systems, file system formats and file system partitions:

NOTE 1: File systems and formats

As mentioned in a previously cited reference, a filesystem is all about the structure for organizing the data stored on a non-volatile memory storage device. This structure is also called the format, and the process of formatting a drive is just applying the chosen filesystem's data structure to the drive/device/partition.

As mentioned previously, I prefer a FAT filesystem for thumb drives due to the fact that FAT can be read and written on all the major OSs: Mac, Linux/Unix and Windows. FAT is a simple filesystem, but this is complicated somewhat by the different flavors of FAT; e.g. FAT16, FAT32, exFAT. I generally prefer the exFAT flavor because it will accommodate larger partition sizes than the other FAT flavors.

NOTE 2: Partitions and their uses

In case you failed to get the definition of a partition provided in the link earlier, let's review that:

The thumb drive itself is a device. It contains a mass of unallocated memory storage, but that unallocated memory is virtually useless to our operating system until it is: 1) partitioned, and 2) formatted. The partitioning process is simply dividing the memory storage on the device into "blocks" of data that can be formatted - or structured - by writing a filesystem onto the partition. A partition can [may] encompass the entire device (i.e. all of its memory), or it can cover only a very small slice of the device's memory. The size of the partition should reflect its intended usage.

Or perhaps this Wikipedia article on disk partitioning will make more sense to you?

Note 3: Evolving Linux Support for exfat

Using the exfat filesystem adds a few wrinkles to the mounting process. Support for exfat in Linux is evolving, and there are some changes we must be aware of if we're to succeed in this task:

First, support for the exfat filesystem has been incorporated in the Linux kernel since version 5.4. In the unlikely event you're still running a kernel earlier than this, you're not out-of-luck, but you will have to adjust your approach, and understand that you'll be limited to the FUSE option.

Verify your kernel version is > 5.4; FWIW I've listed 3 of my systems:

$ uname -sr			# RPi 3B+; Raspbian GNU/Linux 11 (bullseye)
Linux 6.1.21-v7+ 
$ uname -sr			# RPi 4B; Raspbian GNU/Linux 11 (bullseye)
Linux 6.1.21-v8+ 
$ uname -sr			# RPi B+; Raspbian GNU/Linux 10 (buster)
Linux 5.10.103+

Second, some of the tools/utilities for dealing with exfat have been changed, and if you're on a modern version of the RPi OS/kernel, you'll need to have the up-to-date tools. Following old, outdated (or even plain incompetent) blogs and How-Tos can lead you into a world of confusion.

Verify your tools are up-to-date; specifically on a modern, up-to-date system you should be using exfatprogs, and not exfat-utils. You can let apt verify this for you:

$ sudo apt update
$ sudo apt install exfatprogs
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
exfatprogs is already the newest version (1.1.0-1).

On the subject of tools/utilities, I should share this: I tried repeatedly to partition the USB drive in this recipe using fdisk and parted, but all of these attempts failed. I attribute this to my own ignorance. However, I was able to successfully partition the USB drive using gdisk. Consequently, that's what I'll use in the sequel!

Partition:

Following is the sequence I saw/followed for gdisk:

$ sudo gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.6

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): 

# following command sequence used:  n (add new partition), 8300 (filesystem type), w (write), Y (confirm write)
$ lsblk --fs
NAME        FSTYPE FSVER LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
└─sda1

It appears from the lsblk --fs output that we have successfully created a new partition on /dev/sda, namely /dev/sda1.

Format

Since we have kernel support for exfat, we should be able to use mkfs -t exfat:

$ sudo mkfs -t exfat /dev/sda1
exfatprogs version : 1.1.0
Creating exFAT filesystem(/dev/sda1, cluster size=131072)

Writing volume boot record: done
Writing backup volume boot record: done
Fat table creation: done
Allocation bitmap creation: done
Upcase table creation: done
Writing root directory entry: done
Synchronizing...

exFAT format complete!
$ lsblk --fs
NAME        FSTYPE FSVER LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
└─sda1      exfat  1.0          FDD5-6EBA

$ sudo exfatlabel /dev/sda1 CRUZER8GB     # apply a LABEL to the drive
exfatprogs version : 1.1.0
new label: CRUZER8GB

And so we can! We now have a partitioned and formatted exfat USB drive.

3. Mount the USB drive

Before a drive can be mounted, a mount point is needed in the RPi's file system; let's do that. I like to have USB mount points in my $HOME directory, /home/pi. This may not be the best choice for a multi-user system, and others will counsel creating the mount point under /media or /mnt. You can do as you wish, but here's mine:

$ mkdir /home/pi/mntThumbDrv 

That's right - a mount point is just a directory in the computer's file system! Furthermore, that directory is (typically) empty until the mount is completed. Only then does the directory contain any data. Since we've now got everything we need for a mount, let's do that - and confirm it with lsblk --fs:

$ sudo mount -t exfat /dev/sda1 ~/mntThumbDrv
$ lsblk --fs
NAME        FSTYPE FSVER LABEL     UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
└─sda1      exfat  1.0   CRUZER8GB FDD5-6EBA                               7.5G     0% /home/pi/mntThumbDrv

4. Re-Mount each time the RPi reboots:

Now that the mount point has been created, let's consider a question of usage. By usage, I mean "will the drive being mounted be used regularly - or, will it be used only one time?" If one plans that the mounted drive will be used regularly, and it is to be mounted each time the computer is booted, perhaps the easiest approach is to create an entry for that mounted drive in the file /etc/fstab. Otherwise, a one-time mount may be accomplished by running the mount command manually from the terminal - as shown in the preceding code block.

A review of man fstab will get us started toward creation of an entry in /etc/fstab. Open /etc/fstab in your editor, and while reading man fstab, the following line may suggest itself:

#1st field(fs_spec)  2nd field(fs_file)   3rd(fs_vfstype) 4th(fs_mntops)  5th(fs_freq) 6th (fs_passno)
#-----------------------------------------------------------------------------------------------------
# Don't copy headings; they are here for clarity only

LABEL=CRUZER8GB     /home/pi/mntThumbDrv exfat            rw,user,nofail        0      0

We can test this by umounting /dev/sda1, and then running mount -av, and confirm with lsblk --fs:

$ sudo umount /home/pi/mntThumbDrv 
$ sudo mount -av
/proc                    : already mounted
/boot                    : already mounted
/                        : ignored
/home/pi/mntThumbDrv     : successfully mounted
$ lsblk --fs
NAME        FSTYPE FSVER LABEL     UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
└─sda1      exfat  1.0   CRUZER8GB FDD5-6EBA                               7.5G     0% /home/pi/mntThumbDrv 

5. Adjusting permissions and ownership in an exfat drive

Note that the options discussed in this section also pertain to mounting other FAT filesystems (e.g. vfat).

And so we now have a working line for mounting our USB drive in /etc/fstab. Which brings up the question, "Should I have an entry in /etc/fstab if I only use the drive occasionally?"

I would answer that question, "Yes!". The reason is in the user option in the 4th field (fs_mntops); this option allows an unprivileged user to mount the drive:

$ mount ~/mntThumbDrv
$ lsblk --fs
NAME        FSTYPE FSVER LABEL     UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
└─sda1      exfat  1.0   CRUZER8GB FDD5-6EBA                               7.5G     0% /home/pi/mntThumbDrv  

The exfat file system is lightweight; it includes very few of the features of filesystems such as ext4 - the Linux standard filesystem. For example, missing from exfat is the file system's metadata for storing file ownership and file permissions. As a consequence of this, all ownership and permissions in an exfat volume are assigned by the OS at the time the volume is mounted. These values cannot be changed between mounts; i.e. the operations chown and chmod are simply not possible on an exfat partition.

Which gets back to the user option in fs_mntops... If a user mounts the exfat partition, ownership of all files is assigned to that user! If mounted by root (i.e. via sudo), then root has ownership of all files - which can be troublesome under some circumstances. Adding to this flexibility is the option users (plural of user). This option allows multiple users to mount the partition, and gives each of them ownership. [I do not know how file conflicts are managed under multiple users.] I suspect that most RPi systems are not typically used as multi-user systems, but this user/users option still presents some interesting ideas for sharing a USB drive.

The other way to adjust permissions and ownership of an exfat partition is to set it in /etc/fstab. Refer to man mount.exfat-fuse in the FILE SYSTEM OPTIONS section. (I use the reference to mount.exfat-fuse because the Linux kernel maintainers [after 4+ years], have yet to produce any documentation for exfat in man mount! :O ). To set ownership, use the uid= and gid= options. To set permissions, use the umask, dmask or fmask options. Remember that these ownership and permission options apply to all files in the partition, and that they may not be changed while the partition is mounted; i.e.chown and chmod will never work on an exfat partition.

Summary and closing

I think that covers routine mounting of USB drives on RPi. We've focused on mounting exfat partitions connected via USB. That leaves network-mounted drives and the systemd-supported auto mount for another recipe(s).

FINALLY: If you see an error in this "recipe", or you've got an idea to improve it, please fork this repository to your GitHub account, and once it's in your account, submit a "Pull Request" for the corrections or improvements you'd like to see. Tom Hombergs has created a very good tutorial on how to do this.

REFERENCES:

  1. How to format USB with exFAT on Linux - from LinuxConfig.org, covers gdisk usage
  2. Disk partitioning; Wikipedia
  3. 10 fdisk Commands to Manage Linux Disk Partitions
  4. Beginner Geek: Hard Disk Partitions Explained; from How-To-Geek
  5. How to Use Fdisk to Manage Partitions on Linux; from How-To-Geek
  6. Partitioning Disks in Linux; from Baeldung
  7. How To Partition and Format Storage Devices in Linux; from DigitalOcean
  8. Top 6 Partition Managers (CLI + GUI) for Linux
  9. A GitHub repo for exfatprogs
  10. Q&A: Is exfat-utils missing in debian 12?; U&L SE
  11. exFAT filesystem speed and disk usage based on cluster size; from Gabriel Staples
  12. Q&A: Create and format exFAT partition from Linux; from U&L SE
  13. How to Format a USB Disk as exFAT on Linux [Graphically and Command Line]; from ItsFoss.com
  14. Q&A: What is the best way to format a USB stick such that it can be used with both Linux and Windows?; from askUbuntu SE
  15. How to partition USB drive in Linux; from LinuxConfig.org
  16. How to Create and Manage Linux Partitions using Parted; from UbuntuMint
  17. Create a Partition in Linux - A Step-by-Step Guide; from DigitalOcean
  18. How to Format Disk Partitions in Linux
  19. How to Mount and Use an exFAT Drive on Ubuntu Linux; from ItsFoss.com
  20. Search: 'linux format USB drive as exfat'
  21. Search: 'linux format partition'
  22. Search: 'linux disk partition tools'
  23. Search: 'exfat-fuse vs. exfatprogs - which should i use'
  24. Search: 'partition thumb drive with parted'
  25. Search: 'linux exfat FUSE'

-->

-->