Starting pod on system boot using Systemd

[elhananjair]

Hello friends,
I have created systemd unit file and enabled it. The unit file executes podman pod start mypod; I am working on VMware with Fedora Workstation installed. The problem I am facing is when I run systemctl start myunit.service it starts well and the pod starts. But when the server restarts, it won't start showing the following error:

Error: default OCI runtime "crun" not found: invalid argument

Even though it won't successfully start, after the system boot if I start it manually using systemctl start myunit.service it starts fine. I suspected that maybe podman.socket might not be ready when myunit.service is scheduled to begin, therefore I edited the unit file adding Requires podman.socket. Here is the content of the unit file.

[Unit]
Description=Starts ERP container Service
After=network.target systemd-user-sessions.service plymouth-quit-wait.service
Requires=podman.socket

[Service]
User=liyu_admin
Group=liyu_admin
Type=oneshot
RemainAfterExit=true
Restart=on-failure
ExecStart=podman pod start mypod

[Install]
WantedBy=multi-user.target

I appreciate your support, Thank you

[eriksjolund]

Which commands did you run to create the pod mypod?
(I assume you were logged in as liyu_admin when you were creating the pod)

[eriksjolund]

Anyway here are some tips:

To install crun, run the following command

sudo dnf install -y crun

You might want to enable linger

sudo loginctl enable-linger liyu_admin

Then open a login shell for liyu_admin

sudo machinectl shell --uid liyu_admin

Then enable podman socket

systemctl --user enable --now podman.socket

Then create directory

mkdir -p ~/.config/containers/systemd

Create container units and pod unit files under

~/.config/containers/systemd

Check out the man page https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html

[elhananjair]

Hello @eriksjolund
thank you so much for replying and your suggestions.
I still don't think the problem is with crun, because as I stated above, if I execute systemctl start myunit.service it works fine, this shows there is no problem with the service, but instead, since I am running the OS on VM I think the service starting too soon that's what I am suspecting.
Another clue is I have the same setup on my home Laptop, and when I restart my PC the service starts successfully without a problem and the pod is accessible. I am searching for a way to delay the startup of the systemd unit file.

[eriksjolund]

Do you need a dependency on a systemd system service from a system user service?
That is currently not supported by systemd.
There is a GitHub feature request for adding the functionality:

• RFE: monitor system units from user manager systemd/systemd#3312

But you could create a workaround in a similar way as is being done in in
/usr/lib/systemd/user/podman-user-wait-network-online.service
that contains:

[Unit]
Description=Wait for system level network-online.target as user.
Documentation=https://github.com/containers/podman/issues/22197
Documentation=man:podman-systemd.unit(5)

[Service]
Type=oneshot
# Set a timeout as by default oneshot does not have one and in case network-online.target
# never comes online we do not want to block forever, 90s is the default systemd unit timeout.
TimeoutStartSec=90s
ExecStart=sh -c 'until systemctl is-active network-online.target; do sleep 0.5; done'
RemainAfterExit=yes

[eriksjolund]

Another thing:

Using the systemd directive User= is not supported when running podman.
In other words this is not supported:

[Service]
User=liyu_admin
Group=liyu_admin

There is a Podman feature request
for adding Podman support for User= in systemd system services.
The feature request was moved into a GitHub discussion.

[elhananjair]

Another thing:

Using the systemd directive User= is not supported when running podman. In other words this is not supported:

[Service]
User=liyu_admin
Group=liyu_admin

There is a Podman feature request for adding Podman support for User= in systemd system services. The feature request was moved into a GitHub discussion.

Thank you so much let me try and will let you know the result

[elhananjair]

Another thing:

Using the systemd directive User= is not supported when running podman. In other words this is not supported:

[Service]
User=liyu_admin
Group=liyu_admin

There is a Podman feature request for adding Podman support for User= in systemd system services. The feature request was moved into a GitHub discussion.

Ok, I can remove that, although it is not making any difference while runing the service after the system boots.

[elhananjair]

Hello @eriksjolund
Removing User and Group from the unit file doesn't help at all and it causes an issue:

 Error: no pod with name or ID mypod found: no such pod

FYI: I am not generating the systemd unit file using podman-generate-systemd. Therefore it might require specifying a user since podman is rootless by default; podman pod start mypod should be executed by the owner of the pod.

I have added the following parameters after checking on the forum you have suggested:

Type=oneshot
RemainAfterExit=yes
ExecStartPre=nm-online --timeout=30
TimeoutStartSec=90s
ExecStart=podman pod start mypod

and still getting an error with no detailed logs

Feb 07 14:13:18 fedora systemd[1]: Starting erp.service - Starts ERP container Service...
Feb 07 14:14:02 fedora nm-online[1070]: [1002B blob data]
Feb 07 14:14:02 fedora systemd[1]: erp.service: Control process exited, code=exited, status=1/FAILURE
Feb 07 14:14:02 fedora systemd[1]: erp.service: Failed with result 'exit-code'.
Feb 07 14:14:02 fedora systemd[1]: Failed to start erp.service - Starts ERP container Service.

[eriksjolund]

Error: no pod with name or ID mypod found: no such pod

To list pods that were created by root run

sudo podman pod list

To list pods that were created by liyu_admin run

 sudo systemd-run --machine=liyu_admin@ --quiet --user --collect --pipe --wait podman pod list

and still getting an error with no detailed logs

Feb 07 14:13:18 fedora systemd[1]: Starting erp.service - Starts ERP container Service...
Feb 07 14:14:02 fedora nm-online[1070]: [1002B blob data]
Feb 07 14:14:02 fedora systemd[1]: erp.service: Control process exited, code=exited, status=1/FAILURE
Feb 07 14:14:02 fedora systemd[1]: erp.service: Failed with result 'exit-code'.
Feb 07 14:14:02 fedora systemd[1]: Failed to start erp.service - Starts ERP container Service.

I recommend writing quadlet files instead of systemd units.
The quadlet files are automatically converted to systemd units by

# rpm -ql podman | grep generator
/usr/lib/systemd/system-generators/podman-system-generator
/usr/lib/systemd/user-generators/podman-user-generator

[elhananjair]

Thank you for replying and supporting me,
I fixed the "no pod found" issue by adding the user and group back to liyu_admin and it works fine. The issue is still with the Virtual Manager since I have a working systemd service on my laptop.
Quadlets were a good option but I had another issue with it and I was discussing it here and is somehow complex with what I was trying to achieve, therefore I just wanted to auto-start the pod with custom systemd unit file with podman pod start exec in it which works fine on my Laptop and rather not on Virtual Machine.
The auto-starting part is not working but I will try to find the issue related to VM.

Thanks anyway for your time

[elhananjair]

Hello @eriksjolund
To update you on the progress of finding the solution to autostart podman using systemd.

I have tried adding ExecStartPre=/bin/sleep 240 and it seems working although it is not a permanent solution. Furthermore, I am trying to create a timer and use OnBootSec=2min.

It might be helpful for anyone using Linux on a VM environment and coming to this discussion in the future.

[elhananjair]

This finally seems working for me

[Unit]
Description=Starts ERP container Service
Requires=podman.socket NetworkManager.service

[Service]
User=USERNAME
Group=GROUPNAME
Type=oneshot
# RemainAfterExit=yes
ExecStartPre=/bin/sleep 120
Restart=on-failure
TimeoutStartSec=infinity
RestartSec=120s
ExecStart=podman pod start pod_mypod

[Install]
WantedBy=multi-user.target