mismatching manager/executor arches #5805
Labels
Comments
|
The bug didn't reproduce yesterday when running |
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 19, 2025
Dump the whole flatrpc.ConnectRequest to the logs, so that we can better understand the cause of google#5805
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 19, 2025
Dump the whole flatrpc.ConnectRequest to the logs, so that we can better understand the cause of google#5805
|
One possible bug that may lead to this failure mode is some races on the connection buffer (updating/parsing it concurrently). |
github-merge-queue bot
pushed a commit
that referenced
this issue
Feb 19, 2025
Dump the whole flatrpc.ConnectRequest to the logs, so that we can better understand the cause of #5805
That's what Taras suspects as well. But it's strange that these races are so rare given the amount of connections). |
|
It turns out that a very simple program that connects to the manager's RPC socket and sends a TCP message to it may crash the manager if the message can be parsed as a It's not completely clear which program is doing that, but to avoid crashes we need to ensure that the client is an executor before trusting it. |
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection, the executor sends a ConnectHelloRequest containing its ID assigned to it at startup; - the manager responds with a ConnectHelloResponse containing a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection, the executor sends a ConnectHelloRequest containing its ID assigned to it at startup; - the manager responds with a ConnectHelloResponse containing a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
ramosian-glider
added a commit
to ramosian-glider/syzkaller
that referenced
this issue
Feb 20, 2025
As we figured out in google#5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
github-merge-queue bot
pushed a commit
that referenced
this issue
Feb 20, 2025
As we figured out in #5805, syz-manager treats random incoming RPC connections as trusted, and will crash if a non-executor client sends an invalid packet to it. To address this issue, we introduce another stage of handshake, which includes a cookie exchange: - upon connection from an executor, the manager sends a ConnectHello RPC message to it, which contains a random 64-bit cookie; - the executor calculates a hash of that cookie and includes it into its ConnectRequest together with the other information; - before checking the validity of ConnectRequest, the manager ensures client sanity (passed ID didn't change, hashed cookie has the expected value) We deliberately pick a random cookie instead of a magic number: if the fuzzer somehow learns to send packets to the manager, we don't want it to crash multiple managers on the same machine.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Starting Feb 14, I am seeing random local syz-manager crashes looking as follows:
These are quite rare, but when I was running two syz-managers from two different checkouts in parallel, both died almost simultaneously (this happened two times, on Feb 14 and Feb 18).
@tarasmadan also saw a similar crash on an amd64 instance running overnight, and today we found occurrences of the same error messages (for both arm64 and amd64) in syzbot logs.
The text was updated successfully, but these errors were encountered: