GetConfig JunOS issue #73

horseinthesky · 2022-05-05T18:11:43Z

Hello.

Doing my first steps in Go while trying to implement Go agalogue of:

from ncclient import manager

with manager.connect(
    host="10.10.30.4",
    port=22,
    username="admin",
    password="Juniper",
) as m:
    r = m.get_config("running")
    print(r)

which works perfectly fine.

This is the code:

package main

import (
       "fmt"

       "github.com/scrapli/scrapligo/driver/base"
       "github.com/scrapli/scrapligo/netconf"
)

func getConfig(addr string) {
       d, _ := netconf.NewNetconfDriver(
               addr,
               base.WithPort(22),
               base.WithAuthStrictKey(false),
               base.WithAuthUsername("admin"),
               base.WithAuthPassword("Juniper"),
       )

       err := d.Open()
       if err != nil {
               fmt.Printf("failed to open driver; error: %+v\n", err)
               return
       }
       defer d.Close()

       r, err := d.GetConfig("running")
       if err != nil {
               fmt.Printf("failed to get config; error: %+v\n", err)
               return
       }

       fmt.Printf("Get Config Response:\n%s\n", r.Result)
}

func getRPC(addr string, rpc string) {
       d, _ := netconf.NewNetconfDriver(
               addr,
               base.WithPort(22),
               base.WithAuthStrictKey(false),
               base.WithAuthUsername("admin"),
               base.WithAuthPassword("Juniper"),
       )

       err := d.Open()
       if err != nil {
               fmt.Printf("failed to open driver; error: %+v\n", err)
               return
       }
       defer d.Close()

       r, err := d.RPC(netconf.WithNetconfFilter(rpc))
       if err != nil {
               fmt.Printf("failed to get config; error: %+v\n", err)
               return
       }

       fmt.Printf("Get Config Response:\n%s\n", r.Result)
}


func main() {
       getRPC("10.10.30.4", "<get-route-engine-information/>")
       getConfig("10.10.30.4")
}

RPC works but GetConfig doesn't:

Get Config Response:
<nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/19.2R0/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">
<route-engine-information xmlns="http://xml.juniper.net/junos/19.2R0/junos-chassis">
<route-engine>
<slot>0</slot>
<mastership-state>master</mastership-state>
<mastership-priority>master (default)</mastership-priority>
<status>OK</status>
<memory-dram-size>2002 MB</memory-dram-size>
<memory-installed-size>(2048 MB installed)</memory-installed-size>
<memory-buffer-utilization>45</memory-buffer-utilization>
<cpu-user>1</cpu-user>
<cpu-background>0</cpu-background>
<cpu-system>2</cpu-system>
<cpu-interrupt>1</cpu-interrupt>
<cpu-idle>96</cpu-idle>
<cpu-user1>2</cpu-user1>
<cpu-background1>0</cpu-background1>
<cpu-system1>2</cpu-system1>
<cpu-interrupt1>0</cpu-interrupt1>
<cpu-idle1>95</cpu-idle1>
<cpu-user2>2</cpu-user2>
<cpu-background2>0</cpu-background2>
<cpu-system2>2</cpu-system2>
<cpu-interrupt2>1</cpu-interrupt2>
<cpu-idle2>95</cpu-idle2>
<cpu-user3>2</cpu-user3>
<cpu-background3>0</cpu-background3>
<cpu-system3>3</cpu-system3>
<cpu-interrupt3>1</cpu-interrupt3>
<cpu-idle3>84</cpu-idle3>
<model>RE-VMX</model>
<start-time junos:seconds="1651771969">2022-05-05 17:32:49 UTC</start-time>
<up-time junos:seconds="2112">35 minutes, 12 seconds</up-time>
<last-reboot-reason>Router rebooted after a normal shutdown.</last-reboot-reason>
<load-average-one>0.44</load-average-one>
<load-average-five>0.42</load-average-five>
<load-average-fifteen>0.45</load-average-fifteen>
</route-engine>
</route-engine-information>
</nc:rpc-reply>
]]>]]>

Get Config Response:
<nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/19.2R0/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">
<nc:rpc-error>
<nc:error-type>protocol</nc:error-type>
<nc:error-tag>operation-failed</nc:error-tag>
<nc:error-severity>error</nc:error-severity>
<nc:error-message>syntax error, expecting &lt;candidate/&gt; or &lt;running/&gt;</nc:error-message>
<nc:error-info>
<nc:bad-element>running</nc:bad-element>
</nc:error-info>
</nc:rpc-error>
<nc:rpc-error>
<nc:error-type>protocol</nc:error-type>
<nc:error-tag>operation-failed</nc:error-tag>
<nc:error-severity>error</nc:error-severity>
<nc:error-message>syntax error, expecting &lt;candidate/&gt; or &lt;running/&gt;</nc:error-message>
<nc:error-info>
<nc:bad-element>running</nc:bad-element>
</nc:error-info>
</nc:rpc-error>
</nc:rpc-reply>
]]>]]>

What am I doing wrong here?

And there is one more thing: if I change port to 830 (works great from shell or ncclient) I get:

failed to open driver; error: error reading from transport, cannot continue

Could you pls shed some light on how to use that?! Thank you.

The text was updated successfully, but these errors were encountered:

carlmontanari · 2022-05-05T18:16:48Z

And there is one more thing: if I change prot to 830 (works great from shell or ncclient) I get:

Is a juniper thing iirc. they dont allow pty on 22. So either use 830 as you're doing or you can use the standard transport (I think)

<nc:error-message>syntax error, expecting <candidate/> or <running/></nc:error-message>

is the issue, looks like you are passing running though so obviously that seems wrong/bad. Probably need to get logs to tell anything further.

horseinthesky · 2022-05-05T18:25:44Z

Is a juniper thing iirc. they dont allow pty on 22. So either use 830 as you're doing or you can use the standard transport (I think)

I have both 22/830 enabled:

admin> show configuration system services netconf 
ssh {
    port 830;
}
rfc-compliant;

Shell (ssh [email protected] -s netconf) and Python (with ncclient) work on both ports but scrapligo only works on 22. Don't know what standard transport is. I've just taken these code snippets from examples.

is the issue, looks like you are passing running though so obviously that seems wrong/bad. Probably need to get logs to tell anything further.

How can I do it?

carlmontanari · 2022-05-05T18:27:45Z

I have both 22/830 enabled:

Doesn't matter, we force a pty which Junos doesn't like

How can I do it?

https://github.com/scrapli/scrapligo/blob/main/examples/network_driver/logging/main.go

horseinthesky · 2022-05-05T18:35:45Z

Logs

2022/05/05 21:34:19 debug::10.10.30.4::22::"attempting to open netconf transport connection with the following command: [10.10.30.4 -p 22 -o ConnectTimeout=30 -o ServerAliveInterval=45 -l admin -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -F /dev/null -tt -s netconf] 2022/05/05 21:34:19 debug::10.10.30.4::22::netconf transport connection to host opened 2022/05/05 21:34:19 debug::10.10.30.4::22::attempting in channel ssh authentication 2022/05/05 21:34:19 debug::10.10.30.4::22::read: Warning: Permanently added '10.10.30.4' (ECDSA) to the list of known hosts. 2022/05/05 21:34:19 debug::10.10.30.4::22::read: 2022/05/05 21:34:19 debug::10.10.30.4::22::read: Password: 2022/05/05 21:34:19 debug::10.10.30.4::22::found password prompt, sending password 2022/05/05 21:34:19 write::10.10.30.4::22::write: REDACTED 2022/05/05 21:34:19 write::10.10.30.4::22::write: 2022/05/05 21:34:19 debug::10.10.30.4::22::read: 2022/05/05 21:34:19 debug::10.10.30.4::22::read: 2022/05/05 21:34:19 debug::10.10.30.4::22::read: 2022/05/05 21:34:19 debug::10.10.30.4::22::read: 2022/05/05 21:34:19 debug::10.10.30.4::22::read: urn:ietf:params:netconf:base:1.0 2022/05/05 21:34:19 debug::10.10.30.4::22::read: urn:ietf:params:netconf:capability:candidate:1.0 urn:ietf:params:netconf:capability:confirmed-commit:1.0 2022/05/05 21:34:19 debug::10.10.30.4::22::read: urn:ietf:params:netconf:capability:validate:1.0 urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file 2022/05/05 21:34:19 debug::10.10.30.4::22::read: urn:ietf:params:xml:ns:netconf:base:1.0?module=ietf-netconf&revision=2011-06-01 2022/05/05 21:34:19 debug::10.10.30.4::22::read: urn:ietf:params:xml:ns:netconf:capability:candidate:1.0 2022/05/05 21:34:19 debug::10.10.30.4::22::read: urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0 2022/05/05 21:34:19 debug::10.10.30.4::22::read: urn:ietf:params:xml:ns:netconf:capability:validate:1.0 urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file urn:ietf:params:xml:ns:yang:ietf-inet-types?module=ietf-inet-types&revision=2013-07-15 urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring http://xml.juniper.net/netconf/junos/1.0 http://xml.juniper.net/dmi/system/1.0 10180 ]]>]]> 2022/05/05 21:34:19 debug::10.10.30.4::22::ssh authentication complete 2022/05/05 21:34:19 info::10.10.30.4::22::sending client capabilities: urn:ietf:params:netconf:base:1.0 ]]>]]> 2022/05/05 21:34:19 write::10.10.30.4::22::write: urn:ietf:params:netconf:base:1.0 ]]>]]> 2022/05/05 21:34:19 write::10.10.30.4::22::write: 2022/05/05 21:34:19 info::10.10.30.4::22::"sending channelInput: ]]>]]>; stripPrompt: false; eager: true 2022/05/05 21:34:19 write::10.10.30.4::22::write: ]]>]]> 2022/05/05 21:34:19 write::10.10.30.4::22::write: 2022/05/05 21:34:19 debug::10.10.30.4::22::read: 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 0 master 2022/05/05 21:34:20 debug::10.10.30.4::22::read: master (default) OK 2002 MB (2048 MB installed) 44 3 0 3 0 94 1 0 2 0 96 1 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 0 2 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 0 96 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 1 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 0 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 2 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 0 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 94 2022/05/05 21:34:20 debug::10.10.30.4::22::read: RE-VMX 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 2022-05-05 17:32:49 UTC 1 hour, 1 minute, 33 seconds Router rebooted after a normal shutdown. 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 0.49 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 0.57 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 0.49 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 2022/05/05 21:34:20 debug::10.10.30.4::22::read: 2022/05/05 21:34:20 debug::10.10.30.4::22::read: ]]>]]> 2022/05/05 21:34:20 debug::10.10.30.4::22::found prompt match 2022/05/05 21:34:20 debug::10.10.30.4::22::server echo is unset, determining if server echoes inputs now 2022/05/05 21:34:20 info::10.10.30.4::22::server does *not* echo inputs, setting serverEcho to 'false' Get Config Response: 0 master master (default) OK 2002 MB (2048 MB installed) 44 3 0 3 0 94 1 0 2 0 96 1 0 2 0 96 1 0 2 0 94 RE-VMX 2022-05-05 17:32:49 UTC 1 hour, 1 minute, 33 seconds Router rebooted after a normal shutdown. 0.49 0.57 0.49 ]]>]]>

2022/05/05 21:34:20 debug::10.10.30.4::22::transport connection to host closed
2022/05/05 21:34:20 debug::10.10.30.4::22::"attempting to open netconf transport connection with the following command: [10.10.30.4 -p 22 -o ConnectTimeout=30 -o ServerAliveInterval=45 -l admin -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -F /dev/null -tt -s netconf]
2022/05/05 21:34:20 debug::10.10.30.4::22::netconf transport connection to host opened
2022/05/05 21:34:20 debug::10.10.30.4::22::attempting in channel ssh authentication
2022/05/05 21:34:20 debug::10.10.30.4::22::read: Warning: Permanently added '10.10.30.4' (ECDSA) to the list of known hosts.
2022/05/05 21:34:20 debug::10.10.30.4::22::read:
2022/05/05 21:34:20 debug::10.10.30.4::22::read: Password:
2022/05/05 21:34:20 debug::10.10.30.4::22::found password prompt, sending password
2022/05/05 21:34:20 write::10.10.30.4::22::write: REDACTED
2022/05/05 21:34:20 write::10.10.30.4::22::write:
2022/05/05 21:34:20 debug::10.10.30.4::22::read:
2022/05/05 21:34:20 debug::10.10.30.4::22::read:
2022/05/05 21:34:20 debug::10.10.30.4::22::read:
2022/05/05 21:34:20 debug::10.10.30.4::22::read: <nc:hello xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
nc:capabilities
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:capabilityurn:ietf:params:netconf:base:1.0</nc:capability>
nc:capabilityurn:ietf:params:netconf:capability:candidate:1.0</nc:capability>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:capabilityurn:ietf:params:netconf:capability:confirmed-commit:1.0</nc:capability>
nc:capabilityurn:ietf:params:netconf:capability:validate:1.0</nc:capability>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:capabilityurn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file</nc:capability>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:capabilityurn:ietf:params:xml:ns:netconf:base:1.0?module=ietf-netconf&revision=2011-06-01</nc:capability>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:capabilityurn:ietf:params:xml:ns:netconf:capability:candidate:1.0</nc:capability>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:capabilityurn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</nc:capability>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:capabilityurn:ietf:params:xml:ns:netconf:capability:validate:1.0</nc:capability>
nc:capabilityurn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file</nc:capability>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:capabilityurn:ietf:params:xml:ns:yang:ietf-inet-types?module=ietf-inet-types&revision=2013-07-15</nc:capability>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:capabilityurn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</nc:capability>
nc:capabilityhttp://xml.juniper.net/netconf/junos/1.0</nc:capability>
nc:capabilityhttp://xml.juniper.net/dmi/system/1.0</nc:capability>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: </nc:capabilities>
nc:session-id10186</nc:session-id>
</nc:hello>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: ]]>]]>
2022/05/05 21:34:20 debug::10.10.30.4::22::ssh authentication complete
2022/05/05 21:34:20 info::10.10.30.4::22::sending client capabilities:

urn:ietf:params:netconf:base:1.0

]]>]]>
2022/05/05 21:34:20 write::10.10.30.4::22::write:

urn:ietf:params:netconf:base:1.0

]]>]]>
2022/05/05 21:34:20 write::10.10.30.4::22::write:
2022/05/05 21:34:20 info::10.10.30.4::22::"sending channelInput: ]]>]]>; stripPrompt: false; eager: true
2022/05/05 21:34:20 write::10.10.30.4::22::write: ]]>]]>
2022/05/05 21:34:20 write::10.10.30.4::22::write:
2022/05/05 21:34:20 debug::10.10.30.4::22::read: <nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/19.2R0/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">
nc:rpc-error
nc:error-typeprotocol</nc:error-type>
nc:error-tagoperation-failed</nc:error-tag>
nc:error-severityerror</nc:error-severity>
nc:error-messagesyntax error, expecting <candidate/> or <running/></nc:error-message>
nc:error-info
nc:bad-elementrunning</nc:bad-element>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: </nc:error-info>
</nc:rpc-error>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:rpc-error
nc:error-typeprotocol</nc:error-type>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:error-tagoperation-failed</nc:error-tag>
nc:error-severityerror</nc:error-severity>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: nc:error-messagesyntax error, expecting <candidate/> or <running/></nc:error-message>
nc:error-info
nc:bad-elementrunning</nc:bad-element>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: </nc:error-info>
</nc:rpc-error>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: </nc:rpc-reply>
2022/05/05 21:34:20 debug::10.10.30.4::22::read: ]]>]]>
2022/05/05 21:34:20 debug::10.10.30.4::22::found prompt match
2022/05/05 21:34:20 debug::10.10.30.4::22::server echo is unset, determining if server echoes inputs now
2022/05/05 21:34:20 info::10.10.30.4::22::server does not echo inputs, setting serverEcho to 'false'
Get Config Response:
<nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/19.2R0/junos" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">
nc:rpc-error
nc:error-typeprotocol</nc:error-type>
nc:error-tagoperation-failed</nc:error-tag>
nc:error-severityerror</nc:error-severity>
nc:error-messagesyntax error, expecting <candidate/> or <running/></nc:error-message>
nc:error-info
nc:bad-elementrunning</nc:bad-element>
</nc:error-info>
</nc:rpc-error>
nc:rpc-error
nc:error-typeprotocol</nc:error-type>
nc:error-tagoperation-failed</nc:error-tag>
nc:error-severityerror</nc:error-severity>
nc:error-messagesyntax error, expecting <candidate/> or <running/></nc:error-message>
nc:error-info
nc:bad-elementrunning</nc:bad-element>
</nc:error-info>
</nc:rpc-error>
</nc:rpc-reply>
]]>]]>
2022/05/05 21:34:20 debug::10.10.30.4::22::transport connection to host closed

carlmontanari · 2022-05-05T18:39:51Z

Hmm can you try that again in a code block or something -- looks like stuff got cut out -- dont even see any rpc request going out at all

horseinthesky · 2022-05-05T18:45:22Z

Hmm can you try that again in a code block or something -- looks like stuff got cut out -- dont even see any rpc request going out at all

https://ctxt.io/2/AADgJADDFA

here is what happens just before RPC error reply:

2022/05/05 21:41:34 write::10.10.30.4::22::write:
2022/05/05 21:41:34 info::10.10.30.4::22::"sending channelInput: <?xml version="1.0" encoding="UTF-8"?><rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running></running></source></get-config></rpc>]]>]]>; stripPrompt: false; eager: true
2022/05/05 21:41:34 write::10.10.30.4::22::write: <?xml version="1.0" encoding="UTF-8"?><rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101"><get-config><source><running></running></source></get-config></rpc>]]>]]>
2022/05/05 21:41:34 write::10.10.30.4::22::write:

carlmontanari · 2022-05-05T19:27:04Z

Ah thanks, yeah much better.

I am like 99.99999% sure that this is/was because when we send the source tag in the payload the go parser makes it like <running></running> and juniper is furious about this and wants </running>

I found old python logs and that looks to be the case. Offhand not sure how difficult that would be to test -- I feel like I looked into this before and the standard library go xml marshaler wouldn't let m change that but I could be making that up.

You could probably confirm this with me by running the same thing in scrapli netconf (python version) and checking the logs. All the payloads should look identical between the two with I think/hope only that one exception. I did a quick search and I couldn't find anything related to this so that adventure may have happened in chat somewhere or I am hallucinating and making this all up 🙃

horseinthesky · 2022-05-05T19:42:12Z

Ah yes. (Wonderful) Python scrapli_netconf works perfectly fine with:

from scrapli_netconf.driver import NetconfDriver

my_device = {
    "host": "10.10.30.4",
    "auth_username": "admin",
    "auth_password": "Juniper",
    "auth_strict_key": False,
    "port": 22
}

conn = NetconfDriver(**my_device)
conn.open()
response = conn.get_config(source="running")
print(response.result)

But Juniper indeed require self-closing XML nodes. It totally desests doing this:

 ssh [email protected] -s netconf
Password:
<!-- No zombies were killed during the creation of this user interface -->
<!-- user admin, class j-super-user -->
<nc:hello xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
   <nc:capabilities>
    <nc:capability>urn:ietf:params:netconf:base:1.0</nc:capability>
    <nc:capability>urn:ietf:params:netconf:capability:candidate:1.0</nc:capability>
    <nc:capability>urn:ietf:params:netconf:capability:confirmed-commit:1.0</nc:capability>
    <nc:capability>urn:ietf:params:netconf:capability:validate:1.0</nc:capability>
    <nc:capability>urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:base:1.0?module=ietf-netconf&amp;revision=2011-06-01</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:validate:1.0</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:netconf:capability:url:1.0?scheme=http,ftp,file</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:yang:ietf-inet-types?module=ietf-inet-types&amp;revision=2013-07-15</nc:capability>
    <nc:capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</nc:capability>
    <nc:capability>http://xml.juniper.net/netconf/junos/1.0</nc:capability>
    <nc:capability>http://xml.juniper.net/dmi/system/1.0</nc:capability>
  </nc:capabilities>
  <nc:session-id>10390</nc:session-id>
</nc:hello>
]]>]]>
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
 <capabilities>
   <capability>urn:ietf:params:netconf:base:1.0</capability>
 </capabilities>
</hello>]]>]]>
<rpc>
   <get-config>
    <source>
      <running></running>
    </source>
   </get-config>
 </rpc>
]]>]]><nc:rpc-reply xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:junos="http://xml.juniper.net/junos/19.2R0/junos">
<nc:rpc-error>
<nc:error-type>protocol</nc:error-type>
<nc:error-tag>operation-failed</nc:error-tag>
<nc:error-severity>error</nc:error-severity>
<nc:error-message>syntax error, expecting &lt;candidate/&gt; or &lt;running/&gt;</nc:error-message>
<nc:error-info>
<nc:bad-element>running</nc:bad-element>
</nc:error-info>
</nc:rpc-error>
<nc:rpc-error>
<nc:error-type>protocol</nc:error-type>
<nc:error-tag>operation-failed</nc:error-tag>
<nc:error-severity>error</nc:error-severity>
<nc:error-message>syntax error, expecting &lt;candidate/&gt; or &lt;running/&gt;</nc:error-message>
<nc:error-info>
<nc:bad-element>running</nc:bad-element>
</nc:error-info>
</nc:rpc-error>
</nc:rpc-reply>
]]>]]>

Self-closing nodes were one of my other questions regarding Go (it is so painful to transform YAML config info XML payload manually writing structs after Python T_T).

carlmontanari · 2022-05-05T19:44:12Z

Ah man, thanks for confirming... I thought I was going crazy for a bit there... 😁

Ok, I'll keep this open and maybe have a look at if we can figure that out this weekend or next weekend. Will be low priority for me since I dont do anything w/ Juniper, but eventually I'll get to it.... if you want to experiment in the mean time and open a PR/chat about that works too!

Carl

horseinthesky · 2022-05-05T19:49:14Z

I would gladly after I figure out how to make these damn self-closing nodes even happen :P

I was playing with one of my Python projects trying to make a Go flavour of it. And this is where I am at the moment:

package parser

import (
        "encoding/xml"
        "errors"
        "fmt"
)

type Action int

const (
        Discard Action = iota
        Accept
)

func (a Action) String() string {
        return [...]string{
                "discard",
                "accept",
        }[a]
}

func (a *Action) UnmarshalYAML(unmarshal func(v interface{}) error) error {
        var action string

        if err := unmarshal(&action); err != nil {
                return err
        }

        switch action {
        case "accept":
                *a = Accept
        case "discard":
                *a = Discard
        default:
                return errors.New("unknown action: " + action)
        }

        return nil
}

func (a Action) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
        var s string

        switch a {
        case 0:
                s = "discard"
        case 1:
                s = "accept"
        default:
                return errors.New("unknown action int: " + fmt.Sprint(a))
        }

        return e.EncodeElement("<" + s + "/>", start)
}

This e.EncodeElement makes me feel pain:

<route>
    <name>BLABLA</name>
    <match>
        <destination>103.3.62.64/32</destination>
        <destination-port>14433</destination-port>
        <destination-port>14444</destination-port>
    </match>
    <then>&lt;discard/&gt;</then>
</route>

carlmontanari · 2022-05-05T20:38:59Z

Hehe, yeah I feel you... go can be "fun" :D

I have been dealing w/ custom json marshallers/unmarshallers for some unrelated things, so I think I can maybe tackle it.... just wont be a priority for me so may take a while.

Thanks for the help hunting this stuff down!

Carl

hellt · 2022-05-05T20:47:27Z

@carlmontanari is python scrapli uses self closing tags with juniper exclusively?

Also here Kirk says it works on his vmx
https://twitter.com/kirkbyers/status/1522315970613702657?s=21&t=BKJam5-uvwUSTZZfmAwH-A

Which version exhibits this error @horseinthesky ?

ktbyers · 2022-05-05T20:48:09Z

Yeah, I happened to be doing a bunch of NETCONF testing recently, so here is what I sent (obviously wrapped in the rpc tags):

(Pdbr) print(get_cfg2)

    <get-config>
        <source>
            <running></running>
        </source>
    </get-config>

ktbyers · 2022-05-05T20:50:32Z

I was using ncclient (as I had that code all handy):

nc_reply = m.dispatch(etree.fromstring(get_cfg2))

(Pdbr) p nc_reply
... a bunch of stuff
        </services>
        <host-name>vmx1</host-name>
    </system>
</configuration>
</data>
</rpc-reply>

This was Junos <version>18.4R1.8</version>

ktbyers · 2022-05-05T21:09:53Z

As a test, I also tested crazy old SRXs that I have and they also worked fine using:

    <get-config>
        <source>
            <running></running>
        </source>
    </get-config>

    <version>12.1X44-D35.5</version>

hellt · 2022-05-05T21:27:23Z

@ktbyers the version in questions is 19.2+
If you have it, it would be nice to put it to test

ktbyers · 2022-05-05T21:33:50Z

I don't have 19.2+. The two above are the Junos versions that I have.

horseinthesky · 2022-05-06T06:53:04Z

Which version exhibits this error @horseinthesky ?

It's Junos: 19.2R1.8. And it is even older than my production VMs/Hardware boxes. So at the moment I don't have anything older than that to test if older versions we fine to accept double tags ¯_(ツ)_/¯

Anyway self-closing tags are super widely used and it is necessary to support them. I've spent a couple of days and still don't know how to do it with encoding/xml T_T

hellt · 2022-05-06T07:40:26Z

One thing to try out is this command
https://www.juniper.net/documentation/us/en/software/junos/netconf/topics/ref/statement/rfc-compliant-edit-system-services-netconf.html

horseinthesky · 2022-05-06T07:43:07Z

One thing to try out is this command juniper.net/documentation/us/en/software/junos/netconf/topics/ref/statement/rfc-compliant-edit-system-services-netconf.html

This is always enabled:
#73 (comment)

hellt · 2022-05-06T08:05:32Z

Nice. Funny, that looks like a Juniper issue to me
I'm curious to see if you can check the same on an even newer version that you have in lab

horseinthesky · 2022-05-06T08:52:30Z

I just checked this morning on a couple of my prod devices. Found a few with 18.3, a few with 19.4 and the most run 21.1.
QFX, vRR, MX960 all have this behavior.

It kind of goes without saying that root cause of this issue is JunOS doesn't accept double tags. But it doesn't seem as an issue to me.
All Juniper NETCONF docs say "use self-closing tags", all examples use self-closing tags.

So I would say it is the right way to do it.

hellt · 2022-05-06T09:00:15Z

it is opportunistic to violate encoding standard by requiring self-closing tags IMO
why have standards then?

since you mentioned that you have tested it with 18.3 and @kbyers have tested it with 18.4R1.8 I am wondering why you have different results...

horseinthesky · 2022-05-06T09:53:30Z

it is opportunistic to violate encoding standard by requiring self-closing tags IMO
why have standards then?

Is there a standard saying XML must always use double tags? I've always been using self-closing tags if I had no content but only due to minimalism and clarity principle.

It is not just about NETCONF but widely used everywhere:

export default function App() {
  return (
    <Container fluid className="App">
      <Header />
      <Body sidebar>
        <Posts />
      </Body>
    </Container>
  );
}
``

> since you mentioned that you have tested it with 18.3 and @kbyers have tested it with 18.4R1.8 I am wondering why you have
> different results...

18.3R3-S2.5 qfx5120-48y-8c to be exact.

hellt · 2022-05-06T10:13:22Z

Is there a standard saying XML must always use double tags?

of course there is - https://www.w3.org/TR/xml/
an important note here is that this document doesn't say that empty elements MUST use empty element tags (<tag/> notation)
You may use any open XML validator to check that <running></running> is a valid XML

from https://www.w3schools.com/xml/xml_elements.asp:

An element with no content is said to be empty. In XML, you can indicate an empty element like this:

<element></element>

You can also use a so called self-closing tag:

<element />

The two forms produce identical results in XML software (Readers, Parsers, Browsers).

I am not saying that using empty tags is wrong. I am saying that not using empty tags is not wrong.

hellt · 2022-05-06T10:21:27Z

@carlmontanari this thread tracks the (lack of) progress on the Go side to support empty tags golang/go#21399

ADD:
@carlmontanari @horseinthesky
this example https://go.dev/play/p/W5cQkqS0h_ shows how one can squash tags to empty notation after marshalling

hellt · 2022-05-06T11:51:05Z

@ktbyers do you mind to perform the test without ncclient (to exclude lxml interaction) and do it the same way as here #73 (comment) ?

ADD: because lxml automatically squashes non empty elements with empty value to empty tags - https://stackoverflow.com/questions/34111154/how-can-i-prevent-lxml-from-auto-closing-empty-elements-when-serializing-to-stri

horseinthesky · 2022-05-06T12:19:39Z

I am not saying that using empty tags is wrong. I am saying that not using empty tags is not wrong.

I don't mind and/or argue. JunOS does :D

this example go.dev/play/p/W5cQkqS0h_ shows how one can squash tags to empty notation after marshalling

This looks terrible but works nicely.

I wonder what could be the reason for Juniper to do validation the way they do it. But what bothers me more is what would be the best solution?!

carlmontanari · 2022-05-06T15:21:33Z

This looks terrible but works nicely.

lol, don't look too closely at scrapligo code then 🤣

I'll play around with this stuff over the weekend and see what shakes out and keep ya'll posted!

ktbyers · 2022-05-06T17:38:55Z

Okay, I tested manually via ssh using:

ssh [email protected] -s netconf

And observed the same failure as @horseinthesky (i.e. it worked for <running/>, but not for <running></running>.

That caused me to dig into more what ncclient was doing. Digging into this more, ncclient actually converted <running></running> to <running/>.

(Pdbr) print(req)
<?xml version="1.0" encoding="UTF-8"?>
<nc:rpc xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="urn:uuid:eaa99530-6bd3-470e-939b-02ac95ed9d6e">
<get-config>
        <source>
            <running/>
        </source>
    </get-config></nc:rpc>

Where what I sent is:

(Pdbr) p get_cfg2
'\n    <get-config>\n        <source>\n            <running></running>\n        
</source>\n    </get-config>\n'

Digging in more to this, I see that lxml actually converts the:

<running></running>

To an empty tag:

(Pdbr) print(get_cfg2)

    <get-config>
        <source>
            <running></running>
        </source>
    </get-config>

(Pdbr) my_xml = etree.fromstring(get_cfg2)
(Pdbr) my_xml
<Element get-config at 0x7f46a42fcc00>

(Pdbr) etree.tostring(my_xml)
b'<get-config>\n        <source>\n            <running/>\n        </source>\n   
</get-config>'

It looks like per the netconf-1.0 yang model and per the XML spec it "should" be an empty tag, but a bit surprising Juniper doesn't allow it both ways.

https://stackoverflow.com/questions/7231902/self-closing-tags-in-xml-files

From the above page (which is quoting from the XML 1.0 spec):

Empty-element tags may be used for any element which has no content, whether or not it is declared 
using the keyword EMPTY. For interoperability, the empty-element tag SHOULD be used, and SHOULD only 
be used, for elements which are declared EMPTY.

And netconf 1.0 yang model declares source > running as empty.

hellt · 2022-05-06T18:23:29Z

@ktbyers where do you see that running is declared empty?

ktbyers · 2022-05-06T18:37:40Z

https://github.com/YangModels/yang/blob/main/standard/ietf/RFC/ietf-netconf%402011-06-01.yang#L382-L386

@hellt

hellt · 2022-05-06T18:53:30Z

that is a YANG type, which I think has nothing to do with actual EMPTY definition XML standard talks about.
XML has its own XML schema that defines a data model for data encoded with XML, and I think this xml schema has a type EMPTY the original standard refers to.

For example, netconf RFC has the following line (https://datatracker.ietf.org/doc/html/rfc6241#section-6.2.4):

For filtering purposes, an empty leaf node can be declared either with an empty tag (e.g., ) or with explicit start and end tags (e.g., ).

which kind of proves that both options are fine for a netconf server

ktbyers · 2022-05-06T20:06:59Z

@hellt Yeah, I probably disagree on that one.

YANG is the DTD/xmlschema equivalent for NETCONF and the YANG-model says it is an empty leaf node which when translated into XML probably should be as per the XML spec for something declared empty.

The RFC section you referenced above is in the context of filters so I don't think I would generalize it the way you did. I also had read that earlier.

Anyways there is a good chance that is why Juniper did it that way and it is a bit moot as there are thousands of field Juniper devices with this behavior.

horseinthesky · 2022-05-07T10:21:31Z

That's true. I have jinja2 XML templates describing 95% of JunOS functionality and they use self-closing tags literally everywhere.

hellt · 2022-05-07T13:04:04Z

I am not buying that =) To me requiring self closing tags is a mistake of a parser. It should be accepted in an expanded mode.
We need to have an authoritative answer from someone from netmod wg

carlmontanari · 2022-05-07T16:24:31Z

Well, good news re the self closing being ok or not thing -- I (and by extension scrapligo!) dont care which way is right/best/good 🤣

TL;DR here is I'll muck around with trying to make this an option in the least ugly way possible, then probably expose it as an option at driver creation that way we can leave standard xml behavior then users can enable this if they care. Probably wont work on this till next weekend but at least we know what needs doing to sort this issue. Thanks for all the work double checking all the things from everyone!

Carl

earies · 2022-05-14T15:45:59Z

@hellt - what we have here can be left up to interpretation based off the angle in which you are looking at this from.

The primary source of truth for the XML encoding of YANG data are the YANG 1.0/1.1 RFCs themselves. As previously pointed out, the YANG module that defines the RFC6241 RPCs for <get-config> utilizes leaf nodes of type empty for the datastore sources.

XML encoding of type empty is conveyed (albeit a bit loosely) in RFC6020/RFC7950 https://datatracker.ietf.org/doc/html/rfc7950#section-9.11.4 which indicates a single self-closing tag (There will never be text content). In JUNOS, we follow this and could be interpreted as strict compliance (and common usage). Any libraries that generate XML content would be unaware of this as both flattened self-closing and start/end tags without text content are viewed as one in the same.

The reference you have above to https://datatracker.ietf.org/doc/html/rfc6241#section-6.2.4 is mostly unrelated.

Now at the same time, since RFC7950 brought in the ability for leafs of type empty to be valid list keys, this raised the issue of how to encode empty types as predicates (since you have k=v) and Section 9.13 specifies this as a 0-length string which in reality is no different than <running></running>

yanglint for instance will validate both scenarios (and actually collapse the start/end tags post validation) as well as length checking should the start/end elements contain any content

libyang err : Invalid empty value length 1. (Schema location /main:main/test, data location /main:main, line number 2.)

So my opinion is both are valid, this could be relaxed but the common usage would still be to formulate a single self-closing element for nodes of type empty (which will also save you a few bytes on the wire). This is more of an accommodation for the scenarios in which tooling is not collapsing (and unaware of the YANG types which would never have any possibility of text content)

carlmontanari · 2022-05-15T15:59:42Z

thanks again everyone, closing this and following up in #76

horseinthesky · 2022-05-15T17:39:52Z

To make this issue complete to someone else facing it, adding base.WithNetconfForceSelfClosingTags(true) to the driver makes it work:

package main

import (
       "fmt"

       "github.com/scrapli/scrapligo/driver/base"
       "github.com/scrapli/scrapligo/netconf"
)

func getConfig(addr string) {
       d, _ := netconf.NewNetconfDriver(
               addr,
               base.WithPort(22),
               base.WithAuthStrictKey(false),
               base.WithAuthUsername("admin"),
               base.WithAuthPassword("Juniper"),
               base.WithNetconfForceSelfClosingTags(true),
       )

       err := d.Open()
       if err != nil {
               fmt.Printf("failed to open driver; error: %+v\n", err)
               return
       }
       defer d.Close()

       r, err := d.GetConfig("running")
       if err != nil {
               fmt.Printf("failed to get config; error: %+v\n", err)
               return
       }

       fmt.Printf("Get Config Response:\n%s\n", r.Result)
}

func main() {
       getConfig("10.10.30.4")
}

horseinthesky · 2022-05-15T17:43:22Z

@carlmontanari It is probably another noob question from me but how should I correctly update the dependency?
If I do go get github.com/scrapli/scrapligo@latest - nothing happens. I mean it is not downloading new code.
Even if I manually delete go.mod, go.sum in my package and the whole ~/go/pkg/mod/github.com/scrapli and run go mod init bla and go mod tidy it downloads 0.1.3 version but without new changes with WithNetconfForceSelfClosingTags flag.

carlmontanari · 2022-05-15T17:44:58Z

you can just do go get blah@COMMITHASH should work well enough!

horseinthesky · 2022-05-15T17:50:27Z

you can just do go get blah@COMMITHASH should work well enough!

Yeah. I did this to test the branch version. But I don't get why it downloads not the latest version without pointing the hash?

carlmontanari · 2022-05-15T17:53:17Z

pretty sure it just gets latest release if you dont do it with hash, I didn't make a release, so thats what you're gunna get. for now you'll just need the hash.

horseinthesky · 2022-05-15T17:58:01Z

pretty sure it just gets latest release if you dont do it with hash, I didn't make a release, so thats what you're gunna get. for now you'll just need the hash.

So since there was no new release it uses commit with the latest tag (which is 0.1.3 at the moment) as @latest. Is that correct?

carlmontanari · 2022-05-15T17:58:37Z

afaik yep

philshafer · 2022-05-31T19:42:58Z

FWIW: This is definitely a bug in JUNOS (PR 1668382). It's specific to the the NETCONF tag.

Thanks,
Phil

horseinthesky closed this as completed May 6, 2022

horseinthesky reopened this May 6, 2022

carlmontanari mentioned this issue May 15, 2022

force self closing tag option #76

Merged

carlmontanari closed this as completed May 15, 2022

nemith mentioned this issue Sep 4, 2024

proposal: encoding/xml: Add EmptyElement token type to support self-closing elements. golang/go#69273

Open

GetConfig JunOS issue #73

GetConfig JunOS issue #73

Comments

horseinthesky commented May 5, 2022 • edited Loading

carlmontanari commented May 5, 2022

horseinthesky commented May 5, 2022

carlmontanari commented May 5, 2022

horseinthesky commented May 5, 2022

carlmontanari commented May 5, 2022

horseinthesky commented May 5, 2022

carlmontanari commented May 5, 2022

horseinthesky commented May 5, 2022 • edited Loading

carlmontanari commented May 5, 2022

horseinthesky commented May 5, 2022

carlmontanari commented May 5, 2022

hellt commented May 5, 2022

ktbyers commented May 5, 2022 • edited Loading

ktbyers commented May 5, 2022

ktbyers commented May 5, 2022

hellt commented May 5, 2022

ktbyers commented May 5, 2022

horseinthesky commented May 6, 2022

hellt commented May 6, 2022

horseinthesky commented May 6, 2022

hellt commented May 6, 2022

horseinthesky commented May 6, 2022

hellt commented May 6, 2022 • edited Loading

horseinthesky commented May 6, 2022 • edited Loading

hellt commented May 6, 2022 • edited Loading

hellt commented May 6, 2022 • edited Loading

hellt commented May 6, 2022 • edited Loading

horseinthesky commented May 6, 2022

carlmontanari commented May 6, 2022

ktbyers commented May 6, 2022 • edited Loading

hellt commented May 6, 2022

ktbyers commented May 6, 2022

hellt commented May 6, 2022

ktbyers commented May 6, 2022

horseinthesky commented May 7, 2022

hellt commented May 7, 2022

carlmontanari commented May 7, 2022

earies commented May 14, 2022

carlmontanari commented May 15, 2022

horseinthesky commented May 15, 2022

horseinthesky commented May 15, 2022

carlmontanari commented May 15, 2022

horseinthesky commented May 15, 2022

carlmontanari commented May 15, 2022

horseinthesky commented May 15, 2022

carlmontanari commented May 15, 2022

philshafer commented May 31, 2022

horseinthesky commented May 5, 2022 •

edited

Loading

horseinthesky commented May 5, 2022 •

edited

Loading

ktbyers commented May 5, 2022 •

edited

Loading

hellt commented May 6, 2022 •

edited

Loading

horseinthesky commented May 6, 2022 •

edited

Loading

hellt commented May 6, 2022 •

edited

Loading

hellt commented May 6, 2022 •

edited

Loading

hellt commented May 6, 2022 •

edited

Loading

ktbyers commented May 6, 2022 •

edited

Loading