Further adventures in parsing, or how I miss NSO

February 9, 2020 6-minute read

I've written about my parsing adventure before. In fact, I'm going to talk about it at my upcoming conference, DKNOG10. Having to talk about it, means that I'm prioritizing some of the clean up and refactoring tasks I've been holding off for a while. Some of these tasks are pretty big.

I've engulfed in a fairly big task as the first one, namely splitting the whole thing in to smaller components:

A backend to store everything, accessible via a REST API.
A frontend written in a modern Javascript frontend
The parser, making configuration data available as python objects
A synchronizer, responsible for reading configuration files, calling the parser, and updating the backend via the REST API.

Splitting it up means that I need to do some clean up and enhancements at the same time, so while I'm spending the most time with the synchronizer, I have to write new API features at the same time in the backend.

This is because the backend wasn't being used to write data to before, and I mostly only had the REST actions for reading, and not writing.

Luckily, Django REST Framework is a joy to work with, and most changes are really quick to implement.

I've removed the old frontend codes in the backend, so now it only has API functions. This should force me to implement the few missing functions in the new javascript frontend, and stop maintaining two frontends.

I'm also trying to write functional tests in the backend at the same time.

This is all quite a lot of refactoring, and having even spend a lot of hours on this over the weekend, I'm still not done at all.

It has struck me how much I miss NSO when I'm working on this part, even though I'm not exactly doing service activation here, I would have been done three months ago, if I had used NSO for this.

Before going in to details with my process, let me just briefly get you up to speed on the project I'm working on here.

I'm working project "Alcove", a network inventory system built to support the daily operations of an unknown network.

Trying to understand a network that I was unfamiliar with, and that used a platform I had not worked with before, led me to write a configuration parser and a Django app.

Later the Django app became a REST interface, and I wrote a modern frontend.

I wrote the whole thing to get a smart way to search through the network, and a way to visualize various parts of it, such as VRF's and L2VPNs, and I did it as this overly ambitious level, to make it easy for anyone to get the same level of overview.

I'm going to talk about it on DKNOG10 as mentioned before, so when I've done that, I might update this article with the video of my presentation.

The name Alcove comes from Allan's Configuration tool.

Updating databases smartly is hard

Before this code split, I just deleted the node from the database, and database key contraints cleaned up any associated object.

I wanted to have a more update-oriented approach this time, only updating whatever had changed in the network.

This unfortunately means that I have to do a lot of comparison between parsed objects and database objects. This is a bit messy, and requires custom logic for each case.

Here are some takeaways:

Types are hard, and so is modeling

As my parser picks up everything with regular expressions, everything I parse is saved as strings. My database however has types. This leads to me having to implement a type translation function in the sync tool. Otherwise it will not properly detect whether things stored as integers (or booleans) are changed.

My parser gives me a dictionary corresponding roughly to each database table. This leaves me with a flat structure where I manually have to do name-to-key lookup for each relation.

When I fetch an object from the backend, it will contain these relationships with their correct keys. This makes comparing local parser results with their already stored versions tricky.

Just look at this clunky construction I've had to come up with:

    def _update_interface(self, local, remote):
        # Keys from the schema that can be compared
        delta_interface = {}
        changes = []
        update = False
        for key, valtype in self.schema_types.items():
            # Try to infer type

            delta_interface[key] = self._get_with_type(local, key)
            if self._get_with_type(local, key) != remote.get(key, None):
                update = True
#...
     if update:
            # Add node, vrf to delta_interface
            delta_interface['node'] = remote['node']
            delta_interface['vrf'] = remote.get('vrf', None)
            if local.get('port', None) is not None:
                delta_interface['port'] = self._get_remote_port_id(local['port'])
            else:
                delta_interface['port'] = None

            self.actions.append(
                {
                    'action': 'UPDATE',
                    'object': delta_interface
                }
            )

            if self.dry_run is False:
                response = self.api.l3interfaces.update(
                    remote['id'], body=delta_interface
                )

                if response.status_code != 200:
                    raise AlcoveSyncModelError('Could not update interface')

This basically goes through a schema (of sorts) of keys that could have changed, compare them using their correct data type, while creating a new dictionary that will be sent if changes have been detected.

All the relationships and tables!

I'm not even half way through all the various objects I need to write code to be able to update. I'm questioning all my choices, including the choice of not just wiping the node before import at this point.

I'm dreaming of magic meta-code that fixes this for me, but I don't think I can come up with something that can solve all this for me, sadly, and it's the various object relationships that make is non-trivial. Happy to hear any input about this, if you have something. Leave a comment below, please.

What I miss from NSO

This whole process has left an annoying thought that I could achieve this using NSO much, much easier.

The first thing I would need, would be a YANG model. With YANG I could model each service while enforcing types at the same time.

Secondly, I could use natural keying much better. This is not entirely possible with a simple database model, since I would need to use more than one field as key, which is generally not supported, but YANG handles it.

For example, a logical interface could have both name and hostname as key, as interfaces are unique per node. If you use a regular database, you would probably stick with an auto incrementing ID field, or perhaps a UUID field. None of these could be derived without having to look them up in the database.

Parsing would a completely different problem with NSO, since I would now have access to the NED in NSO, and therefore a modelled configuration, so parsing would then be a matter of reconciling configuration in to my new NSO service.

Also, I would get service activation for free, and that would be even more important.

Why didn't I just use NSO, you may wonder, when I speak so nicely about it?

Well, first of all, I don't have access to a license for this network, and if I got one, some of the devices here are unfortunately very old, so I doubt they would work with NSO at all.

Closing notes

Even though things would be easier with NSO, or indeed just with native YANG support, sometimes things just are complicated. Everything with synchronizing two data sources is complicated by nature and even though I need to write a lot of code to solve it, I feel learn a lot from it, and I also detect problems in my original code while doing it.

Hopefully, I end up with clean, nice code.

However, if someone has any suggestions for easier ways around this, I'm always interested in hearing from you.