Version 2.0.0 Release #132
redruin1
announced in
Announcements
Replies: 1 comment 2 replies
-
Awesome work! Nit: I'm not sure you meant "pneumonic". I'm not sure mnemonic would be right either. Consider idiom? |
Beta Was this translation helpful? Give feedback.
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
2.0 Release
At this moment, Draftsman 2.0 is not quite feature complete, so please keep that in mind before upgrading. The vast majority of the module is tested and complete, but your mileage may vary when using the new Factorio 2.0 features. As a result, it might be best to consider Draftsman 2.0.X to be "unstable", until something like 2.1 or therabouts.
This document outlines what is different in the new version, why those things are different, an the direction that future changes to Draftsman will be directed.
Deferred Validation
A big problem with Draftsman 1.0 is that it doesn't provide any real provisions for users working with blueprint configurations that Draftsman has no knowledge of. For example, consider the following case where we try to import a modded entity under a vanilla Factorio configuration:
Draftsman complains because it doesn't recognize the entity, and thus cannot meaningfully check it's correctness. From Draftsman's perspective, this makes sense; what's the dimension of this entity? What does it collide with? Is it circuit connectable? What attributes does it have? What are their allowed values? Because Draftsman cannot validate this object to the caliber of an entity it does know, it decides to consider it a catastrophic error and defer to the user to either remove it or update Draftsman's environment.
This is great for a number of circumstances, such as the case where you thought you were operating in a modded context but actually weren't, or in the case where the entity's name was simply a misspelled version of a known entity, which you likely want to catch as early as possible.
The problem is that there are other situations where this is too restrictive. A user might very well want to just ignore any unknown entities/tiles and simply return a Blueprint with the known ones, which is in fact how Factorio itself behaves. Going further, you might want to actually preserve these modded entities in the created blueprint; you might want to swap the modded entity to a known one which Draftsman can handle, or maybe you don't want to touch the modded stuff at all and simply pass it to the game, asserting that they must be in a valid format and the game will know how to handle it even if Draftsman doesn't.
Because the members of
draftsman.data
are fully writable, you can add new entries in their corresponding dictionaries or lists to "trick" Draftsman into allowing these cases. Unfortunately, there are no helper methods which actually make this a palatable option. Care must be taken to provide all of the necessary information in the exact correct format Draftsman expects, which is also likely to be inconsistent across Draftsman versions to boot. The only real sanctioned way in Draftsman 1.0 for interacting with modded entities is to modify the entire data configuration by runningdraftsman-update
(or the corresponding method inenv.py
). This is easy if you're creating the blueprint string yourself with a set of mods that you're actively playing with, but difficult if:Clearly, this is a design flaw due to a simplification set early on when designing the tool. Since Draftsman runs the data lifecycle to extract validation information and can do so dynamically, I assumed that all of the needed data would be available at the script runtime, when this is not truly the case. As a result, Draftsman 1.0 is essentially a "greedy" validator, requiring comprehensive information about the entire environment before running, which is useful in some settings, but not in others.
Another related flaw about Draftsman 1.0 is that even if you want Draftsman to panic, you cannot tell it when to do so. Suppose for example that we want to swap all instances of a particular modded entity from a blueprint, but we still want to error if we detect any other modded entities. We would like to then write something similar to this:
... but this is also impossible in Draftsman 1.0. Validation of the
Blueprint
always happens at construction, and cannot be deferred until later; the only other option is to modify the data going intoBlueprint
before constructing it, but this would be much more of a hassle and we wouldn't have access to all of the nice helper methods thatBlueprint
already provides, such asfind_entities_filtered()
or similar.Finally, a user may also desire more control of the manner and types of warnings/errors which are issued. Some users might want to check just the format of the input data so that no fields have the incorrect type; others might want a comprehensive analysis of all of the field values, to check for redundancies or conceptual faults, treating Draftsman like a blueprint linter. You might want to treat errors as warnings, or warnings as errors, or perhaps ignore validation completely. What validation should do is more than a simple "yes" or "no", and so a big goal for 2.0 was to not only allow users to control when they can validate their inputs but also configure it to behave exactly as they want.
As a result, in Draftsman 2.0 all Draftsman objects now have a
validate()
function which can be used to check their contents at any point after they're created. The function takes aValidationMode
parameter, which is an enum which indicates the strictness of the validation, which controls the type and quantity of errors and warnings:NONE
: No validation whatsoever. Every attribute remains exactly as it was; even values in a known shorthand format are not converted. Impossible to know whether or not this object will import into Factorio when using this mode. This tells Draftsman to simply treat every object verbatim.MINIMUM
: Only returns formatting errors, where data members are of incorrect types. For example, if you set the name of an entity to an integer, this would raise aDataFormatError
. Besides this, no other warnings or errors are issued. This tells Draftsman to error if the object is in a form that it absolutely knows will NOT import into Factorio.STRICT
: This is the default mode, most closely related to the behavior of Draftsman 1.0. It returns all above errors, as well as most of the errors and warnings most users of the module will be familiar with, in addition to a few new ones. For example, if Draftsman now encounters an entity it doesn't recognize, it issues aUnknownEntityWarning
instead of anInvalidEntityError
; Draftsman doesn't know about this entity, but it may import into Factorio if the game happens to know about it.PEDANTIC
: Issues all above errors and warnings, as well as providing more linting-like behavior as well. A good example is setting the limiting bar of a container beyond it's total inventory slots; this creates no issue when importing into the game, and the container behaves as if the bar was set at that point; but it might indicate a conceptual failure from the programmers perspective or a simple mistake, and as such it will raise aBarWarning
if detected under this validation mode.Instead of raising the errors and warnings in place,
validate()
returns a wrapper object called aValidationResult
. This object contains anerror_list
and awarning_list
attribute, which can be read, modified, iterated over, saved for later, or any combination thereof. This gives the user the ability to convert errors into warnings or warnings into errors, and it allows Draftsman to retain it's previous concepts of "Factorio-safety" and "Factorio-correctness" as the default while still allowing users to deviate from this behavior if required.Most commonly, you'll probably end up writing something like this snippet, which simply reissues any detected errors or warnings found with a blueprint:
Because this particular mnemonic is likely to appear a lot, it's implemented as a helper method called
reissue_all()
:Creating a
ValidationResult
object also makes it very easy to add other similar helper methods like this one as well as additional functionality later on without breaking code in written in earlier versions of 2.0.Similar to their prior behavior, all
Blueprintable
andEntity
subclasses still support validation during construction, with the addition of now being able to configure exactly how using the new keyword argumentvalidate
:In addition to the
validate
parameter, bothBlueprintable
andEntity
subclasses also have avalidate_assignment
parameter, which configures whether or not to run validation when assigning an attribute of the object:In an effort to provide more flexibility while still keeping the API consistent across many different functions and attributes, Draftsman now has 3 "categories" of manipulating data for all the validatable types:
entity["member"] = ...
; This mode is guaranteed to not run validation ever, regardless of the value ofvalidate_assignment
, and does not abide by shorthand formats. As a consequence, this method is also guaranteed to be computationally cheap.entity.member = ...
; The behavior of this mode is configurable, depending on the value ofvalidate_assignment
. Usually the most terse syntax.entity.set_member(...)
; This mode is guaranteed to run validation always, regardless of the value ofvalidate_assignment
. Also potentially provides additional functionality, such as setting defaults or formatting complex structures such as conditions, connections, etc.And finally, for those who just want to update Draftsman's data on the fly, there are now helper methods for all the class types in
draftsman.data
which allow you to add new or modify existing data:These methods are provided as a way to allow Draftsman to remain maximally strict against unknown data, but permit the user to quickly update said data just for the scope of a single script. This is provided mainly as a stopgap for cases where only a few entities/tiles are needed, which may be faster and/or simpler than grabbing the mod files themselves and running
draftsman-update
. In a future version, it might also make sense to havedraftsman-update
call these functions, meaning that this becomes the dedicated way to add/modify the Draftsman configuration dynamically (and thus only one place where it can break).These new features will allow users of 2.0 to have much more control of the manner in which their structures are validated, hopefully making the module much more useful for a variety of new tasks. Of course, there is tons of room for improvement even with these additions, but while I can't quite remove it from the TODO list I feel much more comfortable now kicking that can down the road.
Validation is now done with
pydantic
instead ofschema
All of the above magic is facilitated with the Pydantic validation library.
schema
was showing it's inflexibility, was lacking a number of features I was looking for, and was altogether not very fast, so I started looking for a good replacement. I went through a number of different libraries before I finally settled on Pydantic; the primary reasons were:entity.items = {...}
issue the same warnings asentity.set_items(...)
, but they both use the exact same code which is defined in one place, eliminating most of the bugs relating to validation being inconsistently performed.json_schema()
static method which can be used to dump this to a dictionary which can then be exported to other software. Furthermore, with a little bit of help it's highly likely that you could generate a human readable digest out of this information automatically, since the schema includes data type, value ranges, allowed values, etc. Because of this, almost all fields are given custom descriptions to not only make this feasible, but also highly likely in the future.Additionally, the backend of Pydantic is written in Rust, which theoretically might lead to a considerable performance improvement (which was another longstanding issue with Draftsman), though due to the increased complexity of the validation, benchmarks will have to made to get any concrete conclusions. See the performance section below for more info on that.
Because Pydantic uses type hints to express it's schemas, this means that the new minimum Python version required will be Python 3.7. This also allows Draftsman to use a number of modern Python goodies that were previously precluded from it due to it's backwards compatibility restrictions. I honestly doubt there was anyone using my library on a version of Python prior to 3.7 anyway, but in case there was, Draftsman 1.0 will retain it's Python 2.0 compatibility and remain available on PYPI and under the
1.0
branch on Github. 1.0 will still receive bugfixes for the forseeable future, but all new features and the majority of new development will likely only exist on the main (2.0) branch going forward.RailPlanner
,Schedule
,WaitConditions
, andTrainConfiguration
(finally)Another longstanding weak point of Draftsman was it's rudimentary API when interacting with rails, trains, and their schedules. In 2.0, this area has seen large improvements.
For placing rails, the long in-development
RailPlanner
class is now feature functional. It allows you to draw rail paths using turtle-like commands, entirely similar to how the game itself does it:RailPlanner
is a superclass ofGroup
, so all of the convenience methods available to it are provided as well, such as filtered searching, transformations, as well as positioning the entire set of rails all at once.TrainConfiguration
allows you to specify a sequence of rolling stock in the community-accepted syntax for describing trains:The syntax that Draftsman uses is a superset of the community version, which allows enough flexibility to specify any arbitrary train format. See
examples/train_configuration_usage.py
for much more info about that syntax and how it works.Once you have a train, you likely want to give it a schedule. Conditions are specified with
WaitCondition
andWaitConditions
objects:Then it can simply be added to a blueprint in 2 convenient ways:
schedule
can of course be omitted in case don't want the train to be set with a schedule. If you do specify a schedule, the schedule object will be added to the blueprint'sschedules
list, which can be inspected or modified in many different ways.If you find a particular locomotive in a blueprint that you want to give a particular schedule, you can use the new
set_train_schedule()
helper method:You can also use the newly added
find_trains_filtered()
function to grab a very particular set of trains in a blueprint/group:For more detail on how to use all the features of the new rail-oriented classes, see the examples folder.
draftsman-update
Command changed todraftsman
Commanddraftsman-update
has now been changed to be the more generic entry pointdraftsman
:draftsman list
lists all the detected mods at a givenGAME_PATH
andMODS_PATH
:draftsman enable/disable
quickly enables or disables specific mods:draftsman factorio-version
allows you to view the current version offactorio-data
, and even allows you to specify a specific version of it using git tags:And finally,
draftsman-update
has been renamed to the almost identicaldraftsman update
:Splitting functionality out into a more generic command makes much more sense from a user interface perspective and it makes it very easy to add new functionality and/or commands going forward. I already have some plans to implement recursive enable/disable, and or a method to fully configure mod settings from the CLI... but I'll add those features when somebody actually requests them.
Numerous Additional Quality of Life Features
2.0 also provides a number of features over 1.0 that should allow to write much more ergonomic code:
Direction
andOrientation
are now more than just normal enumerations, and support all of the methods supported with the Factorio Standard Library:The
Ticks
constant class has also been added, which should make translating time-based periods much easier:And a whole lot more. In the effort of getting this version out as soon as possible, there are a lot of TODOs in function descriptions. Expect those to clear up over the coming weeks, at least as soon as all the bugs are worked out.
Future work
pydantic
still has some limitations which make the whole process rather unweildly still in the backend, which I dislike. Furthermore, after discussing the issue with the maintainers it doesn't seem like the features I want will likely make it into the package anytime soon. There are also other things like valiation caching which I would like to implement, but are entirely outside the scope of all the validation libraries I've investigated so far. This implies that I'll have to in essence roll my own, which I am intentionally avoiding.data.raw
, and might not even necessarily want blueprint string manipulation. Fragmenting Draftsman into multiple components that can be integrated individually seems a smarter solution than the massive file glob I've currently developed.Instead of being tacked onto the README, The list of things TODO has now moved to a dedicated document, with more information about each component. If you're interested in contributing, it should now be easier to see my intentions for the project going forward.
Beta Was this translation helpful? Give feedback.
All reactions