Of course you can deactivate our Page Builder specific Add-Ons (like that for Oxygen Builder for example) at any time you want.
Until Plugin Version 1.4.4 or Lower:
If you had the supported Page Builder of the Add-On set as your Default Page Builder, then you need to re-save the settings in Toolbar Extras and set it to another alternative.
This is necessary so that the system no longer thinks this specific Page Builder were your default. For this setting only a Builder that is active in your install makes perfect sense.
Since Plugin Version 1.4.5 or Higher:
Now it is handled automatically: if you disable the Page Builder meanwhile which was set as default in the plugin settings then the plugin will no longer put a notice or an error.