It's a game made using Unity3D engine, so if they really wanted to, Playtonic could release the project files used to create the game and let modders go nuts.
Unfortunately, this is basically an invitation for unsavory individuals to pawn off Yooka-Laylee as their own, so the dev team would probably code some modding tools to import assets and other things during runtime. Fortunately, there is a plug-in called "JSON.NET for Unity" that allows every "scene" (what you would call a level in Unity 3D) to be serialized in an easy to understand language called JSON. From there, a modder could insert their own files into the game's directory and edit the JSON files so that they can use these new models and textures.
Also, Playtonic's dev team could write a tool that allows editing of JSON files while the game is running, using this framework. As for mods that include new logic for original enemies or scripted events, I'm not too sure about that. Someone with more experience in Unity3D could probably answer that.