Practical Python Modules

I grew up writing a lot of Python, which I guess speaks to how old and unorganized it is. And since then, I've been primarily a JS developer—that ecosystem has its failings, but the rules around how things are imported inside "node_modules" are mostly well established, even despite its quirks.

For modules, though—Python is a mess. But I think there's some simple knowledge that you might be missing. 🤔

So, You're Building A Package

Just some package that has some behavior or functionality. It does whatever. It might have some dependencies. But you want to structure it in a sensible way. 💡

You Want Module Mode!

Unless you're writing a single script file, I assert that you almost always want to run your code as a module.

# where "path/to/module.py" exists
python -m path.to.module

If "path/to/module" is a folder, Python will look for "__main__.py" within that folder. If it's a file, it'll run it directly. In both cases, __name__ == "__main__": you can still use that old trick to determine whether you're the startup script.

Benefits

The benefit of this is you get an actually sane relative import system. You can, and should only, write imports like this:

from .foo import whatever

…to import whatever from the peer file "foo.py".

You can also write things like:

# import "./foo_folder/something.py"
from .foo_folder import something

# import "./foo_folder/another_folder/blah.py"
from .foo_folder.another_folder import blah

# import "./hello.py"
from . import hello

# import "../up.py" (caveat: more on this later)
from .. import up

And best of all, if you have a file called "types.py", you don't have to literally worry about importing Python's built-in "types" module—you import it as .types.

Gotcha: Directory Layout / Working Directory

To run module code, it must be within a subdirectory of your current working directory.

Here's an example. This, unfortunately, works:

touch test1.py
python -m test1

But it quickly fails if we try to import something:

echo "from . import test1" > test2.py
python -m test2
# ImportError: attempted relative import with no known parent package

If you were to go up literally one directory, e.g.:

cd ../
python -m pythonsucks.test2

…this will import and run fine.

In a similar way, you can't escape the top-level package. I can't import ..foo if I'm within one of either "test1.py" or "test2.py", because there's only one directory above them in our module path.

This means that Python packages are gated to the current working directory of your Python interpreter. This restriction is absolutely bonkers (especially coming from a JS background), but once you know it, you'll appreciate what you can and cannot do.

Fun fact: You can run any Python file on your computer in a sane way by going to "/".
cd /
python -m Users.Sam.Desktop.pythonsucks.test2
This doesn't work if your folders have "-" or other weird characters in them, but it works.

Gotcha: Import Syntax Restrictions

You can't import a relative file like this—Python's syntax doesn't allow it:

import .hello  # <-- this does NOT work, so...

You instead have to do one of two options:

# For side-effects only - imports just a single symbol
from .hello import __name__ as _ 

# Import from self directory
from . import hello

The latter option works fairly well, and can be extended to e.g., the parent folder or subfolders:

from .. import peer_of_my_folder
from .subfolder import something_in_subfolder

Caveat: Can't Run File

In the examples above, even though the folder name is "pythonsucks", I'm not running Python by calling "python -m pythonsucks/test2.py". In fact, Python gives us an extremely unhelpful error message:

python -m pythonsucks/test2.py
# ImportError: attempted relative import with no known parent package

So whenever you run a file, you need to (a) replace "/" with "." and remove the ".py" suffix, and (b) make sure you're in a parent directory (as I said above, parent relative imports may fail unless you're above all of your Python code).

Notably this is an absolute pain in the ass when you're doing development work:

Your shell probably isn't going to autocomplete module names
You have to "cd" up to a parent folder whenever you want to test a file

It also means that you should effectively never put a ".py" file in the top-level of e.g., your git repository—it has no way of being executed in any sane way.

This is all a pain in the ass…but at least, now you know. Sigh. 😩

What About Requirements?

You might have some dependencies you can install. Python's dependency management systems are bonkers, too—that's a different post. (I have recently learned about pipenv, which is inspired by sane package managers—it may work for you.)

No matter how you install them, though, you should always import them with a name that does not include any leading dots.

Conversely, you should always import your own local code with a leading dot. (Sorry, I put a lot of emphasis in that sentence, but it's the rule.)

What about virtualenv?

I'm sorry. This post doesn't yet cover how this all interacts with that, but I suspect the answer is not much—the same module problems exist there, it's just that it's a way to include specific requirements/dependencies.

So you might be able to use the learnings here.

That's it.

Modern Python is nuts. I've spent hours pulling my hair out on this. 🫠

What I honestly want is a helper that lets me say:

python-module file.py

This helper would:

Walk my directory tree and find some "module root" marker (maybe this is the ".git" folder, or even "/")
Run "file.py" in the "path.to.file" syntax, treating it as a module

Phew. 😡

Hit me up on Twitter for any comments.