Adventures in crappy architecture, part N

Date Tags python (3 min read)

Remember when I said how great it was to make a IPython startup file? Remember those good ego feelings? Yeah, well, I went on vacation and came back and it broke. Goddammit!

Not only did it stop working, but it stopped working in a weird, dark magic way.

Friendly reminder

The point of the IPython startup file was that, every time I opened a new ipython shell or Jupyter notebook, it would automagically (and, in the notebooks, silently) import a bunch of packages I wanted (pandas, numpy, etc.) as well as a couple convenience functions (e.g. alert()). This would spare me having to copy that cell over every time. Hence, it was a minutes-saving, productivity-enhancing tool for near-daily use.

The problem is that it fails silently in the notebook (it fails noisily in the ipython shell). So, today, I happily opened a new notebook, double-checked a couple of my expected packages (pd returned the pandas object, we are go for launch), and proceeded.

And then - it broke. Specifically, my notebook complained that it had never heard of alert(). Huh, what now? It's in the startup file. How can you see import pandas but not def alert()? How does that make any sense?!

Fixing it, step 1: ipython > notebook

No error message means no clue, so I had to find the error message. The first place I looked was the Jupyter notebook terminal output:

shell

Uh, what? [IPKernelApp] WARNING | Unknown error in handling startup files:... and nothing?

This is where ipython shell is handier:

shell

Ooooh.

Note: That sqlalchemy import is not the actual error. I had a much weirder error in a package I had written. Namely...

Fixing it, step 2: circular imports because I no design software so good

I've had something of a dark obsession with design patterns in software for the past ~8 months. Mainly that I don't know any, but I know they're out there. My colleagues always talk about "code smells". WHAT IS A CODE SMELL? Things like that. I watch YouTube videos. This one by Sandi Metz was good - but I just don't write that much Actual Software to have enough practical experience to know this stuff. So I'm always hungry to learn more and get better at it.

One "pattern" (SCARE QUOTES) I've decided to adopt is putting global vars at the very tip-top of the American pyramid my Python package, i.e. in the __init__.py. Then I can import them in all sundry modules within the package. So I had something like this:

# __init__.py
from package import module1
from .top_module import Class1, Class2

# Global vars
STATISTICAL_SIGNIFICANCE = 0.95
DEFAULT_POWER = 0.9

But then, in my module1.py, I had this:

# module1.py
from . import STATISTICAL_SIGNIFICANCE, DEFAULT_POWER

Oh no. In other words, when I imported my package, the following would happen:

  1. package runs __init__.py to get everything started.
  2. First thing, it tries to import module1.
  3. In module1, it tries to import STATISTICAL_SIGNIFICANCE and DEFAULT_POWER
  4. Error! It doesn't know what those global vars are! Because they're defined further down the __init__.py.

So the __init__.py initialization fails, and everything fails, and - because I was importing this package about halfway down into my IPython startup file, it failed and only managed to import a few things for my Jupyter notebook. As you can guess, the alert() function definition was further down my startup file. It never got there.

So here's my fix!

# __init__.py

# Global vars
STATISTICAL_SIGNIFICANCE = 0.95
DEFAULT_POWER = 0.9

from package import module1
from .top_module import Class1, Class2

Voila! Easy! No more circular stuff. Global vars are defined first thing. Architecture! Design! SUCCESS!