Posts on Angela Ambroz

AI Engineering ⭐⭐⭐

Wed, 25 Feb 2026 00:00:00 +0000

Metadata
#

Author: Chip Huyen
Number of pages: 532
Year published: 2025
Year read: 2026

Review
#

Done! OK, that took a while. Overall, distilling it to a singular purpose - did I learn from it? - the answer is, “ya.” But did I learn a sufficiently satisfying amount from it? No, not at all. This was only a starter!

Quick overview
#

The problem with the book - as Jake said - is that it’s trying to cast a very wide net. From the intro, it seems the author, Chip Huyen, is trying to write an accessible “intro” to “AI” for software engineers, data scientists, and would-be founders. She assumes basically no special background in the technology, but - at the same time - is not aiming to write a proper undergraduate-level textbook on the subject (which is kinda what the subject requires!). As a result, it all feels very quick and surface-level.

So what’s covered? Lemme jog my memory and see how much I retained (lol):

“AI is cool and useful” (soooo many highlights and spicy marginalia here)
AI mostly means large language models (LLMs), but it can also incorporate computer vision, speech-to-text, text-to-speech, and so on.
Brief dive into LLMs and natural language processing: like, embeddings are cool, LLMs are big neural nets, the attention mechanism was a big step up, the transfer mechanism also. Even briefer mention of how all the cutting edge LLMs are basically closed-source, profit-driven moats that don’t share the neural net architecture, the weights, the training data, or the post-training steps. But I’m not bitter…
Long stretch about “evaluating AI models”, and how difficult that is. Why is it difficult? Because human knowledge is sooooo expensive and often inconveniently locked between the ears of actual humans. As a result, Huyen advises (and maybe the industry is coagulating around?) “AI judges” - aka, use an LLM to evaluate the answers of another LLM. Also, defining “better” is hard when you’re subjectively grading generated slop.
Some cool stuff about “post-training”, fine-tuning, RAG (retrieval augmented… something… generation?). Aka, building on top of the “foundational models” (aka, OpenAI, Anthropic, Gemini) by either, in order of difficulty, (a) prompt “engineering” (aka, ask it nicely, ask it a lot), (b) RAG (tell it to look things up), or (c) post-training (actually train the last few layers of the neural net again). Also some intriguing stuff about smushing models together (mixture of experts). No statistical concerns here about correlated training corpuses (corpi! yay Latin of meatspace) and orthogonality and, like, what we learned with ensemble methods in traditional, pre-LLM machine learning (e.g. a Random Forest vs. a bootstrap aggregated model with correlated trees)?
Data engineering issues - again, an alarming section about using LLMs to generate synthetic data so that we can make more LLMs. Insufficient statistical concern here!!
Full-stack, UX research-type discussion about how to integrate your “AI” into an app with actual, human users. Some very interesting, but very brief stuff about human-computer interaction, basically?

OK, so the good stuff: What did I learn?
#

This was definitely a helpful “overview” book: it gave me a lot of structure to the otherwise disparate ideas I have had about LLMs. For example, I found it very helpful to disambiguate the post-training step into specific workflows like prompt “engineering”, RLHF (reinforcement learning with human feedback), RAG, teacher-student models, and so on.

I liked the very brief history of the field, how the “Attention is All You Need” paper/attention mechanism is situated within that field (I thought it was THE thing that unlocked this strange alien intelligence, but it is apparently only A thing). I was VERY tantalized by the discussion of embeddings.

I found the evaluation section super useful to, again, structure this otherwise ambiguous space.

Oh yeah - and I love that this book has a repo with tons of additional resources.

The not-good stuff
#

I think I would have felt very differently about this book had I read it last year, pre-AI Con, pre-AI bubble. Living in this bog of cynicism now, I couldn’t help but roll my eyes at the relatively uncritical discussions around, e.g., the fact that all the cutting edge models are closed weight. That’s not science!

Relatedly, the “how to evaluate your model for your needs” section was frustratingly vague and unprincipled - and never called out the elephant in the room: If I can’t do any descriptive stats of the text corpuses in the training dataset, how can I, well, understand or trust anything? (I did like Huyen’s table on languages which are most disproportionately underrepresented in the main training corpuses out there: apparently Punjabi is like CRAZY underrepresented!) But the evaluation section felt like it boiled down to Vicki Boykis’s joke, that all AI evaluation is a “vibe check”.

Similarly, the discussion of using AI judges, for example, or using synthetic training data, without a discussion of the statistical implications of that, felt really unintuitive. For example: iirc Huyen mentions how training data biases will be amplified in synthetic data creation. That’s true - and it’s important to acknowledge. But it’s only one aspect of the wider problem: when your data is highly correlated (even duplicated), then how can you extract more information from it? For example, if you use Claude 4.5 as your LLM base model - and then use Claude 4.5 (or another Claude, for that matter) prompted to be your “AI judge” for the original model - isn’t that, by definition, uninformative? Like how bootstrap aggregating (bagging, the machine learning method) can lead to correlated trees?

Basically, I would have appreciated more foundational statistical thinking throughout. (And, in Huyen’s defense, this might be something to do with the field. For example: this paper, which is basically like, “Hey Anthropic/OpenAI, you should probably put error bars on your evals…” Basic stuff!!!)

One more thing
#

Having a gorgeous barn owl on the cover is apropos as well - owls are majestic, almost mystical creatures. Like LLMs, they’re hypnotic and affecting. But let’s plz always keep in mind: when plucked of their feathers, they do look very, very silly.

📚 Goodreads

Bedazzling the terminal

Sun, 01 Feb 2026 00:00:00 +0000

Eternal question: Where is my peaceful bottega?
#

If I had my way, I would be working in a little 15th century Florentine bottega, carving and chipping away at my craft in blessed peace and good vibes. Having saltless bread and pasta for lunch. Tiny espressos. Admiring the Duomo. Ample time for my family. It would be solarpunk and sane and full of diversity.

“The Cathedral as depicted in the Codex Rustici of 1447”

Unfortunately, I am living in a different historical context. A quote I’ve been thinking about a lot:

“Might does not make right! Right makes right!”

T.H. White, The Once and Future King

Today: Let’s pretend we’re in that bottega and do something just for the sheer pleasure of it. For the joy of tinkering and making things pretty and functional.

Moving from `gnome-terminal` to `kitty`
#

Tbh, I have very simple desires re: a terminal app. I just want it to:

Allow me to scroll back
Allow me to search the stdout history (ideally with ctrl+F, but I’ll take what I can get)
Allow me to copy+paste
And then do all the normal command-line things I need to do: ssh being principle among them.
Have nice colors and be a bit transparent.

This is surprisingly hard to find! So far I’ve worked with:

macOS’s Terminal app (well, years ago): Fine.
gnome-terminal: Fine.
alacritty: Fine, but characters weirdly got swallowed when I would ssh into my NAS. Annoying!

And now I’m trying kitty. kitty is by the same guy that built calibre, which is great software. It sells itself as being “GPU-accelerated”. I’ve never really noticed my terminal being so slow I need to use a GPU to make it go fast, but, well, okay. I basically needed something to play well with ssh + tmux + copypaste, and kitty has been: okay? I appreciate having an explicit config file.

Rapprochement with `tmux`
#

This was motivated by thinking, repeatedly, that it’d be nice to have Claude/LLMs read my terminal output. Some folks at work shared their workflow of using multiple panes in tmux with Claude Code - and telling CC to look at the other panes. This is about as close to the “chat window sidebar” experience that I think I can get. I hate using IDEs, they feel so bloated, and getting a chat window in Neovim has also been a pain. Furthermore, even if I did get an LLM chat in my Neovim, I’d have to figure out the workflow to also work with the command line/zsh. Using multiple tmux panes - the left one keeping it as Claude Code, the right one as “everything else” (Neovim, zsh, etc) - has been great.

I even, in an effort to Keep Up With the Times™, or at least make a good faith effort to explore the vibe code hype, made a Claude Code “command” (aka, incantation) called /pair-programming-setup that launches the tmux pane, launches my work authentication flow, and so on. That is a handy helper indeed.

Colors
#

I like pastels and neons, and picked the following:

catppuccin/tmux - “frappe” theme
dexpota/kitty-themes - “Grape” theme

Claude Code talking to itself
#

I guess everyone had the same idea this week: what if we made the agents talk to each other? Behold moltbook. Anyway, I’m planning to run a little experiment to see how quickly the LLM disentangles itself and degrades into gibberish. But, for now, I just amused myself by having the Claude Codes talk across tmux panes.

Year in review - 2025 - Books

Wed, 10 Dec 2025 00:00:00 +0000

Okay, this feels a little bit like I’m basically working when I’m not working, but, well, here’s my write-up on what I read in 2025, as digested by a little bit of exploratory data analysis.

2025: The year I read 82+ books
#

❤️ Here’s the full list on Goodreads 🌈

Yes, I put the + because I’m almost done with Bill McKibben’s Here Comes the Sun and I might have 1-2 more books in me before the year is out. But let’s analyze what I’ve completed so far.

☝️ A screenshot of Here Comes the Sun.

💹 Descriptive stats
#

📖 A meta aside
#

What I used to track books: Goodreads
What I used to analyze my reading: my self-hosted notebook server

I kept up a pretty steady pace. Reading was really the only thing I did this year in the evenings. I was usually too beat from work and domesticity to do anything else. I was also recovering from a major surgery, and so that kept me sedentary and fatigued.

KOReader has a calendar view, which is super fun:

Behold my gloriously cheesy word cloud - thank you, Claude-embedded-in-my-Marimo-notebook.

These are my book tags, and you can see that the vast majority of my reading was from the library (yay Libby App) onto my e-reader (beloved jailbroken Kobo). I made a conscious effort to read politics, tech, and climate. Those are, indeed, my main preoccupations these days: authoritarianism (Strongmen), the climate crisis (A Half-Built Garden), and Big Tech (The Internet Con, The AI Con).

The grihastha (Sanskrit/Devanagari: गृहस्थ) is my “householder” shelf - aka, any book that has to do with being a parent, managing a home, domesticity. I put kid’s books, cookbooks, and general “daily dharma” books in here. Example grihastha books in 2025: The Lion and the Mouse (wordless, watercolor picture book), Emmy in the Key of Code (middle grade book-in-prose about a girl + computers), The Home-Maker (phenomenal, newly-rediscovered classic about gender roles).

I’ve also tried keeping track of the historical setting of my books, since I enjoy reading about, e.g., the classical period. But I never seem to get around to it! That’s what that random 1600s-ce book is (apparently Wild Seed).

Here’s a histogram of the publication year of the books I read: huge recency bias, ha.

And a slightly cursed pie chart of predicted gender - using the Python package gender-guesser to predict gender using an author’s first name. That package is super old (2015!), and hasn’t really gotten the memo on gender diversity. But - meh - it looks like I did decent on reading a pretty even spread?

No more data, now only vibes
#

I have already written somewhere on the Internet that I love my e-reader. Oh, how I love my e-reader. People who say “I love the smell of books” can bite me. They don’t actually like reading, they like the aesthetics of reading! And, for what it’s worth, I like the smell of my little Kobo off-gassing its electronic juices.

This image fills me with joy. Someone interrupted me while I was taking this picture. Mortifying.

I read more fiction this year
#

I am a little bit allergic to “mainstream”, aka literary fiction. I mostly read non-fiction and science fiction. That has been my preference for nigh 20 years now (!). Anyway, this year I intentionally and unintentionally read a bit more “prestige”, lit fic: A Moveable Feast (and, as a chaser, some fanfic about it via The Paris Wife), The Vegetarian, Fellow Travellers. All were quite good. This fiction stuff, eh! Not bad!

I read some remarkable historical artifacts
#

Namely:

The Home-Maker by Dorothy Canfield Fisher - Originally published in 1924, then forgotten (?!), then re-published by Persephone Books (a British publisher that “reprints neglected fiction and non-fiction, mostly by women writers and mostly dating from the mid-twentieth century”). I think I found this via Kobo’s recommendation. It was phenomenal and was an anchoring text for this year. Can be paired with the equally phenomenal Counting for Nothing by Marilyn Waring, which expanded my consciousness.
The Oppermans by Lion Feuchtwanger - Originally published in 1933. A book about the Holocaust written before the Holocaust. Tremendously affecting. Can be paired with the equally brilliant Learning from the Germans by Susan Neiman (non-fiction political philosophy) and Swastika Night by Katherine Burdekin (sci-fi published in 1935 about the Holocaust, again, before the Holocaust).

All five of the above books really affected my thinking. They were those instant-classic-reads that I ended up citing all the damn time, to everyone. They were those types of books where I’m shocked I only read just now, since they feel so deeply embedded in my brain.

Anyway, some other stellar, brain-embedding books were:

BoyMom by Ruth Whippman - We need to put emotions back into heteronormative masculinity, for everyone’s sakes.
The Tyranny of Merit by Michael Sandel - Hey, crazy idea, what if we tried to get equality of outcomes, not just opportunity? 🤷
NeuroTribes by Steve Silberman - A deep, compassionate history of autism, and its two competing narratives (“we should fix it” vs. “we should make the world more welcoming of it”).
The AI Con by Emily Bender and Alex Hanna - The spiritual sequel to Weapons of Math Destruction.
A Short Stay in Hell by Steven Peck - Super-short philosophical fiction with the bleakest, blackest of humors.

Planning for 2026
#

What genres?
#

I usually have ambitions to read more widely. At the same time, I also keep making promises to myself to always have certain themes in rotation: dharma (aka, anything Buddhist or philosophical), fem (anything feminist or… uh, philosophical), tech (for fun), scifi (for more fun), climate (for 🌎).That’s a lot!

I tried reading some straight-up romance this year - after last year’s highly successful excursion into the genre via Zemindar (oh, Zemindar, my love). But I couldn’t do it: I couldn’t suspend my disbelief. Similarly, I still struggle with mysteries - it always feels less like a puzzle, more like an unfair riddle.

How to find books?
#

Right now, I mostly find new things to read via Goodreads’s “Readers also enjoyed” + recommendations out in the wild (blogs, newspapers, etc) + my LLM librarian. The Kobo store’s recommender system has also been punching above its weight; I think that’s how I discovered the very enjoyable near future cli-fi novellas by Premee Mohamed.

Onwards and upwards! (Wow, I just re-read my yearly review of books from 2018 and I concluded that one with “Onwards!” My brain really has only a handful of subroutines, sigh…)

Migrating from Wallabag to Readeck

Thu, 04 Dec 2025 00:00:00 +0000

Wow, 2025 really has been the Year of ~~Linux~~ Self-hosting! This stuff is the gift that keeps on giving.

The problem space: Reading it later in comfort and style
#

My latest project has been tinkering with my read-it-later flow. I realized I had several quirks in my workflow that, ideally, my tooling would reflect (and these are roughly ranked by importance):

I knew I always preferred reading on e-ink. So something that could sync to my Kobo was essential.
Similarly, I needed the ability to collect links from my preferred browser (Firefox) (highest priority), my phone (second priority), and my FreshRSS instance (last priority, kind of redundant).
Another must-have was archiving: I didn’t want link rot to eat away at my collection.
A “nice to have” would be the ability to highlight and sync highlights.
A really “nice to have” would be integrating it with my Anki workflow too.
A somewhat ambivalent “nice to have” was some LLM integration which would let me organize or interact with my collection using natural language. (Karakeep does this.)

I also realized I kind of had two distinct use cases. One was for my reading, whether that be for work (where e-ink and highlighting were crucial) or for pleasure (where just e-ink was important). Another use case was for collecting, vibing, mood boarding, you know, Pinteresting. I never really used Pinterest, and instead have historically just scattered my little treasure trove of Internet tschotschkes in various places: my Reddit saves, my hard drive Pictures/ folder, and so on. I did like the idea of gathering my visual media someplace.

My personal pain points on Wallabag
#

I already blogged about it seven months ago, but some pain points had been coming up with my flow. First: I broke the CSS and JavaScript on my self-hosted Wallabag web app instance, because I needed the PHP server to identify itself using both my local-network IP address and its externally available, VPN IP address. The environment variable which handles this, SYMFONY__ENV__DOMAIN_NAME, gets treated as a string in the later PHP code, rather than that later code just using a relative reference. So having an ugly, semi-functional Wallabag web app was already pretty sad (and severely limited my usage of it).

But I was like, fine. Whatever. The main use case, for me, is the e-ink.

And here’s the second pain point, which kinda annoyed me increasingly over time. The KOReader Wallabag plugin lets you sync to a Wallabag server. Fine, good. In theory. But! You’re very limited in how that syncing occurs. You can only specify the number of unread articles you’d like synced, and to which folder on your e-reader, and that’s it. What this meant was that I was often syncing 50 or 100 articles, just to see what I had of interest to read. This was certainly tedious. And it felt inefficient, especially as I was having a blast and much better user experience using OPDS for my Calibre-hosted collection. Seriously, OPDS, I love you.

Choosing Readeck
#

After plunging into research again - and seriously considering Karakeep again - I ended up landing on Readeck. The main reasoning was Readeck’s promises of a flexible syncing solution to e-ink (and how that was, relatively, more important to me than the LLM integration in Karakeep). So, on a slow weekend, I gave it a shot. The rest of this blog post is less a tutorial, and more documentation for my future self.

Not-a-tutorial: Getting Readeck set up
#

The brains: Setting up the Readeck server (20 mins)
#

This was the usual flow of picking one of my “hosts” (a NAS or a Raspberry Pi) and setting up a Docker container. To do this, I relied on the clear and brief docs and used a Docker compose file:

---
services:
 app:
 image: codeberg.org/readeck/readeck:latest
 container_name: readeck
 ports:
 - 8000:8000
 environment:
 # Defines the application log level. Can be error, warn, info, debug.
 READECK_LOG_LEVEL: info
 # The IP address on which Readeck listens.
 READECK_SERVER_HOST: "0.0.0.0"
 # The TCP port on which Readeck listens. Update container port above to match (right of colon).
 READECK_SERVER_PORT: 8000
 # Easier to read log format
 READECK_LOG_FORMAT: text
 # Optional, the URL prefix of Readeck.
 # READECK_SERVER_PREFIX: "/"
 volumes:
 - /path/to/docker/readeck:/readeck
 restart: unless-stopped
 healthcheck:
 test: ["CMD", "/bin/readeck", "healthcheck", "-config", "config.toml"]
 interval: 30s
 timeout: 2s
 retries: 3

Then it’s docker-compose up -d and, a few minutes later, navigating to http://whatever_the_host_is:8000 and making the first admin account. The browser extensions are official and maintained by the Readeck team (guy?) and they work fine. This part was fast and easy, maybe 20 minutes tops.

Client #1: My Android phone (1 hr)
#

Sending articles
#

Technically, there are a few options for getting an Android phone set up to talk to a Readeck server. The docs mention an Android app, Eckard, but - upon inspection - it seems very WIP and not available on F-Droid.

Next, I found some random app on GitHub that claimed to be an Android client for Readeck. I won’t link it, because it didn’t work for me. Maybe another WIP.

Finally, the docs recommend configuring HTTP Shortcuts, an Android app that lets you make HTTP requests from your home screen. Or something. Yes, it’s random. Anyway, I imported - and slightly modified - this template file. After test-sending some articles from around my phone, this worked fine.

Reading articles
#

~~This one’s still a WIP, and I’ll probably just use the website.~~ Yep - just using my phone’s Firefox, saving it as a bookmark on my homescreen. It feels snappy and fine. Nice to not have a middleman-app!

Client #2: My Kobo (20 mins)
#

Readeck really shined in how easy it was to set up the link between my Kobo (running KOReader) and Readeck’s OPDS server. You make an API key in Readeck’s web app, copy that to your Kobo (this was the most annoying bit), and then log into it from your “Add new OPDS connection” screen. For my future self, here were the steps:

The URL is: http://local_network_ip_address:8001/opds
You need to enter your Readeck username and the API token.
You can download the API token from the Readeck UI, as readeck-token.txt. I copied this text file over to the Kobo by opening up its SSH port and SCPing it over:

scp ~/Downloads/readeck-token.txt root@my_kobo_ip:/mnt/onboard

And, fyi, long-pressing the API key didn’t work (it wanted to look it up in my dictionaries). Instead, a more swipey highlighting gesture worked.

Migrating my Wallabag articles to Readeck (1 hour)
#

The Readeck web app technically has an “import from Wallabag” function. It wants to connect to Wallabag via its API. But, heavens, you ask me to figure out Docker networking? No no no. I will simply export.

And export I did! Deep in the recesses of my Wallabag instance, I found the 60MB-sized wallabag.sqlite database. There you are, sucker! I grabbed it, inspected it:

sqlite3 wallabag.sqlite
.schema wallabag_entry

CREATE TABLE wallabag_entry (id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, user_id INTEGER DEFAULT NULL, title CLOB DEFAULT NULL COLLATE BINARY, url CLOB DEFAULT NULL COLLATE BINARY, is_archived BOOLEAN NOT NULL, is_starred BOOLEAN NOT NULL, content CLOB DEFAULT NULL COLLATE BINARY, created_at DATETIME NOT NULL, updated_at DATETIME NOT NULL, mimetype CLOB DEFAULT NULL COLLATE BINARY, domain_name CLOB DEFAULT NULL COLLATE BINARY, preview_picture CLOB DEFAULT NULL COLLATE BINARY, uid VARCHAR(23) DEFAULT NULL COLLATE BINARY, http_status VARCHAR(3) DEFAULT NULL COLLATE BINARY, published_at DATETIME DEFAULT NULL, starred_at DATETIME DEFAULT NULL, origin_url CLOB DEFAULT NULL COLLATE BINARY, archived_at DATETIME DEFAULT NULL, given_url CLOB DEFAULT NULL COLLATE BINARY, reading_time INTEGER NOT NULL, published_by CLOB DEFAULT NULL COLLATE BINARY --(DC2Type:array)

And exported it to a CSV, with the gentlest sprinkling of data cleaning that was needed (very, very minimal):

sqlite3 wallabag.sqlite -csv -header \
 "SELECT url, title, CASE WHEN is_archived = 1 THEN 'archive' ELSE NULL END as state, null as created, null as labels FROM wallabag_entry;" \
 > wallabag_export.csv

Readeck chomped through this .csv with minimal fuss, though I am a little distressed that it could only match about ~60% of my articles. I’ll debug this… someday.

By this point, I was basically up and running! I spent a few days just getting used to its flows, and mostly enjoying it. Then came a couple very nice surprises!

Delightful surprises: Showcasing Readeck at its best
#

Surprise #1: Readeck > Save this image
#

Dude! It’s like a self-hosted mood board! For 10 years now, I’ve been occasionally adding gorgeous art and little delights to my Reddit saves. This was really not ideal, though, because of both Reddit link rot and, just, not actually “owning” a copy of that data. I was definitely feeling anxious about this. There were a lot of treasures there (r/museum and r/UtterlyUniquePhotos and r/ImaginaryLandscapes, my loves). But! With Readeck, it was easy peasy to save all these gorgeous pieces of art - with attribution and a link to what they are - into a lovely self-hosted gallery! WONDERFUL.

Surprise #2: Readeck for work research
#

Today, I had a major brainwave. I wanted to learn a bit more about sequential testing. My normal workflow here would be to, first, collect some things to read or watch: ideally, academic articles, industry blog posts, YouTube lectures, and documentation. I would normally dump all these links into my Obsidian, take my notes there and drop a link to what I read. With blog posts, especially, I’ve been annoyed that I can’t highlight. Well, today, I realized… I can.

So, new workflow:

Instead of dumping everything I found into an Obsidian “reading list” note, I saved them all to Readeck.
In Readeck, I created a collection for work-related readings. I loved this because I could group together articles and videos. I could also label them (manually… LLM would be nice here…) and get time estimates. For videos, I could also immediately get the transcript (of so-so quality). And it was all searchable! Glory be.
Finally, I could highlight these posts at my leisure - and view those highlights on their own, in their own highlights tab. (I can imagine using this for quickly generating Anki cards.)
I really loved the informative sidebar that Readeck has in its UI (bravissimi, devs) - telling me the authors, the original link, the posting date. But I just discovered the coup de grace: an outbound links list at the end! 👏🏻

This was really wonderful, and made accumulating knowledge for my, ahem, Knowledge Work, so much easier and more organized. Here’s some screenshots:

Uncanny Valley - ⭐⭐

Tue, 11 Nov 2025 00:00:00 +0000

Metadata
#

Author(s): Anna Wiener
Number of pages: 281
Year published: 2020
Year read: 2025

Review
#

Hm, an interesting memoir that I consumed like gossip or candy - but am unsure if I’d recommend?

This is a memoir of a very specific experience: being a relatively privileged, liberal arts/literary fiction-type white lady going to work in Silicon Valley in the 2010s. I am a white lady who went to work for tech in the 2010s (though, twist!, I hate literary fiction); I can confirm a lot of the cultural/anthropological reportage. The socio-economic and political points are specific to her perspective. And I totally get it.

But! A couple criticisms:

First, Wiener makes this interesting choice to anonymize EVERYTHING. It’s not Meta, it’s “the social network everybody hates”. It’s not Amazon, it’s “the former e-book seller megacorp” (or whatever, I can’t remember that one). In some cases, this is illuminating. Okay, in ONE case, it was illuminating: when she described Apple’s marketing of itself for its liberal/tolerant values, while its supply chain is an extractive, exploitative human rights nightmare (conflict minerals in the Congo; Foxconn, etc). THAT moment gave me pause.

But the rest of the time, the anonymization is just annoying. And weirdly gatekeeping? I kept asking myself: who is the audience for this?

For tech “insiders”, aka people who work in the industry or pay close attention to it, the anonymization is weird because it’s usually VERY clear who she’s talking about: GitHub, the Electronic Frontier Foundation, etc. For folks NOT in the industry, it’s just… not helpful? Like, I’m all for educating people who are not in tech about tech’s wicked, perverted ways (too numerous to mention). But we can only really do that if we name names! Being oblique just isn’t helpful. (And here is one person’s helpful public good: a “reverse engineered” dictionary of all the references.)

Another BIG missing piece that bothered me quite a lot is the total absence of the joy of, well, tech. Wiener is coming from a liberal arts perspective (and the points she makes about the different work cultures in the NYC publishing industry vs. SF startups were very on-point and authentic to my experiences coming from the non-profit world). She’s not particularly interested in learning how to code. She gives it a try one weekend, working through tutorials, and finds it pleasant but only gratifying to her “obsessive compulsive tendencies, [her] perfectionism”. She seems to imply that even the joy of coding is, in some way, corrupt or corrupting.

As a perfectionist with diagnosed OCD (!), this, well, hurt my feelings! And missed the mark COMPLETELY. (Correlation is not causation…)

Because coding is, for me, a creative craft, akin to making music! YES. And I know several musicians-turned-coders. Music and coding both create something out of nothing. They are tools for expression. They abide by some rules - often mathematical - but you can arrive at a “solution” in many different, creative, delightful ways. Reading code/music is harder, more onerous and less fun than writing it - than playing with it.

So imagine a history of the music industry that had zero appreciation for music? But only criticized the ways rapacious capitalism has poisoned and perverted it? That - in a way - endorsed the “tech exceptionalist” narrative that tech is somehow “beyond [the] good and evil” of capitalism. No! Tech is not exceptional. And a history of music without any understanding of why some people enjoy making (or listening to) music is a very woeful perspective indeed. And its narrowness would be immediately apparent! But for whatever reason, coding’s bullshit public relations has ruined it in the cultural imagination to such a point that seeing accurate portrayals of the Joy of Coding, of the hacker spirit, are few and far between. (I have GREAT hopes for Ellen Ullman’s memoirs.) Seriously, the only ones I can think of are Merlin in Disney’s The Sword in the Stone (he is a true tinkerer, people), Jaron Lanier’s memoir, and, of course, almost anything by my beloved Cory Doctorow.

📚 Goodreads

LLM book recommender system

Tue, 28 Oct 2025 00:00:00 +0000

I now have several different apps to organize my self-hosted media: Plex for movies and TV, Audiobookshelf for audiobooks, and Calibre-web-automated for ebooks.

🎥 Discovery on Plex: really good
#

The UX on these apps is uniformly pretty good, with a few being great. Plex, for example, is a joy: the metadata is super well-organized, and so you can easily traverse your movie collection to find, e.g., all the Colin Firth movies you do and do not own (narrator: she owns enough). Similarly, it enriches your existing (probably paltry) metadata with, for example, IMDb and Rotten Tomato ratings, category, country of origin, etc. This is all super helpful for discovery which - when you have a big media library - is key. You can use either deterministic rules (“sort all my unwatched Italian-language movies by audience rating”). And it includes, on every movie page, some “more like this” suggestions - probably powered, under the hood, by a reasonable recommender system (“recsys”).

🎧 Discovery on Audiobookshelf: also quite good
#

Audiobookshelf is probably next in terms of UX and discoverability: like Plex, it does a great job of organizing your media library’s metadata. It does not have a built-in machine learning-powered recommender system, alas, but it does have very good metadata matching. Each audiobook has a ton of rich tags and categories and so you can, at least, manually look things up by categorical vibe.

📚 Discovery on Calibre-web-automated: not so good
#

Which brings us to Calibre. Oh, Calibre. Salty, salt-o-the-earth Calibre. Calibre is dinosaur software, it’s been around forever and has a huge community. It has lots of quirks, both in its community, its development, and its user interface/UX. But it is, as they say, very powerful. Several attempts have been made to essentially re-skin Calibre into a slightly more modern-looking web interface: Calibre-web being the main one I know about. A fork of calibre-web is Calibre-Web-Automated which… well, to be honest, I’ve read the README.md tons of times and I still don’t see why I’m using it versus just calibre-web. But no matter. It has 3.5k stars on GitHub. They can’t all be wrong?!

Anyway, one thing that Calibre(-web)(-automated) all lack is any sense of discovery. This is mostly because, weirdly (for such well-used and well-loved software!), the way they handle book metadata is kinda garbage? At least, when you add an ebook to your library, the flow is not to immediately give you a best-match metadata chunk (as in Plex or Audiobookshelf). There’s tons of UX love this app (these apps) need (e.g. I can’t click on multiple books unless I first switch to “Book list” mode? whyyyy?). But the big one that I need (NEED) is decent discovery.

Using an LLM to do discovery on my own Calibre library
#

I just finished The AI Con (5 stars, v good, highly recommend), and so I do hesitate to touch LLMs these days - especially due to the water usage. But, well, okay, this one’s pretty cool. Basically, in plain English:

Get my Calibre metadata.db, the sqlite3 database that stores all my book data (title, author, read/not read, description, tags…), from my Calibre container.
Add above .db to my self-hosted Marimo container.
Launch a cool, new notebook called librarian.py, hehehehe.
Connect that notebook to the Anthropic API.
pd.read_sql("SELECT * FROM books", con=conn) from your metadata.db and do some light cleaning (well, hopefully). I realized early on that I was just blowing money by feeding way too much textual data about my ebook library. For now, I just send the title-author pairs of all unread books to Claude.
Ask Claude to be my librarian!

Here’s my very exciting prompt and Anthropic API call:

def get_book_recs(mood: str, books_df: pd.DataFrame) -> str:
 if not mood:
 print("No mood!")
 return pd.DataFrame()

 # create a context string from your books
 book_context = books_df.to_dict(orient="records")

 prompt = f"""
 You are a world-class librarian. You prioritize books from under-represented
 groups (women, POC).

 Here are the books in my library (id, title, author):
 {book_context}

 I'm in the mood for: {mood}

 Please recommend books from this list that most match my mood. If you can't
 find any in my library, please point me to the most relevant books that I
 can look for externally.

 Please use Markdown formatting. Format every recommended book
 with <a href="http://my_super_cool_calibre_ip/book/id>title</a>.
 """

 # hit your llm api here
 message = client.messages.create(
 model="claude-sonnet-4-5-20250929",
 max_tokens=1024,
 messages=[{"role": "user", "content": prompt}],
 )


 # parse response and filter df
 return message.content

TEST_PROMPT = "something solarpunk and light, <200 pages"

# Your API call here
recommendations = get_book_recs(mood_input.value, df)

Results
#

So how does this perform? Honestly, pretty good!

I’m basically pulling from Anthropic’s training data for any scraped data on my title-authors. This works great for relatively well-known (or, at least, well-represented-in-the-corpus) stuff. It struggles, of course, on anything very new or anything obscure. So the recommendations are always going to skew relatively well-known. But that’s okay! This has been a great discovery tool. I have about ~1k ebooks in my library - a mix of technical books (so many Humble Bundles, mamma mia), sci-fi, literary fiction, politics, history, economics. Sometimes I’ll be in the mood for a specific theme or type of book. Being able to “speak” to my library in natural language is great! And fun!

Example run
#

Here’s an example run I did just now.

Can you even see that? Sorry it’s so small. My request was, “what’s some good fiction to read about working in the tech industry?”

The full response is pretty good:

From Your Library:
Close to the Machine

by Ellen Ullman This is a classic memoir/fiction hybrid about a software engineer in San Francisco during the first dot-com boom. Ullman brilliantly captures the isolation, obsession, and strange intimacy of coding work. It’s considered one of the best books ever written about what it actually feels like to work as a programmer.

Uncanny Valley

by Anna Wiener A memoir about working at various Silicon Valley startups during the 2010s tech boom. Wiener chronicles her transition from the literary world to tech with sharp, insightful observations about startup culture, gender dynamics, and the industry’s impact on society.

Several People Are Typing

by Calvin Kasulke A clever, experimental novel told entirely through Slack messages about a worker whose consciousness gets stuck in Slack. It’s a funny, surprisingly moving meditation on remote work culture and digital existence.

Moonbound

. by Robin Sloan While more fantastical, this book features tech industry elements and Sloan (who also wrote Mr. Penumbra’s 24-Hour Bookstore) has a background in tech and writes beautifully about digital culture.

Other Books to Look For Externally:

Microserfs by Douglas Coupland - A novel about Microsoft employees in the 90s

The Circle by Dave Eggers - A dystopian take on a Google/Facebook-like company

The Every by Dave Eggers - Sequel to The Circle

Severance by Ling Ma - Combines office work dystopia with a pandemic narrative

Qualityland by Marc-Uwe Kling - Satirical sci-fi about tech monopolies

Averting the Digital Dark Age - How Archivists, Librarians, and Technologists Built the Web a Memory - ⭐⭐⭐⭐

Sun, 14 Sep 2025 00:00:00 +0000

Metadata
#

Author(s): Ian Milligan
Number of pages: 208
Year published: 2024
Year read: 2025

Review
#

Note: Available for free on the publisher’s website. I have no idea how I found this book (probably my self-hosted RSS feed or Mastodon or other nerd worlds) but I’m so glad I did.

Pretty fabulous. This reads a bit like a very well-written PhD dissertation, or academic textbook. It’s a pretty niche topic - memory work (archiving) in the context of the World Wide Web - but it scratched a very specific itch I was having about my own data, scattered as it is among Big Tech’s platforms. The author actually notes this in the book’s conclusion: that the “digital dark age” remains a bogeyman for many in tech, even though the structures are now in place that it’s basically impossible (barring a “nondigital dark age” happening as well). The real risks, instead, are individual “dark ages”: e.g. Google Photos basically becoming your memory landlord that you rent the right to your pictures from. Horror of horrors!!!

Anyway, this was a wonderfully rich history of the early web - we spend a lot of time in the 90s and early 00s. I love social histories of technology. SO MUCH. There are chapters on the Internet Archive and the various archival strategies used by different national libraries (Canada, Sweden, Australia, the US). Libraries are SO COOL. I’m excited to read more about this work. Anyway, the Internet Archive and these libraries are the spine of digital archiving - I learned a lot about Brewster Kahle and Heritrix and just, like, internet stuff. Early web stuff is so interesting. And a good reminder that we are still grappling with all the usual dramas, WHILE ALSO experiencing interesting changes (web 2.0! and now the totally justifiable and I am 100% on-board backlash against Big Tech). Who knew networking computers could be so interesting and such drama.

Very similar to Finn Brunton’s Spam, another terrific book about internet stuff.

📚 Goodreads

The AI Con - ⭐⭐⭐⭐⭐

Sun, 14 Sep 2025 00:00:00 +0000

Metadata
#

Author(s): Emily Bender, Alex Hanna
Number of pages: 288
Year published: 2025
Year read: 2025

Review
#

As Weapons of Math Destruction was to the 2010s and basic machine learning, so this book is to the 2020s and “AI”.

So: I loved this book. It set my head straight on a lot of things I intuitively knew, as someone with enough expertise to understand how the “AI” sausages are being made, but was struggling to articulate. And hype is strong! We are social apes! Like - yes - natural language processing (NLP) is amazing and gratifying and weird. (Text is such gnarly data!) But NLP has been eerily amazing for a long while now (I have felt the icy showers for at least 10 years! tf-idf?! whaaat!?), and these stochastic parrots are indeed eerily amazing as well. But! Do they deserve all this hype? Where by “hype”, the authors specifically mean: enormous venture capital investments, enormous carbon emissions and re-jiggering of our energy infrastructure, and - perhaps worst of all? - figures of authority waving their arms around a vaguely-defined but definitely civilization-altering “Artificial General Intelligence (AGI)” that is always just beyond the horizon?

The authors argue - aggressively, spicily, wonderfully - that NO! This is all bananas! And I am 100% here for it.

First, they give one of the best plain English primers on neural networks, n-grams, and embeddings. It’s one chapter, and it’s only semi-technical (intended for non-technical audiences), but it covers, imo, the main ideas in a very clear, comprehensive way. So bravi, there! They also offer refreshing clarity on defining “AI” - a term that is, currently, being abused in everyday conversation, but that normally captures distinct fields in machine learning/comp sci: large language models (LLMs), OCR, computer vision, blah blah, I am tired of linking.

Rather than prognosticating about the future (and, indeed, notice how much AI hype is about the very near future… it’s just over the horizon, people!), they instead trace the history of AI (leveling some shots at Minsky and Hinton, wowza), the history of Luddites, and the CURRENT practices of how LLMs are trained, how they are used RIGHT NOW, and how they are talked about. There is a lot about labor (outsourced content moderation is horrible indeed; your boss being sold AI to “boost productivity” == aka, layoffs) and training data bias (duh) and basically plugging the holes in our social safety net with word-prediction machines. All of this was stuff I knew, but they structured it in a clear and helpful way.

The one thing I did NOT know, but blew my mind, was the theory of mind stuff and linguistics (Emily Bender is a linguistics prof at U of Washington). Basically, language includes a lot of “guessing what the other person is thinking/trying to say”. That’s why you can’t teach your baby Italian via TV (believe me, I’ve tried). It’s the interaction that matters. The social learning. Because LLMs are so good at sounding human, our brains naturally start to “fill in the blanks” about what they’re “thinking/trying to say”. This is also why people DO NOT ascribe cognition to “AI artists” when they look at those (frankly very tacky) DALL-E, Midjourney, genAI art outputs. No one thinks an “AI” was trying to “express its consciousness” - we see it as an obviously computer-generated, automated mish-mash of training inputs. But LANGUAGE. Our ape brains get real weird there. Hence all the flailing around “omg AGIIIII”.

Anyway, I loved this so much. Should be required reading for everyone in tech.

📚 Goodreads

Hosting my own notebook server

Tue, 26 Aug 2025 00:00:00 +0000

Like many a data scientist, I’ve spent tons of time in Jupyter notebooks. For many years (2015-2022), I mostly just used vanilla Jupyter - or sometimes Google Colab. In 2023, we started using Hex at work, which was super slick and fun: LLM-powered cells! Dark mode! Cells that remember each other’s state!

I’ve been, indeed, very happy with Hex for these last couple years. But, in my homelab journey, I realized having a notebook environment to explore my data - my transaction history from Mint, my purchase history from Amazon, my job hunt data - would be really cool. This became even cooler as I stood up services with database backends - my audiobooks! my ebooks! - the possibilities for data analytics on my own data were endless!

At first, I just set up the basic data science Jupyter notebook via Docker Hub. I used it for a few days but it felt very, well, 2015. I was like, come on, there must be something slicker out there!

Enter marimo!
#

I did a bit of research (Kagi-ing, LLMing), and landed on marimo. It was tons of good things: open-source, self-hostable, with flexible AI integration - including using a self-hosted LLM backend. Also: dark mode! Turning things into apps! Notebooks getting automatically turned into version-controllable Python .py files! Per-notebook environment management with uv!

That’s it! Glorious.

Homelab overview

Fri, 01 Aug 2025 00:00:00 +0000

Updated: 5 Dec 2025

thing	service	replaces	phone?	ereader?	rating	thoughts
Movies, TV	Plex	Netflix, all the various streaming services	✅	n/a	⭐⭐⭐⭐
Ebooks	Calibre-Web-Automated	Amazon	✅	✅	⭐⭐⭐
Audiobooks	Audiobookshelf	Amazon	✅	n/a	⭐⭐⭐⭐
News/blogs/social media	FreshRSS	Twitter, Facebook, NYT	✅	haven’t tried yet, but theoretically yes?	⭐⭐⭐⭐	Moving everything to RSS
Bookmarks/read it later	~~Wallabag~~ Readeck	Pocket	✅	✅	⭐⭐⭐	~~Reading things later with Wallabag and my Kobo~~ Migrating from Wallabag to Readeck
Syncing	Syncthing	Dropbox, Google Drive	✅	❌	⭐⭐⭐
Notes	Obsidian	Evernote	✅	n/a	⭐⭐⭐	On being an Obsidian power user
Document digitization	Paperless-ngx	Dropbox	✅	n/a	⭐⭐⭐⭐
Photos	Immich	Google Photos	✅	n/a	⭐⭐⭐⭐	Imprisoning your users - an experience with Google Takeout

Those are all the services I have so far, and they are making my poor little Synology NAS choke and wheeze. Sorry, buddy. Yet while my Synology suffers, I am filled with joy!

I still have a few pieces I’d like to put together, to really bring this “self-hosted Internet” thing home (nyuk nyuk nyuk):

Cooking? NYT Cooking is still my go-to for excellent recipes (and comment banter), and Eat Your Books is awesome for indexing my existing, physical cookbooks. I’ve seen some self-hostable FOSS options like Mealie, Tandoor, maybe kinda Grocy? None of them are really inspiring me yet - and Tandoor made my Synology really struggle - but we’ll see.
Passwords? I’m comforted by the idea of locally hosting my password manager - it did always feel a bit weird to have my One Password to Rule Them All chilling in the cloud somewhere (encrypted or not idk).
Git? Another sort of interesting one. I hadn’t considered self-hosting a GitHub replacement. As Working in Public discussed, a lot of the open source ecosystem is platformed by GitHub - and kind of incentivized/promoted/facilitated by GitHub and its “social media”-esque features. I do have some private git repos - e.g. my family holiday card - that I use GitHub to version control and… well, I don’t like doing that. I would prefer something more private for those.
LLMs! This is the big one. I use Claude so much these days and would LOVE a truly private, local, self-hosted LLM option. It’s only a matter of time before Claude/ChatGPT/etc start promoting products, and I want to get out and have my own local solution before then. I’ve played around with Open Web UI and Ollama - just on my wheezing, middle-aged Lemur Pro and just as a proof of concept. It worked! Huzzah. But there’s no way my Synology could handle the compute that even a small model would need. So I’m daydreaming about upgrading my hardware to a GPU and really getting things going… Update, Aug 2025: I’m now halfway there with a local pass-through of cloud LLM providers.
Video calls? FOSS Academic had a cool post about using only, well, FOSS tools - including Nextcloud for video conferencing. I also want to support digital sovereingty/European tech for political reasons - and Nextcloud is French? German? Frerman. I also kinda wonder if I can get my non-tech fam onto Nextcloud? I’ve discovered one of them is paying for Dropbox again. 😬
Minecraft? My kids are actually still too small for Minecraft, but, well, it’s an option. Any other games? I would love to self-host some of my nostalgia games of yore like Earthworm Jim or The Sims.

I also have been thinking about switching out some of these services - e.g. Karakeep for Wallabag? Jellyfin for Plex? 🤷 For now, I’m (very) happy.

Some “economics of open source” thoughts
#

Working in Public was a starter book on economics in open source - but it really just scratched the surface, tbh. I was trained as an economist - it was my first love! 💙 - and I get easily nerd sniped by odd socio-economic arrangements. And open source is odd! It is weird and wonderful! The idea that people bring their technical skills to voluntarily work on hobby projects collaboratively… For no reason except passion???… And that those projects end up, uh, supporting the entire Internet infrastructure (I’m thinking of Linux and git and Python and pandas…). Last I checked, the market cap for “tech” is ~20 trillion USD. And - again - the absolute core foundation is… based on free labor? Driven by passion?

What’s even more exciting (and anarchic…) about open source is that it can only be successful if it’s collaborative. One of the big themes of PyCon this year (beyond politics!) was actually maintainer burnout - see Lynn Root’s keynote about this. Burnout permeated tons of discussions and talks. And, indeed, Working in Public highlighted how many (many many…) open source projects often rely on a single (!) maintainer - doing frankly thankless grunt work to keep it going.

I remember running into that when I made a PR, long ago (maybe for Hacktober?), on some Python timezone stuff. It may have been pytz. Something that was used… everywhere?? I think I was fixing something in black? Or pandas? Anyway, it was like a foundational library - the management of TIME - and it’s basically one guy who’s been maintaining it since forever.

ALL THAT TO SAY… these days, I think a lot about digital sovereignty and self-hosting and “well, we don’t have a lord” and community building and human cooperation as the key to our survival. And I’m like, shit, I need to probably actively maintain these services I’ve just installed from Docker Hub? Like, Audiobookshelf is fantastic - but is it gonna be around in 1 year? 5 years? Do I need to help it? Will I eventually need to fork it and run my own? Should I fork it now and start specializing it to my very specific, tiny community’s needs? For now I am WAY too busy with paid tech job work and unpaid “women’s work” (aka, the house and kids, aka maybe a big reason women’s contributions to FOSS are very rare indeed).

Edited to add: Oh, funny, a very relevant article on these issues just popped up in Shareable.

Imprisoning your users - an experience with Google Takeout

Mon, 28 Jul 2025 00:00:00 +0000

For about 30 years, my grandmother used the same hole-in-the-wall shop to get her photos developed. She loved taking pictures of her paintings, sunsets, flowers, her family, and trips. Every few weeks, she would drop off a roll of film. The shop would do that darkroom chemistry stuff and deliver back to her a couple dozen photos, usually in a little, branded album. We have many of those albums lying around the house.

Fast forward to 2025. I’m very similar to my grandmother: I like to take pictures of flowers, my art, my knitted objects, my family and my trips. I put them in my photo shop. My shop doesn’t give me the photos - there’s far too many of them for me to keep at home, they assure me, but if I pay them a reasonable annual fee, they’ll hold them for me indefinitely. Additionally, I can pop into the shop at any time to look at them. The shop also sometimes sends me cute assortments of my photos, with cheesy background music, especially around the holidays. I do really enjoy this, except when they sometimes callously show me a rosey birthday video of a dead loved one. I need to remind them to please skip those photos from the sentimental collages.

Behind the scenes, the shop uses my photos for some very cool, very advanced computer science projects. I notice that my photos start to get automatically labeled - they can tell it’s a picture of me from the 1990s, wow. I didn’t realize my photos were grist for this machine learning mill; it was probably buried in a “consent” form that didn’t actually gather consent in the typical meaning of the word. Anyway, I definitely learn about this after the fact. And I’m not thrilled about my family pics being used for science projects, but I guess the shop is good about privacy? Right??? Plus, it is amazing to be able to ask the robot shopkeeper for all pictures featuring one of my kids and a guitar. Wow, this robot is really smart! I do feel a bit uneasy that I don’t actually, physically have a copy of any of my family’s photos from the past 10 years but well hmm…

Meanwhile, I read a scary article about a time the shop locked its doors to a customer because they misunderstood one of the customer’s photos as being pornographic (it was medical). The customer was kicked out - and he wasn’t allowed to have any of his photos. My unease increases. Years pass.

One day, I decide I’d like to take all my photos out of the shop. Would this be possible, please? I’m tired of paying the shop the annual fee to house all my photos. I’m worried about having all my precious memory-eggs in one basket. I have a surgery coming up and need to take gnarly medical pictures and I don’t want to get banned! So I’d like to try another shop, please. This shop - very high-tech, maybe the highest-tech shop in the whole world - this shop that can make computers recognize my baby pics as me, using only pictures of adult-me - has only ONE way to deliver photos. No, sorry, you can’t access the photos programmatically using an API - despite the abundance of APIs in the backroom.

Here’s how I’m allowed to get my photos:

I have to enter a different shop and tell them I want the photo place’s photos. “Ugh, fine. We’ll let you know when it’s ready at some indeterminate time in the future.”
I wait.
Finally, my photos are ready to be picked up. The shop warns me: “if you don’t pick these up this week, we’re going to need a fresh request!” I am deeply annoyed at this Kafka-esque runaround.
I go back to the different shop, Takeout, to pick up my photos. My photos are bundled together in 50GB .zip files that I need to, one by one, laboriously remove from the shop and heave into my tiny cart, before wheeling it to my slightly larger pickup truck.
I have 7 of these bundles. It takes about one hour to get one of these .zip bundles from the shop into my pickup truck. I’m something of a pickup truck mechanic/hobbyist, so I know how to do all this. But most of my normie, non-mechanic friends don’t really have anywhere to load their giant bundles; they just pay the fee and let the shops keep their photos and hope for the best.
Also, by the way, every time I enter the shop - I need to show them my ID. Not my usual ID (a chocolate chip cookie they stuffed into my backpack on the way out), no. They want to see a key, as well as my second, plan B, just in case key. Man, they are surly, these types!
Once I get all my stacked cartons of photos back at my house, I start to look through them. The organization is wild! Photo labels are spread out across different boxes, photos that I had meticulously organized into albums when I visited the shop have been re-shuffled in a seemingly random order. This is 10 years of my life, in photographs, and the metadata is just fucked. I am exhausted by the sight of it. It feels vindictive. Frickin’ shop!!!

Yeah. So that was my experience with getting 300GB+ of photos out of Google Photos. After several aborted attempts, over several years (!).

It’s amazing - galling, really! - how we’ve let this Big Tech monopoly get out of hand. And how completely turned around the notions of “user”, “customer”, and “company” have become. Why am I renting my own memories back from a cloud landlord? Like a digital serf? Why are my photos being held hostage for shareholder value? Like, what are we even valuing? The photo of my grandmother is priceless to me - but it’s just one more data point to Google. The way we’ve structured tech into our lives is really crazy-making. Imagine what we could do if we didn’t prioritize the corporations’ interests, but our own?

Anyway… Yeah, I’m not into it anymore. My data is mine! ✊ Once I get these sorted (thank you, open source) - programmatically! sanely! - and running on Immich or Synology Photos or just, like, my hard drive and an offsite backup, I will be VERY excited to delete them from Google’s servers. Wish me luck on the sorting-out part - I embark on that with dread.

Getting ready for local AI

Wed, 16 Jul 2025 00:00:00 +0000

I mentioned in my homelab post that one of my STILL-TO-DOs was self-hosting LLMs. Unfortunately, we run into immediate hardware constraints on that one - I don’t have a single GPU lying around! Sad.

I was ready to put this dream to rest, but a video by YouTuber NetworkChuck was super interesting and inspiring. Basically, the idea is:

In other words, I’m currently using my NAS as a pass-through for any/all LLM work. I’m using OpenWebUI (interface), Ollama (local LLMs), and LiteLLM (proxy/API connector). I’m connecting to Anthropic and OpenAI 🤮 using their APIs, and I have - for now - one local LLM installed on the NAS (Mistral 7B). In some future world, I’ll spend the big bucks to get a GPU machine - maybe… dare I dream… a bedazzled Framework desktop??

Anyway. Why do this now? Pre-GPU?

The basic idea I’ve been having is, I want my cyber<>meatspace interactions to be represented like this:

That is:

My meatspace self (and my friends, family, strangers, employers, etc)
…Interact with some UI…
…To see my data/digital self… which I still own and have control over

If a UI breaks, changes, enshittifies, etc, I can just switch it out. This speaks to Cory Doctorow’s interopability dreams. And yes, it’s all about tearing down the walled gardens’, uh, walls.

I spent most of web 2.0 (2005-2015?) as a digital serf - tilling the big platforms’ fields (Facebook, Twitter, etc) with data that was precious to me. My memories! Pics of college and grad school! Old friends! Parties! Trips! Extracting that data out of the platforms has always been onerous, but I’ve been doing it for a few years now. De-Googling, as you can imagine, has been taking forever. But I have de-gmailed! And that got me thinking. If I move from ProtonMail to FastMail or wherever, I’ll have to keep telling my contacts that my e-mail address changed. What if I can keep that fixed - and just swap out the back-end? HMMMM.

So that’s the idea with the self-hosted UI for the cloud LLMs. I’m retaining a streaming, constant copy of my data locally, on my NAS. (Otherwise I’m just tilling OpenAI and Anthropic’s fields, and have to manually export my data whenever I remember.) This way, yes, they still get my data (“Khaaaaaaaan!” as Kirk would say), but at least I have it too. And I can delete it from their servers whenever I want (presumably…). And once I’m ready to buy a GPU and stand up a local, semi-powered, truly private AI, I can cut the chord completely.

Fun OpenWebUI features
#

OpenWebUI is super powerful; docs here. My use cases for LLMs are fairly basic:

Chatting
Some coding
Sometimes some more involved projects (OCR + translation + summarization)

I don’t really enjoy AI slop art, and haven’t yet gotten into text-to-speech (though I have been thinking about Whisper). I have zero interest in nightmarish video gen.

For chatting, I had a glorious prompt on the Anthropic UI for a “techbro neckbeard”. He was the “persona” I spoke with the most. Basically, I told him to be a typical Redditor, all lowercase, very snide, judgmental, etc. The “WELL ACKshually” guy. I ported that persona over to OpenWebUI, added a profile pic (you can add profile pics for your custom models!!), inserted the same Anthropic Claude-4-sonnet brain, and, voila, now I have my asshole AI assistant back!! Man, this guy makes me laugh and laugh. Also, super interestingly, while both the Claude UI and OpenWebUI neckbeard personas always assumed I was a dude (sigh…), the OpenWebUI techbro was also way more overtly sexist and rude. Wow!! Are there more guardrails on the Claude UI chat vs. the Claude API? I wonder… Anyway, I had to adjust his prompt to be a feminist techbro, jeez.

I have a few more AI personas I enjoy - man, I’m really living the Dan Simmons’s Hyperion dream here. I have a solarpunk lady and a fitness guru lady. I used to have a Thomas Bayes econometrics/stats friend - he would speak in 19th century English minister talk, very amusing. I have had dark sci-fi thoughts about making personas for my loved ones that have died 😱. But no! Must not literally recreate that one Black Mirror episode. 😬

Reflecting on a year of attempted sustainability

Thu, 10 Jul 2025 00:00:00 +0000

In July 2024, I read Kim Stanley Robinson’s The Ministry for the Future and it had a profound effect on me. It appealed to lots of long-time personal and professional passions: rationality, sustainability, technocratic solutions to social failings, saving the world. It triggered one of my hyperfocus periods - where I become all-consumed by a topic and obsess over it and adjust my life around it. These are always very educational periods.

It is now exactly one year later from that Goodreads review - and I spent a lot of time trying to apply the book’s green fire to my daily life. Let’s do a retro and see how I did, and what I’ll change or keep going forward.

What I was trying to do
#

Climate nihilism
#

For context: as Alan Tudyk’s alien would say, I am a basic bitch educated liberal. Pre-the book, I most certainly believed that human-caused climate change was real, but I was also suffering from spoon-fed NYT climate anxiety and climate nihilism (“it’s too late”, or “individual action is pointless”). I was aware of “greenwashing” and knew that I should be mindful of it, I liked the bag tax at the grocery store, but - good Lord - do not give me another tote, please!

I didn’t look too deeply into climate science or the state of climate change work in 2024 because it just made me depressed and anxious. Also I felt overwhelmed. I didn’t know where to start. I didn’t want to think about it too much, though I definitely thought that someone should think about it (please?). Indeed, it took me months to get the courage to read KSR’s book!

All that definitely changed with the book. After reading it, I felt inspired and empowered. I loved KSR’s repeated notion that it will take a multitude of actions - large and small - to bend this curve of climate disaster. There was no one, true way, no “perfect” silver bullet. Just a lot of work! I loved how characters repeatedly talked about “putting [their] shoulder to the wheel”. I loved how the book portrayed this multitude of actions, and was ambivalent on which were actually effective: knee-jerk geoengineering following a murderous Indian heat wave, violent eco-terrorism, multilateral cooperation akin to Bretton Woods, regenerative farms and rewilding. It all felt very believable (something KSR has spoken about - writing “optopias” (best case realistic scenarios) versus dystopias or utopias). Indeed, a lot of it is real!

So! Given all that! What did I want to do?

Guiding principles
#

Basically, I was guided by two principles:

“Be part of the solution, not the problem.”
“Be on the right side of history - aka, what will my descendants think of me?”

Vague - but that was the idea. Instead of climate nihilism, I decided to give it my best shot - specifically on reducing carbon emissions and plastic waste, as much as I could, from where I sit.

How to apply the principles
#

I then spent a LOT of time musing about the most impactful ways to reduce carbon emissions and plastic waste. (Well, the latter is pretty straightforward - more on that below.) Should I protest? Chain myself to an oil rig? Write my legislators? (ha ha sob) Never fly again? Buy an EV? Donate to a climate change group? Go zero waste? Bike a lot? Be vegan?

The world was my oyster! KSR gave me tons of ideas! And, finally diving headfirst into the world of climate change activism, I saw that - well, a LOT is going on. But I wanted to be impactful. I have a full-time tech job and full-time domestic/childcare work, so I knew I didn’t want to exhaust myself on peanuts. I also recognized that I was “doing” a lot of stuff already - buying plastic when I could avoid it, etc - and so maybe I could start changing the underlying systems of my life. What were the passive ways in which I was “part of the problem” and could I switch those out?

🏭 Taking stock
#

I also realized, pretty quickly, that quantifying all of this would be a bit of an art. There are a ton of online calculators for individual carbon emissions - I mostly relied on Earth Hero and UC Berkeley’s CoolClimate Calculator - but I tried several, and my estimates had a pretty wide range. Nonetheless, they gave me a sense of scale - and the main takeaways were: (1) the American lifestyle is not sustainable (per capita emissions are bananas), and (2) flying and food waste are big problems. Also, I waffled a bit on whether I should measure my individual vs. household emissions?

Quantifying plastic use was, again, more straightforward. The city where I live collects paper recycling, compost, and everything else in 3 distinct bins. I just wanted to minimize that last bin. So I could weigh it. I… could. I did not.

Okay, what I actually did (– and whether I think it was a hit 🎉 or miss 😖)
#

📚 Learning:
#

🎉 In gathering info on WHAT to do, I ended up using the Earth Hero app as my anchor. Tons of great ideas in here, highly recommend.
🎉 I discovered and subscribed to a ton of excellent climate reporting: Grist was an amazing discovery, as was Resilience and Yale Climate Connections. The Mothers of Invention podcast was also great and got me regularly fired up.

🏠 ✈ House + movement:
#

😖 Signed up for community solar. It’s one year later and we are still waiting for our solar array to get set up.
🎉 Cargo ebike! This replaced a TON of car rides (e.g. daily commute with kids), and we rode it all year round. Big success!
🎉 Replaced all shampoo, conditioner, hand soaps with bar soaps. Easy. Done!
🎉 Replaced toothpaste with toothpaste tablets.
🎉 Replaced household cleaning supplies (laundry, dishes) with zero waste alternatives. Quite onerous to do - you have to hunt for your local specialty zero waste shop, mamma mia.
🎉 Got composting back online! Fetid stink yey.
😖 Reused those damn tote bags. Sometimes?
😖 I seriously investigated alternatives to flying. I remember meeting a climate activist in Berlin in 2017 who told me he attended some workshop in NYC… by boat. At the time, his behavior felt extreme, overly zealous. Now, it seems perfectly rational. The “flight shame” movement has picked up a bit of steam - mostly in densely-connected Europe. My two big flying commitments are (1) business trips every ~4 months, and (2) family trips twice a year. I didn’t think I could drag the family on a transatlantic sea voyage, but maybe I could find a transcontinental train route to get to the work meetings? Nope. Amtrak does have a way to get across America by train - but the options are very limited, mostly geared towards tourists (not work travellers), and THREE TIMES THE PRICE of flying (if you want a private room… and I did, for 3 days’ worth of travel!).

🥦 👚 Consumption:
#

🎉 Damn, I think I had beef only like 2-3 times this year? Good for me. But!
😖 BUT WELL ACKshually, cheese and chocolate are worse than prosciutto (!), and I cannot bear to give those up. I half-heartedly tried to only eat eco chocolate, but my beloved Lindt - even with its cadmium and lead - still reigns supreme on my taste buds!!
🎉 Reduced online shopping to a bare minimum. I mostly bought digital items (books, etc) and second hand stuff.
🎉 Didn’t buy any new clothes, only second hand! Discovered the vibrant world of online thrifting, glory be.
🎉 Successfully ruined everyone’s Christmas by insisting that we do no gifts - everyone got a World Wildlife Fund animal “adoption” instead. YOU’RE WELCOME.

💰 Wealth:
#

🎉 Moved money out of climate chaos-causing mainstream banks and into institutions that are more sustainable, per Bank.Green. See also this amusing Yale Climate Connections piece.
😖 Retirement accounts are still invested in fossil fuel industries.

✊ Activism:
#

😖 Started a green club at work. My idea was - I can use my leverage as an employee to steer the company towards, e.g., reporting its emissions, investing in advance market commitments(a la Frontier), offering regenerative ag CSAs as a wellness perk, etc. Unfortunately, I just did not realistically have the bandwidth to lead this actively. So, for now, it’s just a collection of eco colleague-stragglers and we send each other articles every month or so. It’s very sad.
🎉 Became a monthly donor to my local climate activist groups. Most of these are pretty mainstream political advocacy groups, not the spicier, civil disobedient ones. I was deemed too old for my favorite one lol, and apparently Europe is where all the cool old people activism is (even one just for me!).
😖 Attended a civil disobedience climate group, but didn’t stick with it. Again, a lot of this is time. I think Gandhi, MLK Jr. and Mario Savio were right - sometimes you have to put your body onto the gears of the machine to make it stop - but I just couldn’t find a way to plug in here that actually worked for me.
😖 Similarly, volunteering with my local “mainstream” climate activist groups also didn’t work: I tried plugging in as a tech volunteer, but that fizzled out. I did some pro bono work for a friend’s biodiversity startup. I attended lots of Zoom intro calls, as well as a couple calls with local legislators (yay). But nothing seemed to click, nothing felt impactful, it felt like I was trying to climb a slippery slope and all I could ever concretely do was sign a petition. There must be a way in that is sustainable for the busy working parent?!

Conclusions from a year of attempted sustainability
#

After a year of this work, how would I rate it? What will I take forward?

Climate nirvana is painful
#

I think most educated people in the world accept that human-caused climate change is (a) happening and (b) bad. When you don’t pay attention to it, though, you don’t really know how bad. My vague, unsettled, NYT-driven climate anxiety was unhelpful. I would swing between fairly big extremes: either fearing apocalyptic, civilizational collapse, or trying to reassure myself that “science would fix it somehow, surely it’ll be okay”.

So, well, THANK YOU so much to KSR - one of my top 2 living sci-fi authors! - for presenting a believable prediction for this century. It was scary, sometimes horrifyingly so. But it was also informative - and galvanizing. Humanity will be OK, we are resilient. Which is another way of saying that humanity can bear a lot of suffering. So let’s try to minimize that suffering!

After reading the book, I did spend a couple weeks in a haze - similar to what Richard Feynman described after working on the Manhattan Project. I looked at everyone as mindless sheeple. I was living encased in a fuzzy bubble of cosmic perspective - who cares about this work deadline when our society is hurtling towards disaster? Did you just buy a pack of toothpaste tubes from Amazon - are you trying to kill us all?! Not a good way to be. But also: the natural part of internalizing something quite stark!

Several climate activism resources - Peter Kalmus comes to mind - speak about this being a normal part of the process. First, you have to grieve. I really agree with that now.

We are going to hit a wall of ecological reality sometime this century. We can either hit the wall going full-speed, causing massive suffering, or we can navigate this change on our own terms, skillfully. We have the knowledge and the resources! We just need… okay, we need global collective action. Which is not looking too hot right now, to be fair.

And individually, trying to live with this reality in mind - just the practical aspects of living within Earth’s limit, as an individual/household - was sometimes massively difficult. And always for quite stupid, irrational reasons. You find yourself pushing against enormous systemic forces. Seriously, have you tried finding toothpaste tablets with flouride? It is a challenge! And why don’t we have way more overnight trains, hello?! Reducing emissions and plastic can become a full-time job. Just like living free of Big Tech is almost like opting out of modern society, living a carbon neutral and zero plastic life was basically impossible.

That said - climate nirvana was oddly joyous?
#

Basically, even though my actions were a drop in the bucket, when things clicked - they really, really clicked!

Cory Doctorow (my other top 2 living sci-fi author, surprise!!) has this quote about fighting a fight that you don’t “win”, you just fight. Joanna Macy also had some stuff to say about this on the We are the Great Turning podcast - a sort of Buddhist non-attachment to a particular outcome, even as you work for… a particular outcome. And I think that’s healthy: balancing the grief with joy and optimism - and pragmatism!

I noticed how, as I dove very deep on the climate change activism reading, I found some doomerist strains - dark and scary nihilism, death culty stuff about how we’d all be living in a Mad Max hellscape by 2100. These would be written by authoritative voices - scientists, etc. Brrrr, I shudder. And just not for me. To paraphrase Kant, being optimistic is a way to stay sane and keep plugging away. (Plus, history is littered with catastrophes and, as far as I can tell, these people cannot see into the future!)

Anyway, on the practical side: the parts of my solarpunk lifestyle that worked really worked. The joy of e-biking was just - chef’s kiss. (And you can park anywhere!) The ease of shopping less: having less junk, less cardboard boxes, less plastic bags. Less clutter! The fun and sensible-ness of thrifting. Not giving or receiving a bunch of stuff at Christmas! LESS STUFF. Mamma mia, we can BREATHE again. And regenerative ag fruit and vegetables! Very good.

Future plans
#

The one thing KSR’s book also brought home was the importance of imagination. I discovered solarpunk through him, and just having access to a positive vision of the future was so, so valuable. It gives you something to work towards. My go-to utopia has always been Star Trek - specifically, The Next Generation - but that’s very far future (and does include, okay, a sentient AI war and major societal collapse between them and us). Solarpunk is much more accessible! Why, I feel like most of the Netherlands is living the solarpunk life right now (all those bike lanes!!!!).

So, given that, I’m going to:

Keep cultivating a positive, solarpunk framing through fiction, art, essays, whatever. It’s important to imagine how things could go right.
Keep doing the 🎉 good stuff.
Pick one “collective action”-y activity and try to really understand how I can plug in and be impactful. There must be something between “sign online petition” and “spend hours during your workday in a local legislator’s office”, right? Right????

Fixing subtitles for obscure movies

Mon, 07 Jul 2025 00:00:00 +0000

One of my favorite movie genres is commedia all’italiana - aka, “Italian-style comedy” - aka, Italian tragicomedies with sharp socio-political edges, mostly made in the 1970s. Two of my favorite examples from this genre are Ettore Scola’s Dramma della gelosia and, uh, Ettore Scola’s C’eravamo tanto amati. (Scola was good at this!)

Anyway, the tragedy of these great commedie all’italiana is that they’re getting a bit lost in the mists of time. They’re still pretty well-known in Italy, but they’re definitely being forgotten in other countries. This is a damn shame! Especially when commedia all’italiana directors like Lina Wertmüller made history by being the first female director to be nominated for a Best Director Oscar.

So I’m always trying to push these movies onto people - and that usually means struggling to find English-language subtitles. In 2012, I think I painstakingly wrote my own subtitles for a few of these films. Those .srt files are buried somewhere in my hard drives. Today, I am way less time-rich, and - after ripping my ancient DVDs into Plex - I was gratified to see that Plex lets you search and select for subtitles from the player UI itself. The problem was that most of these subtitles were just a little bit offset - they’d be a few seconds too early or late, depending on whether the subs came from a slightly different DVD version than mine. Plex has a subtitle offset button that does, well, nothing. So… what do?

Solution: Thank you, `ffsubsync`!
#

Open source to the rescue! Thankfully the ffsubsync does just what I need. After downloading some subtitles for my obscure Italian movies, I tried it out locally:

ffs C_ERAVAMO_TANTO_AMATI.m4v -i ceravamo.srt -o ceravamo_fixed.srt

This worked!!!

Glory be!!! Ah-Romaaaaaaaa… moh si parla!

Doing it on the NAS?
#

Okay, so it works! Now I needed to get my environment set up on my NAS. I’m sure there’s automated packages that do all these steps, but I wanted something very very minimal (both to start and to maintain). For now, that just meant:

Downloading the Python package from Docker Hub. This package creates a tiny Debian box with the latest Python.
apt-get update && apt-get install ffmpeg - Ol’ reliable ffmpeg is the main dependency.
Set up a Python virtualenv and pip install ffsubsync into it.
Download the subs I need from OpenSubtitles.com.
And voila!

Culture REVIVED!! Glory be. 🎉 Let us now gaze lovingly at a young Giancarlo Giannini’s long eyelashes, and chuckle at his amusing comments about Scandi polyamory ho ho ho. Seriously, readers, I highly recommend these movies.

On being an Obsidian power user

Thu, 03 Jul 2025 00:00:00 +0000

My boss recently called me an “Obsidian power user” and - how flattering! So let us take a tour of this, ahem, POWER.

What be Obsidian?
#

Obsidian is a closed source note-taking app. It has cultural roots and overlap with the Zettelkasten knowledge management system/lifestyle/philosophy, and has lots of overlap with Evernote - which is what I used for many years, before it became progressively enshittified.

What I like about Obsidian, and what keeps it very sticky for me, is:

Notes are kept in Markdown (.md) files. Soooo…
You can use Markdown syntax: especially code snippets,

code blocks

and LaTeX-ish math. You can do all your usual Markdown/HTML things, like linking across notes, across the web, embedding images and… uh, that’s it, I think?

But honestly: nice formatting of code, writing, and math = good enough for me!

Obsidian and the Zettelkasten people also tout its pretty graph representation of knowledge, e.g. my knowledge as a ball:

Tbh I never found much use of the ball o’ thoughts view.

How I use Obsidian
#

Every morning, when I open the Obsidian app on my personal or work laptop, my daily note springs up - auto-populated with its daily template (core plugin: daily notes). The title of this note is always a date, string-formatted to YYYY-MM-DD (day of week).md. I also enabled a nifty auto-conversion of natural language dates, so that if I type “@Yesterday”, it automatically converts it to “2025-07-02 (Wednesday)” which also becomes a clickable link to that day’s note 🤯 (community plugin: nldates).

Anyway, this daily note template includes tons of emoji (community plugin: emoji shortcodes), because I am a super-basic visual person and emoji are the new hieroglyphics and they give me joy. The template includes three main sections:

✅ TODOs
1. 🔁 Daily rituals
2. 🎯 Today’s specific TODOs
📓 NOTES
🧠 Some meta-analysis about what I’m working on

All the ✅TODOs are tasks (community plugin: tasks) which I format to allow for partial completion and not-doings (theme: things). I give each task a time estimate (e.g. “30m”) and flag it as #work or #personal, as well as a due date (if applicable). As you can see, this is a bit of a Frankenstein mish-mash of the plugins’ capabilities and what I like.

For 🔁 daily rituals, these are the things I do to “clear my desk” every day - checking work notifications, zero inbox, Anki, and a reminder to frickin’ MOVE (aka some kind of exercise, eating vegetables, etc).

For 🎯 today’s TODOs, I used to just append an ever-increasing number of things here, but lately I’ve switched:

I keep a BACKLOG.md note, where I dump all the stuff I will eventually need/want to do.
I pull 2-3 TODOs into my daily note for today. Keeping this short and focused makes it way more likely I actually achieve something by the end of the day.

For 📓 NOTES, this is where I just brain dump all my thoughts about my tasks - as I work through them. It’s where I scribble pseudo-code snippets, draft docs (including blog posts!), meeting notes, TILs and notes to self, etc. This can be short or long.

And now - the pièce de résistance! - I discovered recently that, since Obsidian is basically a webpage (HTML, Markdown, JavaScript, CSS) - you can… CODE-IFY IT?!! aka I discovered that you can write short JavaScript-esque snippets to analyze your notes (community plugin: dataviewjs). So I have a little code block in my daily template, co-written by Claude, that parses my notes and my BACKLOG.md to give an estimate of how much time I’m planning to work (this is a good sanity check!) and if there’s anything urgent bubbling up from the backlog. Here it is:

function getTodayFileName() {
 const date = new Date();
 const formatter = new Intl.DateTimeFormat('en', {
 year: 'numeric',
 month: 'short',
 day: '2-digit',
 weekday: 'long'
 });

 const parts = formatter.formatToParts(date);
 const values = parts.reduce((acc, part) => {
 acc[part.type] = part.value;
 return acc;
 }, {});

 return `discord/${values.year}-${values.month}-${values.day} (${values.weekday}).md`;
}

// Get today's tasks
const todayFile = getTodayFileName();
let todaysTasks = dv.page(todayFile).file.tasks;
console.log("Found tasks:", todaysTasks.length);

// Get waiting tasks from backlog
const backlogFile = 'discord/BACKLOG.md'
const waitingTasks = dv.page(backlogFile).file.tasks.filter(t => t.text.includes('#waiting'));
console.log("Found waiting tasks:", waitingTasks.length);

// Get tasks that are due soon or past due
// Create a date 7 days in the future
const today = new Date();
const sevenDaysLater = new Date();
sevenDaysLater.setDate(today.getDate() + 7);

// Filter for tasks with due dates in next 7 days
// Assuming due dates are in format 📅 YYYY-MM-DD
const dueSoonTasks = dv.page(backlogFile).file.tasks.filter(t => {
 const dueDateMatch = t.text.match(/📅\s*(\d{4}-\d{2}-\d{2})/);
 if (!dueDateMatch) return false;

 const dueDate = new Date(dueDateMatch[1]);
 return dueDate <= sevenDaysLater && !t.completed;
});
console.log("Found due soon tasks:", dueSoonTasks.length);

// Parse all tasks
function parseTask(t) {
 const timeMatch = t.text.match(/(\d+)([mh])/);
 const timeEst = timeMatch ? parseInt(timeMatch[1]) : null;
 const timeUnit = timeMatch ? timeMatch[2] : null;
 const mins = timeUnit === 'h' ? timeEst * 60 : timeEst;

 const dueDateMatch = t.text.match(/📅\s*(\d{4}-\d{2}-\d{2})/);
 const dueDate = dueDateMatch ? new Date(dueDateMatch[1]) : null;

 const isWaiting = t.text.includes('#waiting');

 return {
 text: t.text.replace(/⏲ \d+[mh]/, '').replace(/📅\s*\d{4}-\d{2}-\d{2}/, '').trim(),
 mins: mins,
 dueDate: dueDate,
 isWaiting: isWaiting,
 completed: t.completed,
 willNotDo: t.status === '-'
 };
}

let parsedTodaysTasks = [...todaysTasks].map(t => parseTask(t));
let parsedWaitingTasks = [...waitingTasks].map(t => parseTask(t));
let parsedDueSoonTasks = [...dueSoonTasks].map(t => parseTask(t));

// Calculate total time (excluding waiting tasks)
const activeTasksOnly = parsedTodaysTasks.filter(t => !t.isWaiting && !t.completed && !t.willNotDo);
const totalMins = activeTasksOnly.reduce((sum, t) => sum + (t.mins || 0), 0);

dv.header(1, `🧠 Today's workload: ${Math.floor(totalMins/60)}h ${totalMins%60}m`);

// Show tasks due soon
dv.header(2, "📅 Due Soon");
dv.table(
 ["Task", "Due Date", "Time Est."],
 parsedDueSoonTasks
 .filter(t => t.dueDate && !t.completed && !t.willNotDo)
 .sort((a, b) => a.dueDate - b.dueDate)
 .map(t => [
 t.text,
 t.dueDate ? t.dueDate.toISOString().split('T')[0] : "",
 t.mins ? `${Math.floor(t.mins/60)}h ${t.mins%60}m` : ""
 ])
);

// Show waiting tasks
dv.header(2, "⏳ Waiting On");
dv.list(
 parsedWaitingTasks
 .filter(t => t.isWaiting && !t.completed)
 .map(t => t.text)
);

Overall, I use syncthing to keep my notes synched between work and personal laptops, as well as my phone.

The once and future king
#

I’ve been using Obsidian now for a few years. It’s very sticky. I like it a lot. The one thing that I miss from my Macbook days is a good time tracker - e.g. I really miss Qbserve, which served me well for many years indeed. I’ve tried ActivityWatch, a Linux FOSS alternative, but it didn’t Just Work ™ in the usual way and I eventually got fed up with it. I might give it another shot, but… well, I do miss seeing my actual time worked (rather than estimated time worked).

Another nice-to-have might be integrated Pomodoro? Though I’ve been pretty happy with using my physical hourglass (!).

Subscribing to GitHub releases

Tue, 03 Jun 2025 00:00:00 +0000

Oh wow, this is handy. Noting this down! You can subscribe to a GitHub repo’s releases via RSS (thanks, trusty old StackOverflow). I wanted to keep an eye on the Wallabag repo, since I made a feature request about some relative URL referencing being broken. You can do that via RSS! GLORIOUS RSS! Just append .atom to any releases page, e.g. https://github.com/wallabag/wallabag/releases.atom.

Reading things later with Wallabag and my Kobo

Mon, 26 May 2025 00:00:00 +0000

Reading things later
#

In 2014, I discovered Pocket. I was living in Tanzania and often took the ferry between Dar es Salaam and Zanzibar. This is about a 2.5-hour trip, and the internet dies out pretty quickly once you leave port. I paid for the “fancy” tickets and normally preferred sitting in the lounge, which had gigantic La-Z-Boy-type chairs and a big TV screen. I watched a lot of Charlie Chaplin movies on those ferry trips. But! I also discovered the Pocket app, and it was a great way to pre-load a bunch of stuff to read while sailing.

After that initial burst in Tanzania, I rediscovered Pocket a couple years ago when I found that my browser (Firefox) and my e-reader (the Kobo) integrated with it. Reading it later… in e-ink?! Truly glorious.

The end of Pocket
#

Unfortunately, the good times are coming to an end - Pocket is shutting down in July 2025. I’ve already been moving as much as I can to self-hosted solutions (e.g. all my news comes in via my self-hosted RSS reader), and so I immediately went hunting for an alternative - ideally self-hostable and open source.

The main ones I came across were:

I ended up going with Wallabag because it was: (a) open source, (b) self-hostable, (c) mature, and (d) I found several blog posts of people integrating it into their Kobo reading flow.

Setting up Wallabag on my Synology NAS
#

As with my other self-hosted services, I relied on the Docker Hub image - I set the minimum environment variables up, got some help from Claude on how to debug the env vars lol, and then had the pain of exporting my Pocket saves and importing them into Wallabag.

For now (an open PR will change this), the process is (1) Pocket exports a CSV, (2) but Wallabag wants to talk to Pocket via its API. Either way, I had 4k (!) Pocket saves, and Wallabag’s connection died a couple times trying to import everything. I’m assuming Pocket didn’t save the plaintext of the URLs, and so Wallabag had to ping 4k URLs to try to download the <body> or whatever tags. How tedious! I haven’t checked yet, but I do hope Wallabag is now just saving the plaintext of these articles.

…and my Android and browser
#

The nice thing about Wallabag has been its maturity, as I said: it has an Android app and browser extension available. Note: the browser extension had a small fight with my browser re: HTTPS vs. HTTP.

The next big piece: the Kobo
#

So the vanilla Kobo OS (?) is integrated with Pocket. (This was one of the big reasons I bought a Kobo! sob) I knew that switching to Wallabag, or any other option, really, would probably require jailbreaking it and loading KOReader. This was actually super easy. There’s an install.sh script, but - honestly - it was literally just plugging in my Kobo, downloading the ZIP file and extracting it into the appropriate folder on the e-reader. Here’s the details. It was totally painless - my Kobo immediately restarted, and now I had the KOReader option available on top of my vanilla Kobo menu.

After setting up KOReader, it was just a matter of configuring the (built-in!) Wallabag integration to talk to my self-hosted instance. Doing this was actually a bit tricky, and took some brainstorming with Claude. Basically, my Kobo can’t get on my Tailscale - which is how I connect to my self-hosted services when I’m not on the local nework. But Wallabag’s PHP server insists that incoming HTTP requests, ahem, call it by its (domain) name. And so I just set the Docker’s env var, SYMFONY__ENV__DOMAIN_NAME to both the local network IP address and the Tailnet IP address. This has broken my instance’s CSS and JS (ughhhhhh), but the basic functionality of save an article on my phone or laptop -> read it on my phone or Kobo works - and that’s the most important thing!

Moving everything into RSS

Sun, 06 Apr 2025 00:00:00 +0000

NAS joy
#

One of the few things that brings me true joy these days (!) is self-hosting. Bit by bit, I’ve been moving things into my little NAS - and each little app or service gives me just a little more feeling of dopamine and control during these unsettling times.

Of course, I don’t always end up using the apps and services I self-host. For example, while I use the heck outta Plex, I set up Tautulli and then… forgot about it. Similarly for Pi-hole (I know, I know) and a few others. I have a little garden of dead Docker containers.

So it was with great pleasure that I both set up - and became addicted to - my newest service: FreshRSS.

RSSing my life
#

A couple months ago, I read Cory Doctorow’s blog, “You should be using an RSS reader” and - as often happens with Cory (my lifestyle guru, I swear lol) - I was stupidly inspired. While some of Cory’s promises did not pan out - namely, I couldn’t find a way to RSS my USPS packages, and several big-name news orgs only provided a title and summary in their RSS feeds (NYT, FT) - I did end up finally finding a way to consolidate all those sporadic “cool people” I found on the internet and wanted to follow. All those blogs, newsletters, webcomics, and lesser-known news orgs. What’s even better is you can follow Bluesky and Mastodon accounts… without having a Bluesky or Mastodon account (YES, PLZ, FOR THE LOVE OF GAWD).

A brief tutorial
#

Some notes for my future self, if my NAS ever dies. The process is pretty easy:

Pull the FreshRSS Docker image from Docker Hub.
Configuring a couple volumes to be mapped into the container: /data and /extensions. I hate this part. Years pass and I am still baffled by what Docker wants of me.
Running the container. There’s an initial check which the container auto-runs to make sure it sees all the directories it needs. I remember some drama here of creating a user account and then promptly forgetting the password. I ended up having to go into the SQL database in the container to manually re-hash a password (LE SIGH).
Then the fun part: adding RSS feeds!

What’s interesting is that RSS is kind of an old technology, and so most websites highlight their social media links. But, what’s even more interesting (amusing?), is that if you just add /rss/ or /feed/ to the top-level domain of some (WordPress?) websites, it auto-magically redirects to an XML-formatted RSS feed. I’m guessing that only a few “front end providers” (WordPress, Squarespace, etc) are the basis of a (vast?) majority of websites - so, as long as WordPress and Squarespace auto-handle RSS generation - then so do all their users’ websites. At least, this worked a surprising number of times.

Of course, all the crunchy intellectual web dorks of left-leaning politics loudly blared their RSS feeds - THANK YOU, to all of those. Oh yeah - and you can turn any Substack into an RSS (again, just append /feed/, as in your_fave_author.substack.com/feed).

Mobile
#

Ah yes, another very important use case is the ability to see all my feeds on my phone. There are a plethora of RSS apps. For now, I’m using Tailscale to create a secure connection to my NAS when I’m not on the local network. Then, in terms of mobile front end, I used one of FreshRSS’s “recommended” apps for Android: Capy Reader. I had to input the Tailscale IP - and I remember some futzing with login passwords and ports. Capy Reader syncs reads (which is the main thing I care about); I tried deleting a feed from the Capy Reader app, but this didn’t work. Similarly, I haven’t tried adding a feed either.

Two months in…
#

Two months in and my FreshRSS app is still one of my most-checked sites that I visit when I’m on my laptop.

Stuff that works well:

All the Substack newsletters are great to read in the app. I’ve (re-)discovered Timothy Snyder, Timothy Garton Ash, and Adam Tooze. Yay, PPE life! I added a couple shortcuts from FreshRSS so that I can also immediately save an article to Pocket - which syncs automatically with my Kobo. This is as close to a perfect flow as I can get!
Another content type that works great has been webcomics - finally an easy way to stay up to date with xkcd or SMBC.

Stuff that doesn’t work well:

The Bluesky feeds are high volume, low content, and often relying on embedded content (which FreshRSS strips). What usually ends up happening is that some Bluesky person posts something “outrage”-inducing, but I can’t even see the embedded context, and so I click through… back to Bluesky. This back and forth is super tedious, and I’m debating just removing (all?) of the Bluesky/Mastodon stuff and just using those apps? This would be a real loss, though.
Other high volume, low content stuff are any of the big name news orgs - e.g. Financial Times, you’re killing me here! While some news orgs - like The Intercept, Mother Jones, and ProPublica - all thankfully post their entire article in the feed, many (NYT, FT, Economist, Wired) only post the summary. This is SUPER annoying right now. I think there might be a way around this - e.g. if I need to log into each of those news orgs, maybe the full article will come through? But for now, I’ve just removed all of that noise.

Overall
#

Overall, this is great. I feel empowered and informed. I feel less like an algorithm is force-feeding me news, and more like I’m seeking out what I’m interested in and then keeping tabs on it. I’m excited to keep tinkering!

This blog is now brought to you by Hugo

Wed, 10 Jul 2024 00:00:00 +0000

Welp! It was time! Time for a big website refresh. Let’s talk through it.

2017 - 2024: Tech debt 😱
#

As a reminder, the previous version of this website was:

originally written in HTML + CSS + a touch of Javascript, with just a few html pages (about.html, scifi.html, etc.)
a static blog generated by Pelican
- using a special Pelican theme on an outdated fork
- modified to match the CSS of my original, manually-written HTML + CSS…

So I would use Pelican to generate the entire website, but I basically had everything spread out over four (4!!) directories/repos, and it was an awful pain to maintain. This was how things had been going since 2017. And it was just very very tech debty.

2022: Thinking about Hugo 🤔
#

I had heard good things about Hugo, and thought it might be fun to try rewriting the blog in Hugo. This was back in 2022. I tinkered a bit, but then life got busy again and I dropped it.

2024: Moving to Hugo ⛵
#

I picked it up again, found a theme I liked (Blowfish), and decided to commit! This was a couple weekends’ worth of work.

1. Quickstarting Blowfish with a bare Hugo site 🐡
#

As a first step, I followed the Blowfish tutorial to get a basic site up. Nothing special here.

2. Converting my Pelican blog to Hugo 🐦
#

I had about ~100 blog posts that needed to get Hugo-fied. I wrote a short Python script to rewrite their Markdown just a touch, mostly the frontmatter and relative references.

def process_file(
 filepath: str,
 read_dir: str = INPUT_DIRECTORY,
 write_dir: str = THIS_DIR,
 verbose: bool = False,
) -> None:
 if verbose:
 print(f"Now doing {filepath}")

 with open(f"{read_dir}/{filename}", "r") as f:
 lines = f.readlines()

 frontmatter = []
 body = []
 in_frontmatter = True
 subdir = None

 for line in lines:
 if in_frontmatter and re.match(r"^\w+:\s.*", line):
 key, value = map(str.strip, line.split(":", 1))
 key = key.lower()
 value = value.replace('"', "'")
 value = value.replace(":", "-")

 if key not in VALID_PELICAN_KEYS:
 continue

 if key == "date":
 value = format_date(value)

 elif key == "tags":
 value = [str(tag.strip()) for tag in value.split(",")]

 elif key == "category":
 if len(value) > 0:
 value = [str(category.strip()) for category in value.split(",")]

 elif key == "slug":
 fpath = f"{write_dir}/{value}"
 if not os.path.exists(fpath):
 os.mkdir(fpath)
 subdir = value

 elif key == "summary":
 value = f'"{value}"'

 frontmatter.append(f"{key}: {value}\n")

 else:
 in_frontmatter = False
 if "```{" in line:
 line = line.replace("```{", "```")
 line = line.replace("}", "")

 if "{static}" in line:
 line = line.replace("{static}", "")

 if "img" in line:
 img_filename = re.findall(r"\((.*)\)", line)[0]

 if "https" in img_filename:
 if verbose:
 print(img_filename)
 continue

 # kinda hacky
 img_filename = img_filename.split("/")[-1]
 fpath = f"{write_dir}/{subdir}"

 if img_filename in os.listdir(fpath):
 print("Already here")
 else:
 os.rename(
 f"{IMGS_DIRECTORY}/{img_filename}", f"{fpath}/{img_filename}"
 )

 line = line.replace("imgs/", "")

 body.append(line)

In short, tons of string parsing - including some dreaded regexing. My workflow was mostly:

Ask ChatGPT for boilerplate
Try out regex in regexr
Tons of print statements to debug {chefs kiss}
Sit at cafe and think about the state of the world.

3. Just do the whole website in Hugo actually 🐡
#

Namely, the static, non-blog pages, like scifi. This just took some understanding of how Hugo configs work.

4. Deploying 🚀
#

I deploy this site manually, just SCPing the files over to the server where it’s being hosted.

scp -r public/* $WWW_USERNAME@$WWW_IP_ADDRESS:~/public_html/

Tadaaaaa! 🎉

80/20 it 😅
#

You may have noticed that some of the writing doesn’t show up, and the colors are a bit weird, and I think I still have draft posts sometimes showing up. Sigh.

Anyway, I kept telling myself: 80/20, 80/20, 80/20. So I’m keeping a running list of things I’d like to fix or polish, but the main website is up and running - and that’s what matters!

The Best American Science And Nature Writing 2020 (2020) - ⭐⭐⭐⭐

Sun, 16 Jun 2024 00:00:00 +0000

Metadata
#

Author(s): Michio Kaku, and Jaime Green
Number of pages: 384
Year published: 2020
Year read: 2024

Review
#

I always enjoy these essay collections. I was flustered that the editions cover the preceding year, so this one was actually 2019 articles. Aaarghhh. I wanted an overview of Covid science!! Oh well - onto the next one!

This one had lots of gems. Stuff that really struck me:
- The horrors of California wildfires, as experienced on the ground.
- Two excellent paleontology articles (who knew I loved paleontology so much).
- An adorable buddy movie about astrophysicists trying to find Planet Nine.
- A heartbreaking one about a young girl’s mental health.
- A great one about immunotherapy.

Less exciting was the stodgy New Yorker article about natural language processing, large language models (aka, LLMs, aka ChatGPT and “AI”). There wasn’t anything explicitly factually incorrect about it - it does a decent job of describing what LLMs are, etc - and I even agreed with some of the opinionated Luddite hand-wringing. But something altogether about it felt cringe.

📚 Goodreads

Conspirituality- How New Age Conspiracy Theories Became a Public Health Threat (2023) - ⭐⭐⭐⭐

Wed, 12 Jun 2024 00:00:00 +0000

Metadata
#

Author(s): Derek Beres, and Matthew Remski, Julian Walker
Number of pages: 384
Year published: 2023
Year read: 2024

Review
#

A long-ish, engaging intellectual screed against the marriage of conspiracy theorizing and New Age woo. This was written with a lot of color and brio, so it’s a lot of fun and quite podcasty.

This is a history of 20th and 21st century wellness movements - yoga, Reiki, all that New Agey stuff - and their underlying politics, as well as their often close relationship with conspiracy theories and cults. The first two parts are a lively sociological and historical study of this stuff, and I learned a lot and filled in lots of gaps that I had had. This was basically what I wanted to learn from <a href=https://www.goodreads.com/book/show/9255017-american-veda?from_search=true&from_srp=true&qid=HGPrL6drUm&rank=1">American Veda, a book I am too lazy to actually buy. So, it was a lot of fun! And cut very close to home, especially the Theosophists and Buddhism in the West stuff. [I look down at my Buddhist tattoo in some chagrin.]

I didn’t care for Part 3, where the authors present a Wall of Shame of the worst “conspiritualists” in recent years. Oh yes - “conspirituality” is basically the tight coupling of alt-right Qanon craziness with New Age business, and how the Covid pandemic was like gas on that flame. The Part 3 Wall of Shame chapters profiled the most egregiously awful of these yoga-practicing, coffee-enema-ing, Covid/election denialist influencers. And honestly, I just don’t have the patience to listen to complete and utter garbage for chapter upon chapter.

The final part zoomed in on more quotidian conspiritualists - the conspiritualist next door, if you will. This was, of course, much less outrage-inducing and much sadder. e.g. The breast cancer patient who refuses mainstream medicine and dies amidst her magical thinking and potions. Ugh. It’s just awful. So the book, ultimately, preaches compassion for these benighted and confused people.

The book also discussed the tropes of conspiritual thinking: namely, I replace your reality with my own. A lot of this has to do with people’s ignorance re: the scientific method - and their disappointments in it (i.e. they expect science to be infallible and - once a scientific question is “answered”, the answer never changes). This is a real problem, indeed, and is the problem of MY CAREER, I would say: how do you get people comfortable with uncertainty? How do you convey the epistemology of the scientific methods(and statistics, do not even get me started)? Without them just completely jumping ship and swimming to crazy island?

📚 Goodreads

Workflow 2024

Tue, 04 Jun 2024 00:00:00 +0000

It’s been a while! Here’s how my workflow looks, in this post-LLM world.

Editor: Neovim
#

After several happy years in Vim, I switched to Neovim. My .dotfiles will probably keep evolving, but, right now, I’m enjoying plugins like:

Lazy - For plugin loading.
Git gutter - For :Git blame all the damn time.
Telescope - For very quick and convenient file search.
DAP - Debugging.
Codeium - LLM autocomplete. And this sometimes fights with Jedi.

Terminal: ZSH
#

Specifically, oh-my-zsh, with some autosuggestions and the pure for aesthetics.

ChatGPT > StackOverflow
#

Ever since the Great LLM Explosion of 2023, I haven’t really gone back to StackOverflow for my daily random questions. Instead, my usual workflow is:

ChatGPT: Help me interpret this code, help me write code in a language I never use (e.g. React), what does X mean?
Khanmigo: Help me (re-)learn the covariance matrix, or basic chemistry, or grad-level stats, and other things. I like Khanmigo (enough to pay for it) since they prompt engineered it to be Socratic and “truthy” (better than the usual bullshit-as-a-service ChatGPT).

Desktop environment: Regolith
#

I’ve been using Regolith for a few years now - I think Primagean had a video about his desktop environment, using some similar tool. What I love is its simplicity and ease of use. I usually have 4 “tab” desktops open (L-R): web browser, Obsidian, work Discord, a terminal, and something miscellaneous like Spotify. I have this for both my personal and work machines, and I love it so much that I’m worried I can’t live/work without it?!

OS: Pop!_OS
#

It’s fine.

Notes/second brain: Obsidian
#

Obsidian is my jam. I really like it. Specifically, I love:

I can take notes in Markdown, and then grep those notes super-quickly straight from the shell.
I can write code blocks, LaTeX math blocks, headings, and tables are all supported.

OK, and galaxy brain here, but I set up automatic “smart” linking of dates (e.g. Today becomes 2024 Jun 6 (Tuesday)) and my daily notes template follows the same strftime pattern. So it’s super easy to link to other notes using smart dates.

The Battle for Your Brain- Defending the Right to Think Freely in the Age of Neurotechnology (2023) - ⭐⭐⭐

Mon, 29 Apr 2024 00:00:00 +0000

Metadata
#

Author(s): Nita A. Farahany
Number of pages: 288
Year published: 2023
Year read: 2024

Review
#

A fun, wide-ranging, and provocative non-fiction book about the ethics and possibilities of neurotech (aka brain-meddling!!).

So I spent most of this audiobook running and arguing with it in my head. I almost threw the headphones off in an angry huff during the “why ADHD meds are cognitive enhancers and totally fine ok” - I stuck with it, if only because I enjoyed the ethical debating. (Even though I do think the ADHD meds chapter was GOODNESS ME, really eliding some big, obvious downsides to ADHD meds (and steroids). Like, that shit can fuck you up.

The author, Farahany, is a law prof and was on Obama’s bioethics committee. So I do defer to her authority. At the same time, I feel like I’m kinda not into the Singularity/transhumanist world anymore - and I did sense some techbro enthusiasm in Farahany’s writing, which made me eye-roll. But I can’t criticize her too much; she did a decent job of outlining the ethical debate.

📚 Goodreads

Dawn of the New Everything- Encounters with Reality and Virtual Reality (2017) - ⭐⭐⭐⭐⭐

Tue, 17 Oct 2023 00:00:00 +0000

Metadata
#

Author(s): Jaron Lanier
Number of pages: 368
Year published: 2017
Year read: 2023

Review
#

Jaron Lanier’s brain is a veritable wonderland. There is no one like him. I was absolutely DELIGHTED by much of this book - especially the memoir of his childhood and adolescence. He is really an inspiration. And so funny! And so smart! It’s like his brain is made of champagne fizz. Like the very best Star Trek: Next Generation episodes.

Anyway, this book inspired me to try out virtual reality.

Also, this book took me forever to read, HAHA. I basically started it, was delighted, got consumed by chess for a couple months, got distracted by my life for the next few months, picked it up and finished it. Womp!

📚 Goodreads

In the Camps- China's High-Tech Penal Colony (2021) - ⭐⭐⭐⭐

Mon, 27 Feb 2023 00:00:00 +0000

Metadata
#

Author(s): Darren Byler
Number of pages: 150
Year published: 2021
Year read: 2023

Review
#

A disturbing and distressing examination of the currently-happening Uyghur genocide, and how it’s accelerated by modern surveillance state technologies to create “smart” labor/reeducation camps.

Just like the Nazis leveraged early 20th century industrialization to organize the Holocaust, so China is leveraging computer vision, smartphones, and machine learning to organize its own destruction of the Uyghur, Kazakh, and other ethnic minorities along its borders. A short book, but an informative and scary one! Darren Byler situates this genocide in the overall history of genocides and state surveillance; this book made me want to restart James Scott’s Seeing Like a State - since this book, and this genocide, is very much the eventual, worst extension of what Scott described: the creation of last names, street names, and order so that the state may “see”.

📚 Goodreads

Digital for Good- Raising Kids to Thrive in an Online World (2021) - ⭐⭐⭐⭐

Tue, 13 Sep 2022 00:00:00 +0000

Metadata
#

Author(s): Richard E. Culatta
Number of pages: 256
Year published: 2021
Year read: 2022

Review
#

Finding pro-screentime parenting advice is about as common as finding pro-formula tips. i.e. Completely uncommon because of an enormous cultural juggernaut that moralizes against it. I was starting to feel crazy trying to parse the WHY screens are demonized so totally for our kids - by pediatricians, by “conventional wisdom” etc. Why did I have so much guilt when plopping my kid in front of Peppa Pig? I think the official guidelines are zero screens before 2 years of age. I saw some parents asking if this meant no Zoom calls with grandparents. What!? What, exactly, do these people think screens are doing to their babies?

I sometimes see some data and evidence trotted out: namely, kids who watch more TV are fatter and have more behavioral problems. Two questions, here:
1. Is this correlation or causation? Maybe hyperactive kids need TV to calm down?
2. If it’s causal (TV displaces physical activity), then isn’t any sedentary activity just as risky? Like, say, reading a book? Or drawing?

Also, TV screens are quite different from phone screens and laptop screens?

ANYWAY. So I was happy to find this book. I needed a sanity check. This book was very sanity restoring. He makes a few excellent points. Namely, we prepare our kids to live in the world - we teach them to look both ways, to brush their teeth, to be kind to most and careful around some. That’s all, ahem, IRL. This book makes the EXCELLENT point, that I - a tech person!!! - honestly never even thought about, which is that cyberspace exists and a lot of stuff happens there and so maybe we should… parent for it? i.e. Just like you look both ways before crossing the street, you double check that sketchy e-mail attachment before downloading. Just like you don’t bully a kid in class, you also don’t troll a kid online. And so on.

Culatta removed the high-pitched shriek of moral hysteria from my brain re: screens, and gave me the space and quiet to just THINK about it for a second. Having this space was liberating. And it scared me - how little ACTUAL THOUGHT I had given this issue of, ahem, what’s in the screens! Because abstinence is, indeed, a very stupid way to teach kids both about sex. AND SCREENS.

One of his best ideas - which, again, duh, I feel stupid I never even thought about this - is that you can gradually introduce computers, the internet, and so on to your kids. Most apps have parental controls. But also: the choice isn’t between giving your tween a smartphone and a dumb phone. You can just dumb down an old smartphone - removing most apps, no data plan - and then gradually introduce things, piece by piece.

The book covers lots of interesting things:
- online bullying (and how our “screen education” is literally only ever about that)
- the rise of echo chambers and fake media
- advertising, especially geared towards kid
- how you can (and should! and must!) turn off that damn autoplay feature on YouTube, Netflix, etc.

And so on.

Anyway, this got me pumped. I’ve been pumped about introducing a Raspberry Pi computer to my kids since forever. Now I am feeling even more emboldened. Perhaps you shall have indeed MORE PEPPA.

📚 Goodreads

Split ergo keyboard

Wed, 04 May 2022 00:00:00 +0000

Oh God. Never have I had so many emotions over something as banal as a keyboard. But, after some research and a moment of madness, I decided to invest in a split ergonomic keyboard.

This is the ZSA Moonlander, a stupid expensive mechanical keyboard that is super duper customizable, and also split and ergonomic (via being split and having “columnar” keys).

Ever since my last pregnancy, I had been noticing some twinges - and occasional spikes - of pain around my wrists, thumb tendons, finger tendons. I started looking into ergonomics. It was especially painful when I picked my kids up after a long, sedentary day of, ahem, Tech Labor.

So I started looking into keyboards and noodling around mechanicalkeyboards.com. And I was very influenced by this Switch & Click video about trying a split keyboard for 30 days. Switching to a split keyboard, especially a very customizable one where I could experiment - in the future - with other layouts (Dvorak, Coleman, stenography!), and take my time off in between jobs to re-learn to type… well, it all seemed very rational. So I summoned my inner Picard and bravely declared MAKE IT SO and ordered it.

And since then, dear reader, it has been, oh, such an emotional rollercoaster. I’ve had it for 4 days and have wanted to throw it out the window maybe 100 times. Multiple times a day I need to remind myself to “keep the faith” and “no, a keyboard cannot literally make you stupid”. Because, oh dear reader, having my hands suddenly turn into two stupid meat paws has made me feel like my thoughts are now locked in my brain. (FWIW, I’m typing this on my laptop’s standard keyboard - so things are flowing! I wanted to give myself a break - and keep my old layout muscle memory going too.) And having my thoughts locked in my brain makes those thoughts wither and die. And that makes me stupid.

But no! I vowed that I would persevere through this learning curve because all learning curves which lead to objectively better tools/thinking/life (like Vim! Anki! Linux!) are worth surmounting. I must keep the faith. I must!!!

I’m also trying to de-mystify the process and make myself less afraid of being locked in stupid meat paw brain forever by reminding myself that when I recently learned to crochet, I struggled oh so very mightily. After years and years of knitting, using my hands in this different, weird, crochet way was very challenging. And after several false starts, I finally surmounted that learning curve and am now happily making amigurumi. (And crochet is 100% more rational than knitting - change my mind!)

Look at the creations now flowing from my hands! JUST LOOK.

Goals
#

My pre-split typing speed was ~100 WPM. My goal is to get to ~75 WPM on the split after 3 weeks. I think I’m on track.
I want to keep my non-split typing speed at minimum 90 WPM.
I want my coding typing speed to be “reasonable” - maybe 50 WPM? The more the better. I didn’t realize what a hit coding WPM would take…
I want to, obviously, not have any new or additional pain in my hands and wrists (or shoulders).
KEEP THE FAITH. TRUST THE PROCESS.

That’s it, man. I’m going to do the “deliberate” practice on Oryx, 30 min a day (10min 3x), and generally pray.

Job hunt 2022

Wed, 27 Apr 2022 00:00:00 +0000

I got a new job. I’ll be starting in a few weeks. I want to talk about the job search process since it’s been a while since I job hunted (I was at Optoro for almost five years).

Some stats
#

Platforms I used to find opportunities: Glassdoor, LinkedIn, Hired. the #jobs channel on PyLadies Slack, a couple friends
Applications I submitted “cold”: 48
Applications I made via responding to recruiters: 10

Almost every position I applied to was titled Senior Data Scientist, though there were a few (Senior) Machine Learning Engineers in there too.

The “average” interview process
#

The most common interview process was:

Recruiter screen (30 min). The questions here were almost always: what’s your background? What are you looking for in a next role? What do you work on currently?
Tech screen (30 min - 1 hour). The majority of these were live coding screens on Coderpad.
Hiring manager call (30-45 min). Meeting the potential boss, or boss’s boss.
Take-home assignment (many many hours). Sometimes instead of the code screen. Sometimes in addition!
Final round (1+ days). These often included an additional tech screen/live coding, one or more case study/technical deep dive interviews, and at least one behavioral/culture interview.

Cracking the coding interview… for data science
#

tldr: Listen to the question. Plan out your code using pseudocode/a bulleted TODO list that your interviewer can see. Execute on that TODO list. Practicing for performance anxiety was more worthwhile than grinding Leetcode.

The dreaded live coding interview!

I dreaded this step when I started applying, since I had had bad experiences previously: namely, having incredible performance anxiety, writing code that would run but still being rejected. Once I “cracked” this part of the interview (and disclaimer: ymmv), I then started passing these interviews… even when my code had errors or I got stuck. I’ve been through about a dozen of these, so my experience may not representative.

The live coding screens varied widely. I went through the following:

“Do Python and SQL stuff, either ‘whiteboarding’ on Coderpad or ‘open book’ (i.e. you can Google syntax).” (most frequent)
“Do data with Python but pretend you have no pandas. Closed book.”
“Use Google Colab, while screensharing, while I watch you use pandas.”
Big-O notation/algorithms and data structures.

When I started the job hunt, I was a bit frantic about how to prepare for these. I was at a loss. Should I spend an hour a day working my way up through LeetCode/HackerRank difficulty? Would I have to take an Algorithms & Data Structures online course? And so on.

What I’ve concluded, having now gone through this round successfully several times, is:

No, Data Scientist applicants were not expected to know big-O notation
#

Only one code screen (out of ~15) asked me about this, and I flatly said I didn’t have a CS undergrad background, had never taken Algorithms and Data Structures, but was willing to take a crack at it. I basically mused aloud about memory stuff (thanks, CS50!), the interviewer nudged me forward, and it was fine. I ended up getting an offer from this place!

Performance anxiety prep is as important - or maybe more important - than code prep
#

In my case, the nerves were real. I went through a few code screens where I was extremely nervous. I would jump in, frantically write spaghetti code, hit small syntax errors, get flustered, and then have a brain shut-down.

They always advise you to “show your work” and “think out loud”; and my code and thoughts were a jumbled-up, gibbering mess. I would invariably get rejected after these screens, which hurt. I felt indignant. “I’ve been deploying Python to production for 5+ years! Of course I know how to zip two lists!” I’d think, annoyed with them and angry at myself the next day. But the anxiety crept into my practice as well. When I would sit down to do some Leetcode problems, I’d be hit with it: a faint voice in my head, screaming, You’re too stupid for this!

It took me a few more failures before I realized: gosh, maybe I’m not stupid, I’m just nervous? And maybe I could try to be less nervous? In college and grad school, I used to have awful performance anxiety; I’ve had panic attacks before and during (!) exams. It sucked. Thankfully, after grad school, I learned a technique that really helped: visualization.

Basically, before the Big Day, I make a bulleted list of all the activities of that day: waking up, getting breakfast, shower, commute, etc. I then have two columns: “pre” and “post” anxiety, out of a scale of 1 (not anxious) to 10 (full-blown panic). I sit down with the list and force myself to really, deeply visualize each part of the day - each item of the list. I mark how anxious that specific image makes me: eating breakfast JUST HOURS BEFORE THE CODING INTERVIEW. Opening the Coderpad link. Saying “hi” to the interviewer. And so on. If my “pre” anxiety exceeds 5, I stop the visualization and do some calming breaths. I then record my “post” anxiety - “post” meaning “after I tried to calm down a bit”. If I get back under 5, that’s fine. If not, that’s fine too.

I do this repeatedly before the Big, Scary Event, maybe 2-3x on average… and it’s worked wonders! It was the way I got over panic attacks before public speaking. It was the way I “cured” my fear of flying. And I realized - belatedly - that it would help me during this interview process! While I never properly panicked, I did realize that nerves were getting in the way of showing my skills. The visualization exercise helped tremendously. To the point that, by about halfway through my job hunt, I no longer felt anxious before code screens… or any of the other interviews! 😱

This was such a strange experience - interviewing while totally relaxed - that I was suspicious of it. Surely nerves also help? But in the end, I think it was a good thing, to be so relaxed. It helped me think much more clearly on the spot, especially when interviewers tossed tricky questions at me during the case study interviews. And I ended up really living the growth mindset: suddenly, I hungered for those hard questions and was disappointed when an interviewer didn’t challenge me! I felt like overqualified drummer. And I think this intellectual restlessness - challenge me! - reflected well; more than one interviewer seemed pleasantly surprised that I said I enjoyed getting stuck (I did!). It also gave me a much better sense of what working with that person would be like (since I normally don’t have performance anxiety while in some random meeting with a work colleague I’ve known for a while).

So, yeah. Anxiety + visualization = relaxation. And relaxation = growth mindset!

Another tip: slow and steady beats fast and “clever”
#

I was once invited to attend a conference (at Harvard!) and then - surprise! - ended up being told “my panel was next”. Cue horror.

I sat myself at the end of the quite-long panel table and waited while the other panelists introduced themselves. While they did - and my mind was spinning - I noticed that (a) I couldn’t follow their introductions easily, whether because they spoke softly or in a roundabout way or whatever, and (b) audience members had similar looks of confusion or glassy-eyed distraction. I resolved that, since I had no idea what to say on this panel, I could at least speak very loudly and clearly and simply, just introducing the research org I worked for (and was, apparently, now representing) and the work we did. This went over swimmingly.

It was a lesson I forgot. But I re-learned it during these coding screens. Just speaking slowly, clearly, and simply goes a long, LONG way. Your interviewer is human. They can’t read your mind. They likely know very little about you - maybe they’ve looked at your resume. Even then, likely only briefly. So just introducing yourself clearly, repeating your understanding of the question prompt back to them, and writing down - in Coderpad - the list of what you’re going to do… oh man. That seemed to go a LONG way. I sometimes just wrote SLOW DOWN on my whiteboard (out of sight).

Before this realization, I would feel nervous as heck while the interviewer introduced themselves and talked in a leisurely way about the interview’s structure. I remember thinking, “Hurry, I need all the time I can get! Oh man!!” After my “go slow, be obvious” realization, I would intentionally take it slow - I would not rush through my intro, I would not try to scan through the entire question prompt as soon as they pasted it in Coderpad. I would sometimes just take a moment to make a joke that felt funny. Basically, I was not in a rush. And then, once I got around to actually answering the question, I would do something like:

# Q: [some easy-level Leetcode Python question]

def my_answer(some_input: some_type) -> some_other_type:
 """
 TODO:

 - create an empty array
 - apply machine learning to the elements
 - return some data type
 - do other things
 """

Meaning: I very methodically (1) defined some function and type-hinted the heck out of it (look, interviewer, I know typing!), (2) made some docstring, (3) filled that docstring with my overkill thoughts. When I was learning how to code in Python, I remember I had to write out my plan in plain English or pseudocode. As I got more comfortable, I stopped needing to plan. WELL, NOW I AM BACK TO PLANNING. And it has made me much more effective. Again, my sample size is small, but once I started doing Angela the Sloth Candidate Who Loves Checklists, I passed all my code screens.

Cracking the knowledge interview (case study, tech deep dive)
#

tldr: Anki! OMG, Anki.

Actually, my math above is wrong. It should be: relaxation + Anki = growth mindset. I, like a lot of other people, read Carol Dweck’s book and was like, yes. That sounds great. Of course. I should be growth mindset. But while the book made me aware of “fixed mindset” thoughts, it didn’t necessarily give me a sense of how to actually, ahem, ~live the growth mindset lifestyle~.

At the same time, a year or so ago, I discovered Michael Nielsen’s amazing post, Augmenting Long-term Memory, where I learned about spaced repetition and the evidence-based way to build memory (aka knowledge). After reading that post, I had messed around with the Anki app for a few weeks, but it didn’t stick. I made the classic beginner’s mistake of importing other people’s flashcard decks and using those. This year, early in my job hunt (thank goodness!), I decided to ditch the imported decks and start making my own Anki cards: specifically, I started working through the lecture slides from Harvard’s CS109A and CS109B courses - which I took in 2016 and 2017, respectively. Doing this was a game changer. In so many ways!

First, instead of vaguely remembering that, oh yeah, regularization is a thing… oh, right… Now I was like a robot that could recite the specific loss function that Lasso and Ridge regressions use.

Second, as I started “putting things in my head” (which is what Anki feels like; “memory is a choice”), I started to get what Nielsen promised: I started seeing connections where I hadn’t previously. I developed a deeper intuition. I saw the patterns!

Third, as I started to “know” (remember) more and feel more confident and creative with that knowledge, the fixed mindset image of a “smart person” started to crumble. I’ve often been deeply impressed when someone rattles off some random, exotic-to-me mathematical formula. A work colleague saying “Jacobian!”. Someone else integrating e on the fly. Whenever this happened, I would think “wow” and be both impressed and crushed. I’d think, I’ll never be that smart.

What I didn’t realize was: I was defining “intelligence” as “knowledge” - as memory - of things I didn’t know. No one impressed me if they said something I also remembered; they impressed me when they said something I thought I should know, but didn’t remember in that moment. And when I started using Anki, and memory became a choice, I realized: I could now choose to remember that as well. And anything else I thought was useful! And Anki is… a flashcard app! There’s no special magic to it. It just leverages how your brain works: flashcards on a schedule to fight against the forgetting curve .

The realization that, with “one weird trick”, I could suddenly know all of these things… well, the ceiling of possibilities just went way up. And my life has gotten so much richer. And now I really am, deeply, growth mindset. I think this directly impacted my desire to get hard interviews. I had a few interviews that were lightweight and easygoing, for roles that nonetheless seemed amazing. Past Angela would have counted her good luck. Current Angela was immensely disappointed, to the point that I worried those roles - even though the teams and projects seemed great - would be “too easy”. I started telling interviewers this, and really meaning it: I want hard problems. Even the overall “basic data science domain” - once I completed the Ankification of CS109A and CS109B - seemed… small? I was like Marlin: “3 stripes? that’s all I have?”

Fourth, I am finally going to put linear algebra in my head. FINALLY. And I purely mean an engineer’s/craftsperson’s toolkit knowledge of linear algebra, not the proofs. Maybe this is foolish. But I’m excited.

To take-home or not to take-home
#

Before I became a parent, I loved take-home assignments in hiring processes. I had ample leisure time to work on them! They removed performance anxiety. They felt more “honest” since, after all, they’re a more accurate representation of what working with a person would be like: it is rare for my colleagues to stand over my shoulder, give me a riddle on Coderpad, and then watch me code. It is much more common for us to discuss some problem, often via Slack or standup, and then for me to come back with some Jupyter notebook.

I still think there’s a benefit to take-home assignments - due to that last point above - but I think keen candidates will spend 2x the recommended time on them. I had three take-home assignments during this job hunt and I spent 7, 3 and 4 hours on them, respectively. I was rejected from the ones I spent more time on. At the time, I thought, well, okay, I probably spend about 7 hours prepping for a code screen. But that’s not true anymore. Once I cracked that screen, I basically didn’t prep for it. Once I cracked the performance anxiety, I just needed to leverage my existing Python muscle memory. And while a job hunt is, by definition, an enormous amount of energy spent on things that don’t materialize, the 7 hours I spent on that one assignment still stand out as… particularly wasteful of my time. Specifically, I hadn’t met anyone on the team yet, had no idea if me and the potential boss would gel well, had no idea what the specific projects I would work on would be. In other words, I was investing a lot of my time on something that might have ended up not even being a good fit.

That information asymmetry, coupled with the opportunity cost of my time as a parent, means I’ve cooled down a bit on take-home assignments. I still like them when I’m the one doing the hiring: they give you good insight into a lot of things - the candidate’s enthusiasm for the position, their knowledge, their creativity in applying that knowledge. Even stuff like code style, written communication (I’m a big fan of well-written Jupyter notebook Markdown cells.) Also, take-home assignments are standardized (everyone answers the same prompt) and much less subject to the random variation of a bad interviewer, bad day, etc. The whole hiring team can look over and discuss a candidate’s assignment at their leisure too.

Hmm. Okay, well now I’ve convinced myself again that take-home assignments are a better option. But maybe a code screen + a case study accomplish much the same differentiation, sacrificing a bit to variance/luck? Either way, I did appreciate when hiring managers stressed, repeatedly, not to take more than X hours on an assignment.

Final rounds, interviewing them
#

The only thing I’ll say about the final rounds - which were almost uniformly grueling, given the time investment (usually they were 4-7 hours, spread out over one or two days) - is that this was often the only opportunity I had to meet more of the people I’d be working with. And, thus, to interview them. I do take this seriously, since job offers need to be about a good fit on both sides. As one friend noted, a job hunt is 6 weeks of pain - but being in a “wrong fit” job can be much more painful and for much longer. So, as much as the emotional rollercoaster (“do they like me? they like me! they like me!”) sometimes clouded my ability to put the brakes on and ask harder questions, I did try. I had to consciously make an effort. It’s worth it!

Billion Dollar Loser- The Epic Rise and Spectacular Fall of Adam Neumann and WeWork (2020) - ⭐⭐⭐⭐

Sun, 27 Mar 2022 00:00:00 +0000

Metadata
#

Author(s): Reeves Wiedeman
Number of pages: 352
Year published: 2020
Year read: 2022

Review
#

A brisk and frisky journalist’s account of the, indeed, spectacular fall of WeWork. I have, like many people, used WeWorks. They’re fine. They’re very Millennial. And Millennials are getting old, people. I’m an old Millennial and even my “young Millennial” friends are using outdated slang. It’s not 2012 anymore!

I wonder what gen Z people think of the WeWork aesthetic, or I guess their offices will be in the metaverse.

Anyway, I had no idea about Adam Neumann. I had tangentially heard about some of his more egregious “crazy billionaire” antics. This book was… wow. I hoovered it up in one sitting. The schadenfreude was, agh, chef’s kiss. Very enjoyable. You gotta read it to believe it. In Neumann’s defense, he’s just a symptom of a crazy system right now - VCs, startups, etc.

📚 Goodreads

Nice bash tchotchkes

Tue, 21 Sep 2021 00:00:00 +0000

Hello! Some nice things I’ve been using lately:

`zoxide`: Smarter `cd`
#

zoxide is great - it keeps a “memory” of the directories you’ve been in and, once you’ve visited them, you can revisit them using something like z blog and it’ll jump you to whichever recent directory had blog in the path. Much, much more streamlined than cd /some/super/long/path, or making a bunch of aliases for such cds (as I have done in the past).

`mcfly`: Smarter reverse-search
#

Same deal as above: you type ctrl+r and it launches a smarter search window of your old commands. HANDY.

Prompt colors
#

I wanted to bedazzle my prompt, own it, express my - ahem - creative individuality. Also I wanted to fem it up. I use the pure theme for zsh, and it was as easy as adding this to my ~/.zshrc:

prompt pure
zstyle :prompt:pure:path color "#af87d7"
zstyle :prompt:pure:prompt:success color "#af87d7"

Irresistible- The Rise of Addictive Technology and the Business of Keeping Us Hooked (2017) -

Thu, 22 Apr 2021 00:00:00 +0000

Metadata
#

Author(s): Adam Alter
Number of pages: 368
Year published: 2017
Year read: 2021

Review
#

He’s preaching to the choir here, but I couldn’t get past the long, looong intro on “addiction is a thing”. Yes, it is. Yep, cocaine. Yep, lab rats pressing pleasure buttons until they die. I thought we’d get to the tech addictiveness-by-design a lot sooner. I ended up putting this down and never feeling like picking it up again.

📚 Goodreads

My own private server

Sun, 11 Apr 2021 00:00:00 +0000

Big changes in 2021, beyond Linux. Yeah, boy. We’re getting serious about computers, man. I shelled out the cash for a network-attached storage (NAS) device, AKA a tiny little cloud that lives in our home. Thrilled with this. Thrilled!

Our use cases
#

1. Consolidated data backup
#

Like most people (I assume), we were backing up our data in a number of places:

External solid state drives for when you’re like “okay, I should really back. up. this. computer. I hadn’t used my SSD to backup in 700+ days, my Macbook Time Machine helpfully screamed at me. Yeah, manual backing up is hard. Also, the SSDs had random other things we kinda needed but didn’t have space to put anywhere else - photos, movies.
Various big tech cloud services (Google, Dropbox). Automated backing up: so easy! So nice. Not free, of course. And… it’s on their servers. Which I have been increasingly uncomfortable with. Stop scanning my loved ones’ faces for your machine learning, Google! I knew I wanted to stop trading privacy for convenience.

The NAS offers one big helpful block of storage space. We can backup automatically. We can connect it to a secure (single!) off-site server for one consolidated stream of our crap. Yessss.

2. Media server
#

I didn’t think I’d enjoy this aspect so much but oh boy, have I been enjoying this. I installed Plex on our NAS, uploaded all our old movies and TVs and eeeeeee, now we can actually watch them! And they’re beautifully organized! Eeeee. And we can set up user accounts and even, if we’re willing to accept the security risk of opening the NAS up to the wild, hairy open internet, we could stream it remotely. Our own private Netflix! Eeeeee. Now I need to rip all our old DVDs and yaaaaas.

3. Photos
#

I really, really was getting annoyed with how entangled with Google I was getting. And the main entanglement was via photos of loved ones (of which I have, apparently, 11k!). I’m not a photographer. I just take a bajillion pictures because my pocket computer surveillance device makes it soooo easy. We have a Synology NAS and, using their native app, Synology Moments, I’ve been able to (a) automagically backup all the 11k loved one pics offloaded from my phone and (b) do the facial recognition thing to sort them into Specific Loved Ones. The latter is not as good as Google’s, I’m feeling, but it’s good enough. And privacy + good enough > no_privacy + slightly better.

In fact, a note to current/future Synology users: I followed a tutorial which advised me to use Google Takeout to manually export all my Google Photos data into 60+ (!) 2GB .zip files which I then started laboriously unzipping and moving to the NAS manually. Why. WHYY. The Synology Moments Android app just automagically backed all those 11k pics up. So… do that instead.

Linux, first impressions

Fri, 02 Apr 2021 00:00:00 +0000

This year was personal laptop rollover year, and I was excited to finally take the plunge: Linux! Yes, Linux. After polling some dev bros and doing a lot of YouTube research, I ended up replacing my 2015 MacBook Pro with a System76 Lemur Pro (specifically, the lemp10). With that new hardware came a whole new operating system: Pop!_OS. I’ve now had the new machine for a few days, and I want to record my initial thoughts/feelings with this brave new FOSS world.

Motivation
#

I grew up using Windows machines. In 2010, I switched to MacBook Pros, both for work and for home. I’ve been thinking about switching to Linux ever since this article about Cory Doctorow’s workflow came out in 2013, but lots of stuff held me back. I kept a spreadsheet of all the Mac apps I thought I needed that Linux wouldn’t have - Evernote, Microsoft Office (this was in my pre-career transition days). Finally - in 2021 (!) - my life is now programmy enough that moving to Linux seemed sensible and fine. In particular, I was really inspired by this entry in the always-fun series, Uses This: Katherine Cox-Buday’s work setup. And the fact that a bunch of my work colleagues used Linux at work. And… a lot of things!

In fact, here’s my motivation list.

I have been getting more and more sick of planned obsolescence. In an age of climate change and environmental degradation, spurred by a ravenously materialistic culture, it feels especially stupid to make something as dazzling and hard as a COMPUTER - with fancy rare Earth parts and feats of engineering and logic - only to have it be a 2-year social media browser. My MacBook 2010 chugged along well enough until 2015. My 2015 MacBook has been declining slowly for the last year or two. My 2017 work MacBook Pro died spectacularly after 2.5 years… because the battery fritzed. A battery that is soldered and therefore not replaceable. As an economist, I understand the model: you don’t make tons of money by selling people something they need every 10 years (water heaters!). You need to keep that consumption happening. Also, as a macroeconomist, you need that GDP growing growing growing. The 2019 Nobel laureates had something to say about that… Anyway, no no no.
Speaking of which, why can’t I just remove my dead battery? Or upgrade my RAM? Or just open it and learn about it and fiddle with it? Why is customization artificially locked away via glued and soldered parts?
I want to learn about computers! As my life and career has gotten more programmy, I’ve become more curious (and more empowered) about different aspects of the computer. My tldr understanding of Linux was: this is an OS that makes you work a bit more - there’s more learning curve - but you get rewarded with a deeper understanding of your machine. We’ll see indeed if this is true!
I could do everything I needed in Linux anyway. As an economist, I used Microsoft Word and Excel a lot. As a data scientist, I’m mostly working in Jupyter notebooks and vim now (glorious vim).
The Pi felt like a breakthrough. I don’t know why, but setting up my first Raspberry Pi also loosened me up from my “chained to MacBooks” feeling.

Moving from macOS to Pop!_OS
#

Much has changed! Oh, the many hours I have spent either (a) in a terminal, (b) accidentally right-clicking, and/or (c) pressing the wrong keyboard shortcut. Ay, it has been a little painful, a little joyous. I think things are a bit more settled now, after that initial shock. My report!

First, and most importantly: all the keyboard shortcuts are weird
#

Arrghh. Keyboard shortcuts. After 10 years, I am a wizard at text editing and dashing around with cmd this and 3-/4-finger gesture that. Pop!_OS dropped me into a new world where keyboard shortcuts that had been unified under a macOS’s cmd were now living under three new keys: ctrl, “Super” (a System76-specific special key), and alt. Aieee. My pinkyyyyy.

This has taken, of course, some MASSIVE getting used to, as well as onerous pecking and mousing/clicking in the earliest hours. I have since discovered the general rule that: (1) Super for “big stuff” (launching/quitting apps, moving across workspaces), ctrl for ye olde stuff (closing browser tabs, jumping forward/backward words), and alt as a wild card (jumping across browser tabs? reloading the previous or the future page?).

All a little unexpected and painful, and I’ve seen that Pop!_OS natively supports some vim movements for things. MAYBE I CAN MAKE IT ALL VIM!?

Replicating macOS trackpad gestures
#

Which brings us to the next big thing I do with my hands (“prehensile paws”): TRACKPAD STUFF. Namely, macOS’s glorious gestures - which I fell in love with in 2010 and, honestly, were another big reason I was hesitant to switch to Linux.

This was actually very easy. Almost trivial? I mean, I spent an hour or two researching and installing and so on, but - once I found what I wanted - it was really straightforward. And suddenly, that whole gesturey aspect was unlocked.

Basically, I’m using libinput-gestures, a (mostly-)Python library that interprets the touchpad gestures and maps them to shell commands. Out of the box, using the default configuration, it replicated the 3-gesture workspace swipe that I’ve grown so accustomed to.

RTFM
#

Actually, in installing libinput-gestures, I ran into a workflow that I’ve been using for years now and I just know is not serving me well. It serves me maybe 70% well. Basically, when I don’t know how to do something, especially something programmy, especially something about installing a package via the shell, I normally:

DuckDuckGo it.
Find the first answer that seems relevant.
Scan, without reading, for the first code block.
Copy+paste whatever is in that code block and see if that works.

Yes, I know. Hackers, here’s your entry point! Just stick a fake search result in a… Well, anyway.

So the worst thing about this is that it does work, like maybe 60% of the time. And so I’m being randomly dopamine-rewarded for a very bad process indeed. With the above, when I was installing libinput-gestures (which has very good documentation), I faced an easy problem and kept opening more and more search tabs, trying random things, shaving yaks randomly, before finally finding the solution… in the README. A solution that had been clearly written in both the project’s GitHub README and in the config file I was struggling with.

I’ve noticed this happening in other ways, but, really, I should just RTFM. It seems more “costly” up front, but I can likely spare myself a lot of browser tabs and hand-waving.

How could someone survive without the shell?
#

So far, I’ve spent many a happy hour installing, debugging, and generally fiddling with things in the Pop!_OS terminal. (Which is called Terminal?) This is fine and well, and I feel confident enough to do what I need to do. But what if the Terminal is strange to you? There do seem to be a lot of Linux GUI things floating around, and maybe they’re sufficient? But it seemed like everything I ran into (e.g. libinput-gestures) assumed some familiarity with the command line. I would have struggled much more in 2013. I do wonder about this. How do non-command line users experience and enjoy Linux? I’m so curious, I might investigate this and see if there’s any blog posts, etc.

Everything is a choice
#

This is both the blessing and curse. I didn’t like the way Pop!_OS’s default file browser looked and worked. I understood that, with Linux, I could change this. Oh man, and cue the Reddit threads and browsing of .debs and .flatpaks (what are these?) and all that. I ended up with Nemo, since it was vaguely pretty and immediately intuitive.

Speaking of pretty…
#

Default macOS is very pretty. So stylish. So Just Works-y. So confidence-building. Linux is definitely a little uglier, harsher, etc. I mean, I love the form factor (NEW WORD I LEARNED) of my beautiful, featherweight Lemur Pro. And the… KDE (?) - the windows, the dock (oh yeah, I installed and replicated the macOS dock) are certainly charmingly okay. But I will work on beautifying this machine.

In praise of exercism.io

Thu, 01 Apr 2021 00:00:00 +0000

I began “learning to code” in 2014, where by “learning to code”, I mean moving away from Stata (a proprietary statistical programming language) and into d3.js, JavaScript more generally, and, finally, glorious Python. You can read the whole journey here.

Anyway, back in 2014, I found a number of fun, puzzle-driven coding practice websites. Among these were:

Since getting off the ground with the basics, I’ve periodically returned to these - mostly LeetCode - but never for very long stretches. I just spent some time working through Exercism though and (a) it’s changed a lot, (b) in a really good way.

Why is it so nice?
#

I’m really enjoying it! It’s so nice! There’s a bunch of nice… nicenesses about it.

First, the aesthetic is inclusive and diverse. They make a point of showing a wide range of humans coding.
The command line interface is lightweight and friendly. You can easily download puzzles and submit solutions.
There are numerous language tracks available. I’ve signed up for Python, Clojure (good ol’ Clojure), Go, and bash.
The puzzles emphasize teaching the language and its idioms, rather than grinding CTCI-style riddles. I still think HackerRank and LeetCode might be quite good places to grind your understanding of algorithms and data structures (and thus high-level coding interviews), but Exercism is more about becoming a master craftsperson for its own sake. You get to know the language in a lower-pressure, friendly way.
The CLI, interacted with the different languages you’d like to learn (e.g. Clojure), means you can set up your own local environment (figuring out both how to do it, and what you like).
You see the tests. This is not the case with LeetCode, iirc, which hides the tests but lets you “unlock” different test cases for $$$. Oof. Exercism is free and you can, well, teach yourself to the test. This has been a nice way for me to think about edge cases (“oh yeah, good point, test file”) and address them.
After submitting your solution, you can seek a “Mentor’s” advice (I haven’t done this yet) and/or peruse other people’s solutions (I have done this). I’ve really enjoyed the latter because often you learn a nice idiomatic way to do something.
The “easy” puzzles are bite-sized snacks, good for when you’re waiting for a meeting to start…

Overall, highly recommend!
#

Great job, Exercism community!

Some shell magic

Wed, 31 Mar 2021 00:00:00 +0000

I just ran:

find * -type f -exec echo "{}" \; -exec awk "NR==26" {} \;

Which I adapted from this StackOverflow post and, tldr, searches all files in my current directory (and subdirectories) and tells me what’s written on line 26.

Or, parsing it out:

find: “Find files or directories under the given directory tree, recursively.” ( - tldr)
*: Anything in the current directory.
-type f: A file (not a directory).
-exec: A flag in find which will run a shell command. If you include {} in that shell command, it will replace it with the “found” filename (or directory).
echo: Print to stdout.
"{}": The current filename.
\;: End of that command.
-exec: Yo, you can chain these guys!
awk: “A versatile programming language for working on files.” ( - tldr) AKA, magic.
"NR==26": Magic syntax. Found this via StackOverflow. Line number 26.
{}: The current filename (still coming from the find command).
\;: The end.

It outputs something like:

file1.txt
README.md
some_subdirectory/file2.txt
some random text on line 26
some_subdirectory/file3.txt
.some_config_file
enabled: true

Cool!

dwebp

Thu, 25 Mar 2021 00:00:00 +0000

A small, short tutorial today: How to convert those darn .webp files into .jpg?

Motivation
#

If you, like me, periodically refresh your Zoom background to something fun and festive, you too may run into the dreaded new Internet image format: .webp. This is a Google-developed image format which is a bit more efficient in compressing images for the Web, with less quality loss. Unfortunately, while they load in a webpage fine, my Preview app was not recognizing them. What’s a saving-images-from-the-Internet-person to do?

Example and 2 okay solutions
#

Consider this image, from r/ZoomBackgrounds: image link

If you right-click and Save Image As..., your only option is to save it as a WebP Image. While this YouTube video shows how you can trim everything after jpg? from the URL, this may not work. If you try it on the above link, you’ll get a 403 Forbidden.

Another solution, which did work for me, but felt very 2003 “I have no idea how to computer”, was this dodgy website which lets you upload your .webp files and re-download them as .jpgs. I won’t bother linking, you can Google/DuckDuckGo that yourself. Since I have no idea what chicanery is happening behind the scenes of such a website.

Best solution: The command line!
#

DuckDuckGo-ing “webp jpg command line” delivered me unto the Google docs, and, lo, there is a simple, Homebrew-installable command line utility that will do the converting for you.

brew install webp
dwebp calvin.webp -o calvin.jpg

Voila! That worked. Much easier. Much more transparent-feeling. Note that the webp Homebrew formula downloads several utilities:

cwebp: For encoding from .jpg -> .webp. I guess if you’re a web developer?
dwebp: For decoding your Calvin & Hobbes backgrounds back to .jpg.

And other stuff, including one utility called webpmux, which the docs helpfully describe as a “WebP muxing tool”. What in the world is muxing? (Apparently combining multiple analog/digital signals into one. Still have no idea what that means.)

Endnotes
#

Here’s German 1920s-revival singer Max Raabe and German rapper Samy Deluxe singing one of my favorites, Der perfekte Moment. A true ode to laziness!

Happy zooming!

The Soul of a New Machine (2000) -

Mon, 22 Mar 2021 00:00:00 +0000

Metadata
#

Author(s): Tracy Kidder
Number of pages: 293
Year published: 2000
Year read: 2021

Review
#

Disappointingly Basic Tech Bro-ey. I thought this would be a bit more like “Halt and Catch Fire”. Oh well.

📚 Goodreads

Pi project

Mon, 08 Mar 2021 00:00:00 +0000

Motivation
#

It is winter. The air is dry. Beyond slathering my face in Burt’s Bees, we have mostly suffered in this dryness. I am anti-humidifier. They seem like diva equipment. And who wants to clean them weekly? Ugh. But no more! It is time. Also, my last Pi project broke important barriers in my brain - mostly emotional - and now it seems much easier to try new Pi things. Also also, I had another Pi lying around.

Hardware setup
#

For this project, I used:

a Raspberry Pi 3 - Model B
the Pimoroni Enviro pHat

Note: I was unable to use the Raspberry Pi 3b case that comes with this CanaKit kit - the Enviro pHat doesn’t fit under the case’s middle layer. Word to the wise!

Software setup
#

This was a bit of a journey.

First, I downloaded the latest version of the Raspberry Pi operating system - specifically, the Raspberry Pi OS with desktop. I flashed it onto the micro-SD card using balenaEtcher. After inserting that SD card into the Pi, as well as attaching all the peripherals (monitor, keyboard, mouse, power), I powered it on and then shaved the yak that is installing everything. At the end of this part of the journey, you have a perfectly serviceable little Linux desktop that I believe all children around 5 years old should have. Free Pis for all kids!

Second, well… I spent a lot of time being ridiculous and tinkering with the shell and dotfiles of the Pi. I just want a comfortable programming environment, dammit! This meant first copying over my .vimrc… and then just going whole hog and copying over all my .dotfiles, installing zsh (z-shell), installing ohmyzsh.sh, and generally futzing. The main learning here was that installing things on OSX means using brew a lot (e.g. brew install whatever), while the Raspberry Pi OS - given it’s a type of Debian system, like Ubuntu (the most popular (?) Linux OS) - used a lot of sudo apt get install. Other than that, this part was basically smooth sailing and not much to report.

Third was getting acquainted with the actual software for the Enviro. I relied heavily on this tutorial from the creators, Getting Started with Enviro+, not to be confused with this very similarly titled tutorial for their older model. HEHHHH. Yeah, I definitely got confused. And that’s the main thing I learned in this Pi project - that these projects are small and often changing and so it takes a bit of detective work to figure out how all the pieces go together. Which was fun! Very gratifying!

Anyway, the pieces I ended up needing were the enviroplus-python library (Pimoroni has, again, a very similar one, enviro-phat, which I won’t link because then you will get confused. I’m pretty sure enviroplus-python is the one we should be using, at least at this point in time.)

More hardware
#

So this library was super interesting. It basically installed a bunch of other Pimoroni Python packages which related to the sensors on the pHat (pHAT? HAT? read more here…). These sensors are, specifically:

A BME280 sensor which records temperature, humidity, and pressure. I believe the official docs are here.
An LTR-559 light and (optical) proximity sensor. I believe the official docs for that are these.

There’s also a tiny microphone and LED screen, though I haven’t explored those almost at all yet. More to come on those!

And more software
#

The main questions I had, after running through the enviroplus-python off-the-shelf examples, was what were the units on these sensors? The proximity sensoring, especially, was kinda baffling. The numbers get very “big” (in the thousands) if you put your hand over the sensor, and quite low (down to zero) if you don’t. So, thousands of… what?

It turned out finding the answer to this question was not straightforward, and the official answer was basically an “uhhhh 🤷”. This GitHub issue includes the Pimoroni thoughts on the matter and the official docs basically say it’s “count”. Though count of what?!

A brief digression on how the optical sensor works
#

I dove into this a little bit. The LTR-559 is an optical sensor. (There are other types of proximity sensors that use other parts of the electromagnetic spectrum to “see” if there’s an object in front of them.) An optical sensor works by using light - it measures the reflected light, as far as I can tell. So the numbers the sensor is returning are a “count” (according to the official docs), which the Pimoroni authors think is either the number of LED pulses the sensor emitted that returned an object (e.g. “I shot out 500 pulses and 499 of them reflected light back, so your number is 499”?) or the actual “number of light units”, typically measured as a lux (wiki - a measure of “luminous flux per unit area”).

Either way, big numbers from that sensor = lots of reflected light = something is close.

This YouTube video by Modern Robotics Inc gives a very nice and thorough (and digestible) explanation.

Final Software: the actual little script
#

To start with, I just wanted to start collecting the data for some of our rooms. For now, I’m just writing that data to a CSV file that the Pi is holding, and every so often I ssh into the Pi and check on it. I’ve been noodling over the idea of (1) making the data more stream-y (Kafka?) and/or (2) making a web UI (like the Pi-hole!) using Flask and Bokeh (Bokeh so pretty…).

#!/usr/bin/env python3

import time
import csv
from datetime import datetime

from ltr559 import LTR559
from bme280 import BME280

ltr = LTR559()
bme = BME280()

header = ['datetime', 'light', 'proximity', 'temperature', 'humidity']
data_file = 'data/room.csv'

try:

 # Write the header once
 with open(data_file, 'w') as f:
 writer = csv.writer(f)
 writer.writerow(header)

 while True:
 f = open(data_file, 'a')

 writer = csv.writer(f)

 today = datetime.now().strftime('%Y-%m-%d %H:%M:%S')

 ltr.update_sensor()
 lux = ltr.get_lux()
 prox = ltr.get_proximity()
 temp = bme.get_temperature()
 pres = bme.get_pressure()
 hum = bme.get_humidity()

 writer.writerow([today, f"{lux:.2f}", f"{prox:.2f}", f"{temp:.2f}", f"{hum:.2f}"])

 time.sleep(60)
 f.close()

except KeyboardInterrupt:
 pass

The units for the other sensor readings (apart from proximity) are:

LTR559 - Light: lux
BME280 - Temperature: Celsius
BME280 - Pressure: hectopascal (hPa)
BME280 - Humidity: relative humidity

Interpreting the results
#

My brain is in Fahrenheit, but I have a general sense of Celsius (20 is comfortable! 0 is freezing! 40 is hotttt). I have no similar sense for luxes, hectopascals, and even relative humidity. The relative humidity in the rooms we measured has always been around ~17%. Which seems awfully low. But what does that mean? Apparently the Sahara Desert is typically around 25%. What the heck.

Oh yeah, and I need to compensate for the heat of the Raspberry Pi, as this Pimoroni example shows.

Conclusion
#

Well, this was a lot of fun, despite what my initial Pi pessimism (“is it all just off the shelf?!”). I do feel like my creativity has been unlocked and I’m learning lots of little things as well (hectopascals!); those were the two things I was after (learning, creativity). I’m very pumped to keep tinkering with this and to BUY MORE PIIIIIIS.

Setting up the Pi-hole (finally!)

Sun, 28 Feb 2021 00:00:00 +0000

Finally getting my Pi-hole act together
#

I have been meaning to do something with the multiple (!) Raspberry Pis I’ve received over the years, and never got around to it. Until this weekend! Let us take a journey through the Pi-hole project.

The Raspberry Pi
#

Chances are if you’re reading this blog, you know what a Raspberry Pi is. But perhaps you don’t! In which case: it is a small, cheap ($30) barebones computer used by hobbyists for numerous silly ideas. You can plug in the various accessories - monitor, keyboard, mouse - load an operating system, and you have a computer. You can also plug in various sensors, cameras, microphones, and such, and create a wide array of random widgets and doo-dads: robots with computer vision and so on. There are many, many books about this.

The Pi-hole
#

One popular project is setting up a Raspberry Pi as a network-wide ad blocker. The Pi-hole project makes this very, very easy and friendly. The basic idea is: you route all Internet traffic on your local network through your Raspberry Pi. The Pi-hole software parses a giant blocklist (as well as an optional whitelist) of domains. If the Internet request contains one of those persona non grata domains, it returns a false result - sending it to a DNS sinkhole. In this way, you:

Spare a lot of your bandwidth from awful, pervasive ad traffic.
For all the devices on your network (computers, phones, TVs, game consoles).
And it provides a spiffy web UI for you to monitor (a) who’s browsing what, (b) what’s getting blocked and let through, and (c) it lets you configure - to a very granular level - which domains are allowed and which aren’t.

I have been very, very tempted to use a Pi-hole to sinkhole all social media domains. The Great Firewall of my house!! This has been vetoed, so far, by the other Internet users in my house.

Why not just use an ad-blocker
#

Since this was the main question I was asked by my other network users. The main reason is: an ad-blocker is device-based. Not even, it’s browser-based. So you have to keep installing it on every new browser, on every new device. Pi-hole solves the problem once and for all.

How I set it up
#

Networking is one of those things I really, really need to learn more about (like Bluetooth!), because otherwise - I swear - my ISP tries to scare me into renting their router or stuff like that. Anyway, so my super basic understanding is that, after setting up your Pi on your network (I wired it via ethernet), you assign it a static IP address and then redirect all traffic to that IP address. Specifically, I followed this tutorial: Quick and Easy Pi-hole Setup 2021.

More Pi-thoughts
#

If you visit websites like Adafruit and Pimoroni, you’ll see lots of fun, inspiring projects. I used to be very intimidated by this stuff, because I couldn’t understand how it all came together. The hardware learning curve was (and mostly is) very steep for me, and seemed daunting. Hence why it took me three years to finally getting around to this. Now that I’ve finally broken that first barrier - turning on the Pi, connecting to it headlessly, and making it my home’s Pi-hole - I have the feeling of: “wait, is it always just off-the-shelf stuff?”

Example: the Pimoroni Enviro pHAT, which is a little board that comes with a bunch of weather-related sensors (humidity, temperature, light, noise). You can connect it to your Pi and monitor your indoor ambience. But, reading through the tutorial, it seems to be basically (a) solder this all together, (b) curl the project and bash install it, and (c) voila all your sensors now do stuff! It feels like cheating. It feels so off-the-shelf! Is the only creative contribution I make the way I present the sensor data? That’s boring.

Anyway, so my Pi feelings are still in flux. I’m excited by the idea of DOING and MAKING and LEARNING stuff (my favorite things ever, after all). But I’m still scrambling a little at the base of this hardware maker mountain, trying to find projects that are both challenging but achievable (just like when I was trying to learn Python! ah, Zen mind, beginner’s mind).

Some further reading and resources
#

“Ad block and you don’t stop”, Normcore Tech
“Spam”, Finn Brunton. One of my favorite books of 2014!
The Raspberry Pi Foundation. They’re British!
Raspberry Pi Weekly, on Lynda

Carrie Nation triptych

Thu, 04 Feb 2021 00:00:00 +0000

Made this for an art class I’m taking. It’s a three-panel exploration of what motivates my character, Carrie Nation (based on the historical Carrie Nation, but with some important liberties taken). It relates specifically to plot points from my sci-fi short story, The Good Deaths, Part II.

Best books of 2020

Thu, 17 Dec 2020 00:00:00 +0000

Overall stats
#

This year, I read 84 books. The vast majority of these were picture books: either graphic novels or children’s books. At a glance, 29 (35%) were by POC, and 37 (44%) were by women. (This was just me guessing based on the author names and pictures, so it might be a little wrong.) So, no improvement from 2018 on POC (2018: 33%), and a very decent improvement on reading women authors (2018: 21%). I noticed that many of the children’s book authors were women, and many of the comix as well.

Going forward, I’d like to:

Again, re-commit myself to reading more by “minority” authors (that term feels outdated?).
I’m really enjoying historical fiction, so more of that.
I have read zero poetry in forever; something to explore.
More STEM.

Now to the awards!

Most influential: Usual Cruelty
#

Alec Karakatsanis’s Usual Cruelty, about the criminal “injustice” system, was incredibly enlightening and informative. It blew my mind. I gifted it to several folks for the holidays.

Most pleasurable in the moment: Bonfire of the Vanities
#

More criminal justice things: Bonfire of the Vanities, that peak of 80s excess. A blazing satire. So much cackling. We watched the movie in our book club as well and it was truly awful. Amazingly bad. Bad as a film. Awful as a (re?)interpretation of a good book.

Honorable mention to Pachinko as well.

Best comix: Good Talk
#

Good Talk by Mira Jacob was innovative aesthetically, with an authentic and poignant story of growing up the daughter of Indian immigrants.

Best kids’ book: TIE! Sheep Out to Eat and Cars and Trucks and Things That Go
#

Oh, this one is hard. But it must be a tie between Sheep Out to Eat and Cars and Trucks and Things That Go. The sheep are delightful for their art, their knitting, and the rhythm and rhyme of their disasters. Boy, do they get up to some hijinx. The things that go are delightful for the absolute busy-ness, the chaos (a pickle truck!), the celebration of going for a ride (in something), and for this review.

via GIPHY

Library Extension developers: Thank you!
#

This year I re-discovered the library, namely that they have physical books. I talk about the glories of the library in my anti-Amazon post. But, srsly, Library Extension has changed my life.

The post-pandemic world

Thu, 17 Dec 2020 00:00:00 +0000

People are getting vaccinated. The end feels near and tangible!

Things I’m looking forward to in a post-pandemic world
#

Wearing a mask on my commute (I don’t know why)
Cafe chatter and bustle
Movie theaters
Art museums!!!!
Running without a mask on
Curbside pickup continuing? Maybe?
Natural history museum!
Indoor gyms and play areas, etc

Things I’m not looking forward to
#

Travel
Commuting
Crowds
Reverse culture shock of “returning to normality”

Play-Doh self-portrait

Wed, 09 Dec 2020 00:00:00 +0000

Made some silly, amusing art. Self-portrait in acrylic yarn, Play-Doh, and a jumbo marker. Claymation is harder than it looks!

The Hard Tomorrow (2019) - ⭐⭐⭐⭐

Fri, 04 Dec 2020 00:00:00 +0000

Metadata
#

Author(s): Eleanor Davis
Number of pages: 152
Year published: 2019
Year read: 2020

Review
#

I read this through once, thinking it took place now. One plot point felt outlandish to me and then, reading other Goodreads reviews, I realized this is meant to take place in the very-near future: a sort of “one minute forward”, bleak Black Mirror-ish speculation. That fit the puzzle pieces together (specifically, that plot point I couldn’t quite believe), but what’s interesting is how I firmly believed every single other part of this - and just assumed it was a slice of life tale about being a 30something in this problematic landscape.

Really well-observed; beautiful structured. The art is amazing. The pacing and execution are really impeccable. I’m so impressed. Can’t wait to read more from Eleanor Davis.

📚 Goodreads

Vim!

Thu, 03 Dec 2020 00:00:00 +0000

seven, +/- two

Hello, world! :wq
#

This is coming to you live from a Vim terminal! Or is it a buffer? An insert mode thingie? Well, whatever it is, I am writing this in Vim.

Long have I sought to vanquish the infamous learning curve. Many a time have I tumbled back to the bottom, into the warm embrace of PyCharm, Sublime, Atom, literally anything else (except Emacs). Well, this year is the year. I am now halfway up (maybe?) the learning curve and enjoying it. I hereby proclaim I am not going back. No! Never! I have dotfiles to prove it!

What Vim?
#

I assume most people have a similarly horrible first impression with Vim. You get dropped into Vim sometimes, on servers and iirc on unconfigured git diffs, and then you flail around there, not understanding how to exit or scroll or where you even are (“Is this an editor? What is this?”). In other words, it is an unfriendly text editor (built into your shell? I think?).

Why Vim?
#

But, beyond supreme nerd cred (always a worthy goal in and of itself), why Vim? Here were my reasons:

Lightweight. Until now, I was using PyCharm for most of my development. It has pros and cons. Several pros: multiple cursors, jumping to definitions and usages of any function or variable, lots of gentle UX nudges of how bad your code is (e.g. grayed-out unused objects, auto-linting), and so on. It had three notable cons: it’s not free, it takes forever to load, and - most importantly - configuring it was not only a dotfile away.
Dotfile-able. Readers will remember I love dotfiles, and, indeed, most of them withstood the ultimate test: getting a new machine set up. My Macbook was basically dead (btw Macbooks die so early these days wtf), and I needed to quickly spin up a new one. And I did! It didn’t take long! Except, oddly and ominously, on PyCharm, which suddenly had a super weird keymap - rendering all my typing so painful, like trying to write with my non-dominant hand. No, no nooo.
Programmatical. I have been quite swayed by the promises of Vim letting you type “as fast as you think”. Thought and words/code! As one! United! Very exciting. Also, this blog post by thoughtbot really convinced me to take the plunge - tldr: they claim it’ll only be a week before you’re “as fast as before”, and that, thus, everything after the first week is just gains.

A few of my favorite Vim things, so far
#

Well, it’s been less than a week, but I do feel like I’m now about as productive as I was before (!!!). Which is already amazing. Also, Vim has been impressing me with lots of little wing-dings and doo-dads. Yes, I am still annoyed I can’t just mouse and scroll IN THE USUAL FASHION, as I have for lo these many many computing years, but maybe the programmaticalness will win me over completely?

Here are some nice things I’ve enjoyed so far:

I love being able to run any shell command from Vim, e.g., I could run !make html && make serve right now and I would see my blog on a local server, and I wouldn’t have to leave Vim. There. I just did it.
To make up for the lack of mouse, there are a great many ways to search and find things, with (seemingly) any and all levels of precison. If you know the line number, it’s just :[line number]. If you know the word, it’s just /[word]. If you know it’s somewhere down there in the code, you can try shift+} or ctrl+F or shift+G. Many options!
yank and put. I have been yanking (copying) and putting multiple lines, and it’s so easy: :[starting line number],[ending line number]y and :[line to move it to]pu.
CtrlP: aka, fuzzy search through files (like in Atom).
<leader> + d goto the definition (thanks to jedi-vim).

Resources
#

Learn Vim by thoughtbot/Upcase - This has been extremely helpful, especially their Big Tip to have a “magical number seven” living document cheat sheet to keep learning. My tiny whiteboard is perfect for that!
Vimcasts
OpenVim
The Vim wiki
VimAwesome

Why is it so hard to make it in America?

Fri, 06 Nov 2020 00:00:00 +0000

One of my favorite songs (and performances), ever, seems especially appropriate today/this week. Charles Bradley was such a phenom. He sadly passed away in 2017, so soon after starting to achieve the recognition his talent deserved.

Wizzywig- Portrait of a Serial Hacker (2012) - ⭐⭐⭐

Fri, 06 Nov 2020 00:00:00 +0000

Metadata
#

Author(s): Ed Piskor
Number of pages: 288
Year published: 2012
Year read: 2020

Review
#

An interesting, troubling book about very early hacking and phreaking.

I’m always interested in the hacking mindset - the protagonist, Kevin “Boingthump” Phenicle, is constantly trying to reverse engineer and find loopholes in the phone system and the Internet. He doesn’t have a clear goal beyond seeing what can be done. Along the way, he accomplishes amazing feats. In a way, it’s like pure creativity. He does it because he can, because he must. There’s a great moment when his friend, growing increasingly uncomfortable with the semi-legal nature of their activities, begs him to stop - Kevin/Boingthump replies that that’s like asking him to stop being him. To him, life is a puzzle. Our modern world is a puzzle. A game to be cracked. An ADDICTIVE game.

The feds are portrayed in all their bumbling incompetence and cruelty, shrieking about “cybercrime” they don’t understand. I immediately thought of Aaron Swartz.

I didn’t love the art - it was grotesque, vulgar, cynical. And I spent a lot of time wondering how much was true (Phenicle is a composite of several historical h4x0rs); it made me want to read a history to learn more.

📚 Goodreads

Breaking up with big tech- $AMZN

Thu, 22 Oct 2020 00:00:00 +0000

Scorecard
#

Facebook: deleted 2014
Twitter: deleted 2019
[] Amazon: actively trying to extricate myself
[] Google: dreaming of breaking up

Motivation
#

Jeff Bezos has way too much money, and I am tired of giving it to him
#

Jeff Bezos has $200 billion. The human brain boggles at such a number, so here’s a picture of it.

No one human being deserves this much money (I don’t care how fast your shipping is). It is just plain wrong. It is unhealthy to society. I’m also personally offended by what Bezos chooses to do with his money: a fat lot of nothing. A toy space company (Blue Origin). When Bill Gates and the Gates Foundation exist as counterfactuals, Bezos should really be ashamed. There’s no reason he could possibly need so much money and I am going to do my small part in not giving him any more of it.

Also, the way Amazon is run: the way the warehouse workers are exploited, the way its corporate workforce is exploited. All while Bezos keeps getting richer? No, thank you.

Putting the brakes on consumerism
#

Amazon fought to patent the one-click buy button because, above all else, they want to make you CONSUME CONSUME CONSUME. This mindless consumerism is destroying the planet (all that cardboard!). I don’t need all this stuff.

The anti-$AMZN lifestyle: How to do it
#

Toilet paper, etc.
#

For big box essentials, Target and Walmart have the same app game, the same super-fast delivery game, and the same prices. These three behemoths are fighting, and consumers are benefiting. Yay, markets? So for all those “oh shit, we’re out of TP” moments, it’s very easy to switch. Bonus: Target and Walmart also exist in the real world, so you can physically go to the store if you really, really need that thing today. (It’s called 1-Day Shipping, only you drive. Bonus: no cardboard boxes!)

Books
#

Ebooks and audiobooks
#

This is where you save a lot of money by not-Amazoning.

If you’re in the US: first, get a library card at your local library. If you’re not in the US, you can probably still get a US library card for either free or a moderate yearly fee (e.g. $50/year for a New York Public Library card).

Second, connect the Libby app to any and all libraries you can get a card with. This will be your library portal and your audiobook app.

Third, get the Library Extension browser extension. It’ll tell you when a book you’re looking at on Amazon or Goodreads is at your library.

Fourth, you’ll notice that many popular ebooks and audiobooks are checked out. Put these on hold, until you’ve maxed out your holds on all your cards. Then, and this is the 🔥pro tip🔥: “pause” all your holds for the max length possible (6 months). You’ll still move ahead in the line. Once you’re ready to actually borrow the audiobook/ebook, unpause your hold. You can probably find something to borrow while you wait for the really good stuff.

If you simply must pay for your audiobooks, Libro.fm competes with Amazon on prices ($15/month memberships) and their credits roll over (unlike Audible’s).

Physical books
#

This is where you might spend more or, if you’re creative, much less.

I advocate buying used books when at all possible. For this, you can see if local used bookstores near you have an online presence. I have a local used bookstore with an incredible catalog and decent online presence. There are also some thrift marketplaces like Mercari (for everything) or Biblio (for books). If you live in a hippie area, you may also have a Little Free Library near you.

If you want to buy a new book as a gift or splurge, then Bookshop.org is my go-to: you pay more than Amazon, but that money goes to a local independent bookseller of your choice.

Super rare physical books
#

Sometimes there is a magical book that you simply must have and it looks like Amazon is the only place that has it. I recently ran into this with Figuring Fibers, by Carolyn Yackel and sarah-marie belcastro. This is a pretty niche book (math + knitting!) and I didn’t see it in my usual places. I was about to resort to Amazon when, thanks to a little bit of Googling, I found that the American Mathematical Society was selling it - and for cheaper than Amazon! I can report that the AMS also shipped it super duper fast.

Basically, trying to avoid Amazon actually exposes you to this whole giant, thriving, diverse ecosystem of other markets - which are often cooler, cheaper, and more interesting! Who knew the AMS sold books! And so on.

Prime Video
#

Again, there’s tons of movies to watch through your library (Kanopy) and via Google’s massive movie archive on YouTube.

A great resource
#

Amazon alternatives A great, curated list of alternatives. I used this as my main jumping off point.

Automated collaboration

Wed, 21 Oct 2020 00:00:00 +0000

Hi. Gee whiz, do I like continuous integration (CI). Though, hmm, that wiki entry has a very different definition from what I mean. I mean CI as in automated testing of your code upon merging into a GitHub or GitLab repo.

Anyway, one of the coolest things I learned from working a little bit on open source was how to run a super-tight ship. Given that these big projects like pandas need to operate with a la carte contributions, a lot of the management of those contributions is automated: automatic testing, yes, but even fancier stuff, like automatic documentation, automatic coverage and super-opinionated formatting.

I just integrated a few of these tools into my own work, and I wanted to talk about them.

Automatic test coverage with `coverage`
#

Ned Batchelder has a great PyCon talk on testing in general: Getting started with testing (PyCon 2014). My main takeaway from this talk was respect the dignity of tests 🙇. They do not have to suck. They can be as beautiful and programmy as the rest of your code.

But anyway, Ned also has a great package, coverage, which can integrate with tools like pytest (beloved pytest) to tell you which functions - nay, even lines - of your code are missed by tests.

I used coverage recently to auto-fail any merge requests on a repo by putting this in my CI config file:

coverage run -m --source=my_cool_code/ pytest
coverage report --fail-under 95

This basically says: your code must have at least 95% test coverage to pass CI. I like this because I often add new features and then de-prioritize the tests for them - thinking, well, I’ll get to those eventually. No longer!

Automatic docstring coverage with `interrogate`
#

Lynn Root has a super cool beta project called interrogate that basically does the same thing as coverage, but for docstrings. Once again, you can easily integrate it into your CI with the following:

interrogate --quiet --fail-under 95 my_cool_code/

As above, I’m setting CI up to fail whenever docstring coverage falls under 95%. Yes, I’m strict! No more cryptic functions.

Automatic code formatting with `black`
#

black is a quite opinionated formatter that will, if run on your code, automatically re-format it according to both the PEP8 standard, as well as the opinions of its maintainers. It’s like me when someone asks me to read their email for the content, not copyediting. I CANNOT NOT COPYEDIT, OKAY. That is how black is.

Anyway, given how it’s tightly coupled with - and seemingly blessed by - the Python Software Foundation, I feel comfortable setting it as a CI requirement as well: all MRs must have “blackened” Python code to be accepted. One format = better readability. Hooray.

This is the line in my CI config:

black --check my_cool_code/

It’ll bounce back if any files fail not to be modified by black.

🎉 Summary 🎉
#

All this to say: now MRs into my project will be more onerous, because my CI has become quite strict indeed. But the marginal added work when merging will, I hope, reap great rewards in the long run - clean, robust code that fails less often, and is easier to read and understand! 😭

The lonely 3%

Fri, 16 Oct 2020 00:00:00 +0000

Hey, it’s October. Which means 2 things:

Hacktoberfest! [make 4 open source contributions, win t-shirt]
and Inktober! [draw in ink every day, buy t-shirt?]

Let’s talk about open source.

Some context
#

I’ve been revolutionizing my learning tools lately - discarding some (arrivederci, 🐝!), trying new ones - in an effort to get really super efficient in learning. I want to really maximize the value-add I get when I embark on new stuff now, since the intrinsic opportunity cost of my time is VERY high now. One of the big new ideas I had in terms of “how can I most efficiently get better and learn a lot?” was contributing to open source 🎉!

At the moment, 3% of open source contributions come from women. When women hide their genders while making GitHub pull requests (PRs), their code is more likely to be accepted than PRs made by men. So that’s interesting.

This makes the 16% proportion of Wiki contributors who are women seem, as Esther Duflo said in a similar comment about underrepresented groups in economics, positively numerous…

Contributing to open source: Pros
#

Learn from the big guns. Contributing to pandas, sklearn, and others was like temporarily joining some high-powered teams of smart people. It was definitely intimidating, and honestly the culture shock and environment setup took a lot of work in the beginning. But I learned so much about how projects can be set up, how Python can be written, and how teams can work efficiently.
Learning a la carte. The other nice thing about OS is that you can pick it up and put it down as and when you have time. My schedule is lumpy!
Build a public-facing portfolio (aka transparency and prestige).
See and be seen. If you contribute a lot, I presume you get more and more well-known with the core devs. So you’re basically networking.

Contributing to open source: Cons
#

I know you like to job, so I put job in your job so you could job while you job. As I said in the first pro, contributing to OS was like doing another job on top of my actual job. I was learning a lot, and the two jobs cross-pollinated for sure, but oof, I was pretty tired by the end.
Hostile work environment. OS has a bad rep - maybe one of the worst reps? - in tech for being unfriendly (to put it mildly). This can really limit contributions - you want to join projects that have good cultures, and it’s hard to read culture from the outside looking in (though looking at discussions on old PRs can help). Thankfully, both pandas and sklearn were making clear efforts to be friendly - or, at least, friendly enough. Super efficient, but not mean.

Greetings from the pandemic

Mon, 28 Sep 2020 00:00:00 +0000

May you live in interesting times / Willkommen in der Zukunft
#

Dark is a three-season German-language Netflix show that was produced and released between 2017 and 2020. It’s a sci-fi show that centralizes time travel, and all the various causal pickles you end up in when you could end up both Marty McFly senior and junior. It’s a lot of fun. The casting of characters at different ages is, mwah, amazing. A delight.

The show takes place in a bunch of different times. At one point in the show’s near future, a nuclear apocalypse happens. We spend a lot of plot time trying to avert this. Characterizations are defined by how they react to it. The show mostly asks: do we have free will? It mostly answers: not if we can’t change our desires. Blah blah.

Anyway. The show was filmed pre-covid, and now we’re all living in this very weird covid reality. And the show is a pre-covid artifact about some imagined terrible dystopia. Hm.

Anyway, hello 👋
#

So I really recommend Dark. It’s so fun. Maybe I’ll make a post about my Covid Lifestyle, and recommend all my other new hobbies.

Either way, I found my SSH keys for this blog (only took me a year!) and so new content is forthcoming. 🎉

Credits
#

The GIF on this post’s summary is by Lukesistegià:

Permanent Record (2019) - ⭐⭐⭐

Fri, 18 Oct 2019 00:00:00 +0000

Metadata
#

Author(s): Edward Snowden
Number of pages: 339
Year published: 2019
Year read: 2019

Review
#

Wow. Very interesting.

This was one of those (rare) books where I am just THIRSTING to read other people’s reviews - because I suspect my feelings are shared - but I kept myself from it, since I wanted to give an unvarnished reaction.

So, I’d give Ed Snowden the whistleblower 5 stars for his clarity of thought, strength of principle, and courage. He did something very important for our civil liberties, and for any hope we might have of averting a real-life 1984 dystopia. He pulled back the curtain on the panopticon, and we should always be grateful for that.

I’d also give the guts of his actual whistleblowing act - particularly since I remember watching it “live” in 2013 - 5 stars for sheer adrenaline. I remember when he made that Guardian video unmasking himself, and when he disappeared off the face of the planet: only to turn up stuck in Russia after a failed escape to Ecuador. High drama!

I give his writing… 3 stars?! Like it’s both unexpectedly good (whoa, he foreshadowed his own life story with a pop 1980s reference that ALSO contextualized his upbringing! bravo), unexpectedly cheesy (boy, can his prose get purple), and unexpectedly - and disappointingly - humble-braggy. I mean, this read like a Tom Clancy novel. It’s this very red-blooded, very American (‘murca!), very 90s, rah rah justice story that seems - like Tom Clancy novels! - to feel so morally basic, so straightforward (and even thin characterizations!). I mean, fair. In a way, it WAS morally basic. The post-9/11 bulk collection/NSA panopticon IS a monster. It just takes guts to point it out, and woke-ness to notice/care. But Snowden paints himself as a hero so thoroughly, humble-brags so much, that I found myself getting pretty annoyed.

It’s actually interesting to compare his authorial voice against the voice of Aaron Swartz, who held a lot of the same beliefs and was (I think?) a greater talent, but took a very different (and tragic) path. Aaron Swartz’s writings revealed a young man who really burned with an inner moral fire - he was almost like a prophet. It made his writings incredibly compelling to read, but also alienating. Snowden, on the other hand, sounds MUCH more down to Earth - but also kinda… vain?! And, like, 90s basic technogeek (fetishizing Japanese culture, for example)? I feel strange criticizing his voice, but his extended digressions on his all-American, all-patriot pedigree jarred me; as did his heroic portrayal of how he really wanted to go to a high risk conflict zone but was “punished” for his uppity intelligence by getting a cushy Geneva job; and just his general portrayal as a heroic, super-smart, righteous hero in the mold of Jack Ryan.

It’s actually funny because the meta of his “voice” - what it could (potentially) reveal about his character - reminded me very much of James Comey and John Dickerson, two other public figures who also first struck me as deeply, admirably ethical, and then that upstanding nobility was sort of complicated with a deeper feeling that they were also strangely vain about that same moral nobility? I remember when Trump called Comey a “showboat”, and how very outrageous that felt at the time, but then several months later, I was like, “you know, maybe he has a point”. Similarly, I love John Dickerson (who can’t!), but I’ve likewise noticed how cultivated that aura of righteous dude is; and how he must relish it!

Which is funny! Because, from what I can tell, Snowden, Comey and Dickerson ARE righteous dudes! They DO embody high ethical standards! I guess I just have complicated feelings when it becomes apparent that THEY hero-worship themselves a bit too. I am, of course, speculating (especially about Dickerson, who seems least “showboaty”).

Other writing stuff: Snowden’s early chapters on his upbringing in 1990s Maryland, as well as his descriptions of the early Internet, the Federal contracting industry, and the rise of social media, were his strongest. I highlighted several passages that were SO insightful (and I was going to give this book 5 stars!). His later chapters, and especially his attempt to weave his relationship with his girlfriend, Lindsay, throughout the rest of the book - felt strangely unbelievable?! Which is funny (this is nonfiction!), but the excerpts from her diaries just felt… very similar to his voice!

📚 Goodreads

Fun with git

Wed, 16 Oct 2019 00:00:00 +0000

Today, I was working through an old repo - trying to understand what the hell had happened during its history - when I thought, “I wish there was some way I could just watch the git commits, like a movie, like a ghostly async pair programming exercise.”

A quick google found me this: gource.io

After some faffery around installing from source, I’ll save you the trouble and tell you that:

brew install gource

worked fine (on my OSX).

And now… voila, some snippets from pandas:

You can jump around the timeline; you can pause time and explore. It seems like you can also click on people or files and follow them. One thing I don’t like is that jumping in time refreshes the screen - i.e. it doesn’t show you how big the project has become e.g. from 2009 to 2014. But there’s probably a way around that.

Very, very cool.

Now someone just needs to write a way to replay git commits in a fake pair programming environment.

Nobel Prize, Economics, 2019

Wed, 16 Oct 2019 00:00:00 +0000

Yesterday, the winners of the 2019 Nobel Prize in Economics were announced: Esther Duflo, Abhijit Banerjee, and Michael Kremer.

This was super exciting! I worked for Esther and Abhijit from 2009 to 2013; I never worked for Michael, though he was in a lot of seminars (and was often remarked on by us lowly minions for how both insightful and kind his comments always were). I like to say I was Esther’s lowly minion (Research Assistant) for a year, and then got promoted to chief minion (i.e. Research Manager, yay) for three more years. Those were salad days indeed!

Their work has had a transformative effect on international development; working with them was also formative and special. A good overview of their work is Poor Economics. Their organization - the Abdul Latif Jameel Poverty Action Lab (J-PAL) - also offers a MicroMasters on EdX, if you want to go more in depth. The World Bank had a good blog post on Monday about their impact on the field at large. A good overview of adjacent activities - the economics and development milieu in which their work can be situated - would be The Idealist (about Jeffrey Sachs, another top academic - who incidentally Esther RAed for long ago! - whose work has suffered in reputation lately), The Undoing Project (about the “founding fathers” of behavioral economics, which informs a lot of DufloBanerjeeKremer’s microeconomic views), and Inside Bill’s Brain on Netflix (the Gates Foundation is a big funder of J-PAL’s work).

And Esther is, as always, remarkable as an individual: she’s the second woman to win the Nobel (which should embarrass the Nobel deciderers, honestly) and the youngest winner too (!).

What wonderful news! Heartfelt congratulations to them!

Addendum
#

I wanted to add a very insightful Twitter thread that Nick Hagerty (a friend and former colleague, as well as former PhD advisee of Esther’s) wrote, which - I think - contextualizes just what is so remarkable about Esther, Abhijit, and Michael:

In earlier econ culture, certain great men held special wisdom that was inaccessible to others without their help.

Esther and Abhijit (+ plenty of others) not only pulled back this curtain but also showed everyone ways to start figuring out what we CAN know (however narrowly).
— Nick Hagerty (@hagertynw) October 14, 2019

I agree.

Thu, 03 Oct 2019 00:00:00 +0000

I’ve spent the past few days at #GHC19, aka the 2019 Grace Hopper Celebration in Orlando, FL. I had heard about this conference for several years, and this was my first year attending.

If someone says they can’t find enough women in tech to hire, encourage, promote, show them this picture of 25000 amazing women in tech and assure them that not all of them came to attend #GHC19 and there are more back home! 😉#womenintech #AnitaB #Equity #5050By2025 pic.twitter.com/YA77UD6oZq
— Meenu Datta (@Me_Datta) October 3, 2019

Thoughts and feelings
#

It is a mega-conference! 25,000 attendees! 😱
I felt like I was living in the last few panels of Y: The Last Man, i.e. a world of women.

Y: The Last Man, issue #60

It was super inspiring, more on the meta (yeah! ladies in tech!) than on specific, technical talks. Though I did wrap up the conference with a long list of TODOs: Julia, Go, Rust, Spark, maybe I should build that Clojure web app…
So apparently company t-shirts are a thing; there were gangs of roving Googlers, Amazon(ians?), Facebookers, etc. This was similar to the family vacation t-shirts I saw in the Disney parks at night: Ambroz Disney Family Vacation 2019! Part of me considers it tacky; most of me hungers for such a t-shirt…
Speaking of Disney, I completed Disney Tech’s coding challenge and won a t-shirt! 🎉
I also really enjoyed the challenge - where do I find more puzzles?!
In other t-shirt news, I made (and had accepted) two open source PRs - one on papers-we-love and one on pandas. This puts me at 50% towards Digital Ocean’s Hacktoberfest promise of:

make 4 PRs to open source projects
+
in October
+
be in the first 50k people to do this
=
get t-shirt

LET IT BE KNOWN THAT I WILL CODE FOR T-SHIRTS.

Some history
#

Grace Hopper (1906-1992) was…

“…an American computer scientist and United States Navy rear admiral. One of the first programmers of the Harvard Mark I computer, she was a pioneer of computer programming who invented one of the first linkers.”

wiki

She, along with Ada Lovelace (1815-1852), are kind of the matron saints of Women in Tech. They are eerrrrrywhere.

I have many thoughts and feelings about Women in Tech. The shortest distillation is: I think we’re at an interesting point in history right now, and I’m hopeful. I also love being a Woman in Tech. I love my job. I love (and sometimes dread the future of?!) my career.

We’re at the groundswell of a reaction against a weird, Mad Men/marketing-driven gendering of tech which started in the 80s and basically gatekept-with-a-padlock a generation of women (and thus a generation of minds) from an industry; right when it was getting so important, socio-economically (the dotcom bubble! the commercialization of the Internet! web 2.0! social media!).

So I’m glad that there are these big, grand, moneyed gestures that shout about diversity and inclusion from the convention center rooftops. Indeed! More women in tech! Keep women in tech! And I appreciate that there are lots of opportunities to increase the number of women in tech:

scholarships! *
communities!
- PyLadies *
- DjangoGirls
- RailsBridge *
- ClojureBridge *
- Girls Who Code
- Women Who Code *
- Systers!
codes of conduct for conferences and open source projects!

(I put an * next to the ones I economically benefited from - as in, I either got a job, got better at my job, or just got plain ol’ cash.)

And so on. I think that’s great. I also think there are powerful economic reasons - both from the labor supply and labor demand sides - that make Women in Tech a very worthwhile objective indeed. Efficiency! From the labor demand side, the need for software devs is expected to grow “much faster than average” - that is, 21% increase over the next ten years (as opposed to the average of 5%). From the labor supply side, I’m inspired by Claudia Goldin’s work noting that tech jobs - and any jobs that allow flexibility in how you get your work done - can close the wage gap.

Thanks
#

Speaking of moneyed gestures, thanks to

the 🐝 College of Computing at Georgia Tech 🐝
and my employer, Optoro

who provided me with the funds and the opportunity to attend this year. It was a real privilege!

Chief Yak Shaver

Thu, 05 Sep 2019 00:00:00 +0000

Here’s a bunch of completely random stuff I learned these past few days, thanks to a devilishly borked up bork I fell into while trying to do what I thought was an easy task:

Just set up CI with some super basic tests on a Rails app I’m writing.

So-called “easy task” ate up my whole week, as I shaved yaks in a variety of domains I have [at present] a very limited understanding of:

Ruby
Ruby environments
Ruby on Rails
Docker
Continuous integration testing, aka CI
Gitlab’s implementation of CI, via Docker

I am presuming that my specific problem will impact no one ever, and that its content will be of limited interest to most everyone, except maybe –

xkcd: wisdom of the ancients

But what I would like to take you on is my, ahem, emotional journey here, people, because it really embodied one of the things I adore - and sometimes despair at - in tech: tenacity, grit, self-teaching, focus. AKA shaving the yak.

yak shaving (uncountable) - Noun
#

Any apparently useless activity which, by allowing you to overcome intermediate difficulties, allows you to solve a larger problem.

I was doing a bit of yak shaving this morning, and it looks like it might have paid off.

wiktionary’s entry on yak shaving

The thing about shaving yaks is that, crucially, it feels useless while you’re doing it. You’re banging your head against a wall, there are perhaps imperceptible cracks in the wall, you have a thousand StackOverflow tabs open in your browser, and you generally feel like you are WASTING YOUR DAYS. At least, that’s how I felt.

But I did break through the wall!

I mean, it’s still all broken, but now the pieces are comprehensible and I’ll just spend tomorrow fixing them, one by one. But the main, confusing, confounding, uber-borked, non-deterministic-computing-makes-no-sense yak has been shaved. I Fixed The Thing.

So let’s talk, first, briefly, about what the thing was and the fix.

Then, let us speak of the journey.

The thing that I wanted to do
#

Context:

I have a very simple, very basic Rails app.
It is on GitLab.
It has 2 very simple, very basic tests.
I ran the rspec testing framework locally. It was fine.
I thought I should use GitLab’s fancy continuous integration uh… integration and add a .gitlab-ci.yml file which basically said: hey GitLab, gimme Ruby version 2.5.1, bundle install from my Gemfile.lock, and let’s run that basic rspec, eh?

Reader, it borked on step 2 (bundle install).

The thing that stopped me
#

The way it borked was dastardly and weird. Basically,

$ gem install bundler -v $(grep -A 1 "^BUNDLED\ WITH$" Gemfile.lock | awk 'NR == 2 { print $1 }')
Successfully installed bundler-2.0.1
1 gem installed
$ gem list bundler
bundler (2.0.1, 1.16.6, default: 1.16.2)
$ ruby -v
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
$ bundle _2.0.1_ install
/usr/local/lib/ruby/site_ruby/2.5.0/rubygems.rb:289:in `find_spec_for_exe': Could not find 'bundler' (1.16.6) required by `$BUNDLER_VERSION`. (Gem::GemNotFoundException)
To update to the lastest version installed on your system, run `bundle update --bundler`.
To install the missing version, run `gem install bundler:1.16.6`
 from /usr/local/lib/ruby/site_ruby/2.5.0/rubygems.rb:308:in `activate_bin_path'
 from /usr/local/bundle/bin/bundle:23:in `<main>'

So many questions here. SO MANY.

Main question: Why does it complain about bundler 1.16.6 when I (a) specifically installed version 2.0.1 (gem install blah), and (b) specifically used that version too (bundle _2.0.1_ install)? (FWIW, when I did plain bundle install, it said I needed to use Bundler >2. 🤦‍♀️)
Why is my site_ruby filepath pointing to 2.5.0 when I specifically pulled the 2.5.1 Ruby from Docker hub? And it says I’m using 2.5.1 when I check ruby -v?
Why does gem list bundler tell me I have (a) a default version of the gem (what does this mean?) and (b) that 1.16.6, the thing it later complains about, is also installed????

I was tearing my hair out here. I was googling like mad. I was facing what can only be described as a deep Zen koan-like test of my patience and resolve and sanity. I suspected something was going on with Ruby environments (that 2.5.0 vs. 2.5.1 thing? those Bundler versions?), I wanted to access the Docker container directly but - dash! - I couldn’t figure out a way, because I was using the (incidentally normally very useful) gitlab-runner command line tool and what the heck does that do anyway!?! It definitely spins up a Docker container but then, well, kills it as soon as it’s done. Dash and blast!

The stuff I learned along the way to fixing this
#

Here’s some useful TILs from this terrible ordeal:

gitlab-runner is a very useful way to not spam your GitLab commit histories, i.e. not embarrass yourself in front of your colleagues.
Interactive web terminals are a way to pause and shell into the online CI. Useful! Not so useful locally, but oh well…
You cannot unset and reset the default bundler gem. It is forbidden. Why!? How?!
Bundler is kinda broken in one specific version of the Docker Ruby image. Aw, come on, man.
You can replicate the above error (i.e. the one from my error log above, and mentioned in the GitHub issue in #4) by doing:

❯ docker pull ruby:2.5.1
❯ docker run -it --rm ruby:2.5.1 bash
root@7b17219881e3:/# bundle install
Could not locate Gemfile
root@7b17219881e3:/# bundle -v
Bundler version 1.16.6

So here I had a solution! That Docker ruby image wanted bundler 1.16.6, it neeeeded it, but the Gemfile.lock was insisting on 2.0.1. Fine. Let’s change the Gemfile.lock file.

You can change the Bundler version needed to install your app - a version which is specified in the Gemfile.lock file (created by whatever version of Bundler you have locally) - by (a) removing the version number after BUNDLED WITH (but keeping the BUNDLED WITH!) and (b) running bundle _version-you-want_ install. Thanks, aloucaslabs post! I changed this to 1.16.6 and - voila! - everyone was happy. Docker was happy. Bundler was happy. And Rails was happy. (Well, Rails was mostly happy.)

The journey
#

So this sort of thing - falling into a bork hole - happens once every, oh, month or two? Maybe more infrequently. It happens when your service has a weird bug-out, some unexpected behavior. Or when your data looks weird. Or when, as this time, you’re building something new and have the hubris to think that “oh this’ll just take a minute…”

And, boy, it is an emotional journey. Namely, it’s an exercise in –

– yes, TRUE GRIT.

Many a moment did I despair and think, “oh let’s just forget setting this up, who will know” and many an excuse did I find, “oh i have more important DATA TO SCIENCE, what am i spending all these days on this for”.

But I reminded myself of previous yak shaving expeditions that were similarly dire and similarly resolved, and how indeed sweet that victory was. And how much was learned! So I persevered. And also, of course, another nice thing that programming does to you is make you addicted to puzzling things out - as one part of me despaired and found reasons to give up, the other part just wanted to frickin’ figure it out. AND I DID. And that boost is addictive. So: hooray!

Recursive SQL queries

Thu, 13 Jun 2019 00:00:00 +0000

Did you know that SQL queries can be recusive? I didn’t! Now I do. It’s glorious.

The problem
#

I had some data in the following format:

thing	sub_thing_id	start_time	end_time
1	1	Sep 1	Sep 3
1	2	Sep 1	Sep 7

And I wanted it in the following format:

thing	date	number_of_sub_things
1	Sep 1	2
1	Sep 2	2
1	Sep 3	2
1	Sep 4	1
1	Sep 5	1
1	Sep 6	1
1	Sep 7	1

How do?!

The solution: A recursive common table expression
#

Ah ha!

A recursive CTE allows you to join all the levels of a hierarchy without knowing in advance how many levels there are.

Snowflake docs

tl;dr, the query that worked was:

select
 exp.thing,
 to_date(exp.start_time) as date,
 count(*) as number_of_sub_things
 from
 (with cte
 as (select
 thing,
 sub_thing,
 start_time,
 end_time
 from some_wide_table where start_time is not null
 union all
 select thing, sub_thing,
 dateadd(day, 1, start_time),
 end_time
 from cte
 where to_date(start_time) <= (to_date(end_time)-1))
 select *
 from cte) as exp
group by 1, 2
order by dte asc;

But whaaat is happening here? I shall explain:

First, create a recursive common table expression (CTE): (with cte ... from cte) as exp
In that CTE, create your “anchor clause” (select thing ... from some_wide_table) and union all it to your “recursive clause” (select thing ... from cte). Your anchor clause is the first, base set of rows you will be comparing your next select statement against. Your recursive clause adds its resulting rows back to that base, and then compares that new “anchor” against the same “recursive clause” select statement.
It keeps doing this until the where to_date(start_time) <= (to_date(end_time)-1)) statement is no longer satisfied. So it’s (kinda) recuuuursive! Magic! And, like all recursive functions, there is a risk that you fall into an infinite loop if your where clause never catches.
Once you’ve completed recursing around, you select * from cte - this is selecting all the resulting rows that were found as the anchor and recursive clauses did their dance.
I then needed to group by the top-level thing and count(*) how many sub-things were present on each date.

Voila!

The next problem
#

So I actually ended up not being able to use this, because I wanted to do this arbitrary recursion over thousands of dates, millions of things, and tens of millions of sub-things. I quickly got to a max iteration count reached error from the database; a safety mechanism it has in place to prevent infinite recursions. It wasn’t infinite, but it was certainly a bit of a combinatorial explosion.

So what do now?!

tl;dr: There was a much more efficient, non-recursive way to do this. Basically, make a table of dates on the fly and join against that. D’oh.

select
 exp.thing,
 to_date(exp.this_date) as dte,
 count(*) as number_of_sub_things
 from
 (select
 x.thing,
 x.sub_thing,
 dates.this_date
 from some_wide_table as x
 right join
 (select
 dateadd(day, seq4(), '2011-01-01') as this_date
 from table(generator(rowcount => 9*365))) as dates
 on to_date(x.start_timee) <= to_date(dates.this_date) and to_date(x.end_time) >= to_date(dates.this_date)
 ) as exp
 group by 1, 2
;

I don’t love this solution, namely because I need to decide (and hard-code) the range of dates to compare against. For now, I’m creating a list of dates from 2011-01-01 to 9 years hence (9*365). It feels hacky. But it took a fraction of the time and no combinatorial problems! So it’s good enough.

It's amazing what you can get done

Mon, 10 Jun 2019 00:00:00 +0000

Recently, I realized I was doing the same thing three different ways. Namely, I was trying to practice mindfulness - and, while I was using different strategies (1) for work, (2) for home, and (3) for “personal development”, it was all the same thing. Here’s some stuff about my work-related mindfulness strategies.

The problem: work in the Age of Distraction
#

What with Slack and slow SQL queries, there has been endless opportunity for me to be distracted - and my brain has generally turned into a big, thin mushy goo, as I have flitted from task to task, nudging everything forward but not thinking deeply about any of it.

The solution: Tomato time + turning everything off
#

After reading Cal Newport’s Deep Work and shaking my Spear of Social Justice at it, I decided to implement the basic idea: focused work. I downloaded a pomodoro timer (after briefly debating buying an actual, vintage, Italian tomato-shaped kitchen timer because AESTHETICS) and got in the habit of closing everything (Slack, email, phone, random browser tabs) during those 25-minute chunks.

Reader, I got so much more work done.

And better work! Suddenly, using my ACTUAL BRAIN and giving tasks my full attention, solutions would pop and I had all sorts of ideas and work just generally felt like a calm, rich, rewarding activity. So that was great. A+.

The challenges: but [my SQL query | Docker | grid searching] is so sloooow
#

The big issue I have is that some of my work tasks, by their very nature, occupy my computer for a very long time. They are computationally slow.

For example: a big, complex SQL query on millions of rows. For example: a machine learning grid search. What to do? What if I’m in the dead middle of a tomato-time when some computationally-intensive task kicks off, and I’m just left staring at my screen?

Sometimes, when my will has been strong and my intentions extra-pure, I’ve allowed the dragging seconds of boredom to wash clean my mind. Sometimes this has led to some “ah ha!” moments - e.g. that query could be made much more efficient! there’s probably a better way to do this! That’s excellent. But it takes extra-strong willpower.

Otherwise, I nudge forward on some work-related book (e.g. Statistics Done Wrong). Something to keep things quiet, and not scattered (i.e. not checking my phone).

OMSCS Course review- Knowledge-based Artificial Intelligence

Mon, 07 Jan 2019 00:00:00 +0000

First OMSCS course is complete!

Disclaimer: This review applies to the Fall 2018 instance of CS7367: KBAI.

tldr
#

So, tldr, I thought this course was fine.

I didn’t know much about the topic - knowledge-based AI systems - before signing up, but it sounded intriguing, like maybe it would combine some sort of sublimated Oliver Sacksian neuroscience wonder with programming fun.

It was sorta like that, but I honestly ended up finding the methods of KBAI frustratingly vague. I mean, people get grumpy about the inscrutability of neural nets, and AI modeled on human-like cognitive systems certainly has a contrasting appeal. But coding such systems? Especially when we don’t have a full understanding of how human cognition works? Oooof.

The structure of the course was great, though I felt an increasing disconnect between the project side of things vs. the everything else side of things. That said, it was a good, gentle intro to the OMSCS program overall.

Structure
#

Twenty-six lectures on Udacity, each about 45-90 minutes long.
Three homeworks, which covered at least one lecture topic as well as some open-ended, creative stuff about, “hey, so how about that crazy AI, eh?!”. If you hate writing, you will be sad. If you love writing, these are awesome!
Three exams, which cover the lecture material and I found very ugghh. These were timed, proctored using an Orwellian webcam auto-proctor (heh), and generally relied on tricksy wordings of concepts as opposed to creative applications of concepts. I liked the exams the least, and felt they contributed to my learning the least. (That said, one of the co-instructors, Dr. David Joyner, released some very cool post-exam analyses after each one, which seems to indicate I may have been an outlier?)
One zany project. This was the heart and meat of the course for me; it was the place I learned the most. Basically, we had to build - from scratch (!), and in the language of our choosing (either Python or Java) - an AI agent that could parse visual intelligence tests of the following kind:

If that sounds daunting, oh man, you bet it was! The project was an emotional roller coaster of “omg how do i even” and “OMG IT’S WORKING IT’S WORKING”.

The project work felt very distinct - even disconnected - from what the lectures were telling me. I basically had to be resourceful and self-teach stuff like (a) how do you parse images in Python (we were allowed to use the Pillow library - a basic visual parsing library - but not the more advanced, computer visiony OpenCV), (b) once you “see” the image, how do you compare images, and (c) how do you develop generalized logic to compare arbitrary transitions and patterns.

It was madness!

And a lot of fun. My guesstimate is that the class - which had ~400 people in it - had a long-tail distribution of a few people who put many, many hours into developing near-perfect agents and a lot of people who put in a decent number of hours to get a decent performance. I was more in the fat part of the distribution (yo, I got a life to live here); my agent performed well-enough (it crossed each increasingly difficult hurdle as we faced harder and harder visual IQ tests, but it often just barely crossed), and I did a fair amount of love/hate refactoring throughout.

I do think I missed out on some “ah ha!” moments when I could truly blend the lectures’ vague-feeling “let’s model incremental concept learning!” with my AI agent (which mostly used heuristics). But I just didn’t have the time to make that leap into KBAI nirvana. (And I’m guesstimating I put about ~20 hours total into the agent?)

Effort
#

Which brings me to effort! So this was less effort than any of the Harvard Extension School classes that I’ve taken. And thank God for that, because those classes had me doing ~20 hours/week of problem sets and, man, that shit is unsustainable. (And yes, I did learn a lot, but I do think there were marginal decreasing returns to some of those p-sets.)

I’d guesstimate that I spent, on average, about 7-10 hours/week on this course, and I wasn’t super disciplined or time-efficient. I watched the lectures at about 1.25x - 1.5x speed, which was normally fine. I read a little bit of Winston. I was a speed-demon on the exams and homeworks, and I spent the bulk of my time agonizing over the project. This was fine. Again, I probably could have invested more and gotten more out of it, but - well - the thing I love about the OMSCS overall is its flexibility.

Would I recommend this?
#

YES for first semester OMSCS students. Dr. Joyner is a great edtech educator, and he seems very dedicated to running these courses well. I think any of his courses - like this, CS 6750: Human-Computer Interaction, or CS 6460: Educational Technology - is a great “intro” to the OMSCS overall. I want to actually take his other courses just because I’m so confident they’ll be so well-run.
NO for people interesting in machine learning, since it felt like this type of AI isn’t really practiced anymore. Sometimes it’s valuable to see a contrasting methodology, even if you never use it, but I found much of the lecture material to be so far in the camp of “I will definitely never do it this way” that it got pretty existentially doubtful. Like, why? WHYYYYY?

Up next?
#

I’m pumped for a bunch of courses. Oddly, the ~~Digital Wolf of Wall Street~~ Machine Learning for Trading has me pretty pumped.

Best books of 2018

Mon, 31 Dec 2018 00:00:00 +0000

Yay! It’s my favorite time of year: the let's reflect time of year! The best books of [year] time of year! YAAAAY.

Overall stats
#

Alright. This year, I read 73 books. Of these, 24 (33%) were by women and 15 (21%) by writers of color. Pretty bad stats, given the actual proportions of who’s writing stuff in English. Once again, I disproportionately read the work of white dudes.

In terms of how I decided what to read, I mostly followed my nose; where by “following my nose”, I mean that it was a combination of what the algorithms and the zeitgeist fed me, as mediated by my long-term reading goals (weighted low) and my short-term mood when selecting what to read next (weighted very high). Maybe I’ll be more intentional about reading next year.

Most influential
#

I read a handful of books early in 2018 that I basically spent the rest of year citing and referencing as much as possible. In all these cases, I was surprised I had read them this year, since they had become so completely absorbed in my thinking - they felt like foundational texts from long ago. These were:

Jonathan Haidt’s The Righteous Mind, which Explains Everything About Politics. Politics are very on my mind these days, thanks to the 2016 election and the hyper-polarization we seem to be experiencing in the US, Italy, Germany, UK, Tanzania, India… okay, everywhere the light touches, Simba. Haidt’s book is a great combination of rigorous experimental evidence and social psychology to explain why us hairless apes get so funky and intense about politics (and other tribal affiliations). I really enjoyed his takeaway of the “five moral tastes”, and how left-leaning Western humans have lost the ability to taste all five. I can’t make myself taste authority! AND YOU CAN’T MAKE ME.
Tony Judt’s Postwar. I didn’t think I needed a postwar Europe explainer. Now I think EVERYONE NEEDS IT AND EVERYONE NEEDS THIS ONE. This reframed my understanding of World War 2 and many European countries’ histories (especially Austria, France, the Netherlands). It provided useful kick-in-the-butts around certain well-known periods as 1968 and the Thatcher Years. It was dispiriting. It destroyed myths. It was so good.
Tim Wu’s The Attention Merchants. A wonderful explainer of the driving economic forces behind our attention economy and, as Bruce Schneier calls it, surveillance capitalism. The roots go deep. Delete your Facebook, for the love of God, it’s the cigarettes of the modern era.
Charles Mann’s 1491. I put Charles Mann and James Scott in the same mental bookshelf: they write God’s-eye-view sweeping, hyper-macro interpretations of history and politics. They’ve both made me totally reframe my understanding of big-deal things like the state (via James Scott’s Seeing Like a State) and, in 1491’s case, the New World. I learned so much about the pre-Colombus Americas via this book. I was inspired about specific indigenous cultures like the Inca. It gave me a strong pan-Americas fellow feeling. But, even more grandly, it made me think big picture about, well, the rise and fall of civilizations - and how much human innovation can be lost at the end of a gun, germs, and steel.
Emily Oster’s Expecting Better AKA how to be pregnant. Ho ho jk jk. I kid. Sort of. TL;DR: Be an economist about it! Oster uses statistical know-how, combined with the decision making framework we learn in Economics School, to talk about the inherent risk and uncertainty of pregnancy. I referred to it constantly, and referred all my friends of childbearing age to it as well. My citation rate on this book was very high.

Most pleasurable in the moment
#

Then there were the books that, while they didn’t indelibly tattoo my DNA, were just a whole lotta fun to read. These were the page-turners, the stay-up-late-ers.

Paul Kalanithi’s When Breath Becomes Air. Weird to call this a page-turner, but this intelligent, sparkling memoir of a 30something neurosurgeon with late stage lung cancer is. You will probably sob at the end. But you will also feel spiritually refreshed.
Gore Vidal’s Julian. A dark, funny drama about early Christianity vs. late stage Rome, as embodied in the idealistic, handsome and short-lived Emperor Julian. Snarky and amusing, and also deeply tactile. You live and breathe the 4th century AD! (And yes, I put Anno Domine instead of Common Era as a nod to the book’s sad conclusion - Christian Dark Ages win, after all.)
Angie Thomas’s The Hate U Give. Great YA, moving and powerful and refreshing. The protagonist is so sensible and lovable, you just wish you had a friend like this when you were a teen.
Dexter Palmer’s Version Control. After such a disappointing slew of mediocre mainstream SF in the last couple years, FINALLY a good one! This time travel story was layered and delightful. I sometimes cite it for its insightful acumen regarding dating apps (their incentive is not to get you into a relationship, but to keep you on the app - and they do this by leveraging a false sense of abundance and decision fatigue!).
And, my unplanned-for “tech culture is so crazy!” books, Emily Chang’s Brotopia (aaah sexism) and Dan Lyons’s Disrupted (aaah Millennial chaos).

Most hated
#

There were a few books that I found pretty lame - that Shashi Kapoor biography was a huge missed opportunity, and the Sex Criminals series went off the rails. But the absolute worst book I read, by far, was Light Years. My goodness. That was a book club book - and, I mean, I love that book club makes me read outside of my comfort zone - but this was so far out of my comfort zone as to be in hell. There were so many layers of horribleness - from the racist caricatures, the “male gaze” POV and mansplainy sexism, the moral vacuousness of the main characters, and the pretentiousness of the writing style - OOOOOOF.

Sad little dharma nook
#

I’m supposed to always have a dharma book in circulation, after a promise to myself made many years ago that I would keep the dharma fires alive in my heart forever. I’ve always interpreted the Three Jewels of Buddhism as a three-legged stool of community (sangha), practice (buddha), and study (dharma). And study usually meant reading a book or maybe a video or podcast.

Anyway, this year I kept a low simmer of dharma books, namely via Stephen Batchelor’s After Buddhism (basically boring) and Robert Wright’s Why Buddhism is True (much better). The New Yorker had a great essay/review of both books which both accurately captures the history of American Buddhism, all its inherent tensions (karma?! ladies kill the dharma?!!), and how these two books are situated within and address that tension. It’s good! Honestly, it’s also probably sufficient.

Yes, so 2 books and 1 essay. I told you dharma was sad this year.

Ursula Le Guin died
#

Also sad, of course, was the loss of Ursula K. Le Guin - AKA my guiding light of speculative fiction. I used to say that Le Guin and Kim Stanley Robinson were battling it out for top SF author spot in my heart, but - after having read more KSR and more Le Guin - I can safely say Le Guin is my number one. She wrote so many masterpieces. Even her short stories are startlingly good (the one about the crazy sex ratio and the crazy society it engenders!). Also her super interesting Taoism stuff!

Anyway, this year, in memory of her, I re-read Left Hand of Darkness (which, with The Dispossessed, I consider one of her magnum opuses and maybe one of the top books of SF ever!?) and also started her YA fantasy series, The Wizard of Earthsea.

2019 goals
#

Well, I really should address a few things.

Better representation. English is a big language, written by a bunch of people. I really should make a better effort of reading outside the white dude demographic.
More science fiction. I am always delighted when I find a new SF book to love, but I never make time for it! The Gollancz SF masterworks has been a pretty reliable source of finding incredible, unknown-to-me, underrated-by-everyone masterpieces (Gateway! Random Acts of Senseless Violence! Floating Worlds!).
Branching out into historical fiction. I got super pumped after reading Julian, and kind of remembered that I used to love history? And the ancients? Big time. I pretty reliably enjoy this stuff, but never, ever think of it as being “my thing”. Well! I hereby declare that historical fiction is also my thing! I’ve got Bring Up the Bodies and Creation somewhere around here, and the author of Floating Worlds, Cecelia Holland, was apparently way big in historical fiction before she wrote her crazy awesome sf?

I also discovered r/52book this year, which is just fun preening about “oh look how much I read!” and a nice place to get pumped/inspired by fellow bookworms. Onwards!

Disrupted- My Misadventure in the Start-Up Bubble (2016) - ⭐⭐⭐⭐

Fri, 23 Nov 2018 00:00:00 +0000

Metadata
#

Author(s): Dan Lyons
Number of pages: 272
Year published: 2016
Year read: 2018

Review
#

Honestly, I enjoyed this at a 5-star level, but I’m trying to avoid hyperbole here.

A scathing memoir by an old school journalist trying to make it in the brave new world of young tech startups, this book was a hoot. The book is divided into three sections: first, after Lyons is laid off from his old boys Newsweek job, he joins a broey Boston tech startup, HubSpot, and is variously amused and horrified by the Millennial antics there. Second, Lyons explores the economics of what he believes is the second tech bubble (this was my favorite section by far). Third, Lyons describes the Kafkaesque corporate bullying that can happen when one is, ahem, “managed out”.

I picked this up after hearing a talk by Lyons which variously impressed and amused me - even if we could argue about whether tech is in a speculatory bubble right now, I think we can all agree that the CULTURE of the tech industry deserves to be popped. I would shelve this book right next to Brotopia, in that both offer painfully astute insights into how tech - dominated, for the last ~15 years, by 20something white dudes and fashioning itself a “brave new world” that “disrupts the future” - is actually just replicating a lot of old social hierarchies, coupled with fierce agism. Lyons is scorching in this regard: he calls HubSpot’s “kooky office culture” an “adult kindergarten”, and ceaselessly (and hilariously) mocks the coddling perks for 20somethings. He also notes, horrifyingly, how alienated he is made to feel because he, for example, would rather go home to the wife and kids than do beer bongs with the bros on Friday night. He constantly notes how “lily white”, “pure as driven snow” white the whole company is, and the incredible chasms between the values it preaches (radical transparency, e.g.) and what it actually practices (dystopian “disappearing” of people - e.g. employees who are suddenly fired are rebranded as having “graduated for their next awesome adventure”).

That the tech industry has - on average - huge problems with age, sex/gender, and race is well-known. That many companies’s cultures are frozen-in-time replications of an extended adolescence - frat houses with a paycheck - is also well-known. For the most part, I enjoyed Lyons’s seemingly accurate take-down of what seems to be an extreme case of a general trend. I’ve worked at tech startups that had some of these Millennial-minded perks (free beer on tap, e.g.) but also were a little more enlightened (our meeting rooms were all named after prominent women in computer science, aww). I do think this book is worthwhile reading for people that work in these industries, if only to show what NOT to do. Indeed - in a way - the speculation and money washing around the tech industry means that a big spotlight is on their culture; I would LOVE to see similarly scathing satires written about, e.g., academia and the development industry (The Idealist is the only one I can think of). No industry is immune from madness of one type or another; we are all in our little cultural tribes in that regard, and Lyons - coming from (what I imagine and what he sorta describes) as the cigar-chomping, dick-measuring, self-congratulatory bombastic cynicism of old school journalism - is a big fish out of water in the “omg that’s awesome!!!” culture of Millennial corporatism.

The second section of the book - when Lyons jokes about trying to awaken some of these young people, via a Norma Rae-style inspiring pro-labor speech, to the way many of these “perks” are just repackaged ways for the company to skimp on its workers (e.g. unlimited PTO being one) - is, in my mind, the best. I LOOOVE labor economics, and I think the tech industry is particularly fascinating in this regard. As I said in my Brotopia review, and as Harvard economist Claudia Goldin says in that one paper, tech has the potential to dismantle a lot of artificial, arbitrarily-imposed constraints to work - it could be really liberating, from a labor perspective. This opportunity and potential is coupled with huge market forces (which Lyons argues are largely speculative and based on hot air) that, instead, perpetuate old power dynamics and exacerbate inequality. Lyons notes that the second tech bubble’s motto is all about “exit strategies” - grow a company very fast, never turn a profit, do an IPO and cash out. This only benefits founders and VC funders. Young workers are cheaper and easier to exploit: free beer takes the place of pensions; “tours of duty” take the place of mutual loyalty. “Passion” and “hackathons” gain prestige; meaning people see glory in working, essentially, unpaid overtime.

The third section is Lyons recounting, well, adult bullying - as HubSpot decides it needs to get rid of him, and engages in the frankly cowardly act of “managing him out”; making his life so miserable that he quits, or hunting for small errors and blowing them out of proportion. This is the least interesting part of the book. I mean, it sucks for Lyons - and toxic workplaces definitely suck - but I don’t believe they’re special to the tech industry. And so this section just feels like a long personal beef, a rant by Lyons. I felt sorry for him, but was also less interested. There IS, however, an unexpected “payoff” to this since - once Lyons quits and announces he’s planning to write a book about his experiences - HubSpot apparently went from being “bozos” to being “monsters” by attempting to hack, extort, and break into (!) Lyon’s house (!!). I found this INSANE. It seemed like an insanely disproportionate response to what is essentially just a catty, snarky book painting them as annoying at best, incompetent at worst. But not EEEEVIL. So that was a surprising turnaround. But maybe, in the age where fake news isn’t just Facebook’s problem - it’s its solution, this shouldn’t be so surprising.

Anyway, a lot of fun, some horror, many amaze. Recommended.

📚 Goodreads

The grumpy guide to better conferences

Sat, 17 Nov 2018 00:00:00 +0000

There’s a wonderful, unique sequence in Kim Stanley Robinson’s hard sci-fi book, Green Mars, where one of the characters - the mad scientist Sax - attends a conference.

Sax is portrayed as an intelligent, deeply academic, pro-terraforming geologist/physicist/chemist; in many ways, he’s a caricature of the professor “on the spectrum”, socially awkward and profoundly enmeshed in his work. Green Mars is book 2 in KSR’s often-sublime, sometimes-dragging Martian Trilogy. And this digression - essentially a really long chapter about a guy going to a conference - is, oddly, one of the better bits.

Because the way Sax attends the conference, the way KSR writes it, is - weirdly, mundanely, inspiringly - fascinating. He flits from lecture to lecture, soaking it all in. And it’s a feast. The chapter is a funny little celebration of that Platonic ideal of good conferences: the kinds that get your brain fizzing.

It’s pretty hard to actually go to one of those. But I have been! So far, the best conference I’ve been to remains OpenVis Conf 2016. I’ve been to a lot of mediocre ones in the meantime, but I will not name names. However, I’d like to talk about how to run a good conference. Since I think some small design nudges can really improve things and, as designers always say, spark delight.

Put the schedule and WiFi password on the back of the lanyard thingie
#

(also, have a WiFi option…)
#

Put the damn schedule on the damn stupid thing I’m wearing. That piece of plastic is useless, an enormous eco-waste, and it could really serve more than its singular function of “Hello, I am a conference attendee, let me into this auditorium”.

Make a conference Slack
#

Bocoup - the web design firm that organizes OpenVis Conf - makes a Slack for their conference each year. This is perfect. You can comment on talks as they’re happening. You can post your job opportunities on a #jobs channel. You can DM the presenters with questions or kudos, or people you met over coffee.

Don’t only offer men’s shirts and treat them as “unisex”
#

Instead, buy women’s shirts and treat THOSE as “unisex”
#

This is one of the most insidious, pet peevey, absurd, annoying things for me. I mean, egregious harassment at tech conferences should be averted by having a Code of Conduct and, you know, EYEBALLS AND A BRAIN. But tech sexism isn’t always so over the top. It can also be exhaustingly petty. Wasting money on shirts that essentially fit only men is one of those things.

Outlets, outlets, outlets
#

Duh.

Childcare, childcare, childcare
#

And other, better ways to spend a budget
#

You’re providing free food, endless coffee, maybe hotel shuttles and maybe a social thing with booze. That’s nice. But all of this assumes a certain set of behaviors: that I’m at that hotel, that I drink coffee and alcohol, that I want to socialize in that way.

I mean, I usually want to have a glass of orange juice and a nap at the end of a big conference day. I know I’m not the only decaf-drinking introvert out there. Clojure/conj 2017 was pretty nice about this - they had a catered board game bonanza as their social night.

I’d much rather they spend the conference budget on childcare given that, again, women at tech conferences is kind of a thing (a “not enough of them” thing) and, again, why are we privileging beer > babies in the budget? What’s the message here? Let’s give parents a leg-up here. Women are often the “primary” parent, responsible for figuring out baby stuff. Help them out.

Instant YouTube
#

Most conferences are pretty good at this: getting the talks up on YouTube soon-ish after they happen. The best, by far, was Clojure/conj 2017, which achieved miraculous speeds and sometimes had conference talks up the very same day. This was great because, at a good conference, you’ll often have multiple talks you want to see that are unfortunately simultaneous.

Ruthlessness towards mic-hogs
#

There’s always one or two abysmal questions at the end; blow-hards wanting to show off, people looking for a platform. The best ways to control for this behavior, I’d say, are (a) a pre-Q&A warning that questions should be questions, and (b) training the speaker to deftly and diplomatically cut people off. It can be done!

World Without Mind- The Existential Threat of Big Tech (2017) - ⭐

Sat, 10 Nov 2018 00:00:00 +0000

Metadata
#

Author(s): Franklin Foer
Number of pages: 272
Year published: 2017
Year read: 2018

Review
#

DNF @ ~30%. More like: threw across room at 30%, after great restraint.

This was tedious, tiresome and totally lame. It’s basically a Great Men history of the tech industry, narrow-minded and completely ignorant of a bunch of contexts (social, economic, historical). The Great Men that I got through were: the Whole Earth Catalog’s Stewart Brand, Alan Turing, Ray Kurzweil, Larry Page, Mark Zuckerberg - and I stopped there. There are also some detours into Great Men of History like Descartes and Leibniz. There is a Great Man of Reporting Tech (which I have particular beef about - the best tech reporter these days is Kara Swisher!).

In between these hagiographies - the only concrete-ish parts of the book - Foer uses vague, hyperbolic language to talk about “algorithms” and “AI” and “monopolies” in a way that feels superficial and like, well, he doesn’t have a background in tech or economics. Example: he attributes the tech giants’ accumulation of monopolistic power as being guided by their “theocratic”, near-spiritual belief in their own, uh, futury-ness (as prescribed by the Whole Earth Catalog). I guess he’s never heard of economies of scale? Or natural monopolies? I’m not saying Google is a natural monopoly, but he kept… NOT acknowledging this basic idea of econ that I begun to wonder, “Wait… maybe he doesn’t know about it.”

In other things he doesn’t know about: algorithms! According to Foer, they were invented by Leibniz (?!). They’re also, apparently, mystical and indecipherable and certainly not just patterns of deterministic steps following logic (not invented by Leibniz), fed into a machine (also not invented by Leibniz). According to Foer, Turing - as he lay in a field after a good run - had the “visionary genius” vision of putting algorithms into machines. Not… Ada Lovelace and Charles Babbage, you know, uh, 100 years before. I mean, this is mind boggling. There’s even a bit where Foer recasts Mario Savio’s “throw yourself onto the gears of the machine” speech as a reaction against the oncoming onslaught of computers’, uh, computeriness.

Descriptions of AI - or, I should say, “AI” - were similarly obfuscated, convoluted, and vague. It seems like whenever Foer didn’t know how to explain a technical concept (which was every time he raised a technical concept!), he would just say “ALGORITHMS” and cue some dramatic music. He conflated neural networks with AI based on cognitive models. He didn’t mention, er, HOW algorithms may be problematic.

ALLLLL THIS would have been vaguely more tolerable if the tone of the book hadn’t been so damn ego-heavy and high-falutin’. If he hadn’t used so many $1 words and philosophical ramblings instead of, well, research evidence and data (of which there is ample to support many of his arguments!).

Awful. I’d recommend Jaron Lanier or Cathy O’Neil or Sherry Turkle instead.

📚 Goodreads

Artificial intelligence (1984) -

Fri, 19 Oct 2018 00:00:00 +0000

Metadata
#

Author(s): Patrick Henry Winston
Number of pages: 527
Year published: 1984
Year read: 2018

Review
#

DNF at about halfway, and won’t review, since - meh - it was fine. I used it during my first Georgia Tech OMSCS course, Knowledge-Based Artificial Intelligence, and that’s over now, so… 🤷

I mean, I love Winston’s MIT AI lectures. They are MIND BLOWINGLY good. Here they are. This book is less about “optimal” AI (in the style of neural nets and machine learning) and more about “human-like” AI (in the style of mimicking cognitive systems and encoding semantic relationships). Which I find inherently confusing and kind of gratingly dull. ML is more exciting! And seems to make more “sense” (even if it’s less interpretable, often). As such, this book feels like a textbook from history - from a field that has changed quite a lot since it was published.

📚 Goodreads

Keeping it 100 in the terminal

Mon, 01 Oct 2018 00:00:00 +0000

Hey. Hi. Here’s a missed connection:

You: Wrote a blog about some super cool bash-based TODO list + journal thing. You kept it all in the Terminal. It was glorious to behold.

Me: Read this in, like, 2014 or something.

Oh, it’s hopeless.

Anyway, I read some blog - SOMEWHERE - by some guy (I think it was a guy? statistically, it was probably a guy) who had this very cool setup of keeping an ongoing journal and TODO list all from the Terminal and all in Vim. He (?) was all about writing writing writing, as a hack to keep his productivity up. And I am all about that too.

So I finally implemented something like it! It took me on a nice journey through bash/zsh, and, oh, how I love my zsh. Here are some previous adventures in it:

Specs
#

OK, for this tutorial/thing I did, I wanted:

A todo list thingie from the Terminal.
A journal thingie from the Terminal.
A way to kill Evernote (sorry, Evernote).

`todo.txt`
#

First step was easy. Someone already built a CLI TODO list manager, and that person is Gina Trapani (thank you, Gina), who made todo.txt.

It’s super easy to install - just brew install todo-txt on a Mac. Then you need to configure your ~/.todo/config file according to this tip:

export TODO_DIR="$HOME/Dropbox/todo"
export TODO_FILE="$TODO_DIR/todo.txt"
export DONE_FILE="$TODO_DIR/done.txt"
export REPORT_FILE="$TODO_DIR/report.txt"
export TMP_FILE="/tmp/todo.tmp"
export TODOTXT_DEFAULT_ACTION=ls

You can change the above to whatever, and SUCH is the glory! I have big plans for this, connecting these folders to a bunch of Python scripts for collating, organizing and archiving my TODOs and DONEs.

Finally, I aliased the actual todo.txt commands to spare keystrokes in honor of the great god Productivity. I did this in my ~/.dotfiles/system/.zshrc

alias t="todo.sh"
alias ta="todo.sh add"
alias td="todo.sh do"
alias tls="todo.sh ls"

And voila:

Journaling from the terminal
#

One of the big things I use Evernote for is keeping an ongoing diary of my workdays - what I’m doing, what’s confusing, ideas I have. It’s where I, for example, paste in charts and SQL queries from the data I am sciencing. But! One thing Evernote does not do well is - okay, two things. The first thing it doesn’t do well is data privacy. I have no idea where my deeply private data is being stored; what do I know about the Evernote servers? Nuffin. So, I don’t like that.

The second thing is doesn’t do well is, well, this: $$p(\mathbf{x} | C_{k}) = \frac{(\sum_{i} x_{i})!}{\prod_{i} x_{i}!}\prod_{i} p_{ki}^{x_i} $$

That is - Markdown+LaTeX math! There’s no elegant way to math in Evernote. It’s very ugly and horrible. So switching to Markdown is a win.

Setting up the Terminal-journal (journinal) was easy:

First, I made a directory where I’d store my Markdown journals. I aliased it to hawaii because, that way, it would make me feel happy every time I typed that.
Second, I wrote the following function into my ~/.dotfiles/system/.zshrc:

function notes() {
 local TODAY_NOTES=$(date +%Y-%b-%d_%A).md;
 echo $TODAY_NOTES;
 # aliased to my notes folder
 hawaii;
 # aliased to my work name
 if [[ -z $1 ]]; then
 SUBDIR=$WORK
 else
 SUBDIR=$1
 fi
 vi $SUBDIR/$TODAY_NOTES;
}

I have subdirectories in hawaii for work, personal and omscs, so that way - when I am lamenting life - I can do it directly from the Terminal by simply typing notes personal. It will either find or auto-create the .md file with today’s date, e.g., 2018-10-01_Monday.md, and enter Vim for me:

`RPROMPT`
#

Okay, so this is a thing I found elsewhere, but the basic idea is: use the right-hand side prompt (RPROMPT) in your zsh to put stuff like: how many TODOs you have in your code, how many project-specific tasks you have coming up, and the like. It is also cool, so check out Wynn’s blog.

TODOs on my TODOs
#

I have a bunch of Python scripty ideas, e.g.:

To properly destroy Evernote, I want to bulk-migrate all my old notes into the Markdown format.
I want a weekly archiver that sweeps through my todo.txt list and my work notes and collates them into one big biweekly update I can send to my boss.
I am debating the version control on my notes and TODOs. I do not 100% trust a private GitHub repo!!! But maybe it’s fine. Debating whether to stick all this stuff in Keybase like some sort of Amyericanski spy (sorry, have been watching a lot of The Americans).

oh no

Sat, 08 Sep 2018 00:00:00 +0000

Semester 1 of Georgia Tech’s Online Master’s in Computer Science.

I’m taking CS7637: Knowledge-Based Artificial Intelligence. My reasons: AI is super awesome; I liked the idea of cognitive science and AI; and one of the instructors - Dr. David Joyner - is very highly spoken of throughout the program for running well-designed courses.

Prereqs are “just some Python”. All seems well.

Lecture 1: “hi here are some blocks. can you order them?”

Me: “yes i can”

Lecture 2-8: “here are some ideas how computers would do this”

Me: “sounds good”

Homework 1: “hi isn’t AI cool. let’s talk freely about that.”

Me: “oh man i am going to crush this course - "

Project 1: “hi PLEASE BUILD YOUR AI NOW”

Me: “wait i”

Project 1: “YOU CAN USE PYTHON”

Me: “…”

Project 1: “LOOK HERE ARE SOME SIMPLE VISUAL INTELLIGENCE TESTS PLZ SOLVE THESE W YOUR AI AGENT”

Me: "”

Project 1: “THEY WILL BE GETTING HARDER”

Numerous forum posts: “HI MY AI USES AMAZING COMPUTER VISION AND IS ALREADY SO ADVANCED THAT IT IS TAKING OVER THE LAND AROUND MY CITY IN A GRAY GOO SCENARIO, HOW DO I DEAL WITH THIS? THANK YOU, KIND TEACHERS”

Me: “oh no”

So there was a bit of a leap between the lectures and homeworks and the project.

But all is well now. It’s even getting fun (after that first icy shower of “oh god what have I signed up for”). Though calling the program I’m writing “AI” is very, very generous…

A Groundhog Day of linear algebras

Mon, 16 Jul 2018 00:00:00 +0000

I think linear algebra is much wonderful magic and lore, and I think it’s a lifelong commitment. As such, I’ve taken linear algebra refreshers and courses many, many times.

While I haven’t found the holy grail of linear algebra courses yet - the course that unlocks linear algebra as a visual and geometric intuition, that is what I seek - I have found some good ones and some OK ones. And I figured I could review them all today! Why not!

MATH-310: Linear Algebra, American University
#

My first experience of linear algebra. I mean, I took this ~15 years ago (mamma mia). I remember enjoying it. I remember finding it deceptively simple. “Oh, it’s all just basic math, only on series of numbers. How very plain!” Little did I know.

Pros	Cons
Very positive experience!	Well, it was a long time ago.
Lots of fun!	Did I really dig into the theory of this stuff?
(False) sense of VERY EASY!

Summer math camp, University of Oxford - MPhil in Economics program
#

Whence young folk go into econ grad school, thence are they usually exposed to a crash course of linear algebra (again, for many of them). I don’t remember learning much here (sorry, Oxford! or maybe you should be apologizing to me, yeesh).

Pros	Cons
Well, it was short.	Too short.
	And a long time ago.

Linear Algebra - Foundations to Frontiers (LAFF), UT-Austin, via EdX
#

This linear algebra course, offered freely via the EdX platform, is very highly rated and I was excited to take it. It also promised a visual-first teaching style, a “different” way of teaching linalg, and a very engaged set of professors and instructors.

Unfortunately, me and this course! We just didn’t connect! I churned my way through about 1/3 of it before giving up.

Pros	Cons
Proof-heavy.	Proof-heavy.
Free!	Forced use of their idiosyncratic/esoteric linear algebra software (instead of just `numpy` or something!)
Self-paced	Mostly theoretical. Most exercises/homework are proofs.
Always (?) available

Linear algebra, Khan Academy
#

Honestly, Khan Academy has consistently blown my mind on such diverse topics as:

Natural selection (MY MIND WAS BLOWN)
Information theory (SO BLOWN)
Renaissance Italian art (OMG AND I THOUGHT I KNEW ALL ABOUT THIS)

A lot of this has to do with Sal Khan’s incredible clarity of thought, inherent pedagogical optimism/growth mindset, and his special talent of breaking everything down into a simple, understandable building block - and then using those blocks to build, ahem, TRULY INCREDIBLE TOWERS OF THOUGHT.

I love love love Khan Academy, it’s one of my monthly charitable donations, and I recommend it to everyone for everything.

Anyway, so the linear algebra track! I switched to this from LAFF (above) and have slowly cobbled my way through it over the past N years (yes, years). Like LAFF, it is relatively heavy on proofs. Unlike LAFF, it starts with getting some of the basics of linear algebra calculations out of the way - there are a good number of exercises in the early sections which help you get some sense of doing basic calculations with vectors and matrices.

That said - it quickly takes off and becomes a lecture series of Sal Khan working through various proofs. This has pros and cons: a main pro is that Sal Khan is one of the few people I’ve found that can walk you through a proof in a way that feels instantly clear and simple. He’s just good at it. And I’ve had many “ah ha” moments. The main con is sticking power: after Sal walks me through a proof, I’m like “ah ha, okay… now what?” There’s no immediate application. There’s rarely even a real world example of how/why this is amazing or useful. And thus I find it basically impossible to retain the takeaway.

Whereas that’s what I want! I want to develop an intuitive understanding of linear algebra such that I can “see” the changes visually. See my post on Jacobian transformations for an example of when it has “clicked”.

Pros	Cons
Proof-heavy.	(Eventually) proof-heavy.
Free!	Exercises dwindle after an initial burst - there aren’t many other ways to stay engaged.
Self-paced	Limited real-world examples/applications to work through or think about.
Always available
Sal Khan is a great teacher. Maybe the best!
More use of visual/geometric examples.

Linear algebra refresher, Udacity and Georgia Tech
#

This one is freshest in my mind (since I’ve been working through it over the last couple days), and it’s also my first semi-official introduction to the OMSCS. I was a little irritated with this course, though, since the idea was great but the execution was uggghhh.

The idea was: let’s refresh/learn the basic ideas of linear algebra by building our own linear algebra library! In any language we like! Sounds great.

The execution though, was really, really painful. The instructor would quickly go over some major, deep linear algebra concept - such as determining whether two lines are parallel - and then would introduce the exercise. All the exercises have taken the same form: you are given - via a screenshot (!) of long, arbitrary floats (!!) - some example vectors, and you have to manually peck these long, arbitrary floats into your program (mine was also in Python, like the instructor’s) to then solve for whatever. This was only after the instructor mind-numbingly read out all of these numbers both before and after you submitted your responses.

Screenshot:

What in the Lord. What in the Lord?! Who thought this was a good idea? I spent like 40% of this refresher course manually pecking numbers and then, oh, making obvious human errors because manually transcribing numbers is an incredible waste of my time. Why didn’t they just print the numbers as text?! Then I could copy-paste? Also, why did the instructor need to literally read out “four point zero four six ex plus two point eight three six why equals one point two one” both before and after, when he reads out the answers?! What an incredible waste of time and awful UX/UI. For shame!

For this reason, I’m only about halfway through this, and I’ll choke the rest down (it’s meant to only take a few hours, and I’ve been doing it here and there when I have a minute), but I’m really doubtful that I’ll have any “ah ha” moments.

Pros	Cons
Choose-your-language build-to-learn pedagogy.	SEVEN POINT TWO ZERO FOUR EX PLUS THREE POINT ONE EIGHT TWO WHY EQUALS PLEASE USE THREE DECIMAL PLACES OF ACCURACY WHY IN THE WORLD ARE WE PROVIDING EXERCISES BY MANUAL READ-OUT AND HANDWRITTEN SCREENSHOT

Ten Arguments For Deleting Your Social Media Accounts Right Now (2018) - ⭐⭐⭐

Thu, 21 Jun 2018 00:00:00 +0000

Metadata
#

Author(s): Jaron Lanier
Number of pages: 146
Year published: 2018
Year read: 2018

Review
#

A brief, decent book about an obvious thing that many people can’t bring themselves to do: stop supporting social media.

Social media has been likened to the tobacco industry. It’s a pervasive problem. We have a bunch of research about how bad it is. The incentive structures are a nightmare. It ruined the 2016 election. It undermines democracy, empathy, actual human connections. It’s false and full of lies. It’s addictive. It perpetuates our worst instincts; it turns us, as Lanier says, into assholes. And so on.

Anyway, billions of people use Facebook. What a nightmare! I have no tolerance for the “concern troll” argument that disadvantaged groups are somehow unable to leave Facebook; I lived in Tanzania for two years and India for one year. In those places, Facebook is now bundled with people’s mobile phone plans and offered as a “free” service. I’ve seen an argument (on Twitter): “this is the only access to the internet these people have!” Oh yes, Facebook offering such succor to the poor and benighted. Give me a break. MPESA gives succor. Facebook sure as hell does not.

Anyway. I love Jaron Lanier. I saw him speak years ago. Appalled when no one in the audience knew of E.M. Forster’s The Machine Stops, he strongly recommended it and I read it shortly afterwards. And it changed my life. So thanks, Mr. Lanier! I also think his tech skepticism is informed and healthy. And, of course, I’m inspired by his weird and quirky interests: his writing is smart, confident, kind-hearted, and wacky. Which is always a balm.

I wonder how many people have actually deleted their FB accounts…

📚 Goodreads

Working remotely or remotely working?

Fri, 15 Jun 2018 00:00:00 +0000

I have a strong interest in labor: that is, the history, sociology, economics and psychology of labor.

Specifically, I think it’s really interesting (and not entirely efficient) the way that we organize our work. We’ve inherited things like the 9-to-5 workday and two-day weekends from the Industrial Revolution, Henry Ford, and factory work.

But not all work is factory work. Of course, there is some variety in the modern work day - medical residents, for example, have their own (insane!) labor culture - and there does seem to be an increasing movement towards alternative forms of work: remote work, flexible hours, and so forth. But I don’t think we’re there yet. And I hope to pioneer (with courage!) new, better ways to work. That is, I hope to worry less about “performing” work and worry more about pursuing actual mastery, actual efficiency, and actual productivity. I’m lucky that my employers have generally supported this.

Laboring in grad school vs. laboring at work
#

I remember, very vividly, my first day of “real work”: I had just finished my economics MPhil at Oxford and was starting as an ODI Fellow in Fiji at the Pacific Islands Applied Geoscience Commission. At Oxford, it had been left up to me to organize my time and prepare for the final exams. In Fiji (and in nearly every job hence), there was - instead - an expectation to “perform work” between certain hours in a certain way. Here’s the comparison:

In Oxford/grad school, we had an entire Trinity Term (13 weeks, iirc) to prepare for two 3-hour exams (mine were Development Economics and - surprise - Labor Economics) and one research thesis. We had no other obligation. No classes, no meetings, nothing. So we all entered our little study caves and only emerged 13 weeks later, pale and with overgrown haircuts.

Because my academic diligence spilled over into anxiety/panic, I spent the first day of Trinity Term calculating the number of waking hours in those 13 weeks, and dividing them between my exam subjects. I allocated, a priori, a higher proportion to weaker topics. I took zero weekends. Instead, I worked every day using a primitive pomodoro technique: 2-hour blocks, followed by N-minute breaks.

I would do three blocks per day, starting at 8am. I was very strict on the start time (no later than 8am!), but much more flexible on the break times (sometimes 15 minutes, sometimes an hour long). Hence, my days would sometimes end in the early afternoon (sweet, sweet release), and sometimes drag into the night (oh God).

That’s 6 hours/day, or 42 hours/week, of dedicated, focused work. Which doesn’t sound like a lot.

But this was, honestly, hell. This was definitely the absolute maximum focused effort I could endure and, by the end of the 13 weeks, I was a wreck. I was also doing things like reading epistemology and aesthetic philosophy for fun, so I guess I had body-built my brain. But I knew I couldn’t maintain this. “How do working people do it?” I thought. “Who can possibly work for 8 hours every day, for years on end?”

At my first day of my first real job, I remember that I woke up very early, got ready very diligently, had butterflies in my stomach, and showed up some 15-20 minutes before start time. (Which, iirc, was 8am in Fiji anyway - work days are shifted earlier in the Pacific.) Naturally, they didn’t have something for me to do right then, so my supervisor gave me some internal reports to read. They helped me get my computer set up. Stuff like that. And I remembered thinking: “Wait a minute… am I supposed to just sit here, obediently, and work for the full 8 hours? What about my N-minute breaks?!”

Of course, I didn’t realize that very few people (I’m guessing zero people) actually put in 8 hours of dedicated, focused knowledge work for years on end. (Some random study found that people actually do three hours of “real” work per day, and that includes e-mails.) There’s all sorts of interruptions: chatting with colleagues, getting lunch, futzing around on the intertoobs, the demon Multitasking and that great False Work God: E-MAIL.

Reading and writing e-mails was so much easier than bashing my head against econometric methods for determining the returns to education. So much easier than parsing academic articles, working through problem sets, all that. And - best ever - it was commonly seen as “working”. Indeed, I would guess that a huge proportion of my pre-tech career was, well, DOING E-MAILS.

So that was all a bit of a culture shock.

Moving into tech: Moving into making
#

My first career was in international development economics. Looking back on those 9 (!) years, I would guesstimate that my aggregate time diary would have been: 40% emails, 30% managing surveys/traveling around, 10% writing reports, 10% doing data analysis, and 10% training people. Actually, remove ~1% from all those chunks for “socializing”/futzing around, let’s be honest.

Then I moved into tech. A couple things really attracted me to a career in coding: I loved the creativity, the creation of something, the constant search for automation and efficiency. I loved that, every day, I had something concrete that I could build. I loved the constant puzzles, and the constant learning.

At the same time, I realized (with some horror) that a career centered around programming brought a new, harsh light to things: either my code ran or it didn’t. Either it broke, days later, or it didn’t. Either I wrote 10 lines or 1000 lines. Everything was plain as day. No more hiding in well-written e-mails that danced with diplomacy!

At the same time, given the great clarity with which we could observe each other’s actual productivity, there was an accompanying culture of lax-ness on the “performance” of work. (Yeah, yeah, I know - there are other ways tech people “perform” their “genius”/competence, and I can complain about those in another blog post.) There seems to be a general understanding that we’re all intrinsically motivated. The office culture is more casual. There’s a general tolerance towards remote work and strange hair (I had lavender hair for a while!).

In a way, this is very, very freeing. I no longer have to expend so much energy on the performance of work. If I hate rush hour traffic, I can come in later and leave later or not come in at all. If I feel like I’m not getting anything done at the office, I can go home and work from there. Of course, a lot of this hangs on the trust of the employer but - well - GitHub commit histories can’t lie!

How I organize my days
#

That said, it’s been very hard for me to break certain “performing work” habits (like feeling guilty about taking breaks). And then there are some “performing work” habits that are beneficial (like being good at writing professional e-mails!).

Lately, though, I’ve found some very freeing ways to be more productive and effective, FOR REAL, and less concerned with “performing work”:

Time tracking
#

I use a time tracker to track my actual productive time every day. There are a bunch of different ones out there, but I wanted one that kept my data local (i.e. no sending it to some server). I use Qbserve, which is not free but I highly recommend.

You need to customize your time tracking software to understand what’s “productive”, “neutral” or “distracting”. Obvious things (like Twitter) are already set as “distracting”, but I’ve also been intentional about correctly labelling stuff that, for example, might be productive for some people, but is just a compulsive and pointless time-sink for me (like LinkedIn!).

I’ve also waffled a lot on Slack: I consider Slack an enormous attention-killer and hence productivity-killer. But it’s also where I’ve had some very useful and informative conversations; for example, working with a colleague to debug some code.

Anyway, what’s nice about time tracking is that it makes you aware of where your time is going - and it makes you think mindfully about how you want to spend your time. I think learning is an integral part of both my happiness and my productivity, so I’ve put all online learning resources (like Udacity, Coursera, Stanford, etc) into the green bucket. Similarly, Qbserve has helped me notice when I’ve worked quite hard on one day (which, for me, means 5-6 hours of focused, dedicated work), that I’ll likely need to take it easier on the next day (say, 3-4 hours of focused work).

Finding my schedule
#

I’m very front-loaded: as soon as I wake up, I’m starving and full of energy. If it’s sunny and 10am, I can basically conquer the world. I progressively lose steam throughout the day until, by the late afternoon/evening, I’m very done.

One thing I noticed recently is that, if I work from home, I strongly prefer starting at 7:30am and working until, say, 3pm or so. (Again, I try not to worry about the exact number of hours between my first productive moment and my last - I use Qbserve to see how much I’ve actually accomplished).

When I first tried this out, it was a revelation: I got more done, felt more well-rested/energetic, and seemed to have more time! Just because I started earlier and followed my natural (circadian?) rhythm.

A focus on outputs
#

This is the hardest thing for me, and something I’m still working on (as is obvious by all my hours-counting above): I have a very strong impulse to focus on inputs - that is, the number of hours I put in. Obviously, though, what matters is what I end up producing at the end of the day. If it takes me 1 hour to write a good data ETL job, or if it takes me 10 hours, the virtue is in the former (duh).

So I try to seek automation and get more efficient and be results-focused. And I try, as much as possible, to have concrete (and brief! and measurable!) TODOs at the start of every day. I use Evernote for this:

This TODO list used to be a loooot longer, dozens of items long. Which was absurd. So now I intentionally try to keep it to <5 items.

Finally, an important note about mortality
#

I read a Guardian article in 2008 that was deeply affecting: Life Before Death, The Guardian, Mar 31 2008. In particular, this quote stayed with me:

“Gerda couldn’t believe that cancer was cheating her of her hard-earned retirement. “My whole life was nothing but work, work, work,” she told me.”

This struck me. A lot of people, on their death beds, have only one regret: working too much, at the expense of other things in their lives. I think this is important even if you love your work (as I do!). We don’t have infinite tomorrows, neither do our loved ones, and so it’s good to be intentional about working enough and working well, but not sacrificing everything for it.

Introducing triestin

Wed, 30 May 2018 00:00:00 +0000

I did it! I’m done! I finished CS50. Now is the time for a feeling of accomplishment, mixed with sadness. GOODBYE, CS50! Will I ever MOOC as well as I MOOCed with thee? Who knows.

Anyway. The final project! Yes! I had a GREAT IDEA for my final project:

To write a programming language in not-English, and thus examine whether not-English languages can convey new programming concepts.

Ambitious! I had originally wanted to do it all in C as well: I envisioned fields of malloc() and free(). But, well, I gave up on that and ended up writing and compiling it to Python. EASY MODE. My middle names.

tldr: Here it is!
#

Writing a programming language: In theory
#

Let’s say you open up https://repl.it/, choose a language, write some stuff, and then run it. What, exactly, happens when you run run?

Well: Under the hood, a programming language is doing three things. This is apparently what all languages do. Let’s go step by step.

Inputting the string
#

The zero-th step for any language is accepting a text (string) input. You type in print("hello world") in Python. That string - 'print("hello world")' - gets accepted by the language. Okay!

Lexing/Tokenizing
#

The next step is tokenizing. This is where some heavy regex comes in. You (the language) need to look at the string and figure out how to split it into its constituent parts. The split is determined by the idiosyncracies of your syntax. That is, how are strings of letters interpreted - is print the name of a variable? A number? Is it a pigeon? Or is it a keyword (that is, a special/reserved word in the syntax)? Spoiler, but print is a keyword in Python - it calls the print() function.

Actually, this is easier with an example. Let’s print “hello world” in a few different languages (including triestin!) and write out the resulting tokenization:

# Python
print("hello world")
# Tokens: "print", "(", ""hello world"", ")"

# C
printf("hello world");
# Tokens: "printf", "(", ""hello world"", ")", ";"

# Clojure
(println "hello world")
# Tokens: "(", "println", ""hello world"", ")"

# Triestin
dimmi hello world fin
# Tokens: "dimmi", "hello world", "fin"

(Note to self: Add syntax highlighting for triestin…)

In triestin, I saved my tokens as namedtuples with a type (tipo) and a value (val). For example, from the above, if triestin found a dimmi token, it would save it as Token(tipo="print", val="dimmi").

One important thing about tokenizing is that it’s ordered - that is, we need to look through our input string and make sure we don’t, e.g., mistake dimmi for a variable name. There’s precedence, and keywords come first!

Parsing
#

Okay, so now we have a bunch of tokens. Or rather: we have an ordered string of tokens, like so:

# input string: dimmi hello world fin
Token(tipo="print", val="dimmi"), Token(tipo="identifier", val="hello world"), Token(tipo="fin", val="fin")

Now what? Now we need to parse them. Parsing means taking those tokens and composing them into an abstract syntax tree (AST). That is, we need to translate the input string -> tokens -> something semantic and meaningful which conveys what the user wanted to do. For dimmi hello world fin, we know that fin is the delimiter and anything after dimmi is what we’ll want to print. Parsing those tokens is easy then. In pseudocode:

From the stack of tokens, pop off the first one.
Inspect it. Is it dimmi?
If so, keep popping off more tokens until you get to fin.
Put all those tokens into a string.
Save all that into a PrintNode in your abstract syntax tree. The val (value) of the PrintNode will just be the string to be printed.

Easy!

The harder stuff is in parsing function definitions and function calls. Our language needs to be able to support an arbitrary number of nested arguments and functions. This can be accomplished by:

From the stack of tokens, pop off the first one.
Inspect - is it a function definition (ecco)?
If so, parse a function definition. This includes parsing its arguments, its body, and - if the body has a function call in it - looping back, parsing that call’s arguments and expressions, and so on. You can keep recursively parsing through a function of functions, etc, to an arbitrary depth of levels.

You can see all this in parser.py.

In the end, though, you’ll have an AST with, one hopes, the input code correctly represented as a tree of calls, variables, functions, and so on.

Compiling
#

And now! What do we do with our AST? This is somewhat tricky, but mostly easy (parsing is the hardest part, I reckon): we crawl through our tree and, node by node, convert each node to the target language - i.e. the language we want to compile to. I chose Python because it’s easy! And triestin made the same assumptions as Python (i.e. it was dynamically typed, it didn’t handle memory explicitly, etc.).

So how do we compile? Well, we just untangle the node into its constituent parts and then create an output string in Python! With a PrintNode (which parsed dimmi, remember), it’s super simple: "print('{}')".format(node.val)! Just pipe that output string to Python and Python will do what it’s told! Voila.

One note: like with parsing, compiling is recursive - that is, you have no idea how many nodes you have, and how nested they are. So you gotta keep generating and generating, node by node, until you run out of nodes and have gone down every “branch” of the tree.

Here’s compiler.py.

Writing a programming language: In practice
#

Yeah, right. That all sounds fine. But how was the process?

Well - my process was OK. I took it fairly easy and it took me about six weeks of dabble-here, dabble-there.

I started with Build Your Own Lisp, which is a (free) book to learn how to write a Lisp-like language in C. I got through 11 (!) of its 16 chapters, but eventually called it quits, since it had just gotten away from me. I had no idea what I was doing, I was confused, I was copy-pecking way too much, and bah. So that was a big dead end.

Thankfully, I found Destroy All Software’s great videos - specifically, A compiler from scratch. In that video, DAS writes a language in Ruby that compiles to JavaScript. I know JavaScript and Ruby is similar enough to Python that I could translate it pretty easily. triestin is heavily based on this video.

All the other speed bumps were small potatoes - figuring out how to do math seemed inordinately hard, until I sat down and employed the Lazy Angela Way - i.e. I just directly fed anything that seemed mathy straight to Python and let Python handle it.

I actually ended up learning a lot of handy Python while building this: breaking everything out into modules, learning about when to use and when not to use a Class, working with namedtuples. It was great!

Linguistic side-eye: A detour into `anzi` and `è`
#

`anzi`
#

The big point of triestin, the big hypothesis, is that because most (all?) programming languages are written in English, we may be limiting the programmatic concepts that get out there. That is, it’s kind of like the Sapir-Whorf hypothesis - tldr, that language affects our thoughts - only for software design. This is beyond the obvious implication that English-speaking programmers have a sociocultural, and not technical, advantage over everyone else just because every damn language is written in their own damn language!

I was super into Sapir-Whorf back in grad school, but I’m less enamored with it now (though I still highly recommend this book). I decided to try to think through some words which are difficult to translate into English. German is famous for having words like this, but, interestingly, I think most languages have words without exact counterparts (languages are not one completely overlapping Venn diagram!). For example:

doch in German, which I find impossible to understand totally, but kinda means “the opposite of what you just said”.
The -va- infix in Hindi, e.g. karna (करना, “to do”) vs. karvana (करवाना, “to have done”), which converts verbs from active to passive.
Swahili’s precisely logical “compounding words”: nimekupenda (“I have loved you”), which breaks out into ni- (“I”), -me- (“have”), -ku- (“you”) and -penda (“love”). Man, Swahili logic is so cool. (Also, infixes.)

All of these could be turned into interesting programmatic concepts! I think programming languages are more than pure logic (though they are mostly that) - there are language design choices that, in turn, determine the design of products built in that language. And our software design world is, for now, monolingual!

So I thought I’d try to implement a not-English concept. Namely, I thought I’d try to implement a programmatic concept for anzi, an Italian word that kinda means “the opposite of what was just said” (so, I think like doch in German?) but can also mean “rather” or “instead of” or “actually”. I feel like, meta-semantically, anzi means “HARK, CONTEXT!” But how to use a contexty word like anzi in a programming language? It would have to know context, i.e. remember what had just been said.

In pseudocode, I decided anzi would be a “reassignment operator”. That is, after assigning a variable, x, to some value, y, if the user said anzi z fin, then x would be reassigned to z!

But this meant, ugggghhh, that I would have to figure out how to have state in my language. Ugghghhhh. STTAAAAATTTEEE. Noooooo. Ohhhhhh nooooo.

OK, I complained a lot. It seemed hard. I ended up hacking an awful solution. Here are the steps:

First, in parser.py, let’s make an AnziNode and parse it:

def parse_anzi(self):
 """
 `anzi` is kind of like 'actually' or 'rather'
 it's kind of like "doch" (German) - it can signify "scratch that"
 programmatic `anzi` will revise whatever assignment you just made (for now)
 e.g.
 x è 1 fin
 anzi 2 fin
 x fin
 >> 2
 """
 self.consume('anzi')
 value = self.consume_next().val
 self.consume('fin')
 return AnziNode(val=value)

That is, when someone uses anzi in triestin, we’ll save the value that they gave us. This is the z value - the replacement value. Now it needs to be reassigned!

In compiler.py, I just fed the AnziNode through, telling the REPL that it was coming and was special and had a value.

Then, in the REPL - here - I did this awful hack:

if __name__=="__main__":

 print("Triestin v0.0.1")
 print("ctrl+C per uscire")

 local_state = State()

 while True:
 try:
 # Python 2 on CLI
 user_input = raw_input('triestin> ')
 output = triestin(source_code=user_input)
 local_state.add_history(output)
 if output['state'] == 'exec':
 exec(output['code'])
 if output['state'] == 'eval':
 if type(eval(output['code'])) == tuple:
 print(eval(output['code'])[0])
 else:
 print(eval(output['code']))
 if output['state'] == 'anzi':
 if local_state.history[-2]['node'] == 'AssignNode':
 prev_node = local_state.history[-2]
 prev_var = prev_node['code'].split('=')[0].strip()
 new_code = '{} = {}'.format(prev_var, output['code'])
 exec(new_code)
 except Exception as e:
 print(e)

Ho jeez. This looks so hideous to me. A local history of whatever has happened in the REPL gets saved and anzi just looks to the previous command. If it was an assignment node (e.g. x è 1) then it reassigns whatever 1 was to the AnziNode value. MADNESS.

(Note to self: let anzi reassign to previous assignment, without that assignment having happened just before…)

Okay, anyway, good enough.

`è`
#

My next stab for polylingual justice was replacing something considered very banal and very basic, the assignment operator (normally =), to something that should be equally banal and basic: the Italian word for “is” - è!

That way, someone could write x è 1 fin, which can be read as “x è uno” - that is, “x is one”. Easy!

No, not easy. È has an accent over it, which causes English keyboards to fuss and fritz and generally complain. It also caused Python to fritz and complain: WHAT ENCODING IS THIS?! Because, of course, the computer hegemony is a 26-letter English alphabet one, and “è” isn’t easy to type on my English keyboard, and rant rant rave. It’s one character! Just as efficient as =! But, because of hardware and software design choices, it’s a giant pain in the ass to use.

Anyway, you get my point. Now every time someone has to Google “e accent grave” to get è so they can copy it over into the triestin REPL, THEY SHALL KNOW MY RIGHTEOUS INDIGNATION.

Resources
#

github.com/angelaambroz/triestin

Computer languages
#

Human languages
#

Language in Thought and Action, S.I. Hayakawa
Anything by John McWhorter! Though The Story of Human Language (audio-lectures) and Lexicon Valley are excellent.

PEMDAS HTML

Tue, 29 May 2018 00:00:00 +0000

A small thing today: I just delighted myself with the the text editor plugin, Emmet (Atom, Sublime). Emmet is useful for HTML/CSS autocompletion. Basically, since HTML is a pain in the butt that always has opening and closing tags (<p>blah blah</p>), Emmet uses tab completion to productivity-hack that process: you just type p and then TAB and, voila, you get your <p></p>.

But that’s not the best part. THE BEST PART is that you can tab-complete your way through more complicated HTML trees of classes and IDs. Emmet uses HTML’s inherent tree structure (parent, child, and sibling elements) to create an intuitive, order of operations-style process.

Below, I needed to make another “project info card” on my projects page. Because I’m using Bootstrap, I needed to pack a bunch of elements together: a <div>, some links, some paragraphs, spans, blah blah. With Emmet, I did it all in one go. MWAAHHH!

Voila:

GLORIOUS.

Yet another dotfiles repo

Wed, 23 May 2018 00:00:00 +0000

Today’s simple tutorial: getting your dotfiles in order!

Note: This post was edited and improved in Dec 2020.

What is dotfiles, precious?
#

Dotfiles are:

Files that start with .
And are technically hidden from you (i.e. they won’t show up in Finder or with simple ls)
But you can actually just see them if you do ls -a
You end up fiddling with them to configure your system when you use the shell a lot

Probably the most popular dotfile you’ll run into is .bash_profile. This configures your shell (if you’re using bash, which - by default - you probably are). But you can have a bunch of other dotfiles - for example, .pgpass (where your Postgres credentials go), .id_rsa (where your SSH keys are stored - which, incidentally, sits in a dotfiled directory, .ssh/), and so on.

Anyway, I recently made a few dotfiles that I wanted to version control - specifically, my zsh and ipython setups.

Making your own dotfiles
#

You can see my completed dotfiles repo here.

There’s a bunch of dotfile guides on the web (see below). My process was Simple Angela’s Version. Namely, in my shell, first I made my dotfiles folder and got it ready for git:

mkdir ~/.dotfiles
cd ~/.dotfiles
mkdir secrets
echo 'secrets/' >> .gitignore
mkdir git
mkdir jupyter
mkdir system
git init
git add .
git commit -m "Dotfiles begin..."

In other words, from the command line, I made the following directory structure:

- .dotfiles/
 - git/
 - jupyter/
 - secrets/
 - system/
 - secrets/
 - .gitignore

And I made sure to .gitignore anything that shows up in .dotfiles/secrets/, since that’s where I would put any sensitive passwords, URLs, and API keys.

Now it was just a process of making the relevant dotfiles for each thingie (e.g. my system/ dotfiles would be stuff like my shell configs, git/ would have a global .gitignore) and make sure that each thingie could find its config. Let’s go through it, thingie by thingie:

Thingie 1: Jupyter
#

I’ve already written about my default IPython startup. But how to get IPython to find it? After all, IPython startup files live in ~/.ipython/profile_default/startup/. The easiest way is to link the file to your .dotfiles repo:

ln -s ~/.ipython/profile_default/startup/00-dotfiles.py ~/.dotfiles/jupyter/startup.py

Thingie 2: `git`
#

I used gibo to quickly create a giant .global_gitignore boilerplate to handle all my usual use-cases: Python, PyCharm, Sublime.

cd ~/.dotfiles/git
gibo Python macOS JetBrains SublimeText >> .global_gitignore

But now how to tell git about it?

Easy! Git has a global config file, stored in your $HOME directory, that looks for a global .gitignore and implements it. (Note that your .gitignore file can’t unfurl $ bashy things, so you have to hard-code the full path to your dotfiles.

[core]
editor = /Applications/Sublime\\ Text.app/Contents/SharedSupport/bin/subl -n -w
excludesfile = path/to/home/dir/.dotfiles/git/.global_gitignore
[user]
name = Angela Ambroz
email = angelaambroz@users.noreply.github.com
[alias]
unstage = reset HEAD --

Thingie 3: `zsh`
#

And finally, the majesty - the cherry on top - the crown jewel: zsh! How was I going to tell my shell to, well, SHELL RIGHT?! To use this wonderful new dotfiles directory? This was two parts: having .zshrc (z-shell startup file) loop and source through all the relevant dotfiles, and making sure my shell knew where to find the z-shell startup file.

For the latter: I learned about symbolic links (aka, symlinks) using the ln utility:

❯ tldr ln

ln

Creates links to files and folders.

- Create a symbolic link to a file (or folder):
 ln -s path/to/file path/to/symlink

- Overwrite an existing symbolic to point to a different file:
 ln -sf path/to/new_file path/to/symlink

- Create a hard link to a file:
 ln path/to/file path/to/hardlink

(Side note but tldr - i.e. plain English man pages - is a lifesaver. I doff my hat.)

Okey dokey:

ln -s ~/.dotfiles/system/.zshrc ~/.zshrc

It’s important to include that -s (symbolic) flag, otherwise you’ll just be linking the file once and updates won’t be reflected.

And you can double check that the symlink was truly linked with a quick grep:

ls -a | grep '\->'

For the former (looping through the relevant dotfiles), I just added the following to ~/.dotfiles/system/.zshrc:

# Load all my secrets
SECRETS="$HOME/.dotfiles/secrets"
for file in "$SECRETS"/.*
do
 source "$file"
done
echo "Loaded secrets."

# Load non-secrets
source "$HOME/.dotfiles/system/.alias"
echo "Loaded non-secrets."

Since I broke out my keys et al. into aliases and EXPORTs in the secrets/ dir, and my less interesting, non-secret aliases in system/.alias.

And that’s it!

Resources
#

Look, I made a hat

Sun, 20 May 2018 00:00:00 +0000

“Studying the hat. Entering the world of the hat.”

God, Sondheim. SONDHEIM! Amazing! Ach.

Anyway, on this Sunday in the apartment with Angela, let’s celebrate a “hat” three (lazy) years in the making: my 2015 “second a day” video! 🎉🎉🎉🎉

Background (artist’s statement)
#

On 31 December 2014, I decided I wanted to do one of those “second a day” projects for the next year. I like the idea of art that plays with time: something that commented on impermanence and something that grew in value over time. Something where time was a strong component and a “second artist”.

A “second a day” project is great for that: when you think about it, you forget most days you live. By capturing a single second, you get insight - you preserve a moment. It’s also great because, by condensing time, you see the slow-moving changes: hair growing, weight gain or loss, all that.

Anyway, in December 2014, I was living in Dar es Salaam, Tanzania, but I knew I would be moving somewhere else after sometime in 2015. I knew 2015 would be a busy year full of change: a lively year!

Yet more nostalgia: Python
#

This project was also my first Python project. While it was painless to film a second a day, I realized it would be painful to edit all those multi-second videos into one, ~6 minute thing. And so I had that one fateful thought:

“Wait, is this something I could use Python for?”

The answer to that question is usually yes. Specifically, I used Udacity’s Programming Foundations with Python and found MoviePy and - I was off to the races!

Three years later, I can safely say my Python code was – yeeechhhh, AWFUL. But hey ¯\_(ツ)_/¯ whatever. Girl’s gotta learn somehow.

That was years ago…
#

What would Sondheim, Mandy, and Bernadette have to say about all this nostalgia?

MOOOOOVEEEE OOOOOOOOONNNNNNNNNNNN

Social media is cigarettes

Fri, 18 May 2018 00:00:00 +0000

The GDPR represents a huge legal shift, which is arriving simultaneously with a technical shift (mass adoption of ad-blockers, which Doc Searls calls the biggest boycott in human history) and an emerging normative shift in the form of the #DeleteFacebook movement, which is on its way to treating your friends’ insistence that you use Facebook to communicate with them as akin to insisting that you inhale their second-hand smoke as a condition of socializing with them.

– The GDPR might actually create an “attention economy”, Cory Doctorow via BoingBoing (emphasis mine)

I’ve talked about social media before. My feelings about it have been on a steady decline since 2011 or so. They’re pretty rock bottom now. What’s the benefit? Why should I waste my time to perpetuate knee-jerk outrage and be advertised to and tracked? I like Cory’s point: social media is the cigarettes of the modern era - culturally pervasive, seemingly innocuous in small doses but with growing evidence of addiction and harm.

Actually, some backstory:

In 2004, I joined thefacebook. In 2007, I discovered the digital civil liberties crowd (mostly via Cory Doctorow’s Creative Commons-licensed sci-fi). In 2010, I started thinking I should delete Facebook. I was noticing that I would “perform my life” for status updates and likes, that I would lurk on the profiles of ex-classmates and compare myself to them (and mostly feel bad, or feel a sick gloating feeling), that I was spending way more time on it than I wanted to. In 2013, I joined Twitter for work, had fun - and then Gamergate happened.

In 2014, I deleted Facebook - prompting one hour of panic, followed by years of contentment. I’m still on Twitter, since I feel like I need to “market” my “tech person” “brand”, but I’ve been implementing all sorts of “tying Odysseus to the mast” stuff to limit its dark patterns, and generally treating it like a dangerous animal that I’m locked in a cage with. (Uh, the cage is my mind? Or societal norms?)

More ranting
#

One narrative is that social media is on its way to derailing our democracy - thanks, 2016 US election. The addictive potential and the automation of propaganda has led to the rise of “alternative facts” and multiple realities; the power of leveraging social networks to reiterate people’s inwardness and cocoon everyone from different realities, only exposing them to the extremist worst of the “Other” (e.g. “You wouldn’t believe what this girl brought to her Kent State graduation!” - saved you a click: someone was obnoxious somewhere). The monetization of outrage! It’s the stuff of dystopias.

I’m actually equally horrified by two things: (1) the global pervasiveness of the social media addiction, and (2) the way my most social media-active friends seem less themselves and more homogenized mouthpieces for whatever the latest outrage meme is.

On the latter: After the 2016 election, especially, I started to notice that some of my friends - especially the most outraged and social media-active - would tell me stuff that felt… packaged? Like, they didn’t feel like themselves to me anymore, not like they did 5 or 10 years ago: their personalities were getting subsumed by whatever Facebook/Twitter/IG meme was flowing around that day, Invasion of the Body Snatchers style. Like, since when do you know what Hamilton said in the Federalist Papers? Wait, and why are you mad about X, but not about equally-repellant Y? Wait, where are you getting your news?!

Then I remembered. Back in the early 2010s, I often thought that Facebook was “where I get my news. How will I know what’s going on in the world?” I think a lot of people think that way. But that is, ahem, VERY BAD.

Like, they’ve become the amalgamation of their Social Media Outrage Diet - they don’t have their own worries, their own thoughts. They don’t pursue their causes; they pursue the causes that an algorithm designed to maximize their engagement (for advertising dollars) has fed them. That’s what it feels like. I know, I know, I’m on my high horse, I’m reading minds, I’m just as manipulated by cultural fads as everyone, blah blah. This is just a feeling I have.

None of my friends have quit social media. My feelings are generally pessimistic; that I’m screaming into the void about it. Oh well.

Tutorial: Defang Twitter
#

So, anyway. Imagine, if you will: Thunderdome. Me and Twitter are in the cage together, circling each other. How can I win!?

The best way to defang Twitter (and any social media), I think, is to defang all its “viral” properties. I don’t need to see the latest tweet that got 100k retweets and 200k likes by someone I don’t know or haven’t explicitly followed. It probably went viral because it scratched some reptile-brain itch in my in-group; some outrage about “the Other” (e.g. the Trump voter, the coastal elite) being horrible, or some pithy witticism. I can live without pithy witticisms! There is more to my life than that. I want more to my life! Okay, enough ranting.

How to turn off retweets
#

Twitter obviously doesn’t want you to do this. They want you to be “engaged” (i.e. to be on the site), addicted. Thus not seeing viral stuff goes against engagement. Anyway, first: turning off retweets. You can do this manually, profile by profile, for each person you follow. This is intentionally onerous.

Or, you can use Julia Evans’s little webapp, https://turn-off-retweets.glitch.me/, and it’ll automate through your list of follows and turn off retweets. Twitter gets MUCH MORE QUIET. Good!

How to turn off `so-and-so liked`
#

This is the same deal: you need a bit o’ JavaScript and HTML tags. I used AdBlockPlus (which already blocks stuff using HTML tags, I’m guessing).

On Chrome, go to:

** AdBlockPlus -> Options -> Advanced -> Edit Filters**

And add in:

twitter.com##[id^=stream-item-recap_entry-]
twitter.com##.tweet.js-stream-tweet.js-actionable-tweet.js-profile-popup-actionable.dismissible-content.original-tweet.js-original-tweet.tweet-has-context.PocketAdded

Looks kinda Greek to me. Here’s what I think it parses into:

twitter.com## = Anywhere on Twitter.
[id^=stream-item-recap_entry-] = Any HTML element whose id tag starts with stream-item-recap_entry-. (I’m guessing that caret, ^, is regex?)
[.tweet.js-stream-tweet.js-actionable-tweet...] = Any HTML element with any (or all?) of those classes (e.g. .tweet class).

You can also do some trial and error with AdBlockPlus’s Block Element button.

Adventures in crappy architecture, part N

Thu, 10 May 2018 00:00:00 +0000

Remember when I said how great it was to make a IPython startup file? Remember those good ego feelings? Yeah, well, I went on vacation and came back and it broke. Goddammit!

Not only did it stop working, but it stopped working in a weird, dark magic way.

Friendly reminder
#

The point of the IPython startup file was that, every time I opened a new ipython shell or Jupyter notebook, it would automagically (and, in the notebooks, silently) import a bunch of packages I wanted (pandas, numpy, etc.) as well as a couple convenience functions (e.g. alert()). This would spare me having to copy that cell over every time. Hence, it was a minutes-saving, productivity-enhancing tool for near-daily use.

The problem is that it fails silently in the notebook (it fails noisily in the ipython shell). So, today, I happily opened a new notebook, double-checked a couple of my expected packages (pd returned the pandas object, we are go for launch), and proceeded.

And then - it broke. Specifically, my notebook complained that it had never heard of alert(). Huh, what now? It’s in the startup file. How can you see import pandas but not def alert()? How does that make any sense?!

Fixing it, step 1: `ipython` > notebook
#

No error message means no clue, so I had to find the error message. The first place I looked was the Jupyter notebook terminal output:

Uh, what? [IPKernelApp] WARNING | Unknown error in handling startup files:… and nothing?

This is where ipython shell is handier:

Ooooh.

Note: That sqlalchemy import is not the actual error. I had a much weirder error in a package I had written. Namely…

Fixing it, step 2: circular imports because I no design software so good
#

I’ve had something of a dark obsession with design patterns in software for the past ~8 months. Mainly that I don’t know any, but I know they’re out there. My colleagues always talk about “code smells”. WHAT IS A CODE SMELL? Things like that. I watch YouTube videos. This one by Sandi Metz was good - but I just don’t write that much Actual Software to have enough practical experience to know this stuff. So I’m always hungry to learn more and get better at it.

One “pattern” (SCARE QUOTES) I’ve decided to adopt is putting global vars at the very tip-top of ~~the American pyramid~~ my Python package, i.e. in the __init__.py. Then I can import them in all sundry modules within the package. So I had something like this:

# __init__.py
from package import module1
from .top_module import Class1, Class2

# Global vars
STATISTICAL_SIGNIFICANCE = 0.95
DEFAULT_POWER = 0.9

But then, in my module1.py, I had this:

# module1.py
from . import STATISTICAL_SIGNIFICANCE, DEFAULT_POWER

Oh no. In other words, when I imported my package, the following would happen:

package runs __init__.py to get everything started.
First thing, it tries to import module1.
In module1, it tries to import STATISTICAL_SIGNIFICANCE and DEFAULT_POWER
Error! It doesn’t know what those global vars are! Because they’re defined further down the __init__.py.

So the __init__.py initialization fails, and everything fails, and - because I was importing this package about halfway down into my IPython startup file, it failed and only managed to import a few things for my Jupyter notebook. As you can guess, the alert() function definition was further down my startup file. It never got there.

So here’s my fix!

# __init__.py

# Global vars
STATISTICAL_SIGNIFICANCE = 0.95
DEFAULT_POWER = 0.9

from package import module1
from .top_module import Class1, Class2

Voila! Easy! No more circular stuff. Global vars are defined first thing. Architecture! Design! SUCCESS!

Sourdough (2017) - ⭐⭐⭐⭐

Fri, 27 Apr 2018 00:00:00 +0000

Metadata
#

Author(s): Robin Sloan
Number of pages: 259
Year published: 2017
Year read: 2018

Review
#

A funny-enough, strange, and well-observed book about Silicon Valley narcissism/tone-deaf transhumanism. It starts VERY strong, but progressively loses steam and ends on a bit of a whimper.

Robin Sloan has a really sharp eye and satirizes the tech industry super effectively: e.g. the romanticized treatment of being “on the spectrum”, i.e. mono-focused ~~PASSION~~, i.e. sleeping at the hipster office so you can churn out code, i.e. a work culture that mostly privileges young men without dependents. The post-human/Singularity types who eat utopian/post-apocalyptic non-food since our evolved digestive system is inefficient and evolution should be ~~DISRUPTED~~. The fetishization of productivity. Even just the frisson of merging code changes to the /master branch without even making a PR! (MADNESS.)

The novel - which focuses on a young programmer named Lois, who codes real good but is pretty repulsed by this Bay Area silliness - is a cry for hedonic pleasure, for inefficient and uncontrollable organisms, for YEAST. Which is great. And I was like, “oh wow is this a smart, fun book or what, heheh that bay area amirite”. And I read on, eagerly anticipating the great, philosophical punchline that I just KNEW had to be coming.

And I read.

And I read…

And, basically, it never comes? Which is why this was a 4-star instead of 5-star experience. The setup is so superb, our crazy world so accurately observed, that I was SO READY for that philosophical whammy. Hoo well.

📚 Goodreads

Default Jupyter startup

Wed, 25 Apr 2018 00:00:00 +0000

I have a running mental list of little inefficiencies that annoy me throughout my workdays. The sand, the pebbles, the annoyances of my working life. I try to follow this xkcd in determining if it’s worth the time to actually fix them. The basic heuristic is: if it’s a small thing, but I do it every day, it’s worth spending a few hours optimizing it.

Today I tackled one: creating a global config file to auto-import all my favorite modules into any newly-created Jupyter notebook.

Background
#

I work a lot with Jupyter notebooks. They’re a standard data science tool. What usually happens, though, is I experience “Jupyter efficiency drift”. I’ll spend a chunk of time (several days/weeks) digging into notebooks, and I’ll get really good at importing the right modules, setting my matplotlib up all nice and friendly, and making little IPython JavaScript helper functions. Then I’ll spend a chunk of time on something else - like deploying stuff (gettin’ real good at bash!), or stats stuff (yay Bayes!), or whatever - and I’ll have forgotten what my One True Jupyter Notebook setup is.

My crappy workaround has always been to dig up the last notebook I was working on, and copy the top few cells:

%load_ext autoreload
%matplotlib inline

import os
import requests
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib as mpl
from IPython.display import Javascript
[blah blah blah]

mpl.rcParams['figure.figsize'] = (12, 8)
plt.style.use('bmh')

def alert():
 js = """alert("All done!");"""
 display(Javascript(js))

%autoreload 2

The problem was - some notebooks didn’t have the entire rigamarole, some were missing my beloved alert() function, and this was a stupid process. Sometimes I’d open a new notebook just to do some small thing, and it seemed like overkill to stay meticulous about copy+pasting that One True First Cell.

Solution: `~/.ipython/profile_default/startup`
#

Because surely Jupyter must have some top-level, global-environment-style config that can do this for me? I mean, Stata has profile.do, bash has .bashrc/.bash_profile, is this some crazy idea? To paraphrase Kenneth Branagh: no, the world must be dotfiled!

And, indeed, Jupyter - or rather, Jupyter’s ur-center, IPython - has startup files (docs). The basic process:

Go to your default IPython profile’s startup folder: cd ~/.ipython/profile_default/startup
Make a Python script for what you wanna import: subl 0-the-world-must-be-peopled.py (Note: You can have multiple scripts, they’ll be run in lexicographic order.)
People that script with your imports! Mine looked like the above import os etc. stuff. (Note: Cell %magics don’t work here, sorry.)

VOILA! PRODUCTIVITY GAIN.

An aside on `alert()`
#

Hey, let’s talk about alert(), by the way.

I often run stuff in my Jupyter notebooks that takes a while. For example: if I’m pulling in lots (and lots) of data from somewhere, and it’s just taking forever to load in memory. Or if I’m waiting for a PyMC3 MCMC sampling, or a neural net’s many epochs.

In those cases, I want to be alerted when the process completes. Specifically, I want it to pull me back from whatever other thing I’ve wandered off to do (e.g. watching this). BEHOLD - that’s what alert() is good for:

from IPython.display import Javascript

def alert():
 js = """alert("All done!");"""
 display(Javascript(js))

def some_long_process():
 sleep(100000)
 return 1

some_long_process()
alert()

So alert() just launches a JavaScript alert window whenever some_long_process() completes. You may want to manually allow popups in your localhost via your browser, since (I guess) most modern web browsers block them by default (after much 1990s abuse).

I mean, you’re in the frickin’ browser anyway, may as well leverage that JavaScript!

Such zsh merit

Wed, 25 Apr 2018 00:00:00 +0000

I just kicked some serious computer butt today. I would like to share.

Original problem
#

I just wanted a way to recursively .gitignore some .csv files in my work folders. CSV files, as you know, can get big (BIG). And GitHub doesn’t like that; in fact, it was bouncing my git push -u origin master back - and complaining that I had exceeded the acceptable size of my commit (wow). My previous .gitignore always only excluded CSV files that were exactly two levels deep:

path/to/top/folder/*/*.csv

But this was no longer true. I had just massively reorganized my folders, and my CSVs were now scattered arbitrary-levels deep in my various project folder trees.

WHAT TO DO?

Well, first, this works in your .gitignore:

path/to/top/folder/**/*.csv

Specifically, that double wildcard, **, says “wildcard recursively”. That is, “search through arbitrary levels of sub-directories”. What joy! Which led me into a rabbit hole: wait, can I wildcard recursively in my shell?

Yes, but it’s time to move to `zsh`
#

I just finished a very nice book about bash, and so I didn’t want to haul over my well-earned shortcuts and aliases and all that onto a totally new shell interpreter. Ugh! (Though, to be fair, wiki notes that zsh is a superset of bash.)

But I saw some StackOverflow post saying that zsh has the double-glob (**) built-in, and that - if I wanted it in bash (my current/the default shell interpreter), I’d have to download something or other. So instead of downloading globstar, I decided to pull out the GUTS OF THE SHELL ENTIRELY and just install zsh.

LET’S DO THIS.

Installing `zsh`
#

zsh is just another shell interpreter, with some nice add-ons and wing-dings that make it a superset of bash. I found oh-my-zsh, which is a “community-driven” zsh configuration helper - basically, people have already futzed with zsh profiles to make things pretty and functional and allow autocomplete, and so on. In general, I think it’s wise to follow crowds on tech stuff - where goeth the crowd, there be-eth the StackOverflow answers.

Anyway, here’s me installing zsh and getting going:

# OSX has zsh installed, but I saw somewhere that it's old? so let's update
brew install zsh
# change default shell from bash to zsh
chsh -s $(which zsh)

Then close and reopen iTerm2! Next was futzing around with the oh-my-zsh stuff, by enabling oh-my-zsh, as well as its plugins and themes in my .zshrc. I also copied over all my various aliases and such from .bashrc and .bash_profile to .zshrc. Everything basically worked fine. (Yes, I’ll put my dotfiles on GitHub someday.) For prettiness, I used the pure prompt theme - which is not part of oh-my-zsh, and so needed some overriding. Here’s what it looks like, when I’ve got my pyenv environment activated (www_env), I’m on a git master branch, and I’m in the blog folder:

Doing the thing I came here to do
#

After spending an hour or so getting zsh all ported over, all I wanted to do was remove the giant CSV files from my git history. They were several commits behind in my local branch’s history, and I was stuck: I needed to roll back to the earliest commit (pre-CSV dump) and remove them (temporarily!) from the directory entirely. I needed to rewrite history. Since I had added the path/to/top/folder/**/*.csv line to my .gitignore, whatever changes I made were not being picked up by git.

So my plan was:

Banish the CSV files to a temporary data/ folder.
Roll back to an earlier commit (pre-CSV dump).
Re-commit everything, now without any CSVs in the git history.
Push up to origin/master! 😍

In zsh, all that was:

# 1. banish any CSVs in any subdirectory to new top-level data/ folder
find ./ -name '*.csv' | xargs -I '{}' mv {} data/

# 2. roll back to earlier commit, pre-CSV troubles
git reset [SOME_SHA]

# 3. re-commit everything
git status
git add .
git commit -m "OMG CSVS"

# 4. Pusshhhhhh!
git push -u origin master

# 5. OH SHIT THE CSVS, WHERE DO I PUT THEM BACK

At this point, my good ego feelings quickly disappeared. Since I had moved all CSVs out of any directory and plopped them down into data/, I no longer knew which CSV was associated with which directory (and thus, which project). I was horrified. And my CSVs had cryptic names like A.csv and eligible.csv. Huh!?

Thankfully, I knew that all the CSVs had been used (either created or imported) by some data analysis in Python. So I could just grep through all my .py files (recursively, throughout all the directories and sub-directories) to find when and where each CSV file appeared.

grep -rI '.csv' ./**/*.py

This showed me that, for example, Python file X in directory Y had been doing stuff with A.csv - voila! I could return A.csv to its rightful directory. And now, since all CSVs were .gitignored, I would never have to worry about this again!

CS50 - No more lectures 😭

Wed, 11 Apr 2018 00:00:00 +0000

I was surprised recently to see that I’m basically on track to finish CS50 with the end of Harvard’s spring 2018 semester. I was surprised because, back during the C period, it felt like I was wallowing away weeks and weeks on some of those problem sets. But, well, they had spring break and a couple off weeks, and so I caught up.

I finished up the lectures and problem sets this week. The final lecture was actually just a pile o’ fluff: opening with one of the TAs playing the piano, and lots of pop music montages of “oh the places you’ll go!” stuff. And, funnily, I WAS VERY MOVED. I felt like it was the last day of high school again! SO LONG, CS50, I’LL REALLY MISS YINZ! Thanks to Prof. Malan and all the TAs and staff for a wonderful resource, truly!

## Syllabus - in review

The full 2018 CS50 syllabus is here. In retrospect, my favorite topics were, ranked descending:

Memory: Huge demystification of what happens when you computer things.
Dynamic programming: I want to write a post about this. I had to watch some supplementary YouTubes about this (since I didn’t get it with just the lecture), but - once it clicked - OH MAN. Mind blowing. And the applications! Aaah. Just look at this.
Data structures: Unexpected elegance, thanks to C’s direct memory management, of linked lists and tries and hash tables. Wonderful!
HTTP: Insufficient HTTP stuff, though I did discover the fun of traceroute.
Algorithms: Meh. This was one lecture. It was okay-fun. But, since we didn’t actually implement any algorithms or “race” any code, it felt remote.

## Problem sets - in review

Pset	Language	Joy	Est. hours	Notes
0 - Scratch	Scratch	🙄	<1	Bah.
1 - C	C	🤔	3	Greedy algorithm, oof.
2 - Crypto	C	😂	2	Already did this once in Clojure.
3 - Music	C	😨	5	HARD. Converting text -> Herz frequencies -> MIDI music. Also, waves.
4 - Forensics	C	😨	5	STILL HARD. Recovering lost JPEGs and bitmaps. Also - I KNOW ALL COMPUTER NOW.
5 - Mispellings	C	😨	5	Hard. AKA how to build a linked list. Not sucking at `malloc()`.
6 - Déjà vu	Python	🤔	4	Real easy Python + understanding dynamic programming!
7 - C$50 Finance	Python, SQL	😒	2	Flask app + API calling. Fiddly, annoying.
8 - Mashup	JavaScript	😒	2	JAVASCRIPT. BAROQUE CATHEDRAL OF CODE.
TOTAL			29

## Final project

I’m about halfway through Build Your Own Lisp, which is a lot of fun. I’m using it to write a toy language in C called triestin. It’ll be based on the Triestine-Italian dialect, which is spoken by ~250,000 people. According to ISTAT, the Italian government’s statistical agency, 56% of Italians of working age are employed, and 2.6% of those (I think) are ICT people. So that’s, hmmm -

$$250,000 * 0.557 * 0.026 = 3,620$$

Or about 3,600 people that could potentially understand this language at all! If we whittle things down to people that are interested in compilers, IT’S EVEN LOWER. Yay!

Anyway, I also found Destroy All Software’s wonderful A compiler from scratch, which demystified how programming languages are written: it’s just tokenizing, parsing, and compiling! Wooooo. Lots of Chomsky stuff too, which I will investigate further.

2600 Magazine- The Hacker Quarterly - Winter 2017-2018 (2018) - ⭐⭐⭐⭐

Fri, 06 Apr 2018 00:00:00 +0000

Metadata
#

Author(s): 2600 Magazine
Number of pages: 238
Year published: 2018
Year read: 2018

Review
#

What’s super interesting about 2600 (which is a long-running magazine for l33t h4x0rs) is how much it looks and smells and sounds like a punk magazine. Wow, is the aesthetic the same. The posturing ego, the slightly obnoxious tone, the idealism, the rebelliousness. See one of my fave Portlandia skits.

Anyway, this was fun and got me suitably paranoid about WHOIS info (aaagh) and suitably inspired about maybe setting up some sort of antennae to emit destructive interference to my neighbors’ WiFi signals JUST BECAUSE I CAN. Actually, okay, I don’t know if you can do that. Wouldn’t that eff up the 2.4GHz spectrum around you for everything? And it’s probably illegal (“probably illegal” being an important slogan of the scene).

I liked the article about the guy checking a Linux OS’s command hashes both upstream (from the OS he forked from, I guess) to downstream (the OS he was the main builder of), since people were spreading rumors that using that command secretly sent secret info stuff to The Man (which is such a a Mr. Robot idea in and of itself). The guy checks the SHA-1s of the command as it was downloaded and as it’s deployed are identical - i.e. no one futzed with that command. I didn’t know commands even HAD SHA-1s, but that makes sense and WOW THAT’S COOL.

📚 Goodreads

Two announcements

Tue, 03 Apr 2018 00:00:00 +0000

Well, troops, I have good news, and I have good news.

First good news: SSL
#

Time to join the modern web! I’m happy to announce that angelaambroz.com is now SSL certified.

Second good news: Georgia Tech’s OMSCS
#

I’m also happy - and nervous and excited - to announce that I was accepted into Georgia Tech’s Online Master’s in Computer Science. Holy moly, Morty!

This was something I had been thinking about for about four years, and actively working towards for three. My background is in economics, with an okay seasoning of math (e.g. my undergrad minor was math). But I had only ever taken one CS course previously, and it mostly talked about how to fashion a spinning hard drive out of vinyl, wood, and the blood of your clan enemies. Ho ho, kidding, little joke to convey how old I am.

Anyway, I didn’t have much CS background. So, in 2015, while I was working at the Harvard Kennedy School, I started taking some CS courses through Harvard’s Extension School. These courses were hugely helpful. I also started building out some additional CS foundations with free online courses, like CS50 and a bunch of other bits and bobs. But now we’ll see - the proof will be in the next few years’ pudding!

Big plans, Morty
#

I’m super pumped about the OMSCS, specifically about pivoting my career more towards artificial intelligence. I’m really excited for the courses in computer vision and AI for robotics. I’ve actually been actively strategizing all the courses for about 18 months (I know, too long). OMSCentral is a great resource.

Regression discontinuity okay
#

Finally public! Exciting new work w Mandy Pallais & @jmelkers: "Does Online Delivery Increase Access to Education?" https://t.co/OU5bhVnnXv pic.twitter.com/LwDc59anKd
— Joshua Goodman (@JoshuaSGoodman) September 26, 2016

Joshua Goodman is a professor of public policy at HKS, and he’s been doing a study on the effects of OMSCS. The tldr is: he, with co-authors Julia Melkers and Amanda Pallais, use a regression discontinuity design to create a “treatment” of admitted students (woohoo) and a “control” of those who just missed an (unknown, but it’s in the paper) GPA admissions threshold. They then compare outcomes for both. The really interesting stuff is in the descriptive stats: OMSCS applicants and students are older (average age of 33), mid-career-er, male-er, American-er, and from more diverse educational backgrounds than their on-campus counterparts. Many don’t apply to other graduate schools. The main finding is:

“By satisfying large, previously unmet demand for mid-career training, this single program will boost annual production of American computer science master’s degrees by about eight percent.”

– Goodman, Melkers and Pallais, 2016

EIGHT PERCENT, damn. That’s a lot. That’s insane.

I wasn’t too surprised by the fact that OMSCS applicants are older and more mid-career and non-traditional - that seems to be how the program was pitched. It was certainly how I heard about it: in the Meetup.com circles of ambitious career transitioners in their early 30s. I’m very surprised that the OMSCS has a worse male/female ratio: only 15% of OMSCS applicants are women, compared to 25% of applicants to the on-campus program.

I’m surprised for two reasons: first, the program seems pitched towards career transitioners, and there is a big cultural push right now to get more women into tech. Ergo RailsBridge, Women Who Code, PyLadies, and so on. Maybe it’s the circles I travel in, but I feel like there are a lot of women entering tech right now?

Second, the OMSCS’s flexibility is perfect for, well, working parents in their 30s! In fact, one of the main reasons I was so excited by the program was that it completely fit into my personal life: I wouldn’t have to uproot myself and my husband to pursue my AI/robots dream. Uprooting yourself is easier in your 20s, but once you’ve got the two-body problem, everything gets stickier. The OMSCS is perfect for that.

In bocca al lupo
#

Me, after learning I got in. Champagne!

A very Dirichlet journey

Thu, 29 Mar 2018 00:00:00 +0000

I’ve been learning about hierarchical models in Bayesian statistics lately. This has meant digging into some YouTube micro-lectures, some Gelman, some PyMC3 documentation. And! Revisiting some of my old problem sets from CS109B.

Ye olde pset: Text prediction with naive Bayes
#

The old problem set had two main problems. The first was a text analysis/prediction problem. Specifically, given 100 articles and two authors, use word frequency and Bayesian methods to predict which author wrote which article. I can see how this could be extended to, for example, automated categorization of articles or maybe even spam filtering. The methods used were, first, a naive Bayes approach - which assumes that word frequencies are independent of each other (probably not true) and just tries to measure the likelihood of observing some words, $w$, conditional on that author (or category, or whatever), $C_{k}$.

Here it is for multinomial naive Bayes (looking at word count frequency): $$p(\mathbf{x} | C_{k}) = \frac{(\sum_{i} x_{i})!}{\prod_{i} x_{i}!}\prod_{i} p_{ki}^{x_i} $$

Which I stole from Wikipedia and is saying:

The likelihood of these words, $\mathbf{x}$, conditional on it being this person, $C_{k}$, (where we have $k$ people/categories/whatever), is
The sum of word frequencies, factorial
Divided by the product of word frequencies, factorial
Multiplied by the product of probabilities that that author used that word ($p_{ki}^{x_i}$).

Phew! But basically, we’re doing our usual Bayesian thing: we observe some data ($x_i$, word frequencies), and multiply that by our prior ($p_{ki}^{x_i}$, the probability that author $k$ uses those words). The assumption is that authors use words at different probabilities. The other big assumption is that we can just do simple multiplication, since these word likelihoods are independent. That’s probably not the case (e.g. words come in phrases, pairs, other stuff).

Dirichlet refresh
#

But hey! What if we just know that that assumption of independence can’t hold. Language comes in chunks, after all! Not. Independent. Words. In that case, the pset had us work through a Dirichlet model.

I really like Dirichlet models! They’re the n-dimensional extension of my favorite model, the beta-binomial model. They’re also (holy music) a model with conjugate priors. That is, something that has an analytical solution. You don’t need to do any Monte Carlo Markov Chain sampling (unless you want to, which whatever, you could want to). But the Dirichlet-multinomial, like the beta-binomial, is Dirichlet going in and Dirichlet coming out.

The Dirichlet is parametrized by an n-dimensional vector of $\alpha$ - this is similar to the beta being parametrized by two parameters ($\alpha, \beta$). Same deal.

Here’s a helpful comparison visualization, courtesy Wikipedia:

From a 2-dimensional beta…

…to a 3-dimensional Dirichlet! Ta da!

So, just like you can set an uninformed prior by using a beta distribution of $Be(1,1)$, you can do the same for the Dirichlet: $B(1,1,1,1,…)$. It’ll be a flat line or plane!

The benefit, of course, of using the Dirichlet is that you can now model all those words jointly: you can say, given this big long list of words that author $k$ wrote, what’s the joint probability of those words?

Porting from R to Python
#

Alright. So, I do think that building stuff from scratch is a valuable way to learn. And translating my old R pset into Python is a valuable way to dig in. But, oooof. This took forever! It forced me to think through each step, and think about stuff like: (a) what libraries are available in Python with which I can give up? And (b) what is happening at stage N of this interminable math journey? Things like that.

Okay, naive Bayes was easy. Here it is in Python, courtesy sklearn (thanks, sklearn):

from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import confusion_matrix

# A little convenience function to make my
# confusion matrix labelled and in % terms (rather than absolute)
def confmat(actual, predictions, columns, index):
 cm = pd.DataFrame(confusion_matrix(actual, predictions), columns=columns, index=index)
 cm['sum'] = cm.sum(axis=1)
 return cm.apply(lambda x: x/x['sum'], axis=1)

# Fitting the model, predicting, running through the confusion matrix
mnb = MultinomialNB()
mnb.fit(X_train, y_train)
y_preds_mnb = mnb.predict(X_test)

confmat(y_test, y_preds_mnb, y_test.unique(), y_test.unique())

Err, I should have probably mentioned that my X_{train|test} are pandas dataframes of word count frequencies for a bunch of words. That is: each row is an article, each column is a word, and each cell is the number of times that word appears in that article. And my y_{train|test} are pandas series of the category (in this case, the author’s name): each row is the author’s name.

Fine. That’s all fine. I recommend you try this out. Maybe using the Enron email archive - see if you can predict email senders! Woohoo. Or analyze Trump’s tweets and see which are written by him vs. others. (“Forensic linguistics”, ooh la la.)

Anyway. What became horrible was when I decided to port a mammoth R function (which was given to us in the pset by the TAs - i.e. we weren’t meant to code this from scratch) into Python.

Oh noez overflow!
#

The Harvard TAs had written this gargantuan function to manually calculate the Dirichlet-multinomial posterior for each article. That is, the function would return a floating point likelihood of whether that article was author $A$ or author $B$. I ported it into Python, but won’t copy it here (probably violates Harvard copyright?).

But I did learn some stuff, which relates back to my CS50 learnings: MEMORY ALLOCATION.

So Python is dynamically typed, i.e. duck typed, i.e. if it looks and quacks like an int, it’s an int! Which is fine and well for 90% of what you do. But sometimes you want to be explicit. Sometimes you also just plain overflow.

Overflow is when you run out of memory. Say you want to hold a really big number in memory. Like, $e^{710}$. Well, you can’t. There aren’t enough bits in your memory for that to happen. Sort of. Python actually, for integers, will just take up ALL the memory available and, dammit, give you your number. But this breaks down for floating points. Because, then you need all the numbers to the right and the left of the decimal point - and that’s a tradeoff! You tradeoff size and precision. This is also why some floating points get “rounded” by Python under the hood - for example:

# Ten decimal places: all looks well
print(f"{1/3:.10f}")
# 0.3333333333

# Twenty, and we see the lies we've been told
print(f"{1/3:.20f}")
# 0.33333333333333331483

ANYWAY. All this to say, I was trying to make my Dirichlet posterior function, and it kept crapping out at, yep, $e^{710}$. And the way it crapped out varied by basic Python vs. numpy vs. my eventual savior, the decimal module (docs).


import math
import numpy as np
import decimal

math.exp(709)
# 8.218407461554972e+307

math.exp(710)
# ---------------------------------------------------------------------------
# OverflowError Traceback (most recent call last)
# <ipython-input-11-0ad1e5224725> in <module>()
# ----> 1 math.exp(710)
# OverflowError: math range error

np.exp(709)
# 8.2184074615549724e+307

np.exp(710)
# RuntimeWarning: overflow encountered in exp
# inf

decimal.Decimal(709).exp()
# Decimal('8.218407461554972189241372387E+307')

decimal.Decimal(710).exp()
# Decimal('2.233994766161711031253644458E+308')

Ah ha! Bottom line, use decimal and you may math with ease. Or use numpy for their amusing “so big, basically infinite!”

Progress pics- Portraits

Sat, 24 Mar 2018 00:00:00 +0000

I really adore progress pics, which is a thing I discovered when I discovered (or, rather, figured out how to use) Reddit. I know, I know. Don’t shame me, I love r/ImaginaryLandscapes and r/colorizedhistory and there are lots of good, sane subreddits.

Anyway. Progress pics are great - super inspiring and motivating. So I wanted to include a few “progress pics” from my drawing earlier this year. I’m trying to improve my general craftsmanship on a bunch of stuff: proportion, perspective, shading, color. I’m also trying to branch away from portraits, which I do too much of. Here’s some more thoughts on my Art Journey, including the tools I’m using.

Despite trying to move away from portraits, here are three portraits - made one after the other - of the wonderful Star Trek: Discovery characters.

First, Michael Burnham (played by Sonequa Martin-Green). Reference pic here.

Next, tentatively trying my new colored pencils on Gabriel Lorca (played by Jason Isaacs). Reference pic.

And finally, getting bolder at colors, Philippa Georgiou (played by Michelle Yeoh). Reference pic.

I see a bunch of flaws with the above - but I’m happy with how much each is an improvement on the last. I’m also happy that, in general, they resemble their reference pics (and the actors), even if it’s still not perfect. I really loved doing the colors on the Georgiou portrait, especially around the eyes. It blows the Lorca portrait colors out of the water! So that’s nice.

PyLadies DC - Intro to data science workflow

Fri, 23 Mar 2018 00:00:00 +0000

I led a tutorial for the DC PyLadies group titled, An introduction to the data science workflow in Python. Here are the materials, which I hope will live on in usefulness.

The workshop covered: (1) getting data (with examples of CSV loading, SQL querying, and API calling), (2) cleaning data (with some visualizations in matplotlib), and (3) modeling data (fitting a linear regression using three libraries - statsmodels, scikit-learn and plain ol’ numpy - and confirming the coefficients are the same).

Prep
#

My goal for the workshop was to give a “tour” of the common Python packages used in data science. This was aimed towards women who might have some experience doing data analysis in R, Stata, or something else, and thus needed to get a handle on where to find their favorite stats/machine learning methods in the world of Python (rather than how those methods worked - i.e. I didn’t want this to be a stats class).

I ran a survey beforehand to also get a sense of how new to Python attendees were (by asking a couple “beginner” questions, like, What does import some_library do?), what other programming languages they knew, and what they were most excited about learning. This helped a lot, since I knew what I wanted to present (supply!) but I also knew I wanted to meet folks where they were at (demand!).

Logistics
#

I used Ned Batchelder’s pizza.py script to calculate how many pizzas to order. Ned organizes the extremely popular (and very great) Boston Python User Group, and he is wise in the ways of Meetup. Specifically, notorious Meetup attrition (only about 50% of RSVPs usually show up). Anyway, would you believe it - we had <1 pizza left over - pizza.py really came through!

To bypass Installation Hell - wherein 33% of a workshop is spent trying to get some version of Python installed on some attendee’s machine - I used Binder, which I found via Julia Evans’s blog. Binder is SUPER convenient: you make your repo with your Jupyter notebooks (and a requirements.txt), point Binder to it, and it’ll launch that repo as a Docker container that people can access and interact with. This was really great, since we could skip the installation pain and get straight to it.

The challenges of pedagogy
#

I love teaching, and I love learning. I also love thinking about learning - that mystical moment when you get something - and reflecting on how and why Things Are Learned. Teaching is a special challenge because you want to be engaging (fun!) and clear. (David Malan, who teaches CS50, is a master at this.) Being clear, especially about stuff that either is complex or is commonly perceived to be complex, is hard. I think it’s very important to be clear - most big human ideas can be boiled down to something quite simple. I strongly believe in this ad by Khan Academy - human knowledge is a set of simple building blocks. Also, to paraphrase Richard Feynman, if you can’t explain it in simple terms, you don’t understand it.

So, anyway, I tried to do that - but it was pretty exhausting. I was definitely pooped by the end - and dehydrated! I did appreciate the opportunity, though, and hope to get better at it. Bottom line is I find a lot of inspiration in teaching/learning, and have become super growth mindset lately.

CS50 - Data structures achievement unlocked

Tue, 20 Mar 2018 00:00:00 +0000

Well, Prof. David Malan liked my tweet so…
#

A volcano. A plume of smoke roars up into the steel grey sky. Angela walks towards you, framed in red fiery light, carrying - on her shoulders - the dead carcass of a hash table of linked lists. This is @cs50 pset5! She raises her C sword and frees some more memory. "Raaaah!"
— Angela Ambroz (@angelaambroz) March 20, 2018

I’m so pleased.

Data structures are complicated
#

I’m also pleased because I feel like I reached peak CS50 this past week. Where by “week”, I mean “month”. Life happened, but this was also the densest part of the course and I wanted to take my time. It was hard, brain-sweaty work and it was awesome. And it was over far too soon. I want to take a full course on data structures and algorithms, because the ~6 hours of struggle on problem set 6 were definitely not enough. Definitely need more hours on this. MORE HOURS.

For my problem set, I implemented a hash table with linked lists, and fumbled through malloc() (nightmare of malloc()!) and freeing memory. Apparently, I free up too much memory every time. valgrind (a command-line utility which checks your C program for any memory leaks) usually complained that I had allocated space for 3 things and freed 5. Woops. I JUST WANT TO BE SURE.

Pedagogical meta-ness
#

These “Dense Materials, Ahoy!” lectures are longer (1hr 45min) than the earlier weeks. The “shorts” - additional videos where a CS50 staff member goes in more detail - are, cumulatively, another ~2 hours or so. This is a lot of content! It takes forever, but - again - I loved it and feel hashtag-blessed that we have this resource available, for free, on the intertoobs. I really agree with this gushing Reddit post, too - the problem sets are challenging grinds, but they’re not impossible, and I feel like I learn a lot from them.

Main takeaways
#

Welp, now I know how to define my own data types in C with typedef struct. Wooo hooooo.
Learned about a bunch of data structures: linked lists, hash tables, tries, stacks, queues. Only implemented a hash table with linked lists, and still feel shaky on pointing my pointers to the right places. I wanted to try a trie, but - at 6-hour pset mark - I was pretty pooped.

Looking ahead: Lectures & psets
#

I feel like I just climbed a hill in my bike, and my thighs and lungs are burning, and I’m pretty sure I’m gonna die - but now it’s the speedy, easy, thrilling descent. The next problem sets are all Python, JavaScript and SQL. i.e. My old friends. The next three (!) lectures don’t even have a pset. So the course’s marginal added value to me will be lower in this second half.

Looking ahead: Final project thoughts
#

After the lectures and psets, there is the final project. I found this book the other day - Build Your Own Lisp - and am thinking of using it in my final project, since I’ve always wanted to know how “higher-level” programming languages work. e.g. The most widely-used implementation of Python is written in C, and I was always like, “But what does that mean?”

I’m also interested in investigating the intersection between natural and programming languages: specifically, every programming language I’ve seen is written in English. I wonder if there’s any effect, then, of how all our technology is written in English - e.g. is there a Sapir-Whorf effect where even fundamental computer science paradigms would be re-envisioned if the higher-level languages had been written in, say, हिंदी ?

(My prior is actually, no, the effect is probably limited, since low-level programming languages are just math and logic, which are a universal language. It’s all binary underneath, after all. Maybe there’d be some effect as you go further up the levels of abstraction, e.g. Python, Ruby, etc.)

Anyway, I’m really looking forward to more CS50!

Writing better Python

Tue, 13 Mar 2018 00:00:00 +0000

I started incorporating a few new tools into my programming lately, and it’s been rad. I wanted to share how and why I think these tools are helpful. They may be standard to software engineers, but - as a data scientist, and someone without a formal CS background (ahem, for now!) - these sorts of programming paradigms aren’t always as prioritized. But it’s good to change that!

I’ve been filling in my CS foundations with CS50 lately, and - beyond the big stuff like data structures and algorithms - I feel like C is giving me good “everyday” habits of being explicit about my data types and more serious about adhering to programming styles. Remember: code is read X times more than it’s written, so standardizing your style improves readability/makes you better!

I’ve started incorporating some better programming practices into my day-to-day, and I really like the results. Namely:

Logging
#

My style of coding - which I assume is pretty common - is something like:

(Semi-optional) Pseudo code/sketch something out, if it’s complicated-seeming
Bash something out in Python/Clojure/JavaScript/C
Run it and see what happens
Try to understand inevitable error message

In the beginning, step (4) was mostly impossible, and so I sprinkled print() statements liberally throughout my code. Then I’d sort of anxiously watch the terminal to see what stdout would say as my program ran.

As I got better at reading error messages and generally more confident, I used print() less. But it’s still something that I’ll use when something breaks in a way I’m not expecting.

And that’s fine. But, at the same time, I saw how some of my senior colleagues used logging - and, generally, talked a lot about logs. Searching logs. Logs getting big. Finding something in the logs. Like… what are these logs you people keep talking about? Time to learn!

Python’s standard library includes the logging module. The general idea is that, instead of using print() statements - which write to stdout (i.e. your shell’s output) - and then disappear into the mists of time, logs are saved to a .log file for as long as you like, so you can peruse as much as you like. (Or, better yet, do cool data analysis on them.)

Logging in Python is really straightforward. The basic steps are:

Configure a log file: its name and file path, its formatting, and what it’ll pick up (there are six levels of logging in Python).
Sprinkle log messages throughout!
Profit!

import logging
logging.basicConfig(static=f"{__name__}.log",
 level=logging.DEBUG,
 format="%(name)s - %(asctime)s - %(levelname)s - %(message)s",
 filemode='w')

That is, configure a .log file named (dynamically) after the __name__ (i.e. the name of the Python module currently being executed), catch all priority levels (from DEBUG and up), format the output with the name of the logger, the local time when that log statement was written (asctime), the level of priority (DEBUG, CRITICAL, etc.), and the message (which you’ll define later). Do all this and write (w) to the file, rewriting if it already exists (you can change this to append, a, if you prefer).

Later in the code, I initiated my logger object and wrote my first message (an INFO message):

logger = logging.getLogger()
logger.info('Starting log...')

A .log file is born!

Yet later, while I was running a gnarly loop that tried to do some stats on a bunch of things, I logged every model like so:

def foo(thing_id):
 logger.info(f'ID {thing_id}')

 try:
 b0, b1 = some_fitting_function(data(thing_id))
 logger.info(f'Parameters: {b0, b1}')
 except:
 logging.warning('Failed to fit')

That way, I kept track of everything I was able to fit a model to, and everything that failed. By using a try/except block, my loop kept running. It was super handy, since I needed this to run for a while, for thousands of things, and I was actually trying to get a sense of how often it would fail to fit (and why).

Type hints
#

Python is dynamically typed - meaning you don’t have to specify your data types (string, integer, float), and they’re inferred when compiling. C, on the other hand, is much more explicit. Consider the same code in Python vs. C:

x = "hello"
y = 2
z = 0.14

char x[] = "hello";
int y = 2;
float z = 0.14;

Like, C doesn’t even have a string datatype! It only has an array of characters! So you have to literally tell it, hey, I’m going to make an array, x[], of characters, char, and those characters are h, e, l, l, o. Mamma mia.

Actually, I kid. I love that C does this. I love that an array name is the pointer to the 0th element. And I actually also love the types! It makes the code more readable.

So now I discovered the best of both worlds with gradual typing.

Google auto-complete drama

When I was recently trying to understand a big, wide codebase, I was starting to get annoyed that I’d have to constantly go up and down the function stack to try to understand what the hell that one argument was. It’s hard to tell what function foo(arg) is expecting for arg if it’s something like:


def foo(arg):
 # do some stuff
 return deep_foo(arg)

def deep_foo(arg):
 # do different stuff
 return deeper_foo(arg)

def deeper_foo(arg):
 # for the love of god
 # what is going on here
 return deepest_foo(arg)

def deepest_foo(arg):
 # arg is an int!! AN INT!!!!

I’d be spelunking up and down the function stack, trying to write down what arg was and keep it in mind. But for a lot of parameters and a lot of functions, spread across multiple modules.

This was when I decided that I would spelunk ONCE AND FOR ALL and just type hint everything up and down. Type hinting is great. You basically state, in your foo(), what type arg is, what type foo() returns. It works for keyword arguments too. It’s very flexible. You can type hint a couple parameters, skip others, and so on. You can use the usual data types (str, int, float, bool), but you can also type hint user-created types and classes.


def foo(arg: int) -> pd.DataFrame:
 # do some stuff
 return deep_foo(arg)

def deep_foo(arg: int) -> pd.DataFrame:
 # do different stuff
 return deeper_foo(arg)

def deeper_foo(arg: int) -> pd.DataFrame:
 # for the love of god
 # what is going on here
 return deepest_foo(arg)

def deepest_foo(arg: int) -> pd.DataFrame:
 # arg is an int!! AN INT!!!!

You can also put your type hints in a stub file. The benefits of using type hints, as I see them, are:

Readability. No more spelunking. Better understanding of what each function is doing.
IDEs. I learned a lot about type hints by watching Joel Grus live-code a neural net in 1 hr. His IDE (Visual Studio Code) and many others are smart about type hints and will raise a flag if you’re not being internally consistent.
Linting. Linters also do logic checks on internal consistency with types.

Which brings me to…

Linting
#

Did you know pylint gives your code a SCORE? YES. A SCORE. No longer will you kind of vaguely feel like your code is fine, but could be improved. NO. Now you know whether your code is a 4.32/10 or 6.16/10 or 8.23/10!

So pylint was another thing I long glossed over, but - when I discovered that it QUANTIFIES YOUR CODE QUALITY - well, you can bet I’m on that train. So now I pylint everything I can. My emails, my blog posts. All PEP8 everything, dammit!

What pylint does is automatically check your .py files for adhering to style standards (PEP8 style guide), but also handy stuff like software design rules of thumb. For example, it’ll encourage you to refactor a function into more functions (or a class!) if that function has more than five arguments. I guess this is what I hear people at work call a “code smell”. When I googled some of pylint’s output messages, I found this wonderful resource: PyLint Messages - and what they’re trying to tell you.

Again, you can configure your IDE to adhere to PEP8 guidelines too - making it even easier.

Resources
#

Logging
#

Type hints
#

Type Hints - Guido van Rossum - PyCon 2015

Linting
#

PyLint Messages

Better Bayesian beta-binomial bits

Tue, 06 Mar 2018 00:00:00 +0000

Yesterday at work, I gave an internal talk on the beta-binomial model, AKA my beloved Bayesian conjugate prior model. I’ve been using it a lot recently for A/B testing analysis, usually as a complement (not replacement!) to the usual frequentist methods (e.g. running a t-test between the cohorts, checking cohort balance before and after the experiment, running regressions to control for stuff that may be imbalanced). I’ve been reading a lot of Gelman lately, and just getting more and more enamored with the Bayesian view, and the failings (ahem) of the frequentists.

Specifically, what I like is:

Rather than worrying about sampling distributions (which is core to frequentist stats), you can speak more intuitively about beliefs.
You have to be explicit. That is, instead of a bunch of hidden assumptions about how your parameters or your data are distributed, you make those choices yourself - they’re knobs to turn.
In the case of conjugate priors, the math is super easy. Updating a conjugate prior - such as updating a beta distribution with binomial data - is just so damn simple. And it’s lovely!

I also just really love how the beta can be generalized into n-dimensional space into the Dirichlet. Back in CS109B, we did a cool homework where we Bayesian-updated the Dirichlet distribution of two texts to see if we could determine the author. Hollaaaa, NLP!

One good question that came up during my talk yesterday was: When is this useful? When is it more useful than the usual (frequentist) methods?

My sense is that it’s most useful when you don’t have much data (or don’t expect to have much data). As $n \rightarrow \infty$, Bayesian and frequentist methods should agree. But there’s a big wide world out there of sparse data and low sample sizes!

For example, in the case of views on a single YouTube video - the distribution of views across videos on YouTube has a very long tail: most videos get little to no views, and a minority get a whole lot. But what if you want to predict stuff on that long tail? A low-data area where you don’t expect much data to (ever) come? That’s where Bayes can really shine.

Or, heh, SHINE WITH THE FIRE OF BAD ASSUMPTIONS! But at least they’re explicit. Not-much-data means your prior grows in importance.

As I type, the Women in Machine Learning and Data Science conference is happening/has happened. One of the talks was from the very cool StitchFix Algorithms team, on - gasp! - using the beta-binomial for predicting sales of slow-moving items! Very cool.

Build Your Own Lisp- Learn C and Build Your Own Programming Language (2014) - ⭐⭐⭐

Tue, 06 Mar 2018 00:00:00 +0000

Metadata
#

Author(s): Daniel Holden
Number of pages: 212
Year published: 2014
Year read: 2018

Review
#

Aghh. This is hovering between a did-not-finish (at 69%) and a “want to read” for me. I might pick it up again.

Basically: I picked this up because I had two goals - (1) I wanted to write a toy programming language for my CS50 final project, and (2) I wanted to do more C.

Unfortunately, this book started strong but then lost a lot of steam for me. Plus, I found Destroy All Software’s A compiler from scratch video, which was basically the final nail in the coffin.

This book has two goals, and they’re admirable: first, introduces people to the basic theory behind how programming languages are interpreted/compiled, and, second, it teaches them C. The process is very similar to Destroy All Software’s video: you read in some input string, you tokenize those strings according to your language, you parse them into an appropriate tree structure, and then you pipe that tree into the compilation language. In the case of this book, you do all of that process: input, tokenize, parse, and compile in C. In Destroy All Software, he does the inputting, tokenizing and parsing in Ruby and then - mysteriously/wackily - compiles to JAVASCRIPT (what in the world). I’m planning to parse/tokenize/etc in Python and compile to C.

I made serious headway with the book, but eventually found myself just pecking out 100s of lines of C without understanding what the eff was going on. And the reason is that I think this book’s second goal - to assume it’s understandable to C beginners - isn’t really realistic. I have about ~20 hours of C under my belt (i.e. 5 problem sets in CS50), and I have a shaky grasp of pointers and memory allocation, and this book quickly got away from me. I ended up just blindly trying stuff, but then I was - well - struggling with C concepts, rather than compiler concepts. With the DAS video, instead, I’ve switched to Python and am finding it much easier to follow - now, instead of getting lost in the syntax of my parsing language, I can just focus on building a compiler.

Oh yeah, and another point: the author abstracts some of the tokenizing away from you with a custom-built C library he wrote called mpc. I’m not sure about the pedagogy, but - in general - I don’t like it when books provide custom-built libraries that you need to do to work, e.g. Allen Downey (who I generally admire very much!) did this with Think Stats and it kinda killed the book for me - let the reader build from the smallest building blocks! Don’t give us a half-built contraption upon which we’re meant to build more - that just means we spend more time trying to figure out the contraption!

Anyway, I feel bad saying all this. I think the book would be GREAT for people that are already up to speed with C. Also, it’s free and the author offers a lot of humor and encouragement. Bah!

📚 Goodreads

CS50 - Algorithms, pointers and malloc, oh my

Mon, 12 Feb 2018 00:00:00 +0000

Ah haaa, and now we enter the densest, most bare metal forest of CS50, I AM SO PLEASED.

Looking at the syllabus, I see that the future brings much Python and JavaScript - two languages I’ve been speaking for a while now - and so I am assuming those weeks will be a breeze. Check back in a month’s time to see if that’s the case.

But now! Now it’s been a wonderful ride through challenging and informative basic CS topics. This is what I came for! (I also came for data structures and HTTP, which are the next few weeks, woohoo.)

## Week 2: Algorithms

This week’s topic was algorithms, the varying measures of their speed, and common ones used for sorting integers:

On speeds: given that processors have different speeds, there’s idiosyncratic noise to computer speed, and blah blah, there is something called big-O notation. Big-O is the number of operations the computer needs to perform to complete an (algorithmic) task.

“We use big-Θ notation to asymptotically bound the growth of a running time to within constant factors above and below. Sometimes we want to bound from only above. […] It would be convenient to have a form of asymptotic notation that means “the running time grows at most this much, but it could grow more slowly.” We use “big-Θ” notation for just such occasions.”

(khan academy)

“Big O notation is a mathematical notation that describes the limiting behavior of a function when the argument tends towards a particular value or infinity.”

(wiki)

Normally written as $O(n)$ where $n$ is the size of the task: e.g. the length of the list you want to sort. Big-O is also the worst case scenario: that is, how long it would take to sort a list of integers if you never got any lucky breaks (as defined by the way the algorithm sorts). It’s our upper bound.

Good big-Os are things like logarithmic algorithms: stuff that, no matter how big of a list you throw at it, won’t ever take longer than the log of that list’s length.

Behold now eight different algorithms, eight different speeds:

(wiki)

There’s also a notation for the best case scenario: omega, $\omega(n)$. If you need to sort a list and it’s already, by chance, sorted - you need to only look through it once to be sure, ergo in that case $\omega(1)$.

And finally, theta - $\theta(n)$ - as the range between the upper and lower bounds.

Beyond measuring algorithms’ efficiency, what about what these algorithms are actually doing? Pictures/videos are worth infinity words here, and I really enjoyed these two resources -

Visualizing algorithms - To watch each sorting algorithm in action.
Sorting Algorithms Animations - To watch several in action, as they race against each other.

Problem set: Music
#

Expectations: This PS didn’t ask students to implement any of the algorithms covered in the lecture (which I was expecting). It also didn’t explicitly ask for any recursion - another lecture topic - and I didn’t (and still don’t) see much opportunity to try solving the puzzles with a recursive function.
Content: This was a lot of fun: converting text to music, via magical music math and how it relates to the wavelengths of each pitch. This spoke to my idiosyncratic love of waves.
Difficulty: Quite a bit trickier than earlier weeks, I ended up arriving at the Cognitive Wall - that place where you’re just banging your head, making no progress, and some time away from the problem works miracles. Taking a break helped. Also, I found r/cs50. :)

## Week 4: Memory

(wiki)

Memory! The stack and the heap! We actually drew it upside down - with the stack at the bottom and the heap at the top. I really enjoyed this lecture, since it started demystifying a lot of stuff - specifically, demystifying the connection between programming variables, values, scope, and functions, and how they’re all connected to (i.e. stored in) memory. This was very empowering. We learned about pointers (which are just addresses/locations in memory). Basically, in C, you can:

Initiate some variable, e.g. an integer, and call it x. When you do this, some space - 4 bytes, specifically - in memory is allocated to an integer called x.
Set its value to x=5. The memory location now holds the value for 5, in binary, written with those 4 bytes.
Grab its address, &x. This returns a memory address, written in hexadecimal (i.e. base-16 numbering).
Dereference it, *x: that is, access and - if desired - change the value at its memory location. Of course, we could just change x’s value by setting x=1 or whatever, but if we want to swap variable names, or use a second variable in general to reference to x’s memory and make changes there, then that’s something pointers are useful for. Here’s the wiki example.

Also exciting to learn about was hexadecimal, dynamic memory allocation (grabbing a piece of memory from the memory heap using malloc()) and the attendant memory leaks, and the call stack (which totally demystified recursion). Wonderful!

Problem set: Forensics
#

I’ve only just started this PS, which looks daunting - but manageable. The first part - Whodunit - had an enormous amount of info on bitmap file storage, which I ended up finding super interesting, and then it kinda took the training wheels off and was like, RTFM - i.e. reading and understanding documentation! Not for the faint of heart! But, like bash and regular expressions and reading code, an important meta-skill.

## Conclusion

Alrightie, these two weeks were some of the biggest value add so far. Really excited to keep going! Huzzah!

A side note: Every time anyone said malloc(), I would hear Allen Ginsberg’s wonderful exclamations from Howl: “Moloch!” Damn Moloch! From the poem:

What sphinx of cement and aluminum bashed open their skulls and ate up their brains and imagination?

Moloch! Solitude! Filth! Ugliness! Ashcans and unobtainable dollars! Children screaming under the stairways! Boys sobbing in armies! Old men weeping in the parks!

Moloch! Moloch! Nightmare of Moloch! Moloch the loveless! Mental Moloch! Moloch the heavy judger of men!

Moloch the incomprehensible prison! Moloch the crossbone soulless jailhouse and Congress of sorrows! Moloch whose buildings are judgment! Moloch the vast stone of war! Moloch the stunned governments!

Moloch whose mind is pure machinery! Moloch whose blood is running money! Moloch whose fingers are ten armies! Moloch whose breast is a cannibal dynamo! Moloch whose ear is a smoking tomb!

(source)

Full stack Python- My first Flask app

Mon, 05 Feb 2018 00:00:00 +0000

I’m a big fan of using the internet to share meaningful life events like weddings and births and so on. But I’ve been growing increasingly uncomfortable with using social media platforms like Facebook or Twitter for that - those platforms aren’t designed, really, for optimizing real, human connection; they’re designed to monetize human attention span by selling it to advertisers.

They’re meant to drive “engagement” (our time, our attention), which is the product that they sell to their clients (the advertising industry). (Incidentally, I’m reading Tim Wu’s The Attention Merchants, which talks about this and is v. good.) I especially don’t feel comfortable sharing stuff about kids/babies; it’s not giving their individual privacy a chance, it offers them up to the advertisers at a very young age. As minors, I feel that we should be protecting them from this stuff, not normalizing it and binding their childhoods/sense of self (?) to it. There’s lots of other reasons not to use these platforms.

There are some off-the-shelf tools to address this - the problem of sharing meaningful, but private, life events with family and close friends, unmediated by advertisers and a profit motive - e.g. 23snaps or TheKnot’s password-protected wedding websites?

But, again, it’s not much control and I’m not sure it really is a good alternative: e.g. I don’t know where the data is or how it’s used - whether it’s ultimately sold to those data aggregation/middleman agencies or used in machine learning work.

Side note: I used to work on household surveys in India (e.g. this), and it was important that consent - to be in the study, to be interviewed by us - was made a very explicit step before any survey began. Our surveyors read the consent form in Telugu, any and all questions from the potential respondent were answered, and only when they explicitly said it was okay would we enter to begin the survey. Right now, our “consent to be researched on” (i.e. included in A/B tests, have our data used in training machine learning algorithms) is buried in Terms of Service and End User License Agreements that no one reads. These TOSs can change, and, de facto, they’re legal CYA for carte blanche data use.

But I digress.

My conclusion was to make something “in house”: where I pay for the server space (instead of having advertisers pay for it), I keep it password protected, and basically the onus is on me to make sure it doesn’t get used for bad purposes. Plus, learning how to make a password-protected website is a useful skill! So here’s a blog post about that journey.

Specs
#

A website
That is password-protected
And isn’t hideous
And is in Python
And hopefully won’t get hacked by bad actors :P

So this was going to be my first foray into “real” full-stack web development. Things like this website are static: they use CSS, HTML, and a bit of JavaScript. There’s no need for a “backend” - there’s no users, and therefore no database.

I had a bunch of questions of how this would work: how do I set up a database backend on a remote server? How do I store passwords? (Do I need to learn more about encryption?) Where do I put it (which webhost is good and cheap and not sexist?) and how do I use a custom domain that I already bought and paid for? I was hazy on all this stuff.

Flask, I choose you!
#

Full-stack development in Python is dominated by two popular frameworks: Django and Flask. Django has a reputation for being big and comprehensive and coming with lots of bells and whistles, while Flask is meant to be more minimalist. Since I wasn’t building an industry-level app, I figured Flask would be fine. (I also liked the Horn of Gondor logo, heh.)

Finding the right tutorial
#

Flask Mega Meta Tutorial for Data Scientists - To find the tutorial that’s right for me!
The Flask Mega Tutorial - The tutorial I chose.

Honestly, the hardest thing on a learning journey is usually finding a compelling project. Once you have one of those, one tutorial is about as good as another. But I poked around the metalist and was ultimately quite happy with Miguel Grinberg’s “mega” tutorial, which was modular and comprehensive.

In Miguel’s tutorial, the project is building a social network-style app: there are users who make posts, and you need a database to keep track of people’s logins, their posts, and so on. I ignored the specifics of the project and ported stuff over to my interest: making a private family website. I really recommend Miguel’s tutorial (and won’t attempt to re-do it here!).

Stuff I learned along the way
#

The model-view-controller pattern
#

Although I attended a RailsBridge weekend a long time ago, it didn’t really meet me where I was. A lot of time was focused on getting our environments set up and basic programming. I think it’s great for women just starting out in tech, since web development is a good career path (especially for people transitioning in) and RailsBridge assumes zero programming experience. It’s awesome to get your feet wet.

But I had only one big takeaway from RailsBridge, and that was that there was something called the model-view-controller (MVC) pattern out there, and this was important for web frameworks. I was still kinda hazy on what was what (is the database the controller? or the model?): working through the mega-tutorial and building my first Flask app, though, really clarified stuff. And what a beautiful pattern it is!

(source)

My current understanding of the MVC is:

Model: The abstractions which your web app expects, and which will be turned into data; in Flask, these were abstracted into Python classes. e.g. A User, a Post, a Status or Photo or whatever. I just needed a User.
Views: The actual HTML/CSS that your web app presents to users.
Controller: The logic which underlies how the model and views (and the user and your website) interact. For example, some Users might get certain views (HTML pages), depending on whether they’re logged in or not.

For the Flask app, my MVC looked like this (as described in Miguel Grinberg’s tutorial):

- README.md
- my_app.py
- config.py
- my_app.db
- my_app/
 - __init__.py
 - forms.py
 - models.py
 - routes.py
 - templates/
 - a bunch of sweet html

Where my_app.py just imports my web app (the my_app/ folder) and makes it ready for running locally/on a server, templates/ is where all my views are (a bunch of .html files for, e.g., the index, login/logout pages, private photo gallery, and other stuff), routes.py is my controller (deciding how and when certain views are shown), forms.py separates out the user forms (e.g. the login form template, etc), and models.py defines a couple Python classes (User).

The thing I like about this pattern, and Flask, is:

All the HTML/CSS beautifying - and I think I’ll use Bootstrap to quickly make it more pretty - is separated out into its own front end/views space.
Separating out the logic (routes.py) from the abstractions of data your web app expects (models.py) was also helpful. Specifically, thinking about what abstractions are appropriate for your web app. I’m making a plain vanilla “my website will have users!” web app, but I can imagine other, cool extensions: a TODO list (User, Task), a movie review website (User, Movie, Actor, Director, Theater), etc.
Flask had a lot of nice off-the-shelf stuff for common patterns (docs): e.g. having users and logging them in via a web form. All of those steps ((1) having users, (2) logging them in, (3) via a web form) had pre-baked stuff that just needed to be tinkered with, e.g. from my models.py:

from werkzeug.security import generate_password_hash, check_password_hash
from flask_login import UserMixin
from datetime import datetime
from my_app import db, login

class User(UserMixin, db.Model):
 id = db.Column(db.Integer, primary_key=True)
 email = db.Column(db.String(120), index=True, unique=True)
 password_hash = db.Column(db.String(128))
 posts = db.relationship('Post', backref='author', lazy='dynamic')

 def set_password(self, password):
 self.password_hash = generate_password_hash(password)

 def check_password(self, password):
 return check_password_hash(self.password_hash, password)

 def __repr__(self):
 return '<User {}>'.format(self.email)

All that stuff - the UserMixin and werkzeug.security stuff for generating secure passwords (without me, as the webdev, ever seeing them in plaintext), was very cool.

`Jinja` HTML templating
#

From the docs:

“Jinja2 is a full featured template engine for Python. It has full unicode support, an optional integrated sandboxed execution environment, widely used and BSD licensed.”

For my family web app, I used Jinja to create my views - i.e. my HTML files. Jinja allows you to create HTML that dynamically responds to stuff - e.g. it can access Python variables and, depending on them, display this or that <div> tag, etc. This is distinct from, say, writing JavaScript that dynamically responds to stuff (which would be another way, I suppose, to do all this). My templates/ folder has a bunch of Jinja-fied HTML, but the important one is base.html -

<html>
 <head>
 {% if user %}
 <title>Welcome, {{ user.username }}</title>
 {% else %}
 <title>Welcome!</title>
 {% endif %}
 </head>
 <body>
 {% block content %}{% endblock %}
 </body>
</html>

The money bits above are the curly braces stuff (e.g. {% if user %} in between <head>...</head>). That’s the Jinja. user is a Python variable defined elsewhere in the project, it’s one of our models, and - in base.html - we’re going to modify how the the <head> attribute will look, based on whether that user has signed in or not.

Another nice thing about Jinja is that, after setting up the general look and feel of your website in base.html, you can create more *.html files that “inherit” that look and feel. Rather than copy-pasting the same header, navigation bar, or CSS link, you can just stick that all in your base.html and then do something like this for your index.html:

{% extends "base.html" %}

{% block content %}
 <h1>Hi, {{ current_user.email }}! Now you're in /index - wow!</h1>
 <p>Lorem ipsem blah blah</p>
{% endblock %}

In the above, index.html inherits from base.html ({% extends "base.html" %}) and then adds its own content ({% block content }).

Like pandas or regular expressions then, Jinja has its own special, additional syntax you need to learn - on top of regular old Python. All those curly braces! Why! But building a Flask app helped a lot in terms of learning it. (FWIW Pelican uses Jinja as well. I wish I had known more about that back in November!)

How to put it on the internet?
#

Once I got a small Flask app running locally, but before making the views non-hideous, I wanted to figure out how to run it on a remote server. I didn’t want to invest a ton of front end time before confirming that I could figure out the back end. So: how to actually put this ugly app on the internet?

Running a Flask app locally is as simple as:

export FLASK_APP=path/to/my_app/my_app.py
flask run

This launches a local server and runs your web app on some port (e.g. 8000) in localhost. You just navigate your web browser (Chrome/Firefox/Safari) to localhost:8000 and voila - you should see your app. You can just CTRL+C the bash console that’s running the server to close everything.

But how does this work on the actual, real internet? This was where PythonAnywhere came in.

“PythonAnywhere is an online Integrated Development Environment (IDE) and Web hosting service based on the Python programming language.[1] It provides in-browser access to server-based Python and Bash Command-line interfaces, along with a code editor with Syntax highlighting. Program files can be transferred to and from the service using the user’s browser. Web applications hosted by the service can be written using any WSGI-based application framework.” (wiki)

What’s nice about PythonAnywhere was that it came with a bunch of “pythonista” stuff included: you can easily launch bash consoles or Jupyter notebooks or set up MySQL databases, a bunch of Python packages are pre-installed, setting up virtual environments is super easy, you can git clone stuff into your PythonAnywhere account, you can also drag + drop files (like a favicon!), and - most easily - you can launch a Django or Flask web app pretty painlessly. And they’ll host it! And it’s pretty cheap ($5/month)! And you can DNS resolve it to your custom domain! So nice.

What is WSGI?
#

But why couldn’t I just buy some server space somewhere and run flask run?

If you want to run Python web apps (Django/Flask/whatever), you need to get Python (and Flask, etc.) installed on a server. So how do you do that? This is where WSGI (Web Server Gateway Interface) comes in. This article by Full Stack Python was very informative:

“A traditional web server does not understand or have any way to run Python applications. In the late 1990s, a developer named Grisha Trubetskoy came up with an Apache module called mod_python to execute arbitrary Python code. For several years in the late 1990s and early 2000s, Apache configured with mod_python ran most Python web applications.

However, mod_python wasn’t a standard specification. It was just an implementation that allowed Python code to run on a server. As mod_python’s development stalled and security vulnerabilities were discovered there was recognition by the community that a consistent way to execute Python code for web applications was needed.

Therefore the Python community came up with WSGI as a standard interface that modules and containers could implement. WSGI is now the accepted approach for running Python web applications.”

Incidentally, linked from the Full Stack Python post, here’s the super interesting story of WSGI’s predecessor, mod_python, and how it was developed, as told by its creator.

Voila!
#

And that’s it! I’ve really enjoyed beginning my full stack journey, and getting a Flask app up and running. I’m not going to link it because - well - it’s hideous and has no CSS or content right now. But now that I got the back end part sorted, it’s time for front end!

A few of my favorite pytest

Wed, 24 Jan 2018 00:00:00 +0000

YAZZZZ. I just raised my Python points over 9,000!

Sort of. I learned how to do something in pytest that blew my mind: creating fake attributes and methods of other Python objects. This meant getting better at understanding how modules and objects work. It was, mwah, so cool. But first, let’s talk pytest and why-test.

Why test?
#

This was another culture shock about moving into tech: unit tests. In academia economica, you write a bunch of Stata code (.do files!), run a bunch of regressions, and hope it doesn’t break and consider yourself replicable. But, ah ha, in tech world, your codebases are big, pieces of code interact in complicated ways, and it gets hard to hold it all in your head at once.

So you unit test!

The benefit of testing, as I see it, is that - as your codebase grows larger and more complex - it becomes harder and harder to know if any one change will break something else. The interactions and inter-dependencies get too complex. Often, multiple developers are working on different parts of the same, overall code. Unit tests, especially if they’re small and discrete, can help. You break things down into small packets and make sure each joint in the pipe is working. They also force you to write small, discrete functions, and - by testing constantly - you can see if a small change on module A will lead to a break in module B, etc.

Then there’s test-driven development (TDD), where the tests are written first and the code is written afterwards, constantly revised until the tests pass. I like this idea, but haven’t really done it (yet!).

My current practice is to use tests as a way to get to know a codebase: I’ll write new ones for code that isn’t covered, and I’ll tinker with old ones to see how they work.

A great video
#

I want to highlight this talk by Ned Batchelder at PyCon 2014, since it really informed the way I think about testing. The one thing that struck me that Ned said was, basically, treat your tests with the same respect and reverence as your code. Your tests don’t need to be repetitive, non-DRY, ugly hunks of semi-identical tests. They can have objects. They can have good structure. pytest can help you write these tests in beautiful ways.

A great essay
#

This post by Kent Beck is also great at thinking through the how and why to test.

`pytest` (specifically, faking stuff in `pytest`)
#

Python has unittest built into the standard library (i.e. comes with Python), but I’ve been digging pytest. I reeeally like pytest. It has lots of very handy stuff either built-in or as easy pip installable extensions. I’ll talk one the SUPER COOL thing I learned about: fixtures and monkeypatching!

One of the challenges of unit testing data-driven code (like a data science service, or ETL pipeline) is that it relies on external dependencies (e.g. a database, an API, a Kafka stream). To address this, we can use a monkeypatch fixture.

Fixtures (docs) allow you to pre-define baseline “states of the world” that you then test your modules against. They are related to the set-up/tear-down pattern where you set up, for example, a “fake” database object or a “fake” Kafka stream object, test your code with that, and then tear down the fake object. The convenient part of pytest’s fixtures is that setting up and tearing down is abstracted away from you. You just use the @pytest.fixture decorator in your test_module.py file, and pytest handles it for you.

Monkeypatching (docs) is one such fixture. It allows you to directly (and temporarily) modify existing modules, both from your own project as well as external libraries. For example, you could monkeypatch the requests library such that requests.get() always returns a specific string - but only in the context of your tests. So you don’t have to worry about sullying your copy of requests, nor do you have to deal with the pain of setting it all up and tearing it all down.

Example: Fixing ye olde Odo bot
#

My beloved odo-bot was a Slackbot I built, long ago, at my old gig. odo-bot is early days Python for me, so forgive me the numerous obvious sins. If I re-wrote odo-bot now (DON’T TEMPT ME), I’d definitely include tests. For now, let’s write a little unit test for one of his functions, Inspiration():

def Inspiration(event):
 """
 A function to provide amusement and inspiration.
 For now, pulls from Reddit's API and pulls a random top-rated post/image/URL.
 """

 response = requests.get(f"http://www.reddit.com/r/{random.choice(SUBREDDITS)}/top.json?t=month")

 if response.status_code != 200:
 inspire = "I have no inspiration for you now."
 else:
 data = response.json()
 data = data['data']['children'][0]['data']
 inspire = "Try this: \n " + reddit + data['permalink']

 # Er, re-writing this old function so that it's not directly calling the Slack API
 # sc.api_call("chat.postMessage", channel=event['channel'], text=inspire, as_user=True)
 return inspire

(source)

Ack! What a mess!

Anyway, the tldr on this function is that it takes an event - which is someone on Slack talking to Odo - and returns some “inspiring” Reddit post. What in the Lord? I don’t even remember why I wrote this. What SUBREDDITS? Why are they global vars? Why did I think to put sc.api_call() in scope?! Oof.

Well, anyway. The core of this function is that it’s calling the Reddit API (request.get()), parsing the response (if response.status_code, response.json()), and then posting it to the Slack API (sc.api_call()). The core of this function is parsing Reddit API responses correctly; that’s what we want to test. But we may not want to test it by making actual API calls - this would waste our rate limits there. Similarly, we may not want to post test responses to our Slack API. Indeed, we want to isolate things to just testing the core functionality and removing any external dependencies.

pytest’s monkeypatching can help us!

We see that response = requests.get() is the important bit. This returns a requests.Response object, which has different attributes and methods (docs). We want to somehow mimic the requests.Response.status_code as well as requests.Response.json(). However, while .status_code is a settable attribute, .json() works on requests.Response.text, which is not settable. Also, requests.get() will call the API; so if we monkeypatch the downstream stuff that’s done to the Response() object (like .text, .json(), .status_code), we’ll still be hitting the Reddit API - and bumping up against our rate limit. So let’s monkeypatch .get() itself and return a FAKE RESPONSE OBJECT (aaaah).

We can do this from within our testing function, test_Inspiration(), itself. We temporarily modify requests to do what you want it to for the duration of the test:

def test_Inspiration(monkeypatch):

 # The monkeypatch function is defined within the larger test of the function
 def fake_get(api_url):
 # OMG I know this looks bad, but I think it's okay...!
 class Fake_Response(object):
 def __init__(self):
 self.status_code = 200 # our fake responses are always OK! yay

 # some fake data, who cares, just gotta make
 # sure the structure is the same as Reddit's actual API responses
 self.text = '''
 {"data": {
 "children": [{
 "data": {
 "permalink": "some_reddit_link"
 }
 }]
 }
 }
 '''
 # Faking the requests.Response.json() method
 def json():
 return json.loads(self.text)

 return Fake_Response

 monkeypatch.setattr(requests, 'get', fake_get)
 fake_event = {"channel": "nowhere"}
 assert Inspiration(fake_event) == "Try this: \n http://www.reddit.com/r/some_reddit_link"

The way this works is:

You define your testing function, test_Inspiration(). Make sure to include monkeypatch as one of your arguments.
Within that, you define your mock object or function, Fake_Response(). This is the function that will temporarily replace the function/attribute/object you are monkeypatching (in our case, requests.get()).
Use monkeypatch.setattr(obj, name, value, raising=True) to temporarily make the object’s method (request.get()) return something else (Fake_Response(), instead of requests.Response()). Here’s monkeypatch’s source.
Proceed with the test, calling Inspiration(fake_event), as usual.

What I think happens is that pytest will catch and parse monkeypatch when it looks for test_* functions in your test_*.py. Here’s the official tutorial, similar to the above.

Gamifying art

Mon, 22 Jan 2018 00:00:00 +0000

Lately, I’ve been working on art. Specifically, I’ve been trying to get better at figure drawing, while also exploring watercolors and portraits. For the former, my two current gurus are:

Kimon Nicolaides’s The Natural Way to Draw: A Working Plan for Art Study, which is salty and dated, kinda like * the Hard Way. Very “no pain no gain NOW GIVE ME THREE HOURS OF BORING CONTOUR DRAWINGS, PRIVATE!”.
Stan Prokopenko’s YouTube tutorials, which are Nicolaides minus pain and a lot more fun. Also YouTube!

This means I need a constant stream of images for ideally-daily gesture drawing practice. For now, I’m using quickposes.com and Line of Action. I like Line of Action especially, since it offers both fixed-time poses and a “class” setting - which starts with quick poses (30 seconds each) and slows down to eventual 5- and 10-minute poses. Also, the images are a lot more, er, tasteful.

I feel like I’m already seeing improvement. Anyway, it’s fun! One of my big TODO side projects is supporting this hobby with some bot helpers, e.g. a command line utility to keep track of my progress and find pics and so on.

For now, I’ve also been thinking of how to do this whole thing.

30 second gestures, Conte black pencil

Acknowledging the reptilian brain, addicted to a screen
#

Social media and modern apps are designed to be addictive. So we can learn from them! I want to make drawing an addiction. One way to build addiction is random bursts of dopamine - that little red notification alert, mmm. So nice. When will it come? Another powerful thing is gamification of inputs (time spent drawing) and the feeling of progress on outputs (drawings not sucking as much). Finally, as with Goodreads, ego/social performance are important motivators too (alas). Finally, I know I should automate as much as possible, reducing any friction to starting my drawing day (or gym visit, or chores). If I can get myself going, I can stick with it more.

Random bursts of dopamine
#

Because I live in, well, 21st century America, I’m addicted to consuming. So I’ve subscribed to ArtSnacks, a monthly art supplies box. It’ll be random (doesn’t always come on the same day), it’ll be dopamine (SOMETHING NEW!), and it’ll be a nice way to explore materials. My first box had some water-based pens which have already upped my game a bit.

following Proko’s The Bean tutorial

Gamification of inputs
#

While my heart/gut still has a “fixed mindset” about art - either you have “talent” or you don’t - my head knows that’s baloney and practice will make me better. Which means, as they say, butt-in-chair time. One good gamifier is Habitica, which casts your life (and chores) as an RPG. TODO is to write my own art-specific, Python-powered productivity game tool.

Progress pics
#

Proko says to draw the same thing multiple times. I’ll do this, and post monthly/quarterly progress pics. Hopefully I’ll improve.

shitty perspective, sigh

getting much better at hands

Social performance
#

Profile pics are great places to share some of my latest art; this blog is also a nice repo. Holiday cards?

Automation
#

Yeah, I gotta do this part. To the Python!

image from a to-be-written scifi story

More than mere flesh
#

Okay, with all that depressing “let’s hack our reptilian brain” stuff out of the way, let’s speak of higher matters. ART!

The thing I’ve repeatedly noticed about art - in any medium - is that it can give you new eyes to see the world. I’ve always liked that feeling. After getting into photography, I would see “beautiful shots” everywhere. My early love of drawing made me love the human form - e.g. my brain sees wrists as one of the world’s most beautiful, intriguing forms, but I only recently began to realize other people might not think that?! Lately, I’ve been marveling at perspective and how it can be mathy.

I like very much that the Renaissance masters followed this same line of reasoning, since I consider them my go-to heroes and inspiration. Lately, I’ve been thinking that I could probably hack my way to better drawing by studying anatomy and geometry. (Yeah, I know, DUH.) But what a sensible and rational and sure-fire path!

Goals & inspiration
#

Another thing I like about art is, of course, its ability to tell stories. I like telling stories! Especially, hopefully, weird and wacky ones. I reeeeally admire the Fiona Staples’s complete control of form and movement (mamma mia), and Brandon Graham’s fevered visions, and - of course - Jean Giraud/Moebius’s precise, meticulously imagined worlds. I guess my DREAM (!!) would be to approach any of these three’s skill in any way possible. Even an inch! So let’s see.

Fiona Staples’s absolute mastery and control! no hesitation! every line is perfect!

Brandon Graham’s intense, feverish, hallucinatory sci-fi style

Moebius’s rich, clear, meticulous visions

Starting CS50

Fri, 19 Jan 2018 00:00:00 +0000

Background: What is CS50, and why am I taking it
#

One big part of my journey to the center of data science has been a journey to the center of computer science. I’m a self-taught programmer, and I’ve now taken four CS courses in my life: an introductory undergrad CS101 course from looong ago, and three data science-centric CS courses through Harvard’s Extension School.

The rest I’ve learned on the fly - which has been fun and whirlwindy - but it leaves me with some foundational, I think, gaps. Specifically, I want to focus on:

Data structures and algorithms
Networking

But, before diving into those two, I thought working through Harvard’s (crazy popular) CS50 would be a great way to fill in a lot of general fundamentals. Also, it has some problem sets in C! Yay!

By blogging and “learning out loud”, I’m hoping it’ll motivate me to find the time to really crush through this and finish it up. It’s 11 weeks of about ~2.5 hours of video and ~4-5 hours of problem sets per week. That’s ~75-80 hours. Thanks to edx.org’s instance of CS50X, everything’s released at once, grading (seems to be?) automated, and it’s free. I love MOOCs that let you set your own schedule and, if possible, just binge. That’s been a lot easier for me to organize around than MOOCs that have enforced week-by-week marches. My schedule is lumpy! Some weeks are free, others are insane.

### Week 0

Lecture
#

The first lecture - or, in compsci nerdery, the 0th lecture - was a fun, edutainment dazzlement of a few core facts: hardware (bits!) and algorithms (ripping up phone books!). What I love about the CS50 behemoth (and it is a behemoth - with merch, even) is that its “meta” - the way it’s produced - is clever and fun. I love using their tools. Did they build their own video player?

Problem set
#

I didn’t care for PS0. It was meant to be a gentle introduction to programming via MIT’s Scratch, a GUI-heavy programming language for kids. I think this could be useful to demystify programming to someone who’s never coded before; I’m not sure. It introduces the basic concepts of loops, conditionals, and variables. I found it a chore and rushed through it. Let’s get to C!

### Week 1

Lecture
#

This explored basic programming ideas with C. If you’ve programmed before, the ideas were the basics: loops and conditionals and syntax gotchas, and so on. The main takeaway I had was about compilers (which translate source code, written in C, into assembly language), clang and its many flags, and make.

There was also some fun stuff about floating point imprecision (bane of all my unit tests!) and overflow/underflow. Data types - such as (unsigned) ints, chars, floats, and so on - were introduced.

Problem set
#

This PS was our first in C - yay! It covered a couple basic programming tasks: “hello, world”, drawing some ASCII art inspired by Super Mario, and either implementing a greedy algorithm for a change-making machine or a credit card checksum, uh, checker. This was all done on CS50’s nice, cloud-based (containerized?) IDE. I found it easy (“hello, world”), medium (Mario), and pretty tricky (greedy algorithm). A good mix!

### Week 1, continued

Lecture
#

My three main takeaways from this lecture were:

Compiling as a four-step process: preprocessing, compiling, assembling, and linking.
#include <some_c_library.h> as the C-equivalent to Python’s import statements.
Command-line arguments.
Debuggers.

Okay, four things. Man, I wish I had known about debuggers back when I first learned to code! Yeesh. I relied (and probably still rely too heavily) on console.log()/print() statements, but debuggers offer much richer information about a program’s current state, variable values in scope, and all that.

Problem set
#

Cryptography! So this was basically implementing the Caesar and Vigenere ciphers, something I had already done in Clojure. Still useful, though.

Impressions so far
#

The thing about CS50 is that the lectures are super slick but, even better, the problem sets are dense and challenging. I really love the problem sets!

I remember marveling at the well-designed problem sets of CS171, CS109A and CS109B: in all those courses, I had probably learned the most by going through the comprehensive and dense PSs. So I really like that! It takes forever - but, thankfully, CS50 offers “less comfortable” and “more comfortable” options. If I have time, I’ll work through the “hard” version (e.g. the Mario ASCII art); if I have less time or feel pooped, I fall back on the “easy” version (e.g. the greedy algorithm - which was still kinda tricky!). Overall, I’m spending ~4 hours on the PSs - which is less than I used to spend on the (not free) Extension School classes (that would be more like 8-10 hours/week, oof).

One bash script to blog them all

Fri, 12 Jan 2018 00:00:00 +0000

One of my desired blog specs was automation. The first automation I made was make_entry.py, a script modelled on this blog’s example that would, from the command line, create a fresh new .md file with my Pelican template stuff filled-in. My latest little automation is a script I named picard.sh (because makeitso.sh was too annoying to type).

picard.sh automates the process of Pelican making -> publishing to my webhost -> updating Twitter. It’s super simple, and I’ll be using this blog entry as the first integration test/guinea pig.

Step 1: Pelican-make the blog
#

# picard.sh
# Jan 2018. Automate publishing + tweeting.

echo 'Pouring blog out into local angelaambroz/ dir'
make html

After creating this blog post, in Markdown and in my blog/content/ folder, I need to run make html to pour it into the local directory where the rest of my website is. This comes built-in with Pelican (docs).

Step 2: Securely copying my local website copy onto my webhost server, using `ssh`
#

Oh man, how I love SSH (secure shell)! I could go on and on about it. And I love scp (secure copy protocol), which lets you copy files and folders to remote servers. It’s awesome.

Before I could get picard.sh (or any bash scripting into remote servers) to work, then, I needed to get an SSH handshake set up between me and the webhost. I did this:

Create new SSH keys, specific to my webhost’s server, with ssh-keygen.
Put those keys on the server via the UI. This is similar to how it works on GitHub. Further reading: Adding a new SSH key to your GitHub account
Set up my .ssh/config file to use only those website-specific keys when SSHing or SCPing into that server. Further reading: SSH - Too Many Authentication Failures. This was to troubleshoot a “too many attempts” disconnection I kept getting. It was trying all my other SSH keys.

Once all that was done, I could now securely connect to my website from the command line. Awesome! I copied a test file over with scp, and voila! Things were ready. I added the following to picard.sh:

echo 'SCPing over to angelaambroz.com'
scp -r /path/to/local/website/blog my_username@my_webhost:~/path/to/website/

Step 3: Let Twitter know
#

Okay, so I’m skipping over a Python script that I wrote, update_twitter.py, that tells the Twitter world about a new blog post when it happens. It’s not terribly interesting, or terribly different from my post on Making a Twitter bot. Here’s the only different stuff:

def _make_link(blog_slug, blog_date):
 """
 Example link: http://angelaambroz.com/blog/posts/2017/Dec/21/data_is_my_copilot/
 Twitter will handle link shortening via t.co
 (https://help.twitter.com/en/using-twitter/how-to-tweet-a-link)
 """

 base_url = 'http://angelaambroz.com/blog/posts'
 date = datetime.strptime(blog_date, '%Y-%m-%d %H:%M')
 year = date.year
 month = date.strftime('%b')
 day = date.day
 return f"{base_url}/{year}/{month}/{day}/{blog_slug}/index.html"

def tweet_about_blog(post, connection):
 """Tweet about post
 """
 robot_emoji = b'\xf0\x9f\xa4\x96'
 url = _make_link(post['slug'], post['date'])
 tweet = bytes(f"{post['blurb']} {url} ", 'utf-8') + robot_emoji
 connection.update_status(tweet)


def get_last_n_blog_entries(n=1):
 """Grab last n (default=1) blog entries from the content/ folders"""
 latest_blog_posts = sorted([x for x in os.listdir('content/') if ".md" in x])[-n:]

 title_list = []
 for blog_post in latest_blog_posts:

 with open(f"content/{blog_post}") as f:
 blog_list = f.readlines()

 title = [x for x in blog_list if 'Title: ' in x][0]
 slug = [x for x in blog_list if 'Slug: ' in x][0]
 date = [x for x in blog_list if 'Date: ' in x][0]
 title_list.append({
 'blurb': title.strip().replace('Title: ', 'New blog: '),
 'slug': slug.strip().replace('Slug: ', ''),
 'date': date.strip().replace('Date: ', '')
 })

 return title_list

The most interesting hm!-inducing stuff in this Python script were just:

Finding the the unicode (or is it hex? what is that thing?) for the robot_emoji.
List comprehensions are always nice? e.g. latest_blog_post = sorted([x for x in os.listdir('content/') if ".md" in x])[-n:]
I’m still in love with Python 3.6’s fancy print formatting, f"Hello, {variable_for_name}!".

All that said, it just pops into picard.sh like so:

echo 'Updating Twitter...'
python update_twitter.py

echo 'Done!'

And that’s it!

And, okay, I just realized I could have used publishconf.py or fabfile.py (both built-in Pelican publishing tools) for this… Oop!

How to understand the big, wide codebase, and your place in it

Wed, 10 Jan 2018 00:00:00 +0000

Learning how to dive into a big, wide codebase is - I’m realizing - an important meta-skill. Kinda like learning how to read an academic article.

I’ve struggled with it, though, since I didn’t realize it was a meta-skill and something to improve. I guess I thought other people just magically absorbed big codebases?! That, coupled with the shrieking sirens of impostor syndrome, made me freeze up whenever someone asked me, “Hey, could you make [X small change] to [Y giant new codebase]?”

So I wanted to note down what I do and what’s been helpful.

Objective
#

Imagine, if you will, this scenario:

SCENE 1. A bustling 15th century RENAISSANCE BOTTEGA. Large lavendar plants spill from big ceramic jars. The work of the great Renaissance masters - Da Vinci, Brunelleschi, Michelangelo - adorn the walls, along with a cyberpunk poster or two. The smell of paper-thin pizza is in the air. Your COLLEAGUE is laboring over some giant convolutional neural net. YOU arrive, stage left, pursued by a bear.

COLLEAGUE: Oh! Angela! Welcome to the team. Glad you could join us.

YOU: (nervously) Glad to be here!

COLLEAGUE: We thought a nice first project for you would be extending some of the features for our service’s chatbot.

YOU: …

COLLEAGUE: Are your familiar with chatbots?

YOU: Um, well, I guess I did build one in my old job, but that was just –

COLLEAGUE: GREAT! Ours is powered by some natural language processing, regular expressions, and a few APIs. Pretty straightforward, really, no big deal. We’d love it if it could start responding to user questions about our DATA SCIENCE SERVICE, so you’ll need to plug it into that as well. We’re just trying to give a window into that infamous black box, eh?

The DATA SCIENCE SERVICE is wheeled out. It is a large black box, thrumming softly and crackling with occasional bursts of electric sparks. There is a small post-it attached that reads: README.md. YOU peer at the post-it, peer at the box, and the blood drains from your face. It reminds me you of that one freaky Dune scene. You think you hear the hiss of snakes inside.

YOU: (to self) Oh God.

COLLEAGUE: Sorry?

YOU: I said, sounds great!

So. What to do?

This has happened a few times, where I’ll be introduced to a codebase that has thousands of lines of code spread out over several modules. It can be really intimidating! So this is how I break it down.

Step 1: Get the general context
#

The biggest mistake I make - and I make it again and again - is that I get into the weeds too fast. Whenever I’m introduced to a black box, I find the first weed and start studying it very, very closely. This is no good. This is not just putting the cart before the horse, this is putting the screws before the wheels before the cart before the horse.

So, one thing I’ve been training myself to do is a top down approach. Mainly, this means asking myself, before anything: What is the point of this codebase? If I can describe it in one line (e.g. “It’s a micro-service that provides a probable classification in response to an HTTP request”), that’s a start.

Beyond getting the one-line tldr of the overall, I then try to build out the context more. Why does this codebase exist? What happened before it existed? Who uses it? If there’s documentation, I try to read it - even if it’s outdated. I try to ask stupid questions, and keep asking them. I try to draw out a diagram of its place in the world (e.g. connected to databases or a website or what).

I love my LCD writer - perfect extension for your brain!

Step 2: Get the software context
#

If you don’t have lots of software development experience, directory structures look pretty arbitrary. When I realized that there were norms around how code projects were organized, this sped things up a lot. Suddenly, I knew where to look; I knew which folders to focus on. The Hitchhiker’s Guide to Python has a good overview of a Python project template, and I’ve also seen and used DrivenData’s cookiecutter data science template.

Step 3: The project outline (drawn)
#

I find it really helpful to draw a diagram linking all the modules together. Sometimes this includes drilling down into each module, mapping the functions too and whether they’re calling external stuff (like a database) or calling other pieces. The main thing I want to do is get a sense of what’s in there.

I also am a big fan of color-coded everything, since - thanks to CS171 - I now know that COLOR REIGNS SUPREME, and the human brain processes color blaaazingly fast (yeah, V1 neurons!).

Blue classes hold red functions

Step 4: Reading and writing docstrings
#

Once the above steps are done, I feel like I am NOW ALLOWED INTO THE WEEDS. At this point, I’ll start digging into specific functions by order of subjective “importance”. I really like PyCharm’s goto definition thing - where I can cmd+some_function_name() and it’ll jump me to the source. I’ll try to follow this goto-stream up and down the layers of abstraction, reading through docstrings.

Unless there aren’t any docstrings! Which sometimes happen! In this case, I might start drafting some docstrings for the functions - just (1) what I think the func does, (2) arguments and their data types, (3) returned values and their types. I’ll try to understand the funcs well enough to write docstrings that could be merged into the develop branch.

Also step 4: Running and writing tests
#

I recently learned about the wonders of testing, and now I use tests as my sanity check for everything. When I’m getting to know a new codebase, I might run tests as the first thing I do - this often pops out whether my installation or virtual environment is missing something. And that helps understanding too! I also try to read through the tests and see what’s being tested; this can be the best place to see the overarching behavior of all the code.

I’ve lately also found that writing tests - especially super small unit tests - and checking out test coverage, are reeeeally helpful ways to get to know things better. Robin Andeer’s three part series –

– was really eye-opening.

The last step 4: Jupyter notebooks
#

And finally, Jupyter notebooks! This has also been a helpful way to get to know code. By this point, I should have an overall sense of what the codebase is supposed to be doing. I might want to dig into an important or main function. A good way to do that is, after creating a virtual environment and pip installing the project’s dependencies, creating a (.gitignored!) folder called notebooks/ and starting to explore the tests, the code, everything. One thing I might do is pull in a function and break it down into each of its parts - that way, I can see what goes where, and what needs what.

A slightly more onerous way, and something I sometimes do with testing as well, is just splattering print() statements everywhere, especially of parameters and returned values. This helps me understand the state of things at each point in the code.

I’m still not great at this
#

That’s been the journey so far. And that’s my current best method. If I were to boil it down, it would be: (1) top-down first, (2) draw, (3) test (safely).

Git history is written by the victors

Tue, 09 Jan 2018 00:00:00 +0000

When I career switched into tech, the two big management shocks were daily standup and git. And I mean that in a good way, a very good way. I love git very much, and am starting to get opinionated about it. It’s not just version control! It’s history, and narrative, and structure. It’s collaboration! It’s TEAM. It’s a beautiful tree spawning many branches. Or a lot of hair with a lot of braids that need to get re-braided. I like to keep my repos like little bonsais or Princess Leia buns of good practice. git is not to be neglected! And often is!

xkcd/1296

Tutorial assumptions
#

This tutorial assumes that you have used git, at least a little bit. That is, you know that it’s a version control system, you’ve made some commits, pushed and pulled, and have an account on GitHub. If you’re very new to git, then I highly recommend Udacity’s How to Use Git and GitHub.

Some philosophy: Git as history (AKA the power of `squash` and `rebase`)
#

I think the main power of git is that it tells a story of your project. Sometimes I get lazy and just rely on git to, well, not lose all my stuff. And this is, yeah, its big sell. But, if you write good (long!) commit messages and commit at the Right Time, and if you wield the majestic power of git squash and git rebase with aplomb, I think you can create quite a beautiful, readable little story, something to be passed down from collaborator to collaborator. This is useful for welcoming new team members, or for dealing with the passage of time (why did you write it that way, all those months ago?).

Here are some stuff I find helpful in git:

1. Visualize your history
#

There are several ways to visualize git histories:

Tower ($80/license) - A very pretty GUI, integrates nicely with GitHub.
gitk - A built-in command-line “graphical repository browser” that ships with git. Uglier GUI than Tower.
A bash alias:

alias glg='git log --graph --full-history --all --color --pretty=format:"%x1b[31m%h%x09%x1b[32m%d%x1b[0m%x20%s"'

I don’t know who wrote the above. Someone at my old gig? It was passed down to me from the ancients. If this was you, please let me know. I feel super guilty sharing some mystery code like this. Please let me assuage my guilt and just say you wrote this.

Anyway, the point is: find some way to visually navigate through the commit history and the various branches. Git is surprisingly/eerily smart about tracking when things are happening in relation to each other. Seeing all that is powerful and informative. I glg repeatedly throughout the day. I just checked now –

history | grep glg | wc -l
>> 34

– 34 times! That’s 13 times an hour, or once every 4.6 minutes.

Okay, maybe I check it too much.

2. Pick a comfy editor
#

Git uses your global default $EDITOR as its own. I think that’s Vim on OSX? Or is it nano? Anyway, I like Sublime as my lightweight “small jobs” editor. If you want to change your default text editor for git to Sublime (but keep whatever $EDITOR for other stuff), then:

git config --global core.editor "subl -n -w"

You can also do Atom or TextMate, or anything that launches from the command line.

Having a comfy/friendly text editor is important so I can write long, involved commit messages (further reading: How to Write a Git Commit Message).

3. Rewrite history (responsibly): `git squash`, `git rebase`
#

When we’re working on our local machines, we can create as many branches as we like, however we like. We can commit whenever and whatever. But this can get messy when we start sharing that history around.

I fucked up Git so bad it turned into Guitar Hero pic.twitter.com/vUKZJAQKWg – Huenry Hueffman (@HenryHoffman) February 1, 2016

One way to keep things clean is to combine many commits into one big commit (git squash) and to rewrite our git history such that our suggested changes are always on top of the newest version of code (git rebase).

Git squashing and rebasing is useful but potentially destructive. It rewrites git history. I think a good general rule of thumb is: rewrite your own history, not other people’s. In other words, right before submitting a pull request, you can combine all your branch commits into one - “This is my final version!” - and you rebase that one commit on top of the latest develop branch such that it looks like you used the freshest codebase to start your branch from.

But you never want to squash or rebase master or develop themselves.

git rebase is distinct from git merge since it doesn’t merge in other people’s commits (thus sullying your story), but rather just pretends you started work on whatever the latest code is.

The end goal is this: “beautifull”

Walkthrough
#

Let’s assume you’ve been working on a branch for a few days. You have some code you want to PR into blog/master.

Update your local copy of the master branch with any remote commits.

git fetch
git pull

If there are any updates, rebase your work on top of those changes in master.

git checkout [my branch]
git rebase master

This might be a pain point: merge conflicts. Untangle each of them in turn (I like grepping for <<< in the conflict files), git add . them and then git rebase --continue.

Now it should look like this:

Source: https://www.atlassian.com/blog/sourcetree/interactive-rebase-sourcetree

Squash your commits (which you can count down from HEAD to where your work begins, or use a specific commit’s SHA) into one final commit. For example, say you wanted to squash the last five commits:

# Rebase by counting down from the HEAD...
git rebase -i HEAD~5

# ...or by selecting a specific, older commit's SHA
git rebase -i 9b20280

The interactive -i flag will launch git’s default editor. There, you select which commits to keep as separate commits (pick, or p) and which to combine (squash, s).

Pro tip: If you forget the -i flag, git will not squash anything, and will just exit out back to your shell.

pick your branch’s first commit, and s (squash) all the others.

Next, git will prompt you to adjust the squashed commit message.

Check your git history, and voila! You should see that all those commits collapsed into one!

4. Keep polishing your git skills
#

In my day to day git, I’ve always tried to note down the random stuff I’ve learned about how to use it. Can I search for a specific piece of code? Even if it’s been deleted? Can I search for a deleted file? Stuff like that. The mystical .git file keeps track of so much, and it’s great to be able to essentially trawl back through all these different versions of the past.

Here are some useful recent TILs:

To search all commit history for when a string changed:

git log -Sstring

To compare the differences between specific commits:

git diff COMMIT_SHA^ COMMIT_SHA

To remove stuff from the staging area:

git reset

(I know, I know. That one should be basic, but I almost never used it, yolooooo.)

Jacobi(a)n stuff

Wed, 03 Jan 2018 00:00:00 +0000

Somebody mentioned a “Jacobian” at work - which sounded exotic and mathy and attractive. I resolved to watch the Khan Academy videos on the Jacobian and learn, dammit, LEARNNNN!

til tldr: You can use the Jacobian matrix, and its determinant, to figure out locally linear transformations to small ranges around points, when they’ve been all warped up by a non-linear transformation. Apparently this is useful for non-linear least squares regression. I’m passively looking for more real world examples.

Linear algebra is linear
#

I’ve been working my way through Khan Academy’s linear algebra series which is (a) very good and (b) very long (it’s been months!). I took linear algebra long ago as an undergrad of yore, and had fuzzy memories of (a) how matrices can be multiplied a bunch of different ways (dot? inner? cross?), (b) eigenvalues are important, and (c) ORTHOGONAL. I remember enjoying it, but I was and am very rusty.

When I got into data science, I loved learning about principal component analysis and, of course, deep learning, but I also realized, the time for a linear algebra refresher was nigh upon me. I highly recommend Khan for these things.

I’m about halfway through, and the biggest takeaway is the elegance and omnipresence and convenience of linear transformation matrices. And what you can do with them! And how they can encode so much information! Also, the frisson of linear independence - i.e. more information.

I think it’s valuable/helpful to visualize the geometry of this stuff, so here’s some great visualizations (Khan again):

– an example of a linear transformation

– an example of a non-linear transformation, a SQUIGGLE FEST, a science fiction opus waiting to be written

A case of the squiggles: What to do with non-linear transformations
#

So most of the Khan linear algebra I’ve been doing has been watching Sal Khan work through proofs of all the conveniences and glories of having everything linear. And just generally demonstrating how these two simple rules:

$$T(\bar{v} + \bar{w}) = T(\bar{v}) + T(\bar{w})$$ $$T(\alpha \bar{v}) = \alpha T(\bar{v})$$

for any vectors, $\bar{v}$ and $\bar{w}$, any scalar $\alpha$ and any matrix transformation $T$, can lead to a lot of helpful results.

But what about non-linear transformations? Those would fail the two rules above, and a lot of linear algebra’s stuff would break down. It’s been likewise YEARS since I did any calculus, so it was fun and interesting to learn - in the Jacobian videos - about how multivariable calculus tries to port the insights of linear algebra onto non-linear problems.

Specifically, the Jacobian matrix is based on the premise that, for local ranges around a point, things stay pretty linear even in the most abominable of squiggle-fests. Or, as Wiki explains,

“The Jacobian matrix is important because if the function $f$ is differentiable at a point $x$ (this is a slightly stronger condition than merely requiring that all partial derivatives exist there), then the Jacobian matrix defines a linear map $\mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$, which is the best (pointwise) linear approximation of the function $f$ near the point $x$. This linear map is thus the generalization of the usual notion of derivative, and is called the derivative or the differential of $f$ at $x$.”

Visually:

– screenshot from Khan Academy: Local linearity for a multivariable function

And in plainer English (explaining it to myself): the insight of local linearity lets us do a nice trick: by taking the partial derivatives of the original (non-linear) transformation for the $x$ and $y$ directions (in the 2D example), we can get a sense of how points would be transformed by a non-linear transformation, $f$, in a local space.

Even more cool was that, in a 2D space, the determinant of that transformation (when written as the transformed basis vector) was the scaled change in that area! WOW! See here and here. So, when you take the determinant of the Jacobian, you get a sense of how that non-linear transformation changed the area of stuff around that local space. If something was of size 1 before, and you calculate the Jacobian determinant after a non-linear transformation, you can say, “Ah ha, now it’s size 0.5!” Or whatever. And this determinant is different, of course, depending on which point in the space you choose to transform! Because it’s non-linear!

Perspective in art
#

So yesterday I doodled Oxford’s Radcliffe Camera –

– and was VERY irritated with my total perspective fail.

And, as I was watching the linear algebra videos, and Sal Khan was mentioning the application to 3D game development, I realized: VANISHING POINTS!

This led me, so happily, to the power and glory of Italian Renaissance genius - that most inspiring of geniusnesses. It was the Italian Renaissance masters - Brunelleschi, specifically, who built Florence’s magnificent cathedral - that realized how perspective was a mathematical and geometric property. So glorious! So reliable! I got really pumped about this, and about how linear transformations could be applied to art.

Jacobian, not Jacobin
#

Har har, well, I found it funny. No, it’s not that Jacobin. But oooh - an article about Italy!

TODO
#

Write a script to calculate the per-video and total duration of various Khan Academy courses. (Honestly, this should just be on the website!)
Apply math to art!?

Making a Twitter bot

Tue, 02 Jan 2018 00:00:00 +0000

I made my first Twitter bot today. It was surprisingly easy! I was worried I’d have to get all up in Oauth’s business (please God no), but I didn’t. Instead, I managed to go from zero to bot in the length of Philip Glass’s Koyaanisqatsi. Aw yiss.

– xkcd/1646

Specs
#

Auto-delete my old tweets from Twitter.
Save them somewhere else.

## Walkthrough

Let’s get started!

import os
import tweepy
import argparse
import pandas as pd
from datetime import datetime, timedelta

So, I grabbed os for file management, tweepy as my Twitter API Python wrapper of choice (docs), argparse for command line arguments, pandas for lazy data munging, and datetime stuff for filename management and defining what an “old” tweet is.

First, let’s connect! After registering my app on apps.twitter.com, I granted it Read-Write access and generated some user tokens. I put these in a local dotfile, because SECURITY. (Incidentally, whoa, check out this cool dotfile zoo…) I made a “hidden” function, _connect(), that’ll grab the permissions from my dotfile, do the whole Oauth thing, and return a Tweepy Twitter API object. “Hidden” because it won’t be accessible from the command-line, but rather called from the other main functions (below).

def _connect():
 print("Connecting to Twitter...")
 consumer_secret = os.environ['TWITTER_CSECRET']
 consumer_key = os.environ['TWITTER_CKEY']
 access_token = os.environ['ACC_TOKEN']
 access_token_secret = os.environ['ACC_SECRET']
 auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
 auth.set_access_token(access_token, access_token_secret)
 return tweepy.API(auth)

At this point, I should really test stuff out… but… I was lazy and continued.

Next, I wanted a function to archive all my tweets. I called this backfill(). No arguments, because - meh - I can’t think of any. It just reads through my entire timeline and archives what’s there into a csv, filenamed to the date.

Word on the street is Twitter is limiting the tweets it shows you on the website. Via the API, you can grab a default of 20 (or a maximum of 200) tweets from the api.user_timeline() method. This StackOverflow post includes the for loop to paginate through and grab up to 3,200 tweets.

def backfill():
 api = _connect()
 all_my_tweets = []
 print("Downloading all tweets...")
 for status in tweepy.Cursor(api.user_timeline).items():
 all_my_tweets.append({'id': status.id,
 'date': status.created_at.strftime('%Y-%m-%d'),
 'text': status.text,
 'retweets': status.retweet_count,
 'favorites': status.favorite_count
 })

 pd.DataFrame(all_my_tweets).to_csv(f"{datetime.now().strftime('%Y-%m-%d')}_backfill_twitter.csv")

After the backfill was done, and my precious old tweets were safe, I wrote the deleter function, delete_old_tweets():

def delete_old_tweets(date):
 api = _connect()

 # Ugh filename
 for filename in os.listdir():
 if '_backfill_twitter' in filename:
 twitter_backup = filename

 df = pd.read_csv(twitter_backup)
 df['date'] = pd.to_datetime(df['date'])
 print(f"{len(df)} tweets found.")

 to_delete = df[df['date'] <= datetime.strptime(date, '%Y-%m-%d')]['id'].values
 print(f"Deleting {len(to_delete)} tweets from before {date}.")

 for tweet_id in to_delete:
 try:
 api.destroy_status(tweet_id)
 except tweepy.error.TweepError as e:
 print(e)

Finally, I prep it for command-line execution. I add two flags/arguments: a boolean for backfilling (-b), and a string date for the oldest tweet you want to keep (-d). The default will be anything older than 30 days. This means that, if I go quiet on Twitter and I let the bot run, it’ll clean out my timeline entirely. Which is fine (but I might want to have it leave a little note there, then, “This timeline has been archived by a bot. Angela will return soon” etc etc).

if __name__ == '__main__':
 parser = argparse.ArgumentParser(description="Blue bird killer.")
 parser.add_argument('-b', default=False, action='store_true', dest='backfill', help='Backup all tweets?')
 parser.add_argument('-d', default=None, dest='date', help='Delete tweets before this date ("YYYY-MM-DD") (Default is anything older than 30 days).')
 args = parser.parse_args()

 if args.backfill:
 backfill()

 if args.date:
 delete_date = args.date
 else:
 delete_date = datetime.now() - timedelta(30)

 delete_old_tweets(delete_date)

Thoughts
#

My main sticking point was, as always: abstraction. I knew I wanted to: (1) archive before I delete, but (2) not archive the entire timeline every time, and (3) delete based on the Twitter object’s ID.

Koyaanisqatsi was wrapping up, so I was rushing and I made a mess instead: I separated out deleting from archiving (backfill() isn’t called in delete_old_tweets(), and I think that makes sense…), but I made a mess with the filenames and duplicate deleting. If you backfill on 2018-01-02 and want to delete tweets tomorrow, what filename do you choose? For now, I just choose any old archive filename, ignoring the date. Silly! Also, an archive csv is the snapshot of your entire timeline on that date - so if an archive is used for deleting more than once, you’ll run into an error: you’ll be trying to delete tweets you already deleted!

Much better would be a living archive, one with a status field that gets toggled: deleted, live, whatever. I guess I could set up a MySQL database or something? Or write over the csv when I delete? I don’t like either idea; one seems like overkill, the other seems brittle.

TODO
#

I could set up a cron job to run this locally, a la my automatic desktops script. I could put it on some server somewhere; buy some space on an AWS EC2 instance. I could (finally) set up my little Raspberry Pi and have it always-on. Or I could just manually run the script as I remember to (WORST OPTION).

For now, though, I’m happy with having a functional bot that only took about a Glass album to write (apparently about an hour).

Favorite track from Koyaanisqatsi
#

Prophecies!

Readings
#

Drawing- Anxiety

Sun, 31 Dec 2017 00:00:00 +0000

Happy with:

Right arm + hand
Squishiness of the tentacles
Shading
Coloring (esp. the blushing)
Movement quivers

Unhappy with:

Left arm + hand
Skull crush not gory/body horror enough
Mouth and nose are very lazy
Weird body position in general

Inspired a little bit by Thomas Nast political cartoons.

Clojure + cryptography = maximum fun

Fri, 29 Dec 2017 00:00:00 +0000

In March 2017, I attended the Boston ClojureBridge. Clojure is a functional programming language; it’s a dialect of Lisp, written on top of the Java Virtual Machine. In October 2017, I attended Clojure/conj in Baltimore. I don’t really use Clojure in my day-to-day programming (yet!), but I’ve definitely been flirting with it for a while now.

The tldr on why Clojure has been interesting and useful to me is:

A different programming paradigm. I learned languages in the following order: Stata -> JavaScript -> Python. I’ve been resting comfortably in PythonLand since 2014. It’s a wonderful place to be. But it’s also, ya know, only one way of doing things! ClojureBridge exposed me to a totally different way of conceiving of code. The ideas of an immutable state, of everything as a function, of functions as arguments, and even just the damn parentheses, were mind-bending and amazing. It’s helped me write better Python.
Exposure to more machine. Since I don’t have a degree in computer science (yet!), there’s a whole big wide world out there in The Machine. I didn’t really know anything about compilers until watching a couple lectures of CS50 and writing a bit of C. I still don’t know much! Like, I don’t really know what the JVM is. So Clojure is a fun way for me to explore!
Ground floor data science. Several Clojure/conj speakers talked a big game about how Clojure (or functional programming, anyway) and deep learning are a, ahem, match made in heaven. This was super interesting! Most of the Clojure deep learning, stats and machine learning libraries are still a bit alpha, but it was great to see the enthusiasm. And writing stuff from scratch is the best way to really learn and understand something (see below…).

Anyway, I wanted to write up my ClojureBridge notes because I ended up FALLING IN LOVE with the project from that day: some simple cryptography tasks. CRYPTOGRAPHY IS AMAZING, PEOPLE!

Whirlwind Clojure syntax
#

Everything is a function. Some functions are built-in (like +). Everything is parentheses. PARENS!

; This is a comment
;; This too

; General syntax
(function arg1 arg2 ..)

; Example
(+ 1 2)
>> 3

; Defining a variable (evaluated only once)
(def answer 42)

; Defining a function (evaluated every time code is run)
(defn hello-world [name] (println (str "Hello, " name  "!")))

; Printing a docstring
(doc func-name)

Project: Cryptography
#

The day’s project (github/track2-ciphers) was written by Elena Machkasova - thank you, Elena! The project takes you through some basic cryptography tools and puzzles: specifically, you write functions to encrypt/decrypt texts using both the Caesar cipher and the Vigenère cipher. It also touches on cryptanalysis using frequency analysis. In the end, Elena presents an encrypted text, and you try to Clojure your way to a solution.

It took me about ~15 hours to crack the final puzzle; 4 of them were at the actual ClojureBridge workshop in Boston, where I bumbled through Clojure and then got pretty tired by the afternoon, and the remaining 9 hours were me at home, puttering away for a few more evenings. Decrypting was hard, but SO FUN too. And the whole journey was FANTASTIC. I loved learning about Clojure, and I loved learning about cryptography - something I apparently love!? It was like the Platonic ideal of a day-long workshop. I felt super inspired by the end, and like I had enough tools and a motivating project to continue.

I’ll go through my method, with some notes on both Clojure the language and cryptography the mission.

Caesar cipher: Encryption and decryption
#

The Caesar cipher replaces each letter with the letter n places to the left. You select your n and that’s your encryption key. To decrypt, just shift everything n steps to the right. For example, if we choose a key of 2, we shift everything to the left by 2 steps and get an alphabet like so:

PLAIN: abcdefghijklmnopqrstuvwxyz
CIPHER: cdefghijklmnopqrstuvwxyzab
PLAIN: hello, world!
CIPHER: jgnnq, yqtnf!

If you choose a key greater than 26, you can divide by 26 and use the remainder. That is, modulo 26.

Here’s a Caesar cipher in Clojure:

;; Return the ASCII position of the first letter of the alphabet, A
(def ascii-a (int \a))

(defn abs-value
 [x y]
 (if (> x y) (- x y) (- y x)))

(defn to-int
 "takes a lowercase letter character and returns its position in the alphabet: a = 0, b = 1, etc."
 [c]
 (- (int c) ascii-a))

(defn to-char
 "take a number and return the lowercase letter character that it refers to"
 [num]
 (def abs_num (abs-value num 0))
 (if
 (<= abs_num 25)
 (char (+ abs_num ascii-a))
 (println "Warning: Number must be between 0 and 25 (inclusive)")))

(defn shift
 "shifting a letter to the left by the key"
 [char key]
 (to-char (mod (+ (to-int char) key) 26)))

(defn caesar-encrypt
 "encrypt a word with a key using Caesar cipher"
 [word key]
 (apply str (mapv #(shift % key) word)))

(defn caesar-decrypt
 [word key]
 (caesar-encrypt word (* key -1)))

Cracking the Caesar cipher
#

The Caesar cipher is pretty straightforward to crack. One way would be to just try all 25 possible keys (1 - 25). Even if the encryption key was originally >26, the decryption key will be modulo 26 of it.

Another way, if you don’t want to brute force it, is to conduct frequency analysis. Letters don’t appear randomly in a language; they have distinct frequencies. In English, e, t and a occur way more often than letters like x, q and z.

– Letter frequency in the English language

If your ciphertext is long enough, you could conduct frequency analysis and compare the resulting frequencies to the plaintext English letter frequency above. Like, you could run a Chi-square test for comparing whether two discrete distributions are the same.

Side note but this was SUPER GLORIOUS to me! What an amazing use of statistics! I felt like John Nash in that touching scene in A Beautiful Mind where he exclaims, astonished, “I would never have thought of that!” when the Nobel Prize guy tells him of all the amazing uses of Nash equilibria. (And, yes, they are pretty wild - behavioral ecology?! cancer?!)

Also, side side note but Nash - of course - did a lot of crypto. GLORIOUS CRYPTO HISTORY!

Vigenère cipher
#

The Vigenère cipher is an extension of the Caesar cipher. Well, sorta. Instead of choosing a key, n, which we shift alphabet letters by, we instead choose a key phrase (a password, if you will…) and use that to jumble up our plaintext. Specifically, the password letters are added to the plaintext letters; that sum is the ciphertext.

PLAIN: abcdefghijklmnopqrstuvwxyz
KEY: passwordpasswordpasswordpa
CIPHER: pbuvatxkxjcdibfsfrklqjnanz

And in Clojure:

(defn encrypt-letter
 "Add two letter's alpha positions together, convert that sum to new letter"
 [c1 c2]
 (def summed (reduce + (mapv #(to-int %) [c1 c2])))
 (to-char (mod summed 26)))

(defn decrypt-letter
 "Subtract two letter's alpha positions together, convert that difference to new letter"
 [c1 c2]
 (def subbed (reduce - (mapv #(to-int %) [c1 c2])))
 (to-char (mod subbed 26)))

(defn vigenere-encrypt
 "Cycling through a keyword, encrypt by summing the plaintext letter + keyword letter"
 [text keyword]
 (def clean_text (get-letters text))
 (apply str (mapv #(encrypt-letter %1 %2) (cycle keyword) clean_text)))

(defn vigenere-decrypt
 "Cycling through a keyword, decrypt by subtracting the plaintext letter + keyword letter"
 [text keyword]
 (apply str (mapv #(decrypt-letter %1 %2) text (cycle keyword))))

Cracking the Vigenère cipher
#

Cracking the Vigenère cipher is similar to cracking the Caesar cipher: frequency analysis. (There’s other methods, but I haven’t worked through them yet.) This only works if you have:

Some idea of how long the key phrase is. (Otherwise, you could be conducting freq analysis for a very long time indeed.)
A ciphertext that is pretty long. (Otherwise, your sample size will be too small and your freq analysis too noisy.)

In Elena’s example, she gave us two hints: the key phrase was between 2 and 8, and the ciphertext was related to ClojureBridge. She also gave a great guide on why subsequence frequency analysis works - this blew my mind.

Trying to explain it back to myself, I broke down my cryptanalysis into these steps:

You guess a key of length n (e.g. [k, e, y] would have length, n = 3), and break up your ciphertext into n subsequences.
Each subsequence is all the letters that that key character (e.g. k) is encrypting.
Use the Chi-square test to check whether the distribution of letters in your subsequence is similar to the English language. If you see a near uniform distribution of letter frequencies, it’s probably the wrong n - that is, if you just got random mush back, you didn’t pick the right key length. Start again.
If you do see patterns in your subsequences’ histograms, then you can check the Caesar encryption distance between the most frequent letter in your subsequence and the most frequent letters in English (e, t, a). You then take the subsequence of the nth+1-th letters, and predict which letter the next character in key ([k,e,y] -> e) is.
You put together all the most-frequent-letters by subspace into your n-length key (which should, hopefully, sound like a word).
Then you apply your key to decrypt the entire text.

(This speaks to the value of long passwords and the inherent insecurity of having lots of data about ourselves out there. The longer your key phrase, the smaller the subsequences, the noisier the frequency analysis. The more encrypted data you put out there about yourself, the more data is available for cryptanalysis. I know the Vigenère cipher isn’t how modern encryption works, but the general principle did give me pause.)

And in Clojure:


(defn count-letters
 "Count the number of times a letter, char, occurs in a string, s"
 [char s]
 (count (filterv #(= char %) s)))

(defn freq
 "Return frequencies of each letter in a string, s"
 [s]
 (zipmap alphabet (map #(count-letters % s) alphabet)))

(defn freqs
 "Sort letter frequencies (desc) in a string, s"
 [s]
 (sort-by second > (freq s)))

(defn take-nth
 "Split text, s, into subsequences of every n letter"
 [n s]
 (take n (freqs s)))

(defn chi-square
 "Taking the observed freq hash, obs, and compute Pearson coeff (lower is better) against English lang freqs"
 [obs]
 (def exp (hash-map \e 12.702, \t 9.056, \a 8.167, \o 7.507, \i 6.966, \n 6.749, \s 6.327, \h 6.094, \r 5.987, \d 4.253, \l 4.025, \c 2.782, \u 2.758, \m 2.406, \w 2.360, \f 2.228, \g 2.015, \y 1.974, \p 1.929, \b 1.492, \v 0.978, \k 0.772, \j 0.153, \x 0.150, \q 0.095, \z 0.074))
 (def e (vals (sort-by first exp)))
 (def o (vals (sort-by first obs)))
 (def subtracts (map #(- %1 %2) o e))
 (def squares (map #(* % %) subtracts))
 (def normed (map #(/ %1 %2) squares e))
 (reduce + normed))

(defn try-key
 "Given an encrypted text, text, try a key of lenth n and return Chi-square test"
 [text n]
 (def every-nth-letter (take-nth n text))
 (def freq-every-nth (freqs every-nth-letter))
 (def freq-hash (freq_hash freq-every-nth))
 (chi-square freq-hash))

Honestly, I haven’t looked at this code in a couple months, and it’s a mess. One thing I need to learn is organizing a Clojure project. For now, though, I just manually ran through try-key with the key lengths between 2 and 8, and noted down, first, whether it looked like non-mush came back and, if so, what the likely key character was.

I did this until I had a key length and key that looked reasonable, and then I checked it against the encrypted exercise text. When it worked, it was AMAZING!

This manual method was not great, though, since my human brain is a pattern-recognition machine, and so I saw a lot of patterns that weren’t there. Automating it more would have probably helped in avoiding false-positive traps that my brain kept laying, as well as removing some of the general manual pain.

Drawing- My commute

Fri, 29 Dec 2017 00:00:00 +0000

Today’s commute included:

The next few chapters of Tony Judt’s Postwar (audiobook). Today’s chapters focused on 1956: specifically, the Suez Canal crisis and the Hungarian revolt.
Sub-freezing temps.

Pencil sketch
#

Inked
#

Final
#

Still looking for a style that I like. I’ve been stuck on this current “art skills” plateau for a loooong time (10+ years). But I also haven’t really been drawing/arting for 10+ years. So I have hope!

TIL- Multiple cursors in Jupyter

Thu, 28 Dec 2017 00:00:00 +0000

This tutorial by perfectlyrandom.org had been on my TODO list for about 5 months. Today I finally did it! Jeez. I basically failed productivity hacking. Anyway.

xkcd/1205

Multiple cursors are great
#

Multiple cursors are great, and they’re available (mostly by default) in a lot of text editors: Sublime, Atom, PyCharm. They are a yuuuge part of my daily productivity. I don’t know what I’d do without them.

Except that, until now, I had indeed been surviving without them in my other main window of choice: the browser-based Jupyter notebook. Until now. UNTIL NOW!

TLDR tutorial
#

The full tutorial is here. But here’s the tldr of what I did:

cd ~/.jupyter
mkdir custom
subl custom/custom.js

Then, once in custom.js, this worked for me (it’s the first comment):

require(["codemirror/keymap/sublime", "notebook/js/cell", "base/js/namespace"], function(sublime_keymap, cell, IPython) {

 cell.Cell.options_default.cm_config.keyMap = 'sublime';

 var cells = IPython.notebook.get_cells();

 for (var c=0; c < cells.length ; c++) {
 cells[c].code_mirror.setOption('keyMap', 'sublime');
 }

 });

Aw yiss.

2017 in review- Books

Sat, 23 Dec 2017 00:00:00 +0000

Meh. This year was just an okay reading year for me. At least, I felt like I read a lot of stuff which has been deeply influencing my day to day thinking (high Influence Factor!), but the Pleasure/Inspiration Factor was just okay. I mean, I did enjoy many books this year. But none of them made me want to shriek with joy, so we’re not talking peak book joy.

I also noticed that, this year, my reading followed a few distinct themes:

More politics/US history books than usual. For obvious reasons.
Less sci-fi than usual. :(
Less graphic novels, more audiobooks.

Anyway, let’s begin Official Book Reflection Time.

Most pleasurable read: The Undoing Project, Michael Lewis
#

Non-fiction. About the intellectual romance between the two giants of behavioral economics: Daniel Kahneman and Amos Tversky.

I LOVE behavioral economics, and I love learning about the humans and humanity behind these big economic ideas. I had already really enjoyed Nina Munk’s The Idealist, about Jeff Sachs and his (somewhat infamous) Millennium Villages project, and this book is in the same vein. If you already know the research, it’s catnip. If you’ve never heard of behavioral econ, I think it gives a good overview. The portrait of these two, very different individuals (it’s an odd:q couple buddy movie!), the history of Palestine and Israel, the drama of academia. It was so good. I was especially charmed by Amos Tversky’s biting wit.

Most guilty pleasure read: Endymion, Dan Simmons
#

This book has no right to be as fun as it is. It’s the third book in a four-book sci-fi series, i.e. it should be VERY BAD. The first one, Hyperion, won the Hugo and Nebula, and it is super good. But my experience with sf series is that the quality of the sequels is inversely related to the quality of the first book. Like, exponentially. The Fall of Hyperion (book 2) was not so bad, but I was still expecting Endymion to, well, suck.

But it doesn’t! It’s basically glittering garbage B-grade wonderfulness. It’s no longer as mystical and awe-inspiring as Hyperion 1 and 2, instead it becomes silly and FUN. What an adventure! And, ho man, did I love Father Captain Federico de Soya, the Catholic space captain with a heart o’ gold. Played by Luigi Lo Cascio in my head. Specifically Luigi Lo Cascio at 0:30 in the Noi Credevamo trailer. SPACE CATHOLIC CAPTAIN FROM ITALIA! (He’s actually Spanish, but whatevs.)

Book that made me reevaluate fiction as an art form: Americanah, Chimamanda Ngozi Adichie
#

Sometime in ~2006, I stopped reading fiction and switched entirely to sci-fi and non-fiction. A friend of mine rightly pointed out that you kinda can’t discredit an entire art form, but I was like, NOVELS HOLD NO TRUTH, I WILL NOT WASTE MY TIME.

Oh, how wrong I was. How wonderful Americanah is. How I laughed and cried and marveled at Adichie insight and acumen. SO GOOD.

Best graphic novel: Sex Criminals, Matt Fraction, Chip Zdarsky
#

Kind of a toss-up with Bitch Planet, with some of the same themes of sexuality, sexual politics, and gender. But whereas Bitch Planet is a screaming ball of fiery rage, Sex Criminals is more relaxed: it’s fun and smart and often touching. I read volumes 1 and 2 and loved them.

Best book to understand American politics: The Unwinding, George Packer
#

Okay, let me preface this by saying that I read A LOT of books about the US this year, specifically: Hillary’s campaign memoir, Katy Tur’s campaign memoir, Alyssa Mastromonaco’s White House memoir, John Dickerson’s book about campaigns, Ta-Nehisi Coates’s Obama memoir, a book about White House Chiefs of Staff, Al Franken’s campaign memoir (cringe…), and a huge tome about 17th century English migration patterns to the US and how that explains all of modern American politics. So… a LOT OF COMMENTARY on the whole Trump stuff. I’m also reading Postwar now, about Europe after WW2, and that’s informing how I see the US political moment.

But none of them had me as horrified or as certain as George Packer’s The Unwinding. Reading that, Trump’s election and the rise of know-nothing populism seems inevitable. It’s told as a series of insightful, humanistic mini-biographies about Americans both famous and everyday. And it’s an incredible portrait of how our politics and culture have changed in the last thirty years.

…and happiness chaser: Best Kim Stanley Robinson book from this year: The Years of Rice and Salt
#

Super Buddhist mystical reincarnation adventure through a crazy epic alt history, WHAT’S NOT TO LOVE!?

Data is my co-pilot

Thu, 21 Dec 2017 00:00:00 +0000

Article report time! Today I read this, Five Ways to Fix Statistics, Nature (Nov 2017). It interviews six statistics luminaries about what’s “wrong” with modern statistics. Here are some choice quotes (all emphasis mine):

“In the past couple of decades, many fields have shifted from data sets with a dozen measurements to data sets with millions. Methods that were developed for a world with sparse and hard-to-collect information have been jury-rigged to handle bigger, more-diverse and more-complex data sets. No wonder the literature is now full of papers that use outdated statistics, misapply statistical tests and misinterpret results. The application of P values to determine whether an analysis is interesting is just one of the most visible of many shortcomings.”

– Jeff Leek

“In many fields, decisions about whether to publish an empirical finding, pursue a line of research or enact a policy are considered only when results are ‘statistically significant’, defined as having a P value (or similar metric) that falls below some pre-specified threshold. This approach is called null hypothesis significance testing (NHST). It encourages researchers to investigate so many paths in their analyses that whatever appears in papers is an unrepresentative selection of the data.”

"Statistical-significance thresholds are perhaps useful under certain conditions: when effects are large and vary little under the conditions being studied, and when variables can be measured accurately. This may well describe the experiments for which NHST and canonical statistical methods were developed, such as agricultural trials in the 1920s and 1930s examining how various fertilizers affected crop yields. Nowadays, however, in areas ranging from policy analysis to biomedicine, changes tend to be small, situation-dependent and difficult to measure. For example, in nutrition studies, it can be a challenge to get accurate reporting of dietary choices and health outcomes."

– Blakeley McShane & Andrew Gelman

“The plethora of options creates a hazard that statistician Andrew Gelman has dubbed the garden of forking paths, a place where people are easily led astray. In the vast number of routes, at least one will lead to a ‘significant’ finding simply by chance. Researchers who hunt hard enough will turn up a result that fits statistical criteria — but their discovery will probably be a false positive.”

– Michèle B. Nuijten

“Norms are established within communities partly through methodological mimicry. In a paper published last month on predicting suicidality, the authors justified their sample size of 17 participants per group by stating that a previous study of people on the autism spectrum had used those numbers. Previous publication is not a true justification for the sample size, but it does legitimize it as a model. To quote from a Berwick report on system change, “culture will trump rules, standards and control strategies every single time” (see go.nature.com/2hxo4q2).”

– Steven Goodman

The tldr is that:

We’re too confident. We hide uncertainty. We overstate claims.
We do that because of (a) incentive structures around academic publishing, (b) cognitive errors and bias around wanting certainty.

Some of the proposed solutions are:

Trial registries like ClinicalTrials.gov and the AEA RCT registry (I used to work on this! oh memories).
Changing norms around p-values for grant funding, article publication, etc.
Accepting, with cold existential dread, that we live in a probabilistic universe.

This article also made me think about the data science hype/bubble. Data-driven decision making is definitely preferable to, uh, gut-driven decision-making? norms-driven decision making? There’s still a lot more data out there to be scienced. So the bubble is, maybe, merited. I mean, I work in the bubble, so please please let’s keep this data thing going, okay!? But I don’t worry too much. We live in a world where enormous amounts of data are being passively generated, and this is only going to increase (exponentially?) as IoT stuff takes off. The day your WiFi-enabled fridge starts judging you will be a day of much glory for statisticians and machine learningers.

But data literacy is so important, and so under-taught. And if my human brain understands Daniel Kahneman and Amos Tversky’s work on human brains, we are also just not great at understanding uncertainty. We want certainty - and positive results! As some colleagues used to say, half-joking, “up and to the right!” But null results are important; counter-intuitive (in a not-sexy, non-“ah ha!” way) results are important.

I feel like, right now, there’s a lot of gatekeeping around data science as being something very complex and difficult, and a treatment that it’s a black box full of dark magic. It’s not: it’s mostly linear algebra (which you can learn here). Worryingly, people - even fancy people - still conflate prediction with causation. It’s not: it’s mostly correlation. Hence machine learning’s infamous reputation for perpetuating bias; if you don’t care to disentangle why something worked, you’re less likely to confront ugly correlations and disentangle them for what they are.

Anyway. Some random thoughts. I definitely recommend Joel Best’s Damned Lies and Statistics. It’s not very technical, and is instead more interested in the history and culture of stats - something that might be just as important as the technical side.

I <3 waves

Wed, 20 Dec 2017 00:00:00 +0000

My formal education in physics ended in high school - what a shame!

As an adult, I’ve had big interests in psychedelic mind-bending modern physics (Kip Thorne et al.) and early 20th century atom-splitting physics (Einstein, Oppenheimer, Bohrs, et al.). I mean, hello, sci fi!

In addition to big physics ideas, I’m also interested in the humble wave. Specifically, sound waves and radio waves. (Also rogue ocean waves and surfing memoirs.)

Recently, I was noticing that my Bluetooth headphones were stuttering whenever I went jogging. They did this near certain busy intersections, and were stutter-free when I was on trails or rural roads. This got me thinking: how does Bluetooth work? How do radio waves work? Clearly it was time for a PERSONAL WAVE WORKSHOP.

“Hearing” is tiny hairs in your ears being moved by sound waves. These are called stereocilia.

Some random stuff I learned about Bluetooth
#

Bluetooth was developed by Ericsson in the 1990s and was named after a 10th century Viking king, Harald Blåtand Gormsen, AKA King Harald Bluetooth, I am not making this up, the jokes write themselves, people.

Blåtand was known as a bringer together of people, helping them to communicate (get it…). The Bluetooth logo is made up of his initials, H and B, written in runes - AKA ye olde Germanic script.

One of the world’s biggest big wave hot spots is Teahupo’o, Tahiti. Here’s some mystical big wave surfing to enjoy.

Modern Bluetooth/Blåtand operates via radio waves in the 2.40 GHz - 2.485 GHz frequency band. It has a range of ~1m (Arduino widgets?) to ~10m (my phone and headphones) to ~100m (industrial strength). It sends data in discrete packets. To avoid crowding problems, these packets can be sent along 79 “Bluetooth channels”. They hop around, looking for the least crowded channel.

Stepping back: What’s a radio wave?
#

That was my next question. Okay, so WiFi and Bluetooth are technologies that send data on radio waves. But what is a radio wave? And how does information get encoded (in binary, I guess?) and sent along it? I literally had no idea. Here’s what I learned:

Radio waves == electromagnetic waves, since they’re made up of electric and magnetic signals. The signals are at right angles to each other. So it’s like two waves, one electric and one magnetic, in a helix-y.

source

These EM waves are the same you use in your microwave, from your WiFi router, from the cell phone tower, the radio antenna and the TV antenna. And Bluetooth!

They travel at the speed of light. You can measure the wavelength of an EM signal by dividing the frequency (e.g. 2.4 GHz for WiFi/Bluetooth) from the speed of light (299,792,458 m/s). Okay, I did this for WiFi/Bluetooth and it’s 0.12m, or 12 cm. Google says I’m right! Scientific notation for “wavelength” is $\lambda$.

There’s a direct relationship between the wavelength and the size of the antennae - bigger (~~slower~~? SPEED OF LIGHT) frequencies, like AM radio stations, have much bigger wavelengths (see equation above - the denominator shrinks, ergo the wavelength expands) and thus require LITERALLY LARGE ANTENNAS. I guess that means that our cell phone antennas, which are tiny, pick up the (~~much faster~~?) EM waves of WiFi and phone signals.

You can jack up antennas to have “gain”, which boosts the signal. This is done by manipulating the shape of the antenna. I think this might be related to what Crash Course was saying about constructive interference, where you amplify waves.

There are:

ground waves, that follow the shape of the Earth
line of sight waves, that get thrown off by the shape of the Earth (at the radio horizon, new band name?)

Short wavelengths, like Bluetooth (!), can be absorbed by rain droplets.

Random stuff I learned about waves in general
#

Frequency is the number of wave cycles that occur within one second. This is measured in hertz. So 2.4 GHz - the frequency band where Bluetooth and WiFi are - is 2,400,000,000 wave, or 2.4 billion, cycles in 1 second.
Waves carry information (through space and time!).
Most wave equations seem pretty simple!
Waves coming at each other in opposite directions can augment (constructive interference) or decrease (destructive interference) each other’s energy. This is now noise-cancelling headphones work - they send out “inverse” sound waves! SO COOL!
Sound waves don’t move perpendicular in motion (like a rope being oscillated) but in a compression-squeezing way (like a spring). WHAT. Is this why, when my phone is on speaker and I have it right up in my face, I can feel small bursts of air coming from the speakers as the person on the other side talks?
Sound perception is logarithmic, not linear, to actual loudness/intensity.
That you can use a method developed by a guy named Huygens to figure out where a wave is going: basically, every point on a wave generates a semi-circular (semi-spherical?) ‘wavelet’ - the line tangent to all the wavelets is where the wave will next be.

Colors are just light waves with different wave lengths. Red has the biggest distance between wave peaks - blue the shortest. There are “colors” beyond this, in wavelengths the human eye can’t perceive. The unit measure of light wavelength is the nanometer, or 0.000000001 meters.

Conclusion: Some hypotheses for why my Bluetooth headphones are crapping out
#

Action plan time!

Bluetooth’s fault: Rain. There’s a high correlation between me running in a dense city rain, and me running on a rural road with clear skies. Just by chance. So maybe I’ve misattributed the problem to city density, and not rain, and the fact that high-frequency EM waves can be absorbed by radio droplets.
- Test: Go running in the city dense area when it’s sunny. Do this enough times for statistical significance.
Everyone else’s fault: Busy channels. Bluetooth travels along 79 channels, one for each megahertz between 2.4 GHz and 2483.5 MHz. When I run in device-dense areas, especially near intersections, all 79 channels might be maxed out.
- Test: Identify places where (I guess) the channels will be more or less crowded. Run there.
My headphones’s fault: Antenna design. Signals have a polarity which can be horizontal, vertical, or circular. Antenna get the best signal when they’re rotated to the same axis as the polarity. If my cellphone emits, say, a horizontally-polarized signal, and my headphones are better positioned for a vertical one, I may get a worse signal.
- Test: Wear headphones and place phone at different spots around them. e.g. Above the headphones, below them, rotated, etc.
My fault: my running pouch. It sounded like high frequency signals, such as WiFi/Bluetooth, are pretty fragile and can get disrupted more easily than long, slow, low frequency signals. While some materials (e.g. cement, trees) are known to disrupt stuff, maybe my cloth and plastic running pouch is disturbing the cell phone-headphone link.
- Test: Run with the phone in my hand.

So I have a bunch of potential experiments now, and - given the amount that I run - my sample size won’t be big enough to detect changes for another, well, several months or so. But I have a plan! Which is awesome.

Resources
#

Bluetooth:

Radio waves:

Priors all the way down

Mon, 18 Dec 2017 00:00:00 +0000

Today I learned something very handy: how to encode subjective beliefs into a Bayesian prior when you only have information on the quantiles.

I hadn’t realized it, but this is super useful: nobody can give you a fully described distribution, but everyone can give you a range of likelihoods, and talking with people is one (important) way to inform your prior. For example, a person might say, “well, I’m pretty sure most people love pizza between 50 and 70 happiness points”, or somesuch. Previously, I wouldn’t have known what to do with that information. How do I fit a probability density function around that pizza love? When I have only two vague points?

In the past, whenever I wanted to set a prior, I would do one of the following:

Set an uninformative one, such as the uniform distribution. This assumes that everyone loves pizza an equal amount.
Set a prior based on some historical data, e.g. fitting a beta distribution to some observed historical data and pray that nothing has changed since that data was collected/it doesn’t bias my updating too hard. I feel like this is Sullied Bayes, though, since we’re not technically supposed to be using data in the “setting the prior” stage. But, well, okay, I’ve done it. I do it.
Build a model in PyMC3, hand-picking some distributions and hand-picking their parameters, based on some subjective and vague understanding of the world. E.g. “I know that I love an average pizza at about 70 points and I guess I’ll use a Normal distribution around that mean with a low variance of, uh, 5 points, because - honestly - pizza is never that bad, right?” What nonsense, I know.

But today I was hanging out on StatsOverflow and came across a very useful idea: how to incorporate quantiles information into a prior. YES. What a useful thing.

Here’s how it works in practice:

SCENE 1. The RENAISSANCE BOTTEGA where you work. Brunelleschi’s DUOMO can be seen via a semi-open window. Enter your COLLEAGUE, stage left, pursued by a bear. Upon closer glance, you see that the bear is not a bear, but instead a shimmering beta distribution of indeterminate shape and size. YOU are enchanted.

YOU: I am enchanted! Tell me more about your beta distribution! What is it about?

YOUR COLLEAGUE: This beta distribution refers to the pizza example from previous paragraphs ago! I know not much yet, but my priors are that most people like pizza between 50 and 70 happiness points! Where 100 is maximum happiness!

YOU (meditatively): What in the world do you mean by “most people”…?

In this case, I would: (a) assume “most people” means “95% of people”, (b) normalize my pizza happiness points down to a beta distribution-able scale (50 - 70 becoming 0.5 - 0.7), and, (c) numerically calculate, using a least squares errors-ish approach and scipy’s optimization algorithms, the beta parameters, $\alpha$ and $\beta$, that yield the correct proportions along a beta’s cumulative distribution function. That is, find the $\alpha$ and $\beta$ where Pr(X < 0.5) = 0.025 and Pr(X < 0.7) = 0.975. You can read more about the method here and solving it using scipy’s optimize.fmin algorithm here. I did such a thing, and voila!

And now we can do descriptive stats of this crazy ol’ prior we just confabulated out of the air!

SCENE 2. The BOTTEGA at night. You are manually debugging a NEURAL NETWORK using a power tool that emits sparks. A 3D PRINTER is printing Brunelleschi’s DOME onto plastic.

YOUR COLLEAGUE (entering suddenly): Do you have it?! Do you have the numbers!

YOU: I do. (raising paper triumphantly) Our prior belief about the average, by the power of the

$$E(X) = \frac{\alpha}{\alpha+\beta}$$

incantation, is that it’s approximately 0.602137284331 -

YOUR COLLEAGUE: Hermahgerrddd STAHP. Don’t use droning floating point precision to imply accuracy! It doesn’t even matter, too. Since we’ll be updating that with actual data soon, it’ll become irrelevant.

YOU: So true. Long live the posterior!

YOUR COLLEAGUE: Long may it reign.

Reading
#

Stats StackExchange: Help me understand Bayesian prior and posterior distributions
Stats StackExchange: Do two quantiles of a beta distribution determine its parameters?
John Cook: Parameters and percentiles
John Cook: Finding Probability Distribution Parameters from Percentiles
Hexing the technical interview - related to this post in style, not content; v wonderful

How I read

Sun, 17 Dec 2017 00:00:00 +0000

This year, I read 104 books. My goal was 100. This is probably the maximum number of books I can read and still get something out of. The last year I read 100+ books was 2014, and that was - oh man - a very good year for reading. I discovered some treasures like Jonathan Strange & Mr Norrell, Asterios Polyp, Spam, The Everything Store, the first set of Michael Palin’s diaries, and forgotten sf classic Floating Worlds. Hooo man, such book nostalgia. What a year. This year has been - meh, pretty good. Fine. Whatever.

Anyway, people sometimes ask me how I read so “fast”. I don’t think I read very quickly: it’s more like a marathon, using many different vehicles, than a sprint. But I love reading, and I love talking books, so here are my current reading habits:

1. Audiobooks
#

This year, 60+ of the books I read were audiobooks. I discovered audiobooks in 2016 when I had a super long commute (1.5 hours daily) and, following the dramatic and emotionally wringing Election 2016, I needed a break from podcasts. Rather than staring into space or awkwardly avoiding eye contact with other commuters (SO AWKWARD), I decided to start climbing my mythical book mountain - a mountain I had been building since the late 90s. Could it be summited?! Audiobooks made the dream… well, a possibility.

Also! Around the time I started audio-reading, a friend of mine said two things which further changed my reading life:

“You should ‘speed’ listen.” Yes! My attention actually wanders less if I speed audio up to 1.5x or 1.75x. I think audiobook narrators read intentionally quite slowly (and enunciate really well!), so 1.5x - 1.75x is just a bit faster than normal speech. This can get a bit weird for highly produced audiobooks or radio plays, where there’s music and sound effects, but meh - it’s not a huge deal. At these speeds, I can get through a standard ~300 page book in 2-3 days, usually.
“There’s no such thing as chore time anymore, there’s just podcast time.” I don’t love tidying, but I love a tidy space, and I realized that I could crush all my daily TODOs if I was jamming to Tversky and Kahneman’s bickering or sobbing uncontrollably over Eleanor and Park’s teen romance. Hell yeah, laundry time! Does that tub need a scrubbing?! LET’S DO THIS.

Also, exercise is another notable chunk of time when I get my audio-reading jam on.

Buying wireless headphones made things even easier - now I could cook to Trevor Noah, jog to Ursula K. LeGuin, and deadlift to Kim Stanley Robinson. Glorious!

2. Technical books
#

My mythical book mountain now has a considerable second peak devoted to computer science and machine learning. Enter… O’Reilly. And Manning. And No Starch. I read these at work, mostly.

3. Graphic novels
#

Graphic novels are great - gorgeous art, witty writing, some really out-there storylines. I didn’t start reading graphic novels until a few years ago, when a friend shoved Y: The Last Man into my hands (well, external hard drive) and, lo! A great love was born.

4. Digital sabbaths
#

Digital sabbathing merits its own post, but the basic idea is: no screens of any kind for 24 hours. After some debate, this includes Kindles. I have a pile of dead-tree books that also need a-readin’, and digital sabbaths are often the only times I actually get to them.

5. Always be reading
#

As I said, it’s a marathon, not a sprint. I always have at least one book in circulation and I’m always (audio-)reading. My favorite feeling is those brief moments being in between books - “OH BOY WHAT SHALL I READ NEXT” - and I love curating and re-ordering my various wish lists.

6. Cut the chord
#

Another thing that I would recommend is getting ruthless about DNFing (did not finish-ing). Life’s too short to plow through terrible/boring books, and languishing in that “must finish boring book” limbo is never worth it.

7. I already bought these books with my taxes
#

It was this year, 2017, that I discovered that: ho my God, you can borrow audiobooks and ebooks from your local library! So magical! I use the Libby app now and max out my holds.

8. Social motivation, i.e. Goodreads, i.e. ego (let’s be real)
#

Most book clubs I’ve gone to have been wine-and-snacks clubs, really, but Goodreads! Ah! How that website has motivated me. I love seeing what my friends are reading. I love showing off what I’m reading. I love writing my reviews and I love reading my friend’s reviews. Hell, I love reading my old reviews. It’s like a diary, with books! It’s like a book club that never ends! It’s inspiring, and fun, and the ego-boost of seeing my friends like one of my reviews, or seeing myself inch closer to my annual reading goal, is big. BIG. I think that’s also the reason I can’t get many of my friends to do Goodreads: the most common complaint is, “Oh, I don’t read enough.” I always tell them not to think of it as a race, but - well - I guess there is a performance aspect to it? I dunno. I just love writing reviews.

9. Insatiable/ravenous curiosity
#

I often overestimate books and expect most of them to change my life. A few do! A FEW DO. The human brain releases the most dopamine (and gets addicted quickest) when rewards are randomly dispensed, so there you go: this book may change your life. Who knows! Pull the lever!

Power bummer

Thu, 14 Dec 2017 00:00:00 +0000

I’ve been thinking a lot about power lately. One of my favorite things! I needed to calculate power for some experiments we wanted to run at work and, as with all things, I was seized by existential doubt.

As a brief intro: statistical power is the likelihood of seeing an effect size when there is one. For those of you that like Type I/II errors, your power is your ability to minimize your Type II errors (false negatives). I think it might be fun to do a power walk-through (or power-walk through, heh), but I’ll leave that for another day. For now, assuming you like stats and you like power, here are some thoughts on some stuff I’ve been reading.

Why we normally use power
#

Say you want to measure the effect of something. For example, what’s the effect of textbooks on student learning (not much)? What’s the effect of nearby job opportunities on parents’ investment in their daughters’ education (big)? What’s the effect of Facebook on workers’ productivity? Or of protein on a person’s likelihood to commit a crime? Or whatever. I’m just making things up. There’s lots of possible experiments out there.

To detect an effect size, you can go about things as a frequentist or as a Bayesian. We’ll stick with the former for now, since that’s what most people do. But a Bayes post - many Bayes posts - will come! I love Bayes. Anyway, under the frequentist worldview, you are faced with this reality:

You can pull a sample from a population.
If you pull a large sample, you’ll probably be more likely to hit the population average. (See the law of large numbers.)
But you could be unlucky and have pulled a crappy sample, whose average is way off the map.
Anyway, you’ll be working on a bell curve, which is convenient for the statistical tests you’ll run. (See the central limit theorem.)
In an experiment, you think you’re pulling from two populations - your control population, and your treatment population. A population which has been unalterably changed by your experiment!

The first thing people usually ask is: so how big should our sample be? This is where power comes in. Power is the likelihood of detecting an effect when there is one (i.e. when there is a separate population you’re sampling from, i.e. when things did change). It looks like this:

It’s that red thing. The “industry standard” is power = 80%, so you solve your sample size for that. How you solve it depends on what you think the shape of those distributions are. For now, following the Wikipedia example walk-through, let’s assume we’re going to run a t-test of significance between two independently-pulled sample means, and we’ll shoot for a 5% statistical significance level. In that case, the math would be:

where

$B(1)$ is power
$\phi$ is the quantile function
$1.64$ is the t-value threshold for significance
$n$ is your sample size
$\hat{\sigma_{D}}$ is the standard error

Normally, you fix $B = 0.8$ (for 80% power) and solve for $n$. There’s lots of online calculators and built-it methods in R (power.prop.test), Stata (power), and, for Python, the statsmodels.power module in statsmodels (docs).

Here’s power for a simple A/B test:

def power_calc(minimum_detectable_effect):
 A_mean = data['column_of_interest'].mean()
 B_mean = A_mean * minimum_detectable_effect
 stat_sig_threshold = 0.95
 desired_power = 0.8

 print(f"Detecting any changes of {A_mean:.2f} -> {B_mean:.2f} and above.")

 std_effect_size = (B_mean - A_mean) / data['column_of_interest'].std()
 n_per_group = sms.TTestIndPower().solve_power(std_effect_size,
 power=desired_power,
 alpha=(1-stat_sig_threshold), ratio=1)

 return n_per_group

This returns the sample size per experimental arm (n_per_group). If you were running an A/B test, then you’d need to multiply that by 2 to get your total sample size needed. Generally, you maximize power by splitting cohorts up evenly, though you don’t necessarily have to.

Why power can fail us
#

I read a super interesting blog from the Airbnb engineering team: Selection Bias in Online Experimentation: A Method for Winner’s Curse in A/B Testing. The tldr on it is that we tend to compound upward biases when aggregating cumulative effects over sequential experiments. In other words: don’t add up wrong numbers! Or you’ll get a very wrong number!

The blog went into some heavy stats/prob math, but my general intuition was: when we calculate power and hunt for significance, we often privilege large effect sizes. Note above that the minimum detectable effect size is just that: a minimum. If we were to observe a huge value in the B cohort, for example, the LLN/CLT would tell us that such an effect would be highly improbably random. In other words, it’s probably an effect! What we then do (which is wrong) is take that point estimate as the effect size. In other words: big estimated effects are more likely to look statistically significant (and therefore “win”) but may not actually be that big. The true effect might be there, just smaller. But we write down the big number. This is the “winner’s curse”: we favor big effects, since they satisfy our need for significance, and we miss small effects since they look insignificant.

Wonderful statistician/contrarian Andrew Gelman has a recent paper about this as well: Type M error might explain Weisburd’s Paradox (co-authors are Torbjørn Skardhamar and Mikko Aaltonen). He makes a convincing case that our quest for statistical significance and power calculations (especially ex post power calcs) - driven by a publication bias that favors positive, significant results - doom us to bias our effect size upwards. In the aggregate, as John Ioannidis would say, most research findings are false. But we can’t even be sure, because most people don’t even publish their data at all! Thus making meta studies even harder to do!

Figure 1 from Gelman et al. 2017

Some other take-aways from Gelman’s paper:

They propose ditching the Type I/II error methodology/obsession, since it doesn’t apply to studies that have non-deterministic, heterogenous outcomes (i.e. basically every study of human behavior). Instead, they discuss a new focus on Type S (sign) and Type M (magnitude) errors - i.e. minimizing getting things qualitatively and exaggeratedly wrong. The above figure is, I guess, how you’d go about determining those likelihoods.
There is a thing called the “Weisburd paradox”, which is that sample sizes have not actually been shown to improve power. (A replication study.)

Conclusion
#

The conclusion I always get from this stuff is that it’s all lies, damned lies, and statistics! In other words, I’ll do power out of interest and maybe even to build intuition, but - if I can - I’ll trust something Bayesian instead.

Reading
#

Automatic desktop backgrounds

Tue, 12 Dec 2017 00:00:00 +0000

One of my first Python projects was automating a daily desktop background. I think it’s a good project for beginners since it’s fun (new desktop delight!), it’s a good exercise in breaking down a vague idea (desktop delight?) into manageable chunks (one of the biggest wisdoms in programming!), and you can learn about a bunch of different useful things: interacting with the web and APIs, interacting with files, and scheduling jobs on your computer.

Tutorial assumptions
#

You have some version of Python installed on your computer.
You know how to run a Python script from the command line.
You have a text editor (e.g. Sublime) that you like.

My setup is: I’m using a pyenv virtual environment on OSX with Python 3.6.0 installed. I also would recommend installing ipython, since it provides such a nice shell.

pip install ipython

Break it down
#

One of the biggest wisdoms of programming is that it teaches you to break large tasks down into manageable parts. As I’ve gotten more experienced, I’ve noticed that this process happens faster: I can break things down more quickly, and I can hold more complex pieces in my head. In the beginning, though, it was more of a challenge - yet it was always very useful - to break down a project into a bunch of sub-tasks, and tackle each sub-task bird by bird. Don’t be afraid to make those sub-tasks very, very sub (e.g. “Figure out how to turn on Python”).

Here, our main objective is write a Python script that automatically changes our desktop background to something cool every morning. We can break this down into the following steps:

Find a place that has lots of cool pictures.
Grab pics.
Set pic as desktop background.
Have this happen on a regular schedule.

Here’s what you learn at each step:

Well, not much, but browsing pics is fun.
How to interact with APIs.
How to interact with your OS.
How to schedule stuff on your computer.

Step 0: Setup
#

For a small script like this, I would probably have my Favorite Text Editor open, in addition to a shell window with ipython’s shell running. On my machine, this might mean I’ll use the ipython shell to test stuff out, and build the script out in Sublime.

me@machine $ ipython
Python 3.6.1 (default, Jul 10 2017, 14:13:01)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

Step 1: Find a place that has cool pics
#

NASA’s Astronomy pic of the day, I choose you!

Other good options:

One thing that might inform where you grab your daily picture from is: does this place have an API? Or will I be scraping?

When grabbing stuff from the web, you usually have two choices: an API, or scraping. An API - or application programming interface - is generally preferable, in that the web developers have already prepared an endpoint for programmatic interactions: e.g. for Python scripts pinging it every day for a picture! Web scraping, where you ping the front end of the website, is not ideal - since you’re trawling through HTML and hoping that the HTML tags don’t change.

Of the above options, Reddit and NASA’s APOD have APIs, but I can’t seem to find one for The Atlantic. This might tip the balance in favor of Reddit or NASA - interacting with APIs is usually way easier.

Step 2: Grab pics
#

EASIER SAID THAN DONE, you may say. Never fear. Let’s break this down even further:

2.1. Find the API endpoint for grabbing pictures.
2.2. Use Python to ping said endpoint.
2.3. Save the returned image file somewhere.

NASA’s API documentation is here. The endpoint we’ll be hitting is https://api.nasa.gov/planetary/apod?api_key=YOURKEY where you’ll need to register with NASA and have them give you an API key. (This lets API developers track the number of pings you hit their API with - they want to ensure people don’t completely kill their servers by pinging a lot. Usually these “rate limits” are pretty high - thousands per day or whatever. Not something we have to worry about when writing a simple personal script - but definitely something that comes up for industrial strength API work.)

Anyway, here’s what gets returned by the NASA APOD API when you hit their endpoint with the example key:

{
 "date": "2017-12-12",
 "explanation": "What's up in the sky this winter? The featured graphic gives a few highlights for Earth's northern hemisphere. Viewed as a clock face centered at the bottom, early winter sky events fan out toward the left, while late winter events are projected toward the right. Objects relatively close to Earth are illustrated, in general, as nearer to the cartoon figure with the telescope at the bottom center -- although almost everything pictured can be seen without a telescope. Highlights of this winter's sky include the Geminids meteor shower peaking this week, the constellation of Orion becoming notable in the evening sky, and many planets being visible before sunrise in February. As true in every season, the International Space Station (ISS) can be sometimes be found drifting across your sky if you know just when and where to look.",
 "hdurl": "https://apod.nasa.gov/apod/image/1712/Winter1718Highlights_Universe2go_4994.jpg",
 "media_type": "image",
 "service_version": "v1",
 "title": "Highlights of the Winter Sky",
 "url": "https://apod.nasa.gov/apod/image/1712/Winter1718Highlights_Universe2go_960.jpg"
}

Ah, beloved JSON, lingua franca of the programmatic internet! Hello!

What’s nice about the JSON data format is that, since it’s structured in key: value pairs, you can access the value you want if you know the key. Here, we want to download a cool space picture - looks like we can find one at the url key.

To the Python! I like the requests library for its wonderful, readable documentation, and its wonderful, readable levels of abstraction. Interacting with APIs is molto painless with requests.

import requests

url = 'https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY'
r = requests.get(url).json()
print(r['url'])

This will print out our desired .jpg. To actually download that .jpg, we can use the Python standard library’s urllib.request module.

import urllib.request
from datetime import date

# Create a string variable that prettifies today's date
today = date.today().strftime("%Y%B%d")
todays_file = "{}_nasa_apod.jpg".format(today)

# Grab the image URL from the earlier API request
image_url = r['url']

# Download said file into current working directory
urllib.request.urlretrieve(image_url, todays_file)

I’m doing a few things above. Beyond just downloading the file (urllib.request.urlretrieve), I’m also formatting the filename such that it’ll have today’s date in there (for a handy reference on string formatting, I recommend http://strftime.org/).

3. Set pic as desktop background
#

This is the bit that depends on your OS - and, honestly, this is the bit that I still have no idea what it’s doing. I just found this code somewhere on the intertoobs and plugged it in:

from AppKit import NSWorkspace, NSScreen

ws = NSWorkspace.sharedWorkspace()
file_url = NSURL.fileURLWithPath_(todays_file)
for screen in NSScreen.screens():
 (result, error) = ws.setDesktopImageURL_forScreen_options_error_(file_url, screen, {}, None)

This uses Apple’s AppKit Python library to set the image on the currently active desktop. I’ve tried fiddling with the arguments in setDesktopImageURL_forScreen_options_error_ to have it, for example, scale and resize images appropriately, and apply the image to all desktops, not just the active one. But no luck - and, okay, not much will either. It works 90% of the time, and I’m happy with that!

4. Have this happen on a regular schedule
#

You can schedule things to run on your machine using cron, a job scheduler for Unix. One key thing about cron is that scheduled jobs won’t run if your computer is asleep - so schedule it to happen when your computer is always likely to be awake (e.g. during work hours). Otherwise, on a Mac, you can use launchd to have a background daemon run your script - even when your computer is sleeping. I haven’t bothered with this.

To have cron run your script, I found this tutorial super helpful. You basically need to edit your crontab by adding a line like the following:

30 7 * * * /path/to/cool/script/cool_script.py

The above says: run cool_script.py every morning at 7:30AM (your computer’s time zone). You can check to see if any cron jobs are currently scheduled on your machine by running crontab -l in Terminal.

Voila!
#

I don’t actually use NASA’s APOD for my own script. Instead, I have a list of image-heavy subreddits that I randomly select from. I’ve had this script running for ~3 years (!) now, and it has given me about 70% delight, 25% meh, and 5% embarrassment (be careful what you automatically download!). Here are some recent desktops that were really top-notch:

Kasai, by Joseph Biwald found via r/ImaginaryLandscapes

Aurora Borealis over Norway, by Max Rive, from NASA’s APOD, found via r/spaceporn

Emerald Lake, Yoho National Park, British Columbia, found via r/waterporn

Star Trek- USS Enterprise NCC-1701-D cutaway- Rusted Gear Art, found via r/StarShipPorn

Journey to the center of data science

Mon, 04 Dec 2017 00:00:00 +0000

I’ve been asked by more than two people about what my “journey to the center of data science” was and, well, one of my core values is automation. So here’s my response to those people. Next, I’ll write a bot that will just auto-reply this appropriately, heh.

Usual warning/caveat: your mileage may vary.

My path
#

This is in chronological order, starting from 2014 to ~now.

Glossary

* = how much each resource is still applicable to my day-to-day, where more = better
💵 = whether I paid any money for it (as opposed to, e.g., having it reimbursed)
🐝 = taken as part of Georgia Tech’s OMSCS program (this costs 💵 for degree credit, but is freely available on Udacity if you don’t want to take it for credit)

I’ve also put the approximate amount of time spent ingesting (not digesting!) the resource.

Background: BS in economics and mathematics, MPhil in economics + several years working as a research manager and economist in university research labs and non-profit sector.

💵 Interactive Data Visualization for the Web by Scott Murray * (~1 month)
Udacity - Data Visualization and D3.js * (~3-4 weeks)
Udacity - Programming Foundations with Python *** (~2 weeks + several months working on first project)
Udacity - How to Use Git and GitHub *** (on and off, maybe 2 weeks)
Coursera - Machine Learning Foundations: A Case Study Approach ** (~4 weeks)
RailsBridge * (1 weekend)
ComputeFest 2016 ** (2 days)
💵 Harvard Extension School - CS171: Visualization * (~240 hours AKA 15 hours/week for a semester)
Boston Python User Group - Talking to other computers with Python *** (1 day)
OpenVisConf 2016 *** (3 days)
Udacity - Intro to Relational Databases *** (~2 weeks)

Got first data science job

Udacity - Intro to Machine Learning ** (~3 weeks, unfinished!)
💵 Harvard Extension School - CS109A: Intro to Data Science *** (~320 hours AKA full semester, 20 hours/week)
💵 Harvard Extension School - CS109B: Advanced topics in data science *** (~320 hours AKA full semester, 20 hours/week)
DataCamp - Deep Learning in Python *** (~4 whirlwind hours, applied to CS109B final project)
Boston Python User Group - Data testing tutorial *** (1 day)
💵 Doing Data Science by Rachel Schutt, Cathy O’Neil ** (~10 hours)
Boston Python User Group - Network analysis made simple *** (1 day)
ClojureBridge ** (1 day + a loooong time working on the crypto puzzle)
Bayesian Methods for Hackers: Probabilistic Programming and Bayesian Inference by Cameron Davidson-Pilon *** (~10 hours)
💵 Mastering Regular Expressions by Jeffrey E.F. Friedl *** (~10 hours)
💵 Learning the bash Shell by Cameron Newham, Bill Rosenblatt *** (~10 hours)

Got second data science job

Clojure/conj 2017, * (~2 days)
EdX - CS50: Introduction to Computer Science *** (~60 hours, see the journey here!)
TomTomFest - Applied Machine Learning Conference 2018, ** (1 day)

Admitted to Georgia Tech’s OMSCS

Lynda - Advanced SQL for Data Scientists *** (~2 hours)
🐝 Georgia Tech/Udacity - Knowledge-Based AI * (120 hours, AKA full semester at ~10 hours/week)
Lynda - Polyglot Web Development * (~2 hours)
PyData DC 2018 - *** (3 days)
Destroy All Software - A compiler from scratch - *** (2 days)
Lynda - Ruby Essential Training *** (~10 hours)
Lynda - Git Intermediate Techniques ** (~3 hours)
Lynda - DevOps for Data Scientists * (~1 hour)
💵Destroy All Software - A test runner from scratch *** (2 days)
Lynda - Kafka Essential Training * (2 days)
Lynda - Parallel and Concurrent Programming with Python I *** (2 days)
Lynda - Learning Flask * (1 day)
Lynda - Ruby on Rails 5 Essential Training *** (only did half of it, many hours)
Thoughtbot - Onramp to Vim *** (~1 week)

You can see what I’m currently doing on LinkedIn.

General resources
#

Those are things I actually went through, top to bottom. In addition, here are some reference resources I use a lot (and find very helpful):

YouTube - Data School: Great for getting a handle on pandas’s syntax.
Chris Albon - Data wrangling tutorials: More pandas.
YouTube - MIT 6.034 Artificial Intelligence, Fall 2010: Clear, precise and deep explanations of SVMs, random forests, neural nets and more. I need to go through this top to bottom.
An Introduction to Statistical Learning by James et al.: The main textbook, with examples in R (unfortunately…).
Elements of Statistical Learning by Hastie et al.: More main textbook.
Artificial Intelligence by Winston: Winston is great.
Khan Academy - Linear algebra: Refreshing the fundamentals!
Khan Academy - Statistics and probability: More excellent fundamentals!
YouTube - mathematicalmonk: Very good explanations of more advanced math and probability topics - his video on Markov Chain Monte Carlo (MCMC) sampling was very helpful.
YouTube - 3Brown1Blue: Enthusiastic math.
YouTube - Vihart: More enthusiastic math.
regexr: A beautiful and well-designed site for debugging regular expressions.
Regex Golf: Think long and hard and don’t touch that keyboard.
Anki: Spaced repetition flashcards app. This changed a lot for me. I’ll post a longer blog post about it sometime, but, for now, read Michael Nielsen’s post, Augmenting long term memory.
YouTube/ThePrimeagen: Helpful Vim tutorials.
Hackerrank: Coding puzzles; helpfully includes SQL (which is rare-ish). The one that slows down my browser and overheats my CPU (why?!).
LeetCode: More coding puzzles. I prefer the UI on this one. But it’s more algorithms + data structures-oriented. The broey one.
Exercism: Yet more coding puzzles. Also a nice way to learn a new language. The friendly one.

And, of course, Google, StackOverflow, and YouTube.

It never ends
#

I consider this a start, and I still have a lot of stuff I want to learn. It’s a big topic, and the TOLEARN list is only growing…!

Hello, Pelican!

Thu, 30 Nov 2017 00:00:00 +0000

Hello, world!

For the past few months, I’d been looking to start a mostly-tech blog.

I already have my dusty ol’ movie review blog, and I have my ragingly active book review sorta-blog on Goodreads but I also wanted a place to document and share what I’m learning along my Journey to the Center of the Machine. Given how much I’ve benefitted from other tech blogs - Julia Evans’s or daiyi’s or Peter Norvig or Ned Batchelder’s - it also seemed like giving back to the community was a good thing and something I should do.

Desired specs
#

In looking for a blog framework, I knew I wanted to:

Minimize front end work. I spent a lot of 2014-2015 learning front end stuff: HTML, CSS, Javascript, Bootstrap, and - most especially - d3.js. As much as I love a good data visualization, I did not want to fuss over all that again.
Automate as much as possible. The above, plus could I haz scripts?
Markdown support. Plz, so lazy.
Python. I once attended a one-day workshop where we built little blogs using GitHub Pages and Jekyll. Unfortunately, Jekyll’s in Ruby - but surely there was something in Python?

Enter Pelican.

TIL: Pelican
#

Pelican is a static site generator, written in Python.

– official Pelican docs

Static site generators take your content, pour it into your templates and output the result as static pre-generated HTML, CSS, JS & image files. You can then just upload the resulting folder of output to your server and you’re done.

– How I built this website, using Pelican: Part 1 - Setup, Duncan Lock

To get started, I followed Pelican’s quickstart and, after fussing over where to put it (my directory structure is a mess, please send help) and fussing over keeping my Python environment clean (please send more help), I just said “whatevs” and pip installed it onto my base Python like a maniac:

pip install pelican markdown
mkdir ~/some/file/path/pelican_hello_world
cd ~/some/file/path/pelican_hello_world
pelican-quickstart

pelican-quickstart runs you through a bunch of questions on how you’d like to set up your project: for example, the title, author, and so on. (This is for convenience; you can change your answers later by editing your pelicanconf.py file.) Once you’ve finished, you’ll have some pre-populated files and a directory structure like this:

pelican_hello_world/
 -- content/
 -- output/
 -- develop_server.sh
 -- fabfile.py
 -- Makefile
 -- pelicanconf.py
 -- publishconf.py

The money bits are in content/ (where you’ll draft your blog posts) and output/ (where Pelican will dump the static site, ready to be published to the internet). You can change where Pelican dumps your site - I changed the $OUTPUT_DIR variable in the Makefile (not pelicanconf.py, per this [issue](https://github.com/getpelican/pelican/issues/11 71)) to where the rest of my website is.

You can write your blog posts in Markdown (or .rst, .adoc, .html) and stick them in content/. You can then run make html and make serve to launch a local server and see the changes at localhost:8000 in your browser.

There are some “meta” fields at the top of a blog post in content/, which Pelican’s (magical?) (CSS?) something parses to auto-populate the resulting .html in output/.

At this point, I had a few things I wanted to figure out:

What does Pelican want from me? (`metadata`)
#

According to the docs:

Title: My super title
Date: 2010-12-03 10:20
Modified: 2010-12-05 19:30
Category: Python
Tags: pelican, publishing
Slug: my-super-post
Authors: Alexis Metaireau, Conan Doyle
Summary: Short version for index and feeds

This is the content of my super blog post.

Okay.

How do I make it look pretty? (`themes`)
#

There is a frankenrepo over on GitHub of 120+ themes. Following their instructions, you can git clone the entire repo to your home directory, and then - in the pelicanconf.py config file, add:

THEME = /path/to/pelican/repo/clone/some-theme-name

I didn’t love this. First, there’s a LOT of themes. (You can browse an endless scroll of them here). Second, it seemed like grabbing a gnarly giant repo - with tons of devilish git submodules (ack) - was not a super efficient way to grab one or two themes. Third, as a Pelican noob, I wanted to make something look like my existing website - and customizing an existing theme didn’t seem much easier than just customizing base Pelican.

The docs discuss creating a theme:

Basically, you follow a directory structure, where you have a static/ dir (of css/ and images/ and whatever other assets you want to just port across to output/), and then a bunch of .html templates for each of the different pages Pelican’s output/ creates.
Ah ha! This could be where, if one wanted, one could dump their Bootstrap navbar, stylish fonts, and soothing CSS.

I spent a lot of time on this, way more than I wanted or intended. One of my specs is “minimize front end”! But, argh, it had to be done. What I ended up doing was:

Using the pelican-bootstrap3 theme as my base.
Rewriting a bunch of that theme’s style.css to look like my original website.
Doing a lot of (fun) fussing over widgets and doo-dads.

Syntactic Scriptastic sugar
#

This blog by Nafiul Islam had a pretty awesome/handy example script - make_entry.py - which you can use to automate your posts. I fiddled with it, added argparse and a few more options, and modified it to output .md formatted blog templates. I’ve been using that now to auto-create drafts.

Chris Albon also has a cool Jupyter -> .md -> Pelican workflow script, though I haven’t played around with it yet. But it looks cool.

How do I integrate it with my existing website? (`subdomains`)
#

Subdomains are so cool. After I created the blog.angelaambroz.com subdomain on my webhost, I thought I would just have to dump the output contents into it, remember to link to it from the rest of the site, and - voila! Now I have a tech blog! Wooo.

Unfortunately, tying it all together took a bit more pain and fiddling. First, there was the problem of relative paths. I had used the ARTICLE_SAVE_AS and ARTICLE_URL options in my config file to auto-create year and month archives of posts (and also clean up the blog/ directory structure). Unfortunately, this messed up a bunch of things: I now had index.html files every which way, and I was referencing up and down directory paths. It was a giant, giant pain.

Also, the CSS didn’t line up at first; though I think this had more to do with running a local server. But there are still some annoying disparities (e.g. the navbar on my blog has slightly dimmer font-color than the navbar on the rest of the site - argh) that, after N hours, I am too pooped to resolve. I also don’t love the idea of maintaining two distinct CSS files.

Reading list
#

Official docs
How I built this website, using Pelican: Part 1 - Setup (blog)
Making a static blog with Pelican (blog)

Learning the bash Shell (2005) - ⭐⭐⭐⭐

Fri, 24 Nov 2017 00:00:00 +0000

Metadata
#

Author(s): Cameron Newham
Number of pages: 333
Year published: 2005
Year read: 2017

Review
#

Bash agility - fluency on the Unix/Linux terminal - is a super foundational meta-programming skill that I feel like gets short shrift as we all rush towards machine learning and cryptocurrency. Yo, but the stronger your bash fu, the easier your management of remote servers and such. Like, I did this the other day and basically floated away on a rainbow chariot pulled by mighty unicorns:
pip freeze > diff - requirements.txt
YES, I AM POWERFUL.

Just like Friedl’s book on regular expressions, another clarifying book on an important meta topic, this bash book was SUPER helpful and I wish I had had it in 2014. I literally remember sitting at my desk in Dar es Salaam, staring at OSX Terminal and watching some Coursera course on some tech topic, and marveling at the instructors’ bash incantations. Wtf was he doing?

“Is there some structured way to learn about Terminal!?” I thought. I didn’t even know it was a shell language called bash! I didn’t know shell != bash! Lots of stuff. I WIIISH I had had this then.

Anyway, yes, you can probably pick up these same bash skills by just osmosis over long periods of time. I did - there was a lot in here I had already learned (or had sorta half-known and used anyway), and I think experienced programmers will consider it real (bashy) basic. But if I had a young lady beginning her career transition journey into tech, I would hand her this and the regex book, and the keys to Udacity, and GODSPEED YOUNG MADAM.

And now, for much meta inspiration on people’s hardware, editor, and shell choices, here’s https://usesthis.com/

📚 Goodreads

Bayesian Methods for Hackers- Probabilistic Programming and Bayesian Inference (Addison-Wesley Data & Analytics) (2015) - ⭐⭐⭐⭐

Fri, 18 Aug 2017 00:00:00 +0000

Metadata
#

Author(s): Cameron Davidson-Pilon
Number of pages: 256
Year published: 2015
Year read: 2017

Review
#

A fun and informative book on applied Bayesian modeling in Python.

Assumes knowledge of Python and, honestly, I wouldn’t recommend this - alone - as an intro to Bayesian stuff. But if you combine this with Allen Downey’s Think Bayes or Khan Academy’s Bayes Theorem video or a course (!), you would probably be able to get off the ground with a couple initial models quite quickly.

This book is pitched towards people a bit beyond noob but not total hackers yet on both the PyMC spectrum (Python’s most popular (?) Bayesian modeling library… after PySTAN?) and on the Bayesian stats spectrum. For example, the author intentionally hand-waves away Monte Carlo Markov Chain (MCMC) sampling as being much too mathy for this book’s purposes, which is… meh, fine. There’s a bunch of intuition on what the MCMC sampling is doing and why we use it. (Though I would have introduced conjugate priors earlier, juxtaposing “easy” Bayesian models that can be solved analytically versus hard/annoying models that need sampling.) If you want a bit more mathiness, I thought mathematicalmonk’s video was helpful.

There’s a good section on multi-armed bandit models (i.e. A/B testing when your sample sizes are super small), which was informative and kinda mind-blowing.

The book is open source and you can read it for free (and make contributions) on their GitHub repo. This is very nice and forward-thinking/Star Trek TNG of them, thank God, but it also means that the later chapters are a bit sloppy. I cloned the repo a couple weeks ago and was gifted with a previously-unseen Chapter 7, which turned out to be a bunch of TODOs. Oh well. There’s also some typos, maybe a bit too much Personality (heh), and the nagging feeling that, gosh, I guess I could edit their book for some open source software street cred - but then I thought, WHAT ME BE A GLORIFIED SECRETARY AGAIN NO THANK YOU, and lo, thus dideth the number of women contributors to open source code remaineth at the 5%, the Devil’s percent.

📚 Goodreads

Mastering Regular Expressions (2006) - ⭐⭐⭐⭐⭐

Thu, 03 Aug 2017 00:00:00 +0000

Metadata
#

Author(s): Jeffrey E.F. Friedl
Number of pages: 542
Year published: 2006
Year read: 2017

Review
#

Mwah, mwah! Thank you, Jeff Friedl, for finally - FINALLY - demystifying this most crucial of meta-programming skills: regular expressions. I had tried online tutorials, videos, even a retro bash game, this and that. Nothing ever helped. This did. FINALLY. FINALLY!

So I didn’t “read” the whole book, but whatever, the second half is reference material for languages I never use nor plan to use (Perl, PHP, etc.). Honestly, the first 2-3 chapters were sufficient; once he got into the guts of different regex languages, I felt like I was teleported back to 1983 Bay Area, sitting next to Steve Wozniak as he pecked at his clackety-clack keyboard and wrote the original BSD man pages. Awwww myyyannn, I just did man grep to double-check my facts and DAMMIT the last update to the grep man page was 2010!!!! Way to kill my joke, Unix.

Henyyyway. Highly recommended, though YMMV - people learn in diff ways. I had been daunted by regexes since the 2009 Stata days, and I just WISH I had had this then.

For my future reference (and yours), a muscle must be exercised, and here are some regex games:
- https://alf.nu/RegexGolf
- https://regexcrossword.com/
- https://www.hackerrank.com/dashboard has a bunch
- https://github.com/bchartoff/regexcalibur (the bash game)

📚 Goodreads

Doing Data Science- Straight Talk from the Frontline (2013) - ⭐⭐⭐

Thu, 11 May 2017 00:00:00 +0000

Metadata
#

Author(s): Cathy O’Neil, and Rachel Schutt
Number of pages: 405
Year published: 2013
Year read: 2017

Review
#

Thus endeth my lunchtime reading book. I intermittently read this, over the course of many months, usually over a sandwich at lunch. For this style of reading, it holds up well: the chapters are discreet packets of data science chat. That said, I agree with other critiques of this book: if you’re an aspiring data scientist, this book is NOT sufficient to get you off the ground. It’s not a good beginner’s book. It’s maybe a good “pop data science” book, a pre-beginner’s book. It’s very light on the technical stuff, and, if anything, it’s more like an anthropological survey of the state of the field.

Each chapter covers a technique or common challenge or strategy, describes the general jist of what’s going on, and then points you in the direction of papers, other books, or tutorials online. Early chapters have some “exercises”, though they’re more like general pointers of “oh, you could try this, I guess?” Later chapters don’t even bother.

For an O’Reilly book, I was disappointed that the GitHub repo didn’t have, for example, the code examples mentioned in the book, or the exercises and toy datasets. (What? Are we supposed to manually copy down several pages of R code?!) Or even just a README.md with a bibliography (given how many shortened Google links are used as citations)? This makes it a starkly UNFRIENDLY book, which is weird since O’Reilly books (well, the good ones) can be very, very rich resources. This, instead, felt thin - and the repo is basically pointless.

I will say that I enjoyed the banter-y tone of the book, and some of the discussions of techniques (e.g. there was a great, intuitive explanation of Principal Component Analysis) and “real world” issues (e.g. how Kaggle competitions are basically data science in a vacuum; what it’s like to be a lady data scientist) were quite good. But, overall, yeah, this isn’t really a “good enough” data science book.

📚 Goodreads

Weapons of Math Destruction- How Big Data Increases Inequality and Threatens Democracy (2016) - ⭐⭐⭐⭐⭐

Thu, 26 Jan 2017 00:00:00 +0000

Metadata
#

Author(s): Cathy O’Neil
Number of pages: 259
Year published: 2016
Year read: 2017

Review
#

Cathy O’Neil is one of my heroes; I love listening to her on Slate Money, and her less pop/more technical book, Doing Data Science, occupies a prominent place on my desk at work - and a special place in my heart. Going beyond the usual data science manuals of what a random forest is, or how to account for large, sparse data, O’Neil and her coauthor - in that book - make an almost anthropological survey of the data science “field”/profession/whatever. It’s enlightening, and it’s very real; less a textbook, more a professional guide.

So if Doing Data Science is the “how to do your job” manual, and Slate Money is the occasional soapbox, Weapons of Math Destruction is the manifesto. With clarity, conviction and intelligence, Cathy O’Neil scalpels away at the utopian, misty-eyed tech worship that has contributed to the data science/big data bubble. She identifies the ways in which algorithms encode and perpetuate existing prejudices, and are hidden from plain sight by the obscuring (both intentional and not) use of Fancy Math.

She notes how anti-human “market efficiency” is, and how it’s often been short-hand for unjust and unequal systems, and big data is just scaling those dumb systems way, way up. For example, the use of social network data to “enrich” lending platforms; or recidivism models that ask about where a former convict grew up, whether his/her friends are in jail, how many encounters with the police he/she has had. These algorithms - because they chase correlations to make predictions (rather than identifying causal factors) - create big-data-driven poverty traps. They are also beyond the purview of the government (and, given the new administration, will probably be for quite some time): just like your FICO score is ONLY about how often you pay your credit card bills, and NOT your demographic group, so too should these algorithms be scrutinized and regulated.

Increasingly, O’Neil notes, the “human touch” comes at a price; in upper income groups, you get more real, live service - hearing about a job through connections, informational interviews with alumni, blah - while lower income groups are serviced by increasingly mechanized, machine learning code. The hiring and HR practices of CVS, McDonald’s, and other low wage places were especially striking (and severe): mindless, well-meaning algorithms deployed to effectively punish workers whose BMIs (itself a stupid measure of health) are too high, or who seem “anti-social” according to a creepy questionnaire, or whatever.

I think my favorite chapter - and, of course, the most relevant one - was about the toxic brew that can be politics and data science. It was especially powerful to think through the way that, as opposed to the past, where we had “agreed upon facts” given by carpet bombing-style hegemonic culture (e.g. four news channels that everyone watched), now we have hyper-tailored DARE I SAY “alternative facts” where, by the power of marketers (who prey on confirmation bias), your neighbor’s political beliefs are the result of a very specific and private relationship between them, their web browser cookies, and some political marketer. This means you may have NO IDEA why your swingy neighbor in swingy state X thinks Candidate Y is running an underground child sex ring literally under the ground of a DC pizza shop. O’Neil (and this book was written Before The Madness) notes that these sorts of micro-targeted ads that follow you around the web are very smartly tailored; the marketers know who would be repulsed by such a stupid conspiracy theory, and who would be intrigued. And that’s just terrifying.

Anywayyyyy. From a data perspective, my (cold dead) economist heart reminds me that OF COURSE we’re going to scale up injustice when we mindlessly maximize profit, instead of humans. We need to go back to our roots: to the philosophy and debates of Jeremy Bentham, and welfare economics, and what “utility” really means. Efficiency is not the end all, be all, despite what our current economic system seems to diktat. O’Neil also makes a good point about the repeated erroneous conflating of correlation and causation; something even the fanciest of linear algebra stochastic process meddlers fall prey to. That’s something that I see a LOT in data science: the field is built on the premise of perfect prediction - and prediction doesn’t need causation, it just needs correlation.

OK, I will step off my soapbox. Highly recommended.

edited to add: OMG and I just read her delightful 2017 resolutions post, and had much LOLs. Cathy O’Neil, you da best.

📚 Goodreads

Think Stats (2011) -

Wed, 04 May 2016 00:00:00 +0000

Metadata
#

Author(s): Allen B. Downey
Number of pages: 136
Year published: 2011
Year read: 2016

Review
#

It’s a textbook. A good one. I didn’t finish it. Wiping the slate clean! I saw Allen Downey give a talk on Bayesian stats, and it was fun and informative. I think he’s great.

One annoyance. I think I’m maybe the perfect audience for this book: someone who took stats long ago, has worked with data ever since in some capacity, but has moved further and further away from the first principles/fundamentals. Someone who speaks Python and wants to port all of her Stata skillz onto pandas (the Python library, not the Chinese bear - okay, also the Chinese bear*). So, in a way, this book was perfect for that. MY ONE COMPLAINT is that Allen provides many helper functions and .py files pre-written for you to play around with. I would have preferred less hand-holding, and more: Now build a function that will give you the cumulative distribution function!

But then: who am I to complain. I didn’t finish it (for now). And it can be hard, sometimes, to find the perfect puzzle piece for your current skillset/desires/time constraint on the Great Learning Journey that is life.

* images of pandas feverishly computing z-scores, while I cackle above them, “Work harder!!! Why are you so slow!!!” images of furry paws clutching pens and notebooks, scribbling

📚 Goodreads

How to Think Like a Computer Scientist- Learning With Python (2002) - ⭐⭐⭐⭐

Mon, 20 Apr 2015 00:00:00 +0000

Metadata
#

Author(s): Allen B. Downey, and Jeffrey Elkner, Chris Meyers
Number of pages: 288
Year published: 2002
Year read: 2015

Review
#

First few chapters, I was like, “Duh, self-evident."
Next few chapters, I was like, “Ooh."
Last few chapters, I was like, “…What?"

Yeah, so Python is great. Flexible, durable, like Wolverine’s adamantium claws. Can it do anything? Perhaps yes. Maybe we can replace our eco-system with Python and just be done with it.

Highly recommended, especially as a companion piece to Udacity’s (wonderful) Intro to Python Programming course, which was bite-sized, basic, and clarified a lot of that Class stuff. (Which was sometimes mysterious in the book.)

I second the request for more (more more) exercises, more code examples, more hands-on stuff. It gets a bit too heady otherwise.

📚 Goodreads

The Visual Display of Quantitative Information, 2nd Ed. (2001) - ⭐⭐

Sun, 22 Feb 2015 00:00:00 +0000

Metadata
#

Author(s): Edward R. Tufte
Number of pages: 197
Year published: 2001
Year read: 2015

Review
#

It feels sacrilege to say this, but not only did I find this Classic Text of Data Visualization a complete bore, but I also disagreed with a lot of Tufte’s principles! Uh oh. Is no one going to let me near a data visualization again? Hope not. But seriously. I’m surprised this remains the go-to graphic design book for social scientists, data scientist people, etc. It feels completely outdated, not only because it’s more 1970s than a pair of huge Dan Aykroyd sideburns, but also because it’s all about static, dead-on-the-page graphics - and we live in a world where we can make our infographics animated and interactive!

Yes, I agree that the Microsoft Office Hegemony is a bad thing - especially the way PowerPoints and Excel homogenize and bleed the passion from our work - but I just could’t agree with some of Tufte’s other principles.

📚 Goodreads

Spam- A Shadow History of the Internet (Infrastructures) (2013) - ⭐⭐⭐⭐⭐

Mon, 27 Oct 2014 00:00:00 +0000

Metadata
#

Author(s): Finn Brunton
Number of pages: 270
Year published: 2013
Year read: 2014

Review
#

Aaah. Superbly brilliant. Maybe the best book I’ve read this year? SO GOOD.

This was basically like a five-hour Idea Channel episode, what with the charismatic host (Finn Brunton’s writing - so clever! so fun!), the combination of technology/futurism/literary criticism/culture theory/sociology, the awesomely interesting asides and mind-blowing anecdotes. This was one of those books that my brain absorbed like a giant sea sponge dumped into crack-filled water; I was just so thrilled, so stimulated, SO INTO IT, all the time.

Briefly: it covers the parallel development of the spam and non-spam Internets. Like yin and yang, you kinda realize that one can’t exist without the other: what with spam being the gummy, gray gooey, reptilian brain, capitalist slush that inevitably fills the tubes (ALL OUR TUBES, not just the Internet), pushing the edges of our global network’s technical capacities and acceptable social behaviors. It runs from the earliest proto-spams of ARPANET to the really creepy blurred-lines spam of clickbaity nonsense like, well, any post-AOL-acquisition Huffington Post article(hoo boy, did they jump the shark there, eh).

The book was amazing because it charted, basically, my experience of the Internet, structuring and contextualizing that experience. Yo, I been online since 1995 (twenty long years, people), and I distinctly remember each spam phase: the proto-spams of AOL, the weird litspam of the mid-2000s, and now the Orwellian awfulness of Upworthy et al. and the linkbaitification of journalism. So, it was like, I FEEL THIS. And, I HAD NO IDEA.

Another great thing about this book is that it’s a bit of an action-packed cyberpunk rollercoaster, better than the best Gibson because, well, it’s real.

Highly highly recommended. Eleventy stars.

📚 Goodreads

Learn to Code HTML and CSS- Develop and Style Websites (2014) - ⭐⭐⭐⭐⭐

Thu, 16 Oct 2014 00:00:00 +0000

Metadata
#

Author(s): Shay Howe
Number of pages: 306
Year published: 2014
Year read: 2014

Review
#

This is a review for the online tutorials on which this book is based. It’s unclear to me how different the book is from the online tutorials, but the latter are GREAT. Extremely thorough, clear, and helpful - with tons of great resources elsewhere. I’ve bookmarked this for future use, since it really is incredibly helpful and I’ll probably want to refer back to some of the lessons as I move forward. It’s really exciting to see the sample website come alive - especially once we add our first CSS stuff! happy sigh

Edited to add: Props also to the Bocoup people, from whom I found my way - via recommendations within recommendations - to this resource.

📚 Goodreads

Getting Started with D3- Creating Data-Driven Documents (2012) - ⭐⭐

Wed, 17 Sep 2014 00:00:00 +0000

Metadata
#

Author(s): Mike Dewar
Number of pages: 66
Year published: 2012
Year read: 2014

Review
#

I’m not sure who this book is pitched to - maybe it resonates with people who already have a pretty good knowledge of JavaScript. For complete newbies, or people at level 1.1 noob, it feels really scattered and mostly baffling. I found Interactive Data Visualization for the Web a lot friendlier - and richer, in terms of both explanations and resources for further study.

One (pretty big) plus point, to Mike Dewar’s credit, is his hands-on example using the New York City subway. I sometimes complain about the examples used in code tutorials (they skew heavily towards the interests of middle-aged white dudes - e.g. program a poker game! replicate Pong!), and I appreciated something that was: (1) civics-ish, (2) urban design-y, and (3) with a lot of depth (there are so many questions one could ask about the New York City subway, amirite?).

📚 Goodreads

Interactive Data Visualization for the Web- An Introduction to Designing with D3 (2013) - ⭐⭐⭐⭐⭐

Sun, 07 Sep 2014 00:00:00 +0000

Metadata
#

Author(s): Scott Murray
Number of pages: 270
Year published: 2013
Year read: 2014

Review
#

Awesome. Basically an intro, from the ground up, to d3.js - the powerful JavaScript library built by Mike Bostock for data visualization. I’ve been meaning to get good at d3 for a looong time. I now feel like it’s somewhat possible. Great overview of the fundamentals, lots of calming encouragement, and - thanks to its links all over the place - an amazing resource.

FYI this book is an expansion and enrichment of Scott Murray’s freely available d3 tutorials on his website. I’d say the tutorials form the skeleton of about 50% of the book. Still worth it, though, as (1) I need this shit offline, and (2) the final chapters are pretty mind-blowing (GeoJSON, where have you been all my life?), and (3) the walk-throughs of the existing stuff are more thorough.

On on on.

📚 Goodreads

Posts on Angela Ambroz

AI Engineering ⭐⭐⭐

Metadata #

Review #

Quick overview #

OK, so the good stuff: What did I learn? #

The not-good stuff #

One more thing #

Bedazzling the terminal

Eternal question: Where is my peaceful bottega? #

Moving from gnome-terminal to kitty #

Rapprochement with tmux #

Colors #

Claude Code talking to itself #

Year in review - 2025 - Books

2025: The year I read 82+ books #

💹 Descriptive stats #

📖 A meta aside #

No more data, now only vibes #

I read more fiction this year #

I read some remarkable historical artifacts #

Planning for 2026 #

What genres? #

How to find books? #

Migrating from Wallabag to Readeck

The problem space: Reading it later in comfort and style #

My personal pain points on Wallabag #

Choosing Readeck #

Not-a-tutorial: Getting Readeck set up #

The brains: Setting up the Readeck server (20 mins) #

Client #1: My Android phone (1 hr) #

Sending articles #

Reading articles #

Client #2: My Kobo (20 mins) #

Migrating my Wallabag articles to Readeck (1 hour) #

Delightful surprises: Showcasing Readeck at its best #

Surprise #1: Readeck > Save this image #

Surprise #2: Readeck for work research #

Uncanny Valley - ⭐⭐

Metadata #

Review #

LLM book recommender system

🎥 Discovery on Plex: really good #

🎧 Discovery on Audiobookshelf: also quite good #

📚 Discovery on Calibre-web-automated: not so good #

Using an LLM to do discovery on my own Calibre library #

Results #

Example run #

Averting the Digital Dark Age - How Archivists, Librarians, and Technologists Built the Web a Memory - ⭐⭐⭐⭐

Metadata #

Review #

The AI Con - ⭐⭐⭐⭐⭐

Metadata #

Review #

Hosting my own notebook server

Enter marimo! #

Homelab overview

Some “economics of open source” thoughts #

Imprisoning your users - an experience with Google Takeout

Getting ready for local AI

Fun OpenWebUI features #

Reflecting on a year of attempted sustainability

What I was trying to do #

Climate nihilism #

Guiding principles #

How to apply the principles #

🏭 Taking stock #

Okay, what I actually did (– and whether I think it was a hit 🎉 or miss 😖) #

📚 Learning: #

🏠 ✈ House + movement: #

🥦 👚 Consumption: #

💰 Wealth: #

✊ Activism: #

Conclusions from a year of attempted sustainability #

Climate nirvana is painful #

That said - climate nirvana was oddly joyous? #

Future plans #

Fixing subtitles for obscure movies

Solution: Thank you, ffsubsync! #

Doing it on the NAS? #

Metadata
#

Review
#

Quick overview
#

OK, so the good stuff: What did I learn?
#

The not-good stuff
#

One more thing
#

Eternal question: Where is my peaceful bottega?
#

Moving from `gnome-terminal` to `kitty`
#

Rapprochement with `tmux`
#

Colors
#

Claude Code talking to itself
#

2025: The year I read 82+ books
#

💹 Descriptive stats
#

📖 A meta aside
#

No more data, now only vibes
#

I read more fiction this year
#

I read some remarkable historical artifacts
#

Planning for 2026
#

What genres?
#

How to find books?
#

The problem space: Reading it later in comfort and style
#

My personal pain points on Wallabag
#

Choosing Readeck
#

Not-a-tutorial: Getting Readeck set up
#

The brains: Setting up the Readeck server (20 mins)
#

Client #1: My Android phone (1 hr)
#

Sending articles
#

Reading articles
#

Client #2: My Kobo (20 mins)
#

Migrating my Wallabag articles to Readeck (1 hour)
#

Delightful surprises: Showcasing Readeck at its best
#

Surprise #1: Readeck > Save this image
#

Surprise #2: Readeck for work research
#

Metadata
#

Review
#

🎥 Discovery on Plex: really good
#

🎧 Discovery on Audiobookshelf: also quite good
#

📚 Discovery on Calibre-web-automated: not so good
#

Using an LLM to do discovery on my own Calibre library
#

Results
#

Example run
#

Metadata
#

Review
#

Metadata
#

Review
#

Enter marimo!
#

Some “economics of open source” thoughts
#

Fun OpenWebUI features
#

What I was trying to do
#

Climate nihilism
#

Guiding principles
#

How to apply the principles
#

🏭 Taking stock
#

Okay, what I actually did (– and whether I think it was a hit 🎉 or miss 😖)
#

📚 Learning:
#

🏠 ✈ House + movement:
#

🥦 👚 Consumption:
#

💰 Wealth:
#

✊ Activism:
#

Conclusions from a year of attempted sustainability
#

Climate nirvana is painful
#

That said - climate nirvana was oddly joyous?
#

Future plans
#

Solution: Thank you, `ffsubsync`!
#

Doing it on the NAS?
#

What be Obsidian?
#

How I use Obsidian
#

The once and future king
#

Reading things later
#

The end of Pocket
#

Setting up Wallabag on my Synology NAS
#

…and my Android and browser
#

The next big piece: the Kobo
#

NAS joy
#

RSSing my life
#

A brief tutorial
#

Mobile
#

Two months in…
#

Overall
#

2017 - 2024: Tech debt 😱
#