TimothyCrosley.com

Introducing QuickPYTHON: The fun of QBasic meets the power of Python

2020-11-26T10:00:00-08:00


project:	9/52
Title/Link:	QuickPYTHON
Pitch:	All the fun of QuickBASIC on old PCs but now using Python.
Read if:	You learned how to code on an old DOS computer using QuickBasic or TurboPascel, or if you are interested in unique coding environments.
Skip if:	You've already learned how to code, don't have any kids you want to teach, and aren't interested in unique coding environments.
Prior Work:	QuickBasic Turbo Pascal 10 years later, QBasic is still the best
Hackernews Article:	Show HN: QuickPYTHON

It's easier than ever to waste time, and harder than ever to invest it

Let's say you're 7 years old with access to a computer and have a couple of hours to spend. What do you do? The least friction option is probably to open Netflix or a similar streaming app and pick something suggested by an algorithm that somewhat matches your tastes. Or, if you want to use it as a computer instead of a TV, you could play one of the countless "free" games designed to have you hooked or browse social media. The action that seems unlikely is that you would decide to spend those hours learning to code without prodding. And even if you did, chances are it could easily take you those full 2 hours to find a good initial starting point.

Those of you who have read my whoami introduction likely know that I started coding at a very early age. Given a few hours and a computer, I did precisely that - I decided on my own that I wanted to code. I didn't have access to the internet, but QuickBasic, with its built-in help, intuitive interaction, one button run, and simple mechanics made coding fun. It was easy to pick up and hard to put down. And, talking to many peers, I've learned I wasn't alone. It feels like Basic inspired a whole generation of programmers who eventually took up a career not because of how much money it made but because of how much fun it was.

Learning to code these days feels different. In many ways, the environment in which I learned was defined by its limitations. I had two games on floppy disks. No internet and no money or means to buy more. If I wanted anything else, I had to make it myself. And the only way presented to do that was QuickBasic. Of course, the only way to learn it was to try things out and read the manual. One of the biggest motivators to code was that the restricted content to consume created a huge contrast to the unlimited things I was told I could make. Today, the content itself feels limitless. In the world I grew up in, given complete freedom, I naturally gravitated toward programming. I have to wonder if I had the same freedom as a kid these days if I would have just become addicted to Netflix and video games.

My daughter isn't yet two and is unlikely to code anytime soon, but I've been thinking about whether it was even possible these days to introduce programming similarly in the back of my mind. Others have had the same thoughts and decided to actually teach their kids to code using QuickBasic. I gave some thought but quickly had to admit how much some of QBasic's idioms, such as GOTO statements, set me back when I did graduate to more advanced programming languages. Python, I felt, would be a much better language to learn to code with initially. I spent some time to look and see if anyone had built anything as intuitive and straightforward as QBasic, but using Python - but I couldn't find anything. So, I decided to give a go at making such a programming environment myself, and QuickPython was born:

What is QuickPython

QuickPython is essentially a QBasic or TurboPascal like programming interface for Python. The idea is to fully contain everything you need to learn how to code and run that code in a single place. No internet or external programs required. It's also just meant to be a fun project that brings back nostalgia feelings for anyone who learned to code on an old PC.

For the most part, QuickPython runs vanilla Python - though a few builtin extensions have been added for easier coding of terminal games:

cls(): Clears the screen
beep(): Makes a beep sound
@main: Defines a method as the main method to start

It also comes with many example games written in Python that users can use as an inspiration to build their own, or just as one more local resource to learn the language.

Under the cover, QuickPython relies heavily on Prompt Toolkit to power much of the UI features and is a single file almost exclusively edited with QuickPython.

Thanks For Reading

Thanks for taking the time to read about this new project! What do you think of QuickPYTHON? Any projects you would like to see in the future? Any projects I should try out?

Happy Coding!

~Timothy Crosley

Introducing isort 5: The Refactoring of a 10 Year Old Project

2020-08-10T04:30:00-07:00


project:	8/52
Title/Link:	isort5
Pitch:	isort your imports, so you don't have to.
Read if:	You're interested in Python auto formatters or the challenges of refactoring widely used code that is a decade in the making.
Skip if:	You're not interested in auto formatters, or are only interested in brand new projects.
Prior Work:	rope, older versions of isort

Well, that took longer than expected.

I know. I know—52 projects in a year. A year has likely already passed, and I'm talking about project #8. I've learned a lot over the last year. I've switched jobs (hello Stripe!), celebrated my daughter's 1st birthday, and witnessed the world enter a pandemic. I have, however, NOT finished 52 projects. I knew it was unlikely I would reach the goal from the start, but I was sure either way, I would enjoy the journey, and I have! Hopefully, I've built some useful projects during that time as well. I indeed spent more of that improving one of my oldest projects than expected, but I'm glad I did. However, this put me at a cross-road. Keep the project goal, and give up the time goal. Or, keep the time limit and give up on the number of projects I wished to release. Or, I guess, just give up. I've decided to give up on the time limit. I've just had too much fun to stop now!

The most downloaded auto-formatter for Python ever?

isort, for me, was a great lesson in how the Python community can take projects with very simple initial ambitions and grow them into useful and complete projects. When I created isort ten years ago, and then released it online six years ago, I envisioned it only as a tool for individual programmers to sort imports. Once. You know, to save a bit of time. Maybe with some editor integration for convenience. I thought I'd release it, get some suggestions for some alternative styles, and be done with it forever. Initially, it was so naive as to auto strip all comments to make this task easier (for isort not for the user). I mean, if you run it manually as a one-off command, you can always manually add back a couple of comments afterward as well?

The community, not me, had much grander ambitions. isort would be included in CI/CD pipelines. It would check every single change to every single Python file committed to master. If possible, it would automatically make those changes as well. And not just for small projects, for some of the largest Python codebases around. All of a sudden, taking such a naive approach as auto stripping all comments, no longer seemed like a good long term strategy. Over time, pull requests after pull request came in that fixed these kinds of problems. Eventually, isort was able to handle not only a large number of styles but also a large number of corner cases. Quickly, its test suite grew to be larger than its codebase. isort became useful. Not perfect. But good enough for many. As of this writing, isort has been downloaded 116 million times. Which AFAIK makes it the most downloaded auto-formatting tool for Python by at least 4x:

This doesn't mean as much as it might at first seem. After all, the other tools on this list encompass a much broader scope than just the formatting of imports. Part of it may be that isort's unique focus allows it to be used alongside any of these other formatters. Still reaching 100+ million downloads for my little import sorter, was something I never expected to see.

The problem with isort's organic growth

Over a hundred people (180 as of this writing) have helped improve isort over the years. Fixing hundreds of bugs and introducing dozens of features. But what never changed, until isort 5, was the internal structure of isort. A structure that each fix and improvement, while pushing the project ahead, seemed to ingrain further.

From the beginning of isort, the core and bulk of functionality happened within the __init__ method of a single class: SortImports. While always arguably a sub-optimal design decision, as a new Python programmer at the time, it didn't seem too bad when that core logic was only dozens of lines long. At the end of the isort 4.x series, that core logic had ballooned to hundreds of lines, with the containing class over a thousand lines long. Everything isort would do, internally, was accomplished by mutating attributes of that single class.

Another pain point was isort's initial design, only being centered around an individual developer running it on their machine. As isort became a part of CI/CD systems, its initial trade-off to using as much magic as possible to determine what section an import belonged to, no longer fit the reality of how isort was run. This kind of magic led isort to work correctly on one machine while failing to categorize imports on another—a very frustrating experience when isort is acting as a gatekeeper for a codebase.

I'm still amazed by how far isort went with this approach. Eventually, however, I no longer even felt like I could take pride in the project I had created. Issues piled up that the current architecture couldn't solve. Corner case after corner case had been worked around, but these workarounds had been pegged onto the initial architecture. An architecture that was never planned.

Refactoring the 10-year-old codebase

There were some initial attempts, from various brave souls, to refactor the isort code base, usually, around the usage of an AST. All of them failed. According to those who had tried, the major problem was that it was tough to keep all the existing configuration options, formats, etc. while also rebuilding the core from the ground up. So, when I finally decided I needed to clean up the code base, and solve the many issues that had been hanging over it, I decided to take a different tact.

To me, the biggest problem wasn't the lack of the usage of a traditional AST, that was actually a feature. isort didn't care about anything but imports. A traditional AST parsing approach would have isort parse an entire file, produce the Abstract Syntax Tree, identify the imports, mutate that tree as needed to sort them, and then output the tree into well-formatted code. Straight forward enough. But, that means isort would be parsing syntax it never cared about. Not only would this waste processor cycles and memory, but it would also unnecessarily make isort brittle, especially for editor based usage. One excellent advantage of isort from an integration perspective has always been that you can run it from any version of Python against any other. When things like the walrus operator came out, this broke many formatters, but isort would keep chugging along.

To me, the biggest problem with isorts internal structure was shared state and the associated lack of separation between concerns. You couldn't parse separately from formatting, and both actions freely mutated the same variables. This behavior resulted in tickets piling up, which desired things like sorting imports within functions, and no clear path for implementing them. So I started there. I would fully separate the parsing from the formatting, clearly defining the boundaries between both.

Much better! Immediately after finishing this separation, however, I noticed something else concerning. Parsing and output formatting both used the same config, which is fine. Maybe even expected. What wasn't anticipated was that both mutated this same config. This made it very hard to call them repeatedly with the same configuration, and just felt wrong. So, I decided to make the config creation happen first and separately, ensuring the configuration was immutable.

Finally, things that seemed out of reach before were easy. I was able to create a core function that simply identified groups of contiguous imports and parsed and then sorted them. I then put this new internal functionality behind dedicated Python APIs for common operations (such as formatting files and streams).

Suddenly, I was able to work through the hundreds of issues open on isort's Github issue page at a steady pace. Still, while the initial refactor was relatively quick, finishing enough clean up that I felt proud of the project again took months. It was the first time in a while that I undertook a big open-source project, literally, one day at a time.

Thanks For Reading

Thanks for taking the time to read about the making of isort 5! If you want to learn more about the 5.0.0 release, you can see the release announcement on the official project page. What do you think of isort? Any projects you would like to see in the future? Any projects I should try out?

~Timothy Crosley

Sign-up for PyCascade Sprints!

2020-02-05T08:00:00-08:00

Coming up VERY soon (this weekend), is my favorite yearly conference: PyCascades! It's a regional conference that takes place in the PNW, this year in Portland, covering all things Python. If you haven't signed up yet, and live in the PNW, I encourage you to check it out! You'll meet so many other great Pythonistas, learn about the latest technologies, and maybe even make some friends.

For the third year in a row, I'll be doing my part to help out by acting as the Sprints Lead. Sprints occur on the Monday following the main conference. They provide a venue for Open Source project leads/maintainers and users to collaborate. Many attendees have made their first-ever contribution at one of these events. And, just as many maintainers have benefited from the growth of the contributor community.

If you are going to the conference, I also hope to see you at the Sprints! You can sign up here. The sprints will be the best opportunity to collaborate on projects, ask any aching questions about one of my projects, or have fun building something new. Additionally, if you have any desire to become a co-maintainer for any of the projects I've created, this is the best venue for me to vet you and add you to the project.

Project Leads / Maintainers: If you would like to lead a project, please submit the project here. This year I'll be giving all leads a special gift packet, including a YubiKey and other goodies, to acknowledge all the hard work you take on. Additionally, before the event, I'll send you my tips on how to best prepare for new contributors, independent of the state of your project.

You can find more information about the Sprints on PyCascade's Website.

Thanks, and I hope to see you there!

~Timothy

Introducing streamdeck_ui

2019-10-08T04:30:00-07:00


project:	7/52
Title/Link:	streamdeck_ui
Pitch:	For the first time, use Elgato Stream Deck devices on Linux without needing to code.
Read if:	You own or have wanted to own an Elgato Stream Deck device and use Linux.
Skip if:	You aren't interested in Stream Decks or other programmable LCD keyboards.
Prior Work:	Python Elgato Streamdeck, Coffee2Code Project

In Defense of Saving Seconds

If you haven't seen one before, A Stream Deck is a programmable keyboard, where each key contains an LCD screen. YouTubers and streamers commonly use it to do such things as trigger tweets, change volume, and switch cameras.

Since the first time I saw someone using a Stream Deck, I wanted one. I immediately envisioned, all my favorite apps, hot-key combinations, and commands only one physical button push away. Sure, it's unlikely such a set up would save much time, but it would save some time. And, if it saved just one minute a day, in 10 years, I would have saved two and a half days! But, it wasn't meant to be, because Stream Deck devices didn't support Linux. And there was no way I was going to stop using Linux.

Being the practical person I am, I didn't give up. I needed that time back. So, I spent 13 days researching how far others had gotten, and finishing support for seamlessly using Elgato Stream Deck devices on Linux.

Of course, I'm aware that I've almost certainly introduced more work for myself than I could ever possibly save from the device. I probably would have been better off setting up and memorizing a consistent set of hot-keys. But, it's hard to underestimate the value of having a tool I utilize every day that symbolizes efficiency at all costs. It appeals to my programmer sensibilities more than enough to make it all worth it.

Getting the Stream Deck Working

When I started researching, I was excited to see that 1) this was a somewhat well-beaten path, and 2) no one had "finished" or even gotten close. That meant I wouldn't have to start from scratch, but there was also still room for me to contribute. It also implied that there was likely demand from other people for something like this to exist.

No one that I could see had gotten close to giving Linux a similar experience using the Stream Deck as existed on Windows. There was no single application that enabled interacting with the device. Instead, every existing solution required users to code. On the plus side, some great libraries had been written for directly talking to the device. In particular, this one, for my favorite language: Python. This library meant, in essence, all I needed to do was build a user interface. I brushed up on QT, including being introduced to PySide2 for the first time, and got to work. I quickly had something minimal working, but realized how much I missed UI programming, and spent a bit more time to add a few extra nice to have features.

In the end, you can now install streamdeck_ui on Linux, and use any Elgato Stream Deck without having to write any code at all. However, if you think of any ways to save just a few more seconds, by writing up some additional logic, I'd be the last one to judge you ;).

Thanks For Reading

Thanks for taking the time to read about this new project! What do you think of streamdeck_ui? Any projects you would like to see in the future? Any projects I should try out?

~Timothy Crosley

Introducing cruft - A tool to manage project boilerplate

2019-09-24T09:30:00-07:00


project:	6/52
Title/Link:	cruft
Pitch:	Create projects quickly from templates and keep them up to date against those same templates.
Read if:	You use Cookiecutter, or another project templating tool, and have struggled with managing the boilerplate they generate.
Skip if:	You don't create enough individual projects to gain value from a project templating system.
Prior Work:	Cookiecutter, Quickly, instantly

What's Wrong with Cookiecutter?

I have to start by saying, I love Cookiecutter. It allows you to create standardized project definitions and then makes those standardized definitions the least friction way to create a new project. This is a beautiful thing. And not just because it saves time. The ease of Cookiecutter templates has often led to them becoming a central place to define how a kind of project should be structured. As a result, popular template repositories receive many pull requests, issues, and improvements over time. What's not to love? Lot's of things. I should mention, I also hate Cookiecutter. In my personal experience, the use of Cookiecutter has directly contributed to hard to maintain graphs of copy-and-pasted code.

We've Automated Copying and Pasting

Copying and Pasting code as a process is terrible. Most engineers I've talked to agree with this. However, with tools like Cookiecutter, copying-and-pasting code has become so efficient that it can be easy to forget. Worse, it is even easier to do so blindly, because you just run the command and the code appears. Why is it so bad? Your project lives after project creation. Each line of code has costs, copy and pasted or not. Over time you start to feel those costs. Even a simple change, like removing a deprecated config option, can have rippling changes as you have to update every project by hand to include the improvement. Worse, because Cookiecutter makes it easy to add a lot of structure, smart engineers often get seduced into using hard to maintain non-DRY solutions. Have a bunch of queue processing projects that all get built the exact same way that is represented by a hundred or more line Makefile? Without Cookiecutter many engineers would do something to automate that and share steps between projects. With Cookiecutter, the cost of that large build file is deferred. So, many will just push the duplicate code approach and then be surprised when down the road they have to make the same changes to all their projects. What is this, but starting a project with a significant continued source of technical debt? And this, in the end, is what much of Cookiecutter output represents, technical debt. Often, it's technical debt from ecosystems the end-user doesn't control (such as unwieldy and large setup files). Cookiecutter lets you ignore that the upstream system has poor abstractions when you create the project, but those poor abstractions still exist. Sadly, tools like Cookiecutter end up celebrating the automation of project creation but rarely highlight to the user the cost associated with this automation.

Introducing cruft

To help solve the maintenance problem associated with project templates, I've created a tool called cruft. It's called cruft for the same reason Chrome is called Chrome: It's an acknowledgment, to both the end-user and the project, that project templates can quickly become sources of cruft and expanding boilerplate in projects. cruft believes this boilerplate, while often necessary, is something to be managed not something to be celebrated. cruft uses Cookiecutter behind the scenes to expand templates, making it fully compatible with existing templates, but it does so in a more maintainable way. So how does cruft help solve this problem? When you create a project with cruft, it adds a single additional file to your project .cruft.json:

        {
            "template": "https://github.com/timothycrosley/cookiecutter-python",
            "commit": "8a65a360d51250221193ed0ec5ed292e72b32b0b",
            "context": {
                "cookiecutter": {
                    "full_name": "Timothy Crosley",
                    "email": "timothy.crosley@gmail.com",
                    "github_username": "timothycrosley",
                    "project_name": "cruft",
                    "description": "Allows you to maintain all the necessary cruft for packaging and building projects separate from the code you intentionally write. Built on-top of CookieCutter.",
                    "version": "0.0.1",
                    "_template": "https://github.com/timothycrosley/cookiecutter-python"
                }
            }
        }

This file contains the template you used to create your project, the git commit hash at time of creation, and every parameter that was passed to Cookiecutter. cruft is then able to utilize this to help you keep your project in sync with the parent template.

cruft check will return an error if the template has received updates that the project hasn't been updated to include. Perfect for inclusion in a CI/CD pipeline.
cruft update will present the changes introduced by the template and ask if you want to apply them.

These simple additions allow you to keep your projects in-sync with the template they came from overtime. Even better, they make these changes visible to you, so you see the costs associated with large and unwieldy templates, and are encouraged to templatize only the subset of code that makes sense.

Find out more about cruft on its website: here

Thanks For Reading

Thanks for taking the time to read about this new project! What do you think of cruft? Any projects you would like to see in the future? Any projects I should try out?

~Timothy Crosley

Introducing hypothesis-auto - An extension for Hypothesis that provides fully automatic testing for type hinted functions

2019-09-19T04:00:00-07:00


project:	5/52
Title/Link:	hypothesis-auto
Pitch:	Have tests automatically generated for you.
Read if:	You are interested in increasing test coverage, utilizing property-based testing, or reducing the number of tests you need to write by hand.
Skip if:	You already heavily utilize Hypothesis or another property testing system.
Prior Work:	Hypothesis, QuickTest.

Why Do We Even Write Tests?

Personally, my two biggest motivators for writing tests are to save time and to save face. Over and over again, I've experienced painless refactors and deploys where there are many useful tests, and hours or days of toil picking up the pieces from when there are not. However, I've always felt something was missing from the tests I wrote. They did a great job of keeping me from reencountering the same bugs! But, I would still often encounter what seemed later to be glaring issues in real-life usage. The cases where there were many good test cases? Those test cases primarily came from issues users of my project had encountered. As the number of regression tests grew, so did time before needing to add a new one. However, each regression test represented an error a user already encountered. In attempts to reduce the number of bugs on the initial release of a project, I invested more and more into testing upfront. Most projects I've released in the last few years were released with at or close to 100% test coverage. Still came the inevitable bug reports.

Of course, no matter how much we test, there will always be bugs in any sufficiently complex piece of software. But, what can be done to reduce the number of genuinely embarrassing ones? Over time, my problem with TDD as the accepted solution to this problem has been that it's too easy for me to fail in the same way twice. Writing the tests first assumes I'm going to pick the right test case to write and then write the code in a way that matches the intentions of the test. It's certainly better than no testing. However, I often miss edge cases on either side of that equation. I often feel that my test cases are an attempt to outsmart myself in implementation. But, often it happens that I'm exactly as smart (or not) in similar ways when writing tests as I am implementing features.

A Friend That Writes Test Cases

At work, I've often found a way to work around this problem is to ask coworkers what edge-cases concern them. The difference in backgrounds and thought processes often leads to their edge-cases being very different from my own. Once they've shared their edge cases, I then write up test cases that map to them. Sometimes they even write the test cases for me. I have great coworkers. However, this isn't something I can rely on for my own small independent and often experimental projects. I need a friend that writes test cases for me. But, I would never impose that on someone I considered a friend.

Property-Based Testing

Clearly, I had to build such a friend. When researching this, I came across property-based testing, and it seemed like a perfect solution. I then remembered a talk at PyCascades that went over the Python implementation of Hypothesis, and how it could, given a set of strategies, generate virtually infinite test cases for you. Who wouldn't want a friend like that? I was sold. However, something felt a little off about using Hypothesis to me. In my newer code, I utilize mypy and type annotations to have well defined and typed public functions. But, when using Hypothesis, it felt like I often needed to repeat these typing definitions:

from Hypothesis import given
import hypothesis.strategies as st


def add(number_1: int, number_2: int = 1) -> int:
    return number_1 + number_2


@given(st.integers(), st.integers())
def test_add(x, y):
    assert type(add(x)) == int

There must be a way to take it one step further and automatically generate test cases based on the type annotations?

Introducing Hypothesis Auto

Rather than build a whole new property-based system from scratch, I decided to create an extension for Hypothesis. This extension would automatically generate test cases using the tested Callable's type annotations while allowing any parameters strategy to be replaced. With hypothesis-auto, the above test becomes:

from hypothesis_auto import auto_pytest_magic


def add(number_1: int, number_2: int = 1) -> int:
    return number_1 + number_2


auto_pytest_magic(add)

You can see this project here.

After building the extension, I immediately used it to improve portray's test coverage and find/fix additional bugs in preparation for the 1.3.0 release of that project. So far it's proving to be so easy to use against existing type hinted and pure functions, as to be a no-brainer addition to my test suites. Combined with Examples to define and document the happy-path, static analysis, and a suite of regression tests as needed, I feel like I now have the tools I need to build code just robust enough not to be embarrassed.

Thanks For Reading

Thanks for taking the time to read about this new project! What do you think of hypothesis-auto? Any projects you would like to see in the future? Any projects I should try out?

~Timothy Crosley

Introducing eXamples - Python Tests and Documentation Done by Example

2019-09-11T03:00:00-07:00


project:	4/52
Title/Link:	eXamples
Pitch:	Examples that double as tests and documentation while enabling interactive discovery.
Read if:	You want your project to have more examples, tests, and documentation, but you have limited time to invest in all of them.
Skip if:	You are happy with doctest or have the resources to dedicate fully to each area, and discipline to keep them in sync.
Prior Work:	doctest, hug.

You May Never be Able to Find this Project Again

... Or you may run into it way too often. I'm not sure what the result of using such a ubiquitous programming term such as "examples" as the project name will have. I suppose we are about to find out. When I saw the name was available on PyPI, I couldn't resist using a name that perfectly reflected what the project did. Luckily, Python3 did away with implicit relative imports, or such a name would likely cause all sorts of local conflicts for those who installed it. Just in case I've also registered the project under the much more unique name "xamples".

What Problem Does eXamples Solve?

When creating, or maintaining a project with a public API, examples are a must to demonstrate to new users (or yourself a few months down the line) everyday use cases against your API. A lot of time, a single usable example that a user can interact with, can answer the same question that may require pages of documentation to understand. However, where to put function level examples has never felt evident to me. You can try your best to keep them in-sync in a docstring. Maybe even using something like doctest, that interacts with your docstring like it is code. Or, you can place your examples in test cases and hope that users know to look there. Of course, at the time of using an API, they may not even have your tests easily accessible. And what if some of the APIs end up exposed over an HTTP service? How could you define your examples so they can be utilized to provide examples to API users over those mediums?

What's the Proposed Solution?

eXamples tries to answer these question for examples, in the same way, that type annotations answer it for parameter types. By moving the examples into programmatic definitions associated with the functions on which they operate, instead of within unassociated code, or worse, a free-text field (the __doc__ string). Once done, it becomes trivial to reuse examples for multiple purposes. The eXamples library enables quickly creating these examples, verifying them against the function signature, and utilizing them as test cases. And, of course, it makes them programmatically available. This means, that over time, projects like hug can automatically detect examples you have on API endpoint functions. It can then provide them to users of your HTTP or Command Line interfaces. Finally, eXamples, by default, adds your examples to your docstring, in a way that renders beautifully on portray and pdocs.

Creating examples using eXamples is as easy as adding an example decorator per example with parameters that match the definition of the attached function:

from examples import example


@example(1, 1)
def sum(number_1: int, number_2: int) -> int:
    return number_1 + number_2

For a full overview of how this works and what it enables, see the projects documentation website.

Things I Experimented with When Making eXamples

Since, unlike pdocs, this was a greenfield project, I had some flexibility to try out some new things when building it.

pydantic

I looked into a variety of approaches to verify the type signatures of provided examples matched that expected. I had hoped, I could find a way to call mypy programmatically on a single function. But, it proved, if nothing else, not to be an intended use case. After looking through several runtime type validation libraries, many now defunct, I decided to use the one I've seen most commonly: pydantic. In testing so far, it seems to work reasonably well. The only unfortunate thing is that it seems to want to validate class schemas only, instead of arbitrary functions that have type hints. I was able to work around this by dynamically creating the schema class using pydantics create_model utility function against the information garnered by introspecting the provided function.

functools.singledispatch

I wanted to enable eXample's core API functions to be callable from multiple scopes (function, module, global). This seemed like an excellent opportunity to utilize singleddispatch for the first time. Overall, it was intuitive and seemed like an excellent way to provide structure around what would otherwise be done in a series of if-else statements. I did get tripped up by the fact that it doesn't allow that single argument to be optional. But, after reflecting, I felt the decorator's limitations were helping to guide me into a better more consistent API design.

Thanks For Reading

Thanks for taking the time to read about this new project! What do you think of eXamples? Any projects you would like to see in the future? Any projects I should try out?

~Timothy Crosley

Introducing pdocs - Documentation Powered by Your Python Code

2019-09-03T04:00:00-07:00


project:	3/52
Title/Link:	pdocs
Pitch:	Modern MIT fork of pdoc. Automatic Documentation from Python code.
Read if:	You want to use pdoc but not AGPL OR are interested in documenting Python projects.
Skip if:	You don't create Python projects. Or, you've already created documentation sites for your Python projects and are happy with the tooling around them.
Prior Work:	pdoc, portray.

Sometimes We Don't Pick our Projects

This next project isn't one I chose. I had many different ideas of what to work on after portray, but after that project unexpectedly reached the top of Hacker News, priorities changed. I work on these projects with the hope that some of them will not only scratch an itch that I have but one that's shared with many other developers. So I saw the interest shown in that post as a confirmation that I was on the right track. The most immediate fallout of this was improvements made to portray in the days following the post, as can be seen in portray's changelog. I was thrilled to be able to solve the most immediate pieces of user feedback quickly.

However, one thread brought up a controversy I noticed late into the project: https://news.ycombinator.com/item?id=20803008. I had hoped I could quickly pivot to using the original pdoc that pdoc3 was based on. If only to be able to bring the focus back to the problems I was trying to solve. However, as I researched, I realized things wouldn't be quite so easy. pdoc hadn't seen a release since 2016. It had, however, received code contribution up until last year. Upon investigation, much of this work was broken, leading to the lack of a PyPI release. I was tempted to give up, but then I reread the GitHub conversation between the original pdoc maintainer and the pdoc3 maintainer, and I knew I couldn't. When pdoc was forked into pdoc3 the license was changed from "Unlicensed" to AGPL. Because of the confusing naming and forking, when I had started working on portray, I saw the unlicense of pdoc and the MIT of mkdocs and missed the AGPL of pdoc3. The license incompatibility of an MIT project using an AGPL project forced my hand. My next project would be a fork of the original pdoc.

Creating pdocs

I decided to call my fork pdocs as I felt it had nice symmetry with the other portray dependency mkdocs while holding a dual meaning well:

Python Docs
pdoc successor

Now that I had a name decided, all I had to do was:

Fork the original project
Fix anything that might be wrong
Update portray to use the new fork
???
Profit

Of course, it is much easier said than done. I have to guess that most Open Source developers don't look forward to forking an old project with a long history. I know I dreaded it. There tends to be technical debt in old Open Source projects, just as there is in commercial ones. The moment you fork, especially if your fork is successful, you own that debt. And, since you likely aren't the creator, you probably don't understand the debt you now possess.

The one thing I did have going for me, is that the work done on pdoc most recently was work to refactor it. While the pdoc repository was in a broken disarray: the refactoring that had already been taken on made it much easier to grasp:

The logic was moved from a single module into many separate ones. Even better the boundaries where these modules were separated made sense (web, cli, doc, etc...)
mypy typing was put in place
Code auto-formatting and linting was put in place
More tests were written

The CLI included numerous settings that didn't go anywhere and most simple commands I tried threw an exception, but the groundwork was in-place.

Over the next several days, I played whack-a-mole with various bugs as I encountered them. Eventually, I took inventory of what I needed for portray, that pdoc even without bugs, didn't have:

Type Annotation support
Markdown output
Python 3.6+ support

And one bonus:

A straightforward API for portray to call

To accomplish this, I decided it would be best to split the API from the CLI as I did with portray while limiting the API and commands to the bare minimum to start. I ended up with a stripped-down version of pdoc that could (given a list of modules):

Serve documentation locally.
Render documentation to HTML within a specified directory.
Render documentation to Markdown within a specified directory.

And, that's it. Of course, it could do so both over HTTP and a simplified Python API. By simplifying the scope, I was able to complete the first working fork in a reasonable amount of time, and quickly update portray to use it. pdocs can't do everything pdoc could do, but it does everything portray used. And, I hope, most of what users of pdoc use.

Using pdocs from portray

Since I knew my intended use from the beginning, I tested pdocs against portray throughout the development. So not only am I able to release pdocs today but also an updated portray that uses it. If you currently use portray this change should be transparent to you unless you have manual pdoc3 settings defined. If so, simply change your settings from living in [tool.portray.pdoc3] to [tool.portray.pdocs]

Thanks For Reading

Thanks for taking the time to read about this new project! What do you think of pdocs? Any projects you would like to see in the future? Any projects I should try out?

~Timothy Crosley

Introducing portray - Your Project with Great Documentation

2019-08-26T04:00:00-07:00


project:	2/52
Title/Link:	portray
Pitch:	Beautiful documentation websites for Python projects without the work.
Read if:	You create Python projects that could use browseable and searchable static documentation websites.
Skip if:	You don't create Python projects. Or, you've already created documentation sites for your Python projects and are happy with the tooling around them.
Prior Work:	MkDoc, pdoc, sphinx.

A Birthday Gift to Myself

I decided to release portray today because today marks my 30th birthday. And, I couldn't think of a better gift to give myself then more and better documentation. I know I'll use portray for the rest of the projects that I create, and retrofit my old ones to use it as well. What would be even cooler is if even one developer creates a documentation website with portray when they wouldn't have created one otherwise.

What Problem Does portray Solve?

In an ideal world, every programming project should have documentation. Python projects being no exception. But, this documentation shouldn't come at the expense of writing elegant code and APIs. Indeed, if things are working well, the two goals should complement each other. One thing that often happens, though, is that documentation systems become complex. This complexity can lead to accomplishing those two goals separately. And, when done independently it can be easy for them to feel like they compete with each other. One common way this dynamic plays out is in stale documentation. If code and documentation live too far from each other but at a similar level of abstraction, this drift seems to be an inevitability.

This fact has lead to a sentiment that I have heard many times. And, indeed shared an appreciation for in the past. I'll summarize this sentiment as:

"Wrong documentation is worse than no documentation."

Which, of course, could be seen as conflicting with another common sentiment. The competing view is captured well on pdoc3's homepage:

"Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing."

In the Zen of Hug we summarize these seemingly competing desires as:

Wrong documentation is worse than no documentation. Everything should be documented.

The reality, as is often the case, is that both sentiments have merit. In the end, almost everything we do as humans that is of significant complexity has errors. If a documentation website helps solve the problems most users have while having a few incorrect sections, it is still a net good thing. But, if a well-documented project is wrong in particularly critical sections - it might have been better with no documentation at all. In the end, the best thing to do may be to reduce the complexity as much as possible. And hope that by doing so, we reduce the separation of the goals and the corresponding errors.

One of the unfortunate things is that I believe these different viewpoints are reflected in the tools available. In general, at least in Python, we have two types of documentation systems:

Automatic Reference Documentation: These systems include pydoc, pdoc, pdoc3 and others. They take your existing code and docstrings and put it on a website. This can undoubtedly be useful. But, generally, this is the kind of documentation you look at as a reference once you are deep into a problem. Often, it feels it is easier to look at the code. This is especially true since these systems tend not to be searchable.
Manual Documentation Systems: If the other systems view code as the first-class entity, these systems see documentation itself as the top priority. They need configuration files. They often suggest you put non-standard docstrings in your project. And, in general, they do everything they can to ensure that the outputted documentation is what is envisioned. These systems produce documentation websites that are searchable, browseable, configurable, and beautiful. Projects that support this line of thinking include: sphinx and to a lesser degree MkDocs.

In the end, what I've always wanted was a documentation generator that put both my code and the documentation I was generating on equal footing. I've wanted a documentation generator that was easy to use, searchable, and configurable only if I needed it to be. One that auto-generated reference docs, while enabling me to make beautiful quick start guides. I hope portray is that project.

What's the Proposed Solution?

portray is an automatic documentation website generator for Python projects. portray tries as much as possible to utilize existing work to accomplish the goals stated above. Under the hood, portray combines the Markdown documentation rendering capabilities provided by MkDocs with the automatic reference documentation generated by pdoc3.

I encourage you to look through the project's self-produced documentation website. I believe that website is the best overview I can give of portray as a documentation solution.

Things I Experimented with When Making portray

Continuing from my last project, I wanted to explore a few new tools when making portray itself:

Documentation

I used two new tools when making the documentation for portray itself.

asciinema - Made it super easy to record CLI interaction. The only downside is the CLI sessions can't be placed directly on GitHub markdown pages. They can, however, be placed on portray produced documentation websites.
peek - Made it trivially easy to record a section of my screen as a gif, which I used for the GitHub README.md file. I did, however, have to download an older version of the .deb due to this issue.

Local Environment Management

For the last project, I gave PipEnv a run-through, for this one I used poetry. From my experience, poetry wins hands down. It managed to replace flit, remove duplicate dependencies, and maintain stability across machines. All while using the standard pyproject.toml configuration file. Over time I'll be switching all my projects to poetry. Sébastien, if you happen to read this, kudos your project is fantastic. That is all.

Thanks For Reading

Thanks for taking the time to read about this new project! What do you think of portray? Any projects you would like to see in the future? Any projects I should try out?

~Timothy Crosley

Project 1/52: preconvert - Supercharge Your Serialization!

2019-08-18T23:00:00-07:00


Project:	1/52
Title/Link:	preconvert
Pitch:	No more `is not JSON serializable` errors
Read if:	You use JSON, MessagePack, BSON, or similar. Or, you are interested in the state of developing Python projects.
Skip if:	You don't use serialization, only use it with basic built-in types, or only use it from the context of a framework that handles it well already.
Prior Work:	https://hynek.me/articles/serialization/, json-defaults, and probably many more.

What Problem Does preconvert Solve?

My first project is a small one that comes the way of a goods friends request made at OSCON. Many of us use Python's built-in json module or one of the similar serialization libraries available on PyPI. These are trivially easy to use, and generally, work great. You pass in your native Python objects to json.dumps and your done:

import json

json.dumps({"project": "preconvert"})

They even tend to follow the same loose specification (a dumps and corresponding loads method) making switching between them for any reason equally straight forward.

import simplejson as json

json.dumps({"project": "preconvert"})

This is pretty awesome! It's made even more so by the years of optimization these JSON libraries have received. Not only is outputting JSON easy, but it's also fast.

Problems occur when you start going beyond the basic built-in types. Search for is not JSON serializable and countless stack-overflow questions will appear with many workarounds for the problem. The simplest of which is to override the default callback method provided by handling just the type that failed:

import json
from uuid import UUID, uuid1


def fallback_conversion(item):
    if type(item) == UUID:
        return str(item)


json.dumps(uuid1(), default=fallback_conversion)

The above works great when you have one type you need to convert, and one place where you serialize data. It works particularly badly if you are serializing data which you don't necessarily have full control of in multiple places. Many web frameworks, including hug, provide mechanisms to get around this. They allow you to extend JSON serialization cleanly and provide built-in default serialization for most common types.

My friend, Brandon, suggested this shouldn't be hidden within the walls of a framework. Everywhere json, or another serializer is used, it should be trivial to expand with custom types and handle common ones right out of the gate. I agreed, and preconvert was born.

What's the Proposed Solution?

preconvert is a small, framework independent, extendable Python library that aims to solve the above problems by:

Providing an easy way to specify custom type serializers

import preconvert

@preconvert.always(UUID)
def convert_UUID_to_str(uuid_instance):
    return str(uuid_instance)

Using this ability to automatically handle common types (UUID, dataclasses, etc)
Adding an easy mechanism to extend this further using entrypoint powered plugins
Exposing the same interface defined by existing serializers to make preconvert an easy drop-in replacement.
```
from preconvert.output import simplejson as json
```

preconvert currently works out-of-the-box with bson, json, simplejson, and msgpack. Currently, there is one plugin to handle numpy types, that can be enabled simply by adding it to the projects package requirements.

pip install preconvert_numpy

For more information about the project, browse the documentation website.

State of Python Project Creation

One of the reasons I wanted to challenge myself to create 52 projects over this next year was because I genuinely believe it shouldn't be hard to create projects. While Python makes many things easy, it's surprising the number of things that should be considered for even a simple project.

Documentation

For this project I used both pdoc3 and mkdocs for the first time. pdoc3 is probably the easiest route to document a Python project and I appreciated how it encouraged me to write more expressive doc strings and better organize my project. mkdocs on the other hand has beautiful templates, integrates very well with the existing Markdown documentation I tend to include in GitHub repos, and has built in search capabilities. However, mkdocs noticeably lacks any auto-documentation abilities at the current time. To get around this, I created a build step that included customized pdoc3 output that was compatible with what is expected by mkdocs. I think it works fairly well, but I was disappointed with the lack of a robust all-in-one solution for simple projects.

Local Environment Management

I gave PipEnv its first serious try. It worked alright, but I found it surprisingly slow. It often was slow enough to make up for any time benefit it could have provided. I also found pipenv run and pipenv shell clunky to use. Finally, I found it's lockfile to be confused when I switched machines constantly, my best guess is because of wheels for different platforms. Still, I'm glad to see project environment management become an increased area of focus. For my next project, I intend to give poetry a try.

Packaging

I found flit to be an absolute joy to use. For the most part, it just worked and got out of my way. It's simplified approach is perfect for small projects. The only downside I encountered, which unfortunately for me is a major one, is the lack of Cython support.

Static Analysis

I'm a huge fan of static analysis, code formatters, and any tool that aims to raise the bar for code-quality on a project automatically. This project included all the ones I've used in the past, but also, mypy. For the most part, it just worked. As someone who is very comfortable with dynamically typed languages, I was surprised how little it impacted my productivity, and how even in this small project it found real errors.

I'm hoping to make my next projects ones that simplify these steps if only to give me more time to devote to each individual project.

Thanks For Reading

Thanks for taking the time to read about this new project! What do you think of preconvert? Any projects you would like to see in the future? Any projects I should try out?

~Timothy Crosley

whoami

2019-08-11T08:00:00-07:00

Hi, I'm Timothy! I'm a Principal Software Engineer and Open Source Evangelist working on technologies to keep the web safe at DomainTools. I live with my wife Amanda and our daughter Penny in the Licton Springs Neighborhood of Seattle.

You may know me from one of my Python projects, such as:

Or, from one of my older, now deprecated Projects:

WebBot Web Application Framework
HoneyDo.es social to-do list.

Or, you may know me from one of the many talks I've given at regional Python meetups and conferences. Finally, you may know me from my local community involvement: the neighborhood blog I run, the cleanups and events I've helped to lead, or North Seattle's Open Source Hackathon.

I've crossed paths with many people, both directly and indirectly. But, I've felt connected to so few. I hope this blog can help build connections with more of you. After all, the few of you I have directly interacted with have left a profoundly positive impact on my life.

As I get close to my 30th Birthday, I've decided to make a home for personal reflection and project updates. This website is that new home. To begin this journey, I thought I would reflect on how I got here. In particular, how I got into technology, what my specific interests are and what things I aim to explore in the future.

"No need to argue momma, I've made us money!"

I'm choosing to start my story before I ever even touched a computer. I'm doing this to highlight a truth about myself. Like many others, I've always hated to admit that money would have anything to do with my motivations. "I want to build great things!" "I want to make the world a better place!" "I want to empower people!". These are the messages I want to say to myself and other people. As is often the case, the reality isn't so simple. I do want all those things. But I know I am also driven by an innate desire to provide.

I grew up in a passionate, loving, evangelical, and very low-income family. My childhood defined by ups and downs in nourishment and accommodations. My first moments of life I lived in, what felt, a mansion in Pennsylvania. Our Grandparents had provided us the housing, in hopes that it would enable our family to thrive. Soon, things changed. Before I had turned 3, my father had decided it was his mission to bring the gospel to the world. We left our comfortable accommodations traveling through the United States sharing the scripture.

I don't have many memories from this time. But, there is one story I've heard that demonstrates well what life was like for the Crosley family at that point. We traveled from city to city, staying with other believers when we had the opportunity. Of course, sometimes we had to use temporary accommodations, and without much money coming in, this didn't mean a hotel. On one night, we made a tent to sleep right outside of a city. With me being the youngest (2 years of age at the time) my parents laid me on their only comfortable piece of bedding. As it grew late, I fell asleep on what resembled a small air mattress. To my family's dismay, a thunderstorm rolled in as we began to sleep. With no other choice, everyone attempted to sleep through it. As we slept, the rain got more and more intense, until it caused flooding that rushed into our tent. Everyone awoke and started grabbing their few personal belongings. In the panic, they didn't realize that I had started to float away. I always was a deep sleeper. While I did float outside of the tent, I didn't make it too far. Soon my parents realized what had happened, panicked, and found me. Everything was okay. But, it was so close to not being okay. There is no better way to put my early years, than always toeing the line between everything being okay and not.

Of course, at some point, my parents grew tired of this. With few options, they reached out to my dad's parents, hoping they would put us up as we got back onto our feet. At four, I found myself, and the rest of the family living at my Grandparents. Here is where I find my first complete memory formed. My Grandparents were always the kindest, most hospitable and loving people I knew. Nonetheless, it was hard for my father to accept help and admit the challenges he faced. He wanted us to go back to being on our own. I remember one night, overhearing my parents arguing in the other room. As I pushed my ear against the wall that separated us, all I could hear was money. I didn't know what exactly it was about money that had my parents in such a fit. But, at the very least, it seemed like more couldn't be a problem. So, I went to work. My Grandparents had left crayons and paper in my room, so I did my best drawing as much money as I could. After drawing two full sheets, I busted into my parent's room. "No need to argue momma, I've made us money!". Everything was okay. But, it was so close to not being okay. I had fixed it. At least, I thought.

"Buy a copy! It's cheap!"

At some point, we did move back out. This time we didn't last too long on the road before settling down, seemingly for good. My dad had secured a stable job at an evangelical home for troubled kids. It didn't pay too much - but the local church rented us out a house they owned, for much less than the market rate. The house was old, in need of love, and full of asbestos. But, it was large enough for our entire family — only 2 of the 4 of us kids needed to share a room. We had found a home.

My father, to his credit, realized technology had to be the future. And he always wanted more than anything for us kids to succeed. We had almost nothing frivolous in the house, but one day he came home with a portal to that future. At around the same time Windows 95 came out, we got our first computer. It, of course, didn't run Windows 95. It came from a pawn shop and ran only MS-Dos. It was perfect.

There was very little of anything on the computer. At first, this disappointed me. I heard technology was limitless! This computer only had Edit and this mysterious application called QBasic. Luckily, my dad did secure a copy of Commander Keen on a floppy. After a brief, but enjoyable stint playing the game, I decided I wanted to discover QBasic. Luckily, it came with built-in help. In retrospect, this was the first time I learned the value of limitations. While boundless options have a broad appeal, limits are often even more empowering. Had I had the entire internet, after all, I would have collected every game there was. And, likely, never learned to code at all.

At the time, and throughout my entire pre-adult life, I was home-schooled. My mom did her best to teach me but was always too ill to give it her full attention. No worries, this year we tried an off the shelf curriculum. Soon, I found myself less and less interested in the curriculum and more and more interested in the computer. When I did do my curriculum, I did everything I could to work the computer into it. I built memorization software for tests and made my own gamified math quizzes. And of course, QBasic functions as a calculator in a pinch. My siblings couldn't understand: "What are you doing, Tim? We already beat Commander Keen!"

Of course, my siblings were right. We had run out of games. I was successful in using QBasic to help me out with homework. But, we all knew that the real purpose of a computer was for interactive entertainment. So, I did the only thing I could. I tried my best to make a game in QBasic. The first few were simple. ASCII art, a choice prompt, with the ASCII art updating after each choice submission. It was enough to get me even more hooked. I would sneak out of my bed at night, as I thought of different ways to make the game better. One night, I remember reading something in the documentation. Something that might enable me to make my games not need the enter key to submit each action. I ran downstairs until I got it working. I then stayed up all night, making my first game that utilized my discovery. It couldn't have been more worth it.

This first game still used ASCII art, but it was completely interactive. You could move your little character on the platforms, and jump on and over enemies. Of course, the enemies still only moved when you did, but I thought it was the coolest thing I had ever seen. It was something I had thought of and created, and it worked. With my limited available canvas, all characters were rendered using one of two available smiley characters. And, with my limited imagination, I called my creation "Smiley."

I remember being so excited when my Grandparents came to visit. They always brought sweets and would take our whole family out to eat! And, I missed them, even though it seemed the family was finally getting along fine. They also had a new gadget, a camcorder, and of course, they wanted to film their grandchildren. Being the youngest at 6, I was a guaranteed target. Of course, for me, it was still about the computer. I took the opportunity to tell the broad audience (that I was sure the camera represented) about my new game. "You should buy my new game! It's super fun, interactive, and only $1!" "Buy a copy! It's cheap!" I exclaimed to the camera. I guess I was still trying to draw up new money. No one ever did buy the game, but it didn't matter, I had already fallen in love with programming.

"Windoze Sucks!"

As the years went on, I kept on programming. And, still, in the one tool, I had available: QBasic. It seemed like not a single day would go by without code going through the keyboard. One day, my parents brought me to the local library, where there was the internet. I scoured as much information as I could about programming. I made many games. Then I made a custom tool to launch my games. Finally, I built a graphical windowing environment I called SystemX. I also occasionally did some traditional studying. Much less than I was supposed to, but it seemed that no one cared as long as I kept scoring well on standardized tests. I learned a lot during this time, but I knew in the back of my head that I also missed out on a lot. One thing I never did was hang out with kids my age, and when I did, none of them seemed to understand me. How can I blame them? I didn't understand them either.

At some point, our lives were stable enough that our parents gave us an allowance of 25 cents a week. It was enough to buy a Little Debbie or another unhealthy snack at the local corner store. It wasn't much, but it didn't matter: I rarely saw those 25 cents. Every once in a while, we make a trip to a book store. There, I would beg my dad to allow me an advance to buy a programming book I thought looked cool. Sometimes, he had no choice but to say no, but more often, he made it work. I gobbled them all up. And everything I learned I immediately tested out. It was addicting and fun. Everything I built is now gone, none of it still useful. But, the lessons I learned back then made me the person I am today. I still owe my dad many quarters.

Three years later, at nine years old, my mom had gotten particularly ill. The rest of my siblings and I became forced to make a trip back to my Grandparents. I couldn't know it then, but I was about to encounter something that would change the course of my life forever. I wasn't allowed to drag along our Family computer. Terrified of being away from it, I was nonetheless thrilled to see my Grandparents. Luckily, they did have a computer — an old IBM PS/2. And a ton of games on floppy disks. Including "Freddies Rescue Roundup." It reminded me of the games I had built and shared a name with my Grandfather: Fred. I loved it! Being tucked away in my Grandfathers office, I never was able to spend too much time on it. And of course, never write much code.

My Grandparents recognized my strong desire to code. They started paying me to do chores around their house and taking me to the local computer store. Over a few months, I ended up with a Frankenstein 386 PC built from the remnants of no longer working computers. The case had a turbo button, but it never got connected to anything on the motherboard. No worries, I didn't need a button to make my computer go slower. I only needed to be compatible with the future. I built the desktop in the room my grandparents had put me up in, and spent hours on it every day. At this point, I got more and more enamored with the graphics the computer could make. I made little animated shorts. I even built a simple personal assistant that would respond to questions thrown its way. All still running on Dos. And, all still powered by an unreasonable number of QBasic GOTO statements.

While I was busy coding, my siblings were busy doing what kids are supposed to during the summer. They were making friends and going outside. I couldn't understand it at the time, but I am forever grateful they did this. My brother, who always liked getting into trouble, made friends with a local kid with the same appetite for chaos. That summer they played with explosives, fought with people in the neighborhood, and tried to teach each-other karate. My brother, no matter how many times we argued, always expressed pride in my technical abilities. One day he invited his new friend over and had me show him the latest program I had been building. My brother's friend, Joe, thought it was cool. However, he asked "Why are you using Dos?", "Why not Windows?". It was a good question, but one with only one answer: "I can't afford it." Joe told me to stay put, while he rushed back to his house and came back with a CD with the text "Mandrake" scribbled on it.

Over the next few hours, we installed one of the first versions of Mandrake Linux on my old 386. The whole experience was dazzling to me. We, of course, ran into several installation issues. But in the end, I had a modern mouse-driven interface. And it had more programs and compilers then I could have dreamed. Mandrake Linux was my first exposure to Open Source, and it had me hooked. Hundreds of programmers from around the world had worked together to give me this gift. They brought me into the future, and they asked for nothing in return. To me, Open Source was and still is, about empowerment. About the recognization, that when we stop fighting and start collaborating, we all go farther and faster. Seeing this, I couldn't understand why anyone would use Windows. I became a Linux zealot. "Windoze Sucks!" I would pronounce, of course, I had barely ever touched Windows. In the end, my zealousness was a childish expression of indebtedness.

"No one likes me. I'm fat."

Time kept turning, the family kept moving, and I kept programming. After that summer came to an end, I came back home - this time with my very own computer. We lasted in the house for one more year, before packing up for yet another adventure. This time, my oldest sister, Charity, was graduating high school. She had her eyes set on going to an evangelical college in Ohio called Cedarville. Smart and driven, she managed to pick up a few scholarships. Still, it was nowhere near enough to pay for the tuition. My father still wanted to help her succeed. Without any significant money to contribute, he decided to take a job at the College. By doing this, he was able to take advantage of a discount provided to the children of faculty. For the next year and a half, my father would work as a janitor. We moved to nearby Xenia, Ohio. A small town known for its history of severe tornado activity.

As a 10-year-old kid, I loved Xenia. Everything felt very safe. So much so that us kids were able to explore the neighborhoods. My parents even felt good enough about Xenia to enroll me in the local public school, and I loved it. I made as many friends and took up as many activities as I could. Every day I would go to school early to perform the duties of a school crossing guard. On weekends, I did my first ever job: delivering the local newspaper. I also worked out agreements with my neighbors to mow their lawn. I made enough money from these jobs to further finance my ambitions. I built up my computer and bought even more programming books. With straight A's everything seemed to be firing on all cylinders. Of course, it couldn't last.

During our short time there, Xenia did have a destructive tornado. But, it wasn't the weather that caused us to leave. My mother's health started to deteriorate again. Her sister asked us to move out to Florida so she could help take care of her. It didn't hurt that my father wasn't enjoying his new gig as a custodian and had a hard time finding other work in the town. We packed up all our belongings, yet again, and headed south.

I immediately missed Ohio. My friends, safety, and freedom. 5th grade was a rare time of socialization for me; I would spend 6th grade once again as a home-schooler. It wasn't all bad, though. My aunt and uncle owned a large apartment complex, and let us stay in one of their units under reduced rent. I also got introduced to my cousins, who would be lasting friends from that point forward. On the plus side, I had a better computer at that point. I would drag my machine to their house, hook up a LAN, and play Counter-Strike. At some point, we decided it was too much work to drag the computer over. We dug a crossover cable a few hundred feet between our two units, allowing us to play without moving PCs. I wasn't the best at Counter-Strike. I was also the only one attempting to run it on Linux via Wine. Nonetheless, I found it exhilarating.

Without school as a "distraction," I threw myself deeper and deeper into programming. There was an acre of woods connected to my relative's property that my cousins and siblings would regularly go exploring in. They would build structures, throw axes, and play BB gun battles. Rarely, I would join them, but generally, I stayed back to code. One of my older cousins, Daniel, was curious about what had me so distracted. Unlike everyone before him, instead of expressing confusion, Daniel got it right away. Over the next several years, I, for the first time, had a friend that understood my passion. I shared with him, everything I thought I knew. When he went to college, he decided to get a Computer Science degree. And, during this time, he shared with me everything he was learning. Daniel is the reason I understood Big O notation, C, and pointer arithmetic, at such a young age.

For the next six years, while we would change cities and my dad changed jobs a half dozen times, we would stay in Florida. The longest straight stretch I would remain anywhere in my pre-adult life. During this time, I transitioned from a naive kid to an angsty hormone-driven teenager. Without the social interaction of school, I looked more and more forward to the youth group at our church. Over time, I became interested in girls. That's when I came face-to-face with the ugly consequences of my single focus life. Avoiding every other activity to spend more time coding left me obese and awkward. One day, at 14 years of age, I looked in the mirror and told myself: "No one likes me. I'm fat." I was 280 Lbs and wouldn't even dare to start a conversation with a member of the opposite sex.

I suppose the lesson I should have learned from this was one of balance. In my 14-year-old mind, though, I didn't have time for balance. Over the next year, I went to work. Not a single day went by without me walking for hours. I used weights, did hundreds of sit-ups and pushups, fasted, and avoided every carb in front of me. Dieting, in many ways, was my new programming. I still found time to program for sure. But, if I had to choose between coding and losing another pound, I would have preferred the weight loss. This focus paid off. As I celebrated my 15th year of life, I had made it down to under 175 lbs, well within the healthy range for a 6'1 male.

The drastic change gave me confidence, and with that confidence, I tried my luck talking to a few girls. To my dismay, it turned out that weight wasn't the only thing girls cared about in a potential partner. I had little exposure to popular culture at the time. And, when I did finally strike up a conversation with one of the girls at our church, we had no shared interest at all. Worst of all: the skill I had spent the majority of my life developing didn't seem to impress anyone. What did seem to impress them? Art, music, and the right amount of rebellion. I wasn't willing to give up. I let my hair grow out, and I started to listen to punk rock. In desperation, I started trying to think of ways to incorporate "art" into my skill set. In retrospect, the interface design concepts I got obsessed with aren't the kind of art that would get a girl. That didn't stop me from diving in even more in-depth.

The most direct thing that fell out of this new interest was an icon set I made for KDE, called the Intrigue Icon Set. I spent countless hours drawing icons by hand and then digitizing them using Gimp. Something I would have never thought to do in the past, and I did it to impress other people. On the plus side, it gave me some of my first experiences giving back to the Open Source community. I interacted with users, listened to feedback, and made improvements. I even became proficient using PHP to build a searchable interface for my icon set. The biggest lesson I learned: Any motivation, no matter how silly, can be redirected into something positive.

Will work for laptop.

Programming had always been exciting for me. But, releasing things that other people used was a whole new level of thrill. It challenged me to be better. Even if only to avoid utter humiliation in the eyes of programmers across the world. Soon, I moved passed icons. I started making working desktop applications using QT and C++. I consumed book after book about OOP, design patterns, architecture, and even management. I had the drive to get better at everything in the development life-cycle not only coding. This built up my confidence while simultaneously building up my uncertainty. I realized more and more that disagreements abounded across the development eco-system. There was never a silver bullet that would make everyone happy. Someone, somewhere, would always be unhappy. The only thing I knew for sure was that I had to keep moving. I had to keep releasing things and improving.

I made a contact manager, an image editor, a custom desktop environment, and a local search engine. Then I made Redhat Linux packages (RPMs) and websites to distribute them all. I felt like I was getting somewhere. I was making steady progress. Still, I hadn't made a single cent from the skill I had spent my whole life developing. I wanted to change that. I wanted to go pro. At 15, I had no idea how. Unsure of next steps, I decided to walk into every storefront within a few miles of our apartment. I would walk up to the front desk and ask if they needed a software developer. Almost no one accepted the offer, and most didn't even know how to take such an inquiry. Luckily, one random clothing store did want a software developer. They wanted a joint Point Of Sale (POS) and inventory management system for their small chain. And, they didn't want to pay what they saw as the high cost of the current off-the-shelf offerings. Luckily for them, there was something I wanted even more. AMD had recently released a mobile version of their pioneering 64-bit processor. I wanted to get my first laptop, and I felt it had to be 64-bit. I told them: "I'll do it for a laptop." I got that laptop and used it to build the system they asked for over the next several months. Meanwhile, a member of our church also heard about my programming ability. They asked me to build their company's website for them in exchange for a few hundred dollars. At 15, In my mind, I had gone pro. Nothing could stop me now.

For the next few years, it continued this way. I made small amounts of money, doing hard, and I felt impressive work. What else was there to do? There wasn't exactly a burgeoning tech scene in Winter Haven. As a teenager, I had a hard enough time convincing people to give me work at all. Not that it mattered. Soon, my parents had moved to an even less dense an area of Florida. As I got closer to turning 18, my days became a mix of studying for my GED, taking community college courses, and caring for my Mother. My family started to struggle, and eventually, I had to take a job at CiCi's pizza to help pay the bills. Embarrassingly to me, I made more money washing dishes than I had writing code. Soon my mom became so ill I had to stay home and care for her instead of taking college courses. I told myself, once I turned 18, I would find a way to make it on my own. And I would do it while doing what I love.

"Give me an opportunity. You won't regret it."

As my 18th birthday approached, I looked for an exit. All my other siblings had left at that point. It was my turn next, and I didn't have to wait long. My father had switched jobs, yet again, to work as a truck driver. While truck driving paid okay, my father still had to spend a large amount of his paycheck on rent. I suggested I could move in with my Grandparents. They could then move into a smaller apartment. The downsize would save more than enough money to make up for what I brought in working at CiCi's. Our lease would be up in 6 months, and my dad loved the idea of saving money. He took it one step further. Once our contract was up, I would go live with my Grandparents and my Mother would join him on the truck. They wouldn't need to pay rent at all.

Over the next few months, after moving back in with my grandparents in Annapolis, MD, I went to work trying to get a job. I built a personal portfolio of projects I had worked on and made sure to highlight both my contracts and Open Source work. I then registered with all the online job boards. In the end, it was my old strategy that worked best. In walking distance from me was an aviation company nestled in a business park called ARINC. ARINC, it turns out, needed all the programmers it could get. As an 18-year-old with no college degree, HR was hesitant to forward my resume along. Even though I wasn't sure it was true, I told HR that if they forwarded my resume, it could only make them look good. After getting connected to a hiring manager for the "ARINC Direct" division, I got my interview. I was nervous as I presented my work and experience. I didn't know or use Python, the primary language of ARINC Direct, but I promised them I would pick it up. "Give me an opportunity. You won't regret it." I left the interview sweating. Two days later, I received an offer. $35K a year. For me, the amount didn't matter. I had gotten my foot in the door. They were willing to take a risk hiring an 18-year-old walk-in who had little relevant experience, and no college degree. I'm forever grateful for that.

I started working at ARINC in June and turned 19 in August. I was so invested in the new job that I worked 10 hours day. I wasn't often the first one in, but I was always the last one out. The next few months were a flurry of learning. There was so much about working in a corporate environment for which I wasn't prepared. I learned how to fill out timesheets, have disagreements with co-workers, and work with others daily over the common code repository at the time: SVN. I also learned how much more important it is to build things for a group instead of for individuals. But, most importantly, I learned never to mix tabs and spaces. At first, I didn't understand why ARINC Direct chose Python. It wasn't compiled, it didn't have static type checking, and there were no pointers to anywhere. I would write code as simple as pseudo-code, paste it into an interpreter, and it would work. It was too easy. Soon, I would question why anyone would choose to start a project in anything else. Up until this point, programming languages were a tool I had to use to solve a problem. They were not something for which I would ever develop an affinity. Python was different. It was productive and pragmatic, and I loved it.

"She's not breathing."

As the year came close to an end, my new coworkers started to leave on vacation. I didn't take a vacation at first, but I did take the opportunity to reduce my hours. For most of December, I was only working 8 hours a day. I started having more time to think about Open Source projects and read even more books. Sometimes, I even went out to meet friends. I felt excellent about my life at that point. Every few weeks I'd email my mother and let her know how I was doing. She would tell me how she was so proud of me! She would also tell me how much she hated living in the truck, and how much she wished they could be home. As Christmas came, my dad decided not to take a break due to the high pay during the holiday season. I understood, but I was disappointed. It would be my first Christmas without them. Instead, I would do my best to enjoy the holiday with my Grandparents.

I still hadn't taken any time off, besides the provided holiday days and immediately went back to work. I was determined to do everything I could to prove my worth. I had already fixed countless production bugs and implemented new product features. Nonetheless, I still felt like I was an impostor, sure I would be fired at any moment. After all, I was only 19, and I didn't even have a degree. Soon, I would stop worrying about that. But, only because something much larger would steal my attention. Five days after Christmas, my Grandparents and I got a call from my father; he was on speakerphone. And he was panicking. "She turned purple!" "I don't know what to do!!" "She's not breathing." My dad was calling as he stood next to my mom in an ambulance. Later we would find out she overdosed on the pain medication she was given due to her health conditions. I blamed myself. I shouldn't have gone off on my own -- I should have stayed and taken care of my mother. I took the next few weeks off.

I didn't know how to react to what had happened. I was sadder than I had ever been, but I knew my sadness couldn't bring my mother back. When I went back to work, I managed my emotions the only way I knew how. I worked even longer days. It felt like I only left work to eat and sleep. I couldn't face the emotions, knowing if I did, they would overcome me.

I worked at ARINC for the next four years. During that time, I made one my best friends, Brandon, who continues to work with me on projects to this day. Over time, I grew more and more in love with Python. I applied the knowledge I had learned from building QT apps to our web development problems. I made a framework that significantly reduced the time it took for our team to produce new features and web apps. We were a small shop with no front-end engineers, yet we were able to move much more quickly than larger shops. AJAX calls, JavaScript, even HTML, became abstracted away from us. Creating new web pages was as simple as dragging and dropping with a graphical user interface builder. It wasn't the perfect option for a big dev shop, but it was an excellent solution for us. Having an innate desire to automate everything, I even built a tool to format and categorize our imports. With ARINC's permission, I would release this tool as isort. In the end, I not only learned how to work within a team at ARINC but how to make money while still contributing back. In doing so, I developed a new guiding passion. I would build tools to empower other new developers like Linux had empowered me so many years ago.

A new beginning.

Over the years, I've taken new jobs, learned new technologies, patterns, and approaches. I've also released countless new Open Source projects. Some have been very successful, others rarely used. All of them have my entire heart put in. And I'm not done. Today is the beginning of my story. After all, I'm almost 30, not almost dead.

I'm hoping to explore ways to simplify and automate development. To enable new developers, experienced developers, and teams alike, to solve their problems. And to do so more quickly and sustainably. Many of my projects will undoubtedly be duds, some of them I hope will help move the industry forward. All of them will be described in detail on this blog.

Over the next 52 weeks, you can expect to read 52 articles posted on this blog. These posts will contain the details of 52 new projects or project updates. I hope you will be part of my story. Let's build some great things together.