Ask HN: How can I pick a side project and stick with it?

You can improve matters by a) scoping projects less ambitiously and b) enforcing a novelty budget on yourself.

A very ambitious project which you will never finish might actually be three small projects which you will finish and seventeen small follow-up projects you'll decide to never take on. Better to get the three under your belt than try to sprint in 20 directions at once and bail with nothing to show for it.

Say that you decide "I'm going to write a personal ontology engine flexible enough to store everything I know, with a snazzy, super intuitive frontend written in , a plug-in system for easily creating visualizations for specific types of data (oh, like, what if my personal knowledge DB could take FEN+PGN and let me replay and annotate interesting chess games in-line, that'd be so sweet!), a bespoke query language, the whole thing will be backed by and, hell, I've been meaning to start containerizing apps at work, I wonder if I should be using docker-compose as I do this or maybe like kubectl, ...also I need to dive into the literature around building ontologies, knowledge graphs, etc. Also, it should probably have emacs keybindings. Probably also support for rebinding keys and defining macros too. Also, it should have a pluggable module interface for scraping semi-structured data sources containing things I don't actually know right now but which I would like to learn at some point. Maybe I can use the hyperlink structure of Wikipedia documents as a scaffold to get the thing started. Oh geez, this is really going to need to support multimedia as well, not just text. Really, this kind of thing would be very valuable in many different contexts. It'd be great if it could be used to automatically generate questions about a topic. I could integrate a spaced repetition algorithm with the question generating bit and then I'd have an amazing study tool. Hmmm, I'd better keep the whole thing as flexible as possible so maybe at some point down the road I can productize it! Sure it's a long shot, but it'd be revolutionary if I could pull it off. I'll start tonight!

To steal from Alan Perlis, "When someone says 'I want to build a personal knowledge DB as a side project and I want it to have perfect ergonomics, top grade discoverability, unlimited extensibility, and unparalleled ease of use,' give him a lollipop."

Even if you don't get stuck in feature daydreams or analysis paralysis and manage to start the thing, your "gee it'd be swell if" side project has a five or ten plus year "labor of love" todo list attached to it. The mismatch will sink you unless against all odds this actually is your labor of love.

As a personal example of the scoping into smaller projects which were standalone (though not a good example of enforcing a novelty budget) I wanted to make a personal activity tracker in 2018 or so. Initial conception was the all-singing, all-dancing, kitchen sink Swiss Army knife of personal data tracking and visualization, implemented in all the technologies I'd been meaning to try out. The backend would be a gleaming beacon of modern infrastructure with everything containerized from the outset, code written in Rust (in which I was a neophyte), phone app written in Kotlin (in which I'd only done the Koans, and with no other mobile dev background, but hey, "Anko" seems to be The Thing to use now, hmm)...

Having been bitten by the bug you ask about in the past, I set that aside. Instead, I did a bunch of smaller projects with each on their own bringing a reasonable sense of completion:

1. Two Python scripts - one to configure desktop OS notifications to fire on a cron schedule ("Nag! Update your data for ") and one for CRUD operations on time series metadata and data points themselves. Just backed by local sqlite. Finished in <1 day. Hey look, MVP! Time to start tracking data!

2. Docker setup for building a Rust binary and then trimming away build tooling to get a deployable artifact. 1-2 days? Containerization itch: scratched. Could be a 101 level blog post.

3. PostgreSQL docker image with CREATE/DROP TABLE scripts. Half a day? I guess I learned about Docker data volumes. I can refer back to this Dockerfile in the future, which is like 9/10ths of how I make progress quickly on new projects.

4. docker-compose setup for a Hello World level Rust binary that reads a record from a postgres db running in a separate container and writes it to stdout. Half a day? Hooray, orchestration.

5. "Hello World"-as-a-Service web app that uses Rocket to send a Hello World application/text response. Not very long to finish. Novelty framework itch, scratched.

6. Learn about Diesel. Use Diesel to make a single SELECT COUNT(*) query against the DB. Send query result as application/text response on a dummy endpoint. Done. Could probably be broken off into a "Intro to X with Y and Z" type blog post.

7. Update SELECT-as-a-Service web app to make a meatier query and serialize response to JSON. Learn about Rocket request guards for managing DB connections. Another day or two? If step six wasn't enough for a 101 level blog post on the topic, seven definitely is.

8. Port the Python CRUD functionality to a Rust REST API atop Postgres. Deliverable: approximately all the backend we'll need.

9. Update the Python CLI to use the REST endpoints. Another day? Done.

10. Start doing Android Hello World apps. Using unfamiliar APIs in an unfamiliar dev environment in an unfamiliar (but nice!) language. Eww, this sucks. I'm not having fun any more. Maybe I should first learn vanilla Java Android app development, then Kotlin, then build on that basis with Anko, then...

And then, at that point, I realized that aside from enjoying daydreaming about the idea of a personal activity tracker, I didn't actually give a crap about having an all-singing, all-dancing personal activity tracker. While I'd been happily chugging along building the thing, I didn't care enough about the underlying functionality to bother taking two minutes a day to, y'know, actually track the data when the MVP nag script ran on its cron. When I bailed though, instead of "ugh, another incomplete useless waste of time," I ended up with several completed small projects and more knowledge about what I actually wanted (to futz around building a personal activity tracker, not to have one).

I can tell what I figured out about myself so far. Sorry for the long post, it's tough to write concise. It also doesn't help that I'm describing insights that took me 10 years to get sharp.

I have thoughts and intentions.

I have behaviours.

It's always puzzling to me how bad I am at predicting my behaviour purely with my thoughts and intentions. I guess people who are addicted to something that want to quit can relate (my addiction: gaming/YouTube/HN -- I'll get back to this).

Because of this weird mismatch I've said to myself: the predictive value of my thoughts and intentions is 0, it's actually negative even. So all I can do is observe myself behaviourally. Humans are quite bad at this, and research shows that friends help with this. People outside of you observe your behaviour better than you do. So either (1) get help from friends to do this or (2) observe your behaviour like a ninja. Luckily for me, I happen to be an introspective person by default, so I chose option 2 (you can also choose both at the same time).

So do not trust your thoughts and intentions.

So observe your own behaviours.

After 10 years, I found out 2 fundamental motivational triggers. One I found out about recently, and I am still figuring out how to (ab)use it, and the other one has been (ab)used for years.

1. I work amazingly well under pressure and stress. This is especially the case when my schedule has no minute of spare time and has some vague accountability system.

Example: I was incapable of studying one degree at university, but I was amazingly capable at studying 2 degrees, while working on the side, while having a social life and a girlfriend. I had too little time in my schedule, so I used 4 hour work week tricks to get it manageable.

What I also did was define a priority list. A lot of priorities on the bottom of that list suffered an ill faith. But my most important priorities (and then some) succeeded.

There is one downside with this. Yes, I was motivated. But I also was stressed the whole time, and while the stressed was manageable, I don't think it was a particularly fun time. It sometimes was, and sometimes wasn't, but the pressure felt painful. And while it worked for my motivation, it does do a lot of damage to me existentially.

However, the good part: I do now have skills that I wished to have. And existentially that actually does really help.

So motivation-wise: it works amazingly well. Regarding life decisions: mixed results.

2. Remember my addictions? I realised that I'm simply motivated by human contact. Not just any human contact, but I am not fully clear on the conditions yet. I know that I must like the person and feel a connection in some cases.

A couple of cases where I've seen this:

A. Making music with someone just watching me (I was super motivated). Making music alone the next day (I didn't do anything).

B. Same story with playing Hack The Box. I hacked 16 hours per day for a month and my buddy was simply putting in a normal work week of it. He said: "you're so much more motivated than me." But I knew if he'd stop playing Hack The Box, then I'd stop playing eventually as well.

C. I thought a lot of university courses were easy, but not easy enough to do nothing about it and pass a test. So I would go to the lectures and be bored out of my mind because psychology is an easy degree :D However, those lectures did actually motivate me to then engage with the course material after the lecture. And the thing is, I thought I would be capable of doing that on my own, but I never was. So I continued going to lectures and be bored, in order to be motivated later.

I am still figuring out how to exploit this, because this motivational factor properly exploited is me winning at life. It has: connection, productivity and a sense of "we're in this together." I'm noticing I find it a bit hard to make friends. I can't just say: oh, I like topic x now and then befriend someone in that topic. If I could, I would be a lot more productive.

(I'm now realising this should become a blog post, moving on :P )

Writing all this gave me a new idea. I'm basically like a framework and I found 2 motivational 'hooks' into it. The thing is, I don't have a lot of control over these hooks, things can stress me out (which motivates me) or things can make me feel connected (which motivates me). If something hooks into my 'motivational code' and I don't want it to be there then I basically tend to say I'm addicted to: games, YouTube and HN (fun fact: I don't like playing games alone that friends did not introduce to me).

This means my mind needs to figure out on how to exploit this, but also it begs the question: how does this work? I am pretty sure it's emotional and it's a need based thing (self-determination theory definitely comes to mind here).

My question to you is: what are your motivational hooks? How does the environment hook into your motivational code?

BONUS FACTOR!!! (I'm not making fun of marketing people :P )

3. I made a side project for a month called doodledocs.com. It partially came out of motivation factor 2. I was interviewing for a company and they gave the most boring 2 day take home exercise that I saw (make tic tac toe). So I decided to make something awesome instead with their technology, and it took me a month. Just having someone judging my work and me being in the same room as that person helped quite a bit. However, another motivational factor was that I am frustrated! I am frustrated with how not far we are with having a pencil natively on the web. So I made this app.

But frustration is definitely less of a factor than the other 2. Frustration on its own doesn't work, but it does reduce the amount of stress (factor 1) or connection needed (factor 2).

So I guess you could say there are necessary motivational factors and sufficient motivational factors. A bit of stress or a bit of connection is necessary but doesn't make it sufficient. A lot of connection, for example, does make it sufficient. Also, for example, a bit of connection + a lot of frustration does make it sufficient.

I hope my self analysis helped you a bit. And if you're curious about why I wrote this all out in the open and not on my laptop: I am motivated by people reading it and reacting to it. My sense of connection (motivation factor 2) is definitely at play here.

I really like this question. If I'm honest, it rarely happens that I don't finish stuff. But this wasn't some magical switch that happened overnight. Rather, a bunch of stuff, so this question kind of gives me an opportunity to reflect on what led to meaningful change:

1. Action vs Motion. A lot of comments alluded to this here, and the best I've seen it articulated was in James Clear's book Atomic Habits. You have 'Motion' which is thinking, imagining, whiteboarding, planning, talking about...everything that goes around the doing. And you have 'Action', which is actually doing. Motion is incredibly pleasant, but it's a trap. Action is never as much fun. Because soon enough you're going to get uncomfortable, stuck, frustrated. At the very least, recognize when you're in Motion and when you're in Action, so you're not trying to fool yourself. See also Shonda Rhimes' "A screenwriter's advice" [0].

2. The closer you get to finishing something, the greater the resistance is to finishing it. This was articulated in Steven Pressfield's "The War of Art". IT becomes a struggle, and it's not fun, and anyone who tells you otherwise is flat out lying. There is fatigue from the project, and there's also the fact that it's not quite what you had in your head when you dreamed about it. Because you couldn't dream the imperfections. This this is not a fault of implementation (any creator will fess up to feeling this way). So acknowledging all of these and fighting through it is a crucial skill for getting things done.

3. Without a Process, there's not going to be an outcome. I started as a developer, then switched to writer, and now I do both. It's the switching to becoming a writer that taught me how to get stuff done, because I honed the process. I knew what it would take to get a project from start to completion. Different projects take more or less time, but the stages are clear. When I tackle something now, be it in code or in writing, I set up a roadmap. One that I can say definitively for each step, when it's done, and what's next.

4. Domain: it really, REALLY, helps to build up domain expertise in something. The tech is the HOW, but the domain is the WHAT and the WHY. Without a good WHY, odds are the project is doomed. A domain is really about mining more interesting questions, and staying engaged with ideas. So that even when you hit into difficulties, the curiosity about the problem helps you push through. I'm kinda not surprised you'd give up quickly on building a solitaire with Y, cause who cares, right? And the answer is...you don't care. You don't really want a solitaire, because you know that problem's been solved to a good enough extent that it simply doesn't matter. It's hard doing things when you can get yourself to feel they matter. With a domain, this happens less.

5. So another thing writers say a lot is "Show, don't Tell". And at some point in my life I decided to take that mantra seriously. I felt I'm done with SAYING (on my CV and anywhere else) that I can do X, Y, Z. I was going to SHOW it. It just makes life simpler and easier when you can just shrug and say, "Yup, done that, here's the link." So the point here is that even if I'm doing something in order to learn, it HAS to have an outcome that can be offered to others. Whether they like it, or buy into it or whatever is a different question, interesting in its own right. But my view on folders filled with half baked ideas is that it's not good at all. If you've read "The Goal" (if you haven't, I recommend it) - this is is inventory, and it's a liability, not an asset.

6. Saying No. This is REALLY HARD. There is more stuff you'll want to learn than time to learn it. There are more projects you'd like to get done than the time you're awarded in life. And there is genuine pain in declining to do something you really want to do, because you need to focus on something else. But the alternative to this pain is, ultimately, those folders filled with half-baked ideas. Though I am getting better at saying no, I have yet to get better at not feeling the pain.

7. Time management. Everyone has the same 24hrs in the day. The only question is, how much of this time gets wasted. A lot of people I respect don't spend time on social media. Author Michael Lewis has talked about how it drains creativity out of you in small bits, like air going out of tires. I wholly agree. [Right now, with the covid situation, deep work is super hard. People are just dealing with it however way they can.]

[0] https://zenpencils.com/comic/161-shonda-rhimes-a-screenwriters-advice/

I used to be in this position. Your question really resonates with me.

I think I have broken the cycle, but it is hard to be sure of something like this of course. I have certainly been able to make more progress on projects than I have been able to in the past.

I think this has helped me:

- Make a Someday/Maybe list. Capture your ideas and put them on the list. This can just be a text file. Feel free to add implementation notes. Then, when you have time, you can come back and see if the idea still compels you. The examples you give, for example, sound like things I might have on my own "someday/maybe" list.

- Evaluate your ideas/projects. Do they spark interests? What are the reasons why you desire to complete the project? Do you think that desire will carry you through to project completion?

I'll give you some examples of myself:

For a time, I was really unsure what I wanted to do after I became dissatisfied with Ruby. I wanted to keep exploring functional programming, especially pure functional programming. I also really wanted to work with the actor model, and at the time Rust 1.0 was just being released, so I started working on that at the same time.

At the time I played around with a lot of things. But I ended up focusing on learning Haskell, and my goal was to completely work through this book https://haskellbook.com/. I was able to finish this after several "false starts" with Haskell over the years.

Why did I keep working on this project, instead of any of the other myriad shiny things that pop up on HN? Well, a few reasons:

- I really wanted to familiarize myself with pure FP. I had become dissatisfied with OO techniques, slowly thinking of them as an evolutionary "dead end", and wanted to explore something else. IMO, Haskell was the best way to explore this.

- I also really wanted to be able to "read" Haskell. Generally speaking, Haskell has become a sort of "lingua franca" in the functional programming community. You often see snippets using its syntax and read concepts explained using its ideas. When I encounter terms like "typeclasses", I wanted to go from "eyes glossed over" to "comprehension".

There are more reasons, but suffice it to say that I thought that given my long term goals, not knowing Haskell had become a stumbling block. So, I decided to change that, and working through the cited book has worked for me.

I didn't only work on this project. I did take a few breaks here and there as needs must, and interest naturally waxes and wanes, but the motivation held and the project is completed.

Having completed it, I have a few new projects that also have compelling reasons for me to work on them. I have flip flopped between them at different times, but there have been good reasons for doing so, and the reasons have not been because I lost interest, but because of external factors.

- Realize that you will still sometimes just desire to experiment with something for a little bit. That's OK. The other day I downloaded and messed with Electron. It was just something I wanted to try out and satisfy my own curiosity for something random. But you should think of this as being a time-boxed experiment, and not something that you are committing several years of your hacking life to. Unless, of course, it is, but in that case, count the opportunity costs before mentally committing yourself.

-----

Edit:

I forgot to include this link; I think it might help you too. http://blog.fogus.me/2015/11/04/the-100101-method-my-approach-to-open-source/ It has been discussed on HN a few times.

So, I'll take a different tack: Are you interested in finishing projects or are you interested in learning new things? Because those goals aren't exactly 100% aligned.

It sounds like your motivation is more about learning new things. If that's the case, lots of small, half-baked projects can definitely be educational.

You're also mentioning projects that take significant ramp-up time before they might start to get a life of their own.

So, that being said, I have a couple of book recommendations:

If what you want to do is learn software development in a growing way, I highly recommend Mastering Software Technique: https://software-technique.com/. It's all about using bursts of time to learn new things as fast as you can. For your situation, it sounds close to what you've been practicing, with just a tiny bit more process.

If what you want to do is actually build something on the side for people other than yourself (which is _not_ a requirement for learning useful things), then I recommend https://jessicaabel.com/growing-gills/.

For me, I've embraced the non-essential nature of my side projects. I've had quite a few over the years, some large, some small. Being able to put one on the back-burner and come back it later has been very handy. Most of my sideprojects these days are ones I can get to "done" in a day or two, and I have a couple longer-running ones that I pick at off and on. My motivation for a particular project varies, some are meant to scratch a _very_ specific itch:

- https://github.com/yumaikas/dirx was basically done in a late night and a morning - https://idea.junglecoder.com/view/idea/274 I've done 3 times, first in bash, then in Go (in a rather late night, to scratch an itch), and then again in Nim (to learn Nim, and compare it against Go).

Some projects I have are a lot longer running. Both PISC (now on indefinite hiatus) and Tabula Scripta have run a lot longer than my typical project. One quality of both is that they have work at quite a few different levels of detail. Am I tired of parsing work? I can work on prototyping a game in PISC. Do I want to make the performance dramatically better for my artificial Fibonacci benchmark? Great time to break out the Go profiler. Do I want to make it way easier to concatenate strings in a stack-based language? Maybe today I figure out how to write an evalbot. (These are _all_ things I did in PISC at different times).

Tabula Scripta is still early days, I haven't had the time to use it for that sort of experimentation, but it will have a similarly broad set of applications and problems to solve.

And, like other people have said, it definitely helps to do you side projects in a different IDE/Language than your work. And if learning is a big motivator, keeping your projects smaller, or making them the sort of things that's easier to break into smaller parts can be very helpful.