Engineering · @jnjosh’s internet weblog

Why Monorepos?

Thu, 06 Oct 2022 23:27:17 -0400

I really like the idea of “asking a lot of dumb questions to learn fast” and while I don’t think this is a dumb question it’s kind of dumb in that it’s a controversial question on the internet. Maybe I can learn something at least? So here it goes.

Why are so many developers so smitten with monorepos?

What’s a monorepo? From Wikipedia, it’s a “software development strategy where code for many projects is stored in the same repository”. All your different apps, libraries, and utilities are in one large repository, a “shared codebase”. As you can imagine stashing a bunch of disparate projects into one large repository leads to some problems, namely size. This leads to tools like Rush for Javascript developers to help manage the monorepo. A lot of large tech companies, including those I’ve worked with use this. Continuing on Wikipedia we see some benefits:

Ease of code reuse - when all the code is in one big directory you can just reference anyone else’s code (libraries ore even projects) without needing any sort of dependency management or semantic versioning.
Simplified dependency management - Since many projects in the monorepo may depend on similar third-party libraries, they can be optimized and provided consistently by the monorepo.
Atomic commits - When you need to make a large change affecting multiple projects and libraries, you can push it all together without chaining releases. For example, a model change can be made in a library, the frontend, and the backend all in one pull request.
Large-scale code refactoring - Similarly to atomic commits, you can do large refactors across multiple projects in the course of one pull request—making sure everything continues to work as expected after the pull request is merged.
Collaboration across teams - With all projects, owned by different teams, the code ownership model is much more flexible. You can contribute to and receive contributions from other teams.

I’ve spent most of my career working on native software (iOS, macOS, Windows, etc), but did do some web development here and there. I say that because I think I’ve mostly—but not exclusively—seen monorepos coming from web development teams. Maybe that’s why this is such a strange idea to me? I know it’s not quite this bad but it reminds me of development before adopting version control systems where we’d just store the code on a network drive—everything in one directory is easier!

That’s a bit extreme and when I think about this more critically there are some clear benefits, namely atomic commits. One team I worked on we built a distributed codebase where a rich model layer was in it’s own repository. In the early days when a team owned that project, it was great. Contributions were open, but the team aimed to work with the frontend team to make sure their needs were met by the time it was needed. No silos, everyone talked! As things change, that project became ownerless and people just started making changes—forced to do a semantic versioned release for every change. This would be fine if there were many consumers of that library, but there was one client app.

This was a design problem that multiple repos highlighted. A design problem that would have made more sense for it to start in the repository of the client application, and only moved to a separate repo if more applications started using the project. So for a single application, even heavily factored into core components and libraries, I think I’m newly open to that just being one repo. You can still maintain code ownership with Github features, and even publish semantic versions if you want, but can take the benefit of those atomic commits.

Maybe that is my core struggle with this idea. The above project was all one iOS application written in Swift with the occasional Objective-C. Should there be another iOS app in that repo? Should the Android app have been in that repo too? I don’t think so but maybe that’s not just about multiple applications, but also multiple technologies all in one large repo. Maybe this is why web folks like it, when everything is Javascript why draw boundaries!?

Also, what about the rest of those benefits? Anyone can change anything? So you save the effort of managing dependencies by opening up the risk of uncontrolled changes. Friction can be good to a software project. If I look at those benefits again, I could probably talk to why it’s good that you can’t do that or why it’s already possible without putting all your code in one repo. Let’s do it!

Ease of code reuse - It is easier if all the code out there is easily accessible, but is that really what you want? That seems like a very stack overflow copy paste solution to just access any solution provided in the monorepo. I’d rather build on a solid foundation, even at the cost of velocity. You make up for it when the worst happens because you have clearer boundaries to debug.

Simplified dependency management - It is simpler to not have to think about versions, but is that better? Package dependencies are still required for third party tooling so you aren’t avoiding it and at best you are losing out on meaningful semantic versioning allowing you to rollback dependencies easily without having to manage all the commits of an entire repository. Additionally a good framework team can predict needs and work with you to best provide APIs that solve more than just your needs.

Atomic commits - As I noted above, this is a benefit I can get behind, but I don’t know that it’s always what you want in a distributed set of apps and libraries. Friction is good and not everything needs to be hacked together in the same pull request. It can be fine to build the foundations in one library at a specific version, then add support in the app in another repo. I do understand that it’s hard when teams are working with fewer people than they need and there may not be anyone to own that library you need to contribute to.

Large-scale code refactoring - If your project is so coupled that you need to do such large scale refactoring where this is valuable, I wonder if your problem is not the repo but the codebase. Code is a liability, not an asset. I’d rather have a heavily factored set of projects that have clear protocols and when one isn’t meeting needs, replace it. This sounds to me like everything is so interconnected that it’s hard to reason about, so you need to have it all in one folder so you can just unravel the spaghetti.

Collaboration across teams - Multiple repos already provide this sort of collaboration in a more sustainable and arguably responsible way—you just contribute to their repository. I know I know, there is overhead in that in time and velocity, but really shouldn’t we be aiming for sustainability? I know in a monorepo you can still use code ownership to force a team to be required to review pull requests, but that’s not always great. One large pull request could end up requiring dozens of code reviews because you’ve touched multiple teams’s owned code. I know that’s the same with multiple repos, but that’s my point.

I think there is an artificial feeling to this that makes me feel like it’s about perceived speed and velocity, not good software organization. In many of these cases it seems to aim to remove all barriers to any contribution instead of building good foundations.

And I’m not even getting into the massively large size of these monorepos.

So… with all that, I’m starting to come around on it to a point. I do think having all applications in one repo is not worth the tradeoffs. However, I do understand why a large web app would want to be a monorepo with all libraries accessible in the repo—it’s all one stack. I think I also understand a platform as a monorepo as well, those scenarios where the application stack is a platform and apps are built on that platform.

There is a certain reality to it that I understand. Teams and organizations change. Ownership and focus changes. When there is no team to own something, it can very quickly lead to a library or project getting duplicated instead of made a true dependency. These realities open my mind up to it a bit more.

I read somewhere that “Understanding requires effortful engagement” and I think I just did this exercise here. I’m not convinced this is the best path for an organization, but I think I understand it more and maybe would choose to group more projects together—especially if they are all in the goal of shipping a single application in a single technology stack. What do you think? I’d love to continue thinking through this and if you have experiences that talk to monorepos, I’d love to hear. Let’s talk on twitter.

Cross Platform Thoughts

Sun, 18 Aug 2019 12:27:06 -0500

Cross Platform development for most mobile apps is just simply not worth it. Sure, there are great open source options like React Native and just plain C++ but you end up spending the same time trying to be clever and “share code” when you could have just built your product with all the platform has to offer. There seems to be a lot of realizations this year with Dropbox dropping their C++ components and last year Airbnb sunsetting React Native components. I’ve played this game before and while it was a great challenge, I would advise anyone not to do it for most cases.

Back in 2017 I joined the Highrise team as the Lead Mobile Engineer and one of the first decisions we needed to make was: what is the future of our game and how could we create it so that it could live on Android?

Highrise is unique in that it’s both an app and a game. We quickly explored some cross platform options for making the game components like Unreal, Unity, libGDX, etc. These options seemed great for the parts of the app that were a game but not so great for the standard app components. We’d end up having “crappy Game UI”™ which was explicitly something we did not want. We wanted the app to feel as good as Instagram or any Apple app when not in game-based UIs. I proposed the best path as either “let’s not be clever and just make the app twice” or “let’s build the game components in a shared C++ layer and the app components natively”. I had done this previously when working on BA3’s mapping engine and felt this could be a great solution for our problem. We could write the hard game-based code once and write specific UIs for each platform. I shared this at local meetups as cross platform shenanigans.

You know what? It was actually pretty fun! We ended up with an extra platform, C++, and a lot of bridging code in Objective-C++ (and JNI-based C for Android) and had to make a lot of calls on how we’d communicate game events between the UI and core layers. We worked with Flatbuffers which gave us simple data to pass around. We now had a platform that we could build on top of, we had a core layer that was compatible with multiple platforms, and we had a UI layer that would not be foreign to most iOS or Android developers. I’d even go as far as saying if you are building a game, this is probably the best way to get cross platform game logic while not sacrificing a great app experience.

Would I do it again? No.

The thing is we really didn’t save much time. Most features ended up being full stack features so we couldn’t isolate a developer from the Core as much as we wanted. This meant that we had to expose bridge objects going both ways—which is not a simple problem—for most features. We also had to deal with many layers of abstraction making things like simple networking much harder to debug. In the end, we spent about the same amount of time as if we had just built the app for each platform. We could have even saved some time when you consider we could us one platform as a guide to build the other platform and not fight issues that worked well in one platform and not on the other.

One of the first things I teach new developers is “don’t fight the platform”. Instead of trying to be clever and solving your too many platforms problem by adding another platform, just build your app simply. Build in Swift with all that Apple offers or build in Kotlin with all that Google offers. You’ll appreciate the simplicity of it when you have to debug any issue. You may have to think a little bit harder to coordinate releases between the platforms, but that is a far simpler problem than trying to build something that works great on whatever platform it runs on. I’ve since moved on from Highrise and BA3, but one lesson I’ll take with me is don’t try to be too clever.

Debugging with Sound

Thu, 08 Feb 2018 22:27:06 -0500

Debugging is hard. You have to remember what you or someone else was trying to do while understanding what’s actually going on. I don’t remember where I heard it, but there is a saying that goes something like “Don’t write code at the peak of your understanding because debugging it requires 2 times the understanding…”. Sometimes it’s not that it’s hard it’s that there is just too much going on. For these cases, I’ve been using sound.

One case where this comes in handy is debugging a network call and cache hit. Imagine you are building a system that checks an in-memory cache, an on-disk cache, and a download operation. You’ve probably done this several times. Debugging this is hard because you really don’t want to stop, you want to know that it’s hitting all the different points appropriately. Sure logging is fine here, but that’s prone to confusion if you have a noisy log. What I find helpful when debugging things like this—or any type of retain/release pattern—is to add a sound-based break point.

Now I don’t have to study and read, I just use the app and listen. If I hear a Ping, a Funk, and a Frog sound when I’m expecting it—I know I’m on the right track. The great thing is you also don’t have to stop the debug session, you can simply add these while debugging.

I think this is a great tool and it’s kind of fun! I don’t use it all the time but every once in a while I find something that fits this debugging tool well. Then I can listen to my code instead of reading it.

Number Rush

Sun, 06 Mar 2016 13:38:49 -0500

Apparently I’m not the only one named “Josh Johnson” that has a problem with German numbers. Here is a web-based game by another Josh Johnson to help with your German number learning. It also uses React.js which is something I’ve been wanting to play with. You can see the code on Github.

Permalink

Guide to Functional Reactive Programming

Tue, 01 Mar 2016 10:43:07 -0500

Realm, which posts amazing videos regularly, has just posted a Guide to Functional Reactive Programming. They discuss the why, the what, the how, and the who all while linking to some great videos. Definitely a nice resource if you haven’t looked into this. I’ve read a lot about it but I don’t think I’ve fully internalized this yet. This will definitely help.

Permalink

Coding Weekend

Mon, 29 Feb 2016 21:57:12 -0500

This weekend I spent some time at the Ticketmaster DevJam in Durham. They are working on a public API with events, venues, and more that they’d like to get developers using. For me, it was a day long event with food and building a weird app with some friends.

The basic idea of the event was for Ticketmaster to get eyes on their new APIs and find things wrong with it. They’d like to get people giving feedback and building things. Several groups broke up and build some cool things and demo them at the end of the day.

So… what did I build? I worked with Ticketmaster Mobile Studio developer, Peter Foti to build something that the world really needs. The “Justin Bieber Warning System”.

Events with Justin Bieber

We pretty quickly mixed the data coming from the Ticketmaster Discovery API with the mapping engine we are building at BA3. By finding the specific ID of Justin Bieber, we were able to generate a list of events and their venue’s location. Now you can know what areas you can avoid this pop star.

We are not Beliebers. There are few places to avoid him in North America.

There were some more serious demos too. One team built began building IFTT integration so that you can connect triggers based on different discovery events. Another team built a Slack plugin.

Overall, it was a pretty fun weekend and I got to hang out with some friends while building comedic software.

Microsoft Acquires Xamarin

Wed, 24 Feb 2016 14:32:19 -0500

Interesting, and unsurprising, news today that Xamarin has been acquired by Microsoft. I’ve never used it directly to build apps but I did to some research into using their test cloud system (which is actually calabash-based, something I’ve worked a lot with and have written about before). I hear of a lot of people loving to be able to build iOS apps without learning Objective-C or Swift.

This gives Microsoft an interesting position as a vendor of an iOS SDK. It will be interesting to see what comes of this.

Permalink

My Developer Tools and Utilities List (2014 Edition)

Sun, 30 Nov 2014 01:37:32 -0500

I’ve often wanted to do a yearly roundup of the tools and utilities I use; if only as a log of things I use over time. I did this two years ago in 2012 and when I was thinking about getting back to blogging I wanted to update the list. The funny thing is not much has changed. I’m actually surprised that I don’t use that many tools.

Hardware

Retina MacBook Pro - Still working with my 2012 Retina MacBook Pro with 2.3 GHz i7, 16 GB RAM, and 256gb SSD running OS X 10.10 Yosemite.
27" Thunderbolt Display - Still love this monitor. After a lot of time using the MacBook Pro in clamshell mode, I’ve been using it as a second screen. It makes it much easier to disconnect.
KBC Poker 2 Mechanical Keyboard - Last year I made the switch to using a mechanical keyboard. I love this little thing. Mine has Cherry MX Blue Switches and custom Dvorak keys.
Apple Magic Mouse - I’ve tried many mice. This is not my favorite, but it allows for some of OS X’s gestures. I think I’m going to attempt switching to tha trackpad this year.
iPhone 6 and iPad mini Retina - These are quite obvious, but I use them every day for daily use and development.
Timbuk2 D-Lux Laptop Messenger - I’ve switched to the messenger version of the backpack I used two years ago. I really like this bag but still love the backpack too. I’ll probably switch again in the future.

Software

Developer Tools

Xcode 6 - I write software for iOS and Mac, this is the tool I use every day. Instruments is another app, but I’ll consider it part of Xcode.
BBEdit - I switched to using BBEdit. I’ve tried it several times in the past, but was convinced to try it again. Now it’s part of my daily tools.
Kaleidoscope 2 - Whenever I push new commits to Github, I check them out first in Kaleidoscope first.
Transmit - Still the best tool for uploading via FTP or to S3.
Dash - Update: Can’t believe I forgot this! Dash is the best tool for reading documentation. I use it so much it was really easy to forget. It feels like it is a part of the system.

Other Software and Utilities

Alfred - Even though Spotlight in Yosemite replaces most of what I use Alfred for, I still like using Alfred more.
Divvy - I still use divvy every day. It’s a great tool and allows me to organize my screen really quickly.
CloudApp - I’m often sharing links, photos or files with people on the internet. Transmitting these files used to be a huge problem. Now it is easy with CloudApp. I simply hit my keyboard shortcut on a file or URL and it gets uploaded and a short link is created. I have ‘jsh.in’ mapped to it so all my links look like mine.
Reeder - I have a huge RSS feed of many Mac and iOS development blogs and reeder has fit as my favorite reeder. I actually stopped using it for a little while but then switched back.
Clock.app - I still keep the clock off my menu bar and use the open source Clock.app.

Implementing a language with LLVM

Fri, 01 Feb 2013 23:30:32 -0500

Writing a programming language is not something I’d ever attempt to accomplish. However, the idea of getting a better understanding of LLVM is very interesting. This tutorial isn’t a complete lesson in the compiler, but I think it will at least help you learn a little more about it..

“This tutorial runs through the implementation of a simple language, showing how fun and easy it can be.”

Sounds fun. I’m in.

Permalink

My Developer Tools and Utilities List (2012 Edition)

Sat, 22 Dec 2012 21:11:32 -0500

I’ve often wanted to do a yearly roundup of the tools and utilities I use; if only as a log of things I use over time. I’ve seen this idea come up often on other blogs, so this is an idea borrowed from other developers on the internet. Anyway, the following is a list of hardware and software I use on a regular basis.

Hardware

Retina MacBook Pro - with a 2.3 GHz i7, 16 GB RAM, and 256gb SSD. It is probably the best computer I’ve ever used. Often, I’d rather work on this display than use the thunderbolt display.
27" Thunderbolt Display - Really great monitor that I use when at the office. Before going to the Retina MacBook, I used the laptop as a second monitor, but the difference between the display and the retina screen is too annoying. So now when at work, I run in clamshell mode and only work off the thunderbolt display.
BookArc Pro - Since I run in closed mode at work, I like to set the MacBook up on this nice stand. It keeps my desk clean and looks really cool.
Apple Wireless Keyboard - I’m not sure if this is the best keyboard in the world, but I like it a lot. Aside from being small and wireless, It more closely matches the keyboard on the laptop so I don’t have to fight to remember where the control keys are. Also, this year I switched to the Dvorak keyboard and the low height of the keys made it easy to learn rapidly.
Apple Magic Mouse - I’ve tried many mice. This is not my favorite, but it allows for some of OS X’s gestures.
iPhone 5 and 3rd Generation iPad - These are quite obvious, but I use them every day for daily use and development.
Timbuk2 Q Backpack - Not really hardware, but important for carrying around that Retina MacBook. I’m very picky on the bag I use. I often change bags on a whim. This bag has been a great one though, I haven’t had that feeling to change bags yet!

Software

Developer Tools

Xcode - I write software for iOS and Mac, this is the tool I use every day. It can be frustrating, but I think it is improving greatly. Instruments is another app, but I’ll consider it part of Xcode.
Sublime Text 2 - I’ve been jumping around with text editors a lot. Finally, I stopped on Sublime Text. It is a great editor that hasn’t seemed to explode on me yet. It crashes less than Xcode.
Photoshop CS6 - I occasionally try to design things and often need to cut some assets. I’ve been using Photoshop since the late 90s and it is just my comfort zone for design.
AppViz 2 - I finally have returned to the app store and have an app of my own on the store. This is a great tool to see how your app is performing on the store. If you care about reviews, it also gives you a great way to view them.
Charles - Occasionally, I like to verify that my network requests are doing what I expect, or like to snoop on other apps network use. Charles is the app for this, even with it’s non-native interface.
CodeRunner - Sometimes when learning a new language or verifying something I want to do in an app, I’ll pop into CodeRunner to test the snippet of code out. Probably the most common thing is testing KVC collection operations.
Kaleidoscope 2 - I’ve been using this app since version 1 as a git difftool. Version 2 adds merge support. Great tool.
RESTed - When working with a remote API it is great to be able to create and store requests without having to fumble with cURL. Don’t get me wrong, I’m a terminal guy but I can’t ever seem to remember all the different options in cURL. RESTed solves this for me and you can save the requests for later.
Transmit - Hands down the best tool for uploading via FTP or to S3.
xScope - Great tool for being a pixel perfect snob. xScope has a lot of tools, I probably use the horizontal guides to make sure things are lined up as I expect.
Marked - We use markdown a lot at Two Toasters for functional specs to blogging. I blog here in markdown. Marked is a great tool I use all the time to view these files.
Terminal - I can’t have this list without including the terminal. It is the first application I open when starting to work. I use oh-my-zsh to customize the shell and do all my git versioning from the shell. It is the only way! :-)

Other Software and Utilities

Alfred - My favorite of all the quick launch style tools out there. I love it for it’s speed and simplicity. Most of the time I live on the keyboard so quick launching an app with Command+Space is really important. I realize I can do this with spotlight but I enjoy the way Alfred works. Also, it is my quick calculator.
Divvy - I use divvy every day to position windows from the keyboard.
CloudApp - I’m often sharing links, photos or files with people on the internet. Transmitting these files used to be a huge problem. Now it is easy with CloudApp. I simply hit my keyboard shortcut on a file or URL and it gets uploaded and a short link is created. I have ‘jsh.in’ mapped to it so all my links look like mine.
Sparrow - I worry about this app. For some time it has been a great replacement for Mail.app. I still use it as I like the way it feels over the Apple supplied application. This could change though as Sparrow was bought by Google and there doesn’t appear to be a plan to continue development. For now, I use it.
Tweetbot for Mac - I’m often on Twitter reading what others are chatting about and occasionally responding. Tweetbot has been the best interface since the old Tweetie for me.
Evernote - I’ve recently started noting things much more. I used to believe I could remember everything, but it isn’t true. Often I remember that I knew something about a topic, but can’t remember the actual answer. This is what I’m using evernote for. I like to think of it as a hash table, my memory has the key and the value is in evernote. I’ve also started going back to WWDC videos and making notes of each session.
Reeder - I have a huge RSS feed of many Mac and iOS development blogs and reeder has fit as my favorite reeder.
Clock.app - This is an open source project that shows the date and time in a nice HUD window. I don’t like to keep my clock on the screen at all times so this provides a nice quick way to see the time if I need to. With alfred I can hit Cmd+Space -> ‘C’ -> enter and see the time really fast.

Automated Acceptance Testing or: How I stopped worrying and trusted the tests

Sun, 05 Aug 2012 02:15:32 -0500

This month at Triangle CocoaHeads in Durham, I stepped away from being just the organizer and did a talk on automated acceptance testing with Calabash. This is a topic I’m really excited to learn more about and am constantly trying to become a better tester.

I first learned about Calabash when seeing a video of one of the blitz talks from NSConference about it. I started using it on one project and it seemed to do a nice job.

I think the talk went over pretty well, and since I record the presentations from CocoaHeads you can see for yourself:

You can also download the presentation as a keynote file.

Getting kids excited about coding

Sat, 03 Jul 2010 01:31:32 -0500

Want to get your kids excited about coding? Think about looking into arduino. My fashion design and meteorology loving daughter was having a great time and even said “coding is fun!”.