Package Based Development

If I have seen further it is by standing on the shoulders of Giants
-Newton

Thanks to global connectivity and the aforementioned culture of sharing, we're writing less boilerplate and more uniquely useful code.

This is generally considered to be a Good Thing.

Many developers will agree.  Very few haven't embraced the idea of writing only what they must, pulling existing libraries down like crazy from internet repositories, and most of those are working on (relatively) ancient legacy code. Not that there's anything wrong with that.

For some reason though, this logic still sometimes goes out the window when it comes to internal development. We – many organizations at least – continue to live in a culture of monolithic repositories, using code generation to fix repetition (admittedly better than cutting and pasting), and complex interdependencies.

Internal Service Abstraction and Tool Removal

A great use of internal packages is to remove boilerplate requirements from your own developers.  Let's say for example that you've created the absolute best deployment system since sliced bread delivery.  Your devs should be able to access it without knowing anything about it.  If you find yourself writing your own codegen or compilation tools then you're probably a great candidate for using packing instead and giving your devs the benefit of all of the standard ecosystem tools (and reducing onboarding costs).

You can create a basic RESTful JSON-over-HTTP service in any modern language by bringing in a package or two and writing 3-4 lines of code.  Even if you're doing Sanskrit-over-CarrierPigeon, your dev's experience shouldn't change other than bringing in different packages.  At no point should you need to force "don't touch this" boilerplate code onto your developers.  You want them operating with the same speed and freedom as all of the other devs coding at "internet speed".  After all, if you don't, your competitors will.

Service Packages

Most of your packages won't be providing scaffolding, they'll be peeling off complex, potentially messy bits of code into their own safe spaces.  Many of them won't even be considered messy on their own, but could feel glaringly so when presented in line with other projects working a layer or two up, focusing on business logic for example.

The key when writing packages to be reused is to offer up a clean, usable API.  It should be idiomatic for whatever framework you're targeting, and not have any assumptions about the specific project that's using it (even though you almost certainly know what project that will be, and it may only ever be used by that one project).  That's the key that will allow that project to change its own structure without forcing all of these other packages to change to suit.

Packages can - and should - be internally tested to whatever level would be appropriate.  The may be stable for years or may continue to adapt as external services do, adding a layer of insulation between an unstable partner and your own code.  They may even turn in to products or popular open source projects of their own.

Whenever I have to link out to an external service, such as a credit card processor, a marketing analytics provider, or any random service provider, the first thing that I'll do is to look for their SDK in the central package repository for whatever language/framework I'm using at the time.  If its there, and decent, just import it and get on with the real work.

Frequently I find that either its not there or its not what I would consider decent, and I'll end up wrapping either the SDK or the raw network service itself.  I'll almost always do this in a package that will be released into our repository (publicly or privately, depending on the circumstance).  That way if there are any future projects that need to use the service they'll be ready to go, and if there aren't any then I've still done a good job of peeling off the service specific code and putting it out of sight and out of mind.

This may be done for something as simple as taking an otherwise elegant SDK that loves declared exception handling and wrapping it in methods that log and throw our standard runtime exceptions, assuming that (as is usually the case) there's nothing that we could actually do at runtime to fix the situation other than throwing an error.  Its often done when someone publishes an SDK that doesn't feel like the target language too, using under_scores instead of CamelCase or vice versa.  If I'm writing one from scratch its likely woefully incomplete, covering only my target APIs and objects, but the user of the service doesn't need to know that.

The goal is simple: allow the project under development to work at as close to a single abstraction level (often a business one) as possible, offloading all of the messy implementation details where possible.

Packages Provide Most of the Benefits of Microservices

Ignoring the hype, the biggest benefits of microservices for most smaller-than-Netflix companies are that you can rely on services developed by other teams without having to know the devs or even talk to them.  Sound familiar?  That's the global problem that package code repositories have basically solved.  If it can work for everything from REST servers to PDF generators, why not your own internal algorithms or other segmentable software?

Some things really do work better as microservices, but those things are relatively rare and come at the expense of having to deal with microservice deployments.  Especially for smaller teams, being able to build packages and use them with the full versioning capabilities of modern repositories, doing central trunk rebuild/test/deployment cycles, can be incredibly efficient and powerful.  This approach also allows most independent modules to operate at call-stack speeds rather than using a message bus or HTTP communication, and in the rare case that you really do want to have a separate executable running something, the package for that service can easily switch to being a network front-end without anyone having to do more than a rebuild to take advantage of it.

Front Ends Love Packages Too

Or components, or whatever you want to call them.  Need to display a date?  There should probably be a control for that if your framework doesn't already have one, one that understands your projects timezone handling and presentation needs. Even if you just have one that you have to parameterize a certain way consistently, building your own UI package that just does that can really help.

Find yourself cutting and pasting similar code to display a map just the way you want it?  Time for another custom control.  Generally I find that abstracting one usage takes more code (although it often lets me write better code, since I've got a good spot to handle exceptions and other nuances), using it two times breaks even, and after the third time I'm in the black.

Just Do It!

All of this is assuming that you've got a local repository set up, that is - which isn't particularly hard in and of itself, it just needs to be done.  And by local I mean cloud-hosted but secure.  JFrog is a reasonable starting point if you don't have a clue where to look.  And if possible, have it set up to build out from your source control system but that's a topic for another day.  Bottom line, internal repositories and a small sea of stand-alone packages will simplify your systems, reduce your overall code volume, be easier to test, easier to enhance, and allow your developers to focus on what matters most.  Go do it!