Estimating Software Projects: The art of the SWAG, It’s a Modulith, The Worry Police, (missed) Shift Left
For example, I can make a SWAG about upgrading a codebase from Python 2 to Python 3: I’ve done this work myself a bunch, and observed many more porting projects, and so my intuitive guesses about a new codebase tend to be reasonably accurate. I can’t make a good SWAG in the seemingly similar situation of upgrading a Perl 5 codebase to Perl 6: I simply don’t have enough priors to build on.
So what’s happening isn’t a guess: no, you’re using a highly sophisticated neural network that’s been trained on substantial prior experience. When someone asks, “how long with that take?” and an immediate answer springs to mind, that’s the result of some very quick, often unconscious pattern-matching against your prior experience. This is what Daniel Kahneman calls “System 1 thinking”: fast, instinctive, intuitive. (For more, see Thinking Fast And Slow). System 1 can be accurate in some circumstances – including, in my experience, these quick estimates. But only some of them.
Une bonne description du processus mental d’estimation à la volée, et surtout de bons tips quand ne pas en faire !
With this as a background, one can now read “Scaling up the Prime Video audio/video monitoring service and reducing costs by 90% ”, a writeup published by the Amazon Prime Video Streaming Team. They have implemented a functionality which probes frames from the video stream sent to customers, applies tests and recognizes certain artefacts that impair quality, so that they can be corrected.
Amazon being Amazon, they have implemented that as a pile of microservices that interact in an event-driven architecture with lambdas and step functions, in order to shovel data from one S3 bucket into another, running the required analytics on the way. Notification about things found are being put on SNS.
[…]
Unfortunately, the subtitle of the article is “The move from a distributed microservices architecture to a monolith application helped achieve higher scale, resilience, and reduce costs”. This sets wrong expectations and makes it harder to pick up the actual learning.
What the AWS Prime Video team arrived at is most closely described as a Modulith , only that they arrived at it the other way around. Fowler suggests you start Monolith first and then chisel off the components you identify as standalone subservices. They instead used pre-made AWS infrastructure components to build a highly modular prototype, and then identified tightly coupled components and merged their deployment without giving up the modular structure. Fowler himself points to an article by Sam Newman and the book that came from it as a way of doing this.
The result is not really a single-instance monolith, anyway. Components are merged into a single ECS container, but of course ECS will deploy it with the required degree of parallelism, and it is still part of a larger system that communicates in an event-driven architecture using lambdas and messages.
So if this article is anything, it is an example of the benefits good observability has, and how a good architecture allows you to rearrange the layout of well-isolated and structured components with clean interfaces at will. The application can be made to run on different substrates, in order to react to different demands of scale or changing business requirements.
Une autre perspective intéressante sur cet article d’amazon Prime Video, un focus bien plus archi que technologique !
The Worry Police might sound engaged with their questions, but they aren’t. They’re worried. They believe this continued incessant worry will somehow miraculously guide this project in the correct direction, and sometimes they’ll get lucky. The Worry Police don’t have hope because they haven’t been rewarded for hope, for optimism.
[…]
The Worry Police exist for a reason you need to understand. They were granted power because the perception was they added value. There is a cultural reason they exist at your company.
The Worry Police, you might infer, aptly describes a specific job, but the Worry Police is neither a job nor a role, but a learned attitude. The Worry Police might read like a gross exaggeration, but they’re not. They’re in the room right now with you. Worrying.
Un article plutôt descriptif sur un modèle mental qu’on retrouve facilement dans nos boulots, et un modèle cible qui permet d’avancer (mais rien sur le chemin à parcourir entre deux !)
Shift left! Shift left! They say. This is a common refrain when people are talking about DevOps. Deming taught that we have to build quality in, we cannot inspect quality in at the end, so we need to shift left.
So teams set off to shift left, not sure exactly what this means, but often it’s approached as though it will mean more of their work will be given to the developers. They will finally understand what we go through! They will finally learn about our jobs!
Un article court qui pose des mots sur la stratégie Shift Left, et des préconditions pour y parvenir.
