I agree, that played a large part and is also part of why, once I did finally have legit gear and blasted and cruised I netted 12lb that year. The following year though (final time on gear) I really only netted a few more pounds on top of that. It's similar to noob training gains, you will get great results the first year or two that things are done correctly and then things will largely taper off.
Genetics are a huge part of that, as well as other factors like you mentioned. I have always been a good athlete and strong for my size but in terms of muscle growth I am probably below average so it will depend on the individual. My main points here are 1.) Don't expect anything insane unless you use AAS for many years, and often other things like GH, insulin, etc are needed if you want to get freaky and 2.) more importantly, all the health aspects I touched on.
This is not completely true - there are many aspects to keep growth coming. Most of which beginners don't understand. You obviously can't grow linearly, it just isn't possible.
But how much did you really know about compound rotation? Knowing when to lower doses, when and how to increase doses and add new compounds, knowing when your insulin sensitivity was going out of line, when to cut weight to trigger more growth etc.
Unfortunately this takes years of work and can be different for each individual. This can be reduced with a good trainer etc but if you use properly with the right instruction you can make significant year to year changes for a number of years before you plateau/don't want to up the gear any more.