Most data engineering managers learned technical skills, not management. The transition from senior individual contributor to manager of a 30-person team is non-obvious, poorly documented in the literature, and frequently painful for everyone involved. This paper is a personal case study of that transition, documenting lessons learned over several years of growing and restructuring a data engineering organization across 6 business units and three time zones in banking, insurance, and manufacturing environments. The paper addresses three organizational questions: what organizational structures enable effective management and individual contributor growth in data engineering teams of 30 to 50 people; how hiring practices, conflict resolution, and promotion transparency correlate with retention and performance; and at what team size particular management structures break and what the warning signs are. The honest contribution is the failures: starting out of depth at age 38, two bad hires that took nine months to exit, a span-of-control crisis that required restructuring from eight direct reports to five, and a technical disagreement that turned out to be old promotion resentment. The thesis is that management is a learned skill rather than an innate talent, that the patterns that work at 10 people break at 30 and the patterns that work at 30 break at 100, and that organizations whose engineering managers learn these lessons fastest produce substantially better outcomes.

Returning to the three organizational questions: (RQ1) Organizational structures that enable effective management and individual contributor growth in 30-to-50-person teams are matrix or domain-aligned structures with team leads handling day-to-day coaching, a small central platform squad maintaining shared infrastructure, and explicit career layers for the engineers. Functional structures break above about eight directs. (RQ2) Hiring practices that screen for values and collaboration as rigorously as for technical skill, firing practices that are documented and respectful, and promotion practices that are transparent and predictable correlate strongly with retention and performance. The opposite practices correlate with the bad outcomes documented earlier. (RQ3) Functional structure breaks above approximately eight direct reports. Matrix structure breaks when accountability becomes ambiguous. Domain-aligned structure breaks when the domain teams lack engineering maturity. The warning signs are usually visible months before the structure formally fails: missed one-on-ones, repeated conflicts that do not resolve, attrition concentrated in one part of the team, and the manager’s own sense of being perpetually behind. The closing observation is that most of what was learned about management was learned by getting it wrong first. The two bad hires, the span-of-control crisis, the Kafka-vs-Pulsar conflict, the early months of feeling out of depth—these are not embarrassing details to be hidden in a retrospective; they are the curriculum. The managers who get good at this are the ones who pay attention to their failures and update their practice in response. The managers who do not are the ones whose teams suffer. The learning continues because the team keeps changing and the problems change with it.

[1] T. DeMarco and T. Lister, Peopleware: Productive Projects and Teams, 3rd ed. Addison-Wesley, 2013. [2] F. P. Brooks, The Mythical Man-Month, Anniversary Edition. Addison-Wesley, 1995. [3] P. F. Drucker, The Effective Executive. Harper Business, 2006. [4] A. S. Grove, High Output Management. Vintage, 1995. [5] C. Fournier, The Manager’s Path. O’Reilly Media, 2017. [6] W. Larson, An Elegant Puzzle: Systems of Engineering Management. Stripe Press, 2019. [7] N. Forsgren, J. Humble, and G. Kim, Accelerate: The Science of Lean Software and DevOps. IT Revolution Press, 2018. [8] N. Forsgren, M.-A. Storey, C. Maddila, T. Zimmermann, B. Houck, and J. Butler, “The SPACE of Developer Productivity,” Communications of the ACM, vol. 64, no. 6, 2021. [9] B. Beyer, C. Jones, J. Petoff, and N. R. Murphy, Eds., Site Reliability Engineering. O’Reilly Media, 2016. [10] A. C. Edmondson, “Psychological safety and learning behavior in work teams,” Administrative Science Quarterly, vol. 44, no. 2, 1999. [11] D. Coyle, The Culture Code: The Secrets of Highly Successful Groups. Bantam, 2018. [12] P. Lencioni, The Five Dysfunctions of a Team. Jossey-Bass, 2002. [13] M. Armbrust et al., “Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics,” in Proceedings of CIDR 2021, 2021.