About the series
This series is an initiative for the engineering community to learn, share and grow together in a journey of professional and personal excellence with the spirit of 10X. We will share this via blogs, references, masterclasses, live streaming and podcasts. It will be done with partnership of friends and community. This is an evolving and live content.
Subscribe to our newsletter for updates.
About me
Hi you may call me Ayush. I am the author and the facilitator of this series. Here I will take you through a journey of exploration of the wonderful and exciting world of technology.
For me engineering is a craft I enjoy very much. I also wear other hats like a founder, innovator, teacher, mentor, family, friend and of course eventually everything human! :-)
By virtue of intention, effort and destiny's touch, I landed in IIT Kanpur in 2002 to study engineering. And by stroke of destiny, I landed in Computer Science department while I wanted to originally take up aeronautics. That's another story. I will be sharing more of my journey through stories as we progress in this series. But here I want so state that couple of times in my lifetime, I considered moving out of tech to pursue other callings like MBA. I am so glad I choose to stay invested in tech. It continues to be my way of expression, innovation and service to the world. I built or continue to build somethings I feel very satisfied about.
For me tech is my ikigai. It does four things for me: I love to play with it. It earns me livelihood. It helps me grow. It is also my means to serve the society.
I founded Godspeed for the mission to democratise technology for the tech ecosystem in a collaborative manner. This series is as much part of the mission as is the tools that we make for the same purpose.
I am starting this series to share what I have learned with the world. Here I will focus on the path of tech, its purpose and engineering.
I will be maintaining the content of this particular series and run this program called 10X Engineer, aimed primarily for tech professionals, students and innovators, who are equally desirous of learning and growing in their journey.
About the series: 10X Engineering
At Godspeed Systems, we believe in excellence and want every engineer to be a 10X version of himself or herself. For the same every engineer needs to
Have a certain mindset called the 10X mindset
Continuously develop understanding, information and knowhow
Keep on to play, serve, learn and earn
Hence we will be running this series where we share with the world whatever little we know or are making, and also continue to learn and upgrade ourselves. Consider 10X Engineering not as a course but a guide designed to empower developers and teams at every stage, providing a systemic approach to thinking, solution design and application.
The choice of tools for the journey
The path to 10X is independent of tools. Hence you are free to choose tools that suit your purpose.
What makes the difference between excellent and poor engineering (and engineer) is not the tools we know or use. It is design and application using fundamentals and first principles. The evidence of excellence or success lies in the final output that we ship from day to day, as a team. The job of tools is to help humans make stuff excellently. So a good engineer or team is strong on concepts, and can apply those fast and properly with any tools.
A 10X engineer is like a really good athlete who can run well with any shoes. A novice athlete can not run well with even the best shoes.
In this series, there will be a recommendation from our side but you are indeed welcome to pick any toys you wish to play the game with. We have the freedom to choose and switch between the tools that serve our purpose.
Topics every dev should know
These are some of the topics which I believe every engineer must know, from a little bit to expert level knowledge.
The chapters
Chapter 0: Making of a 10X Engineer & team (Published)
What is Software Engineering
Types of Careers and Roles in Software Engineering
Chapter 1 - Touching the fundamentals (Published #wip)
Pre-requisite
Knowing atleast one language
JS/TS (preferred for full stack aspirants)
Or Python, Java, Golang etc.
Knowing basic datastructures and algos
Software Engineering Lifecycle and environments
Data
What is data
The four types of data
Software is all about playing with data
- Retrieve/receive, compute, transform and send/store data
What is business logic and what is boilerplate
Software Testing and its importance
Single bad UX can be more powerful than 99 good features combined
Unit, integration and functional testing
Basic Unix commands
Best practices
Coding guidelines
Schema driven development
Configure over code
First principles based approach
Shift left approach
Continuous Integration and Continuous Deployment
Decoupling
Project management and teamwork - best practices
Version Control and Git
Chapter 2 - Programming advanced
Paradigm of programming languages
The four generations of programming languages
All about P and NP
Types of programming languages
- Procedural, Object Oriented, Functional, Aspect Oriented
When to use which languages?
Concurrent data access with mutable and immutable data structures.
Programming - best practices
The art of handling errors and debugging
Eating versus throwing the exceptions
How to inform the user or how to log (best practices)
- Error code, description, context, and tips for resolution
How to debug
Locally
Reading logs & stack trace with eye for detail
Step-through debugging in the IDE
On hosted environments
Using monitoring tools like Grafana, CloudWatch, ELK, Datadog etc.
What are distributed traces and spans?
Alerting
- Pre-emptive and post-error
Let’s write some Unit Tests
Chapter 3 - Modern software architecture - basics
Evolution of software architecture
Monolithic
3 tier
N tier
Service Oriented
Microservices
Serverless
Event driven architecture
Server-Client architecture
Three pillars of any microservice/service
- Eventsources, datasources (APIs, datastores) and workflows/functions
Eventsources and events
Two kinds of events
Sync
- All kinds of APIs whether REST, Graphql, gRpc, SOAP etc
Async - Message Bus, Socket and CRON
API protocols
What is an API schema
What is Schema Driven Development and its benefits
Three main modern standards
REST
Graphql
gRpc
Two kinds of Datasources
- Datastores and APIs (other services)
3rd gen Service development frameworks and ecosystem
- Nodejs, Springboot, Gomicro, Django etc.
4th gen Meta-frameworks for service development
- Godspeed
On the UI side:
Web and native mobile apps
HTML/CSS/JS
Frameworks like React & Angular
Strategies for optimising site load time
Web sockets - what it is, its benefits and real world use cases
Develop some APIs and basic UI
Writing Integration and UI tests for APIs
Chai/Mocha in JS or Robot in Python etc.
Selenium versus no code UI test frameworks
Chapter 4 - The world of data and datastores
Types of datastores
SQL (row based typically used for OLTP, column based popular for OLAP)
NoSQL (key-value, document, graph)
Important aspects of the datastore domain
Model (entities, types, relationships), ACID properties, indexes, constraints, querying, joins, materialized views, normalization/denormalization
Replication and sharding
Popular datastores
- SQL (Postgres, Mysql), Document (Mongodb, Elasticsearch), Search Engine (Elasticsearch), Cache (Redis, Memcached, in Memory), Analytics (Clickhouse, Duckdb, Elasticsearch), Graph (Neo4j, DGraph) , Vector (Elasticsearch)
Search and filtering: Text based and vector based
Caching
Types
OS
In memory
Browser
CDN
Gateway
Caching service
Cache eviction policies
Data warehouse and data lakes
Migration
Model
Model with data
Storage (location) migration
Deep dive: Postgres/Mysql (SQL) and/or Mongodb/Elasticsearch (Nosql)
What is Data federation and its use cases
Popularity of Graphql
Backend for frontend design pattern (BFF)
Dual writes and consistency (real time versus eventual)
About Primary and secondary datastores
When is it needed?
Syncing mechanisms
Mutation subscriptions (via CDC, Graphql, message bus)
Lazy loading versus write through cache
Data Pipelines - ETL and ELT
Subscribing to filtered events: Kafka with KSQL
Chapter 5: Security
Layers of security
- Perimeter, network, endpoint, application, data
User Authentication
- JWT, Oauth2, Simple Auth
Authorization for endpoint and data access
RBAC and ABAC
actor, resource, action, context
Access Control and Policy engines
Application security
Secret management and vaults
Scans (code, image, dependencies)
Data security
Encryption of data at rest and transit
A brief history
Deterministic and non-deterministic encryption
One way and two way encryption
Encryption algorithms
Redaction of sensitive data from logs
Chapter 6: Performance profiling & debugging
Load Profiling using Jmeter or similar tool
Measuring latency (99P, 90P, 50P)
CPU and RAM usage
Debugging performance bottlenecks
- Database, API calls, Code (CPU and mem profiles)
Detecting Memory leaks
DEPLOYMENTS
Chapter 7: OS and networking
Hardware: CPU, RAM, DISK
OS - Unix (Linux, BSD and Mac), Windows
Concepts: Caching, swapping, threading/cores, process, concurrency (locking, counters etc)
Networking (IP address and port), Firewall, Proxy,
Unix basic commands recap
Chapter 8: Containerization and scaling strategies
What is containerization? What is its history and modern trends?
When should it be used?
Tools: Docker, Kubernetes
Chapter 9: Continuous integration & Deployment
What is CI/CD and its benefit in Shift Left Approach of Software Development
Role of the CI process and CD process
CI essentials - automation testing, security scans, containerisation, storing in artifactory
CD - Deployment strategies - Blue Green and Canary
What is infrastructure as code.
Popular tools: Terraform, Ansible, ArgoCD
Tool: ArgoCD: setup argocd and deploy over K8s using ArgoCD
Vertical versus horizontal scaling
Microservice mesh
Proxy sidecar pattern (for left-right traffic)
Gateway (for north-south, egress/ingress traffic)
Gateways
Popular gateways: Istio/Linkerd, Kong, Tyk, and traditional (Nginx, Apache2)
Load balancing strategies
Rate limiting strategies
Chapter 10: Observability (Not just monitoring)
Why do we need observability?
What all can we observe?
What is the role of SRE?
Concepts:
Temple: The six pillars of observability - Traces, Exceptions, metrics, logs, events
Alerts
Open Telemetry standard by CNCF
Advanced: Use of AI/ML
Tools
The FOSS based Grafana stack
- Setup Grafana stack. Integrate with a service in Godspeed meta-framework or any 3rd gen framework. Customize a Grafana dashboard.
ELK, Datadog, Splunk
Chapter 11: ML/AI
When do we need ML? And what is ML?
AI: Its uses and technical overview
Models
Langchain
Vector stores
Clustering
Neural Networks
Semantic analysis and meta-data extraction
Sentiment analysis
Popular services, open source models and tools
Use cases
Chapter 12: Advanced Topics
Serverless functions or Function as a service (FAAS)
Event (Async) driven architecture. Pros and cons.
Distributed workflow management
Orchestration versus Choreography
Distributed transactions (Saga pattern) and distributed locking
Retries with backoff, circuit breaking, “rollback”, retry queue, dead letter queue
What is a rule engine? And its use cases. How can dynamically computed policies and workflows come together for highly agile and fluid product offerings or business use cases.
Chapter 13: Case studies and use cases (study with practical examples and practice)
Food delivery app delivery tracking with mutation subscriptions and notifications
Loan Origination System with rule engine based dynamic workflows
Ecommerce distributed workflows
Real time fraud detection using KSQL, Elasticsearch, Data structures
Movie recommendations: Content based and collaborative filtering based approaches
Shortest path recommendations for a navigation app
(Artificially) Intelligent customer support agents
- Natural language to database or API queries. Dynamically computed workflows. Converting response to natural language.
That's it for now!
For participating in the learning and sharing journey as a community, you can join us on Discord,Github, Youtube, social channels and subscribe to the blog's newsletter.
Stay tuned for the series. We have a long way to go.
The woods are lovely, dark and deep,
But I have promises to keep,
And miles to go before I sleep,
And miles to go before I sleep.Robert Frost. "Stopping By Woods On A Snowy Evening."