Traditional it process -> decision to go ahead , issued purchase order , HW implementation , Software implementation , Go market Cloud better alternative->provision environments(server , network , storage , database) , pay as you go , add/remove capacity , destroy and stop .
Why need cloud-> Business Agility , Customer Experience , Cost
Before cloud ->every company have own data centers , expensive implementations ,and maintains , had maintain specialized engineers to maintain to data center
With cloud ->no need to maintain datacenter , consumer resource as u wish and pay for what u consume , no need to buy datacenter , network also .
Cloud computing->Cloud computing is a model that allows convenient, on-demand access to shared computing resources (like networks, servers, storage, and applications). These resources can be quickly provisioned or released with minimal management or involvement from service providers.
Characterize of cloud -> On-demand Self-service ->Consumers can provision computing resources , Virtual servers, Network, storage ,Can be provisioned do on-demand , No interaction with the service provider is needed .Resource Pooling-> Providers computing resources are pooled , Dynamically assigned and reassigned to demand , Customer has no knowledge over exact location , May specify location at a higher level. Rapid Elasticity -> Ability to elastically provision and release , May happen automatically inline with the demand , Broad networks access-> Capabilities are available over the network , Access is through standard mechanism for heterogenous clients Measured Service-> Automatically control and optimized resources using metering capability , Pay-per-use basis , Resources usage can be monitored, controlled and reported
Characterize of cloud -> On-demand self-service: Access resources like servers and storage anytime without human assistance.,Broad network access: Services are accessible over the internet from any device.,Resource pooling: Shared resources serve multiple users with dynamic allocation.,Rapid elasticity: something immediately happening or automatically happening .,Measured service: Pay only for what you use, monitored and controlled automatically
Cloud is not technology , cloud can be , business model , delivery model .
IAAS ->provider create networking , storage , servers , virtualization. Provider manage underlying physical cloud infrastructure and the virtualization layer. Consumer want create applications , data and os. Compute Content Delivery Networks Storage Service Management Backup and Recovery.
PAAS ->providers create os , virtualizations , servers , storage , networking . Consumer does not manage infrastructure or application And do limited user specific application configuration changes. Email and Office Productivity CRM , Document Management Enterprise Resource Planning(ERP).
IAAS ->provider-(ex: create networking , storage , servers , virtualization). Provider manage underlying physical cloud infrastructure and the virtualization layer. Consumer want create applications , data and os. Compute Content Delivery Networks Storage Service Management Backup and Recovery.
PAAS ->providers- (ex:create os , virtualizations , servers , storage , networking ). Consumer does not manage infrastructure or application And do limited user specific application configuration changes. Email and Office Productivity CRM , Document Management Enterprise Resource Planning(ERP).
SAAS -> Use the providers application running on cloud , Consumer does not manage infrastructure or application , Email and Office Productivity Customer Relationship Management , Document Management Enterprise Resource Planning.
Cloud development model Publish cloud -> Provisioned for open use by the general public Operated by a service provider organization Located on premise of the cloud provider PUBLIC CLOUD Characteristics , Cost-effective, On-demand virtually unlimited scalability , Zero maintenance , Continuous up time , Low level of data security .
Private cloud -> Provisioned for exclusive use by a single organization Multiple business units may consume it Can be owned and managed by the organization, a third party or a combination May be located on or off premises. High data security , Less risky , Reliable ,Single tenant &compliance.
Community cloud -> consumers with shared concerns may be security, regulatory requirements, compliance etc. Can be owned, managed and operated by an organization in the community, a third party or a combination
actors in cloud computing -> Cloud Consumer : is an individual or organization that uses services from cloud providers while maintaining a business relationship with them., Cloud Provider : person, organization, entity responsible for making a service available to interested parties.,Cloud Auditor: is an entity that performs independent assessments of cloud services, focusing on their operations, performance, and security. ,Cloud Broker : manages the use, performance, and delivery of cloud services while facilitating relationships between cloud providers and consumers.,Cloud Carrier : is an intermediary that provides connectivity and transport of cloud services between providers and consumers.
Cloud development model -> Public cloud :Provisioned for open use by the general public, Operated by a service provider organization, Located on premise of the cloud provider ,(PUBLIC CLOUD Characteristics : Cost-effective, On-demand virtually unlimited scalability , Zero maintenance , Continuous up time , Low level of data security .)
Private cloud -> Provisioned for exclusive use by a single organization, Multiple business units may consume it Can be owned and managed by the organization, a third party or a combination May be located on or off premises.(PUBLIC CLOUD Characteristics: High data security , Less risky , Reliable ,Single tenant &compliance.)
Community cloud -> consumers with shared concerns may be security, regulatory requirements, compliance etc. Can be owned, managed and operated by an organization in the community, a third party or a combination. ex: GovernmentClouds
hybrid cloud ->Combination of private and public clouds , May enable portability of data and applications within The clouds by standards or proprietary technologies (Characteristics : Secure and safe ,Cost-effective , Flexibility and scalability , Portability between private &public , Data transfer)
Cloud service providers –> Amazon Web Services (AWS) , Microsoft Azure , Google Cloud Platform (GCP) , IBM Cloud , Alibaba Cloud , Oracle Cloud Infrastructure (OCI)
CLOUD CORE SERVICES AND GLOBAL INFRASTRUCTURE
2. CLOUD CORE SERVICES AND GLOBAL INFRASTRUCTURE
Cloud Global Infrastructure -> provide how they design and build there environment with flexible, reliable, scalable, and with high-quality global network performance.
Benefit of Cloud Global Infrastructure ->security availability performance scalability flexibility
Benefit of Cloud Global Infrastructure ->security, availability, performance , scalability,flexibility
Main component of Cloud Global Infrastructure -> region , availability zone , data enter
Shared responsibility model -> in cloud computing domain , we have each and every cloud service providers have shared responsibility model. Security of the Cloud , Security in the Cloud.
How do you pay for cloud services providers ->pay for what u use , pay less you reverse , play less when u use more.
Services limits ->Default quotas , Quota is region specific , Soft limits – can be increased upon requests Hard limits – cannot be increased , Limit potential excessive charges Limit overuse of resources by a single entity.
CLOUD COMPUTE
3. CLOUD COMPUTE
Virtualization –> application , dependencies , guest os , have hypervisor , hardware ,Containerization –> container , application xyz , app dependencies, have docker engine , host os , hardware
Virtualization –> application , dependencies , guest os , have hypervisor , hardware Containerization –> container , application xyz , app dependencies, have docker engine , host os , hardware
Software define compute ->Software-defined compute virtualizes and abstracts computing functions from hardware, allowing workloads to be distributed across multiple processing units. It uses generic, industry-standard hardware, making it easy to expand resources as needed.
Cloud Compute ->Amazon EC2 AWS’s offering of virtual machines in cloud, Provided as an IaaS ,Pre-configured templates (AMI) , Per second billing
Azure VM : Microsoft’s offering of virtual machines in cloud ,Provided as an IaaS ,Pre-configured templates (marketplace) ,Per minute billing
Cloud Compute ->
1.Amazon EC2 : AWS’s offering of virtual machines in cloud, Provided as an IaaS ,Pre-configured templates (AMI) , Per second billing
2.Azure VM : Microsoft’s offering of virtual machines in cloud ,Provided as an IaaS ,Pre-configured templates (marketplace) ,Per minute billing
Block Storage -> Manages data as blocks within sectors and tracks, Set of blocks can be organized as a file system , Suitable for structured data (file systems, databases, logs) , optimized for block level performances , Performance measured in IOPS , AWS –Elastic Block Storage , Azure –Azure Disks (Page Blob Storage)
File Storage ->Manages data as a file hierarchy , Storage managed the underlying data on disk , Storage server uses block storage with a local file system Protocols supported , Network File System (NFS) Server Messaging Protocol (SMB) , AWS –Elastic File System (EFS) , Azure –Azure Files (File Storage
Object Storage ->Manages data as objects , Each object consists of 1.Data 2.Metadata 3.Globally unique, identifier Clients can access through HTTPS or APIs or SDKs , AWS –Simple Storage Service (S3), Azure –Azure Blobs
Object storage features->versioning , encryption (Client side ,Server Side(Versioning, Service Managed Keys, Customer Managed Keys, Customer Provided Keys) , web hosting , immutability (time based retention , legal hold)
Object storage tiers ->CSPs offer different access tiers designed for different use cases(Standard Intelligent Tiering, Infrequent Access , Glacier) , Azure (Hot , Cold , Archive)
Object storage tiers ->CSPs offer different access tiers designed for different use cases., AWS(Standard Intelligent Tiering, Infrequent Access , Glacier) , Azure (Hot , Cold , Archive)
Other Important Aspects of Cloud Storage -> Archiving and Backup , Hybrid Storage, Bulk Data Transfer
Cloud Networking
4.CLOUD NETWORKING
Software Defined Networking (SDN) ->allows dynamic, programmable management of network resources. Decouples control and forwarding planes, making network infrastructure more flexible and scalable.
Network Function Virtualization (NFV) -> NFV is a network architecture that decouples physical network functions (PNF) from proprietary hardware appliances.
Cloud-native Network Functions (CNF) ->CNF is the successor of VNFs. It is a software implementation of a PNF that runs inside a Linux container.
Similarities ->Both use network abstraction. , Both rely on virtualization.
Differences:->SDN separates control from forwarding functions; focuses on overall infrastructure. , NFV abstracts network functions (like firewalls) from hardware; focuses on specific functions.
Virtual Networks-> Isolated cloud network with IP ranges and subnets. , Supports peering if IPs don’t overlap. Examples: AWS VPC, Azure VNet
Virtual Network Interfaces ->Logical network components representing Network Interface Cards (NICs).Allows instances to communicate with others or the internet.
Features->Attaches to compute instances. ,Can have both public and private IPs. , Multiple interfaces can attach to one instance.
Virtual Network Interfaces ->Logical network components representing Network Interface Cards (NICs).Allows instances to communicate with others or the internet. (Features:Attaches to compute instances. ,Can have both public and private IPs. , Multiple interfaces can attach to one instance.) ,Examples AWS-> Elastic Network Interface (ENI) , Azure: Network Interface Card (NIC)
Subnets -> Divide a network into smaller segments to organize and control IP traffic. (Rules:Subnet ranges (CIDR blocks) cannot overlap.,Control access using Access Control Lists (ACLs).
AWS:Public Subnet: Can connect directly to the internet.Private Subnet: Cannot connect to the internet directly.
Azure: No separate public/private subnets; uses route tables for similar control.
Access Controlling ->Implements a simple firewall functionality to control inbound and outbound traffic to a subnet / virtual network interface.
ACL Rules Include-> Rule number, Protocol , Port ranges , IP ranges , Action.
Usage-> AWS Security Groups ( For EC2 instances) , AWS NACL (Network Access Control Lists) For subnets. , Azure NSG (Network Security Groups) For subnets and network interfaces.
Stateful vs Stateless Rules
Stateful Rules ->Only need to specify the security rule for the request, no need to specify a security rule for the response.
Stateless Rules->Need to specify security rules for the request as well as the response.
Internet Gateway (IGW) ->Connects your cloud network to the internet. No bandwidth limits, scales automatically. Used in AWS; Azure uses default gateways for internet access.
NAT Gateway - > Allows private servers to access the internet while blocking inbound traffic from outside. Must be set up in a public subnet.
NAT Gateway - > Allows private servers to access the internet while blocking inbound traffic from outside. Must be set up in a public subnet. Implementations : AWS NAT Instance, AWS NAT Gateway , Azure Virtual Network NAT
VPN Gateway ->Creates encrypted tunnels over the internet for secure communication between cloud and on-premises networks. Examples: Azure VPN Gateway , AWS Virtual Private Gateway
Two types of VPN services
Site-to-Site VPN: Connects a cloud network with on-premises networks. Point-to-Site VPN: Connects individual devices (like laptops) to a cloud network using special software.
1.Site-to-Site VPN: Connects a cloud network with on-premises networks.
2.Point-to-Site VPN: Connects individual devices (like laptops) to a cloud network using special software.
Azure: Supports both Site-to-Site and Point-to-Site. , AWS: Site-to-Site via Virtual Private Gateway , Point-to-Site via AWS Client VPN.
Network Transit Gateway ->A service that connects multiple virtual networks and on-premises networks using a central hub. Architecture: Uses a hub-and-spoke model for efficient connectivity. Examples: AWS Transit Gateway , Azure Virtual WAN
Network Peering ->Connects and allows communication between two virtual networks. , Can connect multiple virtual networks, even across different accounts/subscriptions. , Each network can peer with multiple others.
Limitations-> No transitive peering: If VNet A is peered with VNet B, and VNet B is peered with VNet C, A cannot communicate with C directly. Implementation: Available on both AWS and Azure platforms.
(Limitations-> No transitive peering: If VNet A is peered with VNet B, and VNet B is peered with VNet C, A cannot communicate with C directly. Implementation: Available on both AWS and Azure platforms.)
Bastion Hosts -> securely connects to private servers without exposing them to the internet, using a public IP in a public subnet.) How it works->Connects to private servers without exposing them to the internet. , Must be set up in a public subnet and given a public IP.
Hybrid Connectivity ->Ways to connect cloud networks with on-premises or remote users.
Connecting Remote Users->Options include direct internet access or VPNs (e.g., Azure Point-to-Site, AWS Client VPN) Connecting Remote Networks-> ptions like Site-to-Site VPN, Azure ExpressRoute, and AWS Direct Connect for secure, dedicated connections.
1.Connecting Remote Users->Options include direct internet access or VPNs (e.g., Azure Point-to-Site, AWS Client VPN)
2.Connecting Remote Networks-> ptions like Site-to-Site VPN, Azure ExpressRoute, and AWS Direct Connect for secure, dedicated connections.
Private Access to Cloud Services
Private Links->Securely connect to PaaS services without using the public internet. ,Private Endpoints->Assign an IP in your virtual network for secure access to services. Azure Service Endpoints->Provide optimized access to Azure PaaS over a private network but still uses the public endpoint for traffic.
Network Flow Logs->Capture information about IP traffic going in and out of your network interfaces in the virtual network
Benefits->Logs are collected without affecting network performance ,Useful for troubleshooting, security monitoring, and compliance.
Examples-> Azure: Logs from NSGs, VPCs, and subnets. , AWS: Flow logs for VPCs and network interfaces.
Private Links->Securely connect to PaaS services without using the public internet. ,Private Endpoints:Assign an IP in your virtual network for secure access to services. Azure Service Endpoints:Provide optimized access to Azure PaaS over a private network but still uses the public endpoint for traffic.
Network Flow Logs->Capture information about IP traffic going in and out of your network interfaces in the virtual network (Benefits:Logs are collected without affecting network performance ,Useful for troubleshooting, security monitoring, and compliance. ) ,Examples-> Azure: Logs from NSGs, VPCs, and subnets. , AWS: Flow logs for VPCs and network interfaces.
Cloud High Availability & Disaster Recovery
5. CLOUD HIGH AVAILABILITY AND DISASTER RECOVERY
Downtime->Refers to a period when a system is unavailable (offline))
High Availability (HA) ->Ensures systems remain operational for long periods. Eliminate Single Points of Failure, enable reliable crossover, and detect failures quickly.
Redundancy ->Duplication of system components to increase reliability
Passive Redundancy-> Extra capacity for failures. , Active Redundancy->Automatically switches to backups
1.Passive Redundancy-> Extra capacity for failures. ,
2.Active Redundancy->Automatically switches to backups
Failover & Failback
Failover-> Switch to standby systems upon failure. , Failback->Restoring to the original system after recovery.
Replication(Sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.) Active Replication: Process on all replicas. , Passive Replication: Process on one replica and sync to others.
High Availability Clusters ->A group of computers ensuring continued service when components fail.
Node Configurations->
Active/Active: All nodes actively handle requests. Active/Passive: One node is active, while others are on standby. Node Reliability: Storage (disk mirroring, redundant SANs), Network , Electricity
Load Balancing ->Efficiently distributes network traffic across multiple servers.
Algorithms-> Round Robin, Least Connections, least time , IP Hash.
Node Configurations :Active/Active: All nodes actively handle requests. Active/Passive: One node is active, while others are on standby.
Disaster Recovery (DR) ->Part of BCP to restore systems after disasters (natural or human-made).
Key Elements->DR Team: Responsible personnel. , Risk Evaluation: Assess potential threats., Business Critical Asset Identification: Prioritize vital assets. ,Backup Strategy: Plan for data backups. , Testing & Optimization: Regular tests to ensure readiness.
Backup Strategy ->A copy of data stored elsewhere to restore after data loss.
Backup Methods->Full Backup: Complete copy of all data. , Incremental: Only changes since the last backup. , Differential: Changes since the last full backup. Near Continuous Data Protection: Almost real-time backups.
Disaster Recovery (DR) Tiers -> Tier 0: No off-site data backup. ,Tier 1: Data backup without a hot site. , Tier 2: Data backup with a hot site. , Tier 3: Uses electronic vaulting., Tier 4: Point-in-time copies for recovery.,Tier 5: Ensures transaction integrity. ,Tier 6: Minimal or zero data loss., Tier 7: Fully automated, business-integrated recovery solution.
Backup Strategy ->A copy of data stored elsewhere to restore after data loss. (Backup MethodsFull Backup: Complete copy of all data. , Incremental: Only changes since the last backup. , Differential: Changes since the last full backup. Near Continuous Data Protection: Almost real-time backups.)
Recovery Objectives -> RTO (Recovery Time Objective): How quickly you need to get systems running again after a disaster. RPO (Recovery Point Objective): How much data loss is acceptable (measured in time). RCO (Recovery Consistency Objective): How much data inconsistency you can tolerate after recovery.
Cloud Security & Monitoring
Software-defined security (SDS)->is a type of security model in which the information security in a computing environment is implemented, controlled and managed by security software.)
Key Features-> Automates controls like intrusion detection, network segmentation, and access management using policy-driven software.
Principle of Least Privilege (PLOP)->A security concept where users/processes get only the permissions needed to do their job. Purpose: Minimizes risk by limiting access to only what's necessary.
Identity and Access Management ->IAM enforces the Principle of Least Privilege. It’s a framework of policies to control access to resources. Purpose: Identifies, authenticates, and authorizes users to ensure they have the right access.
6. CLOUD SECURITY AND MONITORING
Software-defined security (SDS)->is a type of security model in which the information security in a computing environment is implemented, controlled and managed by security software.),(Key Features Automates controls like intrusion detection, network segmentation, and access management using policy-driven software.)
how to apply security mechanisms/concept
1.Principle of Least Privilege (PLOP)->A security concept where users/processes get only the permissions needed to do their job. Purpose: Minimizes risk by limiting access to only what's necessary.
2.Identity and Access Management (IAM)->IAM enforces the Principle of Least Privilege. It’s a framework of policies to control access to resources. Purpose: Identifies, authenticates, and authorizes users to ensure they have the right access.
IAM Components ->Logical Organization: Defines boundaries for users/processes. Users-> Individuals or services needing access. Have credentials but no default permissions. Groups-> Collections of users without their own credentials. Policies->Set permissions to allow or deny actions. Roles->Collections of policies for temporary access. Used in Role-Based Access Control (RBAC).
Multi-Factor Authentication (MFA)->means using two or more ways to verify who you are before giving access Factors->What you have: Security token, smart card. ,What you know: Password, PIN. , What you are: Biometrics (fingerprint, face). , Where you are: GPS location.
Federated Identity & Single Sign-On (SSO)
3.Multi-Factor Authentication (MFA)->means using two or more ways to verify who you are before giving access Factors->What you have: Security token, smart card. ,What you know: Password, PIN. , What you are: Biometrics (fingerprint, face). , Where you are: GPS location.
4. Federated Identity & Single Sign-On (SSO)
Federated Identity->Links a user’s identity across multiple systems so they don’t need separate logins.
Single Sign-On (SSO)->Lets users log in once and access multiple systems without needing to re-enter credentials. Two Types:->Enterprise Identity Federation: For internal company systems., Web Identity Federation: For online platforms and services.
Single Sign-On (SSO)->Lets users log in once and access multiple systems without needing to re-enter credentials.
Two Types:->Enterprise Identity Federation: For internal company systems., Web Identity Federation: For online platforms and services.
Types of Encryption-> Symmetric Key: Uses one key for both encryption and decryption. ,Asymmetric Key: Uses a pair (public and private keys) for encryption and decryption.,Hashing: Converts data into a fixed-size value, irreversible.
Stages of Data Encryption->Data in Use: While data is being processed. ,Data in Transit: While data is being transferred over networks. ,Data at Rest: While data is stored.
Client-Side vs Server-Side Encryption-> Client-Side: Data encrypted before sending to the server. ,Server-Side: Data encrypted once it reaches the server.
Cloud Security Risks & Threats ->Poor Access Management: Weak controls over user permissions can lead to unauthorized access. ,Data Breach/Leak/Loss: Sensitive data might be exposed or stolen. ,Misconfiguration: Incorrect settings can leave systems vulnerable. ,Insecure APIs: Weak APIs can be exploited by attackers. ,Account Hijacking: Attackers gaining control of cloud accounts. ,Lack of Visibility: Difficulty in monitoring and tracking cloud activities. , DoS/DDoS Attacks: Overloading cloud services to disrupt access.
Monitoring in Cloud
Cloud Monitoring->Tracks and manages cloud operations to ensure everything runs smoothly. Methods-> Manual or automated checks to confirm websites, servers, and apps are working correctly. Example Services: Datadog, AppDynamics , Azure Monitor , Amazon CloudWatch
Metrics, Events & Logs ->Metrics: Raw data on resource usage (like CPU, memory). Useful for monitoring performance. ,Events: Notifications triggered by system actions (like updates or errors). Capture details on what, where, and when something happened. ,Logs: Detailed records of system activity, essential for troubleshooting and analysis.
Alerts -> Triggered by changes in metrics, events, or logs to respond to issues.
Types of Alerts->Threshold-based: Activated when a metric crosses a set limit., Anomaly Detection: Alerts on unusual patterns., Heartbeat Alerts: Triggered if regular status signals stop.
Output Types-> Notifications: Inform users via email or SMS. Automated Actions: Systems take corrective measures automatically
Alerts -> Triggered by changes in metrics, events, or logs to respond to issues.
Types of Alerts->Threshold-based: Activated when a metric crosses a set limit., Anomaly Detection: Alerts on unusual patterns., Heartbeat Alerts: Triggered if regular status signals stop. , Output Types-> Notifications: Inform users via email or SMS. Automated Actions: Systems take corrective measures automatically
7.CLOUD DATABASE AND ANALYTICS SERVICE
Cloud database and analytics services
Database Models
Relationl/SQL -> Highly structured table organization, Rigidly defined formats, Dependencies among tables, Enforce ACID (Atomicity, Consistency, Isolation, Durability), Ex: MS SQL, MySQL, Oracle, PostgreSQL, Amazon RDS
Non-relational/No-SQL -> Document oriented, Large and complex queries, Supports rapidly changing design, Ex: MongoDB, Cassandra, CosmosDB, Redis, CouchDB, Aurora
1.Relationl/SQL -> Highly structured table organization, Rigidly defined formats, Dependencies among tables, Enforce ACID (Atomicity, Consistency, Isolation, Durability), Ex: MS SQL, MySQL, Oracle, PostgreSQL, Amazon RDS
2.Non-relational/No-SQL -> Document oriented, Large and complex queries, Supports rapidly changing design, Ex: MongoDB, Cassandra, CosmosDB, Redis, CouchDB, Aurora
Database Workloads
Online Transaction Processing (OLTP) – comparison -> Focus is on operational data, Transaction processing, Small and simple ad-hoc queries, Response in milliseconds
Online Analytical Processing (OLAP) – comparison -> Focus is on historical data, Data analysis and reporting, Large and complex queries, Responses times from seconds to hours
Different types of database models and workloads -> [Relational / OLTP (Online Transaction Processing) - Examples: Oracle, PostgreSQL, MS SQL, MySQL], [Relational / OLAP (Online Analytical Processing) - Examples: Oracle, PostgreSQL, MS SQL, MySQL], [Non-relational / OLAP - Examples: Hadoop, HDInsight, Non-relational / OLTP - Examples: MongoDB, Cassandra, Riak]
Different types of database models and workloads -> [Relational / OLTP (Online Transaction Processing) - Examples: Oracle, PostgreSQL, MS SQL, MySQL], [Relational / OLAP (Online Analytical Processing) - Examples: Oracle, PostgreSQL, MS SQL, MySQL], [Non-relational / OLAP - Examples: Hadoop, HDInsight][ Non-relational / OLTP - Examples: MongoDB, Cassandra, Riak]
Non-relational / No-SQL Database
Key-value store - Data is stored as indexed key-value pairs, Ideal for session data and shopping carts, Flexibility: High, Complexity: None (very simple to use), Performance: High, Scalability: High, Best suited for simple, fast data retrieval scenarios.
Column store - Optimized for retrieving entire columns of data rather than rows, Commonly used in Content Management Systems (CMS) and blogging platforms, Flexibility: Moderate, Complexity: Low, Performance: High, Scalability: High, Great for analytical queries, especially on large datasets.
Document store - Stores data in documents (like XML, JSON) which are schema-less, Useful for e-commerce applications and analytics where flexible data models are needed, Flexibility: High, Complexity: Low, Performance: High, Scalability: Variable (can be high), Ideal for handling semi-structured data with dynamic schemas.
Graph database - Represents data as interconnected nodes (graphs) focusing on relationships, Best for applications where understanding relationships is crucial, such as social networks, Flexibility: High, Complexity: High, Performance: Variable (depends on the use case), Scalability: Variable, Best used where relationships and connections between data are a primary focus (e.g., social networks).
CAP Theorem -> Consistency – All clients see the same view of data, even right after update or delete, Partitioning – the system continues to work as expected, even in presence of partial network failure, Availability – All clients can find a replica of data, even in case of partial node failure.
Database Caching is a buffering technique that stores frequently accessed data in temporary memory. It improves data access speed and reduces the workload on databases. Ex: Redis, Memcached
Comparison Redis and Memcached
Radis -> Open source;in-memory;key-value data store, Sub-millisecond response times, Supports various data structures (strings, lists, sets etc.), Persistent – cache survives reboots, Supports read replicas, atomic operations, backup/restore, HA
Memcached -> Open source; in-memory; object store, Sub-millisecond response times, Supports strings and objects, Not persistent – cache does not survive reboots, Supports scaling out, multithreading
Data warehouse
A data warehouse management system is designed to support business intelligence (BI) and analytics.
Primarily used for querying and analyzing large amounts of historical data. Stores structured, processed data for specific business needs.
Data is collected from various sources like application logs and transaction systems.
Data Loading Approaches -> ETL (Extract, Transform, Load): Data is transformed before loading, ELT (Extract, Load, Transform): Data is loaded first, then transformed.
Data Lake
A storage repository that holds large amounts of raw data in its native format until needed.
Unlike data warehouses, which store data in a hierarchical structure (files/folders), data lakes use a flat architecture for data storage.
Comparison DW and DL
DW (Processed data, Data currently in use, Used by business professionals)
DL (Raw data, Purpose of data not determined yet, Used by data scientists)
Real Time Data Processing (Stream Processing)
It is a fast processing technique that handles data immediately upon input, ensuring quick and accurate outputs.
Requires a constant stream of incoming data for continuous, ongoing processing.
Cloud Migration
8.CLOUD MIGRATION
Cloud Migration is the movement of a meaningful portion of your organization’s existing IT assets to the cloud.
Migration Process
Opportunity evaluation -> reasons to move resources to cloud : Hardware reaching end-of-life, Expiring data center leases, Boosting productivity, Supporting global expansion, Mergers & acquisitions, Standardizing architectures
Portfolio discovery and planning -> What is in your current environment?, What are the interdependencies?, Which systems should you migrate first?, How will you execute the migration?, Tools: AWS Discovery Service, Azure Migrate Service, and third-party migration tools.
Application design (iterate)
Migration and validation (iterate) -> Testing & Decommissioning: Run parallel environments temporarily if needed, Gradually switch traffic and users to the new cloud system, Perform a full cutover once the cloud environment is stable.
Operate -> Modern operating models: Devops, Noops, FinOps, SecOps
1.Opportunity evaluation -> reasons to move resources to cloud : Hardware reaching end-of-life, Expiring data center leases, Boosting productivity, Supporting global expansion, Mergers & acquisitions, Standardizing architectures
2.Portfolio discovery and planning -> What is in your current environment?, What are the interdependencies?, Which systems should you migrate first?, How will you execute the migration?, Tools: AWS Discovery Service, Azure Migrate Service, and third-party migration tools.
3.Application design (iterate)
4.Migration and validation (iterate) -> Testing & Decommissioning: Run parallel environments temporarily if needed, Gradually switch traffic and users to the new cloud system, Perform a full cutover once the cloud environment is stable.
5.Operate -> Modern operating models: Devops, Noops, FinOps, SecOps
Rehost -> Known as lift-and-shift, Does not require application / code changes, migrate applications as-is to cloud, Optimize, re-architect applications once they are in cloud
Re-platform -> Known as lift-tinker-and-shift, Migrating to a different platform without changing the core architecture of the application, Ex: Migrating web sites from VMs to PaaS (i.e. Azure App Service, Amazon Elastic Beanstalk etc.), Migrating databases from VMs to PaaS (i.e. Amazon RDS, Azure SQL etc.), Changing the platform of databases (i.e. from Oracle to PostgreSQL etc.), Changing the platform of application logic (i.e. from Oracle WebLogic to Apache Tomcat etc.)
Repurchace -> Moving to a different platform, mainly to a SaaS platform, Ex: Moving a CRM to Salesforce.com, Moving an HRIS to Workday, Moving a CMS to SharePoint
Refactor/Re-architect/Rebuild -> Re-imagine the application using cloud-native features for better optimization, Enhancing cloud-native features like performance, scalability, agility, and business continuity, Involves migrating monolithic apps to microservices or serverless architectures, Most expensive migration strategy
Retire -> Get rid of applications that are no longer useful and can be turned off, Ex: More attention to important applications, Reduce the security risks, Cost savings.
Retain -> Revisit migrating these applications later, or do nothing for now, Ex: Recently purchased / upgraded applications that cannot be migrated at this point, Sunsetting applications, Low priority applications
Cloud Migration Challenges
Cost of migration -> Focus on planning, Follow an incremental adaptation process, Use a hybrid cloud approach
Resistance to Adapt ->Get the leadership buy-in, Choose intuitive tech solutions
Shortage of Skills -> Invest in expert training and resources, Build a culture of continuous learning, Get help from vendors to skill up in-house resources
Cloud Native Applications
9. CLOUD NATIVE APPLICATION
Monolithic Architecture
Traditional way of building applications, Built as a single and indivisible unit, Usually contains(A client-side UI, A server-side application, A database, A middleware)
Strengths -> Less cross-cutting concerns (i.e. handling functions like logging, caching, monitoring etc.), Easier debugging and testing, Simple to deploy, Simple to develop
Weaknesses -> Too complex to understand, Challenging to make changes, Cannot scale components independently, Difficult to apply new technologies / features.
Microservices Architecture
Break down an application into smaller, independent units called microservices, Each unit performs a specific process as a separate service, Built as independently deployable modules, Services communicate through predefined APIs, Each service can be updated and scaled independently.
Strengths -> Independent components, Easier understanding, Better scalability, Flexibility in choosing the technology, Higher level of agility
Weaknesses -> Extra complexity of a distributed system, Cross-cutting concerns, Harder to debug and test.
Comparison of monolithic and microservices
Monolithic -> Contains all functionality in one unit, organized by layers (web, business, data), Scales by duplicating the entire app on multiple servers/VMs/containers.
Microservices -> Breaks functionality into smaller, independent services, Scales by deploying each service separately across servers/VMs/containers.
State in monolithic and microservices approaches
Monolithic -> single monolithic Database, tiers of specific technologies
Microservices
Stateless - Do not maintain internal state; data can be retrieved externally. Ex: Web frontends, protocol gateways, Azure Cloud Services.
Stateful - Maintain persistent data, such as databases, documents, user profiles. Ex: workflows, shopping carts, etc.
Cloud-native computing
A modern approach to build scalable, resilient applications leveraging cloud capabilities, Pioneered by companies "born in the cloud" like Netflix, Spotify, Uber, and Airbnb
Cloud Native Computing Foundation (CNCF) promotes cloud-native practices everywhere, focusing on containerization, Enabling applications to run in dynamic environments (public, private, hybrid clouds).
Cloud Native Computing Foundation (CNCF) : promotes cloud-native practices everywhere, focusing on containerization, Enabling applications to run in dynamic environments (public, private, hybrid clouds).
12 factor app -> A set of principles for building scalable, resilient, and performant enterprise applications, Complements the principles of microservices architecture, Developed by engineers at Heroku. Ex: codebase, dependencies, config, port binding
Cloud native tools -> Cloud Delivery Model (Manages provisioning, consumption, and resource management), Containers (Enhances application manageability, scalability, and security), Microservices (Breaks down functionality into smaller, specialized services), Service Mesh(Manages service-to-service communication for scalability and security), Orchestration(Automates deployment, operation, and scaling of microservices.)
Cloud native practices
Devops -> integrates Development, IT Operations, and Quality Assurance (QA) to streamline software delivery, Continuous loop of planning, coding, building, testing, releasing, deploying, monitoring, and feedback.
SRE -> A model defined by Google as a set of best practices for ensuring digital business reliability, Monitoring, incident response, root cause analysis, capacity planning, testing, and release management, Focuses on combining software engineering with IT operations to enhance system reliability.
NoOps -> DevOps practitioners are freed from managing operations and can focus entirely on software development, Achieved through full automation of operational tasks, eliminating the need for manual intervention.
CI (continuous integration) -> Automates the integration of code changes from multiple contributors into a single project, Developers frequently merge code into a central repository, Automated builds and tests ensure code correctness before integration.
CD(continuous delivery and deployment)
Continuous Delivery: Automates testing to ensure code changes are stable for deployment. Requires manual approval for release.
Continuous Deployment: Automatically deploys code changes to production once tests pass, without manual intervention.
1.Continuous Delivery: Automates testing to ensure code changes are stable for deployment. Requires manual approval for release.
2.Continuous Deployment: Automatically deploys code changes to production once tests pass, without manual intervention.
Application Modernization -> Involves refactoring, repurposing, or consolidating legacy software to better align with current business needs. To generate new business value from existing applications.
Application modernization journey -> Rehost(Traditional app), Refactor(Existing app hosted as container or executab), Rearchitect(Existing application + new microservices), Rearchitect(Parts of existing application rearchitected), Rearchitect/rebuild(Transformed microservices application)
Introduction to Containerization, Container Orchestration and Infrastructure as Code (IaC)
Application modernization journey -> Rehost(Traditional app), Refactor(Existing app hosted as container or executable), Rearchitect(Existing application + new microservices), Rearchitect(Parts of existing application rearchitected), Rearchitect/rebuild(Transformed microservices application)
10. INTRODUCTION TO CONTAINERIZATION , CONTEINER ORCHESTRATION AND INFRASTRUCTUR AS CODE (IaC)
Containerization is a software deployment method that packages an application’s code along with all necessary files and libraries, enabling it to run consistently across different environments.
Containers vs VMs
Containers -> Shares the host operating system's kernel, More portabl, Faster to start up and shut down, Uses fewer resources, Good for portable and scalable applications
VM s -> Has its own kernel, Less portable, Slower to start up and shut down, Uses more resources, Good for isolated applications
How Containers Work (layered structure)
Hardware: At the bottom, includes CPU, disk, and network interfaces.
Host OS & Kernel: Acts as a bridge between software and hardware.
Container Engine: Sits atop the Host OS, managing containers.
Containers: Run on top, containing application code, binaries, and libraries, isolated in user spaces.
1.Hardware: At the bottom, includes CPU, disk, and network interfaces.
2.Host OS & Kernel: Acts as a bridge between software and hardware.
3.Container Engine: Sits atop the Host OS, managing containers.
4.Containers: Run on top, containing application code, binaries, and libraries, isolated in user spaces.
Benefits of containerization -> Portability, Scalability, Fault tolerance, Agility
Container Orchestration -> A technology that automates the management of containers
Needs of container orchestration -> Manages multiple containers across various hosts, Ensures uptime, load balancing, and scalability, Solves challenges of manual scaling and deployment.
Benefits of Container Orchestration -> Enables automatic starting, stopping, and management of containers, Helps developers scale cloud applications accurately, reducing human errors, Ensures containers are deployed with the right resources.
Infrastructure as Code (IaC) -> automates the provisioning and management of infrastructure using code rather than manual processes, Reduces errors and time consumption, especially when managing applications at scale.
Benefits of IaC -> Easily duplicate an environment, Reduce configuration errors, Iterate on best-practice environments
How IaC Works -> IaC describes system architecture using configuration files (e.g., YAML, JSON, HCL), Treats infrastructure as code, similar to application development, Can be written in languages like Python or Java, Supports integrated development environments (IDEs) for error checking, Managed under version control, with changes tracked via commits.
