This project implements a secure and distributed system integrating RAFT-based leader election, messaging, and real-time communication using various technologies. It ensures modularity, security, and fault tolerance through microservices and message-driven architecture.

Architecture Overview

The system consists of multiple components:

Leader Election with RAFT Algorithm
Message Broker for Distributed Communication
Intermediary Manager Server
Web Server Cluster with Leader Election
Automated File Transfer via FTP
SMTP Client for Email Notifications
Dockerized Deployment

Leader Election Process

Implements the RAFT consensus algorithm for leader election among nodes using UDP sockets.
Runs asynchronously using multithreading or containerized execution.
Handles heartbeat messages to maintain leadership status.
Once elected, the leader communicates updates to the intermediary server.

Message Broker Integration

Uses RabbitMQ as a message broker for inter-service communication.
The scraper publishes data to RabbitMQ queues.
The intermediary server consumes messages and forwards relevant data to web servers.

Intermediary Manager Server

Acts as a middleware between the scraper and web servers.
Listens for messages from RabbitMQ and processes them.
Makes POST requests to the web servers for data updates.

Web Server Cluster with Leader Election

Multiple web servers operate on a shared database.
Uses UDP-based RAFT consensus to elect a leader among them.
The leader updates the intermediary server to manage traffic redirection.

Automated File Transfer via FTP

A separate thread in the intermediary server fetches files from an FTP server every 30 seconds.
The retrieved file is sent as a multipart request to the web servers.
The FTP server is dynamically populated with processed data.

SMTP Client

Implements an SMTP client for sending email notifications.
Ensures successful communication with mail servers.

Deployment

All components are containerized.
Uses Docker Compose to manage services including RabbitMQ and FTP.
Ensures isolated and scalable execution.