Web Scrapper in Python (Horoscope.com)
- PUBLISHED ON: 5 October 2020
- READ TIME: 2 mins
About the Project
The project was made for myself in order to manage my own emails. As you can see there is no account system or authentication system. The security of this service is different and unique, so I don't expect anyone to try to hack it soon. The functionalities that usually should exist, were removed mainly because I didn't need them in my case.
I have started to develop the application and the service mainly because I thought it is a better alternative to use AWS, being free in my case. In the following paragraphs, I will describe how I designed and created this project.
The role of Amazon Web Services
Despite my own colleagues, I have chosen to use the least possible AWS, because I already had a VPS who could process the data.
The most significant role of AWS is to use the Simple Email Service. This service listens for emails and stores them into an S3 bucket then it sends different emails. When an email is stored in a bucket, it is a raw email and it has to be processed to obtain data from it.
When the event of receiving an email occurs a Lambda function starts, the email is processed and the data obtained is sent with an HTTP request to the Email Service. The service has its own database and the stored data are only snippets, mainly for listing the emails and relationships between the emails.
About the Service
Its purpose is to store the email snippets and the devices connected and to expose CRUD routes for the emails. There are special routes for communication with AWS Lambda and special routes for communication with devices, every one of them is secured but I won’t expose how it works.
The device can obtain the emails as snippets, based on multiple criteria, to obtain the full content of an email, to delete an email (which is mainly a pseudo-deletion) and to send an email.
With Firebase Cloud Messaging I was able to send in real-time notifications to every device when an email was processed. The service was written in Node.js with RxJS, MariaDB, aws-sdk, and Mailparser.
About the Application
Angular and Ionic were used for the application. The functionality and design of it can be seen in the video below.