JOB DESCRIPTION
About You!
- Develop and maintain scalable web crawlers using Scrapy (for static content) and Puppeteer (for dynamic content).
- Deploy and manage crawlers using Docker to ensure consistent, isolated environments and facilitate easy scaling across different systems.
- Implement proxy rotation and rate limiting to avoid IP bans and ensure ethical scraping, while respecting website policies (e.g., robots.txt, terms of service).
- Handle CAPTCHAs using manual intervention or low-cost solutions like Anti-CAPTCHA to minimize disruptions.
- Extract, clean, and standardize data using Python libraries (e.g., pandas, NumPy).
- Design and maintain databases (SQL or NoSQL) to store and manage large volumes of scraped job data, ensuring efficient querying, data integrity, and security.
- Monitor, schedule and update crawlers regularly to adapt to website changes, ensuring continuous data flow.
- Ensure compliance with data privacy laws (e.g., GDPR) and ethical scraping practices.
Your Tool Kit
- At least 3 years of proven experience in web scraping
- Strong programming skills in Python and JavaScript, with experience in web scraping frameworks like Scrapy, Puppeteer, or Selenium.
- Solid understanding of web technologies (HTML, CSS, JavaScript, HTTP protocols) and anti-scraping measures.
- Experience with proxy management and rate limiting for large-scale crawling.
- Hands-on experience with Docker for containerizing and deploying crawlers, ensuring scalability and consistency across environments.
- Proficiency in database management (e.g., SQL, MongoDB) for designing schemas, optimizing queries, and handling large datasets.
- Problem-solving mindset to handle dynamic content, CAPTCHAs, and evolving website structures.
- Attention to detail for data cleaning and ensuring accuracy in extracted job data.
- Ability to work efficiently with limited resources, leveraging free or low-cost tools (e.g., free proxies, open-source libraries).
- Familiarity with data privacy regulations (e.g., GDPR) and ethical scraping practices.
- Proactive and adaptable, with a focus on cost-effective solutions and process optimization.
- Strong mindset in terms of engagement, pro-activity, self-management and taking initiative.
Our Commitment to You
- Training and development allowance to help you learn new skills and advance in your career
- Wellbeing allowance to support your physical and mental health
- Medical insurance to ensure that you have access to quality healthcare
- Tier 3 pension contribution to help you plan for your retirement
- Remote working that allows you to work from home and in the office
- Amazing work culture, that supports you and allows you to grow personally and professionally.
Salary
Competitive
Monthly based
Location
, East Gonja Municipal District, Savannah Region, Ghana, Savannah Region, Ghana
Job Overview
Location
, East Gonja Municipal District, Savannah Region, Ghana, Savannah Region, Ghana