Web Crawler Developer (Azure Stack)
RAPSYS TECHNOLOGIES PTE. LTD.
17 hours ago
Posted date17 hours ago
N/A
Minimum levelN/A
Key Responsibilities
Key Skills & Qualifications
- Design, develop, and maintain web crawlers/scrapers to extract structured data from various partner websites.
- Handle complex HTML structures, pagination, and anti-bot mechanisms to ensure reliable data capture.
- Implement data transformation, cleaning, and enrichment pipelines for extracted datasets.
- Integrate with Azure services such as Azure Functions, Azure Storage, Azure Logic Apps, or Azure Data Factory for automation and orchestration.
- Utilize Vector Databases (e.g., Azure Cognitive Search, Pinecone, FAISS, or Weaviate) for semantic indexing and retrieval.
- Incorporate multilingual translation APIs (e.g., Azure Translator, Google Translate) to produce multi-language datasets.
- Ensure consistency in JSON structuring, REST API integrations, and maintain proper version control using Git.
Key Skills & Qualifications
- Proficiency in Python, with experience using BeautifulSoup, Requests, and Selenium.
- Strong understanding of web scraping best practices and data processing workflows.
- Hands-on experience with Azure-based automation and data orchestration tools.
- Familiarity with vector search technologies and semantic retrieval methods.
- Knowledge of translation and localization APIs for multilingual data generation.
- Solid understanding of Git-based version control and RESTful architecture.
JOB SUMMARY
Web Crawler Developer (Azure Stack)
RAPSYS TECHNOLOGIES PTE. LTD.
Singapore
17 hours ago
N/A
Contract / Freelance / Self-employed
Web Crawler Developer (Azure Stack)