Responsibilities:

• Data Crawling & Extraction:
• Design, develop, and maintain high-performance data crawling systems using Python, libraries like Scrapy, Beautiful Soup, and other relevant tools.
• Extract data from various sources, including public APIs (e.g., CoinGecko, CoinMarketCap), social media platforms (e.g., Twitter, Reddit), news feeds, and blockchain networks (e.g., Ethereum, Bitcoin).
• Handle challenges such as rate limiting, dynamic content, and website changes.
• Implement robust error handling and recovery mechanisms.
• Data Processing & Transformation:
• Clean, transform, and enrich extracted data using data cleaning techniques, data transformation libraries, and SQL.
• Structure and normalize data for efficient storage and analysis.
• Develop and maintain data pipelines for data ingestion, transformation, and loading into data warehouses or data lakes.
• Data Quality & Validation:
• Implement data quality checks and validation rules to ensure data accuracy and completeness.
• Monitor data sources for changes and updates, and adjust crawling strategies accordingly.
• Conduct regular data audits and reconciliation checks to identify and resolve data inconsistencies.
• Data Storage & Management:
• Design and implement efficient data storage and management solutions, including databases, data warehouses, and data lakes.
• Optimize data storage and retrieval for fast query performance.
• Research & Development:
• Stay abreast of the latest advancements in data crawling, web scraping, data extraction, and anti-scraping techniques.
• Explore and implement new technologies and tools to improve data collection efficiency and effectiveness.

Requirements:

• 5+ years of experience in data crawling, web scraping, or a related field.
• Bachelor's degree in Computer Science, Computer Engineering, or a related field.
• Strong proficiency in Python or Java, and experience with relevant libraries (e.g., Scrapy, Beautiful Soup, Selenium, Playwright, Nutch, Apify，Puppeteer).
• Experience with data extraction techniques, including API calls, web scraping, and data mining.
• Understanding of data structures, algorithms, and database concepts.
• Experience with data processing and transformation tools (e.g., Pandas, SQL).
• Familiarity with cloud platforms (GCP, AliCloud, AWS) and cloud data services is a plus.
• Experience with distributed systems and parallel processing is a plus.
• Strong analytical and problem-solving skills with a focus on data-driven solutions.
• Excellent communication and documentation skills.
• Experience in the financial services or fintech industry, particularly within the crypto/blockchain space, is a strong plus.

All applications applied through our system will be delivered directly to the advertiser and privacy of personal data of the applicant will be ensured with security.