Web Scraping AI Agent

Web Scraping AI Agent was created as a tool to support the team developing the Datenhalle project. It enables rapid creation, fixing, and adaptation of web crawling scripts to content changes, as well as their testing. The key technology here was the use of LLM models.

Responsibilities: I took responsibility for the entire API architecture and AI agent development. I worked on prompt engineering and ensuring the agent effectively utilizes context provided by users. As project leader, I made technical decisions and collaborated with rest of the team to integrate the agent with our web crawling framework.

Challenges: The key challenge was creating an agent that consistently generates functional scripts for diverse web crawling tasks. Particularly difficult was designing a prompt system that handles edge cases and integrating with our proprietary web crawling framework.