LLMs and Online Privacy: Toward Automated Relevance Assessment

Abstract

As digital footprints grow, individuals face increasing challenges in understanding and managing the personal information available about them online. The OPP tool addresses this by retrieving public URLs related to a user's identity. This thesis enhances OPP by integrating a Large Language Model to automatically assess the relevance of these search results. The project explores how prompt engineering and contextual information, such as user profiles and filters, can improve LLM-based relevance scoring. A redesigned system architecture was implemented, featuring components for prompt generation, result ranking, and user interaction. Five prompt strategies were designed, and three of them were evaluated through system-based and user-based experiments. Results showed that the LLM could personalize relevance scoring effectively, though it exhibited sensitivity to negative context and inconsistencies across prompts. The evaluation also identified trade-offs in execution time, suggesting practical considerations for real-world deployment. Overall, this work contributes to ongoing research at the intersection of LLMs, information retrieval, and privacy tools by demonstrating how automatic relevance assessment can empower users to gain clearer insight into their digital presence.