6 Jun 2025 - Volkan Kutal
In an era where AI systems drive recruitment, recommendation engines, and critical decision-making processes, even the smallest piece of external data can be weaponized. In our latest project, we combined the AI Recruiter with the XPIA Attack. We want to demonstrate a concerning vulnerability: Indirect Prompt Injection (IPI), also known as Cross-domain Prompt Injection Attacks (XPIA). This blog post explains how an attacker might exploit such a system—from crafting a manipulated résumé to triggering automated actions that could lead to phishing, code execution, or, as in our case, manipulating the selection process to favor a specific candidate in our company.
Understanding the AI Recruiter Project¶
The AI Recruiter is designed to match résumés with job descriptions using GPT-4o, while also serving as a testing ground for Retrieval-Augmented Generation (RAG) vulnerabilities through Cross-domain Prompt Injection Attacks (XPIA). It integrates ChromaDB for semantic search, while also leveraging PyRIT’s XPIA Attack to automate attacks, allowing AI red-teaming workflows for security research in AI pipelines.
Key Features¶
Résumé Processing & Semantic Matching: Résumés are extracted from PDFs, with embeddings generated using models like text-embedding-ada-002. These embeddings enable semantic matching, while GPT-4o is later used to assign a match score based on relevance and extracted content.
Automated RAG Vulnerability Testing: Attackers can manipulate résumé content by injecting hidden text (via a PDF converter) that optimizes scoring, influencing the AI Recruiter’s ranking system.
XPIA Attack Integration: PyRIT enables full automation of prompt injections, making AI vulnerability research efficient and reproducible.
The Exploit in Detail: Step-by-Step¶
1. Choosing the Target Job¶
An attacker identifies a job posting they wish to apply for. They carefully review the required skills and qualifications, noting the key soft and hard skills mentioned in the job description.
2. Crafting the Manipulated Résumé¶
Instead of submitting a standard résumé, the attacker uses the PDF converter integrated in PyRIT to inject hidden content into their résumé. The same PDF converter is also an integral part of the XPIA Attack’s workflow, ensuring that the manipulation is both stealthy and effective. Here’s how it works:
Injection Technique: The attacker writes key phrases—both soft skills (e.g., “team leadership”, “communication”) and hard skills (e.g., “Python”, “machine learning”)—directly derived from the job description. Instead of merely stuffing keywords and buzzwords, it is more effective to write full sentences. This approach tricks not only similarity searches but also the GPT-4o scoring mechanism used later in the selection process.
Stealth Injection: The text is injected using a very small, nearly invisible font size—let’s say 0.1px—and the same font color as the résumé background. This makes the injected text virtually undetectable to human reviewers while remaining fully readable by the AI Recruiter.
3. Uploading via XPIA Attack¶
Once the résumé has been manipulated by the PDF converter, it is uploaded through the XPIA Attack:
Automated Flow: The attack takes over, automatically passing the résumé into the AI Recruiter. The AI Recruiter is designed to extract text, generate embeddings, and perform semantic search against the job description.
Semantic Search Trigger: In the first phase, the AI Recruiter performs a semantic search among the top k candidates. Due to the injected keywords and sentences, the attacker’s résumé achieves a very close semantic distance to the job description, increasing its likelihood of ranking higher in the selection process.
4. AI Evaluation with GPT-4o¶
The next stage involves in-depth evaluation:
Detailed Scoring: GPT-4o reviews the résumé and assigns a match score. Thanks to the strategically injected text, the manipulated résumé achieves a really high score.
Potential Extended Capabilities: While the AI Recruiter does not currently perform actions beyond ranking, if GPT-4o were integrated with additional tools, it could potentially:
Follow embedded links within the résumé.
Retrieve external information about the candidate (e.g., from GitHub or personal websites).
Analyze or execute code from linked repositories.
Auto-generate summaries that are forwarded directly to HR. Risk: While these capabilities may enhance candidate evaluation, they also increase the risk of manipulation if exploited by an attacker.
5. The Phishing Risk¶
The consequences of manipulation do not stop at résumé scoring:
Misleading Summaries & Links: AI-generated summaries may include links embedded in résumés, which HR might follow without realizing the potential risks. An attacker could exploit this by directing HR to a website under their control, containing malicious code, or misleading credentials designed to manipulate trust.
Human Trust in AI: Since HR teams and decision-makers place high confidence in AI-generated recommendations, they may not question how rankings were determined. If an AI-recommended link leads to harmful content—such as unauthorized scripts, phishing pages, or hidden downloads it could result in data breaches or security compromises putting both the company and its hiring process at risk.
Beyond Recruitment: The Broader Threat Landscape¶
While our demonstration focused on AI-driven recruitment, the underlying vulnerability extends to any AI system that processes external data for decision-making. Microsoft’s AI Red Teaming efforts, as outlined in the AI Red Team Lessons eBook, have identified similar risks, including prompt injection, manipulated feedback loops, and unauthorized automation. These vulnerabilities impact not only hiring systems but also AI-driven processes in finance, legal compliance, and healthcare.
Manipulated AI-Generated Summaries in Decision Systems¶
AI-generated summaries are widely used in hiring, legal case analysis, financial risk assessments, and medical diagnostics.
Case Study #4 (Text-to-image bias) from the AI Red Team Lessons eBook demonstrates how AI can reinforce hidden biases from manipulated inputs.
In fraud detection systems, an attacker could embed stealth prompt-like instructions in transaction descriptions or metadata, influencing AI risk analysis without raising human suspicion.
Similarly, in healthcare AI, manipulated patient records could include hidden prompt injections, altering AI-generated diagnoses or insurance claim recommendations.
Large-Scale Automated Manipulation in AI Systems¶
Attackers can mass-generate structured documents—such as résumés, loan applications, insurance claims, or academic records—embedding stealth-injected content that AI systems prioritize.
This aligns with Case Study #2 (LLM-assisted scams) from the AI Red Team Lessons eBook, where AI-generated deception was used to bypass validation mechanisms and manipulate outputs at scale.
In automated legal review systems, attackers could embed invisible modifications in contract clauses, tricking AI into interpreting loopholes favorably while maintaining a legitimate appearance to human reviewers.
Conclusion¶
Returning to our demonstration with the XPIA Attack and AI Recruiter, we’ve exposed a fundamental weakness: external data can be weaponized to manipulate AI-driven decision-making. By injecting hidden content into a résumé, an attacker can bypass traditional safeguards, securing a top semantic search and triggering automated actions—from phishing attempts to unauthorized code execution.
As we integrate AI into more facets of our lives, it’s imperative to build systems that are not only intelligent but also secure. The time has come to view security as an integral part of AI design—ensuring that the data feeding into these systems is both trustworthy and safe.
Explore More: