Principal QA Engineer – GenAI Testing - Manager
State Street
Principal QA Engineer – GenAI Testing - Manager
Job Description
Job DescriptionWho we are looking forWe are looking for Software Quality Assurance testers with 8+ years of experience, specializing in LLM (Large Language Model) testing and Agents testing. This role involves analyzing business and technical specifications, defining and executing test cases, and capturing and tracking defects.You will work on Artificial Intelligence initiatives within a multidisciplinary team focused on emerging trends, cutting-edge technologies, and their application in financial services for our clients.This role supports a hybrid work model, allowing you to balance work from home and office based on your needs and role requirements.What you will be responsible forAs an SQA Tester, you will:Analyze business and technical specifications.Write functional test cases and perform regression testing.Report bugs, verify fixes, and maintain the bug tracking system (e.g., Jira).Maintain and execute automated tests using Selenium with Python.Create automated test scripts using the Pytest framework.Automate web services using SoapUI and/or Python.Design and execute test plans for Large Language Models (LLMs), including prompt validation, accuracy checks, and safety compliance.Perform Agents testing for AI-driven workflows to ensure reliability, robustness, and integration quality.What we valueThese skills will help you succeed in this roleAbility to understand customer/business user needs, motivations, and behaviors and translate them into concepts and prototypes to elicit direct feedback.Strong analytical, technical, problem-solving skills along with being detail-oriented, and able to understand complex customer needs and translate them into end-user products/automation.Ability to manage your own time by appropriately prioritizing design and research tasks while working within deadlines.Experience with BDD & TDD (Gherkin)Experience in using Jira as agile management tool.Experience in using selenium with python for test automationExperience in testing Large Language Models (LLMs), including validation of prompt responses, evaluation of model accuracy and relevance, and automation of LLM-based workflowsRed teaming of LLMs to identify vulnerabilities and edge cases through adversarial prompts.Safety validation and ethical compliance testing of LLM behavior.Familiarity with tools and frameworks for LLM evaluation such as OpenAI Evals, Ragas, and custom benchmarking suites.Education & Preferred QualificationsThe candidate must have 8 years of software quality assurance testing experience.Experience in tools like PyCharm, Visual Studio Code, Git, Fiddler, Postman, SoapUI, JMeter, Jenkins, Jira is required.Should have experience in CI/CD-based projects.Hands-on experience in LLM testing and Agents testing is highly preferred.Additional RequirementsBasic understanding of artificial intelligence and machine learning algorithms is preferred.Familiarity with AI governance, bias detection, and safety validation will be an added advantage.About State StreetAcross the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you’ll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.Discover more information on jobs at StateStreet.com/careersRead our CEO Statement
Locations
- India, Hyderabad
Skills Required
- AI governanceintermediate
- tools like PyCharmintermediate
- CI/CD-based projectsintermediate
- LLM testingintermediate
Required Qualifications
- Basic understanding of artificial intelligence and machine learning algorithms is preferred. (experience)
- Familiarity with AI governance, bias detection, and safety validation will be an added advantage. (experience)
- Basic understanding of artificial intelligence and machine learning algorithms is preferred. (experience)
- Familiarity with AI governance, bias detection, and safety validation will be an added advantage. (experience)
Preferred Qualifications
- The candidate must have 8 years of software quality assurance testing experience. (experience, 8 years)
- Experience in tools like PyCharm, Visual Studio Code, Git, Fiddler, Postman, SoapUI, JMeter, Jenkins, Jira is required. (experience)
- Should have experience in CI/CD-based projects. (experience)
- Hands-on experience in LLM testing and Agents testing is highly preferred. (experience)
- The candidate must have 8 years of software quality assurance testing experience. (experience, 8 years)
- Experience in tools like PyCharm, Visual Studio Code, Git, Fiddler, Postman, SoapUI, JMeter, Jenkins, Jira is required. (experience)
- Should have experience in CI/CD-based projects. (experience)
- Hands-on experience in LLM testing and Agents testing is highly preferred. (experience)
Responsibilities
- Analyze business and technical specifications.
- Write functional test cases and perform regression testing.
- Report bugs, verify fixes, and maintain the bug tracking system (e.g., Jira).
- Maintain and execute automated tests using Selenium with Python.
- Create automated test scripts using the Pytest framework.
- Automate web services using SoapUI and/or Python.
- Design and execute test plans for Large Language Models (LLMs), including prompt validation, accuracy checks, and safety compliance.
- Perform Agents testing for AI-driven workflows to ensure reliability, robustness, and integration quality.
- Analyze business and technical specifications.
- Write functional test cases and perform regression testing.
- Report bugs, verify fixes, and maintain the bug tracking system (e.g., Jira).
- Maintain and execute automated tests using Selenium with Python.
- Create automated test scripts using the Pytest framework.
- Automate web services using SoapUI and/or Python.
- Design and execute test plans for Large Language Models (LLMs), including prompt validation, accuracy checks, and safety compliance.
- Perform Agents testing for AI-driven workflows to ensure reliability, robustness, and integration quality.
- Ability to understand customer/business user needs, motivations, and behaviors and translate them into concepts and prototypes to elicit direct feedback.
- Strong analytical, technical, problem-solving skills along with being detail-oriented, and able to understand complex customer needs and translate them into end-user products/automation.
- Ability to manage your own time by appropriately prioritizing design and research tasks while working within deadlines.
- Experience with BDD & TDD (Gherkin)
- Experience in using Jira as agile management tool.
- Experience in using selenium with python for test automation
- Experience in testing Large Language Models (LLMs), including validation of prompt responses, evaluation of model accuracy and relevance, and automation of LLM-based workflows
- Red teaming of LLMs to identify vulnerabilities and edge cases through adversarial prompts.
- Safety validation and ethical compliance testing of LLM behavior.
- Familiarity with tools and frameworks for LLM evaluation such as OpenAI Evals, Ragas, and custom benchmarking suites.
- Ability to understand customer/business user needs, motivations, and behaviors and translate them into concepts and prototypes to elicit direct feedback.
- Strong analytical, technical, problem-solving skills along with being detail-oriented, and able to understand complex customer needs and translate them into end-user products/automation.
- Ability to manage your own time by appropriately prioritizing design and research tasks while working within deadlines.
- Experience with BDD & TDD (Gherkin)
- Experience in using Jira as agile management tool.
- Experience in using selenium with python for test automation
- Experience in testing Large Language Models (LLMs), including validation of prompt responses, evaluation of model accuracy and relevance, and automation of LLM-based workflows
- Red teaming of LLMs to identify vulnerabilities and edge cases through adversarial prompts.
- Safety validation and ethical compliance testing of LLM behavior.
- Familiarity with tools and frameworks for LLM evaluation such as OpenAI Evals, Ragas, and custom benchmarking suites.
Tags & Categories
Answer 10 quick questions to check your fit for Principal QA Engineer – GenAI Testing - Manager @ State Street.
10 Questions
~2 Minutes
Instant Score