OverviewAre you a Senior Software Engineer who thrives in a cutting-edge environment where you get to work with some of the largest data volumes processing? Are you excited to craft the most reliable data processing pipelines with Spark and Flink to process PiB level data every day? Are you passionate about using artificial intelligence tools and practices across the software development lifecycle and data processing? The Data and Insight team in Search Foundation organization leverages open-source technology to process our petabyte scale data of Bing and MSN logs and provide comprehensive insight into experiments. The team also supports the grounding business with Realtime signals and insight information and protects the Bing search through offline bot protection analysis and model training. Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond. ResponsibilitiesYou are a seasonal leader in building scalable data analytics/insights platform with PiB level data processing every day. You have a growth mindset to learn the cut-edge technology and apply it in daily work to change the status quo. You dive deep and resolve the challenging technical problem. You insist on the highest standard for system performance and reliability. You will be working on Bing mission critical analytical data and support the grounding business; processing at big data scale and need for generating real-time insights. You will be working in a hybrid technology space that combines the best from Azure and open-source world. Other responsibilities may include: Design, build and own the optimal data processing architecture and systems for new data and analytics pipelines, dashboards and metrics leveraging Azure and open-source technologies Build core datasets as well as scalable and fault-tolerant pipelines Build data anomaly detection, data quality checks, and optimize pipelines for ideal compute and storage systems while guaranteeing high quality and SLA datasets Understand the data compliance and data governanceCollaborate with cross functional partners including data scientists and product managers to design and develop end-to-end data assets that help improve our products. Hand on at least one data process technology like Spark, Flink, and message queue technology like Kafka, EventHubHand on data storage technology like HDFS, Azure blob etcLeverage AI technology in coding/testing/Data processing/data validation.QualificationsRequired Qualifications: Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or PythonOR equivalent experience.Preferred Qualification:6+ years of experience in one or more coding languages (C#, C++, Java) Familiar at least one data process technology like Spark, Flink, and message queue technology like Kafka, EventHub Have experience on cloud environment like Azure Have experience on data storage technology like HDFS, Azure blob etc Have experience in leveraging AI technology in coding or testing work. Fluence verbal and written English communication. 6+ years’ experience in backend service development, 2+ years’ experience in batch or streaming data processing to process the PiB level data. Domain expert on Spark or Flink to handle PiB level data every day. Experience in data pipeline performance optimization Experience to use AI in data processing and data validation Data compliance or governance experience is a plus Software Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $160,200 - $261,000 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:https://careers.microsoft.com/us/en/us-corporate-payThis position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.