Job Title: Senior Linux HPC Engineer
Posted by: SLAC/Stanford on Apr 21, 2024
Location:
Menlo Park , CAJob Description:
Position Overview:
Would you like to configure and troubleshoot modern high-performance Linux clusters? Does contributing to breakthrough discoveries in science and medicine excite you? SLAC National Accelerator Laboratory seeks an energetic, motivated developer-operations engineer who enjoys teamwork, learning cutting edge technologies and engaging the user community.
SLAC is one of the world's premier research laboratories, with internationally leading capabilities in photon science, accelerator physics, high energy physics (HEP), and energy sciences. The Controls & Data Systems (CDS) Division in the Technology Innovation Directorate (TID) is involved in many national and international projects, which, among others, include the Rubin Observatory, the Linac Coherent Light Source (LCLS) user facility, CryoEM user facilities, the LHC ATLAS detector at CERN, and accelerator controls for LCLS-II.
This position will play a critical role in deploying, maintaining and monitoring the large-scale scientific computing infrastructure that supports SLAC's data analysis and Machine Learning capabilities. Thousands of scientists worldwide rely on these systems to perform their research activities. We are seeking a creative, resourceful system administrator that can assist in the configuration, deployment and ongoing maintenance of various platforms and services across hundreds of nodes utilizing the best in class automation platforms. Our working environment is highly collaborative. Skills and responsibilities are shared across our department. We value strong communication and documentation. Host platforms include bare-metal, virtual machines and kubernetes.
We encourage free-thinking open dialog and provide opportunities to explore and implement new technologies and ideas. There is huge potential for career growth. High performance computing is recognized as a SLAC core competency.
Given the nature of this position, SLAC is open to on-site and hybrid work options.
Your specific responsibilities will be to: Lead the management and system administration tasks across hundreds of Linux hosts Maintain and extend our scientific software catalog - working closely with our scientific partners to build workflows and pipelines Architect, administer and tune our batch scheduling systems Lead contribution to developing and standardizing our configuration management platform Help architect and support core monitoring and alerting (notification) capabilities to track health and performance Support day-to-day operations and troubleshooting of scientific computing services and infrastructure Help direct and perform end-user support via our incident platforms and communication channels Maintain all relevant documentation for administration procedures To be successful in this position you will bring: Bachelor's degree in computer sciences, physics or related field and 8 years of relevant experience in information technology, systems administration, or high-performance computing. Proven ability to work effectively in a team environment with excellent organizational and communication skills In-depth technical understanding and proven success partnering with scientific teams to understand and implement large and small scale computational and data-driven workflows Demonstrated ability to lead projects and teams to completion and help drive our scientific mission Extensive experience with Linux system management, monitoring, open-source software Expertise and experience in frameworks and scripting for large distributed systems (ansible, bash and python preferred Expertise with distributed compute and storage systems, high performance computing systems, and networking Proven ability and experience in deploying and troubleshooting full stack software and hardware systems in large complex clustered environments Experience leading and and promoting best practices In addition, preferred experience include: Expert knowledge of high throughput and high performance frameworks and techniques (MPI, containerization, low-level scientific software libraries) Expert knowledge of configuration management systems (ansible preferred) Knowledge of kubernetes primitives and architecture Expert experience managing and configuring Linux and related applications Extensive experience with system and service monitoring (prometheus, influxdb, grafana, loki) SLAC Employee Competencies: Effective Decisions : Uses job knowledge and solid judgment to make quality decisions in a timely manner. Self-Development : Pursues a variety of venues and opportunities to continue learning and developing. Dependability : Can be counted on to deliver results with a sense of personal responsibility for expected outcomes. Initiative : Pursues work and interactions proactively with optimism, positive energy, and motivation to move things forward. Adaptability : Flexes as needed when change occurs, maintains an open outlook while adjusting and accommodating changes. Communication : Ensures effective information flow to various audiences and creates and delivers clear, appropriate written, spoken, presented messages. Relationships : Builds relationships to foster trust, collaboration, and a positive climate to achieve common goals. Physical Requirements and Working Conditions: You are expected to reside locally and work onsite up to 3 days a week Consistent with its obligations under the law, the University will provide reasonable accommodation to any employee with a disability who requires accommodation to perform the essential functions of the job. May work extended hours during peak business cycles. Work Standards : Interpersonal Skills: Demonstrates the ability to work well with Stanford colleagues and clients and with external organizations. Promote Culture of Safety: Demonstrates commitment to personal responsibility and value for environment, safety and security; communicates related concerns; uses and promotes safe behaviors based on training and lessons learned. Meets the applicable roles and responsibilities as described in the ESH Manual, Chapter 1-General Policy and Responsibilities: http://www-group.slac.stanford.edu/esh/eshmanual/pdfs/ESHch01.pdf Subject to and expected to comply with all applicable University policies and procedures, including but not limited to the personnel policies and other policies found in the University's Administrative Guide, http://adminguide.stanford.edu ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Classification Title: System Administrator 3
Grade: K
Job code: 4833
Duration: Regular Continuing
The expected pay range for this position is $119,000 to $150,000 per annum. SLAC National Accelerator Laboratory/Stanford University provides pay ranges representing its good faith estimate of what the university reasonably expects to pay for a position. The pay offered to a selected candidate will be determined based on factors such as (but not limited to) the scope and responsibilities of the position, the qualifications of the selected candidate, departmental budget availability, internal equity, geographic location and external market pay for comparable jobs.
Pay Rate:
HR. Website URL:
Sign Up to Apply to this position
(if you already have a CGO account, just press the button below)
About SLAC National Accelerator Laboratory
As one of 17 Department of Energy national labs, SLAC pushes the frontiers of human knowledge and drives discoveries that benefit humankind. We invent the tools that make those discoveries possible and share them with researchers all over the world. X-rays Reveal the Atomic World Our 2-mile-long particle accelerator is the lab’s backbone. Once the scene of major discoveries in particle physics, today it generates the world’s brightest X-rays for our revolutionary X-ray laser, the Linac Coherent Light Source (LCLS). Thousands of researchers come to SLAC to use LCLS and the Stanford Synchrotron Radiation Lightsource to probe matter in atomic detail. These X-ray studies help scientists understand the fundamental workings of nature and find solutions to real-world problems. Fundamental Science, Practical Benefits When researchers delve into basic details of the world around us, practical benefits often follow. This is true of research at SLAC. In chemistry, “molecular movies” made with our X-ray laser are capturing all the tiny steps of chemical reactions for the first time. This new understanding will help improve reactions that give us fuels, fertilizers and a host of other products. In biology, X-rays reveal how proteins – one of the key molecules of life – function in our bodies and in nature. This research has contributed to the development of medications for melanoma, flu and HIV and is aiding the fight against COVID-19, Ebola, high blood pressure and other ills. SLAC studies of exotic materials with quirky traits could have a profound impact on society, although it may be far in the future. Meanwhile, scientists use our X-ray beams for experiments to improve materials for computer chips, jet planes, refinery operations and “smart windows” that automatically adjust the amount of light coming in, to name a few.Even the accelerator technology developed for basic physics experiments has had a huge impact in medicine and industry, where it shrinks tumors, sterilizes medical supplies and hardens materials, among many other things. SLAC researchers are working to make accelerators much smaller and cheaper so they can accomplish even more. Solving Energy Challenges Many threads of SLAC research come together in the quest for clean, sustainable energy sources. We study how plants make energy from sunlight with an eye to doing the same, and customize chemical reactions for generating clean fuels. Our specialized X-ray equipment allows scientists to watch batteries, solar cells and fuel cells in operation, a crucial step in improving how they work. An Eye on the Cosmos SLAC started more than 50 years ago as a place to discover fundamental particles and forces. Today, our researchers still explore the universe at the largest and smallest scales. At the tiniest scale, we help search for new particles and forces at the Large Hadron Collider in Europe, where the Higgs boson was discovered. At the most sweeping scale, we’re building the world’s biggest digital camera for the widest, deepest survey of the night sky ever undertaken. Our longstanding expertise in building particle detectors is being put to use in experiments that search for dark matter and dark energy, probe the secrets of ghostly neutrinos, look for signs of cosmic inflation and capture high-energy particles from the most violent events in the universe. Key Partnerships Stanford University operates SLAC for the DOE Office of Science. Our five joint research centers and facilities with Stanford focus on cosmology and astrophysics, materials and energy science, catalysis, ultrafast science and cryogenic electron microscopy. SLAC’s location in Silicon Valley and our connections with DOE, Stanford and other leading research centers speed our progress. We also look for ways to work with industry to solve problems and spread the benefits of research out into society.