Research and Evaluations Lead (1053) - Department of Technology

Are you passionate about shaping the future of technology while ensuring its responsible and ethical use? The City and County of San Francisco (CCSF) is seeking a forward-thinking and driven Research and Evaluation Lead to join our Emerging Technologies team at the Department of Technology. This is your chance to help shape how artificial intelligence (AI) and other emerging technologies are enabled across CCSF, applying user-centered, data-driven approaches to understand staff needs and support effective and responsible adoption.
In this role, you’ll collaborate with City departments and other partners to identify needs, evaluate emerging technology tools, and turn research findings into practical recommendations. Your work will help guide decisions about what to build, procure, pilot, and scale, as well as how to support staff through training and other enablement efforts.
The Research and Evaluation Lead serves as a key member of the Department of Technology's (DT) Emerging Technologies team. This role advances two primary objectives: staff enablement and program evaluation.
1. Staff Enablement
The Lead is responsible for identifying the needs of City staff related to artificial intelligence and other emerging technology tools. These insights guide the prioritization of tools to build, procure, or pilot, and inform the development of training required for staff to use these tools effectively.
2. Program Evaluation
The Lead evaluates whether emerging technology tools can be deployed effectively and responsibly at scale. This includes advising departments on appropriate evaluation methodologies and collaborating with academic and non-profit partners to design rigorous and practical evaluations.
The position requires strong qualitative user research skills combined with quantitative and mixed‑methods evaluation expertise. The Lead works closely with the Director of Emerging Technologies, Lead AI Engineer, and Principal AI Policy Analyst, while partnering with City departments implementing emerging technology pilots.
The essential functions of this position include:
Conduct qualitative and quantitative research on how City staff use, adopt, or discontinue use of AI tools, to identify friction points, unmet needs, and adoption barriers.
Translate research findings into prioritized input for the Emerging Technologies team, helping inform which tools, features, product builds, pilots, or procurements to invest in.
Assess staff training needs and recommend training or upskilling approaches to help staff use AI tools effectively and responsibly.
Present findings and recommendations to the Emerging Technologies team, department partners, and City leadership.
Design and run evaluations of AI and other emerging technology tools the City procures, builds, pilots, and rolls out — including experiments and quasi-experimental designs where feasible — to determine whether they work, for whom, and with what safeguards they're worth scaling.
Support the City's emerging technology pilot evaluations by building the City's capacity to design and carry out rigorous, practical evaluations through academic and non-profit partnerships.
Pair usage-log data with survey responses and departmental operational metrics to measure actual outcomes, and document methodology and findings to inform future purchasing and rollout decisions.
DESIRABLE QUALIFICATIONS:
Experience with qualitative methods (in-depth interviews, contextual inquiry, focus groups, usability testing) and quantitative methods (survey design, statistics, usage-log or behavioral data analysis).
Experience translating research findings into product or feature prioritization, ideally in a fast-moving or pilot-stage environment.
Experience designing and running program or pilot evaluations, including experimental or quasi-experimental designs (e.g., A/B tests, pre/post comparisons, matched comparison groups).
Experience assessing training or skill-development needs and translating them into training recommendations.
Experience with survey tooling (e.g., Fillout, Qualtrics) and data analysis (Excel/Sheets required; SQL, R, or Python a plus).
Experience maintaining and leveraging academic and non-profit research partnerships.
Ability to manage research and evaluation projects in fast-moving, ambiguous environments, balancing rigor with practical constraints, shifting priorities, and tight timelines.
Strong written and verbal communication skills, with the ability to communicate findings across technical and non-technical audiences.
Experience in public service or civic technology, and working with staff across different job classes, tenure levels, and language backgrounds.
MINIMUM QUALIFICATIONS:
These minimum qualifications establish the education, training, experience, special skills and/or license(s) which are required for employment in the classification. Please note, additional qualifications (i.e., special conditions) may apply to a particular position and will be stated on the job announcement.
Experience:
Three (3) years in the information systems field, including system analysis, business process design, development and implementation of business application solutions or IT project management.
Education:
An associate degree in business administration, public administration, information systems, economics, finance, computer science, UX/UI design or a closely related field from an accredited college or university OR its equivalent in terms of total course credits/units [i.e., at least sixty (60) semester or ninety (90) quarter credits/units with a minimum of twenty (20) semester or thirty (30) quarter credits/units in one of the fields above or a closely-related field].
Substitution:
One year of additional experience as described above may be substituted for the required degree.
Completion of the 1010 Information Systems Trainee Program may be substituted for the required degree.
Verification of Education and Experience
Please make sure it is clear in your application exactly how you meet the minimum qualifications. Every application is reviewed to ensure that you meet the minimum qualifications as listed in the job ad. Review SF Careers Employment Applications for considerations taken when reviewing applications.
Applicants may be required to submit verification of qualifying education and experience at any point during the recruitment and selection process. If education verification is required, information on how to verify education requirements, including verifying foreign education credits or degree equivalency, can be found at https://careers.sf.gov/knowledge/experience-education/.
Note: Falsifying one’s education, training, or work experience or attempted deception on the application may result in disqualification for this and future job opportunities with the City and County of San Francisco.

Employment type
Full-time

Work arrangement
No