Research Portal

Early Warning Systems (EWS) developed in response to No Child Left Behind and its focus on graduation and accountability. EWS developed in diverse contexts, with rural, town, and city implementations. They are becoming ubiquitous since many Student Information Systems and Statewide Longitudinal Data Systems have embedded EWS data tools. Few research studies focus on the processes involved, policy or otherwise, of EWS implementation. This technical report focuses on the processes apparent in schools that have implemented the program in Montana including early identification, longitudinal analysis, and progress monitoring. Schools implement the model in various ways. The scope and intensity of the model varies on vision, perceived value, and dissemination. By putting this in the context of a rural state, we are able to see how statewide policies can democratize access and use of the model.

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305S210011 to the Montana Office of Public Instruction. The opinions expressed are those of the author and do not represent views of the Institute or the U.S. Department of Education.

Executive Summary

Technical Report

For over 50 years education policy has been guided by an insufficient understanding of economic disadvantage. One component involves poverty measures as seen in the use of free and reduced-price meal data - National School Lunch Program eligibility (NSLP). Since the incorporation of eligibility data into policy in the 1970’s, there have been questions as to which poverty measure to choose in what context. NSLP eligibility largely filled that gap. Nonetheless, NSLP eligibility data has many emerging insufficiencies, including over identification of students, inaccurate income information, and inaccurate accounting of poor students in Community Eligible Provision districts (Geverdt & Nixon, 2018). Most districts and schools participate in the USDA program. However, the number of schools which do not participate in NSLP in Montana is significant (97 do not have claims data in March 2019). To assess these challenges and potential alternatives, we measure eight alternative poverty measures to see which ones meet the historical standard of NSLP. This study analyzes the use case of Montana (statewide data) for measures such as NSLP eligibility, participation, direct certification, and SAIPE.

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R372A200011 to the Montana Office of Public Instruction. The opinions expressed are those of the author and do not represent views of the Institute or the U.S. Department of Education. 

Statewide

Summary

The challenges of National School Lunch Program data stem from over identification of students, inaccurate income data, issues with coverage in rural areas, and difficulties in accounting for Community Eligibility Provision districts. Recent research literature focuses on the need for an alternative poverty measure. Six poverty measures are used to gauge differences and assess alternatives. By applying a statistical model that uses student outcome measures as predictors and adding/removing controls (poverty measures) the consistency of alternative poverty measures becomes apparent. The Spatially Interpolated Demographic Estimates prove the most consistent and suitable of the alternative measures across locales. Evidence of this occurs with linear regressions involving similar sign, significance, and magnitude as the meal data across city, town, and rural areas.

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R372A200011 to the Montana Office of Public Instruction. The opinions expressed are those of the author and do not represent views of the Institute or the U.S. Department of Education.

Locale

Summary

Variation in income and educational outcomes in rural schools can be calculated based on the distance students live from school. This difference is important since many demographic indicators in many geographies are homogenous, however income is not. Near factors (students living close to school) are highly correlated to NSLP and explain to a greater degree variation in student outcome data in Cities. The analysis in Rural Remote areas is complex since income to poverty ratios for far students are much lower than for near students. Data pertinent to students ‘in town’ in rural remote contexts may not be sufficient to explain school level poverty and student outcome trends.

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R372A200011 to the Montana Office of Public Instruction. The opinions expressed are those of the author and do not represent views of the Institute or the U.S. Department of Education.

Proximity

Summary

Consortiums of education institutions develop based on shared challenges and a common mission. The Circle of Schools was convened on the Northern Cheyenne Indian Reservation in 2009 to bring together regional community, school, and tribal leadership for the purpose of working together to support student success. The common challenges faced by participants include rurality and poverty. Of particular importance is Native student success throughout the consortium as challenges among this population are unique and persistent. A common belief amongst participants is a shared cultural understanding and desire to educate all students to their greatest potential.

The purpose of this document is to provide data which can illustrate the context behind the participating K‐12 institutions (private, public, and tribal). The data provided in this report is by no means a comprehensive summary of all the relevant data from these institutions. It simply represents data that is available at the Montana Office of Public Instruction. The report does provide essential student and teacher characteristics‐ data which can be used to help interpret and address the challenges and opportunities for students throughout Rosebud County. It focuses on district level reporting rather than school level (both elementary districts and high school) to better account for the division in elementary districts between the lower grades and the 7th and 8th grade schools. This 2022 report is a follows up on a 2020 document that used SLDS data to describe schools in Rosebud County.

Circle of Schools Narrative (10/14/2022)

Circle of Schools Narrative (3/13/2020)

On a biennial basis, the OPI produces a quantitative report detailing the American Indian student achievement gap. This report details student outcomes for assessments (SBAC and ACT) and graduation/dropout data points. In order to expand on this report to answer questions about factors influencing the gap, the American Indian Student Achievement (AISA) unit and the Statewide Longitudinal Data Systems unit have launched a three phase study of the achievement and graduation gap between American Indian students that live on- and off-reservation, and their peers throughout the state of Montana.

 

Phase one was a focus group that collected data from 45 stakeholders on-reservation and in off-reservation schools with a high concentration of American Indian students. We found that out-of-school effects were cited nearly as frequently as in school effects. Additionally, the impact of poverty and family and home-life circumstances were more often cited as explanatory factors than issues involving attendance, mobility and dropout. In forthcoming phases (a large-scale survey and interviews with indigenous scholars) these and other areas of inquiry from phase one will be examined in further detail and reported here.

 

Please find below two documents: our executive summary of the focus group findings, and a full technical report.

Executive Summary of the Focus  Group Findings

Full Technical Report

SBAC Interim Report (SY 2018 – 2019)

This report prepared for the Assessment unit in the Montana Office of Public Instruction looks at data from the SBAC Interim Assessment and compares it with data on the SBAC Summative Report. The first analyses compare scores among those students that took the SBAC Interim and Summative. The second analyses compare those scores on the Summative of participating and nonparticipating students. Results were strongly in favor of the use of the SBAC Interim Assessments as a means to gauge student progress mid year, provide feedback to the instructor about each student’s progress, and provide the means to adjust instruction to better meet students needs and help them preform well on the SBAC Summative exam. In ELA, students who took the test scored higher on the Summative, indicating progress between the two time points the tests where taken. In both ELA and math, students that took the SBAC Interim scored significantly higher on the Summative than nonparticipating students. This occurs even though interim test takers were more likely to receive Free or Reduced Lunch.

SBAC Interim Report (SY 2017 – 2018)

This report prepared for the Assessment unit in the Montana Office of Public Instruction compares students scores on the SBAC Interim with the same students scores on the Summative. It looks to the relative effectiveness of each Interim Assessment Block (IAB) and compares it with the summative data found in the SLDS. This research was conducted using all instances a student took a SBAC Instructional Assessment Block (IAB) in the State of Montana. The numbers of students who took at least one IAB is large – 12,491. Most students took the assessment in grade 3-5. Two grade levels stand out as achieving especially meaningful gains for each IAB. The majority of the third grade IABs showed that average student score improved significantly in 5 of the 7 IAB tests.  Grade 6 also experienced meaningful positive changes within 3 of 5 IABs measured. The range for the scale scores is 2280 – 2680. The takeaway from this research is that average student scores show significant gains for those students taking an Interim assessment. This could be due to the student having more experience testing or the teacher using the test to guide instruction, or both.

Brazill, S. (2019). The Relationship of Gender, School Attendance, and Grade Level with ACT English and Composite Test Scores. In G. Marks (Ed.), Proceedings of Global Learn 2019-Global Conference on Learning and Technology (pp. 64-69). Princeton-Mercer, New Jersey: Association for the Advancement of Computing in Education (AACE). Retrieved October 6, 2020 from https://www.learntechlib.org/primary/p/210291/ [learntechlib.org].

Factors that predict ACT science scores from a multicultural perspective. Educational Research: Theory and Practice, 30(2), 1-16.

His study investigated predictors for ACT Science scores, a test used by many universities to rank applicants. This study utilized quantitative research methods using the Montana Office of Public Instruction’s GEMS (Growth and Enhancement of Montana Students) data set. All advanced statistical analysis was conducted using Stata software IC/15. This research is significant for increasing the representation of under-represented groups in STEM education because it helps clarify three important relationships: (1) How well do gender, race, and meal status predict 11th grade ACT Science scores; (2) How well does school size predict 11th grade ACT Science scores while controlling for gender, race, and meal status; and (3) How well does high school GPA predict 11th grade ACT Science scores while controlling for gender, race, meal status, and school size.

This manuscript utilized quantitative research methods to analyze the Montana Office of Public Instruction’s GEMS (Growth and Enhancement of Montana Students) dataset. This research is important because it helps clarify three important relationships: (1) Is there a statistically significant difference in the average 11th grade English scores between males and females; (2) Does the number of days present at school influence students’ 11th grade ACT composite scores; and (3) Do individual students’ ACT English scores improve from 11th grade to 12th grade.

Brazill, S. C. (2020). Race and gender factors in ACT English and composite scores. Educational Research: Theory and Practice, 32(1), 17-28.

This study utilized quantitative research methods using the Montana Office of Public Instruction’s Growth and Enhancement of Montana Students (GEMS) data set. This quantitative research is important because it provides insight on the following relationships (1) Is there a statistically significant difference among students from different races on their mean 11th grade ACT English scores?; (2) Is there an interaction between gender and race on students’ mean 11th grade ACT English scores?; and (3) Is there a statistically significant difference among American Indian students mean ACT composite score from their Junior to their Senior year.

Paper Title: The Effects of State Mandated Financial Education on College Financial Behavior

Authors: Christina Stoddard, PhD & Carly Urban, PhD

Affiliation: Department of Economics, Montana State University

Funded by: Montana Office of Public Instruction and the National Endowment for Financial Education

This paper, which also has been adapted into an executive summary, examines the effect of high school personal finance graduation requirements on student decisions about how to finance college. Personal finance requirements exist in many different forms in twenty-five states. The authors conduct an analysis to evaluate the factors that affect financial decision making during their freshman year. They compare students that are in states with those requirements to students in states where financial education is not required for graduation. Montana is examined in a case study as a state which does not have state required financial education. The analyses are informed by data from the National Post-Secondary Student Aid Study (NPSAS), Current Population Survey (CPS), Integrated Post-Secondary Education Data Systems (IPEDS), and data from the Montana Office of Public Instruction and the Montana University System.

The analysis uses data from 25 states. This approach is used “to compare incoming freshman at four year institutions from states with personal finance requirement before and after implementing the requirement to comparable students whose states lack a mandate” (Stoddard & Urban,2018, p. 2).

Stoddard & Urban (2018, p.1) suggest the following findings from their study:

• Financial education graduation requirements shift students from high-cost to low-cost financing behaviors.
• Requirements increase federal aid applications and acceptance of low cost federal student loans.
• Requirements decrease the likelihood of students holding credit card balances.
• Students from less affluent family backgrounds further reduce their likelihoods of working while enrolled.
• Borrowers from more affluent backgrounds reduce private loan amounts.
• The mandates do not change college attendance or choice of institution type.

The data from Montana suggest:

• There are no statistical differences across students in schools that do and do not offer personal finance course (Stoddard & Urban, 2018, p. 14)
• There are no visible patterns in schools offering and not offering financial education based on their geography or distance from main highways in the state (Ibid).
• School choices to add a personal finance course are idiosyncratic (Stoddard & Urban, 2018, p. 32).
• There are no statistical differences in having subsidized Stafford loans, having unsubsidized Stafford loans or having a grant (Ibid).

There are several implications to the study. Personal finance graduation requirements improve student financial behaviors in college. The benefits may extend beyond college financing decision. The report may be downloaded from the GEMS website.

"College Enrollment and STEM Major Choice in a Rural State: A Statewide Examination of Recent High School Cohorts, examined both institution type enrollment (2-year v. 4-year) and college major selection (STEM v. non-STEM) using the GEMS random data set of high school cohorts graduating in 2013-2017. The data set was released under a cooperation among the Office of the Commissioner of Higher Education, the OPI, and Dr. Fenqjen Luo at the Department of Education, MSU. We are only reporting findings for students who stayed in Montana for college. The study examined 3,119 students and found that Montana students are more likely to enroll into a 4-year institution than a 2-year institution (79.6% v. 20.4%). Also, students who enrolled at a 4-year institution are more likely to consider a STEM major than students at a 2-year institution. When examining institution type, significant explanatory variables included high school GPA, ACT STEM score, ACT English score, free/reduced-price lunch participation, and race/ethnicity. An interesting finding showed that American Indian/Alaska Native students were more likely to enroll into a 4-year institution than White students. When examining major choice, only high school GPA, ACT STEM score, gender, and institution type were significant predictors of selecting a major in a STEM field. Specifically, male students were over three times as likely to select a major in a STEM field when compared to their female counterparts. The authors also discuss limitations and recommendations for future research. The study is currently under review in Theory & Practice in Rural Education. 

This research was conducted at MSU by two doctoral students, Que N. Tran and Monte Meyerink, and two faculty members, Dr. Alexandra Aylward and Dr. Fenqjen Luo. Additionally, we acknowledge the excellent support from Dr. Robin Clausen at OPI and OCHE during this research project. This research was funded by the grant, "Collaborative for Continuous Improvement in Education: Montana GEMS Data and the K-16 Pathway," from the Montana Office of Public Instruction for the College of Education, Health, and Human Development, MSU during the academic years 2017-2019 (Dr. Tricia Seifert, Principal Investigator). 

Obery, A., Sletten, J., Vallor, R. & Schmitt-Wilson, S. (2020) Data Driven Decision Making in Teacher Education: Perceptions of Pre-Service Teachers and Faculty Who Teach Them, Action in Teacher Education, DOI: 10.1080/01626620.2020.1762139 [doi.org]

Integrating data driven decision making into teacher preparation programs has been called for by several researchers in order to help lessen the barriers teachers face in using data in their teaching. This mixed methods study examined the perceptions of data use in education by pre-service teachers (N = 112) and perceived knowledge and skills required for successful use from both pre-service teachers (N = 6) and the faculty who educate them (N = 5). Results have implications for teacher preparation programs considering data literacy integration and for researchers aiming to understand the best ways to encourage data driven decision-making skills and knowledge in pre-service teachers.

Urban, C., Carruthers, C. K., Dougherty, S., Goldring, T., Kreisman, D., & Theobald, R. (2022). A multi-state analysis of trends in career and technical education: Massachusetts, Michigan, Montana, Tennessee, and Washington. Georgia Policy Labs. https://gpl.gsu.edu/?wpdmdl=2704 [nam10.safelinks.protection.outlook.com] [nam10.safelinks.protection.outlook.com]

This report seeks to understand changes in CTE concentration just before and just after the pandemic began using administrative data across five states: Massachusetts, Michigan, Montana, Tennessee, and Washington. The study explores how and where CTE concentration substantively changed across cohorts and by CTE concentration clusters at the start of the pandemic, including by student characteristics such as gender, race, ethnicity, and for students with identified disabilities. In addition, the study explores CTE concentration by urbanicity where comparisons between rural students and students in urban areas reveal differences in some states. Lastly, high school graduation rates of CTE concentrators in these states are explored before and after the start of the pandemic. 

 

For Montana specific findings visit: https://gpl.gsu.edu/publications/trends-in-cte-in-montana/ 

As a uniquely rural state that requires a minimum of 1 CTE credit for a high school graduation diploma, Montana reveals interesting trends for CTE concentrators. In particular, roughly half of Montana high school students are CTE concentrators where participation is largely of rural, male students across the state. In addition, this study reveals that students who concentrated in CTE were more likely to graduate high school than non-concentrators, especially for students with identified disabilities and those living in rural areas.

Clausen, R. (2023, August). The Use Case of Montana: School Level Poverty Measures. STATS DC Conference, Institute of Education Sciences: Bethesda, MD.

Stoddard, C. & Clausen, R. (2023, August). Montana’s Early Warning System: Processes and Outcomes. STATS DC Conference, Institute of Education Sciences: Bethesda, MD.

Clausen, R & Stoddard, C. (2023, June). Predicting Dropout and Graduation with Reliability: Montana Early Warning System. OPI Summer Institute: Bozeman, MT.

Clausen, R. (2023, May). Analysis of Alternative Poverty Measures Applied to the Case of Montana. Data Foundation: Washington DC.

Clausen, R (2023, April). Montana’s Early Warning System for Dropouts. AERA Conference: Chicago IL.

Carter, B. & Clausen, R. (2023, February). Emerging Montana Research: EWS & SLPM. SLDS Best Practices Institute: Washington DC.

 

Stoddard, C. & Clausen, R. (2022, Aug). Evaluating Montana’s Early Warning System. NCES STATS-DC Conference: Washington, DC.

Sharkey, N.; Clausen, R., Stoddard, C., Uretsky, M., Henneberger, A. (2022, June). Statewide Longitudinal Data Systems and Predictive Analytics: Understanding, Measuring, and Predicting K12 Outcomes. Data Foundation: Washington DC.